Vasavi M

Senior Data Engineer @Charles Schwab

Performed data analysis and developed analytic solutions, Data investigation to discover correlations / trends and the ability to explain them.• Worked with Data Engineers, Data Architects, to define back-end requirements for data products (aggregations, materialized views, tables – visualization)• Designing and Developing Oracle PL/SQL and Shell Scripts, Data Import/Export, Data Conversions and Data Cleansing. • Developing Spark applications using Scala and Spark-SQL for data extraction, transformation, and aggregation from multiple file formats. Using Kafka and integrating with the Spark Streaming. • Authoring Python (PySpark) Scripts for custom UDF’s for Row/ Column manipulations, merges, aggregations, stacking, data labeling and for all Cleaning and conforming tasks. • Migrate data from on-premises to AWS storage buckets.• Developed Mappings using Transformations like Expression, Filter, Joiner and Lookups for better data messaging and to migrate clean and consistent data• Designed and implemented Sqoop for the incremental job to read data from DB2 and load to hive tables and connected to Tableau for generating interactive reports using Hive server2.• Used Sqoop to channel data from different sources of HDFS and RDBMS.• Developed Spark applications using Pyspark and Spark-SQL for data extraction, transformation, and aggregation from multiple file formats.• Hands on working experience working with Databricks platform.• Crated Databricks delta tables and worked on performance tuning of the jobs.• Hands on experience working with snowflake Database.• Managed Roles and Grants on snowflake database.• Imported the data from snowflake to s3 and vice versa.• Developed stored procedures/views in Snowflake and use in Talend for loading Dimensions and Facts.• Used Spark Streaming to receive real time data from the Kafka and store the stream data to HDFS using Python and NoSQL databases such as HBase and Dynamo DB.