Varun R

Experience

1. Sr Data Engineer

   Vanguard

   Jul 2020 — Present · Malvern, PA, US

   Migrated the data from Oracle DB to AWS S3 using Spark, Python and EMR. Automated the data quality checks. Converted complex Oracle SQL to Spark SQL\\\\n Worked with Data Engineers, Data Architects, to define back-end requirements for data products (aggregations, materialized views, tables – visualization)\\\\n Developing Spark applications using Scala and Spark-SQL for data extraction, transformation, and aggregation from multiple file formats. Using Kafka and integrating with the Spark Streaming. Developed data analysis tools using SQL and Python code.\\\\n Used terraform for cluster creation.\\\\n Developed Spark applications using Pyspark and Spark-SQL for data extraction, transformation and aggregation from multiple file formats.\\\\n Used Spark Streaming to receive real time data from the Kafka and store the stream data to S3 using Python.\\\\n Collected data using Spark Streaming from AWS S3 bucket in near-real-time and performs necessary Transformations and Aggregation on the fly to build the common learner data model.\\\\n Worked on Dimensional and Relational Data Modeling using Star and Snowflake Schemas, OLTP/OLAP system, Conceptual, Logical and Physical data modeling using Erwin.\\\\n Developed complex Talend ETL jobs to migrate the data from flat files to database. Pulled files from mainframe into Talend execution server using multiple ftp components.\\\\n Developed stored procedures/views in Snowflake for loading Dimensions and Facts.\\\\n \\\\n\\\\n \\\\n \\\\n\\\\n \\\\n