Sai Manikanta

Sr Data Engineer @Visa

TX, US

Worked with project team for developing and maintained the ETL (extract, transform, and load) process in AWS.· Wrote Hive queries for data analysis to meet the business requirements and created and managed Hive tables based on the requirement.· Running SQL scripts in AWS RDS, creating indexes, stored procedures for data analysis.· Experience working with Hadoop 2.x version and Spark 2.x (Python and Scala).· Designed various ETL strategies from various heterogeneous sources (such as flat files, excel, access database, and SQL server) utilizing AWS.· Cleaned data using Spark jobs to make it suitable for ingestion into Hive tables for further analysis.· Utilized Hive tables and HQL queries to generate daily and weekly reports. Worked on complex data types in Hive like Structs and Maps.· Developed Spark scripts using Python & Scala as per the client requirements and used spark for transfor-mations.· Used Spark SQL to load Parquet data, created a dataset characterized by case classes, and used Spark SQL to work on structured information. This was eventually stored in a Hive table for downstream use. A scheduled ETL run that runs twice a day.· Used Spark-SQL, Scala on Spark engine to develop end to end ETL pipeline.· Involved in code/design analysis, strategy development and project planning.· Worked on different data formats such as JSON, XML.· Imported all gathered data from different sources into Spark RDD for further data transformations and analysis.· Better working Knowledge in Spark Core, Spark SQL, Spark Streaming.

Harrisburg University of Science and Technology

Master's degree, Information systems and engineering management