Siva Garlapati

Data Engineer @Tata Consultancy Services

Bengaluru, IN

Developed Data lake on Hadoop on-premise cluster and created Hive tables to store raw, staging and transformed data.Worked on developing PySpark scripts to process and transform large size of raw data from Hive tables and load it to refine tables/databases on Hadoop on-premise data lake.Optimized and improved performance of PySpark ETL pipelines that process huge volumes of data by handling data skewness using spark optimization techniques like Salting, broadcast join and cache.Optimized and improved Hive queries performance to ensure hadoop cluster stability.Implemented ETL pipeline jobs like extracting data from refined hive tables using necessary queries and loaded result data Snowflake using Spark.Worked with PySpark to implement different ingestions like full refresh, incremental and SCD.Implemented data quality checks to ensure data accuracy and consistency.Experience with Agile process and worked with Scrum methodology.