Rashmitha Reddy

About Rashmitha Reddy

As a Senior Data Engineer with 10 years of experience, I specialize in building and enhancing cloud-based data solutions using Python, Spark, and big data technologies on the AWS, Azure, and GCP platforms. I have experience creating advanced data processing frameworks, managing complicated data workflows, and building scalable ETL pipelines. Leading multiple cloud migration projects, I have used services like AWS Lambda, Azure Synapse Analytics, Azure Data Factory, and GCP Dataflow to improve data accessibility and performance. Additionally, I have optimized Spark programs, created real-time data streaming pipelines using Kafka, and designed data lakes and data warehouses using Snowflake and Azure Synapse. Utilizing cloud services like AWS S3, EC2, Glue, Redshift, and Athena, as well as deploying Change Data Capture (CDC) mechanisms, I have a great deal of experience with database technology for complex data transformations. I am skilled in performance optimization and process automation using CI/CD pipelines. I am also proficient in containerization using Docker and Kubernetes. Focusing heavily on automation, data quality, and performance optimization, I produce reliable, scalable systems that support organizational objectives and offer insightful information. I am skilled in Java, PySpark, SQL, Scala, Python, and several big data technologies.

Experience

1. Sr Data Engineer Data Scientist

   Verizon

   Jan 2022 — Present · NY, US

   Analyzed ETL requirements to identify enhancement opportunities, aligning technical solutions with business objectives for improved data management.• Designed and implemented end-to-end data pipelines and Spark applications using Azure Synapse Analytics, leveraging Synapse notebooks for efficient data transformation and processing.• Built and optimized fault-tolerant data pipelines in Azure Synapse, ensuring reliability and efficiency in complex data environments.• Implemented Change Data Capture (CDC) logic in Azure Synapse to enable efficient incremental data loads and maintain real-time data consistency across systems.• Developed and optimized PySpark and Spark SQL scripts in Synapse notebooks for complex data transformations and advanced analytics, achieving significant performance gains.• Led the development of a comprehensive data warehousing solution in Azure Synapse, centralizing and optimizing data from various sources.• Designed and built complex ETL/ELT pipelines in Azure Data Factory V2 and Synapse Dedicated SQL Pools, handling large-scale data processing and transformations.• Led a full migration project from on-premises systems to Azure, developing an ELT framework in Azure Synapse Analytics to facilitate seamless data transition.• Developed pipelines for efficient data transfer between Azure Blob Storage and Data Lake, including hashed and un-hashed data, optimizing storage solutions.• Created stored procedures and complex queries using PySpark and Spark SQL within Synapse Spark Pools, supporting advanced analytics and efficient data retrieval.