Lavanya R

About Lavanya R

Senior Data Engineer with over 11 years of experience designing and building scalable data engineering solutions, ETL/ELT pipelines, and enterprise data platforms in AWS cloud environments. Extensive experience developing high-performance data pipelines utilizing Python, Spark, and PySpark to serve large-scale batch and real-time data processing workloads. Practical experience implementing Databricks-based data engineering solutions, such as data ingestion, transformation, and data lake architecture for analytics and enterprise data platforms. Extensive experience with data migration and integration projects, including moving enterprise data systems to current cloud-based platforms like Databricks and Snowflake. Proficient in developing AWS-based data pipelines with S3, Glue, EMR, Lambda, Redshift, and Databricks, resulting in scalable and dependable data ingestion frameworks. Extensive experience with Spark and PySpark development for large-scale distributed data processing and performance optimization. Extensive hands-on experience in designing, implementing, and optimizing ETL workflows to extract, transform, and load data into data lakes and data warehouses. Proven experience managing end-to-end ETL/ELT processes with current frameworks (Airflow, DBT, AWS Glue, ADF) to ensure high performance, data quality, and production reliability. Experience working with full-stack data platforms, including backend data pipelines and frontend integration via Angular/REST APIs, and CI/CD deployments. Proven experience developing ETL pipelines that integrate CRM platforms like Salesforce with cloud data platforms, enabling downstream analytics and reporting systems. Extensive experience developing durable, production-grade data pipelines with Spark and PySpark, providing high data quality, fault tolerance, and optimal performance for large-scale analytics workloads. Demonstrated ability to work with cross-functional engineering teams to convert business needs into scalable data engineering solutions, resulting in dependable pipelines that support enterprise CRM and advertising analytics platforms. Experience creating CI/CD pipelines with GitHub Actions, Jenkins, and DevOps principles to automate data pipeline deployment and testing. Strong experience with data warehousing, dimensional modeling (Star/Snowflake), data transfer, and corporate data integration. Worked in Agile/Scrum environments, collaborating with cross-functional engineering teams to build scalable data engineering solutions.

Experience

1. Sr Gcp Data Engineer

   Shift4

   Apr 2024 — Present · MD, US

   Designed, developed, and optimized BigQuery data warehouses, ensuring high performance through query tuning,partitioning, and clustering for cost efficiency. Implemented IAM role-based access control, encryption, and audit logging to safeguard data and ensurecompliance. Provided production and on-call support for GCP-based data applications built with Dataproc, Cloud Composer, GKE,and BigQuery. Designed and built Java-based Apache Beam (Cloud Dataflow) pipelines for streaming and batch data processing onGCP. Created Kafka/PubSub to Dataflow and BigQuery/GCS pipelines for real-time and near-real-time ingestion. Dataflow workloads were deployed and managed using GKE, Docker, and Cloud Build to enable regulated CI/CD andenvironment promotion. Implemented GitHub Actions and Cloud Build pipelines to automate infrastructure provisioning and servicedeployments. Created and monitored CI/CD pipelines using Terraform and Cloud Build for automated deployment. Created technical specifications and high-level architectural papers for new data pipelines and backend ingestion services. Utilized reporting tools like Tableau to connect with Big Query for generating daily reports of data. Managed workflow orchestration using Cloud Composer (Airflow) to ensure smooth scheduling and execution ofpipelines.

Education

• Teegala Krishna Reddy Engineering College (JNTUH)

  Bachelor's degree, Computer Science

  2010 — 2014