Sagar Nalawade

About Sagar Nalawade

Data Engineer with 3 years of experience building scalable ETL and Big Data pipelines processing 600–800M records per cycle.Expertise in Apache Spark (Java & PySpark), SQL optimization, and distributed data processing. Hands-on experience with GCP and AWS, along with practical exposure to Airflow-based orchestration and Kafka-based batch data pipelines.Currently working on migrating legacy orchestration systems to Apache Airflow, focusing on building scalable, fault-tolerant, and automated data workflows.Strong in performance tuning, partitioning strategies, and building metadata-driven data pipelines. Proven ability to reduce manual effort, improve data latency, and ensure reliable large-scale data processing.Open to Data Engineer / Big Data Engineer opportunities.

Experience

1. Software Engineer

   SMARTe

   Nov 2023 — Present · Navi Mumbai, IN

   Worked as a Data Engineer building and maintaining large-scale Big Data ETL pipelines in a production environment.Key Contributions:• Engineered a distributed ETL ecosystem with 25+ components processing 600–800M records per cycle.• Designed and implemented a daily batch data pipeline (~20M records/day), reducing data latency from months to under 48 hours.• Leading migration of legacy orchestration framework to Apache Airflow by developing DAGs and modular workflows.• Automated Dataproc cluster lifecycle (creation, autoscaling, teardown), improving resource utilization and reducing cloud costs.• Built fault-tolerant pipelines using checkpointing and idempotent design to ensure reliable processing and recovery.• Developed Spark SQL-based data quality and anomaly detection framework to prevent pipeline failures.• Optimized Spark jobs using partitioning, broadcast joins, and query tuning, improving execution performance.• Implemented Kafka-based batch ingestion using producer-consumer architecture.• Reduced manual operational effort by 70–80% through end-to-end automation and monitoring improvements.• Acted as escalation engineer for RCA, debugging, and performance tuning; mentored junior engineers.

Education

• University of Mumbai

  Bachelor of Engineering - BE, Mechanical Engineering

• SunBeam Institute of Information Technology, Pune-Karad

  Postgraduate diploma, Cdac