Pavan Kumar Reddy

About Pavan Kumar Reddy

Data Engineer with 4+ years of experience designing, building, and optimizing large-scale data pipelines, analytics platforms, and cloud-native solutions across AWS and Azure ecosystems. Proven track record of delivering cost-efficient, high-performance data lakes, ETL frameworks, and streaming architectures supporting business intelligence, machine learning, and real-time analytics. Skilled in Python, SQL, Spark, Airflow, and distributed systems, with hands-on expertise in DevOps automation (CI/CD) and data security best practices (RBAC, PII handling). Adept at collaborating across cross-functional teams to translate business needs into scalable data models and solutions, consistently accelerating project delivery and data driven decision-making.

Experience

1. Data Engineer

   Netflix

   Oct 2023 — Present · CA, US

   Engineered and optimized Petabyte-scale data lakes in AWS S3, AWS Glue, and Amazon EMR, applying partitioning, Z-ordering, and Athena query optimization to improve analytics performance by 35% and cut costs by 40%.• Developed Python-based data frameworks using PySpark, Pandas, and NumPy to automate ETL processes and feature engineering pipelines, reducing manual processing time by 50% while improving data quality for analytics and ML applications.• Led development of auto-scaling Spark clusters using Kubernetes (K8s) and Amazon EMR, reducing infrastructure costs by 25% while maintaining 99.9% system availability (SLA).• Optimized complex SQL queries across Redshift, Athena, and Spark SQL, rewriting joins, indexing strategies, and query plans to reduce runtime by 40% and improve resource efficiency.• Implemented role-based access control (RBAC), encryption, and automated audit logging in Apache Airflow DAGs, increasing security and reducing data pipeline vulnerabilities by 60% when processing personally identifiable information (PII).• Tuned distributed query performance across AWS Athena and Redshift Spectrum by conducting query plan analysis and using cost-based optimization, reducing query runtimes by 45% and lowering cloud costs by 30%.• Collaborated cross-functionally with ML engineers to productionize models, DevOps teams to implement CI/CD pipelines, and data analysts to optimize queries, bridging technical gaps between teams to accelerate project delivery by 30%.

Education

• Sathyabama Institute of Science & Technology, Chennai

  Bachelor of Engineering - BE

  2016 — 2020

• Marist College

  Master of Science - MS

  2022 — 2024