Pavan Alli

About Pavan Alli

Data Engineer | Data Enthusiast With 5.3+ years of experience in Data Engineering, I specialize in building scalable, high-performance data pipelines and end-to-end data solutions. My work focuses on transforming large-scale data into structured, reliable systems that enable analytics and business decision-making. What I Do: • Design and develop ETL pipelines using PySpark, Python, and SQL • Build and optimize data solutions on Databricks, AWS, and Azure • Implement Medallion Architecture (Raw → Cleanse → Curated) • Handle large-scale data processing, migration, and performance tuning • Orchestrate workflows and ensure reliable data delivery across systems Domain Experience: Delivered data solutions for retail clients, working with high-volume datasets and building efficient pipelines to support reporting, analytics, and downstream applications. Tech Stack: PySpark | Python | SQL | Databricks | AWS | Azure | Snowflake | Teradata | Redshift | GitHub | Jenkins (CI/CD)🧩 Previous Experience:Earlier worked as a STIBO PIM Developer for retail client with exposure to master data management and product information management, business rule implementation, enterprise data governance, streamline product data on boarding and enrichment processes. What Drives Me: I enjoy building data pipelines and solutions, optimizing systems, and continuously exploring new technologies in the data engineering and cloud ecosystem. 🤝 Let’s Connect: Open to opportunities, collaborations, and conversations around Data Engineering, Cloud, and building scalable data platforms.

Experience

1. Digital Specialist Engineer L1

   Infosys

   Aug 2024 — Present

   Worked as a Data Engineer on Python, PySpark, SQL, Databricks, Teradata and Snowflake. • Migrated data pipelines from AWS EMR to Databricks implementing Raw–Cleanse–Curate–Solution layers for Finance Sole and automated ingestion into Snowflake as target data warehouse. Implemented Medallion Architecture (Bronze → Silver → Gold layers) in Databricks.• Designed, developed, optimized scalable ETL pipelines using PySpark and SQL from Databricks to Teradata and Databricks to Snowflake. • Implemented Slowly Changing Dimension (SCD) Type 2 logic with effective start/end dates, surrogate keys and active flags to maintain full historical tracking of customer and product dimensions.• Built incremental data pipelines using CDC to capture inserts, updates, and deletes, ensuring efficient and accurate data processing.• Developed scripts in IntelliJ, leveraged GitHub for version control and utilized Jenkins CI/CD pipelines for automated build and deployment.• Developed Automation Scripts for creation of schemas, tables, views and for Data Ingestion from Prod to Dev/QA1/QA2/QA3 environments in Databricks for History Load reducing manual effort and setup time. • Performed data validation, reconciliation, mock validation cycles, prepared validation documentation to ensure data accuracy across systems and monitored production pipelines with timely issue resolution of job failures.• Collaborated with cross-functional teams using Jira for user story tracking, sprint planning, and defect management and maintained technical documentation in Confluence. Delivered user stories in Agile sprints through ensuring timely and quality releases.

Education

• Walchand Institute of Technology, Solapur

  Bachelor of Engineering - BE, Computer Science and Engineering

  2016 — 2020