Shubham Katole

About Shubham Katole

Senior Data Engineer with 8+ years of Proficient in Azure Data Factory, Azure Databricks, and Advanced Data Solutions. Skilled in ETL Processes, Data Warehousing, Data Integration, Transformation, and Analytics on Azure Cloud Platform.Technical Expertise : Azure Data Factory, Azure Databricks, Azure DataLake, Azure Data Explore, KQL, SQL, Python, Spark, ETL etc.Explored and analyzed the existing Databricks PySpark codebase, SQL scripts, reusable functions, ADF pipelines, and Logic App triggers to understand the data flow, transformation logic, and integration architecture.Took complete ownership of the Stock module, working independently to enhance its functionality by reusing existing logic and developing new components using PySpark in Databricks.Designed and implemented Delta tables, SQL functions, and stored procedures to ingest and process data from external APIs, enabling seamless integration into the data ecosystem.Developed end-to-end data integration workflows using PySpark to process SAP payloads, transform data into the SFDC schema, and load it into Azure SQL tables and Salesforce, ensuring data accuracy and consistency.Built reusable PySpark functions for various Amcor data objects to handle token generation, Salesforce API calls, chunking large datasets, and integrated error handling via ADF failure logging and alert emails.Led efforts to review and enhance CI/CD pipelines for ADF, Azure Databricks, and SQL, improving deployment speed, reliability, and consistency across DEV, UAT, and PROD environments.Created and reviewed multiple pull requests to ensure smooth code promotion while maintaining coding standards and deployment readiness. Participated in SIT/UAT calls, collaborating with QA and client teams to resolve issues and validate releases.Received appreciation to team for delivering high-quality work within timelines, showcasing strong collaboration, ownership, and commitment to delivery excellence,

Experience

1. Associate Manager

   EY

   Oct 2024 — Present · Pune District, IN

   Explored and analyzed the existing Databricks PySpark codebase, SQL scripts, reusable functions, ADF pipelines, and Logic App triggers to understand the data flow, transformation logic, and integration architecture.Took complete ownership of the Stock module, working independently to enhance its functionality by reusing existing logic and developing new components using PySpark in Databricks.Designed and implemented Delta tables, SQL functions, and stored procedures to ingest and process data from external APIs, enabling seamless integration into the data ecosystem.Developed end-to-end data integration workflows using PySpark to process SAP payloads, transform data into the SFDC schema, and load it into Azure SQL tables and Salesforce, ensuring data accuracy and consistency.Built reusable PySpark functions for various Amcor data objects to handle token generation, Salesforce API calls, chunking large datasets, and integrated error handling via ADF failure logging and alert emails.Led efforts to review and enhance CI/CD pipelines for ADF, Azure Databricks, and SQL, improving deployment speed, reliability, and consistency across DEV, UAT, and PROD environments.Created and reviewed multiple pull requests to ensure smooth code promotion while maintaining coding standards and deployment readiness. Participated in SIT/UAT calls, collaborating with QA and client teams to resolve issues and validate releases.Received appreciation to team for delivering high-quality work within timelines, showcasing strong collaboration, ownership, and commitment to delivery excellence.

Education

• Datta Meghe Institute of Engineering Technology and Research

  Bachelor of Engineering, computer science

  2013 — 2017