Abhishek Ranjan

About Abhishek Ranjan

Professional Summary: Data Engineer with over 3 years of experience designing scalable ETL pipelines for big data and data warehousing, serving global clients like PepsiCo and Centrica.Core Expertise: Proficient in Azure Data Factory, Azure Databricks, PySpark, and SQL; specialize in data migration, ingestion, and transformation using cloud-native solutions.Data Orchestration Skills: Build fault-tolerant pipelines with medallion architecture (bronze, silver, gold layers) to optimize data processing and performance.Additional Proficiencies: Experienced in data validation, automation, and monitoring using Azure Monitor and Azure SQL Database to ensure data integrity for analytics solutions.Key Achievements:Reduced processing times by 25% through PySpark optimizations.Achieved 99% pipeline reliability via automation implementations.Career Goal: Drive innovative cloud-native solutions and data orchestration to deliver high-impact analytics in dynamic organizations.Technical Toolbox: Languages & Frameworks: Python, PySpark, SQL, Apache Spark,Hive. Cloud Platforms: Azure Data Factory, Azure Databricks, Azure Synapse Analytics, Azure Storage. Integration & Tools: REST APIs, Ms SQL Sever. Specialties: ETL Pipelines, Data Modeling, Data Ingestion, Data Validation, Data Transformation, Data Migration. Practices: Agile, Git/GitHub, CI/CD, Documentation, Peer Training. Extra: Basic understanding of Fabrics.

Experience

1. Data Engineer

   Cognizant

   Feb 2022 — Present · Noida, IN

   Pepsi-Co MDIP Data Migration• Served as an Azure Data Engineer to migrate data from Teradata to Azure Data Lake Storage Gen2, ensuring 100%data accuracy and zero data loss.• Developed end-to-end ETL pipelines in Azure Data Factory, transforming data with PySpark on Azure Databricks, andloading it into Azure Data Lake Storage Gen2, meeting client performance requirements.• Configured schedule and event-based triggers in Azure Data Factory for automated pipeline execution, reducingmanual intervention by 80%.• Implemented Azure Monitor for real-time pipeline tracking, and used Azure SQL Database with watermark columns forincremental load management.• Enhanced PySpark script performance through query optimisation and parallel processing, reducing migrationprocessing time by 20%Centrica- MSM Ingestion • Designed and developed data ingestion pipelines using Azure Data Factory to extract data from APIs, Salesforce, SFTP, S3 loading it into Azure Data Lake Storage Gen2 for e-commerce analytics, following medallion architecture.• Wrote and optimised PySpark scripts on Azure Databricks for data transformation, including deduplication, schemastandardisation, and cleaning, improving data quality for downstream analytics• Automated hourly and incremental data pipelines using Azure Data Factory schedule and event-based triggers,implementing retry logic and error handling, achieving 99% pipeline reliability.• Optimised PySpark scripts by implementing partitioning and caching, reducing data processing time by 25% for largedatasets.

Education

• ABES Engineering College

  B.TECH, Computer science engineering

  2018 — 2022