Atanu Sarkhel

About Atanu Sarkhel

I am a Senior Data Engineer with around 8 years of hands on experience designing and building scalable data platforms using modern lakehouse architectures. My focus is on developing reliable, high-performance data pipelines and enterprise data platforms that enable organizations to leverage large-scale data for analytics and decision-making.Currently, I work extensively with Microsoft Fabric Lakehouse, developing end-to-end data engineering solutions using Apache Spark (PySpark) for distributed data processing and transformation of high-volume datasets.One of my key contributions has been building a custom Change Data Capture (CDC) framework to capture incremental changes from source systems and designing a scalable ingestion framework to load data efficiently into Fabric OneLake. This framework supports reliable and automated ingestion of large datasets into the lakehouse environment.I have implemented Medallion architecture (Bronze, Silver, Gold layers) and developed a data quality framework on the Silver layer to ensure standardized, validated, and analytics-ready datasets for downstream consumption.From a data modeling perspective, I have designed enterprise data warehousing solutions including SCD Type 2, fact tables, dimension tables, snowflake schema, and conformed dimensions to support analytical and reporting use cases.I have also implemented CI/CD and Git-based deployment workflows to manage version control and automate deployments across development and production environments, ensuring reliable and maintainable data pipeline releases.I am particularly interested in modern data platforms built on lakehouse architecture and am currently expanding my expertise in Databricks to deepen my capabilities in Spark-based data engineering.I enjoy building scalable frameworks, solving complex data challenges, and designing data platforms that enable organizations to become truly data-driven.

Experience

1. Senior Data Engineer

   Tech Mahindra

   Nov 2025 — Present · Kolkata, IN

   Designed and implemented enterprise-scale Lakehouse architecture using Microsoft Fabric, enabling scalable data ingestion, processing, and analytics across business domains.Developed large-scale distributed data processing pipelines using Apache Spark (PySpark) for transforming and processing high-volume datasets->Built a custom Change Data Capture (CDC) tool to capture incremental changes from source systems and developed a scalable ingestion framework to utilize the CDC mechanism for efficiently ingesting data into Fabric OneLake->Implemented Medallion architecture (Bronze, Silver, Gold layers) in Fabric Lakehouses to standardize raw ingestion, data cleansing, and curated analytics-ready datasets->Developed data quality and validation framework on the Silver layer, implementing automated checks, schema validation, and data consistency rules to ensure reliable downstream datasets->Built scalable transformation pipelines using PySpark and Fabric pipelines to process raw data into curated and business-ready data models-> Implemented data warehousing solutions including Slowly Changing Dimension (SCD Type 2), dimension tables, fact tables, Snowflake schema, conformed dimensions, and factless fact tables to support enterprise analytics and reporting->Orchestrated and automated workflows using Apache Airflow for pipeline scheduling, dependency management, and monitoring.Worked with cloud data warehouse platforms such as Snowflake and Google BigQuery for analytical workloads, SQL transformations, and query optimization->Used Git-based version control and CI/CD practices to manage deployments across development and production environments while collaborating with architects, analysts, and business stakeholders to deliver scalable data solutions.

Education

• West Bengal University of Technology, Kolkata

  Bachelor of Technology - BTech, Computer Science