Abhishek Singh

About Abhishek Singh

Lead Data Engineer with 8+ years of experience owning the design and operation of large-scale, low-latency data platforms that power real-time analytics and data-driven products.I specialize in building high-throughput batch and streaming systems on Azure, leveraging Azure Databricks, Azure Data Factory, Azure Synapse Analytics, and Apache Spark. My expertise spans Kafka, Azure Event Hubs, Delta Lake, Apache Hudi, and Iceberg, with a strong focus on data correctness, fault tolerance, and platform reliability at scale.At UnitedHealth Group (Optum), I owned the end-to-end architecture, implementation, and production operations of 15+ mission-critical data pipelines, consistently delivering 99.9% availability through resilient Spark workloads, idempotent ingestion patterns, and automated recovery strategies. I led a complex Teradata-to-Azure Synapse modernization, migrating analytical SQL workloads and achieving ~30% latency reduction through query refactoring, optimized data distribution, and effective use of materialized views. I also designed and delivered 20+ governed enterprise data products using Delta Lake and Databricks Unity Catalog, enabling secure, scalable self-service analytics across multiple domains.I have architected real-time and CDC-driven data platforms using Kafka and Azure Event Hubs, processing 500K+ events per second with exactly-once semantics enforced via Delta Lake transactional guarantees and MERGE-based upserts. I designed and scaled a 70TB+ Lakehouse across Azure and AWS, delivering sub-30-second analytical queries through optimized partitioning, Z-Ordering, incremental processing, and compaction strategies. By applying advanced Spark tuning, autoscaling, and workload isolation, I reduced compute and infrastructure costs by ~70% while improving throughput and system stability.I thrive in fast-paced, product-driven environments where data freshness, correctness, and developer velocity directly influence user experience and business outcomes. Contact: a••••••••@gmail.com

Experience

1. Technical Lead Data Engineer Azure

   Publicis Sapient

   Mar 2024 — Present · Gurugram, IN

   Client: UnitedHealth Group (Optum) — Gurugram, IndiaOwned the design, delivery, and continuous enhancement of 15+ cloud-native data products and ETL platforms using Azure Data Factory and Azure Databricks, enabling daily, multi-domain analytics via parameterized orchestration, dynamic schema evolution, and environment-aware deployments.Built and scaled a centralized data quality, validation, and observability framework as a shared platform capability in Databricks workflows, embedding rule-driven checks, anomaly detection, SLA monitoring, and automated remediation to improve data product reliability and downstream consumer trust.Designed, launched, and governed 20+ enterprise-grade analytical data products on Delta Lake with Databricks Unity Catalog, delivering ACID guarantees, versioned datasets, schema enforcement, fine-grained access control, lineage, and auditability for UHG and DMA analytics consumers.Led a large-scale analytical platform modernization (Teradata → Azure Synapse SQL), rebuilding and validating SQL-based data products and semantic layers, optimizing distribution strategies, joins, and materialized views to achieve ~30% query latency reduction with zero functional regression.Drove performance, cost, and reliability enhancements across data products, refactoring 100+ legacy SAS workflows into PySpark-based transformations, applying Spark executor tuning, AQE, shuffle optimization, Kafka consumer tuning, and CI/CD-driven releases, and earning Bravo and Diamond Recognition from UHG for exceptional contributions to high-impact data products and platform initiatives.

Education

• Liverpool John Moores University

  Master's degree, Data Science

• St. Xavier's College, Jaipur

  Bachelor's degree , Computer Software Engineering

  2010 — 2013