Pawan Kumar Singh

About Pawan Kumar Singh

Senior Azure Data Engineer with 7+ years of experience designing and delivering enterprise-scale data platforms on Azure Databricks, Microsoft Fabric, and Azure Data Factory. What I Build:End-to-end Lakehouse architectures (Bronze → Silver → Gold) on Azure Databricks and Microsoft Fabric - handling terabytes of structured and unstructured data daily with high reliability, fault tolerance, and cost efficiency. Core Expertise- Azure Databricks | Delta Lake | Delta Live Tables | Auto Loader- Microsoft Fabric | OneLake | Data Pipeline | Lakehouse- Azure Data Factory | Synapse Analytics | ADLS Gen2- PySpark | Python | SQL | Unity Catalog | MLflow- Medallion Architecture | SCD Type 1 & 2 | CDC Pipelines- AI-ready data platforms | RAG pipeline enablement Key Wins- Reduced pipeline runtime by 30% through Delta Lake optimization- Cut downstream data errors by 95% with automated quality checks- Reduced manual processing time by 80% through report automation- Reduced false positives by 50% in fraud detection models Certifications: DP-700 — Microsoft Fabric Data Engineer Associate DP-600 — Microsoft Fabric Analytics Engineer Associate DP-900 — Azure Data Fundamentals Databricks Generative AI Fundamentals DP-750 — Azure Databricks DE Associate (In Progress) Based in Gurugram | Open to Senior DE, Lead DE, Data Architect roles — India & Global Remote

Experience

1. Senior Data Engineer

   HCLTech

   May 2025 — Present · Lucknow, IN

   Architected and deployed end-to-end Medallion Architecture (Bronze/Silver/Gold) on Azure Databricks processing 2TB+ daily structured and unstructured data - achieving 99.9% pipelineuptime with fault-tolerant Auto Loader ingestion- Optimized Delta Lake architecture using OPTIMIZE, Z-Ordering, and Auto Compaction - reducing pipeline runtime by 30% and cutting compute costs by 25%- Implemented Unity Catalog governance framework covering data lineage, row-level security, and column masking across 50+ Delta tables - achieving full compliance with enterprise data standards- Built AI-ready data platform enabling downstream RAG and GenAI workloads - structured metadata-complete Bronze/Silver/Gold layers with schema enforcement via Delta Live Tables expectations- Led cross-functional collaboration with data scientists, BI teams, and business stakeholders translating requirements into scalable data models, reducing requirement-to-delivery cycle by 20%

Education

• Dr. Ram Manohar Lohia Avadh University (RMLAU), Faizabad (Ayodhya)

  Bachelor's degree, Art/Art Studies, General