Pankaj Lal

About Pankaj Lal

Detail-oriented and results-driven Senior Big Data Engineer with 5+ years of experience in building scalable data platforms, cloud-native applications, and enterprise-grade analytics solutions. A 4x Microsoft Azure Certified professional with deep expertise in Apache PySpark, Azure Databricks, Power BI, and end-to-end data pipeline development. Passionate about solving complex business problems through distributed computing, MLOps, and cloud modernization. Proven track record of leading cloud migration initiatives, optimizing algorithms, and delivering high-impact insights across industries. Technical Highlights 4x Microsoft Azure Certified – Expertise in Azure Data Factory, Azure Databricks, Azure DevOps, and cloud-native architecture.🧠 Algorithm Optimization & Refactoring – Refined core algorithms and migrated legacy Spark apps to Azure Databricks, improving performance and scalability. Cloud Migration Leadership – Successfully migrated 10+ applications from Cloudera to Azure, eliminating legacy footprints and reducing licensing costs. Advanced Reporting & Visualization – Delivered 50+ KPIs and interactive dashboards using Power BI and SSRS, improving reporting accuracy by 40%. ETL & Data Engineering – Automated data pipelines using SSIS, PySpark, and Airflow, reducing manual effort by 45% and accelerating delivery cycles. Python & SQL Expertise – Developed complex SQL queries, stored procedures, and Python scripts for data validation, transformation, and orchestration. Hive to Spark Conversion – Modernized legacy Hive workflows into Spark-based solutions for improved compatibility and performance. Award-Winning Solutions – Delivered 10+ high-impact solutions recognized by clients and internal teams for innovation and business value.🧠 5 HackerRank Ratings – Top-rated in Python and problem-solving, showcasing strong analytical and coding skills. Cross-Platform Proficiency – Comfortable working across MacOS, Ubuntu, RHEL 7, and Windows environments.

Experience

1. Senior Data Engineer - Niq Activate

   NielsenIQ

   Aug 2025 — Present · Pune, IN

   I specialize in designing high-performance data infrastructure using PySpark, Databricks, and Azure, while integrating MLOps best practices to streamline model deployment and performance.Reduced data pipeline execution time by 40% using advanced caching in Delta Lake and Z-order clustering. Optimized model training time by 15+ hours/week via MLOps pipelines built with MLflow and Azure ML SDK. Orchestrated complex batch & streaming workflows using Apache Airflow, Docker, and Kubernetes. Designed robust, cloud-native storage layers with Azure Data Lake, SingleStore, and PostgreSQL for multi-TB scale analytics. Built reusable CI/CD pipelines on GitHub Actions to deploy code and models with versioning and rollback support. 🤖 Contributed to AI Proof-of-Concepts involving transformer models, semantic search, and feature store integration.

Education

• Sagar Group of Institutions - SISTec

  Bachelor of Technology - BTech, Computer Science

  2017 — 2021

• Birla Institute of Technology and Science, Pilani

  Master of Technology - MTech, Computer Software Engineering