Mohit Upadhyay

About Mohit Upadhyay

I have 7+ years of expertise as a Data engineer.My areas of competence include Azure Databricks, Data Factory, Spark structured streaming, Delta Live Tables, Snowflake and dbt, SCD Type 1 and 2 implementation, Power BI(Desktop and Service), Terraform, Azure DevOps supported by SQL and Python with an emphasis on developing robust code for ETL pipelines. I focus on client and growth analytics to support data-driven choices. I have worked in all phases of the software development lifecycle, from gathering requirements, analyzing, buiding, supporting, and deploying applications.I have helped create data solutions for Fortune 100 and 500 clients, showcasing my ability to scale and provide outcomes. In addition, my domain knowledge in the areas of FMCG, Oil & Gas, BFSI and Automotive(OEM) has helped me to tackle a variety of business problems.I possess good excellent analytical capabilities, self-motivation, and good communication in addition to my technical expertise. I work well both alone and in groups, always accomplishing objectives with little guidance. Always strive to advance my leadership abilities and assume positions where I can manage teams, spearhead strategic initiatives, and support game-changing projects.

Experience

1. Data Engineer

   Adastra

   Jul 2025 — Present

   Worked on a real-time manufacturing data platform on Databricks using Medallion Architecture (Bronze → Silver → Gold), processing IoT telemetry and MES streams from 30+ machine types across battery production lines.• Built streaming pipelines using Apache Spark DLT with readStream, watermarking, and RocksDB state store (changelog checkpointing), ingesting from Azure Event Hubs into bronze and transforming to silver/gold with sub-second triggers.• Developed a metadata-driven framework using Python factory functions to generate per-machine telemetry and event gold tables (granular + aggregated), removing repetitive logic across stations.• Engineered NG defect analysis pipelines by parsing nested JSON/VARIANT payloads, extracting NOKBits/PartID/Result, and optimizing dimension joins via broadcast strategies.• Implemented a layered data quality framework: DLT expectations, schema validation across catalogs, and pytest-based checks for PK duplication, nulls, and FK-PK integrity.• Designed real-time parameter monitoring tracking property health (counts, timestamps, hidden fields) with watermark-based incremental processing and Delta MERGE upserts.• Built observability tooling for latency (gateway → EventHub → bronze → silver → gold) and throughput tracking comparing ingestion vs output over rolling windows.• Developed timeseries backfill pipelines moving deduplicated Delta data to PostgreSQL (TimescaleDB) via JDBC for Grafana, including indexing, retention, and anti-join deduplication.• Created CDC-based dimension pipelines, equipment master ingestion, event enrichment, and JSON-driven property mapping across 31 station configs.• Built SFM reporting pipelines for production, shift logs, and safety data with strict schema enforcement.• Enabled environment-agnostic deployments using Terraform-driven Spark configs across QA and production.

Education

• Gujarat Technological University

  Bachelor of Engineering, Mechanical Engineering