Vansh Gupta

About Vansh Gupta

Currently thriving as an Associate Data Engineer at Tredence, where I’m passionate about leveraging data to drive innovation and business growth.Proficient in Python, SQL, and PySpark, with hands-on experience in building robust data pipelines and implementing advanced analytics solutions. I specialize in utilizing Azure Databricks to optimize data processing and extract valuable insights.Eager to contribute my skills and knowledge to cutting-edge projects in the field of data engineering and analytics. Let’s connect and explore opportunities to collaborate on exciting data-driven initiatives!

Experience

1. Data Engineer

   Tredence Inc.

   Jun 2025 — Present · Gurugram, IN

   Phase 1: Unified Data Model (UDM)•Worked in the Healthcare & Pharma domain for a US-based pharmaceutical client, actively participating in client discussions and translating business requirements into scalable data engineering solutions.•Designed and implemented an end-to-end Databricks Lakehouse architecture, ingesting data from 8+ heterogeneous sources including Meta, Reddit, Google Ads, and other marketing platforms.•Built the Silver Conformed Layer using PySpark, implementing SCD Type 1 for current-state updates and SCD Type 2 for historical tracking, along with JSON flattening and schema normalization.•Developed a Gold Unified Data Model (UDM) consisting of dimension and fact tables to enable KPI reporting and dashboarding, and created business-friendly views for downstream BI consumption.•Implemented a master orchestration notebook using ThreadPoolExecutor to run Silver pipelines in parallel, significantly improving pipeline efficiency and reducing execution time.•Automated daily refreshes using Databricks Workflows and followed a Git-based development lifecycle using Databricks Repos and GitHub.Phase 2: Metadata-Driven Ingestion Framework•Core member of a 3-person team responsible for designing and building a fully metadata-driven ingestion framework.•Enabled ingestion from S3, Azure Blob Storage, SFTP, and SharePoint into Databricks Volumes and Bronze Unity Catalog tables.•Implemented JDBC-based ingestion from SQL Server and Oracle directly into Delta tables.•Designed and implemented a comprehensive audit logging and reconciliation framework to track pipeline status, row counts, schema details, and execution timestamps, ensuring accurate source-to-target validation.•Led Komodo data ingestion using Delta Sharing into Unity Catalog, implementing complex business rules and transformations.

Education

• Apeejay School

  12th

• UPES

  Bachelor of Technology - B.Tech (Hons), Computer Science (AIML)

• Apeejay School

  10th