Gopal Ji Gupta

About Gopal Ji Gupta

Senior Data Engineer with 7+ years of experience in architecting, designing, and delivering end-to-end data platform solutions on Microsoft Azure Cloud for the retail, banking, consumer goods and energy sectors. Proven expertise in building modern data architectures, data pipelines, and real-time streaming solutions using Azure Data Factory, Azure Databricks, Delta Lake, and Azure Data Lake Storage. Adept at optimizing data processing, enabling AI-driven insights, and ensuring performance, scalability, and governance across enterprise-grade solutions.Core Responsibilities & Achievements:Architected and delivered cloud-native data platforms leveraging Azure services (ADF, Databricks, ADLS Gen2, Key Vault, Azure SQL DB, and Blob Storage).Designed and implemented scalable batch and streaming data pipelines for enterprise data warehouse, eCommerce, and customer data platform (CDP) solutions.Developed and maintained Delta Lake architecture, enabling incremental data loads, schema evolution, ACID transactions, and time travel queries.Built structured streaming workflows to process and aggregate real-time data from diverse source systems into 20+ curated Delta tables.Designed data ingestion frameworks to handle multiple file formats (CSV, JSON, Parquet, Avro, XML) from on-prem, SaaS, and cloud-based systems.Implemented CI/CD pipelines using Azure DevOps, ensuring automated deployments and release management for data pipelines and ML models.Performed data transformation and enrichment using PySpark, Spark SQL, and SQL to optimize analytics-ready datasets.Optimized complex queries and Spark jobs, improving pipeline performance through partitioning, caching, Z-ordering, indexing, and broadcast joins.Scheduled and orchestrated workflows using ADF triggers (schedule, tumbling window, and event-based) and Apache Airflow.Collaborated with cross-functional business and technical teams, mentored junior engineers, and resolved complex issues with strong problem-solving and interpersonal skills.Technical Skills:Azure Services: Azure Databricks, Azure Data Factory (ADF), Azure Data Lake Storage (Gen2), Azure Blob Storage, Azure SQL Database, Azure Key Vault, Azure DevOpsBig Data & Processing: PySpark, Spark SQL, Delta Lake, Structured StreamingHadoop Ecosystem: Hadoop, HDFS, HiveProgramming & Scripting: Python, SQLScheduling & Orchestration: Azure ADF Triggers, Apache AirflowData Engineering Expertise: Data ingestion, ETL/ELT, schema evolution, data quality, performance tuning, indexing strategies

Experience

1. Senior System Analyst

   IBM

   Apr 2022 — Present · Bengaluru, IN

   Product development on SAP HANA ERS HANA and Athena datasets for use in intermediate tables and new data pipelines in Azure Data FactoryWorking as a developer in architecture and implementation of ETL and product solutions using Azure Databricks notebooks Azure Data Factory Azure SQL and Power BIDeveloping Azure Databricks notebooks in PySpark and Python to extract transform and load data into SQL and ADLS as part of migration activitiesCreating reusable Databricks notebooks using GPC and GSPA datasets for custom product developmentBuilding product tables using reusable datasets from SAP HANA Snowflake and Azure SQLDesigning scalable data pipelines in Azure Data Factory for different modules within QM HubBuilding and optimizing Databricks notebooks for ETL processing using PySparkManaging staging and intermediate layer pipelines across the Azure ecosystemImplementing schema evolution and handling incremental loads in Bronze Silver and Gold layer architectureOptimizing Spark jobs with partitioning caching and broadcast joins for performance improvementIntegrating data from multiple sources including SAP HANA Snowflake Azure SQL and REST APIsAutomating pipeline orchestration with triggers parameters and dependency chains in ADFCollaborating with cross functional teams to deliver high quality data solutions aligned with business requirementsEnsuring data quality validation and reconciliation across different stages of the pipelineSupporting migration of legacy systems to modern Azure cloud based architectureContributing to best practices documentation and reusable frameworks for Databricks and ADF development

Education

• School of Electronics , DAVV Indore

  M.Sc, Electronics & Communication

  2009 — 2011

• Bundel khand University

  B.Sc, Mathematics, Physics

  2006

Skills

• Sql
• App Store Optimization
• Linux Kernel
• Linux
• Google Adwords Professional
• Android
• Hdfs
• Software Project Management
• Css
• Google Analytics
• Brand Management
• Scala
• Yarn
• Embedded Systems
• Embedded C
• Apache Spark
• Microcontrollers
• Javascript
• Web Applications
• Web Development
• C++
• Html5
• Web Project Management
• Sqoop
• Web Analytics
• Hive
• Electronics
• Java
• Tableau
• Hadoop
• Mobile Applications
• Apache Pig
• Flume
• Google Adwords
• Software Development
• Hbase
• Html
• Mapreduce