Shubham Rai

About Shubham Rai

Data Engineer | Azure | SQL | PySparkData Engineer with over 4 years of experience in building and supporting cloud-based data solutions. Skilled at transforming raw, large-scale datasets into clean, reliable, and structured data systems to support analytics and reporting. Hands-on experience working with Microsoft Azure and strong technical proficiency in SQL and PySpark.Core Skills & Responsibilities:ETL Development:Built and maintained scalable ETL pipelines using Azure Data Factory, Azure Synapse pipelines, and Azure Functions to automate data extraction, transformation, and loading from various sources.Big Data Processing (Azure Databricks):Designed and implemented distributed data processing workflows using Azure Databricks and PySpark to handle large-scale batch and incremental data processing with high performance and reliability.Delta Lake & Medallion Architecture:Implemented Delta Lake on Azure Data Lake Storage using Bronze, Silver, and Gold layers to ensure data reliability, ACID transactions, schema enforcement, and scalable data transformations.Cloud-Based Data Architecture:Proficient in using Azure Synapse Analytics for data warehousing and Azure Data Lake Storage Gen2 / Blob Storage for managing structured and unstructured data. Ensured data pipelines are optimized for performance, cost-efficiency, and reliability.Data Modeling & Integration:Translated business requirements into logical and physical data models. Supported integration of structured and semi-structured data from internal and external sources using Azure-native services.Query Optimization:Wrote and optimized complex SQL queries to efficiently process large volumes of data across Synapse and Databricks environments.Monitoring & Debugging:Implemented data validation and quality checks using Azure Monitor, Log Analytics, and Databricks job monitoring. Performed root-cause analysis and resolved pipeline failures to maintain production system stability.Collaboration & Communication:Worked closely with analysts, data scientists, and business stakeholders to deliver high-quality datasets, reports, and dashboards that enable data-driven decision-making.

Experience

1. Consultant

   Hoonartek

   Mar 2026 — Present

Education

• Nagarjuna Public School, Nanded

  Xth, CBSE

  2001 — 2014

• Kusumtai Jr. College, Nanded

  HSC, 12th

  2014 — 2016

• Mahatma Gandhi Mission's College of Engineering, Nanded

  B.E., Computer Science Engineering

  2016 — 2020