Shivansh Gupta
Data Engineer @ Accenture | Azure Databricks | Apache Spark | Delta Lake | DLT | Building Scalable Data Pipelines (10M+ rows/day)
- Role
- Data Engineer at Accenture in India
- Location
- Gurugram, HR, IN
- LinkedIn followers
- 500 followers
About Shivansh Gupta
Data Engineer at Accenture\'s Data & AI practice, building production-grade cloud Data…
Experience
Data Engineer
Aug 2024 — Present · Gurugram, IN
Working as a Data Engineer in Accenture’s Data & AI team, building scalable, cloud-based Data Lakehouse pipelines on Azure Databricks using Apache Spark, Delta Lake, and PySpark. Designed and implemented streaming ingestion pipelines with Auto Loader to continuously load raw CSV/JSON data into Bronze Delta tables with checkpointing and schema evolution. Developed parameterized PySpark workflows to automate Bronze → Silver transformations (null handling, deduplication, joins, schema normalization) processing 10M+ records daily. Built curated Gold Delta tables using Delta Live Tables (DLT) with data-quality expectations (@dlt.expect_all_or_drop) and incremental refresh. Optimized Spark jobs using Adaptive Query Execution (AQE), broadcast joins, and caching, improving runtime by ~60%. Automated full Medallion pipeline orchestration via Databricks Workflows and integrated daily triggers for incremental runs. Developed an API-based Facebook scraping service using RapidAPI, enabling structured data extraction for social media analytics. Implemented token authentication, pagination, and rate-limit handling for reliable daily ingestion. Normalized and stored nested JSON data into structured formats on Azure Data Lake Storage (ADLS) for downstream processing. Integrated asynchronous API calls and caching to boost data-fetch throughput by 35%. Automated pipeline runs using Azure Functions, enabling near real-time refresh and alerting. Delivered reusable, modular Python scripts that integrate seamlessly with analytics pipelines.🧠 Tech Stack : Azure Databricks | PySpark | Apache Spark | Delta Lake | DLT | ADLS | ADF | Python | SQL
Education
CHANDIGARH UNIVERSITY
Bachelor of Engineering
2020 — 2024
Find verified contacts for anyone on LinkedIn
Unifers gives sales teams verified emails and direct dials, enriched profiles, and outreach that lands in the inbox.
Free plan included · No credit card required
This profile is compiled from publicly available professional sources. Unifers is not affiliated with or endorsed by LinkedIn. Request removal of this profile.