Shivam Patel
Data Engineer | Databricks | Microsoft Azure l PySpark | Sql l Python
- Role
- Data Engineer (Specialist Programmer) at Infosys
- Location
- Bengaluru, KA, IN
- LinkedIn followers
- 500 followers
About Shivam Patel
Data Engineer with a strong background in designing, developing, and maintaining data solutions for data generation,collection, and processing. Expertise in creating data pipelines, ensuring data quality, and implementing ETL processes tosupport large-scale data systems. Proven ability to collaborate with cross-functional teams to design optimized datasolutions and infrastructure. Skilled in troubleshooting, mentoring junior engineers, and staying up-to-date with the latestdata engineering trends and technologies.
Experience
Data Engineer (Specialist Programmer)
Jul 2024 — Present · Bengaluru, IN
Received Insta Award (Feb 2026) for resolving critical pipeline failures in E.ON Next project and optimizing Databricks-based data workflows, improving reliability and performance.Led successful delivery of a complex data engineering project within timelines and budget.Designed and built scalable ETL pipelines across multiple systems, improving data accessibility for business users.Developed Databricks pipelines to flatten and process nested JSON data from API endpoints.Optimized Spark jobs using effective partitioning strategies, reducing pipeline runtime by ~50%.Tuned complex SQL queries, cutting execution time from 1+ hour to under 2 minutes, eliminating timeout issues.Built reusable UDFs for data extraction from APIs, Excel, and JSON sources.Enabled rapid business POCs by generating optimized Parquet datasets for augmented analytics.Implemented robust data validation and cleansing checks to ensure data quality and integrity.Created AWS Lambda functions to ingest raw data into S3 for Bronze-layer processing.Collaborated with cross-functional teams to design reliable, scalable data solutions using Python and SQL.
Education
Galgotias University
Bachelor of Technology - BTech, Computer Science
2020 — 2024
Central Board of Secondary Education
12th, PCM
2019 — 2020
Find verified contacts for anyone on LinkedIn
Unifers gives sales teams verified emails and direct dials, enriched profiles, and outreach that lands in the inbox.
Free plan included · No credit card required
This profile is compiled from publicly available professional sources. Unifers is not affiliated with or endorsed by LinkedIn. Request removal of this profile.