Soumya Gupta
Data Engineer at HCLTech | Python | SQL | PySpark | Pandas | Airflow | Trino | GCP | Bigquery | Cloud Composer | Databricks | Delta Lake | Azure Data Factory
- Role
- Senior Technical Lead - Data Engineering at HCLTech
- Location
- Bengaluru, KA, IN
- LinkedIn followers
- 500 followers
About Soumya Gupta
I have total 7 years 9 months of IT experience out of which 6+ years is in Data Engineering and Big Data related roles. Currently I am working with Australia and New Zealand Banking Group. As a Data Engineer, the following are my responsibilities-1. Engineered PySpark pipelines on Databricks to process 6M+ customer data on a daily basis, enabling compliance analysis teams to identify anomalies thereby reducing compliance gap identification time by 30%.2. Designed and orchestrated end-to-end data pipelines in Azure Data Factory (ADF) to ingest multi-source financial datasets (CRM, payments, loans) into Delta Lake, ensuring pipeline reliability and supporting downstream reporting for credit risk and portfolio analysis.3.Wrote L0 to L1, L1 to L2 transformations using HQL and perform storage and query level optimizations in Hive wherever possible.4. Developed optimized SQL code in Starburst (Trino) to build data assets for Customer Performance Dashboard.5. Authored highly configurable Airflow pipeline to schedule DBT jobs in GCP Cloud Composer with robust DQ checks.6. Minimized processing cost in BigQuery by replacing repetitive queries with materialized views.7. Made use of Partition Pruning, Column Projection and Data Shuffle minimization in order to improve runtime of existing queries up to 35%.8. Automated Broker Enhancement Report using Python, SQL and Airflow. This automation resulted in a 25-30% reduction in report generation time.9. Made use of Github Actions for running pipeline validations and Terraform (basic level) for deploying cloud resources in GCP.I have received an individual award namely Inspiring Performance and a team award, namely \"League Extraordinaire\" during my tenure in Wipro and also awarded twice in the \"Deliver What Matters\" category in ANZ. I am eager to learn new technologies and try to be aware of the emerging technologies in Data Engineering and Machine Learning fields.I also have a youtube channel where I try my best to share my knowledge with others. The link of the channel is:
Experience
Senior Technical Lead - Data Engineering
Dec 2025 — Present · Bengaluru, IN
Education
Govt. College Of Engineering and Ceramic Technology
Engineer’s Degree, Ceramic Sciences and Engineering
2013 — 2017
Find verified contacts for anyone on LinkedIn
Unifers gives sales teams verified emails and direct dials, enriched profiles, and outreach that lands in the inbox.
Free plan included · No credit card required
This profile is compiled from publicly available professional sources. Unifers is not affiliated with or endorsed by LinkedIn. Request removal of this profile.