Ankit Yadav
Data Engineer @Alten India | ETL, AWS Glue, Lambda, EMR, S3, Kinesis | Python | SQL | Spark
- Role
- Data Engineer at ALTEN India
- Location
- Gurugram, HR, IN
- LinkedIn followers
- 500 followers
About Ankit Yadav
Aws data Engineer with 3.10 years of working experience in Big Data technologies working experience with AWS services such as S3, EMR, Glue, Lambda,SNS, EventBridge and building Big data pipelines using Airflow.Written spark sql queries to transform data for further downstream processing. Build end to end pipeline using Airflow DAG to send data to new system. developed Automation scripts to automate each month end\'s manual process reuired for each data pipeline. Worked on optimization of spark jobs. Worked on pyspark to load s3 files in dataframe and to analyze the data.
Experience
Data Engineer
Aug 2022 — Present
1.Worked on Batch processing system, Designed, developed, and maintained 15+ data pipelines processing over 200 GB of data.2.Migrated Contour SQL code to pyspark code improved the computation time by 80%.3.Created AWS Glue ETL job which gets triggered through a Lambda function whenever a new file gets uploaded on s3 bucket.4.Designed an alert system using AWS EventBridge, Lambda, SNS which sends email if the glue job fails and the data is not uploaded to s3 before schedule time.5.Experience in building and optimizing data pipelines, architectures.6.Rearchitect the existing big data pipeline improved the run time by 3 hrs daily.7.Implemented various pyspark optimisation techniques to improve processing time by 40% for complex pipelines.8.Build dashboard using slate tool for monitoring the data pipelines reduce daily manual effort by 2 hrs and address the data quality issues reducing anomalies by 20%.
Education
MIT Academy of Engineering, Alandi, Pune
Bachelor of Engineering - BE
2014 — 2018
Find verified contacts for anyone on LinkedIn
Unifers gives sales teams verified emails and direct dials, enriched profiles, and outreach that lands in the inbox.
Free plan included · No credit card required
This profile is compiled from publicly available professional sources. Unifers is not affiliated with or endorsed by LinkedIn. Request removal of this profile.