Ayush Dwivedi
Data Engineer at TCS | Bigdata and Hadoop ecosystem | Apache Spark |Apache Hive |Python | Google Big query | Hbase | SQL.
- Role
- System Engineer at Tata Consultancy Services
- Location
- Gurugram, IN
- LinkedIn followers
- 500 followers
About Ayush Dwivedi
I am a Data Engineer in TCS with 3+years of experience in designing, developing, and…
Experience
System Engineer
Oct 2021 — Present · IN
Client: American ExpressResponsibility:1. ETL Design & Implementation: Designed and implemented ETL workflows to process 1TB+ of data daily fromsources like Gaits and ODL, ensuring data quality and reliability.2. Adhoc and Scheduled Reporting: Processed multiple reports for Adhoc, CM, Merchant, and SE by executingHQL and Python scripts through Event Engine.3. Data Profiling & Validation: Created comprehensive test cases to verify data profiling for MR reports onmultiple cornerstone tables.4. API Debugging & Maintenance: Handled real-time APIs for Credit, Fraud, and Eligibility within the E3environment, troubleshooting and resolving production issues.5. Batch and Real-Time Data Ingestion: Managed both batch and real-time data ingestion workflows for variouscornerstone table use cases using Event Engine (Scheduler).6. Error Recovery & Optimization: Implemented retry mechanisms for failed instances and developed multipleoptimized Spark jobs, reducing data processing failures by 50%.7. Pipeline Optimization: Designed and optimized large-scale data pipelines using PySpark and Apache Hive, reducing data processing time by 40%.
Find verified contacts for anyone on LinkedIn
Unifers gives sales teams verified emails and direct dials, enriched profiles, and outreach that lands in the inbox.
Free plan included · No credit card required
This profile is compiled from publicly available professional sources. Unifers is not affiliated with or endorsed by LinkedIn. Request removal of this profile.