Chinmay Vyapari
Data Engineer @Quantiphi | 1xGCP | 1xSnowflake | Big Data | Gen AI | Spark
- Role
- Data Engineer at Quantiphi
- Location
- Mumbai, IN
- LinkedIn followers
- 500 followers
About Chinmay Vyapari
Passionate about data. Experienced in big data, ETL development from scratch and pipeline…
Experience
Data Engineer
Jan 2023 — Present · Mumbai, IN
Contributed to the implementation of Medallion architecture for Unified Data Platform (UDP), consolidating datafrom multiple sources within the BFSI domain, ensuring streamlined data integration and transformation.• Designed robust ETL pipelines using PySpark on GCP Dataproc processing 50+ Tera Bytes of data. Implementedbatch processing and parallel execution strategies resulting in 40% reduction in data loads.• Implemented and optimized Apache Iceberg for warehousing within the Silver layer. Collaborated with cross-functionalteams to define data requirements. Orchestrated Composer DAGs to manage live data pipelines and containerized thecode using Docker, streamlining deployment. Utilized Github CI/CD for autmoation.• Engineered and optimized ingestion pipelines from Source to Raw layer with Apache NiFi. Improved pipelineperformance by leveraging distributed processing, achieving a 40% reduction in processing time.• Analyzed and converted Source to Target Transformation Mappings (STTMs) into SQL, conducting rigorous dataquality assurance using BigQuery. Implemented automated checks and balances that reduced data anomalies by 20%,ensuring the integrity and reliability of the analytics output.• Developed two key accelerators – the Synthetic Data Generator and SQLShift. Productionized SQLShift by fine-tuningMeta’s LLAMA-2 model, automating the conversion of SQL queries from RedShift to Snowflake, which reduced manualeffort by 50% for data engineers
Find verified contacts for anyone on LinkedIn
Unifers gives sales teams verified emails and direct dials, enriched profiles, and outreach that lands in the inbox.
Free plan included · No credit card required
This profile is compiled from publicly available professional sources. Unifers is not affiliated with or endorsed by LinkedIn. Request removal of this profile.