Hitesh Modi
Data Engineer - II at Omnicom Annalect | AWS | Python | SQL | Pyspark | Airflow | Starburst | EX - IPG
- Role
- Senior Data Engineer at Annalect India
- Location
- Mumbai, MH, IN
- LinkedIn followers
- 500 followers
About Hitesh Modi
AI & Data Engineer with 4.3+ years delivering scalable GenAI and data solutions for…
Experience
Senior Data Engineer
Feb 2024 — Present · Mumbai, IN
Designed and developed scalable end-to-end data pipelines for ingesting over 10 million rows of JSONL data daily using Apache Airbyte, ensuring efficient extraction from REST APIs and seamless integration into AWS Glue for processing and ingestion into Iceberg Data Lakes. Employed dynamic partitioning and schema evolution to support evolving data structures.• Built and optimized data ingestion frameworks with a focus on data quality validation, implementing mechanisms for failure notifications, empty file alerts, unexpected column detection, and schema validation to maintain data integrity and reliability across all stages.• Architected robust schemas and data models tailored to multiple platforms, ensuring efficient data storage, retrieval, and downstream analytics while minimizing latency and optimizing query performance on Iceberg tables.• Developed secure SFTP ingestion pipelines integrated with state management and repair mechanisms, supporting ingestion of 10+ client-specific reports without requiring code modifications, ensuring operational scalability and flexibility.• Implemented robust data validation systems to monitor data ingestion, preventing null values, missing dates, and other inconsistencies, ensuring downstream processes receive clean, reliable datasets.• Collaborated with cross-functional teams and stakeholders, utilizing written and verbal communication skills for requirement gathering, quality assessment, and continuous improvement in pipeline performance.• Engineered Spark jobs for Iceberg table optimization, including tasks such as merging small partitions, snapshot removal, and metadata rewriting, resulting in improved query speeds and reduced resource consumption for datasets exceeding 100 GB across 12 platforms.
Education
University of Mumbai
BMS
JAI HIND COLLEGE, MUMBAI
Hsc
Find verified contacts for anyone on LinkedIn
Unifers gives sales teams verified emails and direct dials, enriched profiles, and outreach that lands in the inbox.
Free plan included · No credit card required
This profile is compiled from publicly available professional sources. Unifers is not affiliated with or endorsed by LinkedIn. Request removal of this profile.