Diti Divekar
Data Engineer @ Think360.ai | AWS | GCP | Pyspark | Celery | K8s | Microservices
- Role
- Data Engineer at Think360.ai
- Location
- Thane, MH, IN
- LinkedIn followers
- 500 followers
About Diti Divekar
Data Engineer, with an experience of over 2 years in developing ETL pipelines end to end…
Experience
Data Engineer
Aug 2024 — Present · Mumbai, IN
Written ETL scripts in PySpark and used AWS EMR to seamlessly transform up to 100 Gb worth of client data into product-specific formats for scrubs and POCs.• Migrated the engine to GCP cloud infrastructure using RabbitMQ as a queuing mechanism.• Analysed engine logs from EFK stack to identify time consuming bottlenecks and fix them with appropriate celery worker pools reducing workflow latency by 70%• Optimized Celery Worker Configurations and available infrastructure of K8s pods to reduce cost by 66%• Examined different traffic patterns by stress testing the system reducing the TAT to 7 seconds.• Established a logging infrastructure for the processing engine layer, enabling real-time monitoring and improved decision-making.• Optimized the existing code and conducted unit testing to ensure quality assurance standards were met prior to releasing updates or new features.• Used AWS Athena to extract Gbs of output data from S3 bucket and sent out the consolidated data to data analysts for further analysis.• Constructed CI pipelines using Bitbucket Pipelines for repositories of all microservices being used in the product Algo360.
Education
KJ Somaiya College of Engineering, Vidyavihar
BTech, Information Technology
2020 — 2024
Smt Sulochanadevi Singhania School
ISCE + ISC, PCM+Computer Science
2006 — 2020
Find verified contacts for anyone on LinkedIn
Unifers gives sales teams verified emails and direct dials, enriched profiles, and outreach that lands in the inbox.
Free plan included · No credit card required
This profile is compiled from publicly available professional sources. Unifers is not affiliated with or endorsed by LinkedIn. Request removal of this profile.