Shivam Indani

About Shivam Indani

Early in my career, I discovered that data is far more than just numbers stored in…

Experience

1. Software Developer

   Impetus

   Oct 2022 — Present · Gurugram, IN

   Scaled Kafka pipelines from 150K to 600K messages/min by tuning batching logic and partition handling and addressed growing data volume without adding infrastructure.• Refactored MongoDB interactions by replacing Python-side filtering with aggregation pipelines, reducing query time from 60s to 4s and unblocking downstream processing.• Led part of the transition from MongoDB to ClickHouse to support faster analytics — improved performance by ~40% while cutting storage and infra costs.• Processed over 50M telemetry records to support ML use cases — structured and validated data for model readiness and automated test data creation.• Built ingestion pipelines for multiple OEMs, handling raw vehicle data with custom cleaning, filtering, and standardization logic to ensure compatibility across systems.• Used AWS Glue, Lambda, S3, and SQS to design scalable, event-driven workflows for batch and near real-time data processing.• Improved filtering and memory efficiency in existing pipelines, cutting processing time by 40% with better resource utilization.• Ran POCs for Airflow, Snowflake, and PySpark to evaluate trade-offs and guide tooling decisions for future architecture needs.• Debugged and resolved critical production issues — including data loss, delayed messages, and infra bottlenecks in streaming jobs.• Collaborated directly with clients to translate business logic into technical workflows, speeding up feedback cycles and cutting iteration time by 25%.