Jishan Ahmad

About Jishan Ahmad

I design high-performance data pipelines and lakehouse architectures capable of processing 100TB+ daily, enabling enterprises to make faster, data-driven decisions. With deep expertise in real-time streaming (Kafka, Flink), cloud platforms (Azure, AWS), and modern data stacks (Apache Iceberg, Druid, Spark), I help organizations modernize from legacy systems to scalable, cost-efficient, cloud-native solutions. Key Achievements Improved query performance by 70%(10s → 3s) using Apache Iceberg & PySpark in lakehouse environments. Built real-time pipelines (Kafka, Flink, PyFlink) handling 100TB+ per day—enhancing agility and operational responsiveness. Reduced deployment time by 50% by standardizing ETL with StreamSets Control Hub and Apache Airflow. Led critical migrations from PL/SQL to Hive/Impala and implemented NoSQL solutions (MongoDB, Redis, MinIO). Increased user engagement by 25% through interactive dashboards (Apache Superset, Imply Pivot). What I\'m Passionate About Building real-time and batch data pipelines using technologies like Kafka, Flink, Spark, and Airflow. Automating data workflows using Jenkins, Autosys, and Kubernetes. 🧠 Solving distributed systems and scalability challenges. Designing event-driven data platforms for real-time monitoring and insights. 🤝 Mentoring teams on modern data engineering practices and architecture patterns.

Experience

1. Data Engineer

   IBM

   Oct 2019 — Present · Bengaluru, IN

   Scalable Data Pipeline Development & Modernization Designed and optimized high-volume data pipelines (100TB+ daily) using StreamSets, Kafka, Spark, and Airflow—enhancing reliability and reducing data latency by 30% for enterprise analytics workloads. Automated deployment and orchestration with Kubernetes and OpenShift, improving system scalability and cutting infrastructure costs by 20%. Led the modernization of ETL frameworks with StreamSets Control Hub, reducing deployment times by 50% and enabling real-time and batch processing across 10+ business units. Architected scalable data storage and query solutions using Hive, Impala, HBase, MongoDB, and Druid—boosting query performance by 40% and increasing user engagement by 25% via dynamic dashboards. Built a real-time platform monitoring and event correlation system using Kafka, Solr, and Zookeeper, enabling proactive issue detection and reducing system downtime by 15%.

Education

• Kendriya Vidyalaya

  High school

  2006 — 2007

• GLA University

  Bachelor's degree

  2010 — 2014