Harish Kumar Challa

About Harish Kumar Challa

9.5 years of IT industry experience in Data Engineering and Software application development.Experienced in ETL activities on Azure Data factory, Azure Databricks, PySpark. and Azure data lakeDeveloped end-to-end DevOps pipelines to facilitate seamless deployment of Azure Data Factory pipelines, Databricks notebooks, SQL stored procedures, and SQL table schemas across DEV, QA, and PROD environments.Experienced in building reliable software applications and highly scalable REST API using python and Django.Experienced in using Agile Methodologies.

Experience

1. Senior Consultant

   Deloitte

   Dec 2023 — Present · IN

   Scalable Event‑Driven Streaming FrameworkDeveloped a fully automated, metadata‑driven streaming framework that accelerates onboarding of event‑based datasets across the organization. Key capabilities include: Automated creation of end‑to‑end streaming pipelines with minimal inputs (table name + Event Hub configurations). Utilizes Spark Structured Streaming for reliable, real‑time ingestion from Azure Event Hub into Delta Lake. Processes ~3 million events per hour per minute) with an architecture designed to scale effortlessly to higher throughput. Robust checkpointing and recovery mechanisms ensuring high reliability, fault tolerance, and exactly‑once processing. Seamless ingestion into Delta Lake Bronze → Silver layers with consistent processing logic.Achieves low‑latency processing, making data available in the Silver layer within ~2 minutes of arrival in Event Hub. Auto‑discovery of new event streams and automatic creation of corresponding Delta tables in ADLS. Built‑in auto schema evolution to handle new or changing fields without manual intervention. Advanced column‑conflict resolution using intelligent hashing to manage duplicate or case‑variant columns and maintain clean schemas at scale.

Education

• Sri Chaitanya College of Education

  12th, Maths, Physics, Chemistry

  2010 — 2012

• Shanmugha Arts, Science, Technology and Research Academy

  Bachelor of Technology - BTech, Electronics and communication engineering

  2012 — 2016