Nishant Kumar

About Nishant Kumar

I don\'t just build pipelines. I build platforms that make pipelines unnecessary.At IBM, I architect event-driven, metadata-driven data infrastructure on AWS — where entire teams onboard new datasets through configuration alone, with zero custom pipeline code. What used to take weeks now takes under an hour.Here\'s what that looks like in practice:→ Designed a fault-tolerant ingestion platform handling production-scale financial data across 50+ daily feeds — using Spark, Kafka, PySpark, Glue, Lambda, Step Functions, and DynamoDB→ Built an automated backfill recovery system that replays months of missed data without a single manual intervention or change request→ Implemented PII tokenization and masking for regulated financial data — maintaining full analytical usability while staying compliance-ready→ Cut AWS infrastructure costs significantly through S3 lifecycle and KMS restructuring→ Reduced pipeline manual intervention by 90% through observability controls — stuck-job detection, late data handling, schema mismatch mitigationMy stack: Python · PySpark · Apache Spark · Kafka · Airflow · AWS (Glue, S3, Lambda, Step Functions, Athena, DynamoDB) · Databricks · Snowflake · DBT · Delta Lake · SQLBefore IBM, I spent 2+ years at Cognizant leading DB2 and mainframe → AWS cloud migrations, building Kafka streaming pipelines, and improving Spark ETL performance on large-scale financial datasets.I\'m also actively building at the intersection of Data Engineering and GenAI — designing RAG pipelines, integrating LLM APIs into data workflows, and working with vector databases (FAISS, Pinecone) for semantic search and data enrichment use cases.Certified: Databricks Certified Data Engineer Associate AWS & IBM Professional FoundationsOutside of engineering — I\'ve built a 110K+ Data Engineering community on LinkedIn. Not to teach basics, but to share what actually works at production scale. 650+ engineers mentored. Real problems, real solutions. Open to Senior Data Engineer roles at product-based companies. n••••••••@gmail.com

Experience

1. Data Engineer

   IBM

   Aug 2024 — Present · Bengaluru, IN

   Building production-grade, event-driven data infrastructure for regulated financial analytics on AWS.→ Architected a metadata-driven self-service data platform where teams onboard datasets via config — eliminating custom pipeline development and reducing time-to-production from weeks to under 1 hour (Spark, Kafka, Glue, Lambda, Step Functions, DynamoDB, EventBridge)→ Engineered batch and streaming ETL pipelines with Parquet output, business-date partitioning, incremental processing, and automated Glue catalog registration — ensuring immediate Athena queryability for financial analytics→ Built a declarative data curation framework enabling analytics teams to generate production-ready tables via configuration — with dependency-aware execution preventing partial outputs at scale→ Designed and delivered an automated backfill recovery system (Lambda + EventBridge + DynamoDB) — recovering months of missed production data with zero manual intervention, triggered via a single touch-file→ Implemented PII tokenization and masking for regulated financial data — preserving full analytical usability while meeting compliance requirements→ Cut AWS infrastructure costs through S3 lifecycle optimization and KMS restructuring — significant reduction in encryption costs with no impact on security posture→ Built end-to-end pipeline observability: stuck-job detection, schema mismatch mitigation, late data handling — reducing manual operational intervention by 90%

Education

• Vel Tech Rangarajan Dr. Sagunthala R & D Institute of Science & Technology, Chennai

  Bachelor of Technology - B-Tech, Computer Science

  2018 — 2022