Sandeep Tiwari

About Sandeep Tiwari

I am a Senior Data Engineer with 13+ years of experience designing, building, and optimizing scalable data platforms across banking and financial services. I specialize in building reliable batch and real-time data pipelines that support analytics, reporting, and business-critical decision-making.Currently working as a Senior Data Engineer (AVP), I have hands-on experience with end-to-end data engineering—from data ingestion and modeling to governance, quality, and performance optimization. I have designed and maintained large-scale ETL pipelines using Spark, PySpark, Kafka, Databricks, and Airflow, processing high-volume data across cloud platforms such as AWS (S3) and Microsoft Azure (ADLS, ADF, Synapse).I bring strong expertise in data architecture, data modeling, and CDC-based ingestion, with a deep focus on data accuracy, reliability, and governance. I have worked closely with cross-functional teams including business analysts, architects, and stakeholders to translate complex business requirements into scalable technical solutions. I am also experienced in CI/CD, containerized deployments using Kubernetes, and production support for mission-critical systems.I enjoy solving complex data problems, improving pipeline performance, and building systems that scale with business growth. I am always keen to work on challenging data engineering problems and contribute to high-impact data initiatives.Core Skills & Technologies:• Spark, PySpark, Spark SQL, Python, SQL• Kafka, Airflow, Databricks, CDC• AWS S3, Azure Data Lake, Azure Data Factory, Synapse• Data Modeling, Data Architecture, Data warehouse and Data Governance• Kubernetes, CI/CD, Jenkins, GitHubCertifications-• Databricks Accredited Lakehouse Fundamental• Azure Certified Microsoft Azure Fundamental: DP-900• Oracle certified associates

Experience

1. Assistant Vice President

   Citi

   Jan 2024 — Present · Maharashtra, IN

   Led the design and implementation of a large-scale data engineering platform to process high-volume batch and real-time data for regulatory reporting and enterprise analytics. Architected end-to-end ingestion pipelines using Kafka for streaming and batch sources, with transformations implemented in Spark / PySpark and data stored in AWS S3 and Azure Data Lake using optimized file formats.Defined data architecture, partitioning strategies, and data models to support downstream analytics and BI workloads. Established CDC-based ingestion, data quality frameworks, and data governance controls to ensure accuracy, lineage, and compliance with banking regulations. Orchestrated workflows using Apache Airflow and implemented CI/CD pipelines using Jenkins and GitHub.Led containerized deployment of Spark jobs on Kubernetes, enabling horizontal scalability, fault tolerance, and high availability. Provided technical leadership by mentoring team members, conducting code reviews, and enforcing best practices for performance tuning and cost optimization. Actively partnered with architects, product owners, and compliance teams to deliver resilient, production-grade data solutions aligned with enterprise standards.Technologies : Python, SQL, Spark SQL, Apache Spark, PySpark, Kafka,Azure Data Lake,Databricks, Apache Airflow, Delta Lake

Education

• IMS ENGG. COLLEGE

  B.TECH, CSE

  2008 — 2012

• Gic Sultanpur

  12th, Mathematics

  2002 — 2006