Aditya Singh

About Aditya Singh

I am a Big Data Engineer with extensive experience architecting and maintaining large-scale distributed systems within the BFSI and Credit Analytics sectors.Currently, as an Engineer at American Express, my focus is on the end-to-end ownership of complex ETL pipelines. I specialize in optimizing Spark applications to ensure high availability and strict SLA compliance for enterprise-level reporting and ML workloads. By leveraging advanced features like Adaptive Query Execution (AQE) and fine-tuning resource allocation, I focus on building cost-efficient and resilient data infrastructure on GCP.My career has been defined by a deep understanding of the intersection between financial domain logic and high-performance engineering. From migrating legacy data workflows to modern distributed frameworks to building robust data governance and validation tools, I aim to provide downstream teams with high-fidelity, analytics-ready data.Core Technical Expertise:Frameworks: Apache Spark (Core/SQL), Hadoop, Hive, Kafka.Cloud: GCP (Dataproc, BigQuery, Dataflow).Orchestration: Airflow, Autosys.Languages: Java, Python, Scala, SQL.I enjoy solving \"hard\" data problems involving data skew, memory management, and cloud cost-optimization. Let’s connect to discuss the future of distributed computing and data engineering in the financial world.

Experience

1. Engineer

   American Express

   Jul 2024 — Present

   Own and optimize Spark ETL pipelines processing TB-scale datasets, achieving ~85% runtime reduction through partition tuning, join optimization, and executor memory tuning.• Resolved production issues related to disk spill, data skew, and resource contention, improving SLA adherence for downstream analytics and ML workloads while increasing cluster efficiency.• Designed and maintained data lake pipelines on GCP Dataproc, BigQuery, and Dataflow supporting batch and near-real-time ingestion.• Delivered trusted, analytics-ready datasets enabling ML-driven personalization, reporting, and decision-making across business teams.

Education

• Ganga

  High School Diploma

• Feroze Gandhi Institute Of Engineering & Technology

  Bachelor of Technology, Computational Science

• Dr. A.P.J. Abdul Kalam Technical University

  Bachelor of Technology - BTech, Computer Science

  2015 — 2019