Abhishek Upmanyu

About Abhishek Upmanyu

Engineering Manager with deep expertise in building and scaling distributed data platforms using Apache Spark.Currently leading teams working on Spark runtime in Microsoft Fabric and Azure Synapse, focusing on performance, reliability, and cost efficiency of large-scale data workloads.Hands-on experience across Apache Spark, Hadoop (YARN, ABFS), Hive, and large-scale distributed systems. Passionate about performance optimization, benchmarking (TPC-DS, TPC-H), and improving developer productivity using AI-assisted workflows.Skills: Apache Spark, Apache Hadoop, YARN, Hive, Scala, Java, Big Data, Distributed Systems, Performance Optimization, Azure Synapse, Microsoft Fabric

Experience

1. Senior Software Engineer Manager

   Microsoft

   Nov 2023 — Present · Bengaluru, IN

   Leading a team working across multiple components of Apache Spark to improve performance, reliability, and scalability of Spark in Microsoft Fabric and Azure Synapse- Driving initiatives in autoscaling and dynamic resource allocation, reducing infrastructure costs and improving utilization for large-scale customer workloads- Leading adoption of AI-assisted development workflows to enhance engineering productivity, accelerate debugging, and improve delivery velocity- Collaborating with partner teams to improve Spark distribution quality, ensuring reliable releases with strong CI/CD and integration validation- Supporting critical production incidents and guiding teams to improve system resilience and customer experience- Fixing performance gaps in Apache spark offering

Education

• Indian Institute of Technology, Kharagpur

  B.Tech and M.Tech (Dual Degree), Computer Science

  2010 — 2015

Skills

• Data Visualization
• Functional Programming
• Statistics
• C++
• Data Structures
• Html
• Java
• Python
• Data Science
• Netbeans
• R
• Algorithms
• Linux
• C