Souparno Mukhopadhyay
Advisory Data Engineer @IBM
Signup · Get unlimited contacts
WORK HISTORY
Advisory Data Engineer @IBM
Kolkata, IN
Project1Real-Time Data Streaming (Spark & Kafka) – POCs & Production Readiness1. Designed and developed production-grade Kafka producers and consumers for real-time data ingestion, including stress testing, benchmarking, and performance tuning.2. Delivered POCs and production-ready streaming pipelines using Apache Spark and Kafka, ensuring scalability, fault tolerance, and low latency.3. Conducted show-and-tell sessions with stakeholders to demonstrate pipeline performance, key KPIs, throughput, and latency metrics.4. Built monad-based functional Scala scripts to create a highly configurable, function-driven codebase with a low learning curve for development teams.5. Developed plug-and-play custom transformation JARs, enabling dynamic, runtime transformations without code redeployment.Project2Azure Data Engineer – Enterprise Batch Data Pipelines1. Designed and developed end-to-end batch ETL pipelines using Azure Data Factory (ADF), Azure Data Lake Storage Gen2 (ADLS Gen2), Azure Databricks, and Azure SQL / Synapse (where applicable).2. Built scalable and fault-tolerant ingestion frameworks to process large-volume datasets from multiple upstream systems into curated analytical layers.3. Implemented Databricks-based transformation pipelines using Spark, focusing on performance optimization, partitioning strategies, and efficient resource utilization.4. Provisioned and managed Azure infrastructure using Terraform, enabling infrastructure-as-code (IaC), repeatability, and environment consistency.5. Developed parameterized and metadata-driven ADF pipelines, reducing manual effort and accelerating onboarding of new data sources.6. Delivered globally consumed data pipelines used across the client organization to power analytics, reporting, and business insights.7. Collaborated with cross-functional teams (business, analytics, platform, and security) to align data solutions with enterprise standards and compliance requirements.
EDUCATION
Visvesvaraya Technological University
Bachelor of Engineering (B.E.), E.C.E
ABOUT SOUPARNO MUKHOPADHYAY
With 10+ years of experience in Fullstack Development, Data Engineering and devops I excel at building and scaling products from concept to maturity. I’ve driven impactful solutions handling millions of transactions and petabytes of data, delivering seamless operations for enterprise clients.As a founding engineer in the past, I helped grow a startup from a 10-member team to 100+ in 3 years, taking on diverse roles—from Backend and Platform Engineer to Integration and Customer Engineer—while leading cross-pod teams and navigating complex technical challenges.My journey includes designing scalable data pipelines, enabling digital transformation, and driving 15+ successful integrations with various open source and cloud platforms. I\'ve also managed teams of 10-15 members, fostering collaboration and delivering results through strong leadership and a focus on creating positive team culture.From discovery to MVP to enterprise-scale deployments, I\'ve consistently aligned technical execution with product strategy, ensuring measurable success. I thrive in dynamic environments, combining innovation, technical excellence, and leadership to deliver impactful outcomes.Let’s connect and explore opportunities to create and scale extraordinary solutions!
This profile is compiled from publicly available professional sources. Unifers is not affiliated with or endorsed by LinkedIn. Request removal of this profile.