Swapnil Chougule

About Swapnil Chougule

Highly skilled professional with overall 12+ years of experience with a proven track record of architecting, designing and building large-scale, scalable, fault-tolerant & highly-available state-of-the-art data engineering & data science products primarily based on open source technologies (in domains like E-commerce, AdTech, MarTech, IT Data Analytics, Cloud Native Data Offerings, IT Services Automation etc)Blogs: https://swapnil-chougule.medium.com/Github: https://github.com/swapnil-chouguleStackoverflow: https://stackoverflow.com/users/63•••84/swapnil-chougule* Own major business problem or product vision, translate it into executable technical design and roadmaps to solve the problem & realize the product vision* Lead cross team efforts & coordinate across multiple teams with all stakeholders(data engineers, data scientists, tech leads, software engineers, solution engineers, product managers) to deliver high quality results* Vast experience of designing & implementing complex & scalable data processing pipelines & data science platforms on Peta Byte scale Volume.* Production level hands on experience on cutting edge technologies like Hadoop, Spark, Flink, Kafka/Pulsar, Hive, Airflow, Delta, HBase, Cassandra/Scylla, File formats like Avro, Parquet etc.* Built Auto-ML framework for full stack datascience capabilities to realtime recommendations & AI predictions* Built large scale data components across all phases of data - Ingestion, Processing(ETL), Storage, Analyze, Feature engineering, ML Model scoring* Architected & developed large scale realtime stream processing systems (with lambda architecture) with sub-second latency to get actionable insights for business analytics, data science & realtime reporting* Experience of working with complex data problems like fast data problems, skewed data & sub-second latency* Experienced in tuning, optimizing and managing large scale big data production processing ecosystem* Team building experience* Mentor engineers by providing oversight and direction product roadmaps & technical architecture* Experience demonstrating technical concepts including presenting and whiteboarding* Build and prototype optimized pipelines/components iteratively to provide insights at scale.* Adhere to best practices and architecture with deep domain knowledge.* Open source contributor, Tech blogger* Presented PubMatic Streaming Analytics use cases in Apex Bigdata World 2017 Conference, Spark meetup speaker* Always curious, quick-learner, self-motivated & self-starter with a \'can do attitude\'

Experience

1. Staff Data Engineer

   Linkedin

   Oct 2023 — Present · Bengaluru, IN

   Staff Data Engineer (Data Science): Leading (tech leadership) & building large-scale data platforms and AI-driven analytics systems powering multiple LinkedIn business and customer-facing products.• GTM Data: Building an AI-native unified data platform for LinkedIn’s Go-To-Market ecosystem including sales intelligence and customer success, introducing AI-powered capabilities across data engineering workflows such as pipeline development, automated data quality & cataloging, metric/dimension consolidation, semantic layer via dbt, and analytics features like text-to-SQL, anomaly detection, and reasoning-driven insights.• LinkedIn Business Platform (LBP): Established the foundational medallion-based data platform architecture powering LinkedIn’s business ecosystem including Premium, Online Jobs, executive reporting, and product analytics, enabling scalable pipelines and standardized datasets for downstream business teams.• LinkedIn Talent Insights: Revamped the data platform for LinkedIn’s workforce analytics product built on the Economic Graph, standardizing entities such as members, jobs, employers, industries, and education, modernizing the stack with Spark, Airflow, Iceberg, and Pinot, and enabling AI-driven insights consumption (through MCP server) across products like Premium, Sales Navigator, Jobs, and GTM systems.• Axia (Recruiter Reporting): Re-architected the data foundations for LinkedIn’s ATS-integrated recruiter analytics platform combining ATS and LinkedIn signals, redesigning Spark pipelines and optimizing Pinot ingestion/query layers to significantly improve reliability, latency, and accuracy for recruiter funnel and job performance analytics.

Education

• Jawahar Navodaya Vidyalaya - JNV

  AISSCE (CBSE Board), +2 (Science)

• Indian School of Business

  Executive Execution, Product Management

  2022 — 2022

• Shivaji University

  Bachelor of Engineering - BE, Computer Science & Engg.

  2006 — 2010

• Harvard Online

  Technology Entrepreneurship: Lab to Market

Skills

• Scala
• J2ee Application Development
• Apache Kafka
• Spark
• Xml
• C++
• Modeling
• Hadoop
• Servlets
• Flink Streaming
• Jquery
• Jsp
• Apache Spark
• Sql
• Linux
• Spark Streaming
• Java Enterprise Edition
• Java
• Pl/Sql
• Amazon Web Services (Aws)
• Core Java
• Big Data
• Struts
• Mysql
• C
• Javascript
• Html 5