Shubham Singh

About Shubham Singh

Most enterprise data sits in warehouses that only engineers can navigate. I build the systems that change that. I\'m a Data Engineer with 3+ years at General Mills, working at the intersection of cloud data infrastructure and intelligent systems. My stack lives in GCP, Python, SQL, BigQuery, Apache Airflow, DBT, Vertex AI - and increasingly in LLM agents, metadata intelligence, and conversational analytics that make data accessible to everyone, not just the technical few. Some things I\'ve shipped: an AI-powered metadata intelligence platform that brought 400+ enterprise datasets into a searchable, natural-language-queryable system; a conversational analytics agent built on Vertex AI that lets business stakeholders explore data without writing a single line of SQL; a re-architecture of 50+ legacy ETL pipelines that improved throughput by 40% and cut onboarding time by 70%. I\'ve led DBT adoption across 160+ models, driven cloud cost optimization on GCP, and worked on large-scale SAP S/4 HANA migrations across global regions. I\'ve also had the chance to win and finalist at hackathons at General Mills, Nomura, and Meta. Right now I\'m deeply focused on the next frontier: intelligent data platforms where pipelines are observable, metadata is AI-generated, and any analyst can have a conversation with their data warehouse. If you\'re working on hard problems in data engineering, AI infrastructure, or GCP-native platforms - I\'d genuinely love to connect.

Experience

1. Associate Data Engineer

   General Mills

   Jul 2024 — Present · IN

   Led development of an AI-powered metadata intelligence platform integrating DBT lineage pipelines, LLM-generated documentation, and BigQuery - enabling natural language search and conversational discovery across a large-scale enterprise dataset catalog, significantly improving analyst productivity and data accessibility. Built a conversational analytics agent using Vertex AI and semantic search tooling, enabling business stakeholders to explore data, lineage, and documentation through natural language - reducing dependency on engineering teams for routine data discovery queries. Re-architected 50+ legacy ETL workflows into modular, cloud-native Airflow–BigQuery pipelines with a reusable DAG factory pattern - improving data throughput by 40% and reducing new pipeline onboarding time by 70%. Led DBT adoption across 160+ data models, implementing automated schema contract testing with dbt-expectations and establishing data quality standards that improved enterprise analytics reliability. Executed large-scale cloud data migrations from on-premises ERP systems to GCP, ensuring reconciliation accuracy and seamless integration with downstream analytics and reporting workflows across multiple global regions. Drove GCP cost optimization through BigQuery workload management, query performance tuning, and cloud resource right-sizing - reducing infrastructure spend while maintaining pipeline SLAs. Partnered with BI and analytics teams to generate business-ready dataset documentation at scale, advancing data governance practices and enabling self-service analytics adoption across global business units.

Education

• Vivekanand Education Society's Institute Of Technology

  Master of Computer Applications - MCA

  2023

• Thakur Ramnarayan College of Arts and Commerce

  Bachelor of Science - BSc , Computer Science

  2018 — 2021