Vishwaant Kannaiyan

About Vishwaant Kannaiyan

I operate at the intersection of Data Engineering and Solutions Architecture, combining over 19 years of hands-on distributed systems development with high-level strategic design.Throughout my career, I have carved out a specific niche: I don\'t just build pipelines, I build the systems that build the pipelines.My obsession with developer velocity started early. I spent years architecting custom parsers and metadata-driven code generators to automate boilerplate ETL development, effectively solving the repetitive parts of data engineering before it was industry standard. Today, that passion has evolved into the GenAI frontier. I am now focused on building Autonomous Agents and leveraging LLMs to not only speed up development cycles but to fundamentally rethink operational excellence and observability.I function as a hands-on architect bridging the gap between legacy enterprise frameworks and modern cloud-native ecosystems. I write the code,optimize the execution plans, and design the self-healing architectures that allow organizations to scale from terabytes to petabytes with minimal operational overhead.I am looking to bring this blend of deep distributed systems expertise and next-gen automation to a team tackling the industry\'s hardest data challenges.Core Expertise:Automation & GenAI: Evolution from Regex/Parser-based code generation to LLM-based Autonomous Agents for workflow acceleration.Engineering & Compute: Advanced Spark (Optimization & Tuning), Python, SQL, Distributed Computing, and High-Throughput ETL.Architectural Design: Enterprise Data Lake/Lakehouse patterns, Cloud Migrations, and Self-Healing System Design.Orchestration: Designing metadata-driven frameworks (Airflow, Luigi) for dynamic pipeline generation.

Experience

1. Principal Data Engineer

   Shutterfly

   May 2021 — Present · Charlotte, NC, US

   Leading the architectural modernization of Shutterfly’s data ecosystem. I focus on bridging the gap between legacy reliability and cloud-native velocity.Petabyte-Scale Clickstream & Attribution: Architected the Unified Marketing Attribution (UMA/MTA) engine using raw Adobe Clickstream data. Engineered complex sessionization logic over billions of daily events to stitch user journeys across devices, unlocking granular marketing ROI insights and replacing legacy \"last-touch\" models.GenAI & Autonomous Agents: Spearheading the shift from static automation to probabilistic AI. Architecting LLM-driven agents to automate data observability, root cause analysis, and documentation, reducing developer onboarding time and operational toil.Cloud-Native Architecture: Designing and scaling self-healing ETL frameworks on AWS. Led the migration of critical workloads from legacy on-prem systems to a resilient Data Lakehouse architecture, ensuring 99.9% availability during peak holiday traffic.System Optimization (FinOps): Championing \"FinOps\" for data engineering by optimizing Spark execution plans and cluster sizing. Reduced Databricks runtime costs by ~40% while improving SLA adherence.Force Multiplier: Built a metadata-driven ingestion framework that auto-generates pipeline code, effectively removing the need for manual boilerplate and standardizing quality across the engineering org.

Education

• SRM IST Chennai

  B.Tech, Computer Science

  2003 — 2007