Karthik Shivanna

About Karthik Shivanna

I design and scale enterprise-grade data platforms that power business growth.My work spans distributed systems, large-scale data engineering, and cloud-native solutions, combining hands-on architecture with strategic leadership. Over the years, I’ve led complex initiatives across data warehouses, Hadoop ecosystems, and modern cloud platforms, driving development, automations, performance tuning, cost optimisation and operations at scale.From architecting distributed platforms to guiding teams that deliver enterprise deployments of Airflow, Spark, and other big-data technologies, I lead with a focus on scalability, operational excellence, and continuous improvement.Core strengths:• Distributed data platforms and MPP systems• Orchestration and workflow automation at scale• Cloud-native infrastructure and cost optimisation• Innovation through vendor partnerships and open-source contributionsI thrive at the intersection of technical depth and leadership, helping organizations build data platforms that enable innovation, operational excellence, and confident growth.Technical Capabilities include the following:Cloud Platforms- Hands-on: Google Cloud Platform (GCP) – Dataproc, Compute Engine, IAM, Monitoring and Alerting, Kubernetes- General Awareness: AzureData Platforms & Big Data- Teradata, Netezza, Vertica, Cassandra, Greenplum, Neo4j, Hadoop, Spark- Hadoop expertise: cluster build, upgrade, management, Ambari API automation, monitoring, troubleshooting, HDFS, custom alerting strategies- Big data workloads on KubernetesDatabases- MySQL, PostgreSQL, H2Reporting & Visualisation- MicroStrategy, Business Objects, Tableau, Grafana, LookerProgramming & Scripting- Shell, Perl, Python, SQL, JavaCI/CD & Orchestration- CI/CD: Looper (Jenkins backed), Concord- Orchestration: Airflow (Committer since Sept 2025), Automic, CA7Teradata Specialties- BTEQ, Fastload, Multiload, NPARC, TD Datamover, workload management, Viewpoint, query tuning

Experience

1. Principal Engineer

   Walmart Global Tech India

   May 2023 — Present · Bengaluru, IN

   Leading modernisation of Walmart’s cloud-native data platforms, including but not restricted to BigQuery, Spark, Airflow, Kubernetes- Driving strategic initiatives that blend technical execution with architectural leadership across distributed platforms- Guiding multiple stealth projects focused on next-generation distributed systems- Responsibilities include platform architecture, performance tuning, cost efficiency, and ensuring compliance at enterprise scale- Evaluating and piloting emerging technologies to shape Walmart’s long-term data platform strategy- Led Several Cost Optimisation initiatives, leading to cost savings of ~2.5 million dollars- Driving automations to leverage auto-scaling based on rule-based engines, to save cost (Automated down-scalers)- Work with product vendors to drive the alignment of Walmart\'s roadmap with that of the product vendors- As a personal choice, started contributing to open-source Airflow.

Education

• Cloudera University

  Apache Hadoop Administration, Hadoop

  2015 — 2015

• Visvesvaraya Technological University

  B.E, Computer Science

• Swami Vivekananda Yoga Anusandhana Samsthana

  Master of science , Yoga

Skills

• Sql Tuning
• Netvault
• Database Administration
• Unix
• Shell Scripting
• Teradata Data Warehouse
• Sql
• Apache Spark
• Databases
• Etl
• Rdbms
• Performance Tuning
• Perl
• Business Objects
• Bteq
• Unix Shell Scripting
• Business Intelligence
• Database Design
• Spark
• Data Warehouse Architecture
• Microstrategy
• Javascript
• Hadoop
• Netezza
• Software Development
• Python
• Data Warehousing
• Hive
• Teradata