Harsha R.

About Harsha R.

Senior Lead Data Engineer with over 11 years of experience in Big Data, Cloud Data Engineering, and Data Warehousing, leading end-to-end delivery of enterprise data platforms across financial, insurance, and healthcare domains. Proven track record in architecting and managing scalable, high-performance ETL/ELT pipelines and modernizing on-prem to cloud data ecosystems using Azure (ADF, Databricks, Synapse, ADLS, Azure SQL, Cosmos DB) and Snowflake.My expertise includes real-time data processing using Apache Spark, PySpark, Kafka, and Event Hubs, with a strong focus on data migration, transformation, and analytics optimization. I’ve successfully modernized legacy systems and delivered insights across finance, insurance, and healthcare-adjacent domains.I’m also well-versed in DevOps practices, using Azure DevOps, Terraform, GitHub, and Jenkins to build automated CI/CD pipelines. Having worked across multi-cloud environments, I’ve integrated Azure and AWS (S3, RDS, Redshift) to support seamless, production-grade data pipelines. Core Skills: SQL, PySpark, Scala, Snowflake, Databricks, ADF, Synapse, Power BI, Microsoft Fabric, Kafka, Terraform, Airflow, CI/CD Work Types: Open to C2C & C2H | Remote / Hybrid / Onsite (NY/NJ preferred) Let’s connect if you’re hiring for roles in data engineering, analytics, or AI/ML pipelines!

Experience

1. Senior Data Engineer

   Capital One

   Jan 2023 — Present · VA, US

   Designed and implemented scalable data pipelines for batch and streaming workloads using Azure Data Factory, Databricks, and Spark, enabling real-time analytics and cost-effective processing. Led a successful 20TB migration from on-prem to Azure Data Lake, improving query performance by 30% via indexing, partitioning, and materialized views. Developed real-time ingestion pipelines with Kafka, Event Hubs, and Structured Streaming, reducing latency in fraud detection systems by 60%. Created dynamic Power BI dashboards linked to Synapse for executive reporting, reducing decision-making delays. Automated CI/CD for ADF and Databricks pipelines using Azure DevOps, Terraform, and GitHub, cutting deployment time by 50%. Integrated Snowflake and Databricks for hybrid lakehouse architecture with zero-copy cloning and time travel. Secured data infrastructure using Unity Catalog, RBAC, and Key Vault, aligning with compliance standards. Built SCD Type 2 data models and CDC frameworks for incremental data processing with ADF and Snowflake. Designed and implemented data ingestion and transformation pipelines integrating Snowflake, Palantir Foundry, and Azure Data Factory for enterprise data analytics and operational reporting.Collaborated with business and data governance teams to establish Foundry Ontologies and enforce data access policies, lineage tracking, and compliance frameworks. Engineered Foundry Transforms and SQL pipelines using PySpark and Python, supporting downstream analytics and financial reporting requirements. Delivered high-throughput data solutions in financial, insurance, and healthcare-adjacent domains using modern data stacks. Collaborated in Agile teams, supporting cross-functional engineering efforts with strong communication and delivery focus.

Education

• New England College

  Master of Science - MS, Computer Information Systems

• Jawaharlal Nehru Technological University

  Bachelor of Technology - BTech, Computer Science