Sethu Kadiyala

About Sethu Kadiyala

Senior Data Engineer with 7+ years of hands-on experience designing, building, and…

Experience

1. Senior Data Engineer

   Cathay Bank

   Feb 2024 — Present · New York, NY, US

   Designed and implemented real-time data pipelines using Azure Data Factory and Spark Structured Streaming on Databricks to process high-volume financial transactions with low latency.• Migrated critical data assets from on-premise databases to Azure Data Lake, improving data scalability, query performance, and regulatory audit readiness.• Architected hybrid warehouse models in Azure Synapse, supporting both batch and streaming ingestion for fraud analysis, transaction classification, and reporting.• Tuned Spark clusters on Databricks to enhance memory management, caching, and query planning, resulting in a 20% improvement in execution time for ETL and model scoring jobs.• Built reusable PySpark transformation layers to standardize feature engineering and streamline ML integration across multiple pipelines.• Enabled Power BI self-service analytics by deploying dimensional models and star/snowflake schemas using Kimball principles.• Integrated LLMs and RAG techniques for transaction-level contextual summaries, improving triage and reducing manual analyst workload.• Deployed synthetic data generation using Generative AI and Agentic AI models to simulate fraud scenarios and improve ML training coverage.• Managed end-to-end CI/CD lifecycle using Azure DevOps, GitHub Actions, and IaC for automated deployment and versioned rollbacks.• Monitored DAG execution and alerting using Apache Airflow on AKS, ensuring production pipeline reliability and real-time visibility.

Education

• Velagapudi Ramakrishna Siddhartha Engineering College

  Bachelor of Technology - BTech

• Northern Arizona University

  Master's