Revanth Chaganti

Senior Data Engineer @Centene Corporation

St Louis, MO, US

Designed and implemented end-to-end ETL and ELT pipelines using Azure Databricks, ADF (Azure Data Factory), PySpark, Scala, Python, and SQL to process high-volume structured and semi-structured Big Data into curated Data Lakes on Azure.Developed parameterized ingestion pipelines using ADF, enabling incremental loads, reusable ETL components, and scalable Data Migration patterns across enterprise source systems.Developed PowerShell scripts to automate DevOps tasks for Azure Data Factory and Databricks job deployments across dev, QA, and prod environments.Collaborated closely with UX/UI teams and product managers to enhance the usability of data visualization dashboards in Power BI.Built batch and streaming ETL frameworks leveraging Apache Spark (PySpark, Scala) and Azure, supporting near-real-time ingestion, CDC processing, and reliable delivery into Data Lakes, Snowflake, and downstream Data Warehouse layers.Applied strong problem-solving skills to resolve performance issues in DLT pipelines and investigate data inconsistencies across curated layers.Refactored and converted legacy SQL-based ETL jobs into optimized Delta Lake pipelines, leveraging ACID transactions, schema evolution, and partitioning to improve performance, reliability, and Data Governance across Azure Data Lakes.Define schema evolution and implement Change Data Capture (CDC) strategies within Delta tables to support audit and regulatory traceability.Utilize Azure Data Factory (ADF) to orchestrate batch ingestion workflows, integrating nightly extracts from Oracle and Teradata into Azure-based data lakes.Implement partitioning and Z-Ordering on Delta tables to optimize query performance on time-series data such as trade logs and balance sheets.Configure and optimize job clusters in Azure Databricks based on workload profiles scaling for batch jobs during off-peak hours and streaming jobs 24/7.