Santhosh Kumar

About Santhosh Kumar

Lead/Senior Data Engineer with 9+ years of experience building large-scale data…

Experience

1. Senior Data Engineer

   Optum

   Jul 2023 — Present · MN, US

   Developed Notebooks in Databricks for data extraction of various file formats and transforming and loading the detailed, aggregate data into Azure Data Lake and transmitting data into external data warehouses.• Designed and implemented a reusable test framework for validating data pipelines and transformations using PyTest, Azure DevOps pipelines, and DataBricks.• Automated data quality tests using Azure Test Plans to ensure ongoing validation of datasets as they are processed and ingested into Azure data platforms.• Created detailed test use cases to validate data transformations, calculations, and business rules within Azure Data Lake and Synapse environments.• Managed the end-to-end defect life cycle, from logging and triaging data quality issues to resolution, using Azure DevOps and JIRA. • Tracked and reported defects related to data validation and transformation logic, improving resolution times by 20% through automated notifications and reporting.• Collaborated with development and QA teams to resolve data-related issues, reducing data quality defects by 25% post-deployment.• Automated data quality tests using Azure Test Plans to ensure ongoing validation of datasets as they are processed and ingested into Azure data platforms.• Developed unit and integration test cases for validating complex ETL processes, ensuring seamless integration with existing CI/CD workflows.• Worked on migration of data from On-premises Source systems to Azure Cloud databases /Storages like Azure SQL, Azure Synapse Analytics, and Azure Data Lake in OLTP environment.• Built ingestion pipelines in Azure Data Factory (ADF) using various activities, linked services, and datasets to extract the data from Oracle DB, On Premise and write back into Azure cloud storage.• Conducted data quality checks and implemented data governance processes to ensure the accuracy and reliability of Unity Catalog data in Azure Databricks.

Education

• Shanthi Nikethan High School

  Middle School Diploma

• University of Central Missouri

  Master's degree, Computer Science

• Swami Ramananda Tirtha Institutue of Science & Technology

  Bachelors, Electrical, Electronics and Communications Engineering