Satish Kumar

About Satish Kumar

6+ years of experience as a Data Engineer in areas of Database Development, ETL…

Experience

1. Azure Data Engineer

   McKesson

   Jan 2022 — Present · US

   Facilitated requirements gathering and designed a common architecture for storing Retail data within the Enterprise, establishing a Data Lake in Azure cloud. • Developed Spark applications for efficient data extraction, transformation, and aggregation from diverse systems, storing processed data on Azure Data Lake Storage using Azure Databricks notebooks.• Utilized PySpark to integrate data from various sources, including ftp and csv files, processed using Azure Databricks, and written into Snowflake for enhanced data management. • Implemented Unzip and decode functions using Spark with Scala, parsing XML files into Azure Blob Storage to enhance data accessibility and usability. • Developed PySpark scripts to ingest data from source systems like Azure Event Hub into Delta tables in Databricks, supporting reload, append, and merge modes for seamless data integration.• Designed and implemented ETL processes in DataBricks to migrate Campaign data from external sources like Azure DataLake and gen2 in ORC/Parquet/Text Files formats. • Optimized PySpark applications on Databricks, resulting in significant cost reduction and improved efficiency. • Created Pipelines in Azure Data Factory to facilitate the seamless transfer of parquet files from ADLS Gen2 location to Azure Synapse Analytics Data Warehouse, ensuring data integrity and accessibility.• Replaced existing Hive scripts with Spark Data-Frame transformations and actions, enhancing data analysis speed and efficiency. • Developed PySpark scripts to reduce organizational costs by 30% through the migration of customer data from SQL Server to Hadoop, optimizing data storage and processing. • Demonstrated proficiency in handling JSON datasets and writing custom Python functions to parse through JSON data using Spark, enabling efficient data processing and analysis.