Varun Kumar

About Varun Kumar

Highly skilled Azure Data Engineer with 9.7 years of extensive experience in the design…

Experience

1. Azure Data Engineer

   Freddie Mac

   Oct 2021 — Present

   Involved in complete BigData flow of the application starting from data ingestion from upstream to HDFS, processing and analyzing the data in HDFS\\\\nWorking knowledge on Azure cloud components (Databricks, Data Lake, Blob Storage, Data Factory, Storage Explorer, SQL DB, SQL DWH, CosmosDB).\\\\nExperience in analyzing data from Azure data storages using Databricks for deriving insights using Spark cluster capabilities.\\\\nExtract Transform and Load data from Sources Systems to Azure Data Storage services using a combination of Azure Data Factory, data bricks, Pyspark, Spark SQL and U-SQL Azure Data Lake Analytics.\\\\nCreated Pipelines in ADF using Linked Services/Datasets/Pipeline/ to Extract, Transform and load data from Azure SQL, Blob storage, and Azure SQL Data warehouse\\\\nWorked with Azure BLOB and Data lake storage and loading data into Azure SQL Synapseanalytics (DW).\\\\nInvolved in developing data ingestion pipelines on Azure HDInsight Spark cluster using Azure Data Factory and Spark SQL.\\\\nDesigned and implemented a real-time data streaming solution using Azure EventHub.\\\\nConducted performance tuning and optimization activities to ensure optimal performance of Azure Logic Apps and associated data processing pipelines.\\\\nDevelop a Spark Streaming application to process real-time data from various sources such as Kafka, and Azure Event Hubs.\\\\nBuild streaming ETL pipelines using Spark Streaming to extract data from various sources, transform it in real-time, and load it into a data warehouse such as Azure Synapse Analytics\\\\nUse tools such as Azure Databricks or HDInsight to scale out the Spark Streaming cluster as needed.\\\\nDeveloped Spark API to import data into HDFS from Teradata and created Hive tables\\\\nCreated Partitioned and Bucketed Hive tables in ParquetFile Formats with Snappy compression.\\\\nLearner data model which gets the data from Kafka in real time and persist it to Cassandra.\\\\nHands on programming experience in scripting languages like python and Scala.\\\\n \\\\n\\\\n \\\\n \\\\n\\\\n \\\\n