Akhil M

About Akhil M

Around 6+ years of professional experience in IT industry including designing…

Experience

1. Senior Data Engineer

   Fifth Third Bank

   Dec 2021 — Present

   Proficient in working with Azure cloud platform (HDInsight, DataLake, DataBricks, Blob Storage, Data Factory, Synapse, SQL, SQL DB, DWH and Data Storage Explorer). • Designed and deployed data pipelines using DataLake, DataBricks, and Apache Airflow. • Enabling other teams to work with more complex scenarios and machine learning solutions. • Worked on Azure Data Factory to integrate data of both on-prem (MY SQL, Cassandra) and cloud (Blob storage, Azure SQL DB) and applied transformations to load back to Azure Synapse. • Evolved in Spark Scala functions for mining data to provide real time insights and reports. • Configured spark streaming to receive real time data from the Apache Flume and store the stream data using Scala to Azure Table. • DataLake is used to store and do all types of processing and analytics. • Ingested data into Azure Blob storage and processed the data using Databricks. Involved in writing Spark Scala scripts and UDF\'s to perform transformations on large datasets. • Utilized Spark Streaming API to stream data from various sources. Optimized existing Scala code and improved the cluster performance. • Involved in using Spark DataFrames to create Various Datasets and applied business transformations and data cleansing operations using DataBricks Notebooks. • Efficient in writing Python scripts to build ETL pipeline and Directed Acyclic Graph (DAG) workflows using Airflow, Apache NiFi. • Tasks are distribution on celery workers to manage communication between multiple services.

Education

• SRM IST Chennai

  Bachelor's degree

  2013 — 2017