Dipak Shaw

About Dipak Shaw

I am driven by curiosity to learn, which keeps me moving to great heights. I started my career as a Big Data Engineer, which gave me a huge boost in my career. I have around 12+ years of experience in the data engineering and analytics fields. I learned so much from being able to work in multiple roles, right from solving technical problems to interacting with clients and building meaningful relationships. Every phase gave me an opportunity to learn, and I took ownership and have shown a positive approach. Currently, I am working with Microsoft. My role and responsibility are to build the ingestion pipeline to ingest and consume the data from multiple sources, clean, transform, and export the data into the target table and destination location for the reporting and make it available for further application using Azure services, Databricks, Power BI, SQL Server, etc by handling the several challenges with the data - privacy, unstructured and volume.Prior to Microsoft, I worked with HCL Technologies and Brillio Analytics, helping customers build data lakes, use an ETL pipeline to bring data into them from multiple sources, and analyze data to gain more insight into the business. I am also experienced in performing RCA on data and processes to answer specific business questions and identify opportunities for improvement.In addition to this, I am also having expertise experience in the Telecom, Retail, Logistics, and HR domains with multiple data-driven solutions, which has matured me as an IT professional with strong execution, methodology, and coding skills.Skills Highlight: Hadoop Components (HDFS, Hive, HBase, Sqoop, Map Reduce, Oozie, etc), Hortonworks, Cloudera, Spark, Scala, Python, AWS, Azure, Machine Learning, AI, Aurora, RabbitMQ, Shell Script, CICD, GitHub, Data Modeling & Designing, Data Warehousing and many more.I am an independent individual, and never stop at one thing, keep on upgrading myself, and believe in moving forward. He also loves motivating people and helping them achieve great things.As a wise man said once:\"You are always a student, never a master. You have to keep moving forward!\"

Experience

1. Senior Data Engineer

   Microsoft

   Sep 2023 — Present · IN

   Currently, I am working with Microsoft. My role and responsibility are to build the ingestion pipeline to ingest and consume the data from multiple sources, clean, transform, and load the data into the target table and destination location for reporting and make it available for further analysis. Provide Timely Trusted & Relevant Data to HR Services Insights business (for the prioritized business process) with the best Engineering practices and Modern technologies adhering to Data Governance and Compliance. Lead migration of an on-premises Data Warehouse to Azure using Modern TechFrom the technical stack, it includes - Apache Spark, Azure Active Directory, Azure Data Lake Storage, Azure Blob Storage, Azure Databricks, Azure Synapse, Azure Data Flow, Azure DevOps, Azure Data Explorer, Event Hub, Function App REST API, Azure Data Factory, Power BI, SQL Server, Cosmos DB, Azure Open AI, Azure Purview, etc.Currently, I am working with Microsoft. My role and responsibility are to build the ingestion pipeline to ingest and consume the data from multiple sources, clean, transform, and load the data into the target table and destination location for reporting and make it available for further analysis. Provide Timely Trusted & Relevant Data to HR Services Insights business (for the prioritized business process) with the best Engineering practices and Modern technologies adhering to Data Governance and Compliance. Lead migration of an on-premises Data Warehouse to Azure using Modern Tech From the technical stack, it includes - Apache Spark, Azure Active Directory, Azure Data Lake Storage, Azure Blob Storage, Azure Databricks, Azure Synapse, Azure Data Flow, Azure DevOps, Azure Data Explorer, Event Hub, Function App REST API, Azure Data Factory, Power BI, SQL Server, Cosmos DB, Azure Open AI, Azure Purview, etc.

Education

• MCKV Institute of Engineering

  Bachelor’s Degree, Computer Science & Engineering

Skills

• Hive
• Oozie
• Pyspark
• Hadoop
• Sql
• Apache Pig
• Flume
• C#
• .net
• Cicd
• Java
• Shell Scripting
• Eclipse
• Github
• Git
• Python
• Mapreduce
• Hbase
• Mysql
• Big Data
• Hdfs
• Rabbitmq
• Microsoft Sql Server
• Scala
• Mongodb
• Sqoop
• C
• Programming
• Azure
• Spark