Vishal D.

About Vishal D.

Experienced with 5+ years in Big Data and Azure Data Services by providing cloud based Data platform solution. I excel at creating scalable and robust data pipelines that can handle large volumes of data efficiently. My expertise in Azure Data Factory, Azure Synapse and Azure Databricks, Apache Spark with over 3 years of experience allows me to leverage the power of the cloud to build secure, high-performance and cost-effective data pipeline solutions. Worked in Big Data and Data warehouses like Hadoop, Hive, Sqoop around 2 years. Proficient in working with Agile methodology. Armed with skills in Java, Python, SQL and C#

Experience

1. Data Engineer

   Tata Consultancy Services

   Jun 2021 — Present · Mumbai, IN

   Integrating data from HTTP clients, Azure Blob Storage and Azure Data Lake Gen2 using Azure Data Factory.• Developed Batch Pipelines to ingest the data from numerous Oracle DB system and store in ADLS Gen 2 in JSON and Parquet formats• Performed the Curation Like Data Transformation and then store the data into Curated Data zone which is then stored into Azure Cosmos DB into the Delievery Zone. • Scheduled pipelines using triggers such as Event Trigger, Schedule Trigger and Tumbling Window Trigger in Azure Data Factory (ADF) to automate the process.• Implemented an end-to-end pipeline on event hub using stream analytics to sink the data into Azure Cosmos DB and ADLS Gen 2 in JSON and Parquet Format • Created dependency between pipelines to orchestrate the data flow• Created ADF pipelines to execute Databricks Notebook activities to carry out transformations.• Created dependency between pipelines to orchestrate the data flow

Education

• University of Mumbai

  Bachelor of Engineering, Computer Engineering

  2014 — 2018