Vishal Sharma

About Vishal Sharma

Highly-skilled Data Engineer with over 5 years of experience in developing, implementing, and optimizing ELT/ETL processes.• GCP, DataProc (Managed/ Serverless), BigQuery, DataFlow, Composer, Airflow, Pub/Sub, GCS.• Hands-on experience in creating data-driven Pipelines to perform data ingestion and data transformation in Azure Data Factory.• Created Schedule Trigger, Tumbling Window Triggers, Event Triggers for scheduling the pipeline based on business requirements.• Experienced in working with Virtual machines, Virtual Networks, Storages and storage management Replications to LRS,GRS,RA-GRS. • Worked on Resource Groups, Load Balancer, Application Gateway, Azure AD,RBAC.• Created Parameterized linked services and datasets in Azure data factory.• Migrated the entire data system to Azure Cloud, utilizing ETL/ELT methodologies and tools such as Azure Data Factory ADF, Databricks ADB, Azure Data Lake, Spark SQL, PySpark, and Azure Cosmos DB.• Created Keys, Secrets in Azure Key Vault, resulting in a 50% increase in visibility for admin and authorized owners.• Developed, transformed and maintained tables, views, and functions in Databricks to optimize data extract, transform, and load (ETL) processes.• Strategically scheduled jobs and triggers based on different time frames for each source system to minimize RU consumption.• Optimized NoSQL database structure (COSMOS DB) using incremental and full load techniques.• Generated, maintained, and analyzed Azure monitoring dashboards, reports, and trends.• Monitored and troubleshooted data loads and data quality issues.• Created and maintained documentation of data models, ETL processes, and data security policies,• Worked with Delta Live Tables(DLT) in Azure Databricks.

Experience

1. Engineer Ii

   American Express

   Sep 2024 — Present · Bengaluru, IN

Education

• Amity University, Jaipur,

  Bachelor of Technology - BTech

  2015 — 2019