Mayur Nayak

Azure Data Engineer @Verisk Crime Analytics, Inc

NJ, US

Develop, design data models, data structures and ETL jobs for data acquisition and manipulation purposes.Develop deep understanding of the data sources, implement data standards, maintain data quality and master data management.Expert in developing JSON Scripts for deploying the Pipeline in Azure Data Factory (ADF) that process the data.Expert in using Databricks with Azure Data Factory (ADF) to compute large volumes of data.Performed ETL operations in Azure Databricks by connecting to different relational database source systems using JDBC connectors.Developed Python scripts to do file validations in Databricks and automated the process using ADF.Developed an automated process in Azure cloud which can ingest data daily from web service and load in to Azure SQL DB.Developed Streaming pipelines using Azure Event Hubs and Stream Analytics to analyze data for dealer efficiency and open table counts for data coming in from IOT enabled poker and other pit tables.Analyzed data where it lives by Mounting Azure Data Lake and Blob to Databricks.Used Logic App to take decisional actions based on the workflow.Developed custom alerts using Azure Data Factory, SQLDB and Logic App.Developed Databricks ETL pipelines using notebooks, Spark Dataframes, SPARK SQL and python scripting.Developed complex SQL queries using stored procedures, common table expressions (CTEs), temporary table to support Power BI reports.Implemented complex business logic through T - SQL stored procedures, Functions, Views and advance query concepts.Worked with enterprise Data Modeling team on creation of Logical models.Development level experience in Microsoft Azure providing data movement and scheduling functionality to cloud-basedtechnologies such as Azure Blob Storage and Azure SQL Database.Independently manage development of ETL processes - development to delivery.Environment: ETL, Azure Data Factory (ADF), Azure Databricks, Azure Event Hubs, Stream Analytics, Azure Data Lake and more.