Mayuri S

Experience

1. Senior Azure Data Engineer

   PDS Health

   Mar 2022 — Present · Irvine, CA, US

   Developed ELT pipelines using Azure Data Factory and Azure Databricks.• Designing and implementing scalable and reliable cloud-based data storage solutions. Scheduling and Monitoring ADF pipelines.• Implementing data warehousing solutions using AWS Redshift or Snowflake to enable efficient data storage and retrieval.• Developed PySpark & Spark SQL scripts for multiple data products based on the business logic.• Performed optimizations for existing ADF pipelines, pyspark and data bricks notebooks which improved overall performance of existing implemented architecture.• Experience in configuring custom email notifications using Azure Logic Apps. Implemented incremental load, SCD type 1 and SCD type 2 to capture the data changes. Performance tuning of delta tables with Optimize and ZORDER operations.• Cleaning of delta tables old snapshots through VACUUM procedure.• Worked with Azure DevOps to do the code commits and code migrations across dev, test and prod environments.• Creating and managing data models, schemas, and database objects in AWS RDS (Relational Database Service) or DynamoDB.• Strong skills in visualization tools Power BI, Confidential Excel - formulas, Pivot Tables, Charts and DAXCommands.• Extract Transform and Load data from Sources Systems to Azure Data Storage services using a combination of Azure Data Factory, T-SQL, Spark SQL and U-SQL Azure Data Lake Analytics.• Data Ingestion to one or more Azure Services -(Azure Data Lake, Azure Storage, Azure SQL, Azure DW) and processing the data in In Azure Databricks.Azure Data Factory, Azure Databricks, Azure Logic Apps, Azure SQL, Azure DW) Amazon Web Services (AWS Redshift, AWS Glue, AWS Athena, AWS S3, AWS RDS), AWS S3 PySpark, SQL, T-SQL, U-SQL, Apache Spark, Power BI, CosmosDB, MongoDB, Snowflake, Blob Storage, Scala, Spark v2.0.2, Hadoop 2.x (HDFS, MapReduce, Yarn), Airflow, Hive, Sqoop, HBase.

Education

• Vishwakarma Institute Of Technology

  Bachelor of Engineering - BE, Computer Science