Varun R
Azure Data Engineer @IHG Hotels & Resorts
Signup · Get unlimited contacts
WORK HISTORY
Azure Data Engineer @IHG Hotels & Resorts
Atlanta, GA, US
Developed Databricks ETL pipelines using notebooks, Spark Dataframes, SPARK SQL, and python scripting.• Wrote PySpark code in Databricks notebooks to fetch the data from disparate data sources to land the entity in ADLS Gen2.• Parameterized the notebooks to have a single laptop to land all the entity data.• Created numerous pipelines in Azure using Azure Data Factory v2 to get the data from disparate source systems by using different activities like Move & Transform, Copy, filter, for each, Databricks, etc.• Created tables with different distribution strategies Hash, round robin, replicated based on the ETL in Azure Synapse. • Implemented ELT data loading strategy on Azure Synapse dedicated pool using polybase, data lake, external tables, and stored procedures.• Orchestrated jobs using triggers like Events, Schedules, and Tumbling in Azure Data Factory.• Created snow pipe to auto ingest data into Snowflake Target table on new file arrival from an external azure blob using Eventgrid, Storage Queue, and Snowflake external stage & COPY INTO. • Ingested data into Snowflake using Azure Databricks (DataFrames & Spark SQL) and ADF (Copy activity).• Used Common Table Expressions (CTE) to create multiple tables in a single notebook using SQL queries.• Built Data Pipelines and stored the data in ADLS with different layers like Raw, Refined and Trusted.• Created and maintained Jobs using PySpark to run the necessary pipelines in batch-wise processing.• Worked with building Data Warehouse structures, and creating facts, dimensions, and aggregate tables, by dimensional modeling and Star schemas.• Experience in working with Azure DevOps as a code repository and code deployments.• Used Databricks SQL to analyze the data to find any data inaccuracies and to work with the missing data.• Designing and implementing highly performant data ingestion pipelines from multiple sources using Apache Spark and/or Databricks Delta Lake.
EDUCATION
JNTUH College of Engineering Hyderabad
Bachelor's degree
ABOUT VARUN R
Senior Data Engineer at IHG | Actively Seeking New Opportunities | Big Data | Python | SQL | Azure | AWS | Snowflake | PySpark | Databricks | ETL | Visualization |
This profile is compiled from publicly available professional sources. Unifers is not affiliated with or endorsed by LinkedIn. Request removal of this profile.