Neville Smith

Azure Data Solutions Engineer @Skillsoft

Architecture of an Azure ADF and Databricks solution to unify five existing data warehouses. ADF pipelines to load data from sources including Salesforce and Google Cloud Storage into raw data lake storage, and ADF pipelines to orchestrate Databricks notebook tasks to load data through to bronze and silver data lake (parquet and delta – SCD2) using autoloader (pyspark and spark SQL). Azure data bricks notebooks to load data into gold layer - lake warehouse - for PowerBI connection (tabular…