Ned A.

Sr Bi Etl Developer @Elastic

Vancouver, BC, CA

Designed Pyspark ingestion pipelines with JDBC for Postgres and Snowflake connectors, handling 10M+ records daily from PostgreSQL to Snowflake, ensuring fault-tolerant data pipelines.• Designed and implemented Medallion Architecture (Raw, Curated and Published layers) on top of Azure Data Lake Gen2 to improve data quality of data being loaded into data mart.• Created and designed Power BI data reconciliation dashboard for monitoring ELT/ETL process.• Enhanced operational visibility of the data pipelines by integrating custom logging, error handling and control tables in ADF and databricks workflows, resulting in 30% faster troubleshooting and near zero data loss SLAs.