Banin Zaidi

Experience

1. Senior Data Engineer

   Walgreens

   Apr 2022 — Present · NY, US

   Developed complex SQL queries for data transformation, aggregation, and analysis in both relational and distributed environments, ensuring optimal performance and accuracy.• Utilized PySpark for efficient data processing, leveraging RDDs, Data Frames, and SQL functions.• Designed and orchestrated end-to-end ETL workflows using Azure Data Factory, enabling automated data ingestion, transformation, and loading from multiple sources into Azure Data Lake.• Integrated Azure Key Vault to securely manage sensitive data, including API keys, connection strings, and credentials across Databricks and Azure services.• Implemented Delta Lake to build transactional data pipelines with ACID compliance, supporting batch and streaming processes with versioning and time travel features.• Set up monitoring, logging, and alerting for pipeline health in Databricks and ADF, ensuring smooth operations, quick issue resolution, and data integrity.• Automated and scheduled batch jobs in Databricks, optimizing cluster usage for cost efficiency and parallel processing.• Integrated Salesforce CRM data with Azure Data Factory and Databricks pipelines for Azure Data Factory and Databricks marketing and sales analytics.• Developed custom User Defined Functions (UDFs) in PySpark to extend Spark’s capabilities for specialized data processing tasks.• Created reusable Python modules for handling schema drift, incremental loads, and automated file processing.• Applied data partitioning strategies in Spark and Delta Lake to optimize query performance and reduce job runtimes.• Implemented data lineage tracking and auditing for improved traceability, transparency, and compliance with data governance standards.

Education

• Queens College

  Bachelor's degree

  2011 — 2015