Syed Kazmi

Experience

1. Senior Data Engineer

   Advantage Solutions

   Apr 2020 — Present · Richmond, TX, US

   Involved in data warehouse implementation on Azure Synapse using Synapse pipelines, notebooks, SQL Serverless, and dedicated SQL Pool• Created dynamic data pipelines to ingest data from Salesforce, SQL Server, Oracle, Marketo, and other proprietary data sources into Azure Data Lake Gen2.• Implemented incremental data loads into Data Lake using a watermark table, variables, and parameters.• Architected 3-layered architecture using Delta Lake to store raw data in the bronze layer, transformed data in the silver layer, and fact and dimension tables in the gold/curated layer.• Developed data ingestion workflows using Java in combination with Apache Kafka for real-time streaming and batch processing.• Involved in sprint planning, backlog grooming, daily standups, and sprint retrospectives to ensure efficient project execution and continuous improvement.• Implemented slowly changing dimensions (Type II) using Data Factory and Databricks.• Involved in building CI/CD pipelines for deploying Data factory pipelines and Databricks notebooks to UAT and production.• Design and develop end-to-end big data pipelines, leveraging Spark for various processing tasks.• Integrate Spark with other big data technologies (e.g, Kafka, Hive, Cassandra) to create a seamless data flow.• Provide technical leadership and mentorship to team members on the use of Spark and other big data technologies.• Utilized Java for processing large datasets, building custom solutions for data transformations in a data pipeline architecture.• Utilized various optimization features of Databricks to improve code performance such as partitioning, bucketing, avoiding data skew, salting, and z-order clustering.• Design and implement self-service data platforms that enable users to access, explore, and analyze data using Spark and other big data tools.• Integrate Apache Spark with other technologies (e.g, Kafka, Hadoop, databases) to create end-to-end data pipelines.

Education

• Baruch College

  ● Bachelor of Business Administration

  2011 — 2015