Syeda Tahira Fatima

Senior Data Engineer @Scotiabank

Houston, TX, US

Involved in data warehouse implementation on Azure Synapse using Synapse pipelines, notebooks, SQL Serverless, and dedicated SQL Pool• Created dynamic data pipelines to ingest data from Salesforce, SQL Server, Oracle, Marketo, and other proprietary data sources into Azure Data Lake Gen2.• Implement data ingestion, transformation, and analysis workflows• using Apache Spark APIs (RDD, DataFrame, and Dataset).• Implemented incremental data loads into Data Lake using a watermark table, variables, and parameters.• Architected 3-layered architecture using Delta Lake to store raw data in the bronze layer, transformed data in the silver layer, and fact and dimension tables in the gold/curated layer.• Developed data ingestion workflows using Java in combination with Apache Kafka for real-time streaming and batch processing.• Involved in sprint planning, backlog grooming, daily standups, and sprint retrospectives to ensure efficient project execution and continuous improvement.• Implemented slowly changing dimensions (Type II) using Data Factory and Databricks.• Involved in building CI/CD pipelines for deploying Data factory pipelines and Databricks notebooks to UAT and production.• Designed and implemented CI/CD pipelines using Azure DevOps for automated deployment of Data Factory pipelines, Synapse Analytics, and Databricks notebooks to staging and production environments.• Implemented Data Governance frameworks to ensure data integrity, compliance, and security across multiple Azure data services, including Synapse, Data Lake, and Data Factory.• Design and develop end-to-end big data pipelines, leveraging Spark for various processing tasks.• Integrate Spark with other big data technologies (e.g, Kafka, Hive, Cassandra) to create a seamless data flow.• Provide technical leadership and mentorship to team members on the use of Spark and other big data technologies.

NED University of Engineering and Technology

Bachelor of Engineering - BE