Karim Hasanul

Experience

1. Data Engineer

   BNY

   Jan 2023 — Present · Manhattan, NY, US

   Implemented Azure Data Factory (ADF) to automate ETL processes, designing and developing pipelines, data flows, and activities, resulting in streamlined data integration and improved efficiency. Managed Azure Blob Storage to optimize data storage, ensuring proper file organization, implementing access controls, and enhancing data security and accessibility. Utilized Azure Databricks and PySpark API to develop data transformation workflows, enabling efficient data processing, cleansing, and enrichment, leading to improved data quality and reporting accuracy. Implementing data validation routines and data quality checks to identify and resolve anomalies or inconsistencies in the Marsendise product data. This involves utilizing Pyspark and Python data manipulation libraries to perform data cleansing, outlier detection, and data profiling, ensuring data integrity and accuracy. Implemented data lineage and impact analysis using PySpark, enabling traceability of data transformations and understanding the downstream impact of changes. Implemented data caching and in-memory processing techniques using PySpark to improve query performance and reduce latency. Contributed to real-time data analytics initiatives by configuring and managing Azure Stream Analytics jobs, allowing timely insights and actionable intelligence from streaming data sources. Assisted in the development and maintenance of a scalable NoSQL database solution using Azure Cosmos DB, supporting data availability, scalability, and performance for critical applications. Contributed to serverless data processing using Azure Functions and Event Hub, developing Python-based triggers and bindings to handle real-time data streams and automate data processing tasks. Supported the administration and optimization of MySQL and PostgreSQL databases, including schema design, indexing strategies, and query optimization, ensuring efficient data retrieval and system performance.