Momen

About Momen

Experienced Data Engineer with over 12 years of expertise in designing and delivering…

Experience

1. Lead Data Architect Data Engineer

   Statefarm Insurance

   Sep 2023 — Present

   Involved in converting Hive/SQL queries into Spark transformations using Spark RDDs, Python, and Scala. Responsible for writing Map-Reduce job which joins the incoming slices of data and picks only the fields needed for further processing.Developed transformation logic in PySpark and SQL on Azure Databricks for large-scale data preparation and machine learning integration.Built scalable ETL pipelines using PySpark in AWS Databricks to process large volumes of structured and unstructured dataLed the design and implementation of scalable and efficient data architectures for diverse business requirementsProvide guidance to key stakeholders for data compliance, quality, governance/anonymization, and maintenance.Leveraged AWS services such as S3 for data storage, EC2 for computation, and Lambda for serverless data processing, enhancing scalability and cost-effectiveness.Performed Data cleaning and Preparation on XML files. Developed and managed end-to-end data pipelines using Azure Data Factory (ADF) for orchestrating and automating ETL workflows.Created and managed Databricks jobs on AWS for scheduled and triggered batch processes.Used SQL queries and other tools to perform data analysis and profiling. Routinely use python (Pandas, Polars, scikit) and noSQL to create, maintain, and optimize data pipelines.Utilized Spark SQL API in PySpark to extract and load data and perform SQL queriesCreate and optimize NoSQL routines for large historical timeseries data sets in order to increase speed of data requestsOptimized HDFS storage for efficient retrieval of product information, user logs, and order data to support real-time analytics and personalization. Decreased data pipeline downtime by 60% by implementing robust error handling and retry mechanisms in Airflow workflows.