Mulla

Experience

1. Cloud Data Engineer

   Verisk

   Aug 2022 — Present · Jersey City, NJ, US

   Responsibilities:Led ingestion of raw data files from multiple clients via SFTP, ensuring secure, timely, and automated data flow in AWS cloud infrastructure.Designed and maintained an AWS S3-based data lake to store raw, transformed, and curated data for downstream analytics and ML workflows.Engineered scalable ETL pipelines with AWS Glue and Lambda, improving data transformation throughput by 35%.Implemented data validation using AWS Glue and profiling with AWS Glue DataBrew, improving data quality and reducing manual QA by 40%. Developed Python-based data validation and audit tools using Pandas and Matplotlib to automate schema checks, detect missing values, duplicates, and skewed distributions, significantly reducing data quality issues in production. Processed and transformed vendor data from Amazon S3 with Pandas, generating structured JSON for seamless monthly ingestion. Deployed Lambda-based loaders from S3 to Snowflake, cutting ingestion time by 60% for recurring data pipelines. Used PySpark on Databricks for deduplication and transformation; loaded results into AWS Redshift and Snowflake. Built real-time streaming pipelines with Amazon MSK (Kafka), ensuring scalable and secure ingestion from multiple sources. Created reusable Terraform modules to provision infrastructure (VPCs, ECS Fargate, Lambda, EC2), reducing provisioning time by 70%. Orchestrated containerized workloads with Docker and ECS Faregate; automated deployments via GitLab CI/CD and Terraform. Collaborated with data scientists to integrate APIs within Databricks, enabling analytics workflows and model feedback loops. Secured AWS environment via VPC configuration, IAM roles, security groups, and private subnet routing. Developed scalable and serverless solutions using AWS Step Functions, SNS, and SQS for orchestrating workflows. Wrote complex SQL queries to validate data integrity in Redshift and Snowflake, including window functions, CTEs, and multi-table joins.