H Rahaman

About H Rahaman

Senior AI/ML Engineer with 12+ years of experience…

Experience

1. Sr Data Analyst Sr Data Engineer

   T-Mobile

   Oct 2021 — Present · Seattle, WA, US

   Responsibilities: • Designed and implemented data pipelines consists of launching several Hive, Spark equipped EMR clusters that reads the datasets from various data sources and perform analytics, transformations and finally write to RDS database.• Developed automated AWS Glue jobs to ingest data coming from SFTP vendor systems. Implemented Crawler and catalog tables to automatically manage the Schema changes of incoming files.• Developed AWS Lambda jobs to update metadata tables between cross-job dependencies. Build AWS Lambda job to segregate files based on schema into sub-directories. Setup IAM roles on top of S3 buckets so that AWS Lambda has specific access required to access respective buckets. Setup configuration such that AWS Lambda gets triggered automatically as soon as file lands into source S3 bucket, also called as landing zone.• Built Data models in Redshift (data warehouse), The ETL pyspark jobs write data into Redshift after the transformation is complete. Build SQL scripts on Redshift to perform Change data capture to de-duplicate data for SCD type-2 tables.• Built ETL jobs to load fixed width files from mainframe server to get loaded into Redshift on a daily basis using batch jobs.• Built streaming jobs using Spark Streaming which transfer data from transaction severs had been loaded into Cassandra tables in real-time workloads. The tables are further used by machine learning models to verify transaction authenticity.• Developed the report merging different vertical feeds from same source using custom python functions. The script was implemented on EMR.• Written Unittest scripts using Unitest and pytest packages for Spark scripts and submitted the test results to a test bucket in s3. Developed the test automation framework which is used to test the module during CICD phases.Environment: AWS, S3, Sqoop, Kafka, Spark, Spark SQL, Hive, LINUX, Oozie, Java, Scala, Eclipse, Tableau, UNIX Shell Scripting, Putty.