Imran Mohammad

Senior Data Engineer @Manulife

Waterloo, ON, CA

Involved in requirements gathering, analysis, design, development, change management, deployment.• Involved in the development of real time streaming applications using PySpark, Apache Flink, Kafka, Hive on distributed Hadoop Cluster.• Utilizing Apache Spark with Python to develop and execute Big Data Analytics and Machine learning applications, executed machine learning use cases under Spark ML and Mllib.• Extracting data from heterogeneous sources and performed complex business logic on network data to normalize raw data which can be utilized by BI teams to detect anomalies.• Designing and developing Flink pipelines to consume streaming data from Kafka and applied business logic to massage and transform and serialize raw data.• Developing common Flink module for serializing and deserializing AVRO data by applying schema.• Developing Spark streaming pipeline to batch real time data, detect anomalies by applying business logic and write the anomalies to HBase table.• Implementing layered architecture for Hadoop to modularize design. Developed framework scripts to enable quick development. Designed reusable shell scripts for Hive, Sqoop, Flink and PIG jobs. Standardize error handling, logging and metadata management processes.• Indexed processed data and created dashboards and alerts in splunk to be utilized/ action by support teams.• Responsible for operations and support of big data Analytis platform, Splunk and Tableau visualization.• Managed, developed, and designed a dashboard control panel for customers and Administrators using Tableau, PostgreSQL and RESTAPI calls. Skills: Apache Spark · PySpark · Scala · Extract, Transform, Load (ETL) · hdfs · Hadoop · Continuous Integration and Continuous Delivery (CI/CD) · Google BigQuery · Microsoft Azure ·