Abuzar Khan

Senior Data Engineer @GAP Solutions, Inc.

US

Involved in full Software Development Life Cycle (SDLC)- Business Requirements Analysis, preparation of Technical Design documents, Data Analysis, Logical and Physical database design, Coding, Testing, Implementing, and deploying to business users. Designed and implemented end-to-end data pipelines on AWS using services such as AWS Glue, AWS Lambda, and AWS EMR. Written complex SQLs using joins, sub queries and correlated sub queries. Expertise in SQL Queries for cross verification of data. Created ingestion framework for creating Data Lake from heterogeneous sources like Flat files, Oracle Db, mainframe, and SQL server Databases. Design and Develop ETL Processes in AWS Glue to load data from external sources like S3, glue catalog, and AWS Redshift. Used DynamoDB to log the errors of the ETL process while validating input files with target table structure data type mismatches and all. Developed complex ETL mappings for Stage, Dimensions, Facts, and Data marts load. Involved in Data Extraction for various Databases & Files using Talend. Optimized Spark jobs in Databricks to enhance performance and reduce processing time, utilising AWS EMR integration where necessary. Collaborated with data scientists to deploy machine learning models using Databricks on AWS for real-time scoring and inference. Integrated Databricks with AWS services such as Amazon S3, Amazon Redshift, Amazon RDS, and AWS Glue.