Hmd S

Experience

1. Sr Data Engineer

   AbbVie

   Jul 2022 — Present · Vernon Hills, IL, US

   Responsibilities • Experience in building and architecting multiple Data pipelines, end-to-end ETL, and ELT processes for Data ingestion and transformation in AWS. • Used AWS API Gateway and AWS Lambda to get AWS cluster inventory by using AWS Python API • Implemented a Continuous Delivery pipeline with Docker and GitHub • Worked with Lambda function to load Data into Redshift on the arrival of CSV files in an S3 bucket • Devised simple and complex SQL scripts to check and validate Data Flow in various applications. • Performed Data Analysis, Migration, Cleansing, Transformation, Integration, Data Import, and Data Export through Python. • Worked on JSON schema to define tables and column mapping from AWS S3 data to AWS Redshift and used AWS Data Pipeline for configuring data loads from AWS S3 to AWS Redshift • Performed data engineering functions: data extract, transformation, loading, and integration in support of enterprise data infrastructures - data warehouse, operational data stores, and master data management • Responsible for data services and data movement infrastructures. Good experience with ETL concepts, building ETL solutions, and Data modelling. • Architected several DAGs (Directed Acyclic Graph) for automating ETL pipelines • Hands-on experience in architecting the ETL transformation layers and writing spark jobs to do the processing. • Gather and process raw data at scale (including writing scripts, web scraping, calling APIs, writing SQL queries, and writing applications) • Imported data from AWS S3 into Spark RDD and performed actions/transformations on them. • Created Partitions, Bucketing, and Indexing for optimization as part of Hive data modelling. • Involved in developing Hive DDLs to create, alter, and drop Hive tables.