Demis A.

About Demis A.

9+ years of data engineering experience in analysis, design, development, testing and maintaining data infrastructure solutions. Proficient in various programming languages and technologies, data modeling, ETL processes, and data warehousing. Delivered high-quality data solutions to drive business growth and improve decision-making.• Experienced in using Apache airflow pipeline development.• Expertise in writing SQL queries, views, triggers, complex queries, stored procedure, and functions on SQL database.• Hands on experience in using Azure Data Factory to extract, process and transform datasets from various sources, create pipelines and load the data into ADLS.• Developed custom DBT macros to streamline common data transformation tasks and improve code maintainability.• Collaborate with data analysts and data scientists to provide them with clean, reliable, and well structured data sets.• Experienced in Ingesting data from S3 to Azure blob storage(staging) and load the transformed dataset to ADLS in Azure Synapse. • Integrated Logic APPs and Azure Runbook automation accounts to automate ADF pipeline workflows and integrate with different services. • Experienced on using AWS services like DynamoDB, Lambda, SQS, SNS, EC2, S3, ELB, Route 53, RDS, AWS Glue, Redshift and CloudFront, CloudFormation• Experienced in developing ETL workflows using AWS Glue and Python to extract, transform and load data from various sources into AWS data stores.• Extensive experience in performance tuning of ETL pipeline and DB components• Gathered and translated complex business requirements into comprehensive data models, reports, and interactive dashboards. • Experienced in using Document/MongoDB NoSQL database on AWS cloud.• Collaborate with teams to understand data requirements and design scalable data pipelines for data ingestion, transformation, and storage.• Wrote pre-hook and post-hook scripts to create temp tables in DBT.• Worked with Business Analysts and Data Modelers to understand requirements.• Developed ETL/ELT pipelines using Snowflake, extract, transform and load data from various sources.• Write SQL transformation in DBT to clean, enrich and aggregate data.• Well versed with Dimensional modeling, Star, Snowflake Schema for designing Data warehouse & Data Mart• Build ETL process using DBT to provide a structured way to define the transformation in code, making it easy to version control and collaborate on data transformation.

Experience

1. Senior Data Engineer

   State Farm

   Mar 2020 — Present

   Developed and optimize ETL workflows using PySpark and SQL for large scale data processing and optimization.•Ingest data from S3 bucket, transform and load it into ADLS using Azure Data factory.•Participated in building ETL pipeline and source to target mapping to load data into DataWarehouse.•Used Data Bricks to ingest datasets from various sources into ADL•Implemented a pipeline to ingest data from Salesforce into ADLS Gen2 using Azure DataBricks. •Used Data frame API to read and write data into and from Azure data lake• Collaborate with teams to understand data requirements and design scalable data pipelines for data ingestion, transformation, and storage.•Prepared documentation that describes all required information about the data base, tables, relationships, business logics and development directions•Partners with business, architecture, and infrastructure to develop service level agreements to ensure consistency and customer satisfaction. Created a pipeline on Azure fabrics.•Supervises the preparation of designs and technical and/or functional specificationsMentor others on coding standards and performs code reviews.•Experienced in Integrating Snowflake with various data sources and tools such as Informatica, AWS S3, and Azure Data Lake. •Working knowledge on generative AI, Large Language Model (LLM) and Langchain. •Experienced in using Azure Data Factory to extract, process and transform datasets from various sources.•Developed ETL/ELT pipelines using Snowflake, extract, transform and load data from various sources.•Used various transformations techniques in Azure DataBricks to ingest data from files, database, http endpoints and blob storage to ADLS.• Developed ETL workflows using AWS Glue and Python to extract, transform and load data from various sources into AWS data stores.

Education

• Trent University

  Master of Science - MS

  2009 — 2012