Alok Kumar Aryan

About Alok Kumar Aryan

Certification: Postgraduate Diploma in Big Data Analytics from C-DAC, Noida- Enthusiastic about tackling challenges and eager to explore new technologies. Possess over 6.4+ years of hands-on experience in product development, technical architect design, big data asset development, big data migration, web application development, and ETL. Demonstrated expertise in offering direction, formulating strategies, and providing a visionary approach in the realm of technology and product development- Designed and implemented a scalable financial data ingestion and transformation pipeline for multiple business units. Built robust workflows using Apache Airflow, DBT, and Trino, with historical tracking via SCD Type 2. Created reusable SQL models, optimized data access through unit-specific views, and validated schemas against Source-to-Target (S2T) mappings. Enabled efficient storage and querying using Iceberg and Parquet formats on AWS S3, supporting business logic execution and reporting- Leveraged Airflow as the orchestration service for the backend data ingestion pipeline, optimizing data engineering processes through AWS EMR. Implemented a robust backend pipeline for daily scheduled data ingestion, incremental load & full load harnessing AWS services such as MWAA, EMR, S3, Athena, and RDS (Postgres, MySQL) alongside PySpark and Hive. Orchestrated Managed Apache Airflow (MWAA) to streamline the backend data ingestion pipeline, further enhancing data engineering processes through EMR- Developed a scalable ML model execution pipeline automating the ML model execution process for each batch of data fetched from the data warehouse (Redshift). Implemented the pipeline with Python, R, Docker, and various AWS services (EMR, ECS, Lambda, SQS, Redshift, MySQL).Established an automated data ingestion flow for clinical data, pulling information from diverse sources, flat files and ingesting it into databases after applying business rules, DQM checks, standardization and SCD2 implementation- Utilized a diverse technology stack: Python, Databricks, PySpark, DBT, Starburst, Trino, Control M, Hadoop, Hive, Hbase, Airflow, Docker, Kubernetes, Shell Scripting, AWS: EMR, MWAA, SQS, ECS, EKS, ATHENA, GLUE, ECR, EC2, S3, S3 Glacier, ELB, API Gateway, Lambda, CI/CD, PostgreSQL, MySQL, Aurora, Amazon Redshift, Cloud9, CloudWatch, Amazon Managed Airflow, CloudFormation, Auto Scaling, IAM, SNS, SES, Terraform, RESTful API, Flask, FastAPI, GraphQL, GitHub, Bitbucket, SonarQube, AZURE, Elasticsearch & more

Experience

1. Senior Data Engineer

   IBM

   Jun 2024 — Present · Bengaluru, IN

   Hubble Data Engineering | Domain: Financial Data -Designed and built a scalable financial data platform on AWS, implemented SCD Type 2 incremental pipelines for multiple business units to load raw data from AWS S3 landing zones into validation and integration layers with historical tracking. Deployed Airflow Dag on AWS EKS for workflow orchestration and distributed job processing. Executed containerized DBT models on EKS via Airflow DAGs, enabling incremental transformations-Implemented transformation logic through DBT (Data Build Tool), created SQL models and macros for generic business logic and loaded transformed datasets into Apache Iceberg integration tables on S3, ensuring ACID compliance, schema evolution, and optimized performance. Developed business unit-specific views on top of validated data using Trino for optimized data access-Automated ingestion workflows with schema validation, S2T mapping, data quality checks, and metadata-driven execution using Airflow. Enabled high-performance analytics using Presto/Trino SQL through Starburst, and supporting federated, distributed querying across the data lake.

Education

• Ajay Binay Institute of Technology, (ABIT), Cuttack

  Engineer’s Degree, Electrical and Electronics Engineering

  2012 — 2016

• Centre for Development of Advanced Computing (C-DAC)

  PG, Big Data Analytics

  2019 — 2019