Prathap K

Experience

1. Data Engineer Ii

   O\'neil Global Advisors

   Jul 2023 — Present · Los Angeles, CA, US

   Converted legacy jobs into modular Airflow DAGs, enhancing orchestration and maintainability.• Added functionality to track test progress and handle user-triggered cancellations for reliability.• Built a scalable database connection engine for efficient retrieval of metrics and test IDs.• Stabilized end-to-end pipelines by automating file uploads to AWS S3 and integrating with GitLab.• Decommissioned Hortonworks and migrated processes to Kubernetes Pod Operator.• Debugged complex DataFrame expectation errors, ensuring high data accuracy.• Automated workflows to ingest files from S3, create backup tables, and load data into Amazon Redshift using the COPY command.• Migrated workflows from PySpark to Python, reducing infrastructure overhead and improving runtime performance.• Implemented rigorous data validation checks to guarantee record accuracy and integrity.• Designed archival processes to store source files in S3, supporting compliance and cost-efficiency• Eliminated EMR usage in volatility DAGs by re-engineering them in Python, reducing costs and simplifying maintenance.• Introduced multi-threading for faster data processing and optimized performance.• Executed large-scale historical loads (2016 onwards) while maintaining incremental loads.• Designed and implemented an Airflow-based archival process to separate historical vs incremental pipelines.• Added timestamp logic to incremental files copied into history folders, improving traceability.• Migrated DAG execution to Kubernetes Pod Operator, leveraging containerization for scalability.• Containerized ETL workflows by packaging code into Docker images and deploying via Kubernetes Pods• Developed AWS Glue jobs to automate transformations and optimize data flow.• Built scripts for chunked and incremental data loads, ensuring efficient handling of large datasets

Education

• Jawaharlal Nehru Technological University

  Bachelor of Technology - BTech, Computer Science

  2013 — 2017

• University of North Texas

  Masters, Data Science