Vasant Deware

About Vasant Deware

Proficient Data Engineer with over 3.5 years of experience in designing, developing, and…

Experience

1. Specialist Programmer Big Data

   Infosys

   Aug 2022 — Present

   Project description : Architected a scalable data ingestion pipeline utilizing Apache Spark and Python, processing diverse file formats, and accelerating asset-level data availability by 60% for downstream analytics. • Developed robust batch processing logic with dynamic file discovery, ingestion horizon control, and atomic asset handling, reducing manual intervention by 90% • Built custom asset control modules to enforce scheduling rules, early/late file detection, and failure actions using JSON-driven configurations. • Integrated error handling and alerting mechanisms using custom exception classes and mailer alerts to ensure traceability and operational resilience, engineered dynamic table creation routines for raw and processed layers, supporting schema evolution and ingestion consistency. • Implemented file level ingestion orchestration enhanced metadata enrichment, ingestion buffers and holiday/weekend scheduling logic,leveraged Spark JDBC connectors to retrieve metadata from relational databases and convert results to pandas for configuration-driven ingestion. • Designed reusable helper classes for file pattern matching, folder path resolution, and OS-level refresh logic, improving code maintainability,followed best practices in logging, exception handling and configuration management, resulting in a production-grade ingestion pipeline