Chintan Rajesh Pamnani

Software Engineer in Voice @Comcast

Integrated Great Expectations Python framework with Apache Airflow to schedule and monitor batch-oriented workflows from every data source in the ingest flow and ensured the quality of the data is good while it is loaded from one data source to another• Seamlessly integrated the new framework in the ingest flow to make the system more reliable• Developed ETL (Extract, Transform, Load) pipelines to process large amounts of movie metadata, managing persistence and retrieval of data in JSON & parquet formats to and from S3 buckets• Engaged in troubleshooting ETL-related issues to ensure data accuracy and data consistency throughout the pipeline• Helped reduce the cost of the overall system by 20% in the cloud services area by eliminating some S3 buckets and making the ingest flow more agile