Harshit Gupta

About Harshit Gupta

Data Engineer with experience in Python, PySpark, SQL, and GCP for building data-driven solutions.• Currently working on Datalake Project for American Express, optimizing workflows and orchestrating Spark jobs with GCP Cloud Composer (Airflow DAGs).• Expertise in extracting and processing data from GCP BigQuery and transferring it to mainframe systems.• Strong background in developing robust data solutions using Apache Spark and Dataproc, achieving a 25% increase in job performance.• Skilled in automating workflows, writing Pytest scripts, and employing BDD testing to ensure code quality.• Proficient in full-stack web development with projects featuring CRUD operations, authentication, and RESTful APIs.• Certified in Google Cloud Platform (Data Engineer, Big Data) with a solid understanding of cloud infrastructure.• Strong knowledge of Agile methodology, with hands-on experience using tools like Jira and Rally for project management.

Experience

1. Engineer Ii

   American Express

   Jan 2023 — Present · Bengaluru, IN

   Developed PySpark scripts to extract data from GCP BigQuery.* Utilized Airflow DAGs in GCP Cloud Composer to automate workflows * Employed Spark Clusters for efficient job orchestration and cluster management.* Used Dataproc to optimize job performance and enhance efficiency.* Implemented Pytest scripts to ensure script reliability, code quality and robustness.* Utilized mainframe systems to validate data file accuracy* Leveraged Agile methodologies and related tools (Rally and Jira) to optimize workflows.Skills Acquired: Python3, PySpark, SQL, Airflow, Google Cloud Platform - BigQuery, Cloud Composer, Dataproc

Education

• Chinmaya Vidyalaya, Bokaro Steel City

  CBSE(12th)

  2015 — 2017

• Carmel School, Giridih

  ICSE(10th)

  2003 — 2015

• Visvesvaraya Technological University

  B.E. in Electronics and Communication Engineering, Engineering