Rohith C.

Experience

1. Data Engineer

   American Express

   Jun 2021 — Present

   Cornerstone & LUMIDesigned and implemented ELT pipelines on Google Cloud Platform (GCP) using BigQuery, Cloud Data Fusion, and Dataflow. Developed real-time streaming data pipelines using Google Pub/Sub and Dataflow (Apache Beam) for event-driven analytics. Orchestrated and automated data workflows using Cloud Composer (Apache Airflow), ensuring smooth job execution and monitoring. Built distributed data processing frameworks with PySpark on GCP Dataproc, reducing data transformation time. Partnered with business teams to optimize data warehouse schemas in BigQuery, improving query efficiency. Built data lake solutions on GCP using Cloud Storage and Dataproc, optimizing storage and compute costs. Designed data warehouse schemas in BigQuery, implementing partitioning and clustering for performance optimization. Developed streaming data pipelines using Pub/Sub, Dataflow, and Apache Beam for real-time analytics. Worked on Kafka to produce the streamed data into topics and consumed that data. Implemented Spark using Python and Spark SQL for faster processing of data and Worked on migrating MapReduce programs into Spark transformations using PySpark. Created Hive tables to load the data and wrote Hive queries to analyze the data. Developed Shell scripts to automate routine tasks, increasing operational efficiency and reducing manual intervention in production environments.

Education

• National Institute of Technology Jamshedpur

  Bachelor's degree

• Clemson University

  Master's degree