Rithwik K

Experience

1. Sr Data Engineer

   Huron

   Sep 2022 — Present

   Created batch Data pipelines in order to extract data from Amazon Aurora and Amazon Dynamo DB and load it to S3 data lake using Glue jobs and AWS Lambda.• Implemented a Glue Crawler to glue catalog which fetches data from S3 which allowed the creation of tables and subsequent querying of the data in Athena.• Involved in Designing and developing Batch pipelines using Glue Jobs with Lambda triggers to extract data from S3 and stored that in AWS Redshift. • Extensively involved in handling large datasets using Partitions and Spark in Memory capabilities, Broadcasts in Spark, Effective and Efficient Joins, Transformations and other during Ingestion process.• Designed EMR cluster for Data ingestion and made use of DBT for transforming data into Redshift.• Implemented automatic scaling of EMR instances on data processing and deployed multi-cluster env.• Utilized Apache Spark to write RDD transformations, actions, Data Frames, and case classes to process required input data, converting RDDs to Data Frames and employing various Spark tools.• Created real-time streaming pipelines to ingest data from applications using Amazon Kinesis live streaming capacity, to consume data from application APIs for making it available to other AWS services.• Created several Spark pipelines in Databricks for monitoring the feature engineering pipelines and scheduling notebook jobs for daily sync jobs.• Responsible for migrating the real time data process monitoring from Redshift to an Athena-based architecture, which reduces the workload on Redshift cluster.• Developed Python Lambda functions for AWS Lambda and executed Python scripts for data transformations and analytics on big data sets in EMR clusters and AWS Kinesis data streams.• Implemented build run quality with Python library BOTO3 so pipelines do not fail which reduced effort by 70%.