Surya R

Aws Data Engineer @Chewy

Used AWS Athena extensively to ingest structured data from S3 into other systems such as RedShift or to produce reports. • The Spark-Streaming APIs were used to conduct on-the-fly transformations and actions for creating the common learner data model, which receives data from Kinesis in near real time. • Performed end- to-end Architecture & implementation assessment of various AWS services like Amazon EMR, Redshift, S3, Athena, Glue and Kinesis. • Hive As the primary query engine of EMR, we have built external table schemas for the data being processed. • AWS RDS (Relational database services) was created to serve as a Hive meta store, and it was possible to integrate the meta data from 20 EMR clusters into a single RDS, avoiding data loss even if the EMR was terminated. • Involved in the development of a shell script that collects and stores logs created by users in AWS S3 (Simple storage service) buckets. This contains a record of all user actions and is a good indicator of security to detect cluster termination and safeguard data integrity. • Partitioning and Bucketing ideas were implemented in the Apache Hive database, which increases query retrieval performance. • Using AWS Glue, I designed and deployed ETL pipelines on S3 parquet files in a data lake. • Created a cloud formation template in JSON format to leverage content delivery with cross-region replication through Amazon Virtual Private Cloud • AWS Code Commit Repository was used to save programming logics and scripts and then replicate them to new clusters. • Used the Multi-node Redshift technology to implement Columnar Data Storage, Advanced Compression, and Massive Parallel Processing • Worked with Snowflake cloud data warehouse and AWS S3 bucket for integrating data from multiple source system which include loading nested JSON formatted data into snowflake table. • Worked on the code transfer of a quality monitoring program from AWS EC2 to AWS Lambda.