Sampa C

About Sampa C

Over 8+ years of IT experience in Analysis, Design, Development, and Big Data in Scala…

Experience

1. Data Engineer

   First Republic

   Sep 2021 — Present · New York, NY, US

   Used AWS Athena extensively to ingest structured data from S3 into other systems such as RedShift or to produce reports. • The Spark-Streaming APIs were used to conduct on-the-fly transformations and actions for creating the common learner data model, which receives data from Kinesis in near real-time. • Performed end-to-end Architecture & implementation assessment of various AWS services like Amazon EMR, Redshift, S3, Athena, Glue, and Kinesis. • Hive As the primary query engine of EMR, we have built external table schemas for the data being processed. • AWS RDS (Relational database services) was created to serve as a Hive meta store, and it was possible to integrate the metadata from 20 EMR clusters into a single RDS, avoiding data loss even if the EMR was terminated. • Involved in the development of a shell script that collects and stores logs created by users in AWS S3 (Simple storage service) buckets. This contains a record of all user actions and is a good indicator of security to detect cluster termination and safeguard data integrity. • Using AWS Glue, I designed and deployed ETL pipelines on S3 parquet files in a data lake. • Created a cloud formation template in JSON format to leverage content delivery with cross-region replication through Amazon Virtual Private Cloud • AWS Code Commit Repository was used to save programming logic and scripts and then replicate them to new clusters. • Used the Multi-node Redshift technology to implement Columnar Data Storage, Advanced Compression, and Massive Parallel Processing • Worked with Snowflake cloud data warehouse and AWS S3 bucket for integrating data from multiple source systems which include loading nested JSON formatted data into snowflake table. Environment: Amazon Web Services, Elastic Map Reduce cluster, EC2s, CloudFormation, Amazon S3, Amazon Redshift, Dynamo DB, Cloud Watch, Hive, Scala, Python, HBase, Apache Spark, Spark SQL, Shell Scripting, Tableau, Cloudera.