Pavan Reddy

Experience

1. Aws Data Engineer

   JPMorganChase

   May 2021 — Present

   Implemented Installation and configuration of multi-node cluster on Cloud using Amazon Web Services (AWS) on EC2. • Handled AWS Management Tools as Cloud watch and Cloud Trail. • Stored teh log files in AWS S3. Used versioning in S3 buckets where teh highly sensitive information is stored. • Experience in Developing Spark applications using Spark - SQL in Databricks for data extraction, transformation, and aggregation from multiple file formats for analyzing & transforming the data to uncover insights into the customer usage patterns. • Working experience with data streaming process with Kafka, Apache Spark, Hive. • Analysed the SQL scripts and designed the solution to implement using Scala. • Designed both 3NF data models for OLTP systems and dimensional data models using star and snowflake Schemas. • Optimized the PySpark jobs to run on Kubernetes Cluster for faster data processing. • Developed parallel reports using SQL and Python to validate the daily, monthly, and quarterly reports. • Designed and Developed Real time Stream processing Application using Spark, Kafka, Scala, and Hive to perform Streaming ETL and apply Machine Learning. • Imported data from AWS S3 into Spark RDD, Performed transformations and actions on RDD\'s. • Primarily Responsible for converting Manual Report system to fully automated CI/CD Data Pipeline that ingest data from different Marketing platform to AWS S3 data lake. • Designed and developed AWS architecture, Cloud migration, AWS EMR, DynamoDB, Redshift and event processing using lambda function • Conducted ETL Data Integration, Cleansing, and Transformations using AWS glue Spark script. • Worked on ETL Migration services by developing and deploying AWS Lambda functions for generating a serverless data pipeline which can be written to Glue Catalog and can be queried from Athena.