Shivaram Kothapally

About Shivaram Kothapally

A highly skilled data engineer with expertise in AWS Redshift, Athena, Workflows, Lambda, and Glue Crawlers. Proficient in SQL, Python, and other programming languages with a proven track record of designing and implementing scalable and performant data pipelines while ensuring data quality and integrity. Strong analytical and problem-solving skills to troubleshoot complex issues and provide effective solutions. Passionate about leveraging technology to solve complex business problems and staying up to date with emerging trends in data engineering.

Experience

1. Software Developer

   Amazon Web Services (AWS)

   Jun 2022 — Present · VA, US

   Designed and implemented ETL jobs for data processing using various AWS services, including Glue Crawlers for automatic schema discovery, S3 buckets for storage, and Athena for data transformation which led to a 90% reduction in processing time for the application. •Designed and created a serverless AWS Glue ETL pipeline to transform and load data from S3 buckets into Amazon Redshift, using CloudWatch events to trigger scheduled Lambda functions that automate the ETL process, which was monitored with CloudWatch Metrics. •Converted and delivered 300M+ CSV records to 3.5M Parquet files using AWS Glue Jobs and Python scripts. •Redesigned an architecture and changed the triggering of long-running ETL jobs from Lambda functions to Workflows. •Developed and maintained ETL pipelines to extract data from various sources and load it into Redshift. •Troubleshooting and resolving issues related to Redshift cluster performance, capacity, and security. •Participating in code reviews and contributing to the development of Redshift-related tools and frameworks •Designed various dashboards for the Code Quality team to have insights into developer\'s productivity based on the number of code reviews, number of bugs logged, number of bugs resolved, and number of bugs reopened using Metabase and Python. •Developed generic framework by using spark and bash scripts which involves data ingestion, processing of different data formats(CSV, JASON, PARQUET), and feeding the data into Hive tables. •Chunking the data to convert them from larger data sets to smaller chunks using Python scripts which will be useful for faster data processing. •Designed and implemented Python scripts to fetch build data and test results from Jenkins to pull into AWS S3 and Redshift to enable faster reporting.

Education

• Northwest Missouri State University

  Master's degree, Computer Science

  2021 — 2022

• Gurunanak institute of technology

  Bachelor of Technology - BTech, Ece

  2015 — 2019