Deekshith Reddy Gopidi

About Deekshith Reddy Gopidi

Data enthusiast with 8 years of Experience as a Data/Software Engineer with experience in…

Experience

1. Senior Data Engineer

   Deutsche Bank

   Sep 2023 — Present · Cary, NC, US

   Involved in writing Spark applications using Python to perform various data cleansing, validation, transformation, and summarization activities according to the requirement.Used Spark API over Hortonworks Hadoop YARN to perform analytics on data in Hive.Using Spark Context, Spark-SQL, Data Frame, Pair RDD\'s, and Spark YARN, improved the performance and optimization of current Hadoop methods.Performed transformations, cleaning, and filtering on imported data using Hive, Map Reduce, and loaded final data into HDFS.Involved in converting Hive queries into spark transformations using Spark RDDs, PySpark and Scala.Exploring with Spark improving the performance and optimization of the existing algorithms in Hadoop using Spark Context, Spark-SQL, Data Frame, Pair RDD\'s, Spark YARN.Imported the data from various sources like HDFS/HBase into Spark RDD and developed a data pipeline using Kafka and Storm to store data into HDFS.Used Spark streaming to receive real time data from Kafka and store the stream data to HDFS using. Scala and NoSQL databases such as HBase and Cassandra.Developed Spark code using PySpark and Scala and Spark-SQL/Streaming for faster testing and processing of data.Executed Hive queries on Parquet tables stored in Hive to perform data analysis to meet the business requirements.Configured Oozie workflow to run multiple Hive and Pig jobs which run independently with time and data availability.