Supraja Ganne

Experience

1. Big Data Engineer

   Capital One

   Jan 2023 — Present

   Imported data from various sources into Spark data frames for processing. • Responsible for the design, implementation, and architecture of very large-scale data intelligence solutions around big data platforms.• Involved in all the phases of SDLC including Requirements, Collection, Design & Analysis of the Customer Specifications from Business Analyst.• Used Jenkins, Git Stash, and Ansible like CI/CD tools to make daily builds and deploys. Used JSON for validation, and documentation data required by a given application.• Used AWS Infrastructure and features of AWS like S3, EC2, Glue, Lambda, CloudWatch. Worked on reading multiple data formats on HDFS using Python.• Strong development background in creating pipelines, data flows and complex data transformations and manipulations using AWS Databricks.• Executed Spark workloads on transient EMR cluster.• Implemented Spark using Python (PySpark) and Spark SQL for faster testing and processing of data. Load the data into Spark RDD and do in-memory data Computation.• Worked on troubleshooting spark application to make them more error tolerant.• Experienced in handling large datasets using Spark in Memory capabilities, using broadcasts variables in Spark, effective & efficient Joins, transformations, and other capabilities.• Have done end-to-end ETL testing.• Worked on scheduling jobs on Autosys.• Involved in converting Hive/SQL queries into Spark transformations using APIs like Spark SQL, Data Frames, and Python.• Worked with data in multiple file formats including Avro, Parquet, Sequence files, ORC, and Text/ CSV.• Writing Custom PySpark scripts to replicate the function of the Teradata code in the process of data transformation from Teradata to AWS one lake.• Experience working with an on-shore / off-shore model

Education

• KL University, Vaddeswaram

  Bachelor of Technology - BTech

• George Mason University

  Master of Science - MS