Vasavi A

Experience

1. Data Engineer

   Lowe's Companies, Inc.

   Nov 2021 — Present

   Implemented Spark using Python and Spark SQL for faster processing of data and Worked on migrating Map Reduce programs into Spark transformations using Spark and Scala, initially done using python (PySpark)• Designed and implemented Sqoop for the incremental job to read data from DB2 and load to Hive tables and connected to Tableau for generating interactive reports using Hive server2.• Worked on Implementation of a log producer in Scala that watches for application logs transform incremental log and sends them to a Kafka and Zookeeper based log collection platform• Experience other Hadoop ecosystem tools in jobs such as Zookeeper, Oozie, Impala• Developed a Python Script to load the CSV files into the S3 buckets and created AWS S3buckets, performed folder management in each bucket, managed logs and objects within each bucket.• Good Exposure on Map Reduce programming using Java, PIG Latin Scripting and Distributed Application and HDFS.• Worked with NoSQL databases like HBase in creating tables to load large sets of semi structureddata coming from various sources.