Vaishu Shree
Senior Etl Engineer @Macy's
Signup · Get unlimited contacts
WORK HISTORY
Senior Etl Engineer @Macy's
NY, US
Design, Develop and test ETL Processes in AWS Glue to migrate Campaign data from external sources like S3, ORC/Parquet/Text Files into AWS Redshift. Implemented AWS IAM for managing the user permissions of applications that runs on EC2 instances. Worked on AWS EMR to run spark and hive applications. Moved on-prem jobs to run on AWS cloud. Hands on experience working with snowflake to move the data from s3 to snowflake vice versa. Worked on performance tuning of snowflake ETL jobs and implemented row level security solution in snowflake. Worked with kafka and spark structured streaming to build sales pipeline that support reporting needs. Hands on experience working with Databricks and Delta tables. Deployed applications onto AWS lambda with http triggers and integrated them with API Gateway Developed multiple ETL Hive scripts for data cleansing and transformations for data. Developed spark applications in python (PySpark) on distributed environment to load huge number of CSV files with different schema in to Hive ORC tables. Exported data from Hive to AWS s3 bucket for further near real time analytics. Ingested data in real time from Apache Kafka to Hive and HDFS. Developed the streaming applications using spark structured streaming, Kafka, and s3 integration project to do a real-time data analysis. Use of Sqoop to import and export data from RDBMS to HDFS and vice-versa. Exporting data to Teradata using SQOOP. Comparing the results of traditional system to Hadoop environment to identify any differences and fix them by finding the route cause. Create a complete processing engine, based on Hortonworks distribution, enhanced to performance. Analyzed the sql scripts and designed it by using PySpark SQL for faster performance. Implemented Kerberos and Ranger security Authentication protocol for existing cluster.
EDUCATION
JNTUH College of Engineering Hyderabad
Bachelor of Technology - BTech, Computer Science
ABOUT VAISHU SHREE
Overall 10 years of technical IT experience in all phases of Software Development Life Cycle (SDLC) with skills in data analysis, design, development, testing and deployment of software systems. 9+ years of industrial experience in Big Data analytics, Data manipulation, using Hadoop Eco system tools Map - Reduce, HDFS, Yarn/MRv2, Pig, Hive, HDFS, HBase, Spark, Kafka, Flume, Sqoop, Flume, Oozie, Avro, Sqoop, AWS, Spring Boot, Spark integration with Cassandra, Avro, Solr and ZookeeperExperience in developing data pipelines using AWS services including EC2, S3, Redshift, Glue, Lambda functions, Step functions, cloud Watch, SNS, Dynamo, SQSData Engineer knowledge seeker, working on improving my machine learning and statistical skills to deal with different types and sizes of data. The aim of my career is to optimize already found solutions and find effective mathematical solutions to the business problem using machine learning algorithms.My current focus is to find the best way to work with large datasets using Spark and Python, to optimally use the computing power of the available machine memoryOver the years, I diverse my skill sets in software, digital marketing, and finance fields. My roles changed from a software engineer to senior data analyst.Key Skills :Python, EMR, Snowflake, Big Data, Salesforce, Machine Learning, StatisticPyspark, AWS S3, EC2, GCP, R, SQL, Tableau, RDBMS, Oracle VirtualBox, Hadoop, Map Reduce, HDFS, Hive, Spring Boot, Cassandra, Swamp, Data Lake, Sqoop, Oozie, SQL, Kafka, Spark, Scala, Java, GitHub, Talend Big Data Integration, Solr, Impala.Email: v••••••••@gmail.com
This profile is compiled from publicly available professional sources. Unifers is not affiliated with or endorsed by LinkedIn. Request removal of this profile.