Purushotham T.
Senior Data Engineer at Eliassen Group
- Role
- Senior Data Engineer at Eliassen Group
- Location
- Charlotte, NC, US
- LinkedIn followers
- 500 followers
About Purushotham T.
Senior AWS Data Engineer with 8+ years of experience in IT industry as Hadoop and spark developer and a year of experience as SQL developer.Competencies: Hadoop, Apache Spark, Hive, Sqoop, Oozie, SQL, Scala, Python, Pyspark, AWS, EMR, S3, Glue, EC2
Experience
Senior Data Engineer
Apr 2020 — Present · Charlotte, NC, US
Extracting data from a variety of RDBMS sources like Oracle, MySQL and storing in Hadoop Distributed File System.• Involved in querying and analysing large amount of data on Hadoop HDFS using Hive and Spark.• Involved in designing and developing ETL solutions to support efficient data analytic models which include creating derived and business ready Datasets.• Troubleshooting production support issues post-deployment and come up with solutions as required. • Worked on fine tuning Spark applications for high performance and throughput.• Involved in upgrading the cluster from HDP 2.6.3 to HDP 3.1.5• Migrated codebase and made relevant enhancements and code changes to support 3.1.5 environment.• Involved in copying the history data of multiple sources across the cluster using Hadoop distcp.• Participated in covid-19 related projects to help the business team analyze its impact economically.• Worked with data visualization team closely to help them create dashboards on top of the materialized views created.• Assisted data scientists to convert and scale algorithms developed in python into spark and run-in distributed environment.• Migrated big data applications from on-premises Hadoop to AWS.• Created IAM roles and policies to limit privileges according to security requirements. • Implemented usage of Amazon EMR for processing Big Data across a Hadoop Cluster of virtual servers on Amazon Elastic Compute Cloud (EC2) and Amazon Simple Storage Service (S3).• Installed application on AWS EC2 instances, used EMR for computing and configured the storage on S3 buckets.• Developed terraform scripts to automate and orchestrate EMR jobs using step functions and triggered them using event bridge.• Used Kafka S3 source and S3sink connectors to produce data into Kafka topics from s3 and consume data from Kafka and write to S3 respectively.• Developed AWS lambda functions which run in response to an event like change in s3 bucket.
Education
University of the Cumberlands
Doctor of Philosophy - PhD, Information Technology
2019 — 2022
Jawaharlal Nehru Technological University
Bachelor's degree, Computer Science
2010 — 2014
University of Houston-Clear Lake
Master's degree, Computer Software Engineering
2014 — 2016
Find verified contacts for anyone on LinkedIn
Unifers gives sales teams verified emails and direct dials, enriched profiles, and outreach that lands in the inbox.
Free plan included · No credit card required
This profile is compiled from publicly available professional sources. Unifers is not affiliated with or endorsed by LinkedIn. Request removal of this profile.