Avinash Reddy
Data Engineer / Spark Developer | Hadoop | Hive | Sqoop | PySpark| Spark Streaming | Kafka |AWS(S3,EMR,Ec2,Glue,Athena,Dynamo DB and Redshift) | Azure Databricks | HBASE | Cassandra |Snowflake| Airflow|
- Role
- Data Engineer Big Data Engineer at Thomson Reuters
- Location
- Norfolk, VA, US
- LinkedIn followers
- 500 followers
About Avinash Reddy
So I\'ve been working with big data technologies for more than seven years. So I\'m having…
Experience
Data Engineer Big Data Engineer
Nov 2021 — Present · Eagan, MN, US
o Worked in Azure environment for development and deployment of Custom Hadoop Applications.• Developed workflow in Oozie to manage and schedule jobs on Hadoop cluster to trigger daily, weekly and monthly batch cycles.• Configured Hadoop tools like Hive, Pig, Zookeeper, Flume, Impala and Sqoop.• Deployed the initial Azure components like Azure Virtual Networks, Azure Application Gateway, Azure Storage and Affinity groups.• Responsible to manage data coming from different sources through Kafka.• Working in big data technologies like spark, Scala, Hive, Hadoop cluster (Cloudera platform). • Making a data pipelining with help Data Fabric job SQOOP, SPARK, Scala and KAFKA. Parallel working in data side oracle and MYSQL server for data designing to source to target.• Write programs using Spark to move data from Storage input location to output location by running data loading, validation, and transformation to the data.• Designed highly efficient data model for optimizing large-scale queries utilizing Hive complex datatypes and Parquet file format.• Used Cloudera Manager continuous monitoring and managing of the Hadoop cluster for working application teams to install operating system, Hadoop updates, patches, version upgrades as required.• Developed data pipelines using Sqoop, Pig and Hive to ingest customer member data, clinical, biometrics, lab and claims data into HDFS to perform data analytics.• Analyzed Teradata procedure and imported all the data from Teradata to My SQL Database for Hive QL queries information for developing Hive Queries which consist of UDF’s where we don’t have some of the default functions in Hive.• Worked in Azure environment for development and deployment of Custom Hadoop Applications. • Developed workflow in Oozie to manage and schedule jobs on Hadoop cluster to trigger daily, weekly and monthly batch cycles. • Configured Hadoop tools like Hive, Pig, Zookeeper, Flume, Impala and Sqoop.
Find verified contacts for anyone on LinkedIn
Unifers gives sales teams verified emails and direct dials, enriched profiles, and outreach that lands in the inbox.
Free plan included · No credit card required
This profile is compiled from publicly available professional sources. Unifers is not affiliated with or endorsed by LinkedIn. Request removal of this profile.