Rakesh T

About Rakesh T

Having 8+ years of experience as Big Data Engineer/ Data Engineer and Hadoop Developer including designing, developing and implementation of data models for enterprise - level applications and systems.• Experienced in managing Hadoop clusters and services using Cloudera Manager.• Experience in developing custom UDFs for Pig and Hive to incorporate methods and functionality of Python/Java into Pig Latin and HQL (HiveQL) and Used UDFs from Piggybank UDF Repository.• Expertise in using various Hadoop infrastructures such as Map Reduce, Pig, Hive, Zookeeper, Hbase, Sqoop, Oozie, Flume, Drill and spark for data storage and analysis.• Implemented various algorithms for analytics using Cassandra with Spark and Scala.• Experienced in Creating Vizboards for data visualization in Platfora for real - time dashboard on Hadoop.• Collected logs data from various sources and integrated in to HDFS using Flume.• Creative skills in developing elegant solutions to challenges related to pipeline engineering• Strong experience in core Java, Scala, SQL, PL/SQL and Restful web services.• Extensive knowledge in various reporting objects like Facts, Attributes, Hierarchies, Transformations, filters, prompts, calculated fields, Sets, Groups, Parameters etc, in Tableau experience in working with Flume and NiFi for loading log files into Hadoop.• Experienced in running query - using Impala and used BI tools to run ad-hoc queries directly on Hadoop.• Good experience in Oozie Framework and Automating daily import jobs.• Experienced in troubleshooting errors in Hbase Shell/API, Pig, Hive and map Reduce.• Highly experienced in importing and exporting data between HDFS and Relational Database Management systems using Sqoop.• Designed and implemented a product search service using Apache Solr.• Good Understanding of Azure Big data technologies like Azure Data Lake Analytics, Azure Data Lake Store, Azure Data Factory, Azure Databricks, and created POC in moving the data from flat files and SQL Server using U-SQL jobs.• Good knowledge in querying data from Cassandra for searching grouping and sorting.• Extensive hands-on experience in using distributed computing architectures such as AWS products (e.g. EC2, Redshift, EMR, and Elastic search), Hadoop, Python, Spark and effective use of Azure SQL Database, MapReduce, Hive, SQL and PySpark to solve big data type problems.

Experience

1. Sr Ml Data Infrastructure Engineer

   Arrow Electronics

   Jun 2019 — Present

   Worked in AWS environment for development and deployment of custom Hadoop applications.• Worked extensively on spark and MLlib to develop a regression model for cancer data.• Involved in Designing and Developing Enhancements of CSG using AWS APIS.• After the transformation of data is done, this transformed data is then moved to Spark cluster where the data is set to go live on to the application using Spark streaming and Kafka.• Created and maintained various DevOps related tools for the team such as provisioning scripts, deployment tools, and development and staging environments on AWS, Rack space and Cloud. • Hive tables are created as per requirement were Internal or External tables defined with appropriate static, dynamic partitions and bucketing, intended for efficiency.• Created data pipeline for different events of ingestion, aggregation and load consumer response data in AWS S3 bucket into Hive external tables in HDFS location to serve as feed for tableau dashboards.• Developed Python scripts, UDF\'s using both Data frames/SQL and RDD/MapReduce in Spark for Data Aggregation, queries and writing data back into RDBMS through Sqoop.• Consumed XML messages using Kafka and processed the xml file using Spark Streaming to capture UI updates.