Mike P

About Mike P

8+ years of experience in IT industry with strong emphasis on Object Oriented Analysis,…

Experience

1. Hadoop Spark Developer

   Inovalon

   Mar 2014 — Present

   Responsibilities: • Expertise in designing and deployment of Hadoop cluster and different Big Data analytic tools including Pig, Hive, HBase, Oozie, ZooKeeper, SQOOP, flume, Spark, Impala, Cassandra with Horton work Distribution. • Installed Hadoop, Map Reduce, HDFS, AWS and developed multiple MapReduce jobs in PIG and Hive for data cleaning and pre-processing. • Assisted in upgrading, configuration and maintenance of various Hadoop infrastructures like Pig, Hive, and Hbase. • Used Spark API over Hortonworks Hadoop YARN to perform analytics on data in Hive. • Exploring with the Spark improving the performance and optimization of the existing algorithms in Hadoop using Spark Context, Spark-SQL, Data Frame, Pair RDD\'s, Spark YARN. • Developed Spark code using scala and Spark-SQL/Streaming for faster testing and processing of data. • Import the data from different sources like HDFS/Hbase into Spark RDD. • POC on Single Member Debug on Hive/Hbase and Spark. • Configured deployed and maintained multi-node Dev and Test Kafka Clusters. • Performed transformations, cleaning and filtering on imported data using Hive, Map Reduce, and loaded final data into HDFS. • Load the data into Spark RDD and do in memory data Computation to generate the Output response. • Loading Data into Hbase using Bulk Load and Non-bulk load. • Experience in Oozie and workflow scheduler to manage hadoop jobs by Direct Acyclic Graph (DAG) of actions with control flows. • Expertise in different data Modeling and Data Warehouse design and development. Environment: Hadoop, HDFS, MapReduce, Pig, Hive, Sqoop, Kfaka, Solr, HBase, Oozie, Flume, Spark – Streaming/SQL, java, SQL Scripting, Linux Shell Scripting, Cloudera CDH 5.X.