Santhosh G

About Santhosh G

8+ years of professional experience in Requirements Analysis, Design, Development and…

Experience

1. Sr Hadoop Developer

   State Street

   Mar 2017 — Present

   Convert existing Mart creation application from MYSQL to Hive.• Performed dynamic transformation on Hive external tables to create the data mart.• Developed data pipeline using Spark, Hive, Sqoop and Kafka to ingest data into Hadoop platform for analysis.• Developed Spark code using Scala and Spark-SQL/Streaming for streaming of data. • Performed Data Ingestion from multiple internal clients using Apache Kafka.• Ingested data into HDFS using Sqoop and analyzed the data using Spark (Dataframes,DataSets and Spark-SQL), and series of Hive scripts to produce summarized results from Hadoop to downstream systems.• Used Spark Dataframes,DataSets, Spark-SQL extensively• Developed simple to complex Spark streaming jobs using Scala for real time processing• Created microservice to read from kafka stream and storing it in s3 or file system.• Developed an application from scratch and wrote code on KafkaProducer and Consumer as per the requirement.• Implemented installation and configuration of multi-node cluster on the cloud using Amazon Web Services (AWS) on EC2. • Used cloud computing on the multi-node cluster and deployed Hadoop application on cloud S3 and used Elastic Map Reduce (EMR) to run a Map-reduce.• Reading the file from s3 Storage using spark streaming and storing to s3 inparquet format and query using Athena.• Created Java component for Athena to query back from s3 parquet files.• Exported the data sets from Hive to RDBMS using Oozie.• Implemented business logic in Hive and written UDF’s to process the data for analysis.• Implemented dynamic partitions, bucketing and compression techniques in Hive External Tables and optimized the query performance.• Created HBase tables and column families to store the user event data.• Used Cloudera Manager to manage and monitor Hadoop Stack.• Actively participated in analysis, requirement gathering and development of various POC\'s for various business units.