Sai B

About Sai B

Having 8+ years of over all experience as a Data engineer with expertise in Big Data, Cloud Technologies and Hadoop components like HDFS, Map-Reduce, Yarn, Apache Pig, Hive, Sqoop, WOOPRA (Web-Analytic Application), shell scripting, Kafka and Spark in Scala.An aspirational and results-oriented professional with a track record of developing large-scale data processing systems and data warehouse solutions for data analytics.

Experience

1. Senior Data Engineer

   Experian

   Sep 2020 — Present

   Expertise in designing and deployment of Hadoop cluster and different Big Data analytic tools including Pig, Hive, HBase, Oozie, Sqoop, Flume, Spark, Impala. • Implemented advanced procedures like text analytics and processing using the in-memory computing capabilities like Apache Spark written in python • Implemented Spark using python and Spark SQL for faster testing and processing of data. • Involved in converting Hive/SQL queries into Spark transformations using Spark RDDs, Scala. • Worked with Spark to create structured data from the pool of unstructured data received. • Implemented intermediate functionalities like events or records count from the flume sinks or Kafka topics by writing Spark programs in java and python. • Documented the requirements including the available code which should be implemented using Spark, Hive, HDFS.• Chosen and produced information into csv records and put away them into AWS S3 by utilizing AWS EC2 and afterward organized and put away in AWS Redshift. • Extract Real time feed using Kafka and Spark Streaming and convert it to RDD and process data in the form of Data Frame and save the data as Parquet format in HDFS. • Experienced in transferring Streaming data, data from different data sources into HDFS, No SQL databases • Created ETL Mapping with Talend Integration Suite to pull data from Source, apply transformations, and load data into target database.• Used PySpark and Pandas to calculate the moving average and RSI score of the stocks and generated them into data warehouse.• Fostered the clump contents to get the information from AWS S3 stockpiling and do required changes in Scala utilizing Spark system.• Chipped away at a python content to extricate information from Netezza data sets and move it to AWS S3.• Developed a PySpark program that writes dataframes to HDFS as avro files. Environment: Hadoop, Hive, Flume, Map Reduce, Sqoop, Kafka, Spark, Yarn, Cassandra, Oozie, shell Scripting, Scala, Maven, MySQL

Education

• JNTUH College of Engineering Hyderabad

  Bachelor's degree, Computer Science