Venkat

About Venkat

Data Engineer knowledge seeker, working on improving my machine learning and statistical…

Experience

1. Senior Big Data Engineer

   LendingTree

   Mar 2020 — Present

   Designing the business requirement collection approach based on the project scope and SDLC methodology. •Loading data from different sources to a data warehouse to perform some data aggregations for business Intelligence using python. •Designed and implemented Sqoop for the incremental job to read data from DB2 and load to Hive tables and connected to Tableau for generating interactive reports using Hive server2. •Used Sqoop to channel data from different sources of HDFS and RDBMS. •Developed Spark applications using Pyspark and Spark-SQL for data extraction, transformation, and aggregation from multiple file formats. •Collected data using Spark Streaming from AWS S3 bucket in near-real-time and performs necessary Transformations and Aggregation on the fly to build the common learner data model and persists the data in HDFS. •Installing, configuring, and maintaining Data Pipelines •Transforming business problems into Big Data solutions and define Big Data strategy and Roadmap. •Files extracted from Hadoop and dropped on daily hourly basis into S3 •Used SSIS to build automated multi-dimensional cubes. •Led the installation, configuration and deployment of product soft wares on new edge nodes that connect and contact Kafka cluster for data acquisition. •Created multiple dashboards in tableau for multiple business needs. •Prepared and uploaded SSRS reports. Manages database and SSRS permissions. Environment:Cloudera Manager (CDH5),Hadoop, Hive, S3, Kafka, Pyspark, HDFS, NiFi, Pig, Scrum, Git, Sqoop, Oozie. Pyspark, Informatica, Tableau, OLTP, OLAP, SQL Server, Python, Shell Scripting, HBase, Cassandra, Informatica, XML,Unix.