Danish F.

Experience

1. Hadoop Developer

   Florida Blue

   Sep 2022 — Present · Jacksonville, FL, US

   Perform real time analysis on the incoming data on a consistent basis to avoid job failures. • Monitor production logs for errors, reporting and troubleshooting. Raise tickets for scheduling new jobs and modifying existing jobs in control-m, depending on the business requirements. • Implemented Spark using Scala and Spark SQL for faster and efficient testing and batch processing of data. • Developed shell scripts to read the file from the source path and load it into the HDFS target paths. • Modified existing shell scripts to call the spark process to parse and enrich the data. • Wrote queries in spark and Scala to import data into the SQL tables. Performed joins among tables. • Developed python programs to clean and pre-process the data coming from multiple sources. • Import data from Hadoop edge node and Linux/Unix file system to Hadoop File System (HDFS). Handles data in various formats such as Json and csv. • Hands on experience in writing Queries, Stored procedures, Functions and Triggers in SQL. • Support development, testing, and operations teams during new system deployments. • Supported Jenkins Deployments for production release. Developed Jenkins pipeline. • Used Sqoop to import the data from Hadoop Distributed File System (HDFS) to RDBMS. • Evaluate business requirements and prepare Technical Design Documents (TDD) with detailed design objectives. • Implemented Script for data optimizations and clean ups of bad records in data. • Coordinated with QA testers for end-to-end unit testing and post-production testing and issues. • Maintained regular communication with client and team members for keeping check on flow of project in each phase until successful production release. • Review test cases and collaborate with the QA testing team for implementation and development. • Managed and reviewed Hadoop log files.

Education

• Farmingdale State College

  Computer Programming and Info Systems