Ahmed Hussain

Experience

1. Big Data Developer

   Visa

   Oct 2018 — Present · Foster City, CA, US

   Gather business requirements and design and develop data ingestion layer and presentation layer.• Highly motivated and versatile team player with the ability to work independently & adapt quickly to the environment.• Performed ad-hoc queries on structured data using Hive QL and used Partition, bucketing techniques and joins with Hive for faster data access.• Develop Hive queries on external tables in order to perform various analysis.• Used HUE for running Hive queries. Created partitions according to data using Hive to improve performance. • Importing and exporting data into HDFS and HIVE using Sqoop. • Responsible for loading data from UNIX file systems to HDFS. Installed and configured Hive and written Hive UDFs. • Developed Spark Applications by using Scala and Implemented Apache Spark data processing project to handle data from various RDBMS and Streaming sources.• Used Spark SQL on data frames to access hive tables into spark for faster processing of data• Designed and developed jobs to validate the data post migration such as reporting fields from source and designation systems using Spark SQL RDDs and Data Frames/Datasets.• Used Spark Data Frame API to process Structured and Semi Structured files and load them into S3 Bucket.• Used Spark Data Frames Operations to perform required Validations in the data and to perform analytics on the Hive data.• Used Different Spark Modules like Spark core, Spark SQL, Spark Streaming, Spark Data sets and Data frames.• Worked on apache Solr for indexing and load balanced querying to search for specific data in larger Worked on Spark Streaming and Structured Spark streaming using Apache Kafka for real time data processing. • Responsible for developing multiple Kafka Producers and Consumers from scratch as per the software requirement specifications.