Andy Rahman
- Role
- Spark Developer at General Motors
- Location
- New York, NY, US
- LinkedIn followers
- 500 followers
About Andy Rahman
Around 7+ years of data processing and analysis experience, handled high volume data and…
Experience
Spark Developer
Apr 2017 — Present
Designed and developed various modules in Hadoop Big Data platform and processed data using Sql, MapReduce, Hive, Sqoop, Kafka and Oozie. Responsible for building scalable distributed data solutions using Hadoop. Experience in Job management using Fair scheduler and Developed job processing scripts using Oozie workflow. Used Spark-Streaming APIs to perform necessary transformations and actions on the fly for building the common learner data model which gets the data from Kafka in near real time and Persists into Cassandra. Configured deployed and maintained multi-node Dev and Test Kafka Clusters. Developed Spark scripts by using Scala shell commands as per the requirement. Used Spark API over Cloudera Hadoop YARN to perform analytics on data in Hive. Developed Scala scripts, UDFFs using both Data frames/SQL/Data sets and RDD/MapReduce in Spark 1.6 for Data Aggregation, queries and writing data back into OLTP system through Sqoop. Experienced in performance tuning of Spark Applications for setting right Batch Interval time, correct level of Parallelism and memory tuning. Optimizing of existing algorithms in Hadoop using Spark Context, Spark-SQL, Data Frames and Pair RDD\'s. Implemented ELK (Elastic Search, Log stash, Kibana) stack to collect and analyze the logs produced by the spark cluster.
Find verified contacts for anyone on LinkedIn
Unifers gives sales teams verified emails and direct dials, enriched profiles, and outreach that lands in the inbox.
Free plan included · No credit card required
This profile is compiled from publicly available professional sources. Unifers is not affiliated with or endorsed by LinkedIn. Request removal of this profile.