Diyya Q.
Technical Lead @JPMorganChase
Signup · Get unlimited contacts
WORK HISTORY
Technical Lead @JPMorganChase
Data Warehouse and Azure Synapse Analytics.Worked with spark to consume data from Kafka and convert that to common format using Scala. Worked extensively with importing metadata into Hive and migrated existing tables and applications to work on Hive and Spark. Converted existing MapReduce jobs into Spark transformations and actions using Spark RDDs, Data frames and Spark SQL APIs. Wrote new spark jobs in Scala to analyze the data of the customers and sales history. Involved in requirement analysis, design, coding and implementation phases of the project. Used Spark API over Hadoop YARN to perform analytics on data in Hive. Experience in both SQLContext and Spark Session. Developed Scala based Spark applications for performing data cleansing, data aggregation, de-normalization and data preparation needed for machine learning and reporting teams to consume. Worked on troubleshooting spark application to make them more error tolerant. Involved in HDFS maintenance and loading of structured and unstructured data and imported data from mainframe dataset to HDFS using Sqoop and written the PySpark Script to process the HDFS data. Used Spark API over Hadoop YARN to perform analytics on data in Hive. Extensively worked on the core and Spark SQL modules of Spark. Involved in Spark and Spark Streaming creating RDD\'s, applying operations -Transformation and Actions. Created partitioned tables and loaded data using both static partition and dynamic partition method. Implemented POC’s on migrating to Spark-Streaming to process the live data. Executed Hive queries on Parquet tables stored in Hive to perform data analysis to meet the business requirements. Worked on troubleshooting spark application to make them more error tolerant. Stored the output files for export onto HDFS and later these files are picked up by downstream systems. Load the data into Spark RDD and do in memory data Computation to generate the Output response.
EDUCATION
Albertus Magnus College
Master of Science - MS
Central Connecticut State University
Bachelor of Applied Science - BASc
ABOUT DIYYA Q.
Close to 9 years of IT experience as a Developer, Designer & platform integration experience using Hadoop Ecosystem. Experienced in Technical consulting and end - to-end delivery with data analysis, data modeling, data governance and design - development - implementation of solutions. Have experience in Apache Spark, Spark Streaming, Spark SQL and NoSQL databases like HBase, Cassandra, and MongoDB. Experience in developing Map Reduce Programs using Apache Hadoop for analyzing the big data as per the requirement. Worked with NoSQL databases like HBase, Cassandra and MongoDB for information extraction and place huge amount of data. Expertise in Big Data Ingestion/Integration Tool like flume, Kafka. Used Informatica Power Center for Extraction, Transformation, and Loading (ETL) of information from numerous sources like Flat files and Databases. Expertise in Developing Big data solutions using Data ingestion, Data Storage. Experience in cluster monitoring tools like Apache hue. Good experience in using Sqoop for traditional RDBMS data pulls. Strong experience in database skills in IBM- DB2, Oracle and Proficient in database development, including Constraints, Indexes, Views, Stored Procedures, Triggers and Cursors. Extensive use of Open-Source Software and Web/Application Servers like Eclipse 3.x IDE and Apache Tomcat 6.0. Experience in designing a component using UML Design-Use Case, Class, Sequence, and Development, Component diagrams for the requirements. Hands on experience on developing UDF, DATA Frames and SQL Queries in Spark SQL. Highly skilled in integrating Kafka with Spark streaming for high-speed data processing. Understanding of data storage and retrieval techniques, ETL, and databases, to include graph stores, relational databases, tuple stores. Logical and physical database designing like Tables, Constraints, Index, etc. using Erwin, ER Studio, TOAD Modeler and SQL Modeler. Experienced in writing Storm topology to accept the events from Kafka producer and emit into Cassandra DB. Capable at using AWS utilities such as EMR, S3 and Cloud watch to run and monitor Hadoop/Spark jobs on AWS. Good understanding and exposure to Python programming. Developed PL/SQL programs (Functions, Procedures, Packages and Triggers). Involved in reports development using reporting tools like Tableau. Used excel sheet, flat files, CSV files to generated Tableau ad hoc reports. Broad design, development and testing experience with Talend Integration Suite and knowledge in Performance tuning of mappings.
This profile is compiled from publicly available professional sources. Unifers is not affiliated with or endorsed by LinkedIn. Request removal of this profile.