Shafi Khan

Experience

1. Sr Data Engineer

   AT&T

   Mar 2022 — Present · Plano, TX, US

   Implemented Spark using Scala and utilizing Data frames and Spark SQL API for faster processing of data. • Ingested data from RDBMS and performed data transformations, and then export the transformed data to Cassandra as per the business requirement. • Performed wide, narrow transformations, actions like filter, Lookup, Join, count, etc. on Spark DataFrames. • Worked with Parquet files and Impala using PySpark, and Spark Streaming with RDDs and Data Frames. • Worked on python Openstack APIs and used NumPy for numerical analysis. • Aggregated logs data from various servers and made them available in downstream systems for analytics by using Apache Kafka. • Improved Kafka performance and implemented security. • Built and published Docker container images using Azure Container Registry and deployed them into Azure Kubernetes Service (AKS). • Developed batch and streaming processing apps using Spark APIs for functional pipeline requirements. • Experienced in transferring Streaming data, data from different data sources into HDFS, No SQL databases • Created ETL Mapping with Talend Integration Suite to pull data from Source, apply transformations, and load data into target database. • Transformed data from different files (Text, CSV, JSON) using Python scripts in Spark. • Loaded data from various sources like RDBMS (MySQL, Teradata) using Sqoop jobs. • Well versed with the Database and Data Warehouse concepts like OLTP, OLAP, Star Schema • Azure provides a secure global infrastructure, plus a range of features that use to secure the data in the cloud • Worked and learned a great deal from Azure Azure cloud platform like DataLake, DataBricks, Data Factory, Synapse, SQL, SQL DB, DWH, and Data Storage Explorer• Developed multiple Kafka Producers and Consumers from scratch to as per the software requirement specifications. • Performed advanced procedures like text analytics and processing, using the in-memory computing capabilities of Spark using python.

Education

• Jawaharlal Nehru Technological University

  Bachelor of Technology (B.Tech.)

  2010 — 2014

• Trine University

  Master of Science - MS, Engineering management