Mujeeb Ali Shaik

Data Engineer @Intact

CA

Collaborated closely with business/user groups to elicit requirements and played a key role in the creation and development of pipelines.• Utilized Airflow for building batch data pipelines, while Kafka was employed for constructing streaming data pipelines.• Created a Spark Streaming script that consumes topics from Kafka for real-time processing, enhancing data accuracy.• Developed and managed Azure Data Factory, incorporating Blob storage for storage and backup on the Azure platform.• Leveraged JavaScript, HTML5, CSS3, Bootstrap, Ajax, JSON, XML, jQuery, Angular.js, React.js, and Node.js to design interactive front-end interfaces.• Applied Spark applications in Python to establish distributed environments for loading high- volume files using PySpark into Azure SQL DB tables.• Wrote Java API for Amazon Lambda to manage specific AWS services.• Designed and automated pipelines using Databricks for ETL processes, ensuring efficient workflow management.• Engineered ETL packages in SSIS to extract and maintain data from various sources, such as Access databases, Excel Spreadsheets, and flat files.• Operated ETL processes using Azure Databricks, employing Kafka for connecting to relational databases and Informatica for session and workflow creation and monitoring.• Automated data ingestion, transformation, and storage using Apache Spark and Delta Lake, ensuring data quality and integrity.• Developed data workflows utilizing Data bricks, Scala, and Spark, capturing data from Delta tables in Delta Lakes.• Implemented streaming pipelines via Azure Event Hubs and Stream Analytics, facilitating data- driven workflow analysis.• Utilized Delta Lakes for consistent data unification and worked on ACID transactions using Apache Spark for data processing.• Took charge of identifying and implementing SQL Server enhancements to optimize query performance and ensure data integrity.

Teegala Krishna Reddy Engineering College

Bachelor's degree