Komal Khakal

Data Engineer @Mastercard

Pune, IN

Developed and maintained 100% reusable, modular Python codebases in Databricks to build scalable ETL workflows, significantly improving code maintainability, reusability, and overall team productivity during the POC phase.• Optimized PySpark and SparkSQL pipelines in Databricks for large-scale, incremental processing of semi-structured data (JSON) across bronze, silver, and gold layers, leveraging AQE, delta cache, auto-compaction, Z-order, and broadcast joins to boost performance by 60% and improve cluster efficiency.• Used Unity Catalog for centralized access control and leveraged Databricks system tables for auditing, job tracking, and billing insights.• Built equivalent ETL logic and procedures in Snowflake, replicating the same data processing architecture using Snowflake SQL, streams, and tasks for validation and cross-platform flexibility.• Fetched and ingested raw data from Amazon S3 into Databricks using Auto Loader and Spark Structured Streaming, enabling near real-time data ingestion and seamless integration with downstream systems.• Built and managed Databricks Notebooks, SQL-based transformations, and stored procedures to support KPI-driven reporting and analytics.• Engineered high-performance ETL workflows using proper job clusters for execution to reduce storage and processing costs by 20%, increase data availability, and scale pipelines for future production readiness.• Managed CI/CD processes and version control using Bitbucket, ensuring consistent, automated deployment of notebooks, jobs, and workflows during the POC lifecycle.• Designed scalable data models and system architectures with strong expertise in ETL patterns, data warehousing, and cloud-native integration across modern platforms.Developing a Spark Structured Streaming pipeline on Cloudera Hadoop to ingest real-time data from Kafka into Amazon S3, ensuring reliable and scalable data ingestion.

Maharashtra Institute of Technology

Bachelor of Engineering - BE, Electronics and Communications Engineering