Goutam Kumar Ghadai

About Goutam Kumar Ghadai

Senior Data Engineer at EY GDS, currently working in a data migration project from Oracle…

Experience

1. Senior Data Engineer

   EY

   Sep 2024 — Present · Bhubaneswar, IN

   Leading the migration of complex ETL workflows from Informatica to PySpark and Spark SQL on the MicrosoftFabric platform, ensuring seamless data transformation and query optimization across large datasets- Designed reusable Python utility packages with Lakehouse functions such as SCD (Type-1 and Type-2), INSERT,MERGE, and UPSERT and deployed through Azure DevOps (ADO) for enterprise-wide use- Successfully loaded 50+ TB of data from Oracle DB to Fabric Lakehouse and Warehouse using Medallionarchitecture with 100% data accuracy and validation checkpoints using Spark SQL- Implemented an automated validation framework which can handle 4+GB of data per run, performing end-to-enddata validations through SQL aggregations and window functions to ensure consistency post-migration- Developed 200+ PySpark scripts and Stored Procedures replicating Informatica workflows, and achieved a 30%improvement in processing speed- Migrated complex Alteryx workflows to PySpark and Spark SQL within Azure Synapse using CTE-based logic,and analytical functions for improved maintainability- Used LLMs to automate the PySpark code generation from Alteryx workflows- Optimized complex queries using window functions and indexing, achieving a 40% improvement in processingspeed- Resolved critical production issues to prevent outages and minimize system downtime by identifying andtroubleshooting potential failures.