Suraj Nair

Data Engineer @ICICI Lombard

Mumbai, IN

I have been actively involved in migrating legacy workflows from SAS to Databricks using PySpark. This migration modernized the data processing framework, improved scalability, and optimized performance. Led the end-to-end migration of 15+ legacy SAS-based workflows to Databricks using PySpark, improving data processing speed by ~40% and reducing compute costs by ~30%. Re-engineered SAS logic into optimized, modular PySpark scripts, ensuring 100% feature parity and delivering accurate, reliable output. Developed and deployed automated data pipelines for ingestion, transformation, and reporting, increasing workflow reliability and scalability. Tuned Apache Spark jobs by leveraging caching, partitioning, broadcast joins and resource allocation strategies to reduce execution times from hours to minutes in some cases. Continuously enhanced existing Databricks pipelines by incorporating advanced Spark capabilities (e.g, Delta Lake time travel, broadcast joins, Z-order clustering). Collaborated with cross-functional stakeholders to ensure smooth transitions, validating business rules and data integrity at each step. Created detailed migration documentation and conducted training sessions for team members, boosting adoption of the new framework. Implemented monitoring and logging solutions for proactive pipeline management and faster issue resolution.