Meyya Kannappan

Lead Data Engineer @Assurant

US

Designed and implemented high-performance data pipelines using PySpark, Python, and Auto Loader by reducing execution time by 40% and improving overall system efficiency. Optimized Azure Cloud storage and compute resources, implementing partitioning and intelligent caching strategies, resulting in 60% cost savings annually. Developed an ELT solution which ingesting raw data from AWS S3 to target DWH by following Medallion architecture.(Bronze, Silver, Gold)Architected and deployed Delta Lake-based data solutions, schema evolution, and improved query performance for analytics teams. Enhanced SQL and Spark jobs through query optimization, partition techniquesEstablished automated deployment pipelines for data workflows using Azure DevOps, reducing deployment errors and improving release cycles. Mentored junior engineers on ADF, PySpark, Delta Lake, and cloud best practices fostering a high-performing data engineering team. Partnered with business analysts / integrators to ensure seamless data accessibility, providing refined data for ML model building.