Satheeshkumar Devarajan
Senior Data Engineer @IBM
Signup · Get unlimited contacts
WORK HISTORY
Senior Data Engineer @IBM
CA
Designed and delivered Azure-based scalable data pipelines (ETL/ELT) using Azure Data Factory (ADF), Databricks, ADLS Gen2, Azure DevOps (CI/CD), and Delta Lake, Databricks Workflows, Delta Live tables following Medallion Architecture (Bronze/Silver/Gold).• Built metadata-driven schema evolution and ingestion frameworks, reducing onboarding time by 70% and enabling automated processing across 60+ data sources.• Implemented Unity Catalog–aligned governance, improving lineage, access control, and cross-domain data sharing, reducing data exchange cycles from weeks to hours.• Developed batch and streaming pipelines using PySpark and Structured Streaming for real-time ingestion and analytics.• Built GenAI-powered search chatbot for aerospace/defense, reducing document retrieval time by 90% and achieving 85% accuracy.• Designed reusable ingestion frameworks (schema drift processor, multi format converters, UDF libraries, PII handling, Data masking) enabling standardized ingestion across diverse sources and reducing onboarding effort by 40% from Bronze to Gold layer.• Collaborated with analytics teams to build Power BI dashboards on curated Delta Lake tables, improving data-driven decision-making for operations and finance stakeholders.• Managed and optimized 50+ node Databricks clusters, improving performance through partitioning, caching, and skew mitigation.• Led PoCs, technical workshops, and solution demos for enterprise clients.
EDUCATION
Sengunthar Arts and Science College
Bachelor of Computer Science, Mathematics and Computer Science
Adhiyamaan College of Engineering
Master of Computer Applications (MCA), Mathematics and Computer Science
SKILLS
ABOUT SATHEESHKUMAR DEVARAJAN
Summary• Senior Data Engineer with 10+ years of experience designing scalable data pipelines using Medallion Architecture, Databricks Spark/PySpark, ETL/ELT workflows, Delta Lake, Azure Data Factory, ADLS Gen2, DevOps CI/CD, and cloud data engineering.• Designed metadata-driven ingestion frameworks and Medallion Architecture solutions handling 60+ data sources, reducing onboarding time by 70% and powering analytics, AI, and operational reporting. • Experienced in collaborating with clients, leading workshops, and delivering production‑ready solutions in insurance, financial services, and enterprise environments.• Proven ability to build batch and streaming pipelines, implement Unity Catalog–based governance, optimize Spark workloads, and deliver end to end ELT/ETL solutions supporting analytics, AI, and operational reporting• Seeking a Senior Data Engineer role to drive scalable architecture, automation, and data platform modernization. SKILLS• Programming & ScriptingPython, SQL, PySpark, Java, Scala, Shell Script, PowerShell• Data Engineering & Big DataAzure Databricks, Spark, Delta Lake, Delta Live Tables, Structured Streaming, Unity Catalog, ETL/ELT, Kafka, Databricks Workflows, Airflow/Jenkins), dbt, Hadoop, Hive• Cloud & DevOpsAzure (Azure Data Factory or ADF, ADLS Gen2, Synapse, Azure SQL, Key Vault, Logic Apps, Event Hubs), Azure DevOps CI/CD, Git, AWS S3• Databases & StorageSQL Server, PostgreSQL, Snowflake, Cassandra, Elasticsearch• Data Modelling & GovernanceMedallion Architecture, Dimensional Modelling, Data Fabric, Data Lineage, Data Quality, Data Observability, Data Cataloging, Data Sharing• Analytics & BIPower BI• Other ToolsREST APIs, Docker, Kubernetes (basic)• ConceptsData Pipeline Architecture, ML/AI workflow integration, Data Fabric, Performance Optimization, Data Migration, Data Skewing, Query Optimization, Cloud Data Engineering, Data Quality, monitoring, Data Contracts, Azure Purview / Microsoft Fabric
This profile is compiled from publicly available professional sources. Unifers is not affiliated with or endorsed by LinkedIn. Request removal of this profile.