Syam N
Data Engineer @Databricks
Signup · Get unlimited contacts
WORK HISTORY
Data Engineer @Databricks
US
At Databricks, I designed and deployed scalable ETL pipelines on the Databricks Lakehouse Platform using Apache Spark and Delta Lake, resulting in a 25% improvement in data processing speed for high-volume fintech transaction logs. I developed modular data transformation workflows in PySpark and SQL, integrating structured and semi-structured data from S3 and Kafka, which reduced batch processing latency by 30%. I also implemented Change Data Capture (CDC) and real-time streaming pipelines using Structured Streaming and Delta Live Tables, enabling sub-minute response times for fraud detection systems. To enhance operational reliability, I orchestrated end-to-end data workflows with Apache Airflow and dbt, integrating automated lineage tracking and CI/CD, which increased deployment reliability by 30%. I further improved performance and cost-efficiency by applying Z-ordering, partitioning, and caching techniques on large-scale datasets in Databricks and AWS S3. Additionally, I led the migration of legacy ETL processes to Databricks on Azure, achieving a 15% reduction in infrastructure costs and consolidating analytics across Snowflake and Azure Synapse. I worked closely with data governance teams to implement column-level lineage, access controls, and audit logging using Unity Catalog and Azure Purview, ensuring full compliance with SOC 2 and GDPR standards.
EDUCATION
Narayana Junior College - India
Intermediate
University of Alabama at Birmingham
Masters, Computer Science
Dr.kkr's gowtham school
secondary Education
Vit
Btech, Computer Science
ABOUT SYAM N
Experienced Data Engineer | 3+ Years in Fintech & Healthcare | Databricks, Spark, Azure ExpertI\'m a results-driven Data Engineer with 3+ years of experience designing and deploying scalable, high-performance data pipelines across fintech and healthcare domains. I specialize in modern data stack tools including Databricks, Apache Spark, Delta Lake, and Azure Data Factory, with deep expertise in real-time and batch ETL/ELT workflows using PySpark, SQL, Airflow, and dbt.I’ve built streaming architectures for fraud detection, implemented Change Data Capture (CDC) systems, and optimized performance through Z-ordering, caching, and cost-based query tuning. I also bring experience in data governance and compliance (HIPAA, GDPR, SOC 2) through tools like Unity Catalog and Azure Purview. Whether migrating legacy systems to cloud or building lakehouse architectures from scratch, I focus on delivering reliable, compliant, and business-driven data solutions.Currently pursuing my Master’s in Computer Science at the University of Alabama at Birmingham, I combine hands-on engineering with a strong foundation in data modeling, DevOps (CI/CD, Terraform), and cross-functional collaboration in Agile environments.
This profile is compiled from publicly available professional sources. Unifers is not affiliated with or endorsed by LinkedIn. Request removal of this profile.