Durgaprasad S

Experience

1. Senior Data Engineer

   Discover

   Dec 2024 — Present

   Designed and implemented real-time and batch data pipelines using Spark, Scala, and Kafka to support risk analytics, fraud detection, and credit decisioning platform. • Designed and developed robust ETL pipelines to extract, transform, and load large volumes of financial data into Snowflake, improving processing efficiency and data availability. • Designed and maintained scalable data pipelines on Snowflake to support real-time financial analytics and credit risk modeling. • Implemented CI/CD pipelines using Azure DevOps to automate deployment of Snowflake code and infrastructure changes. • Used Terraform to manage infrastructure as code for Snowflake environments and related AWS resources (e.g, S3, RDS). • Automated ingestion of financial data from multiple sources (Kafka, APIs, flat files) into Snowflake using Python and Airflow. • Managed RBAC policies and data masking in Snowflake to ensure compliance with PCI-DSS and internal governance standards. • Designed and developed scalable data pipelines using Azure Databricks and Azure Data Factory, enabling real-time ingestion and processing of financial data from multiple sources. • Developed and optimized scalable ETL pipelines using Databricks and Apache Spark to process high-volume financial transactions in near real-time. • Integrated Kafka streaming with Databricks Structured Streaming for continuous ingestion of payment and credit card data. • Automated deployment of Databricks clusters and jobs using Terraform and Databricks REST APIs to reduce manual efforts and ensure consistency. • Designed data models and schema evolution strategies in Delta Lake to support regulatory reporting and audit requirements. • Collaborated with risk management and fraud analytics teams to embed ML models within Databricks notebooks, enabling real-time fraud detection. • Implemented role-based access controls (RBAC) in Databricks workspaces, ensuring data security.

Education

• University of New Haven

  Master's degree

  2023