Spandana M

About Spandana M

Experienced Data Engineer with around 7 years of expertise in building robust, scalable…

Experience

1. Data Engineer

   American Express

   Sep 2024 — Present · Phoenix, AZ, US

   Architected, developed, and optimized scalable ETL pipelines using Apache Spark, Hive, Scala, and Java, significantly enhancing data processing capabilities.• Designed real-time data streaming solutions with Apache Kafka and AWS Kinesis, enabling instant analytics and reducing latency in decision-making processes.• Automated end-to-end data workflows and CI/CD pipelines utilizing Jenkins, Terraform, Kubernetes, and advanced shell scripting techniques.• Implemented machine learning and predictive analytics models leveraging GCP Vertex AI, Dataproc, BigQuery, and Cloud Composer, delivering actionable business insights.• Conducted extensive performance tuning and optimization of complex Spark and Hive applications, improving execution speed by up to 60%.• Spearheaded data governance frameworks and ensured stringent compliance with enterprise security protocols and data quality standards.• Collaborated with stakeholders to define strategic roadmaps, driving innovation in data engineering practices aligned with business goals.• Led initiatives on cloud migration strategies, enabling seamless transitions and modernization of legacy data systems GCP infrastructures.• Using Databricks and ADF, create pipelines, data flows, and complicated data transformations and manipulations. • Developed sophisticated fraud detection analytics solutions, significantly improving fraud detection rates and reducing false positives.• Streamlined and automated the ingestion, processing, and delivery of financial transaction data, enhancing real-time monitoring and reporting.• Managed large-scale data warehousing solutions using Redshift and Snowflake, supporting complex analytical queries and reporting needs for AMEX financial products.• Leveraged advanced analytics and visualization tools (PowerBI, Tableau) to deliver impactful dashboards for senior management and stakeholders, facilitating informed decision-making.

Education

• The College of Saint Rose

  Master of Science - MS, Computer Science