Bhargavi

Data Engineer @BNY

Sunnyvale, CA, US

Led the design and architecture of the data processing pipeline, selecting Scala and Apache Beam to handle complex data transformations and processing logic.• Collaborated with cross-functional teams to define requirements and integrate Google Cloud services such as Pub/Sub for data ingestion, Dataflow for data processing, and BigQuery for data storage and analytics.• Implemented custom data processing logic in Scala to perform real-time aggregations, enrichments, and filtering of streaming data, ensuring data quality and accuracy.• Worked closely with data scientists to integrate machine learning models into the pipeline for real-time predictions and insights.• Optimized the pipeline for performance and scalability, fine-tuning resource allocation and parallel processing to meet SLAs and handle large data volumes efficiently.• Implemented monitoring, logging, and alerting using Stackdriver to track pipeline performance, detect anomalies, and ensure high availability and reliability.• Successfully deployed and maintained a real-time data processing pipeline that processed millions of events per day with low latency and high throughput.• Reduced data processing costs by optimizing resource utilization and implementing cost-effective storage and processing solutions in GCP.• Orchestrated data workflows using Apache Airflow, ensuring efficient ETL processes.• Developed custom Airflow operators and orchestrated workflows with dependencies across multi-cloud platforms (AWS EMR, Azure HDInsights, MapR CLDB).• Build data pipelines in airflow in GCP for ETL related jobs using different airfow operators both old and newer operators• Migrated previously written cron jobs to airflow/composer in GCP• Improved data-driven decision-making by providing timely and accurate insights from the streaming data, enabling stakeholders to respond quickly to changing business conditions.