Anantha Rao Yanamula

Data Engineer, Governance & Management Architect @Accenture

Dallas, TX, US

Designed and implemented end-to-end Azure-based data solutions to support data warehousing, advanced analytics, and real-time reporting.• Designed scalable data pipelines in databricks to ingest, process, and analyze data from various sources (eoperational databases, external data feeds, and logs). This involved leveraging Apache Spark capabilities within Databricks for distributed data processing.• Architecting databricks end-to-end solutions from data ingestion, transformation, storage, and analysis, ensuring the solution integrates smoothly with Azure services like Azure Data Lake, Azure Blob Storage, Azure Synapse Analytics, and Azure SQL Database.• Designing workflows in Databricks to ensure data processing jobs are optimized for performance, cost efficiency, and scalability. leveraged Databricks features like Delta Lake, Auto-scaling, and Spark Jobs to meet the needs of Travelers\' business.• Collaborating with data engineers to build the underlying infrastructure and with data scientists to ensure that the platform supports advanced analytics and machine learning workloads. The aim would be to ensure that Databricks supports various use cases, from real-time fraud detection to claims processing optimizations.• Implementedbest practicesin databricks data pipeline design, cluster management, and the use of Databricks notebooks. This would include setting up MLflow for model tracking or integrating Azure Machine Learning (AML) for model deployment and management.• Implemented databricks best practices for securing sensitive insurance data. This would include setting up access controls, managing encryption, and ensuring compliance with industry regulations• Monitored and optimized Spark jobs and Databricks clusters for performance. This includes tuning Spark configurations, leveraging Delta Lake for efficient data storage and transaction management, and optimizing SQL queries for fast retrieval.