Anand Sagar T

Senior Data Engineer @PayPal

Scottsdale, AZ, US

Responsible for developing data pipeline with Amazon AWS to extract the data from weblogs and store in Amazon EMR and worked with cloud-based technology like Redshift, S3, AWS, EC2 Machine, GLUE etc. and extracting the data from the Oracle financials and the Redshift database.•Developed a comprehensive strategy for migrating and integrating existing on-premises data sources and workflows into AWS Glue. This involves assessing current systems, designing a migration plan, and implementing Glue to seamlessly integrate data pipelines.•Architected and implemented a scalable and cost-effective data lake solution using AWS Glue as the ETL (Extract, Transform, Load) engine. This includes designing data ingestion processes, defining data catalog schemas, and optimizing performance for querying and analytics •Established data quality standards and governance policies within AWS Glue to ensure consistency, accuracy, and compliance with regulatory requirements. Implement data validation checks, monitoring mechanisms, and metadata management practices.•Setting up comprehensive monitoring and logging infrastructure for Apache Airflow to track workflow execution, task status, and performance metrics. Configure integrations with monitoring tools such as Prometheus, Grafana, and ELK stack for real-time visibility and troubleshooting.•Optimize performance of AWS Glue jobs and workflows by fine-tuning parameters such as partitioning, parallelism, and resource allocation. Conduct performance analysis, identify bottlenecks, and implement optimizations to enhance overall throughput and efficiency.•Implement continuous integration and delivery (CI/CD) pipelines using Apache Airflow to automate deployment and testing processes for data pipelines and applications. Utilize Airflow\'s scheduling.•Designed, documented operational problems by following standards and procedures using JIRA.

Vignan's University

Computer Science Engineering