Anurag D.

About Anurag D.

Senior Data Engineer | Cloud & Big Data Specialist | AWS | Azure | GCP With 10+…

Experience

1. Senior Data Engineer

   Ecolab

   Jun 2023 — Present · St Paul, MN, US

   Key Responsibilities & Achievements:•Designed and implemented AWS data pipelines using AWS API Gateway, AWS Lambda, AWS Glue, AWS Redshift, DynamoDB, and S3, improving data retrieval speed by 40% and conversion accuracy by 30%.•Developed and optimized ETL workflows using AWS Glue to process and import campaign data from external sources (S3, Parquet, ORC) into AWS Redshift, enabling efficient analytics and reporting.•Configured and deployed AWS EMR clusters to process large-scale data efficiently using PySpark and Scala, optimizing real-time and batch processing workflows.•Automated AWS infrastructure provisioning by designing and implementing CloudFormation templates, ensuring rapid deployment and scalability.•Built PySpark transformations in AWS Glue to extract, transform, and load data from AWS S3 into Redshift, improving data accuracy and accessibility.•Led the migration of large-scale multistate-level data from SQL Server to Snowflake using Python and SnowSQL, enhancing query performance and data accessibility.•Designed and managed data ingestion workflows with AWS Glue and Step Functions, ensuring efficient data transformation, cleaning, and validation.•Optimized complex data transformations using in-memory computing capabilities of PySpark, leading to faster text analytics and aggregations.•Implemented AWS CloudWatch monitoring and logging, reducing failure resolution time by 35% and improving observability of AWS data pipelines.•Developed and maintained Airflow DAGs to orchestrate and schedule ETL workflows, integrating AWS Glue and Redshift for seamless data pipeline automation.•Deployed Dockerized environments for testing and deploying ETL workflows, improving pipeline stability and scalability.

Education

• JNTUH College of Engineering Hyderabad

  Bachelor's degree, Computer Science

  2010 — 2014