Ananya Kapoor

Data Engineer @Erie Insurance Group

PA, US

Developing and maintaining scalable data pipelines using Python (NumPy, Pandas) and SQL to handle large datasets, improving data processing speed by 30% in AWS cloud platform using Terraform as infrastructure as a code. Designing and managing ETL processes with Apache Spark, Hive, and Pig, ensuring seamless data flow and integration from various sources. Optimizing data processing workflows using Hadoop and MapReduce, resulting in a 25% reduction in processing time. Deploying big data solutions on AWS RDS, Aurora,Postgres,Redshift, utilizing services like S3, Athena, AWS Glue, Dynamo DB, Kafka, EMR, Secret Manager, Cloud Watch, Sage Maker including defining IAM roles and policies. Developing and optimizing several big data applications using Apache Spark, significantly reducing batch processing times and resource utilization across projects. Implementing complex ETL processes using SSIS, SASS,BCP and orchestrating data flow between SQL databases and cloud platforms, ensuring seamless data integration and consistency across AWS. Conducting thorough data validation and quality assurance through SSRS and MS Office, ensuring high standards of data accuracy and reliability were met in reporting and analysis. Managing databases using MySQL and ensured robust version control using Git, contributing to improved data integrity and collaboration across the data engineering team. Utilize MS Office tools to prepare detailed documentation and reports on data engineering processes and outcomes, facilitating clear communication and process standardization.

ABV-Indian Institute of Information Technology and Management

Master of Business Administration - MBA, Banking and Financial Support Services

Northeastern University

Master of Science - MS, Data Analytics

Shri G S Institute of Technology & Science

Bachelor of Engineering - BE, Electronics and Instrumentation Engineering