Teja Varma Yeggadi

Senior Data Engineer Big Data Engineer @Capital One

Toronto, ON, CA

Developed end-to-end ETL data pipelines using AWS Glue, integrating large datasets from on-premises MySQL, Cassandra, and Amazon S3, applying transformations, and enabling interactive querying through AWS Athena, enhancing data accessibility and reducing query latency.• Utilized AWS EMR to run large-scale Spark jobs for processing high-volume datasets efficiently, integrating with S3 for storage and Hive Metastore for schema management, enabling scalable and cost-effective data transformations.• Migrated SQL databases to Amazon S3, Amazon RDS, and Amazon Redshift, while maintaining data integration across Hadoop and relational database environments within the AWS ecosystem.• Designed real-time data streaming pipelines using AWS Kinesis and Kinesis Firehose, with AWS Lambda for on-the-fly data transformation and delivery to Amazon S3 and Redshift, enabling real-time analytics and faster decision-making. • Developed and maintained AWS Glue jobs and Python scripts to automate data transformations and achieve data consistency across various data sources. • Developed ETL pipelines for Snowflake using Python, Pyspark, and SQL, and created Spark-SQL, Spark Data Frames transformations with Azure Databricks and Data Factory. • Processed real-time streaming data with Kafka and Spark Streaming, developing data pipelines for ingestion, transformation, and analysis.• Scheduled and orchestrated end-to-end data workflows using Apache Airflow, Terraform, and Azure Data Factory, enabling automated data pipeline executions by ensuring timely data availability for business analytics and reporting. • Migrated data from OLTP systems to OLAP systems and built and maintained database structures to support robust data operations and analysis.• Utilized Power BI to create various analytical dashboards that depict critical KPIs.