Durga Avva

Senior Data Engineer @Scotiabank

Toronto, ON, CA

Vast experience working as a Data Engineer on Google Cloud Platform (GCP) within the financial sector, utilizing GCP tools for data engineering and analytics.• Skilled in designing and deploying efficient, scalable data pipelines utilizing GCP services such as Google Cloud Storage, Big Query, Dataflow, and Pub/Sub.• Developed custom operators and sensors within Google Cloud Composer (Airflow) to enable seamless integration with various data sources and destinations.• Proficient in leveraging DBT (Data Build Tool) for data transformation, including data modeling, cleansing, and aggregation, to enable robust reporting and analytics.• Expert in designing and optimizing data architectures on GCP, applying best practices for storage, partitioning, and indexing to improve performance and reduce costs.• Expertise in migrating on-prem Scala framework to GCP.• Contributed to converting Hive/SQL queries into Spark transformations using Spark RDDs, Python, and Scala.• Improved Spark job performance by implementing persist, cache, broadcast variable, Spark UDF conceptsEnvironment: GCP (Big Query, Cloud Functions, Pub-Sub, Dataflow & Data-proc, cloud storage, cloud composer, cloud SQL), Airflow, dbt cloud and core, Python, Spark (Scala), Git, IntelliJ, Jupiter Notebook, MySQL, MinIO, Trino(presto), Docker, Kubernetes.

Sri Chandrasekharendra Saraswathi Viswa Mahavidyalaya, Kancheepuram

Bachelor of Technology - BTech