Rajeshwar K
Sr Data Engineer @Scotiabank
Signup · Get unlimited contacts
WORK HISTORY
Sr Data Engineer @Scotiabank
This project involves the end-to-end processing of customer data in GCP, starting with Cloud Storage ingestion and transforming data using Apache Spark on Dataproc. The system leverages services such as Dataproc, Dataflow, Cloud Functions, and Compute Engine to process large datasets, supporting analytics and machine learning feature engineering. Key tasks include data remediation, generating dashboard graphs, and delivering CSV files. Additionally, it processes historical financial data to support analytics and client reporting while ensuring pipeline support and monitoring in financial production environments. Used Dataproc to process and transform large amounts of data, moving it in and out of various GCP data stores such as Cloud Storage and Bigtable to support data-driven use cases. Created Hive tables and partitions on Dataproc and implemented incremental imports to optimize performance and enable ad-hoc querying on structured data for analytics. Developed various ETL transformation scripts using Hive and Spark to refine datasets for analytical use cases and further processing within the data pipeline. Designed ETL processes in Dataflow to migrate campaign data from external sources (e.g, Cloud Storage, ORC/Parquet/Text Files) into BigQuery. Stored log files in Cloud Storage, implementing versioning for sensitive information. Experience with GCP services related to data processing, such as Dataproc, Cloud Storage, Dataflow, and Compute Engine. Used microservice architecture with Spring Boot-based services, interacting through REST APIs and Pub/Sub for decoupled communication between services. Worked with Spring Core for Dependency Injection (DI) to build and maintain scalable Java applications, adhering to coding standards and best practices. Scheduled and orchestrated data pipelines and workflows using Apache Airflow on Composer, automating and monitoring the flow of data across various services.
EDUCATION
JNTU College Of Engineering
Bachelor's degree
ABOUT RAJESHWAR K
Experienced Big data developer / Data engineer with 9 years of expertise in designing and…
This profile is compiled from publicly available professional sources. Unifers is not affiliated with or endorsed by LinkedIn. Request removal of this profile.