Krishna Shah

Senior Data Engineer @Tata Consultancy Services

CA

The Bank of Nova Scotia\'s Project - Orchestrated the sourcing and integration of trading order data from external venues into the bank\'s strategic system, ensuring conformity to standardized formats- Engineered an end-to-end pipeline leveraging Nifi transformations to seamlessly convert raw data from diverse sources into a uniform data structure- Spearheaded performance optimization initiatives, implementing a spectrum of strategies to enhance system efficiency- Directed the routing of transformed data to Kafka topics for real-time processing, with subsequent storage in Elasticsearch, while archiving source data in Minio for historical preservation- Developed Dremio views to empower the business team with comprehensive data insights, facilitating informed decision-making- Developed a Python script utilizing a Docker image and deployed it on a Kubernetes cluster to generate dashboard-style reports showcasing key end-of-day business activities, which includes the number of files received, processed,etc. and sent email notifications for the same- Developed a centralized utility to streamline the loading and processing of historical files, effectively managing and cleansing older data sets.

Vishwakarma Government Engineering College

Computer Engineering