Krishnakanth B

Team Manager @MUFG

Newyork, NY, US

Integrated Hive and HBase for streamlined operations and data management. Developed Spark programs in Scala for data transformations, utilizing Datasets, Data Frames, and Spark p queries, including Spark Streaming and windowed streaming applications. Utilized Azure Databricks with PySpark to optimize data transformation speed and accuracy. Created on-demand tables from S3 files using Lambda functions and AWS Glue with Python and PySpark. Designed and implemented PyUnit framework for unit testing. Developed complex Transact-SQL queries and SSIS packages for data warehouse loading. Built Admin API for managing Kafka objects such as topics and brokers. Implemented data ingestion modules with AWS Glue, loading data into S3 layers and facilitating reporting via Athena and QuickSight. Managed data storage on Azure Data Lake Store (ADLS) and Azure Blob Storage, ensuring data durability and accessibility. Leveraged Kubernetes features like Ingress controllers, Persistent Volumes, and Secrets to configure networking, storage, and security policies for applications. Developed data quality scripts using SQL and Hive to validate data integrity post-load. Integrated Azure Databricks with Azure Data Lake Analytics for advanced analytics. Utilized Python and Tableau to create various data visualizations. Proficiency in containerizing applications and services using Docker for easier deployment, scalability, and isolation of environments. Experience in defining and managing multi-container Docker applications using Docker Compose for local development and testing. Proficient in monitoring and managing Hadoop clusters through Cloudera. Implemented data lineage and cataloging using Azure Purview.