Subhasish Karmakar

About Subhasish Karmakar

Experienced with a demonstrated history of working in the information technology and services industry. • A budding professional having firm belief in hardworking, but in smart and honest way.• A suitable team player with a great communication and inter-personal skills. • Highly efficient Data Solution professional specializing in Data Engineering, Data Warehousing, Business Intelligence and Data Analysis domain with demonstrated history and track record of building data driven solutions and facilitating digital transformation for global clients across varied industries. • Responsible for Design and Maintenance of ETL Pipelines for Multiple Clients, Products and POCs. • Collaborated with multiple teams to carry out new Proof of Concepts and Potential Services using the wide array of azure cloud-based analytical tools. • Leading all data initiatives required for creating final demo ready prototypes for clients including but not limited to database design, data simulation, data anonymization and data harmonization. • Key skills and experiences include – Requirement Gathering, Business Logic design, ETL/ELT and Data Pipelines, Data Engineering on Microsoft Azure, Strong SQL Coding, Python for Data Processing and Analysis. Experienced Azure Data Engineer with a demonstrated history of working in the information technology and services industry. Strong engineering professional skilled in Azure Data stack including Azure Blob storage, Azure ADLS gen 2, Databricks, Pyspark, Azure Data factory, Azure Synapse DWH, Sql Server, Azure Fabric, Unity Catalog, Azure Devops for CI/CD pipeline.• Ability to quickly learn and grasp new skills and technologies to incorporate end to end analytical solutions independently to cater to client needs.

Experience

1. Manager

   Capgemini

   Nov 2023 — Present · Kolkata, IN

   Responsible for performance optimization using tools ADF, Databricks, Python(Pyspark) and SQL 1) Introduce delta tables, as It provides optimized performance for analytics workloads. 2) Identify which repo and tables taking time to load data, after converting to delta table, verify join conditions and apply necessary logics to get required data. 3) Parallel processing implemented by identifying the cross dependencies on repos which has been taking longer time. 4) Proper usage of worker nodes in data bricks and cluster Upgrade 9.2LTS to 13.3LTS 5) Distribution of jobs on multiple cluster. 6) Network Optimization : Peering between storage and data bricks subnet 7) Removal of unnecessary partitions. File size optimization at storage using vacuum & Z - Ordering. 8) Repo and Config consolidation. Notebook optimization on critical path. 9) Cluster optimization for cost optimization. 10) Replace Azure Synapse with Databricks SQL DWH.• Introduced Logging, Alerts & Notification for ETL Job & Objects Using tools ADF, Azure Databricks, Python, Azure Logic Apps 1) Incorporate audit logging in ADF data load pipeline, using python & SQL code logic to capture job & object level success/failure. 2) Send email alerts and MS Teams notification, using python & Azure Logic Apps.• Introduced Unity Catalog in our lakehouse project, migrated all L0, L1 & L2 tables from legacy hive metastore to unity catalog to simplify the security, as well as, the governance of the data by providing a Centralized Place, where it is possible to administer the access to the data, and, also audit the access to the data.

Education

• Maulana Abul Kalam Azad University of Technology, West Bengal (MAKAUT, WB)

  Bachelor of Computer Application, Computer Science

  2012 — 2015

• Hooghly Collegiate School

  (10+2) Std, Science

  2004 — 2012

Skills

• Bootstrap
• Web Application Development
• Visual C#
• C#
• .net
• Javascript
• Ajax Toolkit
• Windows 8.1
• Html5
• Phpmyadmin
• Ajax
• Css
• Jquery
• Php
• Jquery UI