Santanu Ghosh

About Santanu Ghosh

14 years of experience in IT Industry including 4+ years in Big Data - Spark/Databricks Ecosystem & Azure Cloud and 9 years of experience in Java & PL SQL • In-depth understanding of Spark architecture including Spark Core, Spark SQL, Data Frames • Experienced in using Databricks Notebook with Python and implementing ETL pipelines • Hands on experience in Azure Cloud with Data Lake, Data Factory, Databricks, Key Vault, Logic App, GIT Integration and DevOps Pipeline. • Ingested Data from SAP Sources, SQL Server, Excel, CSV & Rest API. Applied transformation on the data to store in Parquet format using Delta Lake • Implemented Databricks Autoloader to ingest the Telematics data for US/Canada Data Integration Project and build ingestion framework, which is reusable, dynamic, configurable and robust to load the parquet data into delta lake tables. This data is accessed using SQL query and published to Power BI using Databricks Connectors. • Reading event hub transactional data using Spark Structured Streaming with checkpointing to prevent the data loss and load the data to the Datawarehouse using polybase directly from Databricks. • Logic Apps to upload multi sheet excel data to sharepoint from adls: Data Extract Project which is to find the list of Customer onboarded via Partnership Model and remunerate the Partners for their Services. The requirement was to generate the data and upload the excel files with two sheets (summary and aggregated per country wise) containing the list of those accounts with purchased details to sharepoint from adls using Logic Apps. Used Databricks library com.crealytics.spark.excel to create multi sheet excel. • Alert Engine: API for user subscriptions and alerts on criteria like card transactions, location, usage. Processes data from Event Hub & using Service Bus to send alerts. • Designed Motor Claims Insurance Product using Spring MVC and deputed in Dubai as Onsite Coordinator for Motor Claims • Good Level of experience in SQL using Cursor, Procedures, Packages, Functions, Exceptions • Good level of experience in Core Java, J2EE technologies as JDBC, Servlets, and JSP • Experience in designing the User Interfaces using HTML, CSS, JavaScript and JSP • Microsoft Certified: Azure Data Engineer Associate • Microsoft Certified: Power BI Data Analyst Associate (PL-300) • Scrum Certified (in Agile methodologies) • Completed 32 hours trainings with hands-on and labs on Microsoft PowerApps Power Automate on PL-200 Power Platform Functional Consultant (Canvas App, Model Driven App, Power Pages, Chatbots & AI Builder)

Experience

1. Senior Data Engineer

   IBM

   Oct 2021 — Present · Kolkata, IN

   Salesforce Implementation: Ingested Fuel Cards Data for 30+ countries for europe based Oil and Gas Company from multiple API of Salesforce which are collected daily in Landing Layer and then further curated using Databricks & joined to load into Synapse using External table and Polybase• Azure Databricks, ADF and Datawarehouse components are build & deployed using Azure Devops (CI/CD)• Myadmin data integration: Implemented Databricks Autoloader to ingest the Geotab Telematics Device Admin data for US/Canada/Mexico users & build ingestion framework, which is configurable and robust to load to Databricks delta lake and then to Synapse DWH. Common dashboard provided for Device, Order, Monthly Billing & Purchases reports. Enabled Users to build Self-Service customized reports in Power BI• Real Time Cards Data: Ingesting fuel card transactional data swiped at petrol pumps or gas station. Scheduled Databricks jobs using workflows to get the Real Time event hub data using Spark Structured Streaming with checkpointing to prevent the data loss and load the data to the Azure Synapse DWH using polybase directly from Databricks• Telematics CDM: Accumulating multiples TSP(Telematics Service Provider) data into one place. Leveraging cutting-edge technologies Delta Live Tables in the Azure Cloud, harnessed real-time telematics data to gain insights into vehicle engine status, fuel capacity, driving behaviours, etc. This allowed us to structure the telematics data and empowers business users to generate dashboards and real-time alerts using Azure SQL Server, Azure Function & Cosmos DB and elevated customers\' vehicle management experience to new heights• Fuel Fill Up Fraud Detection: Compare the Cards Transactions with Telematics Device Data and raise fraud alerts if there is fuel volume mismatch, fuel fill up records missing, location mismatch etc.• Telematics CDM API: Providing API for Overview and Summary Data includes sales transaction and device data via Azure Function