Paramik Dasgupta

Data Engineer @Tata Consultancy Services

Toronto, ON, CA

Built Source-Target Mappings (STMs) for seamless data integration across various systems using SQL and PySpark, specifically for different banking data products such as Mortgage, Deposits and Investments.• Used Azure Databricks to code STMs and created notebooks for each table under different sub-data products, resulting in a 20% reduction in processing time for data transformations.• Passed parameters to existing Azure Data Factory (ADF) pipelines for RDBMS and file ingestion into Azure Data Lake Storage (ADLS), facilitating the ingestion of over 1 million records per day.• Developed a consumption framework using Pyspark for managing source and target tables in Azure Data Lake Storage (ADLS) to optimize data usability.• Implemented data security measures for role-based access, improving data protection compliance and reducing unauthorized access incidents by 40%.• Created SQL-based Databricks notebooks for one-time data loads, Delta table updates, and rollback processes, contributing to a cost savings of annually through process efficiencies.• Utilized Function Apps to monitor the status of tables during data ingestion, ensuring timely alerts and status updates.• Managed Autosys jobs and AzCopy jobs for scheduling and orchestrating data transfers efficiently.• Performed Development Integration Testing (DIT) to ensure data accuracy and system reliability, achieving a data accuracy rate of 99.5%.• Used in house DevOps framework to pass parameters and deploy code into upper environments via CI/CD pipelines.• Resolved production issues and applied fixes, ensuring system stability and performance with an uptime of 99.9%.• Integrated data from multiple sources (RDBMS, ASCII) into a cloud environment to enhance scalability and performance.