Pratyush Kumar

Experience

1. Azure Data Engineer

   Tata Consultancy Services

   Mar 2022 — Present · Kolkata, IN

   Responsibilities include writing SQL queries, extracting and manipulating data from multiple sources within a real-time bigdata environment, collaborating with business analysts and customers to address business needs and gathering detailed requirements/specifications.• Curated large-scale datasets, applied advanced data cleaning and transformation techniques to handle missing values, duplicates, outliers etc. and prepared data for analysis and reporting.• Performed unit testing and presented insights to technical and non-technical audiences, thus rectifying errors before deployment and ensuring overall data quality and accuracy.• Implemented and managed CI/CD processes and pipelines using Azure DevOps, streamlining the development and deployment processes to maintain data integration, delivery consistency, and detect potential issues.• Designed and implemented manual data pipelines in Azure Data Factory to orchestrate and automate data movement and transformation. • Utilized Azure Databricks for data processing and transformation, ensuring data integrity and optimization.• Led and managed a team of data engineers, fostering a collaborative and high-performance work environment.• Utilized Azure Databricks to compile, clean and manipulate data using ETL tasks, thus maintaining the data integrity and accuracy.• Gained experience in Azure Data factory and its building componentslike Datasets, Pipelines and Activities, Integration Runtime, Scheduling, Execution, and Linked Services.• Architected & implemented medium to large-scale BI solutions on Azure using Azure Data Platform services (Azure Data Lake, Azure Data Factory, Azure SQL DW, and Azure Databricks).• Documented CI/CD processes, pipeline configurations & deployment procedures, thus facilitating knowledge sharing.• Fostered a culture of collaboration and adaptability within Agile teams to promote effective communication, rapid response to changing requirements.