Saikat Bhattacharjee
Information Technology Analyst - Big Data Engineer @Tata Consultancy Services
Signup · Get unlimited contacts
WORK HISTORY
Information Technology Analyst - Big Data Engineer @Tata Consultancy Services
Kolkata, IN
Project Highlights: 1. Streaming data from Azure cosmos DB using Azure Databricks and storing data into Azure Datalake as delta tables. Sha1 hash of source data is validated against the sha1 hash of data inserted into delta tables to avoid any corrupted data getting stored in delta lake. Log4j is used to generate application Log.3. Hive and sqlDW tables are maintained for delta tables.4. Reading data from Azure Blob in batches and storing it into Azure datalake as delta tables which in turn is used to generate reports using PowerBI.5. Use of delta lake time travel feature to rollback the data if reconciliation of data between source and target fails.6. Packaging of pyspark utility functions into wheel or egg file.7. Code developed in Pyspark and spark sql Working on a transformation project for Australian Regulatory body to change the 30 mins spot price settlement to 5 Mins spot price settlement. This is purely a Data Engineering project hosted in Azure Cloud. This will effect the whole Australian Energy market and its world\'s first project for 5 Mins settlement.Developed Enterprise Data Platform on Microsoft Azure that aggregate data from all relevant internal and external applications into a consistent structure and context.We receive data from varied sources such as oracle(structured data) and cosmos db (unstructured data). Oracle data are extracted as CSV files and cosmos db data are extracted in JSON format. Structured Streaming as well batch processes has been used in the project to write data into Enterprise data platform as parquet files of format “delta”. Have experience in packaging pyspark, python code base into egg, wheel or pypi library. Developed backend services written in Node.js using Typescript, containerized using Docker and deployed to a Kubernetes platform hosted on Microsoft Azure cloud in an Agile delivery environment.
EDUCATION
BENGAL COLLEGE OF ENGINEERING AND TECHNOLOGY
Bachelor of Technology (BTech), ECE
SKILLS
ABOUT SAIKAT BHATTACHARJEE
Lead Data Engineer with 11 years of success in conceptualizing technical solutions and system development predominantly in Big Data platforms.• Databricks Partner Solution Architect Champion, certified in multiple Databricks and Azure certification related to data and architecture.• Hands on expertise in Cloud Architecture, Big Data processes and tools.• Experienced in Distributed and Co-located Agile - Scrum methodology. • Proficient in Pyspark, Databricks, Python, Data Factory, Synapse Analytics, Azure SQL, Logic App, Storage Accounts, cosmos Db, Function App, Azure Devops.• Extensive experience as technical consultant in customer facing roles for Mobility, Utilities and Telecom domains at multiple geographies.• Well conversant with Software Development Life Cycle (SDLC) and have carried out its various phases like Requirements Analysis, preparing Systems Design development, testing and implementation.Core Competencies-• Requirement Gathering & Analysis• Technical Architecture• Data Warehouse/ Data Visualization• Data Modelling / Data Governance• Automation/ Process Improvement• Solution Delivery• Cloud Deployment, Configuration• Transition & Transformation• New Product Conceptualization Team Management/ TrainingTechnical Skills-• Technology - Pyspark, Python, T-Sql, PostgreSQL, Spark Sql• Database - Azure SQL, Synapse, Cosmos DB, Mysql• Cloud Services - Azure Databricks, Data Factory, Azure Storage, Logic App, Function App, Azure Purview, Azure Kubernetes• Version Control - Azure DevOps, Git• Management Tools - Azure Board, Jira, confluence.• Secondary Skills - PowerBI, Kafka, Snowflake, AWS S3, Nodejs, Docker, Unix Scripting, PowerShell
This profile is compiled from publicly available professional sources. Unifers is not affiliated with or endorsed by LinkedIn. Request removal of this profile.