Vivek Sharma
Senior Data Engineer @Tata Consultancy Services
Signup · Get unlimited contacts
WORK HISTORY
Senior Data Engineer @Tata Consultancy Services
London, GB
Working as a Senior Data Engineer for easyJet Airline Company Limited.Tech Stack: Databricks on AWS, PySpark, Python, SQL, Apache Airflow, AWS Glue, AWS Lambda, Amazon Kinesis, Amazon SQS, Amazon Event Bridge- Collaborated with Business Analysts, Product Owners and data scientists to understand data needs- Engineered a scalable, metadata-driven streaming ingestion framework for ingesting data from streaming sources into Databricks using PySpark & Delta Live Tables (DLT). The framework leverages Databricks Asset Bundles(DAB) and is integrated with GitHub Actions for CI/CD. This framework has reduced the data latency by approximately 90%- Engineered 10+ near real time & Batch ingestion pipelines leveraging AWS Glue, AWS Lambda, Apache Airflow and PySpark to ingest & transform the data into Databricks, along with implementing multiple data curation pipelines using DBT (Data Build Tool)- Enhanced the QA Automation test framework by automating manual data validation scenarios. This framework enabled automated validations eliminating human errors and improved QA efficiency by 40%- Developed POCs leveraging Amazon Event Bridge, Kinesis, Data Firehose, Structured streaming & DLT pipelines to stream the data into Databricks.
EDUCATION
Sanatan Dharam Inter College
SSC, Mathematics
MEERUT INSTITUTE OF ENGINEERING AND TECHNOLOGY, MEERUT
Bachelor of Technology, Computer Science
ABOUT VIVEK SHARMA
I\'m a Senior data engineer with 7+ years of experience in data engineering scalable ETL/ELT optimized data pipelines with built-in data governance and quality controls. Skilled in building streaming and automation frameworks, real-time data processing and monitoring systems across cloud-native and enterprise environments. Experienced in delivering resilient solutions using Python, Databricks, AWS & Azure cloud services, Apache Spark, DBT, Terraform and ELK stack. Specialized in stream processing, enabling CI/CD with Jenkins & GitHub Actions and improving operational efficiency through automation.I’m passionate about solving complex data problems, ensuring data quality, integrity, and privacy, and collaborating with cross-functional teams to drive measurable business impact.Currently I am upskilling in Azure and GCP cloud, Kafka streaming, Architecture & design.
This profile is compiled from publicly available professional sources. Unifers is not affiliated with or endorsed by LinkedIn. Request removal of this profile.