Shivam Mahajan

About Shivam Mahajan

Senior/Lead Data Engineer with 9+ years of experience in designing, building, and optimizing cloud-native, scalable, distributed data systems. Specialized in big data processing (Spark, PySpark, Databricks), cloud platforms (AWS), and streaming systems (Kafka, Structured Streaming). Proven track record of leading data engineering initiatives that power mission-critical applications, reduce latency, and improve reliability. Passionate about writing clean, testable, production-grade code and enabling data-driven decisions at scale. Expertise in Python, SQL, PySpark, Databricks, and Data lake, with a strong focus on data modeling and automation. Proven ability to lead data engineering teams, mentor junior engineers, and collaborate with cross-functional stakeholders to deliver high-impact solutions

Experience

1. Lead Data Engineer (Sde Iii)

   Adobe

   May 2022 — Present · Noida, IN

   Designed and built petabyte-scale ETL pipelines on Databricks + PySpark, transforming structuredand semi-structured data for security analytics.•Optimized Spark jobs by tuning partitioning, caching, and broadcast joins, reducing runtime of criticalpipelines by up to 60%.•Implemented Delta Lake with schema enforcement and time travel, enabling safe and scalable datamutation operations.•Built streaming pipelines with Kafka and Spark Structured Streaming, enabling near real-timealerting with <5 minute latency.•Automated end-to-end deployment of pipelines via Databricks Workflows, GitHub Actions, andparameterized notebooks.•Migrated on-premise data workflows to AWS, leveraging S3, Lambda, and Glue, enhancing scalabilityand reducing costs by 25%.•Optimized complex SQL queries, improving data retrieval efficiency by 50%.•Led a team of 4 engineers, driving code reviews, architectural decisions, and establishing testing + CIbest practices.

Education

• National Academy of Science & Technology (Nepal)

  Intermeditae, Science

  2008 — 2010

• Invertis University

  Bachelor of Technology (B.Tech.), Computer Science

  2011 — 2015

Skills

• Web Services
• Openerp
• Representational State Transfer (Rest)
• Cascading Style Sheets (Css)
• Git
• Shell Scripting
• Rabbitmq
• Html
• Customer Relationship Management (Crm)
• Agile Methodologies
• Software Development
• Python
• Elasticsearch
• Object-Oriented Programming (Oop)
• Aws
• Java
• Celery
• Javascript
• Solr
• Enterprise Resource Planning (Erp)
• Postgresql
• Css
• Xml
• Ubuntu
• Jira
• Odoo
• Mongodb
• Django
• Sublime Text
• Redis
• Ethical Hacking
• Mysql
• C
• Programming
• Team Management
• Sql
• Linux
• Core Java
• Pycharm
• Svn