Jidnasa Pillai

About Jidnasa Pillai

As a Senior Data Engineer with 5+ years of experience, I thrive on transforming complex data challenges into scalable, high-performance solutions—especially within the banking and financial sectors. My expertise lies in building and optimising end-to-end ETL pipelines, migrating legacy platforms to modern cloud-native architectures, and ensuring robust data quality for regulatory and analytical needs.Core Expertise:1) Big Data frameworks — Spark (Scala & PySpark), Hadoop Ecosystem (Hive, HDFS, Sqoop, Oozie, MapReduce)2) Real-time ingestion with Kafka and Avro → Parquet conversion, partitioning strategies, and Hive optimisation for regulatory and analytical use cases.3) DevOps integration with Jenkins, Git, SonarQube, Blackduck, and automated testing using ScalaTests.4) Shell scripting for automation and workflow orchestration (Autosys).Cloud & Modern Data Stack Exposure:1) Hands-on experience with AWS (S3, Glue, Redshift, Lambda), Azure (Data Factory, Synapse, Databricks), and GCP (Pub/Sub, Dataflow, BigQuery) through real project implementations.2) Focused on re-architecting on-prem Spark/Hadoop pipelines to cloud-native designs with lakehouse patterns.3) Delivered personal & collaborative side projects showcasing ingestion, transformation, orchestration, and visualisation across multiple clouds.Certifications:1) AWS Certified Data Engineer – Associate2) AWS Certified Solutions Architect – Associate3) Databricks Certified Data Engineer Associate4) Additional training on Azure Data Engineering and GCP Data EngineeringStrengths:1) Strong understanding of distributed systems, data governance, and schema evolution.2) Experienced in optimising Spark jobs for large-scale workloads and ensuring data quality in production environments.3) Adept at bridging traditional data platforms with cloud-native architectures for scalable analytics.4) Recognised with multiple awards (Pat on the Back, Rising Star) for technical ownership and delivery excellence.

Experience

1. Senior Data Engineer

   LTIMindtree

   Apr 2022 — Present · Mumbai, IN

   Delivering high-impact ETL and streaming data pipelines in the banking and financial domain using Spark, Hive, and Kafka, with a strong focus on performance optimization, data accuracy, and regulatory analytics.Key Achievements:1) Engineered and maintained large-scale Spark (PySpark/Scala) ETL pipelines, processing 10B+ records monthly, improving data throughput by 35% through optimized partitioning and job tuning.2) Reduced data processing time by ~40% by implementing Avro-to-Parquet format conversion, enabling faster downstream query performance and analytics.3) Lowered data latency by ~45% by designing real-time Kafka ingestion frameworks, enabling near real-time dashboarding for compliance reporting.4) Developed automated validation and data quality checks, improving accuracy to 99.9% across regulatory datasets and reducing manual interventions by 60%.5) Built reusable transformation modules, decreasing pipeline development time by 30% and increasing maintainability.6) Worked extensively with Hive for partitioning and query optimization, cutting average job runtime by 25%.7) Collaborated with business and regulatory teams to deliver stable production pipelines aligned with strict compliance SLAs.8) Recognized with multiple internal awards (Pat on the Back, Rising Star) for technical ownership and delivery excellence.Tech Stack: PySpark, Scala, Hive, Kafka, Avro, Parquet, Autosys, Git, SonarQube, Blackduck, Datalens

Education

• SVKM's Narsee Monjee Institute of Management Studies (NMIMS)

  Master of Business Administration - MBA, Information Technology and Systems Management

• MES'​ PILLAI COLLEGE OF EDUCATION AND RESEARCH

  Bachelor of Engineering, Computer Engineering

  2016 — 2019

• Sardar Vallabhbhai Patel Polytechnic

  Diploma, Computer Technology

  2014 — 2015