Vaibhav Saini
Data Engineer @BUSINESSNEXT
Signup · Get unlimited contacts
WORK HISTORY
Data Engineer @BUSINESSNEXT
Noida, IN
Engineered a policy-driven Data Archival Tool (PySpark, Apache Iceberg, Hive Metastore on S3) that pulls data per retention rules, processes in configurable micro-batches/chunks, performs archive validation (checksums/reconciliation), and purges sources only after successful archive validation — migrating 10B+(20TB+) records across columns with zero data loss and automated lifecycle management. Optimized RDBMS update/upsert workflows in PySpark using dynamic SQL, parallel task distribution and batched inserts (pyodbc/JDBC), improving RDBMS write efficiency by >60%. Developed a PySpark-based pipeline to retrieve AI predictions from ML models and apply trans-formations with performance enhancements like broadcast joins, lazy evaluation, and data cachingfor memory-efficient execution. Engineered scalable RDBMS→NoSQL migration patterns with micro-batching and parallelism, enabling robust, resumable transfers from Oracle/SQL Server/MySQL/Postgres into MongoDB without operational downtime. Integrated Apache Iceberg with S3, Hive Metastore and Trino and implemented catalog APIs for schema evolution, time-travel and upsert, enabling multi-cloud data lake writes and automated lifecycle management for large datasets. Implemented a custom repartitioning strategy to standardize data chunk sizes to 128 MB, maximiz-ing Apache Spark performance across varied input data sizes.
EDUCATION
Indian Institute of Technology, Mandi
Bachelor of Technology - BTech, Computer Science
ABOUT VAIBHAV SAINI
As a Data Engineer, I am currently involved in designing and optimizing data pipelines, working with cloud platforms (AWS, GCP) and big data technologies like Hadoop and Spark. I focus on building scalable data architectures that enable seamless data flow and support data-driven decision-making. Collaborating with data scientists and analysts, I transform raw data into actionable insights that drive business outcomes.Previously, as a Software Engineer, I worked on languages like Java and C++. My experience as software engineer has honed my problem-solving skills and ability to deliver high-quality, efficient solutions in fast-paced environments.
This profile is compiled from publicly available professional sources. Unifers is not affiliated with or endorsed by LinkedIn. Request removal of this profile.