Saksham Khanna

Lead Data Engineer @Tata Consultancy Services

Pune, IN

Modernized Sqoop and Hive workloads by replatforming them to PySpark-based pipelines on Databricks. Ensured data reliability and consistency through Delta Lake–backed validation and reconciliation. Enhanced Pyspark job performance to process 40 mn of transaction table Automated daily source-to-target count reconciliation with email alerts for 100+ tables. Optimized techniques such as Partitioning, Liquid Clustering, and usage of correct executors for a spark pipeline Implemented Databricks Auto Loader to ingest fraud detection data incrementally from cloud storage with minimal latency Fixed existing Sqoop pipeline for backfilling of past data as the job had failed many times for past dates Collaborated with team members to troubleshoot and resolve technical issues in data pipelines Involved in migration of cloudera cluster to latest spark and hive jars also validated the existing jobs to support the newer versions and fixed the jobs when needed