Azhar Baig

Experience

1. Lead Engineer

   HCLTech

   Jul 2024 — Present · Pune, IN

   Requirement AnalysisUnderstand existing data pipelines, workflows, and business logic.Analyze source systems, data models, and target architecture.2. Data Pipeline Development (Spark-Scala)Design and implement scalable data pipelines using Apache Spark and Scala.Translate Java-based data processing logic into Spark transformations/actions.Optimize Spark jobs for performance (e.g, partitioning, caching, broadcast joins).3. Data Migration & TransformationDevelop ETL/ELT processes to extract data from source, transform as per business rules, and load into target systems.Ensure schema mapping and data type compatibility during migration.4. Data Validation & TestingWrite data validation scripts to compare source (Java pipelines) vs target (Spark-Scala) outputs.Use tools like Spark, ScalaTest, or custom logic for data quality checks.Perform row-level, column-level, and aggregate-level comparisons.5. Code Management & DeploymentUse Git for version control and CI/CD pipelines for deployment.Work with tools like Jenkins, Airflow, or Azure Data Factory for orchestration.Ensure code modularity, reusability, and documentation.6. Monitoring & LoggingImplement logging (Spark UI) and monitoring for Spark jobs.Handle job failures, memory issues, and data inconsistencies effectively.7. Documentation & Stakeholder CommunicationDocument migration strategy, data flow diagrams, job logic, and test results.Provide regular status updates, risk assessments, and troubleshooting support.8. Performance TuningProfile and tune Spark jobs for better memory usage and reduced execution time.Apply best practices like efficient joins