Varahalaraju P

About Varahalaraju P

Designed and maintained CI/CD pipelines using Jenkins. • Integrated GitHub…

Experience

1. Software Engineer

   HCLTech

   Oct 2022 — Present · Chennai, IN

   Software Engineer | GCP Data EngineerAs a GCP Data Engineer, I build and optimize cloud-based data solutions, enabling efficient data processing, analytics, and real-time insights.Key ResponsibilitiesDeveloped ETL pipelines using BigQuery, Dataflow, and Cloud Composer for processing large datasets.Designed real-time data streaming solutions with Pub/Sub & Dataflow for instant data processing.Built data warehouse models in BigQuery, improving query performance and cost efficiency.Automated workflow orchestration with Apache Airflow (Cloud Composer) for seamless data processing.Integrated PySpark for large-scale batch data transformations.Migrated on-prem databases to GCP, reducing infrastructure costs and improving scalability.Current ProjectsReal-time Customer Analytics: Using BigQuery & Dataflow to reduce reporting time from hours to minutes.Data Lake Implementation: Building a centralized data lake on GCS & BigQuery for structured and unstructured data processing.GCP Cost Optimization: Implemented BigQuery partitioning & clustering, cutting storage costs by 30%.Pre-Joining Training & TransitionBefore joining full-time, I completed 4 months of pre-joining training covering SQL, Python, Cloud Computing, Linux, and Data Structures, which helped me transition from Electrical Engineering to Data Engineering.