Sourabh Lakhera

About Sourabh Lakhera

I am a results-driven Data Engineer with a strong background in Python, Apache Spark, and…

Experience

1. Technical Lead

   HCLTech

   Jul 2022 — Present · IN

   Experience at Western Union • Proficient in SQL, Linux, Python, Talend (ETL), Cloud Storage/Computing. Certified in Databricks and AWS. • Led 3 successful POC projects using SQL, Bash, Python, Apache Spark, Kafka, Airflow, Docker, and Cassandra. • Directed cross-functional teams in data migration projects, consistently meeting deadlines and exceeding client expectations. • Designed and developed data pipelines for migration and transformation from on-premise databases to cloud storage. • Expert in ETL processes, data migration, and database architecture. • Strong problem-solving skills: Ensured data quality and performance, conducted reporting tasks, and performed root cause analysis. • Communicated technical concepts to non-technical stakeholders, fostering collaboration. • Updated skills with emerging technologies like Prefect, MAGE, and chatbot creation with LLMs. • Developed APIs/frameworks for data wrangling, scraping, and ingestion using Spark, with support from Snowflake, AWS S3, Databricks, and BigQuery.Additional Projects • Data Engineering - Ingesting Data to GCP with Airflow (2024): Developed a DAG and ingestion pipeline to load compressed raw data into a data lake. • Data Ingestion with Apache Cassandra (2024): Streamlined data ingestion into Cassandra using Docker and Python. • Data Pipeline - Docker and Postgres Integration (2023): Used Docker with Postgres for data management, ensuring consistent application performance. • EDA on Vehicle Insurance Customer Data (2023): Conducted exploratory data analysis and cleaning for future analysis. • Real-Time Data Monitoring with Evidently AI (2024): Developed a POC integrating tools to monitor machine learning systems using Evidently AI, Grafana, and PostgreSQL. • Ride Duration Prediction Model (2024): Deployed a model for predicting ride duration based on Yellow Taxi Trip Records. • Setting up dbt Cloud with BigQuery (2023): Configured dbt Cloud with BigQuery to enhance automation and security.