Lidya H.

About Lidya H.

Data Engineer with 5+ years of experience bridging the gap between business needs and technical solutions. Proven ability to design, develop, and implement large-scale data systems on cloud platforms (AWS, Azure, GCP) for enhanced accessibility, efficiency, and scalability. Skilled in Agile methodologies and data processing technologies (PySpark, SQL) to deliver robust data solutions. Extensive experience in user training and implementation, ensuring smooth user adoption.

Experience

1. Senior Data Engineer

   Tata Consultancy Services

   Sep 2019 — Present · US

   Designed and implemented end-to-end data pipelines using Apache Spark and Azure Data Factory, enabling efficient processing of high-volume machine data, logs, and time-series datasets for real-time analytics.Developed and optimized ETL workflows with Apache Kafka, PySpark, and Azure Data Lake Storage for seamless ingestion, transformation, and storage across distributed systems.Built and managed real-time streaming solutions leveraging Kafka and Spark Structured Streaming, ensuring sub-second latency for monitoring and event-driven analytics.Created and deployed custom data processing components to enhance parsing, indexing, and visualization capabilities, tailored to business and compliance needs.Optimized query performance and reporting dashboards using Databricks SQL and Power BI, reducing compute costs while delivering faster insights to stakeholders.Partnered with cross-functional teams to define real-time alerting, monitoring, and KPI dashboards, providing business units with actionable insights and system health metrics.

Education

• Seattle Central College

  Associate of Science (AS), Associate of Science

  2014 — 2016

• University of Washington Bothell

  Bachelor of Science (B.Sc.) , Computer Science

  2017 — 2019

Skills

• Sql
• Microsoft Office
• Microsoft Excel
• Java
• Html
• Python