Navya e

About Navya e

Senior Frontend-focused Full Stack Developer with around 12+ years of experience building…

Experience

1. Data Engineer

   高盛

   Jun 2023 — Present · US

   Developed a comprehensive data quality assurance framework for the batch processing environment, implementing data validation and error handling mechanisms to ensure high data integrity and accuracy. Leveraged GCP services, including Dataproc, GCS, Cloud Functions, and BigQuery to optimize data processing and analysis, achieving improved system efficiency and cost reductions. Developed data pipelines using Airflow to ingest data from various file based sources such as (FTP, SFTP, API, Main Frame) in GCP for ETL-related jobs, utilizing various airflow operators to streamline data processing workflows. Leveraged the power of Airflow\'s scheduling capabilities to automate data pipelines, ensuring timely data updates and delivery. Implemented data encryption and access controls to ensure the security and privacy of sensitive data in transit and at rest within BigQuery. Integrated machine learning models with BigQuery and Cloud ML to enable advanced analytics and predictive insights, contributing to data-driven decision-making processes. Conducted performance tuning on BigQuery queries to optimize execution times and reduce query costs, resulting in faster data retrieval and lower expenses. Utilized PySpark in-memory processing capabilities to accelerate data computations and enhance data querying performance, leading to faster data transformation and analysis. Created custom monitoring and alerting solutions using Stackdriver and Cloud Monitoring to proactively identify and resolve data processing issues. Performed data validation and quality checks during the ETL process to ensure data consistency and integrity, identifying and resolving anomalies and discrepancies. Contributed to the development of a data catalog for organizing and discovering datasets, enhancing data discovery and reuse across the organization. Implemented serverless Cloud Functions to automate lightweight data processing tasks, enabling efficient routine data operations.