Govardhan Digumurthi

Experience

1. Senior Data Engineer

   ENGIE North America Inc.

   Feb 2023 — Present · Houston, TX, US

   Built a maintenance framework for Redshift to automate data vacuuming, manage outdated datasets via system tables, enforce data retention policies, and track data skew metrics for optimized distribution. This streamlined Redshift performance, minimized storage bloat, and maintained high query efficiency across complex workloads- Engineered a real-time pipeline for Texas Power Plants using PITS (PI Server), Kinesis Data Streams, Firehose, and Lambda to stream live sensor data to S3 and Elasticsearch- Designed and implemented a scalable, cost-saving pipeline that handles high-volume refresh of approximately 150 million records from on-prem SQL Server deprecating the outdated, long-running stored procedures- Led data governance initiatives emphasizing data quality, metadata management, and security for 50 user groups with over 350 active database users. Developed a custom in-house security model to ensure compliance across data pipelines, integrating with internal services to prevent unauthorized access and streamline access management- Implemented a serverless data processing pipeline using AWS Lambda and Step Functions to ingest, transform, and load real-time streaming data from IoT devices into Amazon Redshift. This solution reduced data latency by 40% and improved scalability, handling thousands of events per second- Implemented an adaptable strategy to make near real-time data from Yes Energy available in Snowflake using event-driven, serverless services like SNS, SQS, Lambda, and Snowpark for data transformations and fact table creation.