Andrei Badescu Knal

Senior Site Reliability Engineer @Qwilt

Portland, OR, US

Designed a metric pipeline with LABEL enrichment, using Logstash layer and near real-time 1m aggregations based on enriched labels on a single homed OSS InfluxDB cloud compute instance.Built Dashboards using variables to allow viewing metrics at different aggregation levels based on Topology keywords such as ALL, Network, Site, Group, and Individual machine, which help Operations engineers to drill down from the Network level to identify which machine is misbehaving.Helped debug and troubleshoot a Cloud infrastructure Load-Balancer bug that was previously ignored for a long time due to misunderstood or unclear logging and the complexity of troubleshooting network traffic in cloud environments. The bug was affecting Critical routing heartbeat health-check systems, which caused healthy systems to randomly flap health states. Suggested a design that was accepted and implemented for fully automating OS deployment on Cisco UCS and other server installations over CIMC | IDrac | other BMC.Deployed and configured Telegraf using Puppet due to operational constraints, migrated Logstash enrichment logic and dynamic dictionaries needed for LABEL enrichment to Telegraf metric shipping scripts to reduce points of failure, simplify the design, and facilitate TSDB transition to PromQL-based backends.Migrated metrics from OSS InfluxDB server to Grafana Cloud Mimir, translated dashboards from InfluxQL to PromQL, implemented anomaly prediction and outliers machine learning to reduce operational toil.