feat(runs): the investigate prompt names the labels and the figure (archie #2005)
An alert about a filesystem carries instance and mountpoint, and those are the labels the forecast query needs, so the prompt names both and gives the query that answers what an operator asks: the seconds until the disk is full, from predict_linear and deriv over the same window. A negative figure is the filesystem not filling, which the prompt says to report as that rather than as a time.
This commit is contained in:
1 file changed
+11
-6
@@ -13,12 +13,17 @@ The event is a claim, not a fact. Check it against what you measure, and if
|
||||
they disagree (an alert says a disk is full and it is at 40%), say so first:
|
||||
that disagreement is a finding in itself.
|
||||
|
||||
A trend is a query, not an estimate. When something is filling up, get the
|
||||
time until it's full from the metrics server rather than working it out from
|
||||
raw points: Prometheus answers `predict_linear` through `query`, so with node
|
||||
exporter's filesystem metrics a figure comes from a query such as
|
||||
`predict_linear(node_filesystem_avail_bytes{instance="host:9100",mountpoint="/"}[6h], 24*3600) < 0`.
|
||||
Report that figure and the query it came from.
|
||||
A trend is a query, not an estimate. An event about a filesystem carries
|
||||
`instance` and `mountpoint`, and those labels go into the metric's own
|
||||
selector: ask the metrics server for the forecast rather than working one out
|
||||
from raw points. Prometheus answers `predict_linear` through `query`, and with
|
||||
node exporter's filesystem metrics the seconds until the disk is full are
|
||||
|
||||
-predict_linear(node_filesystem_avail_bytes{instance="<instance>",mountpoint="<mountpoint>"}[6h], 0)
|
||||
/ deriv(node_filesystem_avail_bytes{instance="<instance>",mountpoint="<mountpoint>"}[6h])
|
||||
|
||||
Report the figure that query returns and how long it is. A negative figure
|
||||
means the filesystem isn't filling: say that instead of giving a time.
|
||||
|
||||
When you're done, reply with your findings in plain text:
|
||||
|
||||
|
||||
Reference in new issue
Block a user