docs: add symptom-first troubleshooting guides - #1296
Conversation
The metric registry says what each instrument measures, and the observability guide says how telemetry is collected, but neither takes an operator from a symptom to a cause. A user reporting a slow request or a 503 had no path through the data. These three guides each start at a symptom and end at a component. Every step gives a query, says how to read the result, and names the next step or the guide to switch to. The shared index holds what all three need: the two spellings of the instrument names and which ingest path produces each, the four reasons a query returns nothing, and the pointer to the subsystems that emit no metrics at all. Every query was run against a live cluster, and the readings drove the text: the resume label is only meaningful next to the outcome, a fleet gauge that every replica reports needs max rather than sum, and a phase histogram has to exclude failures before its quantile means anything.
|
I noticed several drifts between this guide and the metrics definition (like the error code etc) and I think it will become more programatic in the future because it will be very hard to keep the guide the metrics definition synced. I think we should make the guide very thin. The durable structure is: guide owns the decision tree and interpretation, registry owns definitions, and the guide links rather than paraphrases. Every value taxonomy or default it copies inline is future debt. |
| Calculate the hit ratio as `hit / (hit + miss)`. Keep `error`, `cancelled` and | ||
| `timeout` out of the denominator. | ||
|
|
||
| If the outcome is `error`, `error.type` holds the HTTP status of the registry. |
There was a problem hiding this comment.
It claims ate.imagecache.requests error.type is only 401/403 (credential), 429 (rate limit), and _OTHER for everything else, but the registry (docs/metrics/registry/metrics.yaml:973-976) and internal/imagecache/metrics.go pass 404, 500, 502, 503, 504 through as literal values
2 participants
Fixes #1294
It's converted from this public doc. Please feel free to drop comments in the doc and I can update this PR.