List recent failures under each failure count on /ui/analytics

[simonchrz]

The failure counts under Runtime on /ui/analytics say how often something failed, but not what. When npm · fetch_failed shows 7, there is no way to see which packages failed without digging through the logs.

This adds an expandable Latest N list under each failure count, newest first. Each entry shows the package URL, the filename, the upstream URL or storage path, the error, and how long ago it happened.

Changes

• RecordUpstreamError, RecordStorageError, RecordIntegrityFailure, RecordHealthProbeFailure and RecordScanError take a metrics.FailureDetail (PURL, filename, URL, path, error). All existing call sites pass what they already have in scope.
• The last 20 failures per counter series are kept in memory and cleared on restart, like the counters themselves.
• Userinfo, query strings and fragments are stripped from URLs before they are stored, including URLs inside error messages (e.g. Get "https://user:token@…?sig=…").
• Package names and URLs never become Prometheus labels.
• New setting ui_failure_details / PROXY_UI_FAILURE_DETAILS, off by default for the same reason as ui_request_sources: /ui has no authentication. While it is off and failures exist, the page shows a note naming the setting.
• Documented in the README, docs/configuration.md and config.example.yaml.

Circuit-breaker trips and blocked scans are left as they are: their call sites have no per-failure detail beyond the labels.

Testing

• Unit tests for ordering, the per-series cap, keeping series separate, credential stripping, and returning a copy.
• Handler tests assert that a failed fetch and a failed store record the package, URL or path, and error.
• Page tests: the detail is absent from the HTML by default and rendered when the setting is on, without the query string.
• The failure history is process-wide, so every test that records or reads failures directly empties it before and after it runs (metrics.ResetRecentFailures). The tests pass with -count=2 and with go test -count=3 -shuffle=on ./internal/metrics ./internal/server ./internal/handler.
• go test -race, go vet and golangci-lint v2.13.1 pass.
• Manually: requested a missing left-pad tarball; the npm · fetch_failed row expands to pkg:npm/left-pad@99.99.99, its registry URL, and artifact not found.

[andrew]

Please isolate the process-global recent failure history in the new tests. Cleanup alone does not prevent records from earlier tests or earlier repetitions from reaching these assertions.

go test ./internal/server -run '^TestRuntimeViewAttachesRecentFailuresWhenEnabled$' -count=2 fails on the second run at analytics_failures_test.go:38: it finds two recent failures instead of one.

go test ./internal/metrics -run '^Test(RecordUpstreamOperations|RecentFailuresNewestFirst)$' -count=2 fails at failures_test.go:20: it finds three failures instead of two because TestRecordUpstreamOperations leaves a record behind.

Reset or isolate the history before assertions and clean up every test that writes it, so repeated runs and changes in test order remain reliable.

[simonchrz]

Confirmed both, and fixed in b131db3. Tests only cleared the history after running, and not all of them did. TestRecordUpstreamOperations and TestRecordStorageOperations in metrics, TestGatherSeesRegisteredMetrics, and every server test that records failures left records behind.

• ResetRecentFailures is now exported, so the server and handler tests can use it.
• Every test that records or reads failures directly now calls an isolateRecentFailures(t) helper. It empties the history before the test runs and again in t.Cleanup.
• The handler tests for a failed fetch and a failed store now assert exactly one failure instead of checking only the newest. With -count=2 they found two before this change.

Both of your commands now pass, and so does go test -count=3 -shuffle=on ./internal/metrics ./internal/server ./internal/handler. The race suite and lint pass too.