Skip to content

promtop: no rising/falling notes for the runtime's allocation and GC counts - #9

Merged
igorrius merged 2 commits into
mainfrom
ccr-fe082a8a-egtxyr
Nov 4, 2026
Merged

igorrius merged 2 commits into
mainfrom
ccr-fe082a8a-egtxyr

Conversation

@igorrius

@igorrius igorrius commented Nov 4, 2026 •

Copy link
Copy Markdown
Owner

go_memstats_mallocs_total got a falling note in the summary's notable list whenever a burst of work ended, for example a series of /metrics renders. The counter is read as KindRate with UnitNone, so moved() holds it to Poisson noise (3*sqrt(c1+c2)). With millions of allocations that threshold is almost nothing, so every change over bigChange counted as significant. The gcHeap names catch heap gauges but not these counters.

Change

internal/summary: runtimeWork(p) drops trend notes for the runtime's per-second counts of its own work. These counts follow the service's load in bursts:

Metric Decision
go_memstats_mallocs_total, go_memstats_frees_total, go_memstats_alloc_bytes_total no trend note
go_memory_allocations_total, go_memory_allocated_bytes_total (OTel Go) no trend note
go_gc_cycles_* no trend note (GC count follows allocations)
jvm_memory_pool_allocated_bytes_total, jvm_gc_memory_allocated_bytes_total, jvm_gc_memory_promoted_bytes_total no trend note
jvm_gc_collection_seconds, jvm_gc_pause_seconds (summaries without quantiles plot collections per second) no trend note for that rate
go_gc_duration_seconds (quantiles of pause times), jvm_gc_pause_seconds_max notes kept: pauses are short by design, so pauses that grow are news

The check only applies when the plotted value is a rate. A GC summary that plots a pause quantile keeps its notes. The metrics stay in the summary and on the panels with their trend; only the note goes.

promtop/dash fix: CI on this PR hit a race that is already in v0.5.0. A Render call made while the scrape loop's first read was still in the handler failed with "metrics handler still busy with an earlier scrape" (TestRenderSummaryNotable, macOS). The loop and Server.Draw now take turns reading the source, and Draw uses the loop's scrape if it lands first. The public API is unchanged.

Tests

TestRuntimeChurn covers each metric with three cases: allocations fall tenfold, allocations rise tenfold, and steady allocations with GC pauses ten times longer. In every case a growing process_resident_memory_bytes gets rising and 5xx responses get errors. Without the fix the test failed for every allocation and GC count. Removing the rate-only condition makes the "pauses grow" case fail.

TestRenderDuringFirstScrape calls Render while the loop's first scrape is held in the handler. It failed before the fix every time.

make test passes, and so do 300 -race runs of the dash render tests. The docs (README, the skill and its reference) list the new exclusions.

🤖 Generated with Claude Code

https://claude.ai/code/session_01US3HyG56Ex8D8KAVo4hjax

…counts

go_memstats_mallocs_total got a "falling" note whenever a burst of work
ended, a series of /metrics renders for one: as an event counter it was
held to Poisson noise, and with millions of allocations any change of
50% passed. The gcHeap names covered heap gauges, not these counters.

Counts of the runtime's own work follow the service's load in bursts,
so they are no news: allocations, frees, promotions and collections per
second (go_memstats_{mallocs,frees,alloc_bytes}_total,
go_memory_{allocations,allocated_bytes}_total, go_gc_cycles_*,
jvm_memory_pool_allocated_bytes_total,
jvm_gc_memory_{allocated,promoted}_bytes_total, and the collection rate
that jvm_gc_collection_seconds and jvm_gc_pause_seconds plot without
quantiles). They stay in the summary and on the panels with their trend;
only the note goes. GC pause times (go_gc_duration_seconds quantiles,
jvm_gc_pause_seconds_max) keep their notes, as do a growing RSS and 5xx.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01US3HyG56Ex8D8KAVo4hjax
…failing

Render with no history yet reads the metrics itself. When the scrape
loop's first read was still in the handler, the in-process source
refused a second call at once and Render returned "metrics handler
still busy with an earlier scrape" (TestRenderSummaryNotable failed on
macOS CI that way). The loop and Draw now take turns reading the source,
and Draw uses the loop's scrape when it lands first.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01US3HyG56Ex8D8KAVo4hjax
@igorrius igorrius self-assigned this Nov 4, 2026
@igorrius
igorrius marked this pull request as ready for review November 4, 2026 13:19
@igorrius
igorrius merged commit e481956 into main Nov 4, 2026
5 checks passed
@igorrius
igorrius deleted the ccr-fe082a8a-egtxyr branch November 4, 2026 13:20
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants