Repository navigation
promtop: no rising/falling notes for the runtime's allocation and GC counts - #9
Merged
Merged
Conversation
…counts
go_memstats_mallocs_total got a "falling" note whenever a burst of work
ended, a series of /metrics renders for one: as an event counter it was
held to Poisson noise, and with millions of allocations any change of
50% passed. The gcHeap names covered heap gauges, not these counters.
Counts of the runtime's own work follow the service's load in bursts,
so they are no news: allocations, frees, promotions and collections per
second (go_memstats_{mallocs,frees,alloc_bytes}_total,
go_memory_{allocations,allocated_bytes}_total, go_gc_cycles_*,
jvm_memory_pool_allocated_bytes_total,
jvm_gc_memory_{allocated,promoted}_bytes_total, and the collection rate
that jvm_gc_collection_seconds and jvm_gc_pause_seconds plot without
quantiles). They stay in the summary and on the panels with their trend;
only the note goes. GC pause times (go_gc_duration_seconds quantiles,
jvm_gc_pause_seconds_max) keep their notes, as do a growing RSS and 5xx.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01US3HyG56Ex8D8KAVo4hjax
…failing Render with no history yet reads the metrics itself. When the scrape loop's first read was still in the handler, the in-process source refused a second call at once and Render returned "metrics handler still busy with an earlier scrape" (TestRenderSummaryNotable failed on macOS CI that way). The loop and Draw now take turns reading the source, and Draw uses the loop's scrape when it lands first. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01US3HyG56Ex8D8KAVo4hjax
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
go_memstats_mallocs_totalgot afallingnote in the summary'snotablelist whenever a burst of work ended, for example a series of/metricsrenders. The counter is read asKindRatewithUnitNone, somoved()holds it to Poisson noise (3*sqrt(c1+c2)). With millions of allocations that threshold is almost nothing, so every change overbigChangecounted as significant. ThegcHeapnames catch heap gauges but not these counters.Change
internal/summary:runtimeWork(p)drops trend notes for the runtime's per-second counts of its own work. These counts follow the service's load in bursts:go_memstats_mallocs_total,go_memstats_frees_total,go_memstats_alloc_bytes_totalgo_memory_allocations_total,go_memory_allocated_bytes_total(OTel Go)go_gc_cycles_*jvm_memory_pool_allocated_bytes_total,jvm_gc_memory_allocated_bytes_total,jvm_gc_memory_promoted_bytes_totaljvm_gc_collection_seconds,jvm_gc_pause_seconds(summaries without quantiles plot collections per second)go_gc_duration_seconds(quantiles of pause times),jvm_gc_pause_seconds_maxThe check only applies when the plotted value is a rate. A GC summary that plots a pause quantile keeps its notes. The metrics stay in the summary and on the panels with their
trend; only the note goes.promtop/dashfix: CI on this PR hit a race that is already in v0.5.0. ARendercall made while the scrape loop's first read was still in the handler failed with "metrics handler still busy with an earlier scrape" (TestRenderSummaryNotable, macOS). The loop andServer.Drawnow take turns reading the source, andDrawuses the loop's scrape if it lands first. The public API is unchanged.Tests
TestRuntimeChurncovers each metric with three cases: allocations fall tenfold, allocations rise tenfold, and steady allocations with GC pauses ten times longer. In every case a growingprocess_resident_memory_bytesgetsrisingand 5xx responses geterrors. Without the fix the test failed for every allocation and GC count. Removing the rate-only condition makes the "pauses grow" case fail.TestRenderDuringFirstScrapecallsRenderwhile the loop's first scrape is held in the handler. It failed before the fix every time.make testpasses, and so do 300-raceruns of the dash render tests. The docs (README, the skill and its reference) list the new exclusions.🤖 Generated with Claude Code
https://claude.ai/code/session_01US3HyG56Ex8D8KAVo4hjax