runtime,cl: add Go-compatible sampled memory profiling#2027
Open
cpunion wants to merge 6 commits into
Open
Conversation
Codecov Report❌ Patch coverage is
📢 Thoughts on this report? Let us know! |
cpunion
force-pushed
the
codex/stage5-memprofile
branch
4 times, most recently
from
July 4, 2026 12:52
c647b14 to
2b22d07
Compare
cpunion
force-pushed
the
codex/stage5-memprofile
branch
2 times, most recently
from
July 8, 2026 07:02
4695ad3 to
9d9f6b0
Compare
cpunion
force-pushed
the
codex/stage5-memprofile
branch
3 times, most recently
from
July 9, 2026 05:52
e6bbe61 to
2e36dbf
Compare
8 tasks
cpunion
added
go-test-compat
20 tasks
Replaces the size-class counters with gc-shaped heap profiling: sampled allocations are attributed to physical call stacks at exact statement lines, and records hold RAW sampled counts — consumers (pprof, goroot heapsampling.go) apply the Poisson correction themselves, exactly as with gc. - Sampling mirrors gc's mcache.nextSample: bytes count down to an exponentially distributed threshold (mean MemProfileRate), sample once on crossing, redraw. The memoryless distribution is load-bearing: with any bounded-support threshold a near-periodic allocation pattern phase-locks the sample points onto the large sites (observed 1.6x per-site skew on heapsampling's interleaved sizes). ln() is a small local approximation — the runtime core cannot import math. - Stacks come from the FP walk at sample time (fpCallers via a hook the public runtime registers), bucketed by stack hash; allocator plumbing (including __llgo_stub. wrapper frames of the hook) is trimmed at read time. A reentrancy flag spans the whole decision path: threshold drawing and bucket allocation themselves allocate, and a recursive sample overflows the stack. - Heap allocations get statement anchors in tracked functions, and a package that reads the memory profile (runtime.MemProfile / MemProfileRate under either the "runtime" or the patched lib-runtime spelling) pins all its trackable functions: per-site attribution loses sites to inlining otherwise. Profiling packages are rare and accuracy beats inlining there; gc gets both via its inline tree (P4). goroot heapsampling.go passes on darwin/arm64 and linux/arm64 (the latter was a pre-existing platform gap). Depends on --icf=none from the line-directive PR: heapsampling's three identical wrapper functions must keep distinct pcs. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
An acceptance regression asserts exact per-line attribution at rate=1 (raw counts are exact there), and a cl unit test covers the memprofile-package pinning criterion under both runtime spellings. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
… capture path The frame-table init allocates; when one of those allocations crossed the sampling threshold, captureMemProfileStack -> fpCallers re-entered initRuntimeFuncPCFramesSlow on the thread that already held the Busy latch and usleep-spun forever. First testing.callerName call of a test binary triggers the init, so whole test binaries hung at startup — which packages hit it depends on the deterministic threshold sequence meeting the binary's pre-init allocation volume: net/rpc and net/rpc/jsonrpc under go1.24 stdlib, net/http/expvar/cookiejar under go1.26 (CI shard timeouts on ubuntu, both attempts). Entering an Uninit latch from the capture path is safe (the whole sample runs under memProfileInSample, so init's own allocations cannot re-sample); only Busy must not be waited on. Drop that one sample.
cpunion
force-pushed
the
codex/stage5-memprofile
branch
from
July 18, 2026 23:37
2e36dbf to
ef62b2d
Compare
cpunion
changed the title
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Implements Go-compatible, stack-keyed sampled heap profiles for LLGo.
Problem
LLGo's
runtime.MemProfileonly kept size-class counters, so heap profiles had no allocation stacks or source-line attribution. During integration with currentmain, the first sample could also lazily initialize frame metadata from insideAllocZ/AllocU; that reentrant allocation corrupted reflected closure calls (_demo/go/reflectcallfn).Changes
runtime.MemProfileRateand stores raw counts keyed by physical call stack.Coverage and validation
heapsampling.goconformance now passes and its xfails are removed.packageReadsMemProfilecoverage is 95.7% on macOS/arm64 and Linux/amd64.reflectcallfnexecutions on the current-main merge tree.Uses the frame-pointer/funcinfo infrastructure already merged in #2024.