Proposal: runtime funcinfo metadata and fast FuncForPC lookup

[cpunion]

Proposal: Runtime Funcinfo Metadata and Fast Runtime Lookup

Goal

Give LLGo Go-compatible runtime introspection — runtime.Caller, Callers, CallersFrames, FuncForPC, Func.FileLine, runtime.Stack, panic tracebacks — without DWARF, llvm-symbolizer or libunwind on the primary path, with DCE safety, controlled binary size, and performance at or beyond Go.

Architecture (three layers + a conformance gate)

1. Metadata model (runtime: add Go-style funcinfo find index #2012, Stage 1) — Go-1.26-style compact function records + shared string pools + statement-level pcline records, emitted as DCE-safe sections: ELF SHF_LINK_ORDER (honored by --gc-sections), Mach-O live_support + one linker-private atom symbol per record (verified under ld64/lld -dead_strip, incl. LTO). findfunctab geometry follows Go (4KiB buckets × 16 subbuckets) with one deliberate deviation: uint16 subbucket deltas — LLGo has no 16-byte MINFUNC guarantee, so Go's uint8 could overflow; uint16 makes it structurally impossible. Runtime falls back to first-use table construction when the link-phase table is absent.
2. Link-phase table (runtime: link-phase ftab/findfunctab generation (Stage 2, P1–P3) #2016, Stage 2 P1–P3) — after linking, internal/pclnpost deduplicates records against the symbol table (FNV: a record is canonical iff its anchor's owning symbol hashes to its symbolID, or is that symbol's __llgo_stub. wrapper — LTO inline copies fail exactly), sorts, and rewrites the entry section in place with a prebuilt ftab+findfunctab the runtime adopts zero-copy. Mach-O specifics proved out: chained-fixup chain surgery, BIND-node decoding through the imports table (exported-function anchors are binds, not rebases), the header base slot spliced back as a live rebase so dyld writes the slid address, ad-hoc re-sign only when the input was signed. Overflow spills the blob into the larger stub section behind a LLGOFTB2 redirect — rows are never dropped (a gap-y table silently mis-names function-value pcs). Startup pre-warm (adopt + page-touch + one synthetic lookup) mirrors Go, whose pclntab is warm by main.
3. Physical unwinder (runtime: LLGo-owned frame-pointer unwinder (Stage 5) #2019, Stage 5) — every Go function on supported targets keeps the frame-pointer chain ("frame-pointer"="non-leaf"); Caller/Callers/CallersFrames/Stack and the unrecovered-panic dump walk [fp]/[fp+w] directly, symbolized through layer 2's tables with Go's pc-1 return-address convention. The walk is bounded to the program's own text (libc frames without FP discipline decode as wild pcs). Shadow-stack instrumentation is no longer emitted (LLGO_SHADOW_STACK=1 keeps the legacy emitters one release); linux binaries no longer link -lunwind (the clite fallback walks FP with dladdr names). Semantics are gc ground truth — physical stacks show every real frame; skip counts verified against go (interface chain MARK at skip 3, closure at 4). Two hardening rules: the compiler records its FP decision in a per-binary __llgo_fp_chain byte the runtime trusts (no inference from table presence), and every lookup path refines function records through one refinePCSymbolLine helper (statement record at pc, then pc-1) so FuncForPC/FileLine/CallersFrames cannot disagree on line attribution — the class of bug behind all six amd64 debug rounds.
4. User-visible conformance (runtime,cl: Go-style panic tracebacks and exact log/slog/testing locations, with gc-verified acceptance suite #2023, acceptance) — what application developers actually see, held to an agreed bar: format and user-code file:line match gc exactly; runtime-internal, patched-stdlib and startup frames may differ. Delivered on top of runtime: LLGo-owned frame-pointer unwinder (Stage 5) #2019: gc-style unrecovered-panic tracebacks (goroutine 1 [running]: + fn(...) + file:line +0x<pc-entry>, via a runtime hook with clite fallback); exact log.Lshortfile/slog AddSource/t.Errorf locations (previously ???:1 — the blanket stdlib tracking exclusion is gone, the same per-package reaches-runtime.Caller analysis now applies uniformly, callers of pc-consuming functions keep frames + statement anchors); main display naming regardless of module name, closures as gc's .funcN; main.main/init///go:noinline frames pinned (noinline is free there) so panic bottom frames are line-exact. Locked in by a gc-verified acceptance suite (test/go/caller_acceptance_test.go: panic, log/slog text+JSON, llgo test failure lines, goroutine/init/defer callers, FuncForPC names, errors-with-stack idiom). Cost: hello-world +368 bytes, plain.* untouched, no golden changes.

Data structures and lookup algorithm

compile time (per package)          link phase (pclnpost, per binary)      run time
--------------------------          ---------------------------------     --------
func f() {            emitted as    1. parse entry/stub anchor sections    adopt zero-copy
  funcinfo record ──┐  DCE-safe     2. FNV ownership: record canonical     (validate magic,
  pcline anchors  ──┤  sections        iff anchor's owning symbol hashes    read dyld-slid
  (per call site)   │  (ELF            to its symbolID (LTO inline           base pointer)
}                   │  link-order /    copies fail exactly) → dedup
string pools  ──────┘  Mach-O       3. sort by entry, rewrite in place:
(pkg/name/file,        live_support)  ┌─────────────────────────────┐
content-deduped)                      │ header: magic, text base ●──┼─ spliced into fixup
                                      │ ftab: {entryOff, funcIdx}[] │  chain as live rebase
records: {symPkg,symName,namePkg,     │ findfunctab: 4KiB buckets   │ (overflow → LLGOFTB2
 nameName,fileRoot,fileName,line}     │   × 16 × uint16 delta       │  spill into stub sec)
 as u16 indexes into string pools     └─────────────────────────────┘

PC → location (the only symbolization rule, refinePCSymbolLine):

pc (return address) ── pc-1 ──► off = pc − base          ← Go's convention: attribute
bucket = off >> 12   (4KiB)                                the call, not the next stmt
sub    = (off >> 8) & 15
i      = bucketBase[bucket] + uint16 subDelta[sub]
while ftab[i+1].entryOff ≤ off: i++                      ← short forward scan
funcIndex → record → symbol / name / file / decl line
pcline nearest-below(pc, same entry), then (pc-1)        ← statement-exact line for
  → statement line                                          tracked frames
caches: 4-way set-assoc pc cache → per-ftab-row *Func → frameSymbolResult (per-pc memo)

Physical stack walk (fpCallers; same discipline in the clite C fallback):

stack (grows down)      every Go function keeps x29/rbp chained
┌─────────────┐         ("frame-pointer"="non-leaf"):
│ caller frame│◄─┐        ret  = [fp + wordsize] → pc[n++] (must lie in program text)
│  saved ret ─┼──┼─►pc    prev = [fp]            (strictly increasing, ≤1MiB stride,
│  saved fp ──┼──┘        fp   = prev             word-aligned, else stop)
└─────────────┘         availability = per-binary __llgo_fp_chain byte × tables present

Frame-keeping analysis (per package, cached): track a function — noinline + no-tail-call

• statement anchors — iff it (1) transitively reaches runtime.Caller/Callers within its
  package, or (2) statically calls another package's criterion-(1) function (the frame a
  fixed Caller depth attributes), or (3) is main.main / init (program-unique, noinline is
  free), or (4) carries //go:noinline anyway. Everything else stays fully optimizable.

Stage / PR status

Stage	Scope	Status
1	Metadata model	#2012 — all green (codecov 91.3%), merge-ready
1.5	ns monotonic clock	#2015 — done, carried in #2012; close after it merges
2	Link-phase table	#2016 — all green (codecov 88.9%), merge-ready
3	LTO policy	data collected; PR pending (main+LTO mac crash worth root-causing there)
4	Zero-copy names, prebuilt pcline, !pcsections, section shrink, stub removal	staged (ordered below)
5	FP unwinder	#2019 — all green (48 checks incl. codecov), merge-ready; review cleanup folded in (#2022 closed)
5b	Acceptance & conformance	#2023 — based on #2019; first CI round all green, conformance round in CI
5c	goroot conformance	#2024 — based on #2023: //line spelling, --icf=none pc identity, 12 xfails retired
5d	Panic-site snapshots	#2026 — based on #2023: deferred callers & fault stacks see panic frames (C→Go fault chains, SIGFPE/SIGBUS recoverable); 3 more xfails retired
5e	Heap profiling	#2027 — based on #2024 (needs --icf=none): gc-shaped sampled memory profiles, exponential thresholds, raw counts; heapsampling green on darwin+linux arm64
5f	Fault stacks	#2028 — based on #2019: hardware faults recoverable (incl. SIGFPE) with fault-site C→Go stacks via dynamically-resolved libunwind (no link dependency; nongnu flavor names static C symbols from .symtab) + FP-chain fallback; overlaps #2026's fault half — second lander rebases

What each PR owns (with final numbers, best-of, mac/linux)

	#2012	#2016	#2019
Contribution	metadata + first-use tables + shadow stack	prebuilt table, cold/scale, LTO integrity, pre-warm	physical stacks, instrumentation & libunwind removal
hot.Caller0	54/65ns	55/66ns	17/37ns, +LTO 14/29
hot.CallersOnly	129/154ns	131/153ns	31/56ns, +LTO 28/47
hot.FuncForPC / FileLine (mac)	9/10ns	9/10ns	2/3ns, +LTO 1/1
deep.Direct512	87–95µs	87–95µs	2.8µs (Go 9.7µs)
deep.CallersFramesAll(512)	105–120µs	105–120µs	3.7µs (Go 12µs)
cold.FirstFuncForPC	40.7/13µs	3–5/2.9µs	2.5/1.5µs (Go 2.8/1.2µs)
bigfunc.Work (16×2000, mac)	433µs	433µs	18.1µs (Go 30µs)
bigfunc.FileLineMid	—	176ns (+LTO 49)	57ns, +LTO 27 (Go 86µs)
Binary size	+18–29% vs main	= #2012	≤ #2016; mac all scenarios ≤ Go
Semantics	instrumented frames only	same	gc-conformant full physical stacks

Reference bar: Go 1.26 hot.Caller0 155/241ns; per-fresh-pc FuncForPC 40–125ns (zero-copy pclntab views) vs LLGo 0.4–1.7µs (string materialization — the headline P4 item). plain.* (fib/json/sort/map) is identical across all variants: code that never introspects pays nothing. Scale sweeps (48×48/96×96 pkgs×methods, depth 128/512, big methods 32×200/16×2000) hardened #2016 (overflow spill, per-row Func cache — 96×96 batch lookups 8–19ms → 147µs +LTO, beating Go's 179µs) and quantified the shadow-stack tax (~160ns/frame) that #2019 removed.

Differences from Go's implementation (by design or accepted)

Aspect	Go (gc)	LLGo	Consequence
Table generation	linker emits pclntab	compiler emits DCE-safe sections; post-link rewrite prebuilds ftab/findfunctab; first-use fallback	same lookup model; needs no linker changes
Line granularity	per-instruction pcvalue tables	statement/call-site anchors (pcline records)	tracked frames statement-exact; plain mid-traceback frames declaration-adjacent until P4b
Inlined frames	logical unwinding via inline tree	physical FP walk; tracked functions keep noinline	inlined frames invisible (like C); P4 inline tree lifts
Defers during panic	run on top of the panicked stack	siglongjmp unwinds first; #2026 snapshots the panic pcs and splices them into caller-info views (validity tied to the recovering frame's liveness)	conformant incl. Caller depth through the gopanic-equivalent frame; panicking-statement lines in untracked functions await P4b
Names	zero-copy views into pclntab	strings materialized per function (P4a); display names normalized to gc forms (main., .funcN), linker symbols keep module path / $N	per-fresh-pc lookup 0.4–1.7µs vs Go's 40–125ns until P4a
Identical functions	never merged; distinct pcs	LLVM/linker may fold same-body functions to one address	FuncForPC reports the surviving symbol (issue58300; #1924 reimpl)
Traceback format	fn(args…) + file:line +0xoff	fn(...) + same shape; offset values differ (different codegen)	format-compatible; no argument values (needs arg metadata)
Goroutine tracebacks	goroutine N [state], runtime.main/goexit tail, Stack(all=true)	header fixed goroutine 1 [running], walk stops at program text, Stack(all) = current only	accepted (runtime frames out of conformance bar)
Availability gating	always on	per-binary __llgo_fp_chain byte + funcinfo tables; embedded/wasm keep tables, shadow-stack escape hatch	graceful degradation per platform table below
find index geometry	4KiB buckets, uint8 subbucket deltas (16-byte MINFUNC)	same geometry, uint16 deltas (no MINFUNC guarantee)	overflow structurally impossible

Platform support (three gates: FP attr → walker → symbolization)

Target	Status
darwin arm64/amd64, linux arm64/amd64	green in CI; AAPCS64 mandates FP, attr forces it on x86-64
embedded (ESP32…) / wasm	attr off by design (NeedsFramePointer): unwinder doesn't exist there (runtime gated !baremetal && !wasm); FP layout change also perturbed the conservative GC on ESP32-C3. Behavior = pre-Stage-5; LLGO_SHADOW_STACK=1 restores caller tracking — on wasm the shadow stack is the only possible mechanism
BSDs / android / ios	same ELF/Mach-O + SysV/AAPCS mechanics; expected to be a GOOS-whitelist change only
windows	needs COFF equivalents of the DCE-safe sections first (bigger than the unwinder); then either clang-kept RBP chains with the same walker, or SEH RtlVirtualUnwind. Currently attr off → graceful degradation
riscv64 / arm32	known gap: frame layout differs (RISC-V: ra at fp−8, prev fp at fp−16), walker offsets are arm64/x86-64-specific; the FP gate should whitelist arm64/amd64 explicitly until per-arch offsets land

Obstacle assessment for full support: BSD/android/ios — mechanical (GOOS whitelist). riscv/arm32 — per-arch walker offsets (~10 lines each; arm32 thumb/arm FP registers unify under clang's attr). Windows — bounded, path known: COFF IMAGE_COMDAT_SELECT_ASSOCIATIVE is the direct link-order analog, PE .reloc rewriting is simpler than the Mach-O chained-fixups we already handle, and clang keeps RBP chains on Win x64 so the same walker works (SEH optional); the real gate is LLGo's Windows runtime base, not funcinfo. wasm — physical unwinding is impossible by spec: full support means shadow stack as the permanent wasm backend (escape hatch exists; flip to default there), and wasm-ld section-GC semantics for the metadata need dedicated verification. Embedded — full support should be opt-in metadata (LLGO_FUNCINFO=0 exists) + P4d shrink given flash budgets; the conservative-GC/FP interaction is gated off today and only a precise-scanning project would lift it.

Fallback ladder guarantees no platform regresses below Stage 1: tables (FuncForPC/FileLine) work everywhere; Callers falls back shadow-stack → clite FP walk.

Follow-ups (priority order)

1. P4a zero-copy joined names (unsafe.String views over pre-joined strings): removes the 0.4–1.7µs per-function materialization — the only remaining structural gap vs Go on fresh-pc and >4k-target batch lookups; also enables per-funcIndex Func caching. Size cost ~+150–250KB stdlib (still ≪ Go's 1.41MB pclntab).
2. P4b prebuilt pcline + pcvalue-style instruction lines: kills the first-use pcline build (~55–400µs once at scale), the two remaining first-use cells in runtime: LLGo-owned frame-pointer unwinder (Stage 5) #2019, and the last line-exactness gap — plain user functions mid-panic-traceback show declaration-adjacent lines because statement anchors structurally require the owning function to survive (ELF link-order symbol), so full coverage needs a metadata mechanism instead of body asm.
3. P4c !pcsections as the site mechanism: removes the ±% inline/layout perturbation of body-embedded site asm.
4. P4d section shrink: name pool dominates at extreme function counts (96×96: 15.4 vs Go 9 MiB; typical scenarios already ≤ Go).
   4b. Foreign-function hole sentinels (pclnpost sees the full symbol table): insert non-Go boundary rows so a C pc between Go functions stops resolving to the preceding Go record — fixes linux fault-stack C-frame naming/misattribution (darwin already corrected by a dladdr cross-check) and general FuncForPC on C pcs. Also evaluate ELF .addrsig + --icf=safe here if the +1–2% of --icf=none ever matters.
5. Stage 4 stub removal (stubs already merged into the prebuilt ftab; mainly deletes generation), Stage 3 LTO policy.
6. Unwinder arch ports per the platform table; inline-tree (with P4b) to lift noinline from tracked functions (stdlib.Work's residual ~23µs vs Go 6.2µs is noinline + LLGo baseline, not unwind cost).

PR housekeeping / merge order

• Superseded & closed: runtime: add statement line caller frames #2002/runtime: speed up funcinfo entry lookup #2009/runtime: speed up funcinfo hot paths #2010/benchmark: add runtime funcinfo scaling probes #2011 (pre-runtime: add Go-style funcinfo find index #2012 line; runtime: track caller frames for LLGo metadata #1884 closed earlier). Backbone PRs runtime: add Go-style funcinfo find index #2012/runtime: link-phase ftab/findfunctab generation (Stage 2, P1–P3) #2016/runtime: LLGo-owned frame-pointer unwinder (Stage 5) #2019/runtime,cl: Go-style panic tracebacks and exact log/slog/testing locations, with gc-verified acceptance suite #2023 are closed as folded into runtime,cl,build: Go-style caller info — funcinfo tables, link-phase pcln, FP unwinder, gc-verified conformance (stage-5 backbone) #2024, which now carries the whole backbone as one review unit (rebased onto current main).
• Merge: cltest: share one go-build cache across captured runs #2035 and runtime: nanosecond monotonic clock; fix Linux clock IDs #2015 are MERGED. The backbone (runtime: add Go-style funcinfo find index #2012/runtime: link-phase ftab/findfunctab generation (Stage 2, P1–P3) #2016/runtime: LLGo-owned frame-pointer unwinder (Stage 5) #2019/runtime,cl: Go-style panic tracebacks and exact log/slog/testing locations, with gc-verified acceptance suite #2023) is folded into runtime,cl,build: Go-style caller info — funcinfo tables, link-phase pcln, FP unwinder, gc-verified conformance (stage-5 backbone) #2024 (those four closed); merge runtime,cl,build: Go-style caller info — funcinfo tables, link-phase pcln, FP unwinder, gc-verified conformance (stage-5 backbone) #2024 first, then in parallel: [Based on #2090] runtime,cl: preserve panic-site frames across defer and recover #2026, runtime,cl: add Go-compatible sampled memory profiling #2027 (on runtime,cl,build: Go-style caller info — funcinfo tables, link-phase pcln, FP unwinder, gc-verified conformance (stage-5 backbone) #2024), ssa,runtime: gc-conformant type-assertion panics and go:nointerface support #2031, [Based on #2090] ssa,runtime: tighten recover to direct deferred calls #2033 -> [Based on #2033] ssa,runtime: return runtime.Error values for assertions and faults #2034, cl,ssa,runtime: stack-object finalizer liveness #2036, runtime: recoverable hardware faults with fault-site stacks (dynamic libunwind + FP chain) #2028 (all based on runtime,cl,build: Go-style caller info — funcinfo tables, link-phase pcln, FP unwinder, gc-verified conformance (stage-5 backbone) #2024's branch; each shows only its own diff after runtime,cl,build: Go-style caller info — funcinfo tables, link-phase pcln, FP unwinder, gc-verified conformance (stage-5 backbone) #2024 lands). [Based on #2090] ssa,runtime: tighten recover to direct deferred calls #2033/[Based on #2090] runtime,cl: preserve panic-site frames across defer and recover #2026 recover contact point: second lander rebases. All 8 rebased onto main@8bb9e4e55 (post-hchan/PanicNilError/MethodByName-LTO), conflicts resolved, builds green.
• Old funcinfo PRs, superseded and closed: runtime: preserve line directive caller filenames #1903/runtime: preserve FuncForPC names for function values #1924 → runtime,cl,build: Go-style caller info — funcinfo tables, link-phase pcln, FP unwinder, gc-verified conformance (stage-5 backbone) #2024 (backbone); runtime: attribute panic caller lines #1925 → [Based on #2090] runtime,cl: preserve panic-site frames across defer and recover #2026 (deferred-caller/fault visibility done; panicking-statement lines in untracked functions are the P4b residual — bug347/348, issue29504/4562/27201 stay xfailed with that reason); runtime: add memprofile attribution support #1905 → runtime,cl: add Go-compatible sampled memory profiling #2027; ssa: panic with runtime type assertion errors #1892 → ssa,runtime: gc-conformant type-assertion panics and go:nointerface support #2031 (its ~10k-line diff was an early funcinfo draft; the surviving value — go:nointerface, scope-disambiguated assertion messages, MatchesClosure/reflect.Method fixes — re-expressed on runtime,cl: Go-style panic tracebacks and exact log/slog/testing locations, with gc-verified acceptance suite #2023, retiring mdempsky/15 + issue16130 + issue29735 xfails; mdempsky/16 residual: message lacks source interface type and prints command-line-arguments — same class as funcinfo main-name normalization).
• Semantics PRs re-expressed and closed: runtime: tighten recover to direct deferred calls #1918 → [Based on #2090] ssa,runtime: tighten recover to direct deferred calls #2033 (Defer-node recover ownership; retires issue4066/73916/73916b xfails), runtime: preserve recover error values #1882 → [Based on #2033] ssa,runtime: return runtime.Error values for assertions and faults #2034 (real runtime.Error values for asserts + SIGBUS; retires recover2/4, zerodivide, issue73917/73920; recover1.go = recursive-panic sub-call masking residual on the Defer-node model). Fix stack object finalizer liveness #1906 → cl,ssa,runtime: stack-object finalizer liveness #2036 (stack-object finalizer liveness on runtime,cl: Go-style panic tracebacks and exact log/slog/testing locations, with gc-verified acceptance suite #2023; retires deferfin/stackobj/stackobj3 xfails). Correction to the earlier survey: all three carried the pre-runtime: add Go-style funcinfo find index #2012 funcinfo draft (~11k lines each) — none rebase cleanly; extraction was the right path.
• Benchmarks: benchmark/runtime_funcinfo (in-tree since runtime: link-phase ftab/findfunctab generation (Stage 2, P1–P3) #2016) covers hot/deep/multipkg/cold/stdlib/plain plus -scales/-depths/-bigsizes; codecov ignores benchmark/chore/cltest; coverage uploads from both OSes (single-platform uploads mark the other OS's branches dead).

Acceptance criteria

• DCE-safe metadata — holding (both linkers, incl. LTO).
• API coverage by tests — holding (cl, test/go, LLDB 194/194, ssa, internal/build, both platforms + amd64).
• Hot lookups ≤ Go — exceeded (2–10× faster across hot/deep/statement-level).
• First-use ms → µs — met (first-in-process ≈ Go ±; per-fresh-pc gap is P4a).
• Size ≤ Go — met on mac all scenarios & linux typical; extreme function counts are P4d.
• User-visible conformance (agreed bar: format + user-code file:line == gc; runtime/patched-stdlib/startup frames may differ) — met for panic tracebacks, log/slog/testing locations, Caller/Callers/FuncForPC scenarios (runtime,cl: Go-style panic tracebacks and exact log/slog/testing locations, with gc-verified acceptance suite #2023 acceptance suite, gc-verified); mid-traceback plain-function lines are the documented P4b residual.

[cpunion]

Follow-up ledger addition: #2049 — scope the runtime-caller tracking memoization (runtimeCallerBaseCache/runtimeCallerExtendedCache, introduced with the Stage-5 uniform tracking) to the compilation instead of package globals. #2048 clears them per build.Do to fix the in-process memory accumulation (cl.test 2.0GB→0.5GB, suite co-resident anon 6.2GB→2.9GB); per-compile scoping removes the global state entirely and makes concurrent in-process compiles safe. Not urgent: single-compile-per-process is the only pattern today.