Repository navigation
gc(measurement): poll-time moving buys ~nothing the 16 MB nursery cap does not already buy — delete the loop back-edge poll, do not build pin-based poll moving #7056
Description
Measurement run to settle one decision: under the plan to replace shadow-stack precise roots with a conservative stack scan plus per-object pinning, the shadow stack's only remaining unique job is enabling the moving minor at loop back-edge polls. If poll-time moving buys ~nothing, that arm gets deleted outright; if it buys something real, pin-based poll moving has to be built first.
Bottom line
Poll-time moving buys ~nothing. Delete the arm; do not build pin-based poll moving.
Every gram of the footprint benefit attributed to the loop back-edge poll is actually the 16 MB nursery cap — a pacing number that merely shares an enable gate with the poll. Two independent isolations show it:
- Keep the poll, raise the cap to 128 MB: peak RSS becomes identical to the kill switch on 7 of 7 workloads — while the poll is still firing and still relocating a million objects.
- Drop the poll, keep the 16 MB cap: peak RSS is better than shipped (231.9 MB vs 272.2 MB) at the same wall time, copying more objects.
Meanwhile the poll's codegen half costs up to 3.7× on tight loops, 19 of 24 suite benchmarks never collect at all, and on the server-shaped workload the shadow stack contributed 3 rewritten slots across 63 moving minors that relocated 10.6 million objects.
Recommendation: delete the loop back-edge poll, decouple the 16 MB cap from gc_moving_loop_polls_enabled() and keep it. See §9 for what this does not settle — in particular it is not a verdict on conservative scanning, which §6–§7 measure separately and which #6981 has already shown is load-bearing for correctness.
Separately and more urgently: this run turned up a deterministic wrong answer produced by the shipped default configuration on an ordinary server-shaped program. It is not a poll bug — the discriminator is whether a copying minor runs at all. Filed separately as #7055; see §10.
Provenance
| Host | perry-macos — Mac mini, 8-core M1, 8 GB, macOS 26.5.1, idle |
| Load average | 1.5–2.8 throughout, recorded per run inside every result JSON (loadavg_start/loadavg_end) |
| Commit | e279b2d54, detached worktree, isolated CARGO_TARGET_DIR |
| Build | cargo build --release -p perry -p perry-runtime-static -p perry-stdlib-static |
perry |
aeac6a6e562cf6b13e585b7e49e53c136c97d9dc1a8493ef81ec10717ea2d08a |
libperry_runtime.a |
58a83af57503a84e1163e041f64aa355476b3354c16e7acb22b3a362f95a39cf |
libperry_stdlib.a |
645fb9c483d46bb5fe4846b401f42dbcc2bc846b9c5763e3cf3ae6cdccf9d282 |
| Oracle | ~/node-v26.5.0-darwin-arm64/bin/node, asserted v26.5.0 |
PERRY_NO_AUTO_OPTIMIZE |
not set — production path, not the deterministic one |
Instrumentation is benchmarks/gc_ratchet/gc_ratchet.py's run_once (peak RSS from this child's os.wait4 ru_maxrss, not RUSAGE_CHILDREN) plus the per-cycle JSON from PERRY_GC_TRACE=1. Arms are interleaved — repeat i runs every arm before repeat i+1 — so drift in machine state perturbs all arms equally. Run-to-run spread is reported next to every delta; nothing below is claimed as a difference unless it exceeds the spread, and most of it exceeds the spread by 50–300×.
The stale-archive check, adapted
The usual failure mode — linking a stale .a so both arms behave identically — cannot occur here, because both arms link the same archives. PERRY_GC_MOVING_LOOP_POLLS changes the compiled TypeScript, not the runtime. The correct analogue is that the compiled binaries must differ, so I checked something stronger than a hash: I counted call void @js_gc_loop_safepoint() in the emitted LLVM IR (--trace llvm) for every workload under every arm. Every workload: N>0 with polls on, exactly 0 with polls off.
That check earned its place. 05_fibonacci reports binaries_differ=true with poll_sites=0/0 — differing bytes, no behavioural difference. A hash comparison alone would have called that arm "changed".
One run discarded
My first driver compiled the off arm correctly but ran it without the runtime env, producing exactly the incoherent mixed mode gc_moving_loop_polls_enabled's doc comment warns about: codegen emits no poll, the runtime still defers collections to a poll that never drains. It measured +14.3% wall / +50.6% peak RSS vs shipped on w1_srv_pump — the opposite sign from the real kill switch (−71.6% / +54.9%). Discarded; every number below is from the corrected driver. Recording it because mixed-mode being genuinely bad is a real datum, and because it is a good example of a plausible-looking A/B returning a confidently wrong answer.
What flipping the flag actually changes
Read by two processes; moves four things at once, which is why a single on/off flip cannot attribute anything.
- codegen (
stmt/loops.rs:5227) — whetherjs_gc_loop_safepoint()exists in the binary at all. Part of the.ocache key, so no stale-object hazard. - runtime, routing (
policy.rs:1294) — alloc-point triggers defer to a poll instead of collecting inline non-moving. - runtime, pacing (
policy.rs:80-102) — the nursery is capped at 16 MB; with the flag off it reverts to the legacy adaptive trigger with a 128 MB ceiling. - runtime, eligibility (
copying.rs:26-51) —CopyingPointerSet::neweagerly builds the malloc registry; without it the always-on event-loop-safepoint moving minor falls back to non-moving whenever malloc-tracked objects exist. Workload-wide, nothing to do with loops.
Arms:
| arm | polls (codegen) | polls (runtime) | nursery cap |
|---|---|---|---|
on |
on | on | 16 MB — shipped |
off |
off | off | legacy 128 MB — full kill switch |
off_rt |
on | off | legacy 128 MB — codegen frozen |
on_cap128 |
on | on | 128 MB — isolates pacing |
off_scav |
on | off | 16 MB via PERRY_GC_SCAVENGE=1 — isolates the cap |
Also worth recording, because the doc comments disagree with the code: both gc_moving_loop_polls_enabled (policy.rs:369) and moving_safepoint_polls_enabled (loops.rs:5227) open with a paragraph saying the poll is opt-in/default-off, then a second paragraph saying DEFAULT ON. The code is unambiguously default-on. The stale first paragraphs are worth deleting before they mislead the campaign.
1. Primary A/B — benchmark suite (24 --full benchmarks, 5 repeats, interleaved)
Median Δwall, off vs on: −20.6% (range −76.5% … +1.1%). Median Δpeak RSS: 0.0% (range −7.7% … +160.6%). Spread on the affected benchmarks is 0.2–1.5%.
| benchmark | polls on/off | on ms |
off ms |
Δ | off_rt ms |
spread |
|---|---|---|---|---|---|---|
bench_buffer_readwrite |
12/0 | 416.7 | 98.0 | −76.5% | 381.6 | 1.22% |
13_factorial |
2/0 | 400.8 | 97.7 | −75.6% | 368.4 | 1.13% |
02_loop_overhead |
2/0 | 396.5 | 97.6 | −75.4% | 363.7 | 0.88% |
14_closure |
2/0 | 201.1 | 50.6 | −74.8% | 184.1 | 0.62% |
06_math_intensive |
2/0 | 200.2 | 52.9 | −73.6% | 183.6 | 0.62% |
04_array_read |
3/0 | 97.6 | 42.6 | −56.4% | 93.8 | 0.89% |
10_nested_loops |
6/0 | 102.7 | 51.5 | −49.9% | 97.0 | 0.33% |
15_mandelbrot |
6/0 | 39.6 | 24.9 | −37.2% | 36.6 | 1.09% |
bench_json_roundtrip |
5/0 | 330.3 | 261.8 | −20.7% | 265.5 | 0.43% |
bench_gc_pressure |
5/0 | 37.9 | 35.5 | −6.4% | 37.8 | 1.35% |
bench_string_heavy |
5/0 | 66.9 | 67.7 | +1.1% | 67.7 | 0.72% |
Correctness: every arm exits 0 and every checksum is byte-identical to Node 26.5.0. (The suite prints its own timing line, so a raw stdout diff is red by construction — the checksums are what was compared.)
19 of the 24 benchmarks never collect at all (minor_cycles = 0 in every arm), yet 22 of 24 carry poll sites. For those 19 the poll is overhead that can never do any work.
2. Which half costs: codegen or runtime
off_rt runs the polls-on binary with the runtime flag off — call sites present, body returns immediately.
02_loop_overhead:on396.5 →off_rt363.7 →off97.6 ms. The poll body costs 8%; the mere existence of the call costs 3.7×. That is the vectorization defeat the doc comment predicts, measured.bench_json_roundtrip:on330.3 →off_rt265.5 →off261.8 ms. The reverse — runtime half ~20%, call sites ~1%.
Why so many non-allocating loops get a poll: loop_purity::loop_may_allocate's whitelist accepts Compare{Eq,Ne} but none of the relational operators, accepts LocalSet but not GlobalSet, and no arithmetic Binary at all. for (let i = 0; i < N; i++) { sum = sum + 1 } therefore fails purity on its loop condition, and the "only emit the poll for allocating loops" mitigation misfires on 22 of 24 suite benchmarks. Cheap and separable win regardless of what happens to the poll.
3. Server-shaped workloads
Nothing in the repo is simultaneously server-shaped and sync-stretch-heavy: the batch benchmarks have long synchronous allocation stretches but no event loop; honest_bench's Fastify workload has an event loop but trivial handlers (and needs oha, not installed). So I wrote three, each diffed byte-for-byte against Node 26.5.0:
w1_srv_pump— event-loop-driven request pump, 600 requests; each handler builds 1,500 rows + tags + JSON in one long synchronous stretch, then yields viasetImmediate. Ordinaryforloops.w2_srv_pump_while— identical, every hot loop awhile(generic lowering path).w3_mono_sync— one long synchronous allocating stretch that never reaches the event loop.
| workload | polls | on |
off |
Δwall | off_rt |
peak RSS on → off | spread |
|---|---|---|---|---|---|---|---|
w1_srv_pump |
7/0 | 6746 ms | 1918 ms | −71.6% | 1927 ms | 271.9 → 421.1 MB (+54.9%) | 0.89% |
w2_srv_pump_while |
5/0 | 6278 ms | 2007 ms | −68.0% | 2021 ms | 240.5 → 426.6 MB (+77.4%) | 1.02% |
w3_mono_sync |
10/0 | 104.0 ms | 76.5 ms | −26.5% | 84.7 ms | 27.8 → 66.0 MB (+136.9%) | 1.29% |
off ≈ off_rt here, so on server-shaped work the cost is entirely the runtime half. Read naively this says the shipped configuration is 3.2–3.5× slower for 1.5–2.4× less peak RSS, and the trade is at least arguable. §4 is why it is not the poll's trade to make.
On w1_srv_pump the shipped arm spends 5.97 s of its 6.76 s wall inside GC pauses (88%), across 63 minors that copied 10.6 M objects.
4. The decisive isolation: it is the cap, not the poll
Same binary; only the runtime env differs. on_cap128 keeps the poll fully enabled and only raises the nursery cap.
| workload | on (16 MB) |
on_cap128 (128 MB) |
off_rt |
|---|---|---|---|
w1_srv_pump |
6756 ms / 272.2 MB | 2678 ms / 422.1 MB | 1935 ms / 421.1 MB |
w2_srv_pump_while |
6303 ms / 242.0 MB | 3242 ms / 434.4 MB | 2017 ms / 426.6 MB |
w3_mono_sync |
104.2 ms / 27.8 MB | 84.8 ms / 66.0 MB | 76.5 ms / 66.0 MB |
bench_json_roundtrip |
330.8 ms / 94.5 MB | 265.4 ms / 100.4 MB | 265.2 ms / 100.4 MB |
bench_string_heavy |
67.0 ms / 22.4 MB | 67.5 ms / 58.4 MB | 67.5 ms / 58.4 MB |
bench_gc_pressure |
38.0 ms / 21.7 MB | 37.7 ms / 24.2 MB | 37.9 ms / 24.2 MB |
02_loop_overhead |
403.6 ms / 4.0 MB | 402.9 ms / 4.0 MB | 369.4 ms / 4.0 MB |
on_cap128 peak RSS equals off_rt peak RSS on all seven, to within 0.2–1.8%, inside spread. And not because the poll stopped working: on w1_srv_pump, on_cap128 still ran 5 moving minors and copied 1,040,734 objects while off_rt copied 0. A million relocated objects bought 1 MB out of 422 MB.
The mirror-image arm closes it. off_scav keeps the 16 MB cap with the poll off (reached via PERRY_GC_SCAVENGE=1, the other gate on effective_next_arena_trigger):
| workload | on (poll + 16 MB) |
off_scav (no poll + 16 MB) |
|---|---|---|
w1_srv_pump |
6764 ms / 272.2 MB / 10,640,721 copied | 6718 ms / 231.9 MB / 11,066,933 copied |
w3_mono_sync |
104.4 ms / 27.8 MB / 16,365 copied | 104.3 ms / 27.7 MB / 16,390 copied |
bench_string_heavy |
67.0 ms / 22.4 MB | 68.8 ms / 22.6 MB |
bench_json_roundtrip |
331.2 ms / 94.5 MB | 372.3 ms / 94.8 MB |
Without any loop poll, the cap alone delivers equal or better footprint at equal or near-equal throughput, while evacuating more. (Caveat: PERRY_GC_SCAVENGE is documented measurement-only and also switches the alloc-point minor to precise roots, so this is not a clean single-variable move. It shows a poll-free configuration reaching the footprint; the clean single-variable evidence is on_cap128 above.)
Decomposing the shipped-vs-kill-switch gap on w1_srv_pump (6756 → 1935 ms):
- nursery cap 16 MB → 128 MB: 4078 ms, 84% of the gap
- poll / deferral / eager-registry: 743 ms, 16% of the gap
- of the 149 MB peak-RSS difference, the poll accounts for 1 MB
The mechanism is unremarkable once stated: peak RSS is set by how large the nursery may grow before a collection triggers — that is the cap. Whether the collection then relocates survivors or sweeps them in place does not move the high-water mark.
5. Does footprint stay bounded? (the property that matters for a server)
Same binary, request count swept. A terminating benchmark cannot distinguish "cheap because efficient" from "cheap because it deferred the work past process exit", so the question is the shape.
Peak RSS, MB — the two 16 MB-cap arms are grouped, then the two 128 MB-cap arms:
| requests | on (poll, 16 MB) |
off_scav (no poll, 16 MB) |
on_cap128 (poll, 128 MB) |
off_rt (no poll, 128 MB) |
|---|---|---|---|---|
| 150 | 162.8 | 168.7 | 174.8 | 174.8 |
| 300 | 236.9 | 219.0 | 309.1 | 309.1 |
| 600 | 271.9 | 267.3 | 384.6 | 422.0 |
| 1200 | 277.0 | 307.5 | 620.2 | 648.6 |
The curves group by cap, not by poll. The 16 MB arms plateau together (1.7–1.8× over an 8× request increase); the 128 MB arms grow together (3.5–3.7×) and would not survive a process that never exits. The poll is on in one member of each pair and off in the other, and it makes no difference to which curve the arm lands on.
(The 1200-request row is the noisiest point in this report — wall-time spread 7.9% on on and 13.6% on on_cap128, against 0.2–1.9% everywhere else. Read it for the shape, not the digits; the shape is a 2.2× separation between the cap groups, far outside that noise.)
So the correct reading of §3 is not "the poll bounds footprint" but "the 16 MB cap bounds footprint, costs the throughput, and is gated behind a flag named for polls".
6. Per-minor conservative-stack-scan cost
One binary, only the runtime env differs, stack depth from argv. phase_us["root_marking"] is shared by four subphases so it cannot isolate the scan alone — but in the precise arm that accumulator reads 0 µs on every automatic cycle at every depth, so differencing against it isolates the scan exactly.
| live stack frames | root_marking µs (median/cycle) |
conservative roots/cycle | conservative_pinned_bytes (max/cycle) |
|---|---|---|---|
| 0 | 551.5 | 11.2 | 3,104 |
| 250 | 608.0 | 205.5 | 11,136 |
| 1,000 | 834.5 | 768.5 | 35,128 |
| 2,500 | 1,275.5 | 1,893.8 | 95,168 |
Clean linear fits:
- scan cost ≈ 543 µs + 0.29 µs per live stack frame
- conservative roots ≈ 15 + 0.75 per frame
- bytes pinned ≈ 1.5 KB + 37 B per frame
For scale, the shipped arm's entire median minor pause is 726–828 µs. A conservative scan adds roughly one whole current minor pause of pure root work before collecting anything, ~1.5× at a server-ish 2,500 frames.
The direct cost is not the important number. The consequential one is: with the scan on, copying_nursery.fallback_reason is conservative_stack and copied_objects = 0 on every cycle of every probe. Today a conservative scan does not degrade the copying minor — it disables it. Median pause 828 µs → 14,745 µs (17.8×); total GC pause 22.3 ms → 58.6 ms on the same workload. On w1_srv_pump, forcing the scan takes wall 6752 → 11,805 ms and peak RSS 272.5 → 413.6 MB.
7. False retention, and what it means for pinning
Reproduced on an independent build; 8 gc_ratchet probes, heapUsed after an explicit gc():
| probe | precise | conservative | ratio |
|---|---|---|---|
01_nursery_churn |
0.77 MB | 3.57 MB | 4.6× |
02_survivor_promotion |
4.79 MB | 24.07 MB | 5.0× |
03_cross_gen_writes |
0.67 MB | 6.19 MB | 9.2× |
04_dead_after_deep_stack |
0.95 MB | 8.76 MB | 9.2× |
05_closure_capture |
0.99 MB | 8.00 MB | 8.1× |
06_string_retention |
0.71 MB | 4.53 MB | 6.4× |
07_array_grow_evacuate |
1.73 MB | 94.76 MB | 54.8× |
08_map_set_sidetables |
0.44 MB | 6.44 MB | 14.6× |
+360% to +5378% — the recorded 364%–5371% band, reproduced independently on a different build. Peak RSS rises 44%–274% alongside.
The finer characterisation asked for is the gap between what gets pinned and what gets retained, and the two are orders of magnitude apart:
- pinned: 328 B – 419,624 B per cycle across the 8 probes; ≤3.7% of the retained heap, usually far less. On
w1_srv_pump, 13,280 B against a 413 MB heap. - retained: 4.6×–54.8× the precise baseline. At depth 2,500, 1,894 conservative roots pin 95 KB directly but retain ~3.1 MB extra.
This cuts against the intuition the question is usually asked with. Pinning does not reduce retention. A conservative root marks its object live whether or not you also pin it; pinning only preserves the movability of everything else. So the retention blow-up is the unavoidable price of conservative rooting, pinned or not — and what the pinned-bytes figures actually say is the good news: 96–99% of the heap would remain movable.
That mechanism already exists and works, for the old generation. The cons_evac arm shows the old-gen evacuator pinning 7,537 objects (95 KB) under a conservative scan and still relocating 435 KB in the same cycle. It is only the nursery copying minor that bails outright on conservative_stack. Pin-based moving is therefore not novel machinery — it is machinery that exists one generation over.
8. What the shadow stack is actually carrying
From root_sources in the per-cycle trace:
w1_srv_pump, shipped arm, 68 cycles / 63 moving minors that relocated 10,640,721 objects:compiled_shadowcontributed 199 slots scanned, 79 pointer roots, 3 rewritten slots — for the entire run.runtime_mutable_scannerscontributed 1,451,297 slots and 354,421 pointer roots over the same run. The shadow stack is 0.02% of pointer roots.w3_mono_sync: 3–4 shadow slots per cycle.scan_costat depth 2,500: 105,028 slots, 35,000 roots, 10,000 rewritten — about 31% of roots.
So the shadow stack's contribution is essentially a function of JS stack depth with live locals: negligible on server-shaped work, substantial under deep recursion. It is load-bearing precisely where a conservative scan is also most expensive (§6), which is the real tension in the campaign and is not resolved by this measurement.
9. Recommendation
Delete the loop back-edge poll arm. Do not build pin-based poll moving.
- Held at a constant cap, the poll's peak-RSS contribution is 1 MB out of 422 MB while it relocates a million objects, and it costs 0–38% of throughput (§4).
- A poll-free configuration with the same cap gets better footprint at equal throughput while evacuating more (§4), and tracks the bounded curve as requests scale (§5).
- The poll's codegen half costs up to 3.7× on tight loops, and 19 of 24 suite benchmarks never collect at all (§1–§2).
- The shadow stack's unique remaining job is worth 3 rewritten slots across 63 moving minors on the workload it exists to serve (§8).
Concretely: decouple the 16 MB cap from gc_moving_loop_polls_enabled() in effective_next_arena_trigger and keep the cap — it is what buys the footprint and it is what costs the throughput; that trade should be made explicitly and tunably, not as a side effect of a flag named for polls. If the poll survives for a reason this measurement does not cover, fix the loop_may_allocate whitelist gaps in §2 first — they are most of the codegen cost and independent of everything else here.
10. Incidental, and more urgent than the above
While building the request-count sweep I hit a deterministic wrong answer produced by the shipped default configuration. Same binary, same input, only the GC runtime env differs:
| arm | minors | copied objects | checksum @150 | checksum @600 |
|---|---|---|---|---|
on (shipped) |
8 | 915,961 | -90240474 ❌ |
1986912948 ❌ |
on_cap128 |
0 / 3 | 0 | -341887099 ✅ |
1735266323 ✅ |
off_rt |
0 / 1 | 0 | -341887099 ✅ |
1735266323 ✅ |
off_scav |
8 | 1,045,097 | -90240474 ❌ |
1986912948 ❌ |
Node 26.5.0 says -341887099 / 1735266323. Deterministic across repeats in every arm.
The discriminator is exactly copied_objects > 0 — a relocating minor. Not the poll, not the cap, not minor_cycles: at 600 requests on_cap128 runs 3 minors and gets the right answer because none of them copied anything. The error is a constant 251,646,625 at 150, 300, 600 and 1200 requests — one corruption of fixed magnitude, occurring once, early, independent of workload length.
This is the #6951/#6972/#6981/#6982 defect class — a reference held across a relocating collection — reproducing in the shipped default, with native_stack_fallback.decision = skip_disabled on 62 of 63 cycles, i.e. with no conservative scan masking it. #6981 concluded the forced scan at gc_check_trigger was what kept this class out of production; on the deferred/moving path measured here, that mask is not present.
It is also a concrete instance of exactly what #6993 says the per-PR gate cannot catch — and it needs no special arm at all, just the default configuration. Filed separately with a reproducer: #7055.
11. What this does not settle, and what I could not measure
- Not a verdict on conservative scanning. §6–§7 price it at 543 µs + 0.29 µs/frame and as the thing that disables the copying minor outright (17.8× pause). Those costs stand whatever happens to the poll, and gc: a relocating minor with PRECISE roots breaks 14 of 20 representation-corpus files (5 crashes, 9 mismatches) — the conservative stack scan is load-bearing for correctness #6981 has already shown the scan is currently load-bearing for correctness. Deleting the shadow stack still has to answer that.
off_scavis not a clean single-variable arm. It reaches the 16 MB cap via a flag that also switches alloc-point roots to precise. The clean evidence for "it's the cap" ison_cap128;off_scavcorroborates by construction from the other side.- No real socket server.
node:httpneedslibperry_ext_http.a, which a-p perry -p perry-runtime-static -p perry-stdlib-staticbuild does not produce — the link fails on_js_node_http_create_server_with_options/_js_node_http_server_listen. My workloads are event-loop-driven request pumps with heavy synchronous handlers (the shape that matters here) but do not exercise the accept loop, sockets, or the tokio paths. - No isolated timing for the conservative scan. Four subphases share the
root_markingaccumulator (cycle.rs:544-549). Differencing against a zero-valued precise arm works today but would not survive a change that makes precise root marking non-trivial. - No exact false-retention counter.
conservative_root_count/conservative_pinnedare structurally over-inclusive — the conservative subphase runs before the precise ones every cycle, so they count objects precise roots would have reached anyway. Upper bounds. - Pin-based nursery moving does not exist, so its cost could not be measured directly; the old-gen evacuator's pinning is the proxy §7 uses.
- The
setImmediate-driven workloads are not bit-deterministic in their GC scheduling, unlike thegc_ratchetprobes.w5at 600 requests measured 248.6 MB peak RSS in one session and 271.9 MB in another (58 vs 56 minors, 11.64 M vs 11.66 M copied). Within-session spread is 0.2–1.9%; between-session peak-RSS drift on these workloads is up to ~9%. That is far below the effects being claimed (55–137%), but it means single-digit peak-RSS differences on the async workloads should not be read as signal. Theon_cap128-vs-off_rtequality in §4 rests on 0.0% differences on five workloads and 0.24%/1.83% on the other two, all measured within one session. - Single-threaded only (
spawn/parallelMaparenas untested). Single host, single architecture (arm64 macOS).
Activity
proggeramlug commentedon Jul 30, 2026
Shadow stack vs stack maps: measuring the ceiling
Sequel to this issue's measurement, answering a different question with the same
hosts and method. Stack maps have essentially zero mutator cost, so the most
they could ever buy is exactly what the shadow stack costs today. That number
bounds the decision without anyone building a prototype.
Gate used:
PERRY_SHADOW_STACK=0(perry-codegen/src/codegen/helpers.rs).Bottom line: the ceiling is large and shape-dependent — 0% on numeric loops,
+5.7% on allocation-shaped code, +71.5% retired instructions on the most
application-like workload, +78% (Pi) / +110% (M1) on deep pointer recursion.
A prototype is justified. But two cheaper fixes should land and be re-measured
first, because a large share of that ceiling is implementation overhead in the
shadow-stack runtime ops, not something inherent to shadow stacks.
Two secondary results that matter more than the headline:
On allocation-shaped workloads that actually collect,
PERRY_SHADOW_STACK=0either crashes or exits 0 with a deterministic wrong answer — the same
class as gc: the relocating minor produces a deterministic wrong answer in the SHIPPED default on an ordinary event-loop workload #7055/gc: a relocating minor with PRECISE roots breaks 14 of 20 representation-corpus files (5 crashes, 9 mismatches) — the conservative stack scan is load-bearing for correctness #6981, reproduced bit-for-bit on both aarch64 hosts. Those
workloads are therefore unmeasurable by this method and are excluded.
loop_may_allocatewhitelist bug contributes 0% of the ceiling — itis gated by a different env var and its callsite count is identical in both
arms. It is separately worth 34× retired instructions / 9.1× wall on a
pure counting loop and +8.05% instructions on app-shaped code, which
independently corroborates this issue's "delete the loop back-edge poll"
recommendation.
Provenance
lego.local— Raspberry Pi 5 Model B Rev 1.1, Cortex-A76 ×4, 8 GB, Debian 13, kernel6.12.47+rpt-rpi-2712perry-macos.local— Mac mini, 8-core M1, 8 GB, macOS 26.5.1c0d98624a(clean tree)c0d98624a(git archive, identical tree)cargo build --release -j3 -p perry -p perry-runtime-static -p perry-stdlib-static-j3perrysha256ba2e7fd0f4654414413b…03abf59ae3f5227fcdf7…libperry_runtime.a3e7c11e863509aee8a3e…4605e6648462d42cc4cd…libperry_stdlib.a7f473ea12fb7c4d41633…1de17b0d048bb5ccc7f6…/usr/local/bin/node26)~/node-v26.5.0-darwin-arm64)performanceon all 4 cores, runs pinnedtaskset -c 3, load ≤0.5PERRY_NO_AUTO_OPTIMIZE=1on both armsPi thermal record. Rebooted before the session to clear the sticky throttle
bits left by the provisioning build. Every run samples
vcgencmd measure_clock arm/get_throttled/measure_tempimmediately before and after; any runwhose clock left 2.400 GHz ±1% or that set any throttle bit is discarded.
Across the 288 measured runs: clock 2.400017–2.400037 GHz, throttle word
0x0throughout, temperature 60.4–67.5 °C, zero runs discarded for thermalreasons.
scaling_cur_freqwas not used — it lies on this board.Cache-key check (the null-result trap)
Confirmed rather than assumed, per host, per workload: the two arms hash to
different object-cache entries, so they cannot silently share a cached
.o.…and so on for all 8 workloads on both hosts (
PERRY_SHADOW_STACKis hashed atcompile/object_cache.rs:795and listed inBUILD_CACHE_ENV_VARS).Two workloads emit no shadow-stack code at all, so their two arms produce
byte-identical binaries (
sha1matches) — these act as the harness's ownnull control and calibrate its measurement floor. See
w0_noop/w5_puritybelow.
Workloads
The shadow stack's cost is extra opaque calls, register pressure, and blocked
optimisation, so the set is deliberately allocation-shaped, call-heavy and
loop-heavy. A trivial program measures nothing here.
w0_noopconsole.logonly — startup floorw5_purityfor(i=0;i<400M;i++) sum=sum+1— no user functionw2s_allocring_smallw6_recordsweight/pick/scanhelpersw1_calldepthw3_treewalkw2_allocringw2sbut 3 M iterationsw4_strchurnSoundness discipline. Tier A workloads run zero collections in both arms,
verified per workload per arm per host with
PERRY_GC_DIAG=1(count of[gc-step]lines = 0). With no collection, removing precise roots cannot changethe answer — the OFF arm is sound by construction. Additionally, every one of
the 288 measured runs had its stdout compared against the Node 26.5.0 oracle;
0 mismatches across all Tier-A runs on both hosts. Both guards, not one.
★ Tier B is unmeasurable: the OFF arm is unsound, and one case is silent
The moment a collection actually runs,
PERRY_SHADOW_STACK=0stops being"the same program, faster":
w4_strchurnis the dangerous one: exit 0, plausible-looking number, wrong,and deterministic — identical across repeated runs and bit-identical on both
the Pi and the M1 (
738577316on each). A live ring/string reference iscollected because nothing rooted it. This is exactly the #7055 / #6981 class.
24 of 168 rows in the first session were discarded on this basis (12 reps ×
2 workloads). No timing from those runs is reported. The consequence: this
method cannot measure the shadow-stack ceiling on GC-heavy workloads at all.
The only allocation-shaped datapoint below (
w2s) is capped at ~15 MB totalallocation, i.e. just under the first minor.
Static delta (load-independent)
Emitted callsites (strict
call … @name(in the traced LLVM IR, user moduleonly). Identical on both hosts:
js_gc_loop_safepointON vs OFFw0_noopw5_purityw2s,w2w6_recordsw1_calldepthw3_treewalkw4_strchurnThe OFF arm emits exactly zero shadow ops, and the loop-safepoint count is
identical in both arms — the first of three proofs that the whitelist bug is
outside this ceiling.
.textof the user object (the whole-binary.textis ~8.9 MB of sharedruntime and its ±0.16% delta is linker-layout noise — ignore it):
w0_noopw5_purityw6_recordsw2s_allocring_smallw1_calldepthw3_treewalkw4_strchurn★ Static size is a bad proxy for this.
w6_recordshas the largest dynamiccost in the set (+71.5% instructions) and one of the smallest static deltas
(+0.29% on the Pi). The shadow ops sit in hot loops; code size does not see that.
Report the dynamic number, not this one.
Dynamic ceiling — Raspberry Pi 5 (primary host)
12 interleaved reps per workload (ON, OFF, ON, OFF …), warmup discarded,
0 of 144 rows dropped, 0 oracle mismatches. Values are medians;
pairedis the median of per-rep ON-vs-OFF ratios.w0_noop(identical binaries)w5_purity(identical binaries)w2s_allocring_smallw1_calldepthw6_recordsw3_treewalkAbsolute anchors:
w6_records11.80 G → 20.23 G instructions, 1.42 s → 2.37 s.w3_treewalk11.00 G → 19.57 G instructions, 1.58 s → 2.69 s.(
LLC-loads/LLC-load-missesare the explicit LLC events. The genericcache-references/cache-missesalias to L1-dcache on this PMU and were notused.)
Variance, and what counts as a difference
w138.71 / 38.71,w377.92 / 77.93,w2s5.72 / 5.70,w5−0.00 / −0.00w367.8 / 69.9,w2s6.22 / 5.93,w50.00 / −0.01;w145.4 / 38.0 (session 1's wall was noisy — its instruction count was identical)Instructions retired is the reportable metric here; wall and cycles only
corroborate it. The measurement floor is set by
w0_noop, whose two arms arebyte-identical binaries yet read +0.70% instructions / +0.36% wall — that is
the harness's systematic bias (the ON arm always runs first in a pair).
Anything below ~0.7% instructions is not a difference.
w5_purity, alsobyte-identical, reads −0.00% instructions, which is the tightest available
validation that the pipeline is not inventing a delta.
Peak RSS (
/usr/bin/time, max of 5): flat.w0+0.00%,w2s−1.84%,w6+0.00%,w1+3.45%,w3−3.20%,w5+0.71%. No memory story.Architecture delta — Mac mini M1
Same commit, same workloads, wall + peak RSS only (no PMU access on macOS).
12 interleaved reps, 0 dropped, 0 oracle mismatches, cv < 1% — far tighter
than the Pi despite the noisier host.
w5_purity(identical binaries)w0_noop(identical binaries)w2s_allocring_smallw1_calldepthw6_recordsw3_treewalkw0_noop's +10.95% is on byte-identical binaries over a 5 ms process — a fixed≈0.5 ms ordering bias, i.e. the macOS harness floor. It is 0.04% of
w6'sruntime and can be ignored for everything else. An order-swapped control
(OFF first, then ON, 6 reps) reproduces the large deltas exactly —
w60.63 s → 1.11 s (+75%),w30.65 s → 1.37 s (+109%) — so they are not anordering artifact.
★ The delta goes the wrong way, and the counters say why
The expectation going in was that the Pi's 64 KB L1d / 512 KB L2 would expose a
cost the M1's 128 KB / 12 MB hides. The opposite happened: the M1 is worse
on the call-heavy shapes (
w3+110% vs +70%,w1+46% vs +38%) and equal on therest.
The Pi counters explain it: on every workload
LLC-loadsandLLC-load-missesmove ≤ +2% and ±3%, while instructions rise 5–78% and frontend stalls rise
150–250%. The shadow stack's cost is not a data-cache-footprint effect at all
— it is retired instruction count plus I-side pressure from non-inlinable calls
in hot loops. A cost of that shape does not care about D-cache size, so the
small-core proxy argument does not apply, and per-op call overhead dominates
instead. A plausible reading of the M1 being worse is macOS dynamic TLS
(
tlv_get_addr) versus Linux initial-exec TLS, since every shadow op takes oneTLS access — the runtime's own comment in
gc/roots/shadow_stack.rsalreadyrecords
tlv_get_addrat 20.9% of leaf samples in an earlier profile. I couldnot verify that on the Mac (no PMU access); it is a hypothesis consistent with
existing code comments, not a measurement.
How much of the ceiling is the known
loop_may_allocatewhitelist bug?Zero. Proven three ways:
js_gc_loop_safepointcallsite counts are identical in both arms forevery workload (table above).
PERRY_SHADOW_STACK=0does not remove the poll— the poll is gated by
PERRY_GC_MOVING_LOOP_POLLS(stmt/loops.rs:5227).w5_purity— whose only GC-related emission is that safepoint — reads−0.00% instructions ON vs OFF, on byte-identical binaries.
502 856 323. The shadow stack contributes nothing once the poll is gone.
So none of the ceiling above is attributable to the whitelist bug, and fixing it
does not shrink the case for stack maps. But the bug's own cost is enormous,
measured at shadow-ON with
PERRY_GC_MOVING_LOOP_POLLS=0:w5_purityw6_recordsw1_calldepthw2s_allocring_smallw3_treewalkThe root cause is unchanged and still unfixed:
expr_alloc_free(
perry-codegen/src/loop_purity.rs:115) whitelistsCompareonly forEq | Ne, and has noExpr::Binaryarm at all. Soi < Nandsum + 1both fail purity,
loop_may_allocatereturns true, and a provably non-allocatingloop gets a back-edge poll that costs ~42 instructions per iteration and defeats
vectorisation. This independently corroborates this issue's "delete the loop
back-edge poll" recommendation, and puts a number on it for app-shaped code: 8%.
Where the cost actually goes
w3_treewalktraverses a perfect depth-14 tree 320 times = 20 971 200walkactivations. The instruction delta is 8 569 162 213, i.e. ≈409 extra retired
instructions per activation. With ~6 emitted shadow ops per activation (push,
pop, and the slot sets for
t/left/right) that is ≈65–70 instructions pershadow op — an order-of-magnitude estimate from the aggregate, not a direct
per-op measurement.
That is very expensive for what these ops nominally do, and reading the runtime
explains it:
js_shadow_frame_push(gc/roots/shadow_stack.rs:59) does three separateVec::resizecalls per activation —stack,slot_ptrs,active— eachwith its own capacity check and zeroing, on top of two header pushes.
js_shadow_slot_set(:134) does a TLS access, two bounds checks, aVec<bool>write, and a call toruntime_write_barrier_root_nanboxonevery non-zero store, plus an indirect store through
slot_ptrs.in registers across it — which is what the +150–250% frontend-stall numbers are
showing.
A large share of the measured ceiling is therefore implementation overhead in
the shadow-stack runtime, not something intrinsic to the shadow-stack design.
Three parallel
Vecs could be oneVecof a packed struct (or the header couldcarry the slot count and the resize be hoisted); the write barrier on slot-set is
a separate question. None of that requires stack maps.
Recommendation
A stack-map prototype is justified — but it should not be the next thing built.
The ceiling is unambiguously large where it matters: +71.5% retired
instructions on the most application-like workload, reproducing at +72% wall on
a completely different microarchitecture, with 0.00% run-to-run variance on the
instruction count and a validated ~0.7% measurement floor. This is not a
rounding-error ceiling that closes the question cheaply.
But two cheaper interventions attack the same number and should be measured
first, because either could move it materially:
loop_purity.rswhitelist, or thisissue's delete-the-arm recommendation). Independent of the ceiling, worth
34× on numeric loops and +8% on app-shaped code. Cheapest win available.
Vec::resizecalls perframe push and a write-barrier call per slot set are not inherent costs.
At ~65–70 instructions per op, this is where the ceiling lives.
Re-measure with this exact harness after each. If a ceiling on the order of tens
of percent survives both, build the stack-map prototype; the counters say what it
would be buying (retired instructions and I-side pressure, not cache
footprint), which is also a useful design constraint on the prototype.
One caveat on the framing: the measured delta is an upper bound, not a
prediction. Real stack maps are not free — LLVM statepoints still force live
pointers to be spilled at safepoints. Stack maps would recover some fraction of
these numbers, never all of them.
What I could not measure
collection runs (crash, or a silent wrong answer). The ceiling for
allocation-heavy programs is therefore unknown; the one allocation datapoint
here (
w2s, +5.70%) is capped below the first minor GC. Anyone wanting thatnumber needs a different method — e.g. a build that keeps precise roots but
strips only the redundant stores, or a real stack-map prototype.
instruction/cache/stall breakdown exists only for the Pi.
runtime's own recorded profile, not measured here.
Reproduction
Workloads, harness (
run_ab.py,mac_ab.py,analyze.py) and raw per-run JSON(
results_run1.json,results_run2.json,results_mac.json— every row carriesits own clock, throttle word, temperature, stdout and keep/drop reason) are in
~/ssbenchon both hosts. No code change was made to the repo; this is ameasurement-only run and there is no PR.
proggeramlug commentedon Aug 3, 2026
Sequencing warning before anyone starts this — see
docs/src/internals/rfc-rooting-by-construction.md("Interaction with the RSS goal", landed in #7292).Two findings change how this should be approached:
The −65% RSS (320 MB → 111 MB) does not come from the copying minor. It comes from the 16 MB nursery cap, which merely happens to be gated on the same flag:
effective_next_arena_triggergates it ongc_scavenge_enabled() || gc_moving_loop_polls_enabled(). A no-poll arm reaches the same 108 MB. So there is a route to the memory win that does not require the risky moving path at all.Do not measure the cost of collecting more often until the minor path actually works. The "~20× wall-clock" figure attached to the cap was measured while gc-matrix: four of the six PR-gating arms run ZERO copying minors corpus-wide (default/verify_evac/cons_scan_off at copy-minor 0/49; header still claims 12/22) #7255's defect was live — in the default configuration
gc_check_triggerfalls through to a direct minor behindManualGcScanGuard::force_full_scan(), i.e. a conservative full scan. Those were not minor collections. Minor-GC cost should scale with survivors, not with collection count, so a 20× penalty is a symptom, not a curve to sit on.⇒ Fix the minor path first, then re-derive these numbers. Measuring now would bake the full-scan cost into the record and make an RSS/speed tradeoff look inherent when it is a defect. That conclusion is why this is sequenced last in the plan, not because the RSS goal is deprioritised.
perf(gc): nursery cap + scavenge on by default — peak RSS -69%
perf(gc): nursery cap + scavenge on by default — peak RSS -69% (#7377)
proggeramlug commentedon Aug 8, 2026
Re-derived under the statepoint default (plan item 9) — the root lowering is not an RSS lever
Every number in this issue was measured under the shadow-stack root lowering. Statepoints became the default in #7370, so the first question is whether that invalidates them. It does not. What has invalidated parts of this issue is everything else that shipped since — including this issue's own §9 recommendation.
Re-derived at
7bde3de24(v0.5.1376) on the same host this issue used —perry-macos, Mac mini, 8-core M1, 8 GB, macOS 26.5.1 — withPERRY_RUNTIME_DIRpinned,PERRY_NO_AUTO_OPTIMIZE=1, node oraclev26.5.1. 94.6–95.8% CPU idle, zerorustc, zerocargofor the whole session. 2x2 over root lowering x nursery cap, 12gc_ratchetprobes, 7 repeats, 3 interleaved rotations (rotation i runs every arm before i+1, the same discipline this issue used). All 72 probe-runs byte-identical to the oracle.1. Statepoint vs shadow stack, at the shipped cap
PERRY_RS4GC=0)gc()104 of 108 deterministic cells are bit-identical. All four that move are on
10_store_receiver_across_allocand are ≤0.31% — the shadow stack keeps 7 more objects live at that collection point than the statepoint map does (copied_objects8,056 → 8,063,heap_used_bytes+0.035%).Between-rotation spread: retention 0.000%, peak RSS max 0.621% (median 0.000%), wall max 2.779% (median 0.449%).
The arms were shown to differ before they were compared, per probe — this issue's own poll-site-count discipline, since differing bytes are not a differing configuration.
--trace llvmcensus across all 12:gc "statepoint-example"+addrspace(1)root allocas present on 12/12 under the default (168–514addrspace(1)sites per probe), 0 of both underPERRY_RS4GC=0, withjs_shadow_frame_enterrising to match (1 → 3–10).2. §9's recommendation shipped, so the arms in this issue no longer name anything
#7377 decoupled the 16 MB cap from
gc_moving_loop_polls_enabled()and made it unconditional; the poll has been default-off since #7161.on/off/off_rt/on_cap128/off_scavdo not describe any shipping configuration today.3. The cap is still the whole footprint lever — now on 12 probes rather than 8
At
PERRY_GC_SCAVENGE_NURSERY_MB=128, against the shipped cap:04_dead_after_deep_stack5.005x07_array_grow_evacuate6.350xIdentical to within 0.6% under both root lowerings, so this is a pacing result and not a rooting one — which is §4's conclusion, re-confirmed on the current default.
4. §7's false-retention table is gone, not shrunk
§7 recorded a 4.6x–54.8x conservative-vs-precise band on probes 01–08.
gc_ratchet.py classifyat7bde3de24reports excess 0.00% and spread 0 on all twelve probes, with nomanual_collectscan site on any of them. #7657 removed the cause (the forced conservative scan at explicitgc()), not the reading.5. §6's real finding survives intact
A conservative scan does not degrade the copying minor, it disables it. Forcing
PERRY_CONSERVATIVE_STACK_SCAN=fullunder statepoints:minor_cycles→ 0 on all twelve probes;copied + promoted→ 0 on all twelveSmaller than the +364%..+5371% recorded here, same sign, same mechanism. (
12_large_live_setretains less under the forced scan, 0.70x, because it stops running minors and its collections become fulls — worth knowing before someone reads a single row.)6. What could not be re-derived
§3–§5's largest numbers (272 MB → 421 MB on
w1_srv_pump/w2_srv_pump_while/w3_mono_sync) came from three server-shaped workloads this issue wrote and never landed. There is nothing in the tree to re-run, and a fresh reimplementation would not be comparable to them. The 12 probes are what replaced them, and #7666 adds a 13th for the large-Eden cadence #7481 named.Suggested disposition
Close. The measurement stands, §9 shipped, and the parts that are stale are stale because the collector moved, not because the method was wrong.
docs/engine-plan.mditem 9 now carries this table; the harness and the re-pin are in #7666.test(gc-ratchet): pin the large-Eden copying-minor cadence (#7481) an…
test(gc-ratchet): pin the large-Eden copying-minor cadence (#7481) an…
proggeramlug commentedon Aug 9, 2026
Closed by #7666 (v0.5.1379) — plan item 9, re-derived under the statepoint default.
The root lowering is not an RSS lever. Pinned quiet mini, 12 probes × 7 repeats × 3 interleaved rotations, all 72 probe-runs byte-identical to the node 26.5.1 oracle:
PERRY_RS4GC=0)gc()104 of 108 deterministic cells bit-identical. The arms were proven to differ per probe before being compared —
statepoint-example+addrspace(1)roots on 12/12 by default, 0 of both underPERRY_RS4GC=0— which is what makes 1.002 mean "no effect" rather than "the same binary twice".Section by section:
classifynow reports excess 0.00% on all twelve.PERRY_CONSERVATIVE_STACK_SCAN=fulltakesminor_cyclesto 0 on all twelve, 1.98× retention, 1.46× peak RSS.fix(gc): an allocation-point collection must not move anything (#7682)
fix(gc): an allocation-point collection must not move anything (#7682) (
fix(gc): the scavenge nursery cap applies only when the minor can eva…
fix(gc): the scavenge nursery cap applies only when the minor can eva…
fix(gc): restore evacuation at precise safepoints — the pacing half of …