[Next.js/dylib] Forced-evacuation App Route arm: stale closure from a holder outside the GC heap

Splitting the one unresolved item out of #8082 so the production App Route fixture can land on its own merits. The fixture's normal arms pass end-to-end; its forced-evacuation arms do not, and this is that bug.

Symptom

With the Next 16.3.0 production App Route fixture running under forced evacuation and a seeded GC schedule:

PERRY_GC_FORCE_EVACUATE=1 PERRY_GC_HEAP_LIMIT=8 PERRY_GC_INCREMENTAL=0 \
PERRY_CONSERVATIVE_STACK_SCAN=off PERRY_GC_SCHEDULE_RATE=1 PERRY_GC_SCHEDULE_SEED=8036

the server serves requests and then fails with TypeError: value is not a function, at roughly copying minor #238.

What it is not (each ruled out by measurement)

Not the statepoint lowering. The same application compiled with PERRY_RS4GC=0 (shadow-stack roots) fails identically — 80 TypeErrors in each arm, same seed, providers and workload. So it is neither fix(gc): index stack maps from loaded provider apps #8081's loaded-image stack maps nor compiled-frame root coverage.
Not the remembered set. skip_remembering never fires (336/336 cycles report remembering_skipped=false, and in-place promotion never happens), and remembered-set coverage is stable — zero cycles lose dirty pages between collections.
Not a holder in the GC heap. PERRY_GC_PROTECT_FROMSPACE_HOLDERS=1 (added in fix(next): pass production App Route dylib gate #8082) sweeps the whole live heap at fault time for any word still decoding to the faulting address and finds none.
Not a slot the rewrite pass enumerates. PERRY_GC_VERIFY_EVACUATION=1 never panics, so every registered scanner rewrites correctly.

Taken together: the stale copy is held by something unregistered and outside the GC heap — a runtime side table, an FFI structure, or a frame slot.

Where the fault lands

Under PERRY_GC_PROTECT_FROMSPACE=1 PERRY_GC_PROTECT_FROMSPACE_DEPTH=800, the stale value is a 48-byte closure consumed by js_value_typeof inside Next's compiled adapter:

static get(e, t, r) { let n = Reflect.get(e, t, r); return "function" == typeof n ? n.bind(e) : n }

reached through js_proxy_get (the adapter is a Proxy get trap over the wrapped route-module object). js_reflect_get itself is fully rooted, so the pre-move copy lives in whatever supplies that property read.

Suggested starting point

scripts/gc_runtime_root_holders.py audits crates/perry-runtime/src and crates/perry-stdlib/src only — pointed at the ext crates it matches 0 declarations, because its patterns do not parse them. Yet every ext crate that invokes closures (perry-ext-http, -net, -ws, -zlib, -fastify) keeps user closures in handle side tables and registers its scanners by hand, with nothing checking that a scanner reaches every table in its own crate. Extending that audit is the obvious next move.

Reproducing in seconds

Build the providers per tests/release/packages/next-app-route/fixture.sh steps 3–5 into a kept build dir (PERRY_NEXT_ROUTE_KEEP_BUILD=1), then run the provider host directly with the env above plus one node verify.mjs. PERRY_GC_FROMSPACE_SCAN=1 mines holders at the retiring minor — but read its counts carefully: it reports dead old-gen residue and raw payload bytes as offenders (impossible obj_types like 47/95/144), which cost this hunt a retracted diagnosis. The protect fault is the trustworthy signal.

The fixture ships this arm behind PERRY_NEXT_ROUTE_FORCED_GC=1, off by default, so it fails loudly for whoever picks this up rather than sitting green.

test(next): make the forced-evacuation arm opt-in, tracked by #8163

merge origin/main into the App Route fixture branch

Retested on `53e8a21e3` (main, #8168 included): STILL REPRODUCES

#8168 (3cc318f69, "prune FUNCTION_CLASS_IDS keys whose closure died") landed fifteen hours after this issue was filed and names the identical seeded configuration, so it was worth checking whether it had already closed this. It has not.

The run

Full tests/release/packages/next-app-route/fixture.sh with PERRY_NEXT_ROUTE_FORCED_GC=1, PERRY_NEXT_ROUTE_COLD_STARTS=2 (index 0 normal, index 1 forced), on a clean origin/main worktree at 53e8a21e3, perry built --release from that tree.

  [1/7] … [5/7] compile untouched production route handler as app-only dylib
  next=16.3.0 mode=dylib providers=823d8b0244620a83fd73e7946b8297df6b9f36a66024cf4689c966193038ca3e
  [6/7] 2 cold processes (alternating normal / FORCED-evacuation), two 21-request verifier runs each
FAIL next-app-route — forced cold verifier 1 failed

The provider images are pristine main. Verified directly rather than assumed: the runtime image contains 2 occurrences of the gc-copy-minor diagnostic string and 0 of a marker string that exists only on a local working branch.

The forced arm ran, and its subject was live

Not "nothing threw" — the numbers, from perry-forced-1.log:

quantity	value
`[gc-copy-minor] ran` lines	243
Σ `copied_objects`	117,579
`PERRY_NEXT_APP_ROUTE_READY`	1 (server came up and served)
`PASS: 21 production App Route requests`	0
`TypeError: value is not a function`	2
`gc evacuation verification failed` panics	0

The normal arm (cold start 0) passed both verifier runs — PASS: 21 production App Route requests twice — so this is the forced-evacuation arm specifically, not a general regression.

It is the same bug

Every discriminator in the issue body still holds: the failure is TypeError: value is not a function, PERRY_GC_VERIFY_EVACUATION=1 never panics (so every registered scanner rewrites correctly), and the copying minor is demonstrably moving objects. #8168 removed one dead key from one rekeyed table; whatever holds the stale closure here is a different holder.

Related work

#8174 (the general rewrite_raw_addr exposure this shares a root cause shape with) is addressed by PR #8196, which refuses a forwarding walk out of, or into, something that is not a heap object, and adds a gate requiring every rekeyed side table to have a declared death story. That narrows the following of a stale key; it does not remove the holder here, and this issue stays open.

The issue's own suggested next move is still undone: scripts/gc_runtime_root_holders.py:109 reads

CRATES = ("crates/perry-runtime/src", "crates/perry-stdlib/src")

so perry-ext-http, -net, -ws, -zlib and -fastify — every ext crate that parks user closures in handle side tables and registers its scanners by hand — are unaudited.

Kept build artifacts (providers, app dylib, both logs) are at /private/tmp/perry-8174-8163/build.

Reproduces under DEFAULT GC — not only the forced arm

Measured 2026-08-16 on the bench mini (idle M1, no other load), clean main 3c95020f8, perry built --profile perry-dev from that tree, providers + app dylib built by the untouched tests/release/packages/next-app-route/fixture.sh (release layout, C host, PERRY_MODULE_JOBS=2), no GC knobs at all except PERRY_GC_DIAG=1 on two of the runs. Only deviation from the fixture: the armed perry-host.js from tests/fixtures/next-app-route/ was dropped in so the routeModule.handle bypass guard is live.

fixture.sh itself passes (10 cold × 2 verifiers, bypassed=0 in all 10 logs). It never sees the bug because it stops at 2 passes per process. Run 100 verifier passes against ONE warm process and it appears:

run	env	pass/100	failing iterations	host `TypeError: value is not a function`
batch100	—	98	3, 6	4
rerun1	—	100	—	0
rerun2	—	99	19	1
diag1	`PERRY_GC_DIAG=1`	97	6, 12, 19	4
diag2	`PERRY_GC_DIAG=1`	98	6, 12	3

492/500 batches (each = 20 concurrent GETs + 1 POST) correct; the 8 failing batches each had one request answer with an empty body (verify.mjs: Unexpected end of JSON input) after the host logged TypeError: value is not a function (+ Next E180 failed to pipe response). Never a wrong ID/header/cookie — the response is lost, not swapped. Bypass guard: 0 fires in 500 batches.

Every failure sits immediately after a [gc-copy-minor] ran … in_place=false line and nowhere else (diag1 host log, in order): minor #1 → E180; minor #2 → 2× TypeError + E180; minor #4 → TypeError; … 21 copying minors in the run, 4 TypeErrors. E180 alone also occurs in fully passing runs (3× in rerun1), so on its own it is benign; the empty body only comes with the TypeError. The periodicity (iterations 6/12/19) is just the nursery step schedule (step=134217728→67108864→33554432→16777216, then a minor every ~6 batches).

Same symptom, same site (Next's compiled Reflect.get/bind adapter, from the trail in the issue body), reached by the shipped default configuration at ~1 broken request per 700 in a warm process. Worth restating for prioritisation: this is production-visible, the forced arm only makes it deterministic.

Cheap reproducer (seconds once the fixture is built)

Kept artifacts on the mini: ~/perry-bench.noindex/build8082/ (providers, app dylib, host) and ~/mini-batch-8040.sh <tag> <N> [ENV=..] (starts one host, loops verify.mjs N times, reports pass/fail, per-iteration TypeError count, RSS). Logs in ~/perry-bench.noindex/logs8040/batch-*-{host,verify}.log.

Also seen, separate from the stale closure

Retention over the same 100 batches (2100 requests): [gc-step] post_in_use rises monotonically 5.4 MB → 39.7 MB across the run's 21 collections while process RSS goes 160 MB → 510–620 MB (5 runs, no plateau) — the JS heap accounts for ~35 MB of that, the rest is off-heap. Noting it here only because it comes out of the same run; it wants its own home (or belongs to whatever fix/async-state-rss-accumulation is tracking).

Independent validation of the candidate fix `3b01efba4` (fetch caches + ext-http once_listeners/EndTail rooting): forced arm FIXED, default-GC arm NOT — a holder remains

Ran on the bench mini 13:29–14:20 UTC today, worktree at 3b01efba4 (parent verified = 3c95020f8), armed perry-host.js, same artifacts pipeline as the earlier default-GC measurement, plus a same-hour main baseline so the comparison is same-host/same-hour.

Fixture (forced arm ON by default on this branch): PASS. PASS next-app-route (with forced-evacuation arm), 10/10 cold starts PASS: 21×2, TypeError=0, bypassed=0, gc evacuation verification failed=0. Each forced cold start: 443 copying minors, Σcopied_objects≈197.7k, liveness assert evacuation live on all five. Pre-fix this arm died deterministically at ~minor #238 — genuinely fixed at fixture scale.

Warm-process batches (one host, N× the 21-request verifier):

run	binary	env	pass/N	failing iters	host TypeError
b1	fix	—	99/100	12	1
b2	fix	—	100/100	—	0
b3	fix	—	99/100	12	1
d1	fix	`PERRY_GC_DIAG=1`	98/100	3, 12	2
d2	fix	`PERRY_GC_DIAG=1`	99/100	3	1
f1	fix	forced+seeded (`SEED=8036` env set)	29/30	22	1
base	main `3c95020f8`	—	96/100	3, 6, 12, 19	6

Fix: 5 failing batches / 500 default-GC (vs 4/100 on main same-hour; vs 8/500 in yesterday's main measurement). Every failure keeps the exact original signature: empty body + TypeError: value is not a function immediately after a [gc-copy-minor] ran in_place=false (d1: minor Support custom menu bar items #1→TypeError, minor Using the fetch API #3→TypeError; d2: minor Support custom menu bar items #1→TypeError).
f1: 3712 copying minors, Σcopied≈1.04M, verify-evacuation panics 0, host survived; its one TypeError landed right after copy-minor Add buffer INSPECT_MAX_BYTES to API manifest #2696 — pre-fix the same env died at ~fix(fetch): #234 real Blob with arrayBuffer/text/bytes/slice instance methods #238. ~10× longer survival, not unbounded.
Rate improvement vs today's baseline is real (≈1.0/100 vs 4/100); vs yesterday's 8/500 it overlaps Poisson noise — treat "reduced" as suggestive.
Retention unchanged by the fix: post_in_use 5.6→40 MB and RSS 160→~600 MB per 100 batches, same as main.

Conclusion: the two rooted holders were real, but at least one more unregistered holder sits on the default path (same consumer site, surviving copying minors under no GC knobs at all). Recommend landing 3b01efba4 on its merits (forced arm + measured rate reduction) while keeping this issue open for the residual, with the 100-pass warm-process batch as the acceptance instrument — the 10×2 fixture cannot see it.

Artifacts on the mini: worktree ~/perry-bench.noindex/wt-validate-8163, build ~/perry-bench.noindex/build8163 (10 cold-start logs), batch logs ~/perry-bench.noindex/logs8163/, runner ~/mini-batch-8163.sh; baseline logs ~/perry-bench.noindex/logs8040/batch-base-*.log.

Root-caused and fixed — PR #8211

Two holders, one behind the other. Both are outside the GC heap and unregistered with any root scanner, exactly as this issue predicted; the second only became visible once the first was fixed, because the first fault kills the process.

1. HEADERS_METHOD_VALUE_CACHE (crates/perry-stdlib/src/fetch/headers_method_value.rs) — a lazy_static! Mutex<HashMap<(handle, method), u64>> of NaN-boxed bound-method closures backing headers.get / .entries / …. Its only "rooting" was js_write_barrier_root_nanbox, which is the incremental-marking shade, not a root registration. This is the one that reaches js_value_typeof through Next's ReflectAdapter.get: the route reads (await headers()).get("x-request-id") twice per request with awaits between, so the closure allocated by the first read is retired by a later minor and the second read hands typeof the pre-move address. Attributed by instrumenting the cache: miss allocated id=262149 name=get bits=0x7ffd044c5059f980, a later hit returned the same bits, and the PERRY_GC_PROTECT_FROMSPACE fault landed at 0x44c5059f978 — the header of that 48-byte closure. FORM_DATA_METHOD_VALUE_CACHE and RequestRecord::signal are the same shape.

2. ServerResponse.once_listeners (crates/perry-ext-http/src/server/response.rs) — res.once(event, cb) stores into a second table that take_event_listeners merges into every emit, and scan_http_server_roots visited only listeners. Next's pipeToNodeResponse registers res.once('close', …). The close snapshot taken at the top of res.end() already held the retired address, i.e. the table itself was never rewritten. The res.end() tail also used listener/callback snapshots across JS calls (rooting inside emit_no_arg_to_listeners roots what it is handed), now fixed by a rooted EndTail.

The issue's suggested next move was right and is done, though not where expected: scripts/gc_runtime_root_holders.py had four blind spots that hid holder #1 in its own audited crates — DECL never matched lazy_static!'s static ref (93 tables invisible), strip_comments blanks string literals so extern "C" fn reached FN_DEF as extern "" fn (no C-ABI function had a body in the walk, including the FFI scanner trampolines), a body-less fn f(...); inside extern "C" {} swallowed the functions after it, and the registration regex stopped at the first ( of SOURCE.as_ptr(). Census 78 → 129 declarations, 53 → 74 reached, 31 new verdicts. The ext crates remain outside CRATES and are now named in the script's "cannot see" list — holder #2 was a struct field, which no declaration scan can reach; a parallel session is extending the census there.

Validation. Same app dylib, same seed, only the provider images swapped: main 59 TypeErrors / 0 PASS with a from-space fault at minor #213; fixed 2× PASS: 21, 0 errors, clean under PROTECT_FROMSPACE depth 800 on seeds 8036/8174/1/8040/4242. Full fixture.sh with PERRY_NEXT_ROUTE_FORCED_GC — now back ON by default — passes all 10 cold starts with liveness asserted (~197k objects copied per forced arm).

fix(gc): root the two holders outside the GC heap behind the forced-e…

The retention noted at the end of the 2026-08-16 comment ("post_in_use 5.4 -> 39.7 MB, RSS 160 -> 510-620 MB, wants its own home") now has one: #8213. Diagnosis: per-request, not per-connection — ~1,100 async box cells + 4 immortal Proxy registry entries per request; 55% of the retained JS heap hangs off BOX_REGISTRY, and the same registry's growth is why the copying-minor root scan (and this issue's failure window) gets hit more often late in a warm run: minor pause grows 59 ms -> ~0.5 s, and past ~pass 135 the host runs ~70 collections per verifier pass.

@proggeramlug

Correction to my "Root-caused and fixed" comment above — this issue should stay OPEN

@proggeramlug's 500-batch measurement (posted while I was writing that) shows the two holders I fixed are not the whole story, and I want my claim corrected rather than left standing:

Forced-evacuation arm: fixed. Their independent run of my branch agrees with mine — fixture 10/10 with liveness asserted, and under the seeded forced env survival goes from copying minor ~fix(fetch): #234 real Blob with arrayBuffer/text/bytes/slice instance methods #238 to ~Add buffer INSPECT_MAX_BYTES to API manifest #2696 (3712 minors, Σcopied ≈1.04M, 0 verify panics).
Default-GC path: NOT fixed. 5 failing batches / 500 on my branch vs 4/100 on a same-hour main baseline, every failure carrying the identical signature (empty body + TypeError: value is not a function immediately after a default-mode [gc-copy-minor] ran … in_place=false). So at least one more unregistered holder sits on the same consumer site.

I have changed #8211 from Closes #8163 to Refs #8163 so merging it cannot auto-close this. The residual is the more important half — it is production-visible without any GC knob, where the forced arm only made it deterministic — and per their offer they are taking that hunt, scoped out of the files #8211 touches (crates/perry-ext-http/src/server/**, crates/perry-stdlib/src/fetch/**, scripts/gc_runtime_root_holders.*).

Two notes for whoever picks the residual up:

The shipped fixture cannot see it — 2 verifier passes per process; the residual needs ~100 passes against one warm process. If this issue is to have an acceptance gate, that instrument is the one, not fixture.sh.
The validated commit 3b01efba4 is now 232f5c6e3 on the branch. The delta is a doc paragraph plus hoisting allocations out of the REQUEST_REGISTRY guard (registering the fetch scanner made those a self-deadlock: std::sync::Mutex is not reentrant and js_request_clone threw under the guard, which under panic=abort unwinding leaves it locked permanently). None of it changes rooting behaviour on the default path, so the 500-batch numbers should carry over — but it is a different binary, so re-baseline if you want the measurement to be exact.

Reopening — closed by automation, not by resolution

#8211 merged as b8d32ab19 at 15:16:17Z and auto-closed this issue one second later, because its body still read Closes #8163 at merge time. I edited that to Refs #8163 a few minutes afterwards, which was too late — the close keyword fires from the body as it stood when the merge happened. My mistake in sequencing, not a judgement that the issue is done.

What merged is a partial fix, and the content is verified on main (the fetch root scanner, the ext-http once_listeners + EndTail rooting, the four census-parser fixes, and the REQUEST_REGISTRY guard hoists are all present in b8d32ab19):

Forced-evacuation arm: fixed. Survival goes from copying minor ~fix(fetch): #234 real Blob with arrayBuffer/text/bytes/slice instance methods #238 to ~Add buffer INSPECT_MAX_BYTES to API manifest #2696; fixture.sh passes 10/10 cold starts with liveness asserted, twice, independently.
Default-GC path: NOT fixed. 5 failing warm batches / 500 on the fixed branch versus 4/100 on a same-hour clean-main baseline, every failure carrying the identical signature — empty body plus TypeError: value is not a function immediately after a default-mode [gc-copy-minor] ran … in_place=false. At least one more unregistered holder sits on the same consumer site.

The residual is the more important half: it is production-visible with no GC knob set, where the forced arm only made it deterministic. The hunt for it is under way, scoped out of the files #8211 touched.

Acceptance instrument for whoever closes this for real: ~100 verifier passes against ONE warm process. The shipped fixture.sh runs 2 passes per process and is structurally blind to this failure — a green fixture is not evidence here.

docs(next): state what the warm soak's confidence figure does and doe…

test(next): arm the routeModule.handle guard and make verifier passes…

test(next): reconcile the header with post-#8211 state (#8163 reopene…

fix(gc): root the Headers/FormData iteration frame slots (#8163, #8217)

Residual holder NAMED and fixed: `js_headers_for_each`'s hoisted closure pointer is a native Rust FRAME SLOT — PR #8220

The remaining default-GC holder is crates/perry-stdlib/src/fetch/headers.rs:452, at the binding on line 460:

let closure = cb_ptr as *const perry_runtime::ClosureHeader;   // hoisted OUT of the loop
for (k, v) in entries {
    let v_ptr = js_string_from_bytes(v.as_ptr(), …);   // allocates -> collection point
    let k_ptr = js_string_from_bytes(k.as_ptr(), …);   // allocates -> collection point
    …
    js_closure_call2(closure, …);                      // runs user JS -> collection point
}

It is not a side table — it is a callee-saved register (x20) in a native Rust frame. The precise root map does not cover Rust frames, and production resolves the conservative stack scan to SkipDisabled (the reproducer's minors all report eligible=true fallback=none), so the collector never rewrites it. Note the same function also creates v_ptr, then allocates k_ptr — a second collection point — and only then reads v_ptr, so the callback could receive a dangling value string too.

Evidence chain

step	instrument	result
1	`PERRY_GC_PROTECT_FROMSPACE=1`, default GC, no other knobs	SIGBUS at `js_headers_for_each + 200` (the `bl` invoking the callback), via `dispatch_headers_method+1176` ← `js_handle_method_dispatch` ← the compiled route chunk
2	fault report	`obj_type=4` (GC_TYPE_CLOSURE), `size=48`, fault at `user_ptr + 12` = `CLOSURE_TYPE_TAG_OFFSET`, i.e. `get_valid_func_ptr`'s `CLOSURE_MAGIC` probe
3	`PERRY_GC_PROTECT_FROMSPACE_HOLDERS=1`	`none in 90262 objects` — confirming the holder is outside the arena, exactly as this issue predicted
4	source + disassembly	the hoisted `let closure`, held in `x20` across two `js_string_from_bytes` calls and the callback

Mechanism end to end. js_closure_call2 → get_valid_func_ptr reads CLOSURE_MAGIC at +12; once the retired block is recycled into Eden and partly overwritten the magic check fails, it returns null, dispatch_proxy_callee_or_throw falls through to throw_not_callable() → TypeError: value is not a function. The consumer is Next's pipeToNodeResponse header copy, present verbatim in the fixture's route chunk: c.headers.forEach((a,c)=>{… b.appendHeader(c,d) …}). The throw aborts the header copy, Next never writes the body → E180 failed to pipe response + empty body, and never a wrong id/header/cookie, because the content was already correct and only the write was abandoned. It also explains why stale derefs outnumber visible failures ~10:1 — from-space is bump-allocated over gradually, so most stale derefs still read an intact magic.

One holder, caught 8 times. All 8 faults are obj_type=4 size=48 at the same faulting code site, retired_by_minor spanning #0/#1/#1/#1/#2/#3. The identical-offset +354604 pair is the same deterministic allocation under an identical schedule (both failed at iteration 6 with 2 minors), not a second site. No second holder emerged in 360 protected passes.

Fix: PR #8220 (draft) — roots the frame slots via RuntimeHandleScope and re-reads the closure at each call. It covers all four sites of #8217 plus five more the new test caught (js_headers_get_set_cookie, js_form_data_for_each, and three in body_metadata.rs), so it closes #8217 and proves the reachability #8217 left unproven.

Matched A/B (identical app dylib and host; arms differ only by the patch):

arm	config	result	copying minors
before	protect, default GC, 12 runs	8 faults	51
after	protect, default GC, 6 runs	0 faults	60
before	default GC, no knobs, interleaved, 300 passes	3 failed batches / 3 TypeErrors	—
after	same	0 / 0	—
before	default GC `HEAP_LIMIT=8`, 75 passes	1 failure	6334
after	same	0	6339
before/after	forced+seeded `SEED=8036`, 30 passes	30/30 both, 0 verify panics	3723 / 3719

Under the before rate (8 faults / 51 minors), P(0 faults across the after arm's 60 minors) ≈ 8×10⁻⁵. The protect arm is the discriminating one: it faults on every stale deref, not only those whose bytes were already clobbered — which is why 6 runs suffice where the unprotected rate needs hundreds of passes. Your collections-not-requests finding is what made the HEAP_LIMIT=8 arm (6,334 minors in 75 passes) practical.

A/B base is 3b01efba4, not merged main, because merged main cannot build the fixture — see #8228. I verified the transfer argument rather than taking it: crates/perry-stdlib/src/fetch/headers.rs is blob-identical (a3120f1600…) at 3b01efba4, b8d32ab19, and current main 2e7a12818, and gc/roots/runtime_handles.rs differs only in doc-comment example text. The code under test and its rooting machinery are the same on all three.

Not needed from you: no gc_runtime_root_holders.json verdict — this fix adds no static or thread-local holder, only frame-local roots. Thanks for the offer, and for releasing the fetch/** glob; without that this would have had to change hands mid-fix.

The class is systemic. A scan for "raw heap pointer bound, then used after another allocating call" over perry-stdlib + perry-ext-* reports ~150 candidate sites, and no existing instrument covers them: gc_runtime_root_holders.py audits declarations, gc_root_dominance_check.py reads IR emitted from compiled JS, and raw_handle_debt.py counts bare reads only inside crates/perry-runtime/src. A ratchet over the stdlib/ext crates is the obvious next gate — worth its own issue.

Qualification on the numbers in my comment above (5308595639): the A/B was measured on base 3b01efba4, which is pre-#8204 — the 56 B → 48 B object-header shrink — while the fix merges onto a post-#8204 base. My blob-identity argument established that the code under test was the same; it did not establish that the object layout the collector operates on was, and for a rooting/evacuation defect that is not a neutral variable. Raised by the #8163 author, verified here. Detail, and what still stands unaffected (the attribution chain, and the defect's presence on current main), in #8220 (comment). A matched post-#8204 re-run is blocked on #8228, for which PERRY_LLVM_INPROCESS=off is being tested as an interim workaround.

test(next): arm the routeModule.handle guard and make verifier passes…

fix(gc): root the Headers/FormData iteration frame slots (#8163, #8217)…

test(next): add the warm-process soak arm #8163's residual needs (#8215)

Follow-up to my qualification above (5308859673): the matched post-#8204 measurement is now done, on clean origin/main 695daf302b with compiler, runtime, stdlib, providers and app dylib all built from one tree.

300-pass warm soak: PASS 300/300, 14,057 copying minors, 0 TypeError: value is not a function. Plus 200 further passes across 20 cold processes (two fixture copies, guard armed, 0 bypass fires) and a forced/verified arm whose liveness assert fired on all five processes (~1,376 minors each, 2.28M objects copied). Zero failures in 500 passes / 10,500 requests. So #8220's fix reproduces on the 48 B layout the merged code actually runs on — the gap I flagged is closed on this axis.

Two things keep this from being the close condition you specified, and I'm not claiming it as such:

Text backend only. [codegen] In-process LLVM backend fails the 5 biggest Next App Route modules on current main (silent per-module failure; regression in 3c95020f8..07c8040bf) #8228 still prevents the native in-process path from compiling the fixture, so this ran under PERRY_LLVM_INPROCESS=off — 16 and 62 modules respectively dropped to -O0, 203 MB dylib vs ~87 MB native. The native path is the default and remains unmeasured. fix(codegen): teach the in-process IR reader the vector instructions codegen emits (#8228) #8241 is ready for review.
Your minors-per-100 request, answered — and it cuts both ways. Passes 1–120: 20 minors/100, against the ~21–22 native baseline, independently reproduced by 15 fresh processes across both fixture copies. So in the small-heap regime the schedules are indistinguishable, which is the genuinely useful result you predicted: =off is a usable GC instrument there. But at pass ~130 (live heap ≈47 MiB) it phase-transitions to exactly 84 minors/pass and stays pinned — 8,400/100 passes for the rest of the run. The soak delivered ~213× the collections a native-rate run would have, so it over-exercised rather than under-exercised the collector; whether that saturation is a -O0 artefact or a property of main's collector at ~50 MiB live is exactly what your native-path soak would settle.

Full table and caveats: #8040 (comment). The native-path 300-pass soak after #8241 remains yours — the mini's fixture tree and providers are built and preserved at ~/perry-bench.noindex/{wt-8040-final,build-final} if you want a matched tree rather than rebuilding.

Native-path warm soak: 265 of 300 passes, 10,995 copying minors, zero residual signatures

The measurement I said should close this issue, on the shipped backend. It did not reach the stated 300 passes — the host box ran out of disk at pass 265 and the process was killed. I am reporting what it got rather than claiming the bar I set, and then arguing about what the bar was actually for.

Matched tree end to end: 183d30c53. Compiler, both provider images and the app dylib all built from that one commit, with no PERRY_LLVM_INPROCESS flag — the default native path, i.e. the configuration users get. Prerequisites verified as ancestors rather than assumed: #8220 310c95ec4 (the js_headers_for_each fix), #8241 f70abf752 (native dialect reader).

The compile itself confirms the native reader was exercised, not routed around:


modules entering the multi-unit native path	5 (`chunks/2.js` 3u, `chunks/430.js` 6u, `jsonwebtoken` 2u, `app-page.runtime.prod.js` 10u, `app-route.runtime.prod.js` 4u)
`bad binary op`	0
modules failed	0

Those are precisely the five #8228 used to reject. Under the PERRY_LLVM_INPROCESS=off workaround this count is 0 — so a soak taken that way could not have said anything about the shipped configuration.

Result

discriminator	count
`Unexpected end of JSON input` (empty body — the residual's signature)	0
host `TypeError: value is not a function`	0
`E180 failed to pipe response`	0
copying minors	10,995
passes	265 / 300

On the 300

I set that number, so I should be the one to say what it was worth. It came from the rule of three at p = 0.01 per pass — and that rate was measured entirely in the pre-knee regime, where a pass is worth ~0.2 collections. The bug is per-collection, not per-pass, so what "300 passes" was really buying was on the order of 60 collections of exposure. This run delivered 10,995 — roughly 180× the intended bar, in the quantity that actually matters.

Against the pre-fix rates measured on this same fixture: ~1 failure per 22 collections (my 100-pass run) predicts ~500 failures here; the most conservative figure anyone measured, 1 per 6,334 under HEAP_LIMIT=8, still predicts ~1.7. Observed: zero.

I am not going to pretend the round number was reached. I am also not going to pretend it was the meaningful quantity once the cadence turned out to be non-uniform by two orders of magnitude.

Cadence — why "minors per 100 passes" is not a property of the collector

pass   1:  0 minors/pass    cumulative      0
pass 100:  0 minors/pass    cumulative     20
pass 140:  4 minors/pass    cumulative     71
pass 160: 84 minors/pass    cumulative  1,370
pass 265: 84 minors/pass    cumulative 10,995

Nearly flat for ~120 passes, then a knee to a saturated ~84/pass. Any per-100 average is a statement about where you stopped. This knee is independently reproduced by the parallel session at ~84/pass after the transition at ~47 MiB live, on a different host and a different backend — so it is a property of the collector, not a -O0 artefact.

Two caveats, both mine

1. The arm's own verdict line is wrong and should be ignored. 12 passes failed, all UND_ERR_HEADERS_TIMEOUT — client-side timeouts (passes 203 and 238), with a clean host log. run_warm_soak counts verifier exits instead of classifying them, so it reports #8163 residual for a run with no residual evidence. That is a bug in the instrument I added in #8215, filed as #8286 and mine to fix. Cite the discriminator table above, not the verdict.

2. The timeouts are a real finding, just not this one. Both timeout passes are collection storms — pass 203 ran 644 minors against a steady 84. That is #8213, and it is user-visible: the caller gets nothing. Posted there as comment 5312115049.

Status

I do not think this issue should be reopened on my evidence. I also do not claim the literal close condition was met. The parallel session has a matched native tree with prebuilt artifacts and 20 GB free; I have asked them to run the literal 300 to remove the asterisk, and I said earlier that if my run died I would take that offer rather than restart on a box that has now hit zero bytes four times.

fix(test): classify warm-soak failures instead of counting them (#8286)

Close condition met on the native backend: 300/300 passes, 13,089 copying minors, zero residual signatures

This is the arm started by the session working this issue and lost to disk exhaustion at pass 265, completed on the bench mini. Same tree, same artifacts, same instrument — not an independent replication. Run 2026-08-17 07:24:23Z → 08:57:17Z, 92.9 min.

Tree 183d30c53a. Providers, host and app dylib reused byte-identical:

next-app.dylib            87,475,088 B  sha256 3b64c74c96f4dcaba25f7d0e26277af7145c78980481e124b129dc02106f0a28
libperry_runtime_provider 33,592,320 B  sha256 756870aa6c4da743815ab8fdd58ea319b6b1cd1e35ed654202d03a5110c95748
libperry_stdlib_provider  74,642,560 B  sha256 9fa17c44f9779b69ce175cbc8c52dd80cb9e8049e07fa119af8bd6ce0efe8012

No PERRY_LLVM_INPROCESS — the default native path. No GC knobs. Only PERRY_GC_DIAG=1, as run_warm_soak sets it; every PERRY_* unset before launch and asserted at 0.

discriminator	meaning	count
`Unexpected end of JSON input`	empty body — residual	0
`TypeError: value is not a function`	the stale closure — residual	0
`E180 failed to pipe response`	residual-adjacent	0
`UND_ERR_HEADERS_TIMEOUT`	client gave up — not the residual (#8213)	0
`generated handler bypassed routeModule.handle`	bypass guard	0
`PASS: 21 production App Route requests`	successful passes	300

Σ copied_objects 5,741,420, Σ promoted 2,247,344.

The arm's own verdict line, quoted separately: [warm soak] passes=300 failures=0 copying_minors=13089 host_type_errors=0. #8286 did not fire — there were zero non-zero verifier exits, so the exit-counting path was never reached. That agreement is luck, not vindication: the bug is still there and would have misreported the previous run.

Exposure in collections, not passes

The defect is per-collection, so that is the quantity that matters:

pre-fix rate on this fixture	predicted failures at 13,089 collections	observed
1 per 6,334 (most conservative measured)	~2.07	0
~1 per 22 (100-pass run, measured pre-knee)	~595	0

Leading with the conservative figure deliberately: P(0 | λ=2.07) ≈ 12.6%, so that arm alone would not settle it and I am not claiming it does. The ~1-per-22 rate was measured pre-knee and extrapolating it across mostly post-knee collections assumes a regime-independent per-collection failure probability that nobody has demonstrated. The verdict rests on both runs together — this 13,089 plus the previous session's 10,995 on a different host, ~24,000 collections of combined clean exposure.

Anti-vacuity — and its one honest limitation

freeze/LLVM pipeline started = 5 in the compile log that produced these artifacts (emitted only by native_emit::compile_module_units_native; 0 on the text path), naming exactly the five modules #8228 rejected — chunks/2.js (3 units), chunks/430.js (6), jsonwebtoken (2), app-page.runtime.prod.js (10), app-route.runtime.prod.js (4). bad binary op 0, modules failed 0.

Limitation, stated rather than glossed: that count comes from the compile that produced the artifacts, not from this run. The artifacts are byte-identical to that compile's outputs (hashes above, log sha256 842e0448b5… preserved in two locations), so the chain holds — but "the arm ran and was clean" and "the arm ran against something proven to be built on the native path" are different claims, and only the second is load-bearing.

Cadence — the knee is a latch, and it reproduces across hosts

passes	per pass	cumulative	`post_in_use`
1–120	~0.24	29	43.3 MB
121–140	1.0 → 3.0	69	49.4 MB
141–150	42.1	490	51.2 MB
151–300	84.0	13,089	123.4 MB

Pinned at 84.0/pass for 150 consecutive passes (mean 83.99, min 83, max 84); onset at post_in_use ≈ 47–51 MB, matching the ~47 MiB the text-path run reported. Cross-host: cumulative 69 vs 71 at pass 140, 1,330 vs 1,370 at pass 160, 10,149 vs 10,995 at pass 265 — within ~8%, explained by their knee landing ~10 passes earlier. Two hosts, same backend: 84/pass saturation is a property of the collector.

The new finding: the storm fails requests, saturation alone does not

The other host's storms (pass 203 at 644 minors, pass 238) did not reproduce here, and neither did any timeout. Maximum single-pass minor count across all 300 was 84; no pass exceeded 100. Their 8 timeouts were confined to the 644-minor storm.

So 150 consecutive passes at a flat 84 minors/pass produced zero client-visible failures. The #8213 latency failure tracks the storm specifically, not the saturated steady state. Saturation is merely expensive; the storm is the hazard — and per the two-host comparison it is stochastic in placement.

(The one non-84 pass, 223, is its opposite: sweep_freed=30,598,680 with pre_in_use == post_in_use and block_reclaim=0 — a tracked-malloc registry sweep, not arena reclaim. It is the 31.7% outlier in the yield distribution and should not be read as the arena recovering anything.)

Verdict

The #8163 residual signature does not reappear on the native codegen backend across the full 300 passes. The native backend is asserted live, the collector is asserted live (13,089 minors, 5.74 M objects copied, 2.25 M promoted), and the run completed rather than being cut short. The round number set by the previous session is reached without an asterisk, and the meaningful quantity — collections — exceeds the original intent by roughly 200×.

Deviation, stated plainly: I drove the shipped run_warm_soak loop against the prebuilt artifacts rather than invoking fixture.sh end to end. The fixture has no reuse path — steps 1–5 unconditionally rebuild, and PERRY_NEXT_ROUTE_BUILD_DIR selects where to write, not whether to build, so pointing it at these artifacts would have overwritten the very things this run exists to test (and Perry binaries are not byte-deterministic, so the result would no longer have been the previous session's arm). The soak's own configuration was used verbatim, with fail()/cleanup_server() eval'd out of fixture.sh rather than copied; additions were bookkeeping only (per-pass markers, a CSV, a disk watchdog).

Not closed by this: #8213 is corroborated and sharpened — post_in_use 5.22 → 123.4 MB (23.7×, no plateau), peak RSS 894 MB, saturated sweep yield median 0.043% at a pinned 1 GiB stride, tail throughput 40 s/pass rising to 53 s against ~0.16 s/pass pre-knee (~300× degradation).

Evidence on the mini: ~/perry-bench.noindex/logs-8163-soak/{perry-warm-soak.log (30 MB), passes.csv, driver.log, df.log}.

fix(test): classify warm-soak failures instead of counting them (#8286)…

[Next.js/dylib] Forced-evacuation App Route arm: stale closure from a holder outside the GC heap #8163

Description

Symptom

What it is not (each ruled out by measurement)

Where the fault lands

Suggested starting point

Reproducing in seconds

Activity

proggeramlug commented on Aug 16, 2026

Retested on 53e8a21e3 (main, #8168 included): STILL REPRODUCES

The run

The forced arm ran, and its subject was live

It is the same bug

Related work

proggeramlug commented on Aug 16, 2026

Reproduces under DEFAULT GC — not only the forced arm

Cheap reproducer (seconds once the fixture is built)

Also seen, separate from the stale closure

proggeramlug commented on Aug 16, 2026

Independent validation of the candidate fix 3b01efba4 (fetch caches + ext-http once_listeners/EndTail rooting): forced arm FIXED, default-GC arm NOT — a holder remains

proggeramlug commented on Aug 16, 2026

Root-caused and fixed — PR #8211

proggeramlug commented on Aug 16, 2026

proggeramlug commented on Aug 16, 2026

Correction to my "Root-caused and fixed" comment above — this issue should stay OPEN

proggeramlug commented on Aug 16, 2026

Reopening — closed by automation, not by resolution

proggeramlug commented on Aug 16, 2026

Residual holder NAMED and fixed: js_headers_for_each's hoisted closure pointer is a native Rust FRAME SLOT — PR #8220

proggeramlug commented on Aug 16, 2026

proggeramlug commented on Aug 16, 2026

proggeramlug commented on Aug 17, 2026

Native-path warm soak: 265 of 300 passes, 10,995 copying minors, zero residual signatures

Result

On the 300

Cadence — why "minors per 100 passes" is not a property of the collector

Two caveats, both mine

Status

proggeramlug commented on Aug 17, 2026

Close condition met on the native backend: 300/300 passes, 13,089 copying minors, zero residual signatures

Exposure in collections, not passes

Anti-vacuity — and its one honest limitation

Cadence — the knee is a latch, and it reproduces across hosts

The new finding: the storm fails requests, saturation alone does not

Verdict

Metadata

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

Issue actions

Retested on `53e8a21e3` (main, #8168 included): STILL REPRODUCES

Independent validation of the candidate fix `3b01efba4` (fetch caches + ext-http once_listeners/EndTail rooting): forced arm FIXED, default-GC arm NOT — a holder remains

Residual holder NAMED and fixed: `js_headers_for_each`'s hoisted closure pointer is a native Rust FRAME SLOT — PR #8220