perf(gc): main regressed the retain cluster 2.2-4.8x — retain now runs 2 full collections where it ran none (suspect #7901/#7902)

Summary

The retain cluster regressed 2.2–4.8× on main between 8260a9e50 and d78efca41. Independently confirmed by a second party on a separate build.

The load-independent evidence is the collection census — main now runs full mark-sweeps on programs that previously ran none:

program	`collection_kind:"full"`	`minor`
`retain`	2	6
`retain1`	1	3
`deeplist`	1	3

(measured at cf9999855, PERRY_GC_TRACE=1 PERRY_GC_DIAG=1)

The last clean quiet-mini sweep at 8039fa798 recorded retain 0.153 s (1.13× node), retain1 0.068 s, deeplist 0.057 s — and that commit predates both suspects, so the regression is invisible to it. Corroborating timing on a loaded dev box (magnitude only, not a benchmark): retain 0.83 s, retain1 0.24 s, deeplist 0.24 s.

The original reporter measured retain 2,840 M → 12,421 M instructions with fulls costing 190 ms + 605 ms.

Suspected cause — not bisected

Two PRs merged in that window, both in the promotion/survival accounting:

memory(gc): copied minor subtracts from-space high-water from an exact live census #7901 / e886b56dd — "subtract live from-space bytes, not the block high-water, in a copied minor". Changes what the survival ratio is computed from.
memory(gc): untraced promotion can retain an old-heap-sized phase-change cohort as live #7902 / 1bd5eeb6b — "bound and heap-scale the untraced-promotion budget". Bounds the untraced budget ⟹ more, smaller promotion steps.

Both are plausible on mechanism, and they interact: the survival ratio feeds the in-place-promotion decision, and promotion step size feeds the old-gen trigger.

The mechanism this most likely trips

A third party independently instrumented old_reclaim_pressure_due's absolute arm (old_in_use >= T && baseline < T) and found it races two quantities that both move with promotion, so whether it fires depends on promotion step size, not on the heap: same program, same live set, same total promotion, and changing only the schedule from (18.7, 34.6) to (17.7, 17.8) makes it fire twice and buy 588 ms of futile fulls.

#7592 exempted the proportional arm via GC_MAJOR_PACING_RETAINING and left this one. #7902 changing promotion step size is exactly the input that would newly trip it.

That also makes this an instance of the rule CLAUDE.md already carries, for the fifth time: never pace a collection on a quantity that collection does not move — and its corollary that a constant band cannot pace an O(live) process.

Asks

Bisect 8260a9e50..d78efca41 on the full count (a counter, so a single run per commit suffices — no quiet host needed).
Owners of memory(gc): copied minor subtracts from-space high-water from an exact live census #7901 and memory(gc): untraced promotion can retain an old-heap-sized phase-change cohort as live #7902 should look at the interaction rather than each PR alone; each may be individually correct.
Whatever the fix, gate it on the full count, not on wall time — that is what makes this cheap to detect and is how it was found.

Note on downstream measurements

Any A/B taken against a base after e886b56dd is measuring against a regressed baseline. #7960 (first-cycle in-place promotion) reports −41% to −77% against current main but −14% to −31% against the pre-regression base; the latter is the honest figure and its author flagged this unprompted.

Bisected to one commit and one line: #7944 (#7902) stopped crediting untraced promotion to the old-reclaim baseline

Independent confirmation, plus the bisect. Four builds, each a full
-p perry -p perry-runtime-static -p perry-stdlib-static in an isolated target
dir; corpus byte-exact vs node --experimental-strip-types 26.5.1 and exit 0 at
every hop. Instructions retired (load-independent — the dev box sat at load
77–92), full = collection_kind:"full" in the PERRY_GC_TRACE=1 JSON.

commit		deeplist	retain1	retain	retain_wide
`0809ada92` (before all three)	fulls / instr	0 / 1117 M	0 / 1286 M	0 / 2567 M	0 / 2967 M
`e886b56dd` (#7900 + #7901)	fulls / instr	0 / 1106 M	0 / 1294 M	0 / 2596 M	0 / 3026 M
`1bd5eeb6b` (+ #7902)	fulls / instr	1 / 3910 M	1 / 3720 M	2 / 12 431 M	2 / 8786 M
`ac52a5c38` (current main)	fulls / instr	1 / 3928 M	1 / 3734 M	2 / 12 457 M	2 / 8755 M

#7900 and #7901 are exonerated — flat to ≤0.6%, which is this A/B's noise
floor. The whole regression appears at #7944 (#7902) and nothing after it
moves. Peak RSS moves with it: deeplist 75.5 → 103.5 MB (+37%), retain1
87.0 → 116.6 MB (+34%), retain 249 → 347 MB (+39%).

★ It is NOT promotion granularity

The leading hypothesis was that #7902's bounded untraced budget produces more,
smaller promotion steps, newly straddling old_reclaim_pressure_due's absolute
arm. Measured, that is false. deeplist's promotion trace is byte-identical
across the boundary — same cycles, same untraced classification, same counts:

e886b56dd:  untraced=false untraced_cycles=0 untraced_objects=0
            untraced=true  untraced_cycles=1 untraced_objects=337035
            untraced=true  untraced_cycles=2 untraced_objects=954928
1bd5eeb6b:  (identical, all three lines)

The actual mechanism — one line

-    credit_promoted_bytes_to_old_baseline(collector.stats.promoted_bytes);
+    if !untraced {
+        credit_promoted_bytes_to_old_baseline(collector.stats.promoted_bytes);
+    }

deeplist and the retain* family promote through the untraced in-place
path. Post-#7902 those bytes no longer advance
GC_LAST_OLD_RECLAIM_IN_USE_BYTES, so old_in_use climbs to 53.5 MB while
baseline stays pinned at 0 — and then both arms of

(old_in_use >= T && baseline < T) || old_in_use.saturating_sub(baseline) >= band(baseline)

are true for the first time (T = GC_OLD_GEN_RECLAIM_THRESHOLD_BYTES = 48 MB).
The resulting full mark-sweep reports sweep.freed_bytes = 0: it is a
whole-heap collection over an old-gen that is live by construction. On
deeplist that one futile cycle is 72% of the program's instruction count.

So the arm's shape is not the bug — its input was silently zeroed. #7902's
reasoning ("an untraced promotion makes no marked-liveness claim, so do not tell
old-reclaim pacing that cohort is clean") is sound about liveness, but the
baseline is not a liveness claim, it is the denominator of a growth band.
Zeroing it does not make pacing more conservative, it makes the band meaningless:
old_in_use - 0 exceeds any band as soon as old-gen exists at all.

Two shapes for a fix, both preserving #7902's intent:

Credit untraced promotions to the baseline but keep them accounted as the
uncertain class elsewhere — the baseline records where old-gen was, not
what was proven clean.
Keep the guard but make the absolute arm unreachable when the last cycle
measured a ~100%-live young generation (young_survival_permille was 1000
on every one of these cycles), i.e. do not schedule a reclaim that the
collector has just finished proving cannot free anything.

Interaction with #7929 / PR #7961

Recorded so nobody double-counts: PR #7961 lowers the nursery band on
small-object programs, which keeps deeplist under the 48 MB threshold and so
dodges this regression without fixing it. Against current main that reads
as −72.3%; against the pre-regression base e886b56dd the same binary is
−1.6% on deeplist — i.e. essentially all of that headline is #7965, not
#7929. retain1/retain/retain_wide still carry the full with #7961 applied.
The #7961 numbers are being restated both ways on that PR.

★ #7961 (merged) is a second, independent trigger for the same latch — this must be fixed together

Follow-up to the bisect above. I rebuilt #7961's change on the pre-regression
base (e886b56dd, before #7902) to measure the term in isolation, and it
reproduces this issue's symptom on its own:

program	`e886b56dd` fulls / instr	`e886b56dd` + #7961	Δ
retain1	0 / 1293.7 M	1 / 3342.4 M	+158%, RSS +25.4%
retain	0 / 2596.4 M	1 / 4635.8 M	+78.5%, RSS +2.6%
deeplist	0 / 1107.2 M	0 / 1093.5 M	−1.2%

baseline (GC_LAST_OLD_RECLAIM_IN_USE_BYTES) is a running sum of promotion
deltas, so the absolute arm's baseline < T is decided by promotion step
size. On retain1:

	steps	final baseline	latch
base	18.74 + 34.60 MB	53.34 MB > 48	disarmed by overshoot
+#7961	18.74 + 26.21 MB	44.95 MB < 48	still armed

retain1's all array is a large object born straight into old-gen; its growth
never passes through promotion, so it never credits the baseline. It then carries
old_in_use past 48 MB and the still-armed latch fires — freed_bytes = 8 257 632, a whole-heap collection for 8 MB of dead array backing stores.

So the arm is disarmed only by accident — by a promotion step large enough to
overshoot T. #7902 removed one source of credit; #7961 shrinks the steps. Neither
is wrong about its own subject; the latch is wrong to depend on either.

Ordering hazard, please do not miss it: on current main both arms already
carry this full, so #7961 measures as a win there (retain1 −10.9%, retain
−63.1%). The moment this issue is fixed, those two flip to +158% / +78.5%.
A fix for #7965 that does not also re-measure #7961's nursery band will land a
regression on retain1/retain that looks like it came from nowhere.

Both are closed by the same durable change — make disarming a function of heap
state rather than of step size. Either credit old-gen growth to the baseline
however it arrived, or refuse to schedule a reclaim when the last completed cycle
measured young_survival_permille = 1000 (it is 1000 on every cycle in every row
above, i.e. the collector had already proved the reclaim could free nothing).

Fixed in PR #7968 — and the mechanism proposed in this issue is refuted

Bisect (on the full count, one run per commit): 0809ada92 → e886b56dd
(#7900 + #7901) is flat at ≤0.6% — both exonerated; e886b56dd → 1bd5eeb6b
(#7902) carries the entire regression. Confirmed independently here:
cdb67021e reproduces 8260a9e50's census exactly (retain 5 cycles / 0
fulls / 2 841 M instr / 250.1 MB).

★ The granularity hypothesis is false here

This issue proposed that old_reclaim_pressure_due's absolute arm is decided by
promotion step size. That is a real property of that arm — it is why #7960
exempted it — but it is not what happened: deeplist's untraced-promotion
trace is byte-identical across the boundary. Step size did not move.

The actual cause is #7902's `if !untraced { credit_… }`

GC_LAST_OLD_RECLAIM_IN_USE_BYTES is not a liveness claim, it is the base of
a growth measurement. A fully-live young generation promotes untraced on every
cycle, so nothing else credits the baseline on exactly the workloads that reach
this path and it stays pinned at 0 — whereupon old_in_use - baseline
collapses into absolute occupancy and gc_old_reclaim_growth_band_bytes's
proportional half (baseline / 2) collapses with it, leaving the constant floor.

So this is the fifth instance of the rule, but reached from underneath: not
"paced on a quantity the collection does not move" but its corollary — a
constant band pacing a collector whose per-cycle cost is O(live), the shape
#7592 removed here and #7594 removed one generation down.

Result

19/19 byte-exact + exit 0 both arms. retain 1 full → 0, 8 237 M → 2 176 M
instructions (−73.6%), RSS 311 → 249 MB; retain_wide 1 full → 0,
5 741 M → 2 859 M (−50.2%). No program gains a cycle or a full. Against the
pre-regression base the fix lands below it (retain −23.4%, retain1 −31.5%,
deeplist −17.5%) — i.e. #7960's independently-reported −14%…−31%, which is the
arithmetic check that the regression is gone rather than masked.

Gated by an_untraced_promotion_credits_the_old_reclaim_baseline: a real
untraced-promoting minor driven through the production entry point must advance
the baseline by exactly the bytes it relocated, plus a replay of retain's
measured promotion series asserting that an uncredited baseline does fire —
so a green run proves the credit is load-bearing rather than that nothing was
tried. That second assertion failed on its own first draft.

Note for #7929/#7961: its band merged on top of this regression and its
reported wins were the spurious fulls moving. Re-measured with the fix in both
arms and posted to #7929 — ±0.2% on the retain cluster, and +42…83% collection
count on cycles/interp/iso_miss.

fix(gc): credit every promotion to the old-reclaim baseline, untraced…

ci: repair three broken gates — Windows cp1252, llvm-inprocess vacuou…

gc: pace a full by the bytes promoted since the last full, at the pro…

Navigation Menu

Repository navigation

perf(gc): main regressed the retain cluster 2.2-4.8x — retain now runs 2 full collections where it ran none (suspect #7901/#7902) #7965

Summary

Suspected cause — not bisected

The mechanism this most likely trips

Asks

Note on downstream measurements

Bisected to one commit and one line: #7944 (#7902) stopped crediting untraced promotion to the old-reclaim baseline

★ It is NOT promotion granularity

The actual mechanism — one line

Interaction with #7929 / PR #7961

★ #7961 (merged) is a second, independent trigger for the same latch — this must be fixed together

Fixed in PR #7968 — and the mechanism proposed in this issue is refuted

★ The granularity hypothesis is false here

The actual cause is #7902's `if !untraced { credit_… }`

Result

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

perf(gc): main regressed the retain cluster 2.2-4.8x — retain now runs 2 full collections where it ran none (suspect #7901/#7902) #7965

Description

Summary

Suspected cause — not bisected

The mechanism this most likely trips

Asks

Note on downstream measurements

Activity

proggeramlug commented on Aug 12, 2026

Bisected to one commit and one line: #7944 (#7902) stopped crediting untraced promotion to the old-reclaim baseline

★ It is NOT promotion granularity

The actual mechanism — one line

Interaction with #7929 / PR #7961

proggeramlug commented on Aug 12, 2026

★ #7961 (merged) is a second, independent trigger for the same latch — this must be fixed together

proggeramlug commented on Aug 12, 2026

Fixed in PR #7968 — and the mechanism proposed in this issue is refuted

★ The granularity hypothesis is false here

The actual cause is #7902's if !untraced { credit_… }

Result

Metadata

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

Issue actions

The actual cause is #7902's `if !untraced { credit_… }`