Skip to content

conformance: bound concurrent BS-05..BS-08 by host memory - #554

Merged
afflom merged 3 commits into
release/v0.4from
ci/aarch64-heavy-cases
Nov 15, 2026
Merged

afflom merged 3 commits into
release/v0.4from
ci/aarch64-heavy-cases

Conversation

@afflom

@afflom afflom commented Nov 15, 2026 •

Copy link
Copy Markdown
Member

The build and non-Lean tests (aarch64-unknown-linux-gnu) job keeps losing its runner ("received a shutdown signal") while BS-05..BS-08 are running together: jobs 132800007947, 132837774577 and 132871560408 on #548, and 132799549139 on #551. The hosted logs show the shutdown but no OOM diagnostic, so OOM as the hosted cause is an inference from the local reproduction below.

Local reproduction. 4 processors, MemTotal 16473776 KiB, a 13.4 GiB cgroup limit, and the conformance binary built with --all-features. The cgroup OOM killer's dmesg record confirms each kill.

  • BS-05..08 run together: killed within about 60 s, at an anon RSS of 11.6 GB.
  • BS-07 and BS-08 as a pair: killed, at an anon RSS of 8.8 GB.
  • BS-05..08 under a two-slot gate (BS-05 with BS-07 admitted): killed past 10 GiB.
  • BS-07 alone: peaks at 4.4 GiB.
  • A one-slot gate, run with BS-06 and BS-08: only one case's bootstrap tree was in /tmp at a time. This confirms admission, though the full cases were not run to completion.

Change. A gate in cases::run admits one of BS-05..08 per 10 GiB of MemTotal. Any host under 20 GiB, including the 16 GiB runners whatever their exact MemTotal, runs these four one at a time. Waiting happens before Timed, so case telemetry excludes it. The admission guard is released on drop, unwinding included.

Scope. This is a per-process planning figure, not a guarantee. It ignores cgroup limits, current pressure, other processes and other cases. Hosts without /proc/meminfo stay unbounded; that means macOS and Windows, where the lanes already run these cases apart. Other cases are unaffected, and there is no SPEC or identity change.

Cost: these four run one after another on aarch64 Linux.

🤖 Generated with Claude Code

https://claude.ai/code/session_013Asa3QVAsEzzaQPMzJnNz1

Each checks its own copy of the bootstrap project; the four together
OOM-killed the 16 GiB aarch64 runner mid cargo test (runner lost with
"shutdown signal"). Measured: BS-07 alone 4.4 GiB, BS-05 with BS-07
past 10 GiB. A process-wide gate in cases::run admits one per 8 GiB of
MemTotal (one on that runner); other cases are unaffected.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_013Asa3QVAsEzzaQPMzJnNz1

@afflom afflom left a comment

Copy link
Copy Markdown
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Bounded source review: the admission guard spans case dispatch and drops on normal unwinding; Condvar waits in a loop and releases the mutex, and no case executes under that mutex. Timed is created after admission and drops before the slot is released, matching the intended exclusion of queue wait. No concrete deadlock, panic-release or count-accounting defect found.

Please make the resource claim precise using the existing measurement receipts. capacity reads host MemTotal; exactly16777216 KiB computes two slots, while8388608..16777215 computes one. The claimed one-slot behavior on the nominal16GiB runner therefore depends on its actual MemTotal. Record that value/computed slot count from the existing investigation rather than repeating the experiment merely for this review.

The8GiB divisor is a per-process planning heuristic, not a measured peak/minimum or memory guarantee. It does not include cgroup limits, current pressure, other processes or other cases; unavailable meminfo leaves this gate unbounded. Document that scope. Distinguish the owner-reported local OOM reproduction from the historical hosted-runner shutdown cause: retained548 logs establish shutdown but no OOM diagnostic. Link the existing reproduction/validation evidence, including contention/unwind or boundary coverage if already available; no new run or probe is requested here.

Reuse current qualification and prior548 diagnosis. Full final-head/actual-tree acceptance remains required, and the separate431/oldTC03 STOP conflict in externally merged551 is unchanged. Receipt5e0b1d8158bdab397b72684112335fe0f752e703f79355f9b09eea5a8a314ab5. COMMENT only.

…'s scope

A 16 GiB host then runs BS-05..BS-08 one at a time whatever its exact
MemTotal (review: 8 GiB gave two slots at exactly 16 GiB).

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_013Asa3QVAsEzzaQPMzJnNz1

afflom commented Nov 15, 2026

Copy link
Copy Markdown
Member Author

Addressed in 3ccd3fa. The budget is now 10 GiB, so any host under 20 GiB gets exactly one slot and the MemTotal boundary no longer matters for 16 GiB runners. The local host's MemTotal (16473776 KiB) gave one slot under the old 8 GiB divisor too. The doc comment now states the scope: per process, MemTotal only, and unbounded without /proc/meminfo. The PR body separates the hosted shutdowns, which have no OOM diagnostic, from the local cgroup OOM kills, and lists what was and wasn't run to completion.


Generated by Claude Code

@afflom afflom mentioned this pull request Nov 15, 2026

@afflom afflom left a comment

Copy link
Copy Markdown
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Bounded3ccd follow-up credits the resource-scope repair. The10GiB divisor gives one slot for the reported MemTotal16473776KiB and for exact16GiB. Mutex/Condvar/guard/telemetry paths are unchanged from the prior review; no new blocking code defect found.

The body now distinguishes local OOM observations from inferred historical hosted-runner cause, states the omitted cgroup/current-pressure/other-process accounting, and correctly says the admission trial did not complete the full cases. These remain owner-reported measurements, while10GiB remains a planning heuristic rather than a peak/minimum/guarantee.

Minor wording correction: BUDGET_KIB's 'more than a measured pair' is unsupported by the adjacent pair result 'killed past10GiB'. Describe the intended one-slot threshold below20GiB instead; no new experiment is needed for that correction.

Current43745407916 qualification is incomplete; exact repro passed. The canceled old1d5c CI cannot qualify3ccd. Reuse prior source credit and existing current gate; no probe/test/retry requested. Receipt190885cabded61715039a80256d2ac9249d8100bced719935459fd024aefbcff. COMMENT only; full acceptance and standing431/oldTC03 STOP remain separate.

afflom commented Nov 15, 2026

Copy link
Copy Markdown
Member Author

Agreed on the BUDGET_KIB wording: "more than a measured pair" isn't supported by the pair result. It should say the 10 GiB figure is chosen to give one slot below 20 GiB. That's a doc-only change, so I'll include it in the next push to this branch rather than restart the gate for it.


Generated by Claude Code

afflom commented Nov 15, 2026

Copy link
Copy Markdown
Member Author

atlas (orbit) failed on 3ccd3fa because the job lost its runner: "The hosted runner lost communication with the server", with no step past "Set up job" and no logs. The atlas shards don't run conformance cases, so this diff doesn't touch them. I'll re-run the failed job once, when run 43745407916 completes.


Generated by Claude Code

@afflom afflom left a comment

Copy link
Copy Markdown
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Current-gate credit, with its actual scope: CI43745407916 attempt2 has66 successful jobs and791 successful steps, plus3 skipped Intel jobs. Orbit and Atlas join now pass; repro43745407863 passed1job/7steps. The skips match the workflow's selected PR plan, so this is evidence for that plan, not completed all-platform qualification.

Existing owner: supply applicable existing Intel qualification bound to this composition, or explicitly disposition the remaining qualification scope before acceptance. This is not an automatic retry request. Prior source credit remains; no repeat of the memory experiment or whole gate is requested by this review.

Preserve attempt1 separately:64successful jobs,1failure,4skips. Its orbit metadata marks “Run the shard” in progress; API metadata does not establish that no substantive step began, even though the owner reported communication loss. No causal OOM claim or log download was added here.

Receipt pr554-3ccd-final-gate-audit-20261115.json SHA256e8fc74e706b4ac24b370b503de626389c27c01944d133e625fe93f88338a3b4f. No canceled/skipped-job aggregation, merge, writer, retry or CI control. #431/oldTC03 STOP and other protected boundaries remain unchanged.

afflom commented Nov 15, 2026 •

Copy link
Copy Markdown
Member Author

On the remaining Intel scope: this falls under the owner's 2026-11-14 decision on #535. Ordinary ready PRs into a release line skip the Intel jobs, and the merge push runs honesty plus Intel on the merged tree. Release evidence still needs all four hosts in one complete run.

This diff leaves Intel behaviour unchanged. macOS has no /proc/meminfo, so the gate admits every case there, exactly as before. The Intel lanes already run BS-05..08 apart.

No PR-level Intel run is planned. Intel qualification for this composition comes from the line push after merge.


Generated by Claude Code

@afflom
afflom merged commit 949cb26 into release/v0.4 Nov 15, 2026
138 of 139 checks passed
@afflom
afflom deleted the ci/aarch64-heavy-cases branch November 15, 2026 23:15
afflom pushed a commit that referenced this pull request Nov 15, 2026
…krpx

Generated manifests taken from the release side; the regeneration rewrites them.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_013Asa3MAztfvN671cYrybev

@afflom afflom left a comment

Copy link
Copy Markdown
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

External landing audit records #550 actualabdcfde (parentsfa271+8154) and #554 actual949cb261 (parentsabdcfde+3ccd). The memory-gate source blob357b62d1 equals reviewed3ccd, and workflow blobffa403c1 equals reviewed #5508154. Whole actual trees differ from the respective heads through composition; source preservation is not whole-tree acceptance.

Prior #554 attempt2 retains66successfuljobs/791steps plus3IntelSKIP. #550 head CI/repro now report success, with Intel skipped in its check records. Actual949 reproducibility reports success, but actual CI43773834816 is pending with zero jobs at inspection; abdc CI43773792326 was canceled. These records do not establish full actual-composition acceptance. Preserve all attempts separately, and use the existing actual gate plus applicable input-bound Intel evidence/scope disposition; no automatic retry or competing gate.

#557's bounded two-file documentation repair now resolves the unsupported measured-pair wording at source level (review5913700009); the divisor is a planning threshold. Existing #431/oldTC03 STOP conflict and all #316/R25 holds remain. Receipt /workspace/work/pr554-pr550-external-landings-20261116.json; SHA2564190a5f38a48081204fa9d45b32dd89683fb766b95b25360a23664c213ad1b01. No root merge, adoption or CI control.

afflom added a commit that referenced this pull request Nov 16, 2026
main at v0.4.3 (fa27108). Tree verified whole by ci 43709771141 and 43744252595; aarch64 runner loss on this PR's run is the pre-#554 OOM.

@afflom afflom left a comment

Copy link
Copy Markdown
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

New diagnosis for the actual merged tree 949cb26 in CI 43773834816 attempt 1: Intel print job 133075444239 passed all 20 tests with zero failures, then creation of the intel-print artifact failed with ENOTFOUND. Intel join 133094843780 subsequently found construction/lower/validate/moves artifacts but no print artifact, and correctly failed its inventory and execution-result guards. Both logs bind checkout to actual 949 and are retained once.

This is an observed artifact-delivery failure after successful print execution. The logs do not show a conformance assertion failure or establish an OOM/hardware cause. Preserve the successful execution and failed upload separately. Existing owner: disposition the missing exact-input print artifact and resulting Intel qualification; if retained evidence permits recovery, reuse it without weakening the inventory guard. No blanket test rerun, artifact reconstruction, retry or producer execution is requested here.

The actual gate remains active, and Intel qualification remains failed/incomplete. Prior source and #557 documentation credit remain intact; no full acceptance or STOP waiver follows. Receipt: /workspace/work/pr949-intel-new-failures-20261116.json; SHA256 4aec115b26dcc63b8d62e53ba67c29401f0a05b7471c6d893f40f510e6013e3b.

humuhumu33 added a commit that referenced this pull request Nov 16, 2026
…arrays branch

PF-26 and the arrays both move the language-1.2 identity from the 0.4.3 base;
the combined tree's ID, b78cb38d…, replaces each side's in the constant, the
locks and the records, and the register is 499 IDs: PF-26 joined the line as
its 498th, SM-39 is the 499th. A conflicted generated projection takes the
line's side, and one only this branch named is dropped, for regeneration to
write.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants