Skip to content

Time measures no longer depend on the runner; one file per oracle source; HW-01's tests run the producers twice - #179

Merged
github-actions[bot] merged 8 commits into
mainfrom
claude/project-thread-jbqzew
Nov 16, 2026
Merged

github-actions[bot] merged 8 commits into
mainfrom
claude/project-thread-jbqzew

Conversation

@afflom

@afflom afflom commented Nov 16, 2026 •

Copy link
Copy Markdown
Owner

Time measures (QR-03, QR-05) no longer depend on the runner

The labeling time (labeling-ms-per-100, at most 250) and the restart time (restart-ms, at most 3000) are wall-clock measures. On GitHub's runners today QR-03 read 275, 330 and, with #174's middle of three, still 257; QR-05 read 3181, 2590, 3157 and 2631, on pull requests that touch neither. The budgets sat inside the runners' own spread, so the machine decided merges. Two causes, two changes:

They were timed while every other suite ran beside them. The page check and the train ran all 31 *.test.mjs files in one node --test, side by side on four cores, browsers included. Now the suites that the product record names as gates of its time behaviour measures (today test/router/labels.test.mjs and test/product/start.test.mjs, read from app/specs/product/spec.json, not listed by hand) run one at a time and apart from the rest. This is in the models app/specs/pipeline/workflows/pages.yml.json and merge-train.yml.json, rendered again.

QR-03 was in milliseconds of whatever machine ran it. The model (app/specs/product/quality/QR-03.json) gives the measure a pace: a fixed piece of work over the same items, and the time it takes on the machine class the budget was set on (520 ms, measured here over nine runs: 494 to 604). test/router/labels.test.mjs times that work in the same process straight after the labeling and states the labeling time at that pace. Threshold and middle of three unchanged. No retry.

What the evidence says so far: here QR-03 reads 177, 208, 203 idle and 210, 206, 234 with eight busy loops on four cores (unscaled about 450). On the runner, the previous head (80c696b, pace only, suites still side by side) read 246, and QR-05 failed there at 3149. So the pace alone is not enough, which is why the suites are now also run alone. This head's page check is the first runner reading with both; I will read the two figures there and say them here.

One file per oracle source

Every pull request that pins an oracle source edited the one sources list in app/specs/oracle/spec.json, so two such pull requests conflict, and a clean textual merge of that list can still be a wrong one. This is the class #164 closed for guests and #174 for standards.

The file had a second problem: generators/person's icfmodule step sends the whole of it in one request, and on main that request was within about 1 KB of the 32 KiB bound, so the next source or gap would have been refused by GN-01. spec.json goes from 29,917 to 12,331 bytes.

  • Each source is app/specs/oracle/sources/<key>.json (19 files). spec.json keeps everything else and says "sources": {"parts": "sources"}.
  • tools/oracle/gen.py and tools/oracle/vcard.py read the sources from that folder; tools/standards/gate.py reads them through the loader the product and standards records already use (tools/specs/record.py).
  • crates/conformance/tests/oracle.rs (OR-01) counts the pinned sources as the files in that folder.
  • Sources are now read in key order, so test/oracle/corpus.json, model/authorities.toml and CONFORMANCE.md are regenerated in that order: the same 6,303 examples and the same authorities, reordered.
  • tools/standards/gate.py no longer falls back to the base's whole debt.json, which main no longer has since One file per standard and its debt; the train runs the browser acceptances once per batch; QR-03 takes the middle of three #174.

To pin an oracle source now: add app/specs/oracle/sources/<key>.json. An open pull request that added to the old list moves its entry into that folder when it next merges main.

HW-01's tests run the producers twice, not six times

The page check was cancelled at its 75-minute limit on #162 with nothing failing; the three tests of test/handwritten/gate.test.mjs were 60 percent of a 19-minute pass, six producer runs in all.

  • tools/handwritten/gate.py keeps what the producers claimed under target/, named by a hash of what they read (every measured file's bytes and the spec's producers and built entries), and only from a run in which every producer passed. A fresh runner has nothing kept, so the gate's own verdict on a pull request is measured as before.
  • The two planted-defect tests are one test with one run; each refusal is still asserted by its own message.

Here the test file took 516 s for two producer runs. On the runner, one pass of all suites on head 80c696b took 309 s.

The train's fast gates

tools/parts/gate.py and tools/product/gate.py no longer exist (#142), so the two guarded lines that ran them are removed from the train's model.

Checks run here

  • tools/generators/run.sh --check (GN-01) on this head
  • tools/handwritten/gate.py --check --base origin/main on the QR-03 commit (25,523 hand-written lines, none added or grown); the commits after it change only models and rendered workflows
  • tools/standards/gate.py --check --base origin/main (ST-01, as on main), before the HW-01 and QR-03 commits
  • node --test test/handwritten/gate.test.mjs: 2 pass; node --test test/specs/*.test.mjs: 5 pass; node test/router/labels.test.mjs passes
  • cargo test -p repo-conformance --test oracle --test person; cargo fmt --check
  • The suite selection run in bash here: 2 timed, 29 others of 31; both rendered workflows parse as YAML

Not run here: the browser acceptances (so QR-05), the full conformance suite, and the new test step as a whole; this pull request's own page check runs that step from this head.

🤖 Generated with Claude Code

https://claude.ai/code/session_013Asa3LuNYtPDBa4FbSEQUZ

…n programs now

Co-Authored-By: Claude Fable 5.2 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_013Asa3LuNYtPDBa4FbSEQUZ
… defects go in one run

Co-Authored-By: Claude Fable 5.2 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_013Asa3LuNYtPDBa4FbSEQUZ
Co-Authored-By: Claude Fable 5.2 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_013Asa3LuNYtPDBa4FbSEQUZ
Co-Authored-By: Claude Fable 5.2 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_013Asa3LuNYtPDBa4FbSEQUZ
@afflom afflom changed the title One file per oracle source; the train drops two gates that are LexLean programs now One file per oracle source; HW-01's tests run the producers twice, not six times Nov 16, 2026
@github-actions

Copy link
Copy Markdown
Contributor

Merge train notice: the merge train merges this pull request once it is green on current main; don't enable auto-merge.


Generated by Claude Code

@afflom afflom changed the title One file per oracle source; HW-01's tests run the producers twice, not six times QR-03 is stated against a pace timed in the same process; one file per oracle source; HW-01's tests run the producers twice Nov 16, 2026
@github-actions

Copy link
Copy Markdown
Contributor

Merge train: 80c696b: not merged, because its check "page / gates" ended failure on this head, so it is not green; push to this branch to run it again. Fix it on this branch (merge main first) and the train tests it again.


Generated by Claude Code

…one at a time, apart from the rest

Co-Authored-By: Claude Fable 5.2 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_013Asa3LuNYtPDBa4FbSEQUZ
@afflom afflom changed the title QR-03 is stated against a pace timed in the same process; one file per oracle source; HW-01's tests run the producers twice Time measures no longer depend on the runner; one file per oracle source; HW-01's tests run the producers twice Nov 16, 2026
@github-actions
github-actions Bot merged commit a507f82 into main Nov 16, 2026
7 checks passed
afflom pushed a commit that referenced this pull request Nov 16, 2026
Co-Authored-By: Claude Fable 5.2 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_013Asa3NKtifq75V8QbFCACT
@github-actions

Copy link
Copy Markdown
Contributor

Live: this change is on the published page, as main a507f8268a8fbf91964130677f068e468f1705f1.

The publish job opened the published page in a fresh Chromium and its live checks passed (PG-01, PD-01, PD-02): https://github.com/afflom/me/actions/runs/43919204957

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants