You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
{{ message }}
Repository navigation
The Intel macOS host leg has crept from ~60 to 100+ minutes; bring its cost down #235
The build and non-Lean tests (x86_64-apple-darwin) leg of ci.yml took 74, 92, and 100 minutes on the three full gates of 2026-10-19/20 (runs 39368166156, 39353454952, 39316818883). On run 39382888986 (PR #226, head f6e43dd) the 430 conformance tests passed in 5,945 s and the job was cancelled at its 120-minute limit during the last test binaries. The limit's comment said the leg takes about 60 minutes after #190's split, so the leg has doubled since.
#226 raises the limit to 240 minutes so a cancelled leg stops reading as a result about the code. That is the bound; this issue is the cost.
Where the time goes
From the timestamps of the ok lines on that run (tests run in parallel, so these are lower bounds on the longest case in each stretch):
Stretch ends at
Gap
conformance_tc_05
1,015 s
conformance_bs_03
513 s
conformance_sp_09
409 s
conformance_sp_10
377 s
conformance_tc_03
254 s
conformance_rb_06
240 s
conformance_sp_03
217 s
conformance_sp_08
163 s
The cases that run a rustc differential on every host (the SP production cases, BS-03/BS-05 bootstrap roots, the TC cost rows, the PD-08 stack-bounds scaling on parseAuditOutput at 250..4000 records) are where the Intel runner is slowest.
A newer measurement, and where the tail is (run 39679288933, PR #248, head 677e88d, attempt 1).
The x86_64-apple-darwin leg took 122 minutes. It was the gate's critical path by 48 minutes: the next longest legs were Windows at 74, aarch64 macOS at 57 and just vv (rest) at 45. Since #190 the leg has taken 68 → 70–100 (2026-10-19/20) → 112 (#250) → 122 (#248) minutes. On the full gates since 10-17, the Intel leg was the longest job on every run.
cargo test runs one test binary at a time, so the formatter binary's 9 minutes come before conformance and add to the leg's time.
In the conformance binary, every case had finished by 23:19:08 except three. The binary ended at 23:36:36, so for its last 17.5 minutes at most three tests were running:
Case
Started (from libtest's 60-second notice)
Ran
conformance_sp_02
about 22:34:56
about 61.7 min
conformance_tc_03
about 23:01:25
about 34.4 min
conformance_tc_05
about 23:02:34
about 30.0 min
libtest starts the cases in name order. The two TC cases are long and sort late, so they start about 87 minutes into the 103-minute binary. That leaves a tail no thread count can shorten. SP-02 alone runs for an hour on this host.
Proposed order, each step measured on this leg before the next:
Extend BS-02's differential runs on every host of the matrix and costs the gate about 140 runner-minutes #190's split to SP-02, TC-03 and TC-05. The host matrix would run their rustc differential on the deterministic vectors, and the scaling, seeded corpora and plants would run once on the normative host. The register statements would say where each runs, as Split BS-01 and BS-02 differential between host matrix and normative host (#190) #192 did for BS-01/BS-02. On this run that would remove the 17.5-minute tail. How much of SP-02's hour moves depends on how much of it is scaling rather than the vector differential, which I have not measured. If most of it moves, the leg drops under the Windows leg (74 min), and Windows then sets the gate.
I would not shard the Intel leg across more jobs. The org's macOS concurrency is capped at 5, and every gate already holds two macOS jobs. More macOS shards would make parallel PRs queue on that cap.
This matters for v0.4: each self-hosting root (BS-06, BS-07 and #267–#276) adds a rustc differential that runs on every host. Without the split, the leg grows with every component. Could #235 move into the v0.4 milestone?
humuhumu33 can take step 1 if you agree. Say which cases you want kept on the host matrix.
One more data point, from #254's landing gate (run 39779897081, head 16ee56c90). Every job but one had finished by 13:52, the last being atlas (join). The x86_64-apple-darwin leg's cargo test ran 12:40–14:38 (118 min), and the merge followed at 14:47. So the landing waited 46 minutes on this leg alone, after the rest of the gate was green.
About a dozen v0.4 landings remain, each with a full gate, so this tail repeats on each of them.
Could you say which cases you'd move to the normative host in step 1 of my earlier comment? I proposed SP-02, TC-03 and TC-05, which were the three still running in the last 17 minutes of #248's leg. Once you name them, I'll open the PR: register statements, ci.yml, the limit's comment with the new measurement, and before/after times on this leg.
I am taking the bounded SP-02 / TC-03 / TC-05 cost correction on a maintenance branch from release/v0.4. The plan keeps the committed vectors and platform-independent assertions on every supported host, runs the full stress families and seeded fuel sweep on the normative Linux host, and renders TC-05's shared fixture modules once instead of once per instance. No build cache, extra macOS shards, or removed acceptance recipe. I will record the before/after Intel measurement before claiming this issue closed. Current baseline: successful run 41478333299 at afe1a00, Intel job 125390659100, about 144 minutes overall and 7,470.84 seconds for its 468 conformance tests; SP-02, TC-03 and TC-05 still form the tail.
Completed by reviewed PR #400, merged into release/v0.4 as 477a508. Actual Intel job time improved from 144m02s (baseline run 41478333299/job 125390659100) to 117m01s (run 41592896657/job 125773640497), 27m01s less (18.76%). All 468 conformance cases passed in 6021.07s versus 7470.84s, 19.41% less. Complete CI passed all 42 jobs, plus honesty and reproducibility.
Full stress remains on normative Linux; every host retains corpus/limit checks and fixed certificate vectors. Shared fixture rendering/parsing now happens once. No timeout, proof or acceptance guarantee was weakened. The actual merge tree e02d1637615872ccf1eff1588cb57b1b9fda3ce6 equals the reviewed clean integration with the independently passing #397 docs. Closing explicitly on this non-default release line; required forward propagation is tracked in #404/#405.
Final acceptance cross-check found one remaining documentation requirement in this issue: the timeout comment in ci.yml must contain the new measured duration. #400 records the actual comparison in its PR, but its committed workflow comment still refers readers to that pending result. Reopening until the comment itself records 117m01s (100m21s conformance), with the unchanged 240-minute bound. This small follow-up will ride in #406's already-required gate rather than starting another full matrix solely for a comment. The optimization and measured passing results from #400 remain valid.
Completed through merged PR #403 on release/v0.4, merge 5a44fe49e5d73569c3bb377bb28b56e463ee0547. The final reviewed head 215a9d591 passed all 43 jobs in CI 41658939956, plus honesty 41658939965 and reproducibility 41658939998. Actual merge tree 88d8b657f1ab621bf736b9d7e5d97fcb9c834e5e exactly equals the reviewed/tested head and synthetic merge tree. Explicitly closing this issue because the landing is on a non-default release branch. The required workflow timing comment now records #400's actual Intel comparison: 117m01s versus 144m02s; 468 conformance cases in 6021.07s versus 7470.84s (19.41% less).
Activity
humuhumu33 commentedon Oct 22, 2026
A newer measurement, and where the tail is (run 39679288933, PR #248, head 677e88d, attempt 1).
The
x86_64-apple-darwinleg took 122 minutes. It was the gate's critical path by 48 minutes: the next longest legs were Windows at 74, aarch64 macOS at 57 andjust vv (rest)at 45. Since #190 the leg has taken 68 → 70–100 (2026-10-19/20) → 112 (#250) → 122 (#248) minutes. On the full gates since 10-17, the Intel leg was the longest job on every run.Inside the leg:
formatter_is_idempotent_on_every_examplebinary (11 tests)cargo testruns one test binary at a time, so the formatter binary's 9 minutes come before conformance and add to the leg's time.In the conformance binary, every case had finished by 23:19:08 except three. The binary ended at 23:36:36, so for its last 17.5 minutes at most three tests were running:
conformance_sp_02conformance_tc_03conformance_tc_05libtest starts the cases in name order. The two
TCcases are long and sort late, so they start about 87 minutes into the 103-minute binary. That leaves a tail no thread count can shorten.SP-02alone runs for an hour on this host.Proposed order, each step measured on this leg before the next:
SP-02,TC-03andTC-05. The host matrix would run theirrustcdifferential on the deterministic vectors, and the scaling, seeded corpora and plants would run once on the normative host. The register statements would say where each runs, as Split BS-01 and BS-02 differential between host matrix and normative host (#190) #192 did for BS-01/BS-02. On this run that would remove the 17.5-minute tail. How much ofSP-02's hour moves depends on how much of it is scaling rather than the vector differential, which I have not measured. If most of it moves, the leg drops under the Windows leg (74 min), and Windows then sets the gate.SP-08..SP-12,BS-03..BS-05,RB-06).I would not shard the Intel leg across more jobs. The org's macOS concurrency is capped at 5, and every gate already holds two macOS jobs. More macOS shards would make parallel PRs queue on that cap.
This matters for v0.4: each self-hosting root (BS-06, BS-07 and #267–#276) adds a
rustcdifferential that runs on every host. Without the split, the leg grows with every component. Could #235 move into the v0.4 milestone?humuhumu33 can take step 1 if you agree. Say which cases you want kept on the host matrix.
humuhumu33 commentedon Oct 22, 2026
One more data point, from #254's landing gate (run 39779897081, head
16ee56c90). Every job but one had finished by 13:52, the last beingatlas (join). Thex86_64-apple-darwinleg'scargo testran 12:40–14:38 (118 min), and the merge followed at 14:47. So the landing waited 46 minutes on this leg alone, after the rest of the gate was green.About a dozen v0.4 landings remain, each with a full gate, so this tail repeats on each of them.
Could you say which cases you'd move to the normative host in step 1 of my earlier comment? I proposed
SP-02,TC-03andTC-05, which were the three still running in the last 17 minutes of #248's leg. Once you name them, I'll open the PR: register statements,ci.yml, the limit's comment with the new measurement, and before/after times on this leg.afflom commentedon Nov 3, 2026
I am taking the bounded SP-02 / TC-03 / TC-05 cost correction on a maintenance branch from release/v0.4. The plan keeps the committed vectors and platform-independent assertions on every supported host, runs the full stress families and seeded fuel sweep on the normative Linux host, and renders TC-05's shared fixture modules once instead of once per instance. No build cache, extra macOS shards, or removed acceptance recipe. I will record the before/after Intel measurement before claiming this issue closed. Current baseline: successful run 41478333299 at afe1a00, Intel job 125390659100, about 144 minutes overall and 7,470.84 seconds for its 468 conformance tests; SP-02, TC-03 and TC-05 still form the tail.
Merge pull request #400 from UOR-Foundation/maintenance/issue-235-hos…
afflom commentedon Nov 3, 2026
Completed by reviewed PR #400, merged into release/v0.4 as 477a508. Actual Intel job time improved from 144m02s (baseline run 41478333299/job 125390659100) to 117m01s (run 41592896657/job 125773640497), 27m01s less (18.76%). All 468 conformance cases passed in 6021.07s versus 7470.84s, 19.41% less. Complete CI passed all 42 jobs, plus honesty and reproducibility.
Full stress remains on normative Linux; every host retains corpus/limit checks and fixed certificate vectors. Shared fixture rendering/parsing now happens once. No timeout, proof or acceptance guarantee was weakened. The actual merge tree e02d1637615872ccf1eff1588cb57b1b9fda3ce6 equals the reviewed clean integration with the independently passing #397 docs. Closing explicitly on this non-default release line; required forward propagation is tracked in #404/#405.
afflom commentedon Nov 3, 2026
Final acceptance cross-check found one remaining documentation requirement in this issue: the timeout comment in ci.yml must contain the new measured duration. #400 records the actual comparison in its PR, but its committed workflow comment still refers readers to that pending result. Reopening until the comment itself records 117m01s (100m21s conformance), with the unchanged 240-minute bound. This small follow-up will ride in #406's already-required gate rather than starting another full matrix solely for a comment. The optimization and measured passing results from #400 remain valid.
Record the measured post-change Intel host runtime (#235)
afflom commentedon Nov 3, 2026
Completed through merged PR #403 on
release/v0.4, merge5a44fe49e5d73569c3bb377bb28b56e463ee0547. The final reviewed head215a9d591passed all 43 jobs in CI 41658939956, plus honesty 41658939965 and reproducibility 41658939998. Actual merge tree88d8b657f1ab621bf736b9d7e5d97fcb9c834e5eexactly equals the reviewed/tested head and synthetic merge tree. Explicitly closing this issue because the landing is on a non-default release branch. The required workflow timing comment now records #400's actual Intel comparison: 117m01s versus 144m02s; 468 conformance cases in 6021.07s versus 7470.84s (19.41% less).Reduce repeated host conformance work for UOR-Foundation#235