Skip to content

The Intel macOS host leg has crept from ~60 to 100+ minutes; bring its cost down #235

Closed

Description

@afflom

What was measured

The build and non-Lean tests (x86_64-apple-darwin) leg of ci.yml took 74, 92, and 100 minutes on the three full gates of 2026-10-19/20 (runs 39368166156, 39353454952, 39316818883). On run 39382888986 (PR #226, head f6e43dd) the 430 conformance tests passed in 5,945 s and the job was cancelled at its 120-minute limit during the last test binaries. The limit's comment said the leg takes about 60 minutes after #190's split, so the leg has doubled since.

#226 raises the limit to 240 minutes so a cancelled leg stops reading as a result about the code. That is the bound; this issue is the cost.

Where the time goes

From the timestamps of the ok lines on that run (tests run in parallel, so these are lower bounds on the longest case in each stretch):

Stretch ends at Gap
conformance_tc_05 1,015 s
conformance_bs_03 513 s
conformance_sp_09 409 s
conformance_sp_10 377 s
conformance_tc_03 254 s
conformance_rb_06 240 s
conformance_sp_03 217 s
conformance_sp_08 163 s

The cases that run a rustc differential on every host (the SP production cases, BS-03/BS-05 bootstrap roots, the TC cost rows, the PD-08 stack-bounds scaling on parseAuditOutput at 250..4000 records) are where the Intel runner is slowest.

What would close it

One of, measured before and after on this leg:

  • Run the host matrix's rustc differential on the deterministic vectors only, once per case, and move the scaling and bootstrap-root builds to the normative host (the split BS-02's differential runs on every host of the matrix and costs the gate about 140 runner-minutes #190 made, extended to the cases added since).
  • Cache the differential crates' build directory per host between cases (the generated crates share dependencies; each case rebuilds them).
  • Build the scaling inputs at one size on the hosts and all four on the normative host.

Whichever it is, the limit's comment in ci.yml is updated with the new measurement so that "twice the slowest observed time" stays true.

Milestone: v0.5.

Activity

  1. humuhumu33 commented on Oct 22, 2026

    @humuhumu33
    Contributor

    A newer measurement, and where the tail is (run 39679288933, PR #248, head 677e88d, attempt 1).

    The x86_64-apple-darwin leg took 122 minutes. It was the gate's critical path by 48 minutes: the next longest legs were Windows at 74, aarch64 macOS at 57 and just vv (rest) at 45. Since #190 the leg has taken 68 → 70–100 (2026-10-19/20) → 112 (#250) → 122 (#248) minutes. On the full gates since 10-17, the Intel leg was the longest job on every run.

    Inside the leg:

    Part Time
    Build about 8 min
    formatter_is_idempotent_on_every_example binary (11 tests) 557 s
    Conformance binary (443 tests) 6,187 s

    cargo test runs one test binary at a time, so the formatter binary's 9 minutes come before conformance and add to the leg's time.

    In the conformance binary, every case had finished by 23:19:08 except three. The binary ended at 23:36:36, so for its last 17.5 minutes at most three tests were running:

    Case Started (from libtest's 60-second notice) Ran
    conformance_sp_02 about 22:34:56 about 61.7 min
    conformance_tc_03 about 23:01:25 about 34.4 min
    conformance_tc_05 about 23:02:34 about 30.0 min

    libtest starts the cases in name order. The two TC cases are long and sort late, so they start about 87 minutes into the 103-minute binary. That leaves a tail no thread count can shorten. SP-02 alone runs for an hour on this host.

    Proposed order, each step measured on this leg before the next:

    1. Extend BS-02's differential runs on every host of the matrix and costs the gate about 140 runner-minutes #190's split to SP-02, TC-03 and TC-05. The host matrix would run their rustc differential on the deterministic vectors, and the scaling, seeded corpora and plants would run once on the normative host. The register statements would say where each runs, as Split BS-01 and BS-02 differential between host matrix and normative host (#190) #192 did for BS-01/BS-02. On this run that would remove the 17.5-minute tail. How much of SP-02's hour moves depends on how much of it is scaling rather than the vector differential, which I have not measured. If most of it moves, the leg drops under the Windows leg (74 min), and Windows then sets the gate.
    2. If the leg is still the longest, apply the same rule to the next cases on The Intel macOS host leg has crept from ~60 to 100+ minutes; bring its cost down #235's list (SP-08..SP-12, BS-03..BS-05, RB-06).
    3. Only then cache the differential crates' build directory per host (The Intel macOS host leg has crept from ~60 to 100+ minutes; bring its cost down #235's second option), since it touches §9.4.

    I would not shard the Intel leg across more jobs. The org's macOS concurrency is capped at 5, and every gate already holds two macOS jobs. More macOS shards would make parallel PRs queue on that cap.

    This matters for v0.4: each self-hosting root (BS-06, BS-07 and #267–#276) adds a rustc differential that runs on every host. Without the split, the leg grows with every component. Could #235 move into the v0.4 milestone?

    humuhumu33 can take step 1 if you agree. Say which cases you want kept on the host matrix.

  2. humuhumu33 commented on Oct 22, 2026

    @humuhumu33
    Contributor

    One more data point, from #254's landing gate (run 39779897081, head 16ee56c90). Every job but one had finished by 13:52, the last being atlas (join). The x86_64-apple-darwin leg's cargo test ran 12:40–14:38 (118 min), and the merge followed at 14:47. So the landing waited 46 minutes on this leg alone, after the rest of the gate was green.

    About a dozen v0.4 landings remain, each with a full gate, so this tail repeats on each of them.

    Could you say which cases you'd move to the normative host in step 1 of my earlier comment? I proposed SP-02, TC-03 and TC-05, which were the three still running in the last 17 minutes of #248's leg. Once you name them, I'll open the PR: register statements, ci.yml, the limit's comment with the new measurement, and before/after times on this leg.

  3. afflom commented on Nov 3, 2026

    @afflom
    MemberAuthor

    I am taking the bounded SP-02 / TC-03 / TC-05 cost correction on a maintenance branch from release/v0.4. The plan keeps the committed vectors and platform-independent assertions on every supported host, runs the full stress families and seeded fuel sweep on the normative Linux host, and renders TC-05's shared fixture modules once instead of once per instance. No build cache, extra macOS shards, or removed acceptance recipe. I will record the before/after Intel measurement before claiming this issue closed. Current baseline: successful run 41478333299 at afe1a00, Intel job 125390659100, about 144 minutes overall and 7,470.84 seconds for its 468 conformance tests; SP-02, TC-03 and TC-05 still form the tail.

  4. added a commit that references this issue on Nov 3, 2026
  5. afflom commented on Nov 3, 2026

    @afflom
    MemberAuthor

    Completed by reviewed PR #400, merged into release/v0.4 as 477a508. Actual Intel job time improved from 144m02s (baseline run 41478333299/job 125390659100) to 117m01s (run 41592896657/job 125773640497), 27m01s less (18.76%). All 468 conformance cases passed in 6021.07s versus 7470.84s, 19.41% less. Complete CI passed all 42 jobs, plus honesty and reproducibility.

    Full stress remains on normative Linux; every host retains corpus/limit checks and fixed certificate vectors. Shared fixture rendering/parsing now happens once. No timeout, proof or acceptance guarantee was weakened. The actual merge tree e02d1637615872ccf1eff1588cb57b1b9fda3ce6 equals the reviewed clean integration with the independently passing #397 docs. Closing explicitly on this non-default release line; required forward propagation is tracked in #404/#405.

  6. afflom commented on Nov 3, 2026

    @afflom
    MemberAuthor

    Final acceptance cross-check found one remaining documentation requirement in this issue: the timeout comment in ci.yml must contain the new measured duration. #400 records the actual comparison in its PR, but its committed workflow comment still refers readers to that pending result. Reopening until the comment itself records 117m01s (100m21s conformance), with the unchanged 240-minute bound. This small follow-up will ride in #406's already-required gate rather than starting another full matrix solely for a comment. The optimization and measured passing results from #400 remain valid.

  7. reopened this on Nov 3, 2026
  8. added this to the LexLean v0.4.1 milestone on Nov 3, 2026
  9. afflom commented on Nov 3, 2026

    @afflom
    MemberAuthor

    Completed through merged PR #403 on release/v0.4, merge 5a44fe49e5d73569c3bb377bb28b56e463ee0547. The final reviewed head 215a9d591 passed all 43 jobs in CI 41658939956, plus honesty 41658939965 and reproducibility 41658939998. Actual merge tree 88d8b657f1ab621bf736b9d7e5d97fcb9c834e5e exactly equals the reviewed/tested head and synthetic merge tree. Explicitly closing this issue because the landing is on a non-default release branch. The required workflow timing comment now records #400's actual Intel comparison: 117m01s versus 144m02s; 468 conformance cases in 6021.07s versus 7470.84s (19.41% less).

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    Type

    No type

    Projects

    No projects

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions