Skip to content

ci: run iOS and macOS smoke on parallel Blacksmith runners - #4418

Merged
thymikee merged 1 commit into
mainfrom
ci/parallel-apple-smoke
Nov 11, 2026
Merged

thymikee merged 1 commit into
mainfrom
ci/parallel-apple-smoke

Conversation

@thymikee

@thymikee thymikee commented Nov 11, 2026 •

Copy link
Copy Markdown
Member

Summary

Run iOS and macOS smoke on independent Blacksmith runners. Preserve affected selection, platform cleanup, diagnostic uploads, draft behavior, and the stable Smoke Tests aggregate check. Three files; no scope expansion or linked issue.

Two controlled comparisons on identical product/test sources:

Layout Trial 1 Trial 2
Combined Apple runner 415s 410s
Parallel completion 242s 228s

Median platform completion improves 43% (412.5 → 235s). Total macOS allocation seconds increase about 11%; the final aggregate adds one small Linux job. No test selection or coverage is reduced.

First comparison, repeat: all six platform/control jobs pass, including source-counted XCTests, fixture E2E, replays, and the iOS focus canary.

Validation

Tested 03c0f8d8a60a8e8d141952f99d3aaa8aeda1641e:

  • pnpm check:affected --run --head 03c0f8d8a60a8e8d141952f99d3aaa8aeda1641e: 42 passed, 25 skipped. Receipt run-G3KPaK: exactCommit=true, checkoutChanged=false.
  • Focused Apple tests: 51 passed; gate-manifest models: 53 passed. Independent review finding fixed.
  • Regression proof: old workflow fails all 7 selected contracts. Removing aggregate failure propagation fails 3/4 status cases; restored head passes all 7.

An earlier cold-preparation assertion did not reproduce on unchanged base or the final head run; cause remains unproven. Native PR checks remain required and run when marked ready.

@github-actions

Copy link
Copy Markdown
Contributor

Size Report

Metric Base Current Diff
Installed (including dependencies) 6.21 MB 6.21 MB 0 B
Package (unpacked) 6.21 MB 6.21 MB 0 B
Package (download) 1.88 MB 1.88 MB -15 B

Startup median (7 runs, lower is better):

Scenario Base Current Diff
CLI --version 35.5 ms 26.8 ms -8.6 ms
CLI --help 76.3 ms 50.9 ms -25.4 ms

@thymikee
thymikee marked this pull request as ready for review November 11, 2026 11:11

@cubic-dev-ai cubic-dev-ai Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

1 issue found across 3 files

Prompt for AI agents (unresolved issues)

Check if these issues are valid — if so, understand the root cause of each and fix them. When an issue isn't valid or won't be fixed in this PR, reply in its thread with the reason and then resolve the thread. If appropriate, use sub-agents to investigate and fix each issue separately.


<file name="scripts/gate/workflows.test.ts">

<violation number="1" location="scripts/gate/workflows.test.ts:349">
P2: This test checks the aggregate's `needs` and `if`, but not the step that fails when the platform job did not succeed. Because the aggregate runs under `always()`, dropping that `test "$RESULT" = success` step would make the stable `Smoke Tests` check pass after an iOS or macOS failure. Assert that a step checks `needs.smoke-apple-platform.result` for `success`.</violation>
</file>

Reply with feedback, questions, or to request a fix.

View guided diff | Turn on auto-fix | Re-trigger cubic

assert.equal(job.if, '${{ !github.event.pull_request.draft }}', workflow.name);
if (workflow.name === 'Apple' && job.name === 'Smoke Tests') {
assert.equal(job.needs, 'smoke-apple-platform');
assert.equal(job.if, '${{ always() && !github.event.pull_request.draft }}');

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

P2: This test checks the aggregate's needs and if, but not the step that fails when the platform job did not succeed. Because the aggregate runs under always(), dropping that test "$RESULT" = success step would make the stable Smoke Tests check pass after an iOS or macOS failure. Assert that a step checks needs.smoke-apple-platform.result for success.

Prompt for AI agents
Check if this issue is valid — if so, understand the root cause and fix it. When an issue isn't valid or won't be fixed in this PR, reply in its thread with the reason and then resolve the thread. At scripts/gate/workflows.test.ts, line 349:

<comment>This test checks the aggregate's `needs` and `if`, but not the step that fails when the platform job did not succeed. Because the aggregate runs under `always()`, dropping that `test "$RESULT" = success` step would make the stable `Smoke Tests` check pass after an iOS or macOS failure. Assert that a step checks `needs.smoke-apple-platform.result` for `success`.</comment>

<file context>
@@ -343,6 +344,11 @@ test('device smoke lanes skip drafts and start when a draft is marked ready', ()
-    assert.equal(job.if, '${{ !github.event.pull_request.draft }}', workflow.name);
+    if (workflow.name === 'Apple' && job.name === 'Smoke Tests') {
+      assert.equal(job.needs, 'smoke-apple-platform');
+      assert.equal(job.if, '${{ always() && !github.event.pull_request.draft }}');
+    } else {
+      assert.equal(job.if, '${{ !github.event.pull_request.draft }}', workflow.name);
</file context>

@apex-sentinel-staging

Copy link
Copy Markdown

Apex Sentinel

Verdict: approve

1 finding(s): 1 minor.

🟡 Minor (1)

Where Finding
.github/workflows/apple.yml:36 90-minute job ceiling leaves no room for the 80-minute phase step plus cleanup and upload

Reviewed with this repository's own Sentinel configuration (3924da9474ed).

Feedback goes in the review comment threads, as a reply under the inline comment of a finding:

Reply @apex-sentinel-staging note ... to add context, or @apex-sentinel-staging pivot ... if it is wrong — I will re-check the cited code.

Useful? Reply @apex-sentinel-staging like, or @apex-sentinel-staging dislike <why> if it is not.

Comment @apex-sentinel-staging review on this conversation to run a full review.

@apex-sentinel-staging apex-sentinel-staging Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Apex Sentinel: 1 new finding on 03c0f8d. The verdict and the full list are in the summary comment.

Comment on lines 36 to +37
runs-on: blacksmith-6vcpu-macos-26
# Keep each platform's former 80-minute ceiling, plus shared setup and cleanup.
timeout-minutes: 170
timeout-minutes: 90

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🟡 Minor · 90-minute job ceiling leaves no room for the 80-minute phase step plus cleanup and upload

The 90-minute job ceiling leaves only ~10 minutes around an 80-minute phase timeout, so Clean the iOS host, Require the selected Apple phases to succeed, and Upload Apple artifacts (if: always()) never execute when the phase hits its ceiling — the PR gets a bare cancellation with no diagnostics. The predecessor used 170 minutes for the same reason: the job budget must exceed the phase ceiling plus setup, cleanup, and upload.

Suggested change
runs-on: blacksmith-6vcpu-macos-26
# Keep each platform's former 80-minute ceiling, plus shared setup and cleanup.
timeout-minutes: 170
timeout-minutes: 90
runs-on: blacksmith-6vcpu-macos-26
# Keep the platform's 80-minute phase ceiling, plus setup, build, cleanup and upload.
timeout-minutes: 110
Details

Also raised here

Also raised at this location by persona senior-developer: Apple job ceiling of 90 min no longer leaves room for setup, cleanup and uploads — An iOS phase that hangs to its own 80-minute step ceiling (line 90) after a cold toolchain install and build exceeding ~10 minutes now hits the job-level 90-minute ceiling instead: GitHub cancels the whole job, so Clean the iOS host, the phase-outcome guard and Upload Apple artifacts never run and the diagnostics that step exists to produce are lost, with the check reported as cancelled rather than a diagnosable failure. The predecessor set 170 minutes with the comment "Keep each platform's former 80-minute ceiling, plus shared setup and cleanup", and this diff keeps the unchanged 80-minute per-phase ceilings while cutting the job budget to 90, so the headroom that comment reserved for setup, cleanup and upload is gone. Splitting the platforms removed the need for 2×80 plus shared setup, but 90 is less than 80 plus the setup/build/cleanup/upload time the same runner still performs.

persona: javascript, senior-developer

Useful? Reply @apex-sentinel-staging like, or @apex-sentinel-staging dislike <why> if it is not.

Reply @apex-sentinel-staging note ... to add context, or @apex-sentinel-staging pivot ... if it is wrong — I will re-check the cited code.

@thymikee

Copy link
Copy Markdown
Member Author

This PR is ready at 03c0f8d, and I found nothing that must change before merge. All 23 checks are passing or skipping, including the head's Apple run, where Smoke Tests (ios) passed in 4m16s, Smoke Tests (macos) in 4m12s and the Linux aggregate in 4s. The skipping check named Smoke Tests (${{ matrix.platform }}) comes from the earlier draft run and is not a failure. There are no conflicts.

Not blocking: apple.yml:37 sets the job timeout to 90 minutes around a phase step that keeps its own 80-minute ceiling, so a hung phase after a slow cold setup would cancel the job before cleanup and artifact upload run. Setting the job ceiling above the step ceiling plus setup and teardown, with one line saying why, would cover it. Separately, each matrix leg runs checkout, toolchain setup and both select steps on a macOS runner before it learns whether its platform is selected. Passing the target through the matrix with one impact step id would remove the echo run=false shim and one select step. You can take or leave both.

On the open threads: the thread on the job timeout (#4418 (comment)) still applies. The thread on the aggregate check (#4418 (comment)) does not apply, because scripts/__tests__/apple-ci-impact.test.ts lines 108-120 already assert the RESULT binding and run the real aggregate step for success, failure, cancelled and skipped, so you can resolve it.

I did not re-measure the timing claim or run a failing platform leg. Failure propagation here rests on the GitHub needs.result semantics and the executed bash tests. Once you have read the two notes above, nothing else stands between this PR and merge.

@thymikee thymikee added the ready-for-human Valid work that needs human implementation, judgment, or maintainer merge label Nov 11, 2026
@thymikee
thymikee merged commit 7d9be67 into main Nov 11, 2026
23 checks passed
@thymikee
thymikee deleted the ci/parallel-apple-smoke branch November 11, 2026 11:53
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

ready-for-human Valid work that needs human implementation, judgment, or maintainer merge

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant