Repository navigation
ci: run iOS and macOS smoke on parallel Blacksmith runners - #4418
Conversation
Size Report
Startup median (7 runs, lower is better):
|
There was a problem hiding this comment.
1 issue found across 3 files
Prompt for AI agents (unresolved issues)
Check if these issues are valid — if so, understand the root cause of each and fix them. When an issue isn't valid or won't be fixed in this PR, reply in its thread with the reason and then resolve the thread. If appropriate, use sub-agents to investigate and fix each issue separately.
<file name="scripts/gate/workflows.test.ts">
<violation number="1" location="scripts/gate/workflows.test.ts:349">
P2: This test checks the aggregate's `needs` and `if`, but not the step that fails when the platform job did not succeed. Because the aggregate runs under `always()`, dropping that `test "$RESULT" = success` step would make the stable `Smoke Tests` check pass after an iOS or macOS failure. Assert that a step checks `needs.smoke-apple-platform.result` for `success`.</violation>
</file>
Reply with feedback, questions, or to request a fix.
View guided diff | Turn on auto-fix | Re-trigger cubic
| assert.equal(job.if, '${{ !github.event.pull_request.draft }}', workflow.name); | ||
| if (workflow.name === 'Apple' && job.name === 'Smoke Tests') { | ||
| assert.equal(job.needs, 'smoke-apple-platform'); | ||
| assert.equal(job.if, '${{ always() && !github.event.pull_request.draft }}'); |
There was a problem hiding this comment.
P2: This test checks the aggregate's needs and if, but not the step that fails when the platform job did not succeed. Because the aggregate runs under always(), dropping that test "$RESULT" = success step would make the stable Smoke Tests check pass after an iOS or macOS failure. Assert that a step checks needs.smoke-apple-platform.result for success.
Prompt for AI agents
Check if this issue is valid — if so, understand the root cause and fix it. When an issue isn't valid or won't be fixed in this PR, reply in its thread with the reason and then resolve the thread. At scripts/gate/workflows.test.ts, line 349:
<comment>This test checks the aggregate's `needs` and `if`, but not the step that fails when the platform job did not succeed. Because the aggregate runs under `always()`, dropping that `test "$RESULT" = success` step would make the stable `Smoke Tests` check pass after an iOS or macOS failure. Assert that a step checks `needs.smoke-apple-platform.result` for `success`.</comment>
<file context>
@@ -343,6 +344,11 @@ test('device smoke lanes skip drafts and start when a draft is marked ready', ()
- assert.equal(job.if, '${{ !github.event.pull_request.draft }}', workflow.name);
+ if (workflow.name === 'Apple' && job.name === 'Smoke Tests') {
+ assert.equal(job.needs, 'smoke-apple-platform');
+ assert.equal(job.if, '${{ always() && !github.event.pull_request.draft }}');
+ } else {
+ assert.equal(job.if, '${{ !github.event.pull_request.draft }}', workflow.name);
</file context>
Apex SentinelVerdict: 1 finding(s): 1 minor. 🟡 Minor (1)
Reviewed with this repository's own Sentinel configuration ( Feedback goes in the review comment threads, as a reply under the inline comment of a finding: Reply Useful? Reply Comment |
There was a problem hiding this comment.
Apex Sentinel: 1 new finding on 03c0f8d. The verdict and the full list are in the summary comment.
| runs-on: blacksmith-6vcpu-macos-26 | ||
| # Keep each platform's former 80-minute ceiling, plus shared setup and cleanup. | ||
| timeout-minutes: 170 | ||
| timeout-minutes: 90 |
There was a problem hiding this comment.
🟡 Minor · 90-minute job ceiling leaves no room for the 80-minute phase step plus cleanup and upload
The 90-minute job ceiling leaves only ~10 minutes around an 80-minute phase timeout, so Clean the iOS host, Require the selected Apple phases to succeed, and Upload Apple artifacts (if: always()) never execute when the phase hits its ceiling — the PR gets a bare cancellation with no diagnostics. The predecessor used 170 minutes for the same reason: the job budget must exceed the phase ceiling plus setup, cleanup, and upload.
| runs-on: blacksmith-6vcpu-macos-26 | |
| # Keep each platform's former 80-minute ceiling, plus shared setup and cleanup. | |
| timeout-minutes: 170 | |
| timeout-minutes: 90 | |
| runs-on: blacksmith-6vcpu-macos-26 | |
| # Keep the platform's 80-minute phase ceiling, plus setup, build, cleanup and upload. | |
| timeout-minutes: 110 |
Details
Also raised here
Also raised at this location by persona senior-developer: Apple job ceiling of 90 min no longer leaves room for setup, cleanup and uploads — An iOS phase that hangs to its own 80-minute step ceiling (line 90) after a cold toolchain install and build exceeding ~10 minutes now hits the job-level 90-minute ceiling instead: GitHub cancels the whole job, so Clean the iOS host, the phase-outcome guard and Upload Apple artifacts never run and the diagnostics that step exists to produce are lost, with the check reported as cancelled rather than a diagnosable failure. The predecessor set 170 minutes with the comment "Keep each platform's former 80-minute ceiling, plus shared setup and cleanup", and this diff keeps the unchanged 80-minute per-phase ceilings while cutting the job budget to 90, so the headroom that comment reserved for setup, cleanup and upload is gone. Splitting the platforms removed the need for 2×80 plus shared setup, but 90 is less than 80 plus the setup/build/cleanup/upload time the same runner still performs.
persona: javascript, senior-developer
Useful? Reply @apex-sentinel-staging like, or @apex-sentinel-staging dislike <why> if it is not.
Reply @apex-sentinel-staging note ... to add context, or @apex-sentinel-staging pivot ... if it is wrong — I will re-check the cited code.
|
This PR is ready at 03c0f8d, and I found nothing that must change before merge. All 23 checks are passing or skipping, including the head's Apple run, where Smoke Tests (ios) passed in 4m16s, Smoke Tests (macos) in 4m12s and the Linux aggregate in 4s. The skipping check named Not blocking: apple.yml:37 sets the job timeout to 90 minutes around a phase step that keeps its own 80-minute ceiling, so a hung phase after a slow cold setup would cancel the job before cleanup and artifact upload run. Setting the job ceiling above the step ceiling plus setup and teardown, with one line saying why, would cover it. Separately, each matrix leg runs checkout, toolchain setup and both select steps on a macOS runner before it learns whether its platform is selected. Passing the target through the matrix with one On the open threads: the thread on the job timeout (#4418 (comment)) still applies. The thread on the aggregate check (#4418 (comment)) does not apply, because I did not re-measure the timing claim or run a failing platform leg. Failure propagation here rests on the GitHub |
Summary
Run iOS and macOS smoke on independent Blacksmith runners. Preserve affected selection, platform cleanup, diagnostic uploads, draft behavior, and the stable
Smoke Testsaggregate check. Three files; no scope expansion or linked issue.Two controlled comparisons on identical product/test sources:
Median platform completion improves 43% (412.5 → 235s). Total macOS allocation seconds increase about 11%; the final aggregate adds one small Linux job. No test selection or coverage is reduced.
First comparison, repeat: all six platform/control jobs pass, including source-counted XCTests, fixture E2E, replays, and the iOS focus canary.
Validation
Tested
03c0f8d8a60a8e8d141952f99d3aaa8aeda1641e:pnpm check:affected --run --head 03c0f8d8a60a8e8d141952f99d3aaa8aeda1641e: 42 passed, 25 skipped. Receiptrun-G3KPaK:exactCommit=true,checkoutChanged=false.An earlier cold-preparation assertion did not reproduce on unchanged base or the final head run; cause remains unproven. Native PR checks remain required and run when marked ready.