Repository navigation
ci(e2e): split the checker journeys, reuse check's builds and capture traces from the collector - #291
Closed
systemfsoftware-maker wants to merge 24 commits into
Closed
systemfsoftware-maker wants to merge 24 commits into
systemfsoftware-maker wants to merge 24 commits into
Conversation
…ry check job `check` restored `turbo-Linux-dist-<sha>` through prefix restore-keys, so a PR got the newest entry its ref could read rather than its base commit's. #269 (one changeset file) restored an older main entry and re-ran 53 of 112 tasks; turbo's own dry-run hashes are identical between that base and head, so the key was never invalidated, the restored entry was simply another commit's. - check restores `turbo-<os>-dist-<base>` in its own step (first parent, or merge-base with main on dispatch), and falls back to the prefix only on a miss; actions/cache searches the current ref before main, so one step with restore-keys would still prefer this ref's stale entry. - Turbo writes run summaries (TURBO_RUN_SUMMARY); the cache is pruned to this run's task hashes and saved only when the Gate passed and missed a task. - cache-sweep (main pushes only, the one job with actions: write) deletes turbo entries by cache id: main keeps its newest three, other refs one, closed pull requests none. - .github/actions/job-report writes each job's summary (duration, Turbo cached/total, slowest missed tasks, restored key) and, on failure, an annotation titled with the reason code: GATE_OK, TURBO_TASK_FAILED, GATE_NON_TURBO_FAILED, CHECK_SETUP_FAILED, CANCELLED, SWEEP_OK/FAILED. - workflow_dispatch input `cache: cold` skips the restore to measure cold. Adds the CI wall-clock plan (docs/plans) this change is unit U1 of.
…cache - The save guard compared only the fallback restore's exact hit, which is empty whenever the base entry restored, so a same-sha rerun tried to save over its own key. It now skips when either restore matched this run's key. - cache-sweep lists and deletes through `gh cache list` / `gh cache delete <id>` (still by id) and checks out only the job-report action. - job-report names its reason vocabulary itself and reports NO_REASON_CODE instead of an empty title when a caller passes none.
cancelled() is valid only in if: conditionals, so the workflow failed to compile on every push of this branch (actionlint: calling function "cancelled" is not allowed here).
…rden the sweep - .github/scripts/turbo-summary.sh reads missed and failed tasks from Turbo's run summaries and exits 2 naming the first missing field. The verdict maps that to GATE_VERDICT_UNPARSEABLE with a next action and fails the job, instead of reading a format change as "nothing missed, nothing failed". turbo-summary.test.sh feeds it summaries missing fields and runs in check before the Gate. - A lookup-only probe of this commit's own entry replaces the key comparisons: a re-run never re-tars and re-saves an entry that exists. A cache: cold dispatch never saves. The summary says "save: attempted" or "skipped (<why>)". - cache-sweep: a failed delete is a SWEEP_DELETE_FAILED warning plus a count, and a failed PR-state lookup keeps that PR's entries; only a failed listing fails the job. - The e2e comment claimed a cancelled job skips if: always() steps; a cancelled run ran them. It now says what timeout buys.
… with the repo scripts #275 runs every scripts/*.test.ts in check, with Deno. The summary reader moves there as scripts/turbo-summary.ts (run with node in the verdict) beside a Deno test that feeds it summaries missing the exit code, the cache status, or the whole summary. The bash reader and its own test step are gone.
…eno test The previous commit removed the bash reader without the workflow and script changes that replace it.
…h install @effect/platform-node, platform-node-shared, opentelemetry and vitest 4.0.3 were published at 10:03 UTC with a peer effect@^4.0.3 that is not on the registry. The bake resolves registry dependencies fresh, so npm resolved forever and every e2e leg of run 38043976128 timed out in global setup. Both bake installs now pass --min-release-age=1, and each runs under a 300 s timeout whose expiry names the fixture and step on stderr.
The bake guest is node:24-alpine, whose timeout is busybox: a killed install exits with its own signal status (143 or 137), never GNU's 124, so the reason line never printed. The script now decides from elapsed seconds, and timeout -k 10 kills an npm that ignores TERM.
… builds The e2e legs restore check's Turbo entry with the same two restore steps (base commit, then newest readable) and never save one, so the pre-test build of unchanged packages comes from cache. LGTM starts detached right after pnpm setup and a separate `Wait for LGTM` step blocks before the lane, so the image pull overlaps the install. `lgtm:up` is now `lgtm:start` then `lgtm:ready`. typescript-checker.e2e runs in two legs from one CHECKER_SPLIT pattern: `-t P` and `-t ^(?!.*(?:P))`, so every scenario matches exactly one leg. The rest legs exclude the file and keep --shard=1/2 and 2/2. Each leg writes a Vitest JSON report beside the default and github-actions reporters, and calls job-report with E2E_OK, E2E_LANE_FAILED (naming failing files), E2E_LANE_TIMEOUT (exit 124), LGTM_NOT_READY, TRACE_SEARCH_FAILED, E2E_SETUP_FAILED or CANCELLED. U4 of the CI wall-clock plan.
…adable reports pnpm 11 needs Node 22.13 or later, so Start LGTM now runs after setup-node. A failed Start LGTM reports LGTM_NOT_READY. A truncated Vitest report or Turbo summary no longer aborts the verdict step under bash -e. A lane killed by timeout after ignoring TERM (exit 137 at the 1200 s budget) reports E2E_LANE_TIMEOUT. Review fixes for U4 of the CI wall-clock plan.
Turbo 2.10 hashes the args after -- into every task of a run, so the legs' builds never matched check's entries (0 cached / 14 on #291's first run). The legs now build stryker-e2e's dependencies in an args-free run, whose hashes equal check's, and run test:e2e --only. U4 of the CI wall-clock plan.
The lane's test:e2e --only run also lists the dependency builds in its Turbo summary, so the verdict counted every build twice (26 / 26 for 13 builds). Only the build step now writes a run summary. U4 of the CI wall-clock plan.
# Conflicts: # .github/workflows/ci.yml
…ch (WIP) The LGTM image's otelcol-contrib 0.160.0 ships the file exporter, so an overlay (test/e2e/lgtm/otelcol-capture.yaml, merged through the image's OTELCOL_EXTRA_ARGS) writes every received span to e2e-telemetry/capture/traces.jsonl. The capture step stops the container (SIGTERM, 25 s bound) and requires the collector's "Shutdown complete." log line, which it writes only after the file exporter flushed and closed the file. Failures report TRACE_CAPTURE_MISSING / _INCOMPLETE / _EMPTY and TRACE_UPLOAD_FAILED with a next action. export-traces.ts and its 70 s settle are removed; import-traces.ts replays the captured OTLP lines. WIP: per-leg trace/span parity against run 38068253763 not yet proven on CI.
Proves on CI that a lane whose collector wrote nowhere the Capture traces step reads fails with TRACE_CAPTURE_MISSING, a next action and an ::error annotation.
The capture step's own ::error duplicated the job-report annotation. The verdict now annotates the capture reason only when it is not already the job's reason, as it does for TRACE_UPLOAD_FAILED.
…t follows)" This reverts commit 9a62a2c.
systemfsoftware-maker
changed the title
Contributor
|
Closing without merging, superseded: e2e is moving off pull requests to its own per-SHA workflow on main (branch ci/e2e-postsubmit), and leg budgets are measured, not enforced. Reusing check's builds doesn't carry across workflows. If a leg's measured wall time on main stays over its target, the shard split for typescript-checker.e2e and the collector trace capture from this branch come back as a fresh PR against the e2e workflow. The branch ci/wall-clock-u4 is kept. |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
The e2e legs no longer rebuild packages
checkalready built, and no longer wait for LGTM before installing.typescript-checker.e2e(6 of rest-1's 9 lane minutes) now runs across two legs, so no e2e leg sits on a single 14-minute critical path. Each leg also captures its traces from the collector in about 15 s; previously it searched Tempo for 1.7–1.8 min. Every scenario still runs exactly once with the same assertions. Each leg's summary names its files, name filter, Vitest counts, pre-test build cache hits, captured trace and span counts, and a reason code.This is U4 and U3 of the CI wall-clock plan (
docs/plans/2026-10-10-0702-perf-ci-wall-clock-plan.md), stacked on #279 (U1: Turbo key,job-report). U5 (fixture bake cache) is #285.Design decisions
check'sturbo-<os>-dist-*entry with U1's two steps (base commit first, then the newest readable entry). They never save, so the cache budget andcheck's pruning are unchanged. Turbo 2.10 stamps the args after--into the hash of every task in the run, dependency builds included (cliArgumentsonstryker-e2e-core#build). Onepnpm test:e2e -- <files>run therefore rebuilt every package under a hash only that leg uses: 0 of 14 cached on this PR's first run. The legs now buildstryker-e2e's dependencies in an args-free run, whose hashes matchcheck's on CI (stryker-e2e-core#build=d5c158b3f6ce9696in both), and the lane then runstest:e2e --only.CHECKER_SPLIT=stryker\.(?:references|preservation)\.config. Leg 1 runs-t Pand leg 2 runs-t ^(?!.*(?:P)), so any full test name matches exactly one leg, including scenarios added later.setup-node, not before it. pnpm 11.27.0 needs Node ≥22.13 and the runner image ships 20.lgtm:upbecomeslgtm:start && lgtm:ready.grafana/otel-lgtm0.33.0, digest475319e8…) ships otelcol-contrib 0.160.0, whose distribution includes thefileexporter.test/e2e/lgtm/otelcol-capture.yamlis merged over the image's collector config through itsOTELCOL_EXTRA_ARGShook. It writes every received span toe2e-telemetry/capture/traces.jsonlas OTLP JSON.Capture tracesrunsdocker stop --time 25 stryker-lgtm. The image stops its components on SIGTERM, and the collector logsShutdown complete.only after its exporters have flushed and closed the file. The step checks for that line: there is no sleep, settle window, or polling loop.export-traces.tsand its 70 s settle wait are deleted.import-traces.tsreplays a captured file into a local stack. Thecheckjob type-checks it against the scripts import map.E2E_OK,E2E_LANE_FAILED(names the failing files),E2E_LANE_TIMEOUT,LGTM_NOT_READY,E2E_SETUP_FAILED,CANCELLED, and for a lane that passed but whose traces were not kept:TRACE_CAPTURE_MISSING,TRACE_CAPTURE_INCOMPLETE,TRACE_CAPTURE_EMPTY,TRACE_UPLOAD_FAILED. Each failed capture is annotated once. When the lane failed for another reason, the verdict still annotates the capture failure separately.U3 evidence
Capture is proven on two green runs, 38082608332 (cee7fb4) and 38085468401 (a5b712e). The baseline is the Tempo-search export of 38068253763. Counts are traces/spans from each leg's
e2e-telemetry-*artifact.vitest.start/e2e.setup.prune/gherkin.stepe2e.setup.prunetrace the baseline run did not export.vitest.module.transform,stryker.mutationReporting.checkpoint,vitest.runtime.open. They move in both directions between the two capture runs. Two of the fourteen samples fall below the baseline: rest-1 run 2 by 8 spans and typescript-checker-1 run 1 by 4.otlp_http/metricsinterrupted at shutdown, not traces.Capture tracestakes 15–16 s on every leg;Export telemetrytook 1.7–1.8 min.Failure path: a planted commit (9a62a2c, reverted in a5b712e) mounted the collector's capture directory elsewhere. On 38083646223 and 38084796362, every leg's
E2E lanepassed andCapture tracesfailed. The job failed with the reasonTRACE_CAPTURE_MISSINGand its next action ("The collector wrote no log to e2e-telemetry/capture; check that process-compose.yaml mounts …") in the step summary and in one::errorannotation per job. The first run showed that annotation twice; b3f7773 leaves it to the verdict.Wall times (a5b712e, run 38085468401)
lifecycle is the critical leg. Its lane alone took 8.3–9.3 min on this PR's last four runs, with walls of 9.5, 10.5, 9.6 and 9.7 min, so it remains at the 10-minute budget and passed it once. U5 (#285) removes its bake.
Coverage: the same 10 non-checker files run once each.
typescript-checker.e2e.test.ts's 5 scenarios split 2 + 3 with none run twice: 16 tests, as before.