From 91befb0cfea68478b928777c8522a5a15ac58c7c Mon Sep 17 00:00:00 2001 From: Nirupam Bhowmick <48842933+heydryft@users.noreply.github.com> Date: Mon, 17 Aug 2026 19:02:46 +0100 Subject: [PATCH 1/2] ci: run the full lane on stacked PRs, and refuse to report green when it did not (D18 #11) MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit `ci.yml` and `cuda_compile_check.yaml` both triggered on `pull_request: branches: [master]`. A PR whose base is another PR's branch matched neither, so not one lane ran — no compile, test, rustfmt, clippy, MSRV or nvcc. The only workflow without a branch filter is analysis.yaml (`pull_request_target`), so those PRs showed exactly one check, `comment`, and rendered green. Six open PRs were in that state at once: the stack #95 -> #100 -> #102 -> #103 -> #104, plus #98. Check-run census: 16 checks on master-based PRs, 1 on every stacked one. This compounds with the macOS gap — a stacked PR touching cuda-gated Rust got neither the nvcc lane nor a local type-check, since cargo check on macOS cannot see those arms. Two changes, because the trigger fix alone is not enough: 1. `pull_request.branches: ['**']` on both workflows. Stacked PRs are normal here and must get the full lane. 2. A `ci-complete` guard job that needs every lane, runs `if: always()`, and fails unless each reports success — skipped and cancelled included — and asserts the lane count. A branch filter is only one route to 'the lanes did not run'; a skipped job, a renamed job, an empty matrix or an over-narrow paths filter all reproduce it. Unlike the individual lanes, ci-complete cannot pass by not existing, which makes it the correct required status check for branch protection. flash_attn_compile_check.yaml is left dispatch-only by design. --- .github/workflows/ci.yml | 64 ++++++++++++++++++++++- .github/workflows/cuda_compile_check.yaml | 5 +- 2 files changed, 66 insertions(+), 3 deletions(-) diff --git a/.github/workflows/ci.yml b/.github/workflows/ci.yml index f8f4434e0..a9e031134 100644 --- a/.github/workflows/ci.yml +++ b/.github/workflows/ci.yml @@ -5,8 +5,14 @@ on: branches: - master pull_request: - branches: - - master + # D18 #11: this was `- master`, so a PR whose BASE was another PR's branch + # (a stack) matched nothing here and ran zero lanes. The only workflow + # without a branch filter is analysis.yaml (`pull_request_target`), so such + # a PR showed exactly one check — `comment` — and read as green while + # nothing had been compiled, tested, linted or nvcc'd. Six PRs sat in that + # state. Match every base branch; stacked PRs are normal here. + branches: + - '**' workflow_dispatch: name: Continuous integration @@ -154,6 +160,60 @@ jobs: command: check args: --workspace + # D18 #11 GUARD — the job that makes "green" mean something. + # + # Fixing the `pull_request` branch filter above stops THIS instance, but the + # class of defect is "the real lanes did not run and their absence read as a + # pass". A branch filter is only one way to reach that state; a skipped job, + # a renamed job, a matrix that expands to nothing, or a `paths:` filter that + # excludes everything all reproduce it exactly. + # + # So: this job depends on every substantive lane, runs even when they fail or + # are skipped (`if: always()`), and fails unless each one actually SUCCEEDED. + # `skipped` and `cancelled` are failures here — that is the entire point. + # It also asserts the lane COUNT, so silently deleting a lane trips it rather + # than quietly shrinking what green covers. + # + # Make THIS the required status check in branch protection. It is the only + # job whose non-success is unambiguous, and unlike the individual lanes it + # cannot pass by not existing. + ci-complete: + name: CI complete + runs-on: ubuntu-latest + if: always() + needs: + - check + - check-metal + - test + - fmt + - clippy + - docs + - typos + - msrv + steps: + - name: Assert every lane ran and succeeded + env: + # Passed via env, not interpolated into the script, so a lane result + # can never be shell-injected. + LANE_RESULTS: ${{ join(needs.*.result, ' ') }} + EXPECTED_LANES: '8' + run: | + set -euo pipefail + echo "lane results: ${LANE_RESULTS}" + n=0 + for r in ${LANE_RESULTS}; do + n=$((n + 1)) + if [ "${r}" != "success" ]; then + echo "::error::A CI lane reported '${r}'. Refusing to report green." + exit 1 + fi + done + if [ "${n}" -ne "${EXPECTED_LANES}" ]; then + echo "::error::Expected ${EXPECTED_LANES} lanes, saw ${n}. The job set drifted — update ci-complete's needs list deliberately, do not let green shrink silently." + exit 1 + fi + echo "All ${n} lanes succeeded." + # run `mlc` (`cargo install mlc`) # markdown-link-check: # runs-on: ubuntu-latest diff --git a/.github/workflows/cuda_compile_check.yaml b/.github/workflows/cuda_compile_check.yaml index b8177e16b..af52755d3 100644 --- a/.github/workflows/cuda_compile_check.yaml +++ b/.github/workflows/cuda_compile_check.yaml @@ -120,7 +120,10 @@ on: - "Cargo.lock" - ".github/workflows/cuda_compile_check.yaml" pull_request: - branches: [master] + # D18 #11: was [master], so stacked PRs never got the nvcc lane — the one + # lane that catches cuda-gated Rust, which a macOS `cargo check` + # structurally cannot type-check. Match every base branch. + branches: ['**'] paths: - "**/*.rs" - "**/*.cu" From 71485376d3f2be474a8d4ea62621be505dd6b194 Mon Sep 17 00:00:00 2001 From: Nirupam Bhowmick <48842933+heydryft@users.noreply.github.com> Date: Mon, 17 Aug 2026 20:17:11 +0100 Subject: [PATCH 2/2] ci: add the concurrency group ci.yml never had MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Widening the pull_request trigger multiplies this workflow across every stacked PR, and ci.yml is 12 jobs — the largest multiplier in the repo, and the only workflow with no concurrency group. cuda_compile_check.yaml has carried the identical block for a while; this is the same shape on purpose. Without it every push leaves its superseded runs sitting in a shared queue. Measured today rather than assumed: 38 in-flight runs, of which 13 were on SHAs that were no longer any PR's head, cancelled by hand to free capacity. perf/qtip-grouped-gemm-arch alone held two queued CI runs, one obsolete. The 'head_ref || run_id' idiom is load-bearing. On pull_request, head_ref is the source branch, so a new push cancels that PR's own in-flight run. On push, schedule and workflow_dispatch, head_ref is empty and run_id is unique per run, so master pushes and scheduled runs are never cancelled by each other — only PR iterations are. --- .github/workflows/ci.yml | 15 +++++++++++++++ 1 file changed, 15 insertions(+) diff --git a/.github/workflows/ci.yml b/.github/workflows/ci.yml index a9e031134..c4e31cd37 100644 --- a/.github/workflows/ci.yml +++ b/.github/workflows/ci.yml @@ -17,6 +17,21 @@ on: name: Continuous integration +# Widening the trigger above multiplies this workflow across every stacked PR, +# so it must stop paying for superseded runs on the same ref — `ci.yml` is 12 +# jobs, the largest multiplier in the repo, and was the only workflow without +# this. `cuda_compile_check.yaml` has carried the identical block for a while; +# this is the same shape deliberately, not a new policy. +# +# The `head_ref || run_id` idiom is load-bearing: on `pull_request`, `head_ref` +# is the source branch, so a new push cancels that PR's own in-flight run. On +# `push`, `schedule` and `workflow_dispatch`, `head_ref` is empty and the +# fallback `run_id` is unique per run — so master pushes and scheduled runs are +# never cancelled by each other, only PR iterations are. +concurrency: + group: ${{ github.workflow }}-${{ github.head_ref || github.run_id }} + cancel-in-progress: true + jobs: check: name: Check