Skip to content

fix: fail the test gate when the runner cannot run tests - #1054

Open
kylerankin wants to merge 3 commits into
projectbluefin:mainfrom
kylerankin:fix/test-baseline-runner-error
Open

kylerankin wants to merge 3 commits into
projectbluefin:mainfrom
kylerankin:fix/test-baseline-runner-error

Conversation

@kylerankin

Copy link
Copy Markdown
Contributor

Summary: This repository has a check that tells contributors whether their change broke anything, but it could report success even when the tests never actually ran — for example when a test file fails to load or the test runner crashes. That meant a contributor could be told everything was fine while the code they touched was never checked. This fix makes the check report a failure in those cases, so a green result now means the tests really did run. Normal results are unchanged: expected, previously-known failures still pass, and a healthy run still passes.

What I need from you: Review it.


What changed

The local test check (npm run test:gate) compares the current set of failing tests against a recorded baseline and fails only on new failures. It shells out to the test runner, reads the JSON report, and counts individual failed assertions while ignoring the runner's own exit code.

Three gaps let a suite that did not run look green:

  • The runner's exit code was discarded, so a suite that failed to load, a setup error, or a runner crash — all of which exit non-zero — were treated the same as a normal red suite.
  • Only assertions marked failed were counted. A suite that failed before any assertion ran produced no such entries, so it read as "no new failures".
  • An incomplete report could be written to the baseline under the re-record mode, replacing it with empty evidence.

The fix

  • The runner's exit code is now captured instead of discarded.
  • A run that ends without any completed failed assertions is rejected with an explicit message, since a normal red suite always has real failures to explain its exit code.
  • The JSON report is validated for completeness before it is read or an updated baseline is written; an incomplete report is rejected with a diagnostic.
  • A suite that fails to load reports a failed assertion with no name; it is no longer counted as a completed failure.
  • Recorded, expected failures still pass, so the normal red suite is unaffected, and neither the public commands nor the baseline format changed.

Tests

scripts/tests/test-baseline.test.ts adds regression cases for failed suites without assertions, runner errors, incomplete reports, and refusal to overwrite the baseline on such failures, alongside passing controls for healthy runs and recorded assertion failures. All 20 tests pass and the changed files pass lint.

Notes

The CI job runs the test runner directly with coverage, not this wrapper, so this change does not affect CI and no workflow edit is needed.

Closes #1053

-- hive: backend=pi model=lemonade/ornith-1.5-35b-a3b-gguf-q6_k


🐝 Hive Agent: contributor | SHA: 9b12dcbd

The baseline gate only counted individual failed assertions and ignored
the test runner's own exit code, so a suite that failed to load, a setup
error, or a runner crash - any failure that stops tests from running - was
reported as no new failures. Contributors could therefore see a green result
while their code never actually executed.

The gate now captures the runner's exit code and rejects a run that ends
without any completed failed assertions, since a healthy red suite always has
real failures to explain its exit. It also validates that the JSON report is
complete before reading it or recording an updated baseline, so an incomplete
report can no longer overwrite the baseline or pass the gate. Recorded,
expected failures still pass, so the normal red suite is unaffected.

Tests: regression cases for failed suites without assertions, runner
errors, incomplete reports, and refusal to overwrite the baseline on such
failures, with passing controls for healthy runs and recorded failures.

Assisted-by: pi (lemonade/ornith-1.5-35b-a3b-gguf-q6_k)

Hive-Run: projectbluefin#1053

Hive-Plan: repair-test-baseline-runner-error

Hive-Spec: interface-violation
Signed-off-by: kylerankin <kylerankin@users.noreply.github.com>
@hivecommons-hive hivecommons-hive Bot added author/clanker A contributor's own hive authored this PR under their login dco All commits are DCO signed-off role/trusted Author is trusted for unattended merge (trusted-author role or registry/claim tier in require_tiers) stage/050-review-pending Review swarm requested; awaiting a verdict for the current head labels Nov 20, 2026

@hivecommons-hive hivecommons-hive Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Summary: This change makes the website's local test check fail when the tests never actually ran, instead of reporting success. It handles the cases it targets, but one common case still slips through: if one test file fails to load while some other test fails in an already-expected way, the check still says everything is fine. That case matters most once the list of expected failures is no longer empty, which is what this check exists for.

What I need from you: Apply the suggested change on the inline comment (or an equivalent), plus a test for it.


correctness (medium, in-scope) - see the inline comment on scripts/test-baseline.mjs:93. The new guard only fires when no named failure exists anywhere in the run, so a suite-load failure is hidden by any unrelated failure that is already in the baseline. I reproduced this against the head script with a stubbed npx (baseline src/tests/a.test.ts :: a fails; report with that failure plus broken.test.ts at status: "failed", assertionResults: []; child exit 1): output test-baseline: no new failures (1 known, baseline 1)., exit 0. Under --update the same run would rewrite the baseline without the broken file's tests. Issue #1053 asks to reject load/setup failures "not explained by completed failed assertions"; a failure in another file does not explain them.

Limit of this review: I did not run real vitest 5.1.0 to confirm the exact JSON shape it emits for an import failure (empty assertionResults vs. the nameless failed entry the PR describes). The suggested fix keys on the file-level status, so it covers both shapes. A runner-level unhandled error (not tied to one file) alongside known failures would still pass; that is worth a note in the code if you do not handle it.

CI on 9b12dcbd: test, wolves-movie-flow, Audit GitHub Actions passed.

No findings from: security, intent-alignment, style, docs-currency.

Confidence: 3/5 (needs attention) — 1 medium finding; 1 perspective requested changes

-- hive: agent=reviewer backend=claude model=claude-opus-5-5 effort=medium claude=2.1.327

Comment thread scripts/test-baseline.mjs
// normal red suite always has completed failed assertions to explain its exit
// code, so the absence of any is itself a failure. Reject it before the
// comparison and before any `--update` could record an empty baseline.
if (childExit !== 0 && current.size === 0) {

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

correctness - medium: this guard only fires when the whole run has zero named failures, so one known failure anywhere hides a suite that failed to load.

Reproduced at this head with a stubbed npx: baseline src/tests/a.test.ts :: a fails; report has that failure plus src/tests/broken.test.ts with "status": "failed" and "assertionResults": []; child exit 1. Result: test-baseline: no new failures (1 known, baseline 1)., exit 0. --update would also drop broken.test.ts's tests from the baseline.

Proposed fix: flag each file that vitest marks failed but that has no named failed assertion, and reject the run if any exist (replaces lines 75–98):

 const current = new Set()
+const unexplained = []
 for (const file of report.testResults) {
+  let named = 0
   for (const assertion of file.assertionResults ?? []) {
     // A suite that failed to load reports a failed assertion with no name;
     // it is not a completed assertion and must not enter the failure set.
     if (assertion.status === 'failed' && assertion.fullName) {
       current.add(`${file.name.replace(cwd, '')} :: ${assertion.fullName}`)
+      named++
     }
   }
+  // A file vitest marks failed with no completed failed assertion did not run
+  // (import/setup error). Failures in other files do not explain it.
+  if (file.status === 'failed' && named === 0) {
+    unexplained.push(file.name.replace(cwd, ''))
+  }
 }
 
 const sorted = [...current].sort()
 
-if (childExit !== 0 && current.size === 0) {
+if (unexplained.length > 0 || (childExit !== 0 && current.size === 0)) {
   console.error(
     'test-baseline: vitest failed to run to completion and reported no completed failures',
   )
+  for (const name of unexplained) {
+    console.error(`  ! ${name} failed without completed assertions`)
+  }
   process.exit(2)
 }

Plus a regression test: non-empty baseline with one known failure in a.test.ts, a second file with status: 'failed' and empty assertionResults, vitestRc: 1 → expect status 2 and the baseline unchanged under --update. The existing test fixtures in this PR omit the file-level status, so add status: 'failed' to the failed-suite reports there too.

@hivecommons-hive hivecommons-hive Bot added stage/072-returned-to-contributor Clanker PR: CI red, changes requested or conflicting; diagnosis posted, its hive must act and removed stage/050-review-pending Review swarm requested; awaiting a verdict for the current head labels Nov 20, 2026
@hivecommons-hive

Copy link
Copy Markdown
Contributor

@kylerankin this PR is back in your court (stage/072-returned-to-contributor). The hive will not merge it until the head changes.

Review asks:

  • the review swarm requested changes
  • review finding: scripts/test-baseline.mjs:93 Suite-load failure still passes the gate when any known failure exists elsewhere

Reply here or push a new head to this branch (same sign-off); CI and the review swarm re-run automatically and the PR returns to the normal path.

Queued in your relay fix queue (1 waiting): fixes are handed out before new work as soon as you have capacity.

@hivecommons-hive

Copy link
Copy Markdown
Contributor

@kylerankin the hive has adopted this PR (stage/074-adopted-by-hive): the review asks stayed open (the review swarm requested changes; review finding: scripts/test-baseline.mjs:93 Suite-load failure still passes the gate when any known failure exists elsewhere) for 2h after the nudge and the relay nudge went unanswered. The hive will rebase this branch on main here; your commits and sign-off stay. The fix here repairs CI only: if the review swarm's finding is still open on this head after the stage dwell, the hive moves the work to a superseding PR that carries the finding. Push a new head yourself at any time and the hive yields it back to you.

@hivecommons-hive hivecommons-hive Bot added stage/074-adopted-by-hive Contributor PR adopted: author unavailable after red CI; the hive is finishing it (takeover path) and removed stage/072-returned-to-contributor Clanker PR: CI red, changes requested or conflicting; diagnosis posted, its hive must act labels Nov 20, 2026
The run-wide "non-zero exit and no failures" check let a file that failed
to import hide behind any unrelated known failure, and --update would drop
that file's tests from the baseline. Track named failures per file and
reject any file that failed (or reported a nameless failed assertion)
without a completed failed assertion. Add a regression test with a
non-empty baseline.

Hive-Fix-Attempt: 1/6

Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
Signed-off-by: scanner <280983584+hivecommons-hive[bot]@users.noreply.github.com>
@hivecommons-hive hivecommons-hive Bot added the merge-method/merge Hive merges this PR with a merge commit (mixed authorship, each commit self-signed); never squash label Nov 20, 2026
@hivecommons-hive

Copy link
Copy Markdown
Contributor

1 of 2 commits on a4a60099d2d9 are the hive's, signed off as the hive App, and the rest are @kylerankin's with their own sign-off. A squash would land the hive's commits under @kylerankin's name with the App's sign-off, so this PR merges with a merge commit instead (merge-method/merge): every commit lands as it is, each with its own author's sign-off. Nothing is rewritten and nobody needs to re-sign anything. Maintainers merging by hand: use gh pr merge --merge, not squash.

@hivecommons-hive hivecommons-hive Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Summary: This change makes the project's local "did I break anything?" check report a failure when tests could not actually run. The latest update closes the gap found in the last review: a test file that fails to load is now caught even when other, already-known failures exist. The logic now looks right, but the project's automatic style check rejects two lines in the new test, so the automated checks on this pull request are failing.

What I need from you: Run the style auto-fix on the test file and push.


Reviewed head a4a60099, delta since 9b12dcbd.

Previous finding - suite-load failure masked by an unrelated known failure (scripts/test-baseline.mjs:93): fixed. Per-file unexplained check at scripts/test-baseline.mjs:95-116 rejects any file with status: 'failed' and no named failed assertion, before --update writes (:125). New regression test at scripts/tests/test-baseline.test.ts:431-458 covers both gate and --update paths.

style

  • CI test job (job 136320457123, head_sha a4a60099) fails at npm run lint: antfu/reasonable-newline errors at scripts/tests/test-baseline.test.ts:443:29 and :456:26, both in the new test. Log says both are --fix-able. PR body claim "changed files pass lint" is false at this head. Fix: npx eslint --fix scripts/tests/test-baseline.test.ts, push.

correctness (advisory, non-blocking)

  • scripts/test-baseline.mjs:95 only flags a file when named === 0. A file with a baseline-known named failure plus a file-level hook error (e.g. afterAll throws) still passes the gate. Narrow residual case; import failures, the issue's main case, are covered.

No findings from: security, intent-alignment, docs-currency.

Confidence: 3/5 (needs attention) — 1 medium finding; 1 perspective requested changes

-- hive: agent=reviewer backend=claude model=claude-opus-5-5 effort=medium claude=2.1.327

Comment thread scripts/tests/test-baseline.test.ts Outdated
{
name: join(root, 'src/tests/a.test.ts'),
status: 'failed',
assertionResults: [{ status: 'failed', fullName: 'a fails' }],

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

CI lint fails here at head a4a6009: antfu/reasonable-newline (443:29). Put the nested object on its own line, or run npx eslint --fix scripts/tests/test-baseline.test.ts.

Comment thread scripts/tests/test-baseline.test.ts Outdated
const { status, stderr } = runGate(root, { vitestRc: 1, report })
expect(status).toBe(2)
expect(stderr).toContain('src/tests/broken.test.ts')
expect(runGate(root, { vitestRc: 1, report, args: ['--update'] }).status).toBe(2)

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

CI lint fails here too: antfu/reasonable-newline (456:26). Auto-fixable with eslint --fix.

Comment thread scripts/test-baseline.mjs Outdated
}
}
}
if ((file.status === 'failed' || nameless > 0) && named === 0) {

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Advisory: named === 0 means a file with a known named failure plus a file-level hook error (e.g. afterAll throw) is not flagged, so the gate still passes. Narrow case; import failures are covered. Consider also flagging files whose status === 'failed' carries a file-level message regardless of named.

A hook error reported on the file (message) was masked when the file
also had a baseline-known named failure. Flag any file with a file-level
error, add a regression test, and fix antfu lint errors in the test.

Hive-Fix-Attempt: 2/6

Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
Signed-off-by: scanner <280983584+hivecommons-hive[bot]@users.noreply.github.com>

@hivecommons-hive hivecommons-hive Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Summary: This change makes the project's local test check fail when tests did not actually run, instead of falsely reporting success. Since the last review, the author fixed both open problems: a test file whose setup or cleanup step crashes is now caught even when it also has an expected failure, and the new test now passes the project's formatting checks. The change is ready from every angle reviewed; one small wording note is left inline.


Delta re-review at head fa974616, against previous head a4a60099.

Previous findings:

  • [correctness] scripts/test-baseline.mjs:95 hook error in a file with a named failure was not flagged: fixed. scripts/test-baseline.mjs:98-99 now flags any file with a non-empty file-level message, and the new test at scripts/tests/test-baseline.test.ts:466-487 covers it (named baseline-known failure plus afterAll hook threw, exit 2).
  • [style] scripts/tests/test-baseline.test.ts:443 new test failed repo lint: fixed. At this head the test job (136327445082, head_sha fa974616) passed npm run lint, npm run typecheck and npm run test:run -- --coverage.

One low-severity wording note is inline. It does not block.

No findings from: correctness, security, intent-alignment, docs-currency.

Confidence: 5/5 (safe)

-- hive: agent=reviewer backend=claude model=claude-opus-5-5 effort=medium claude=2.1.327

Comment thread scripts/test-baseline.mjs
// file itself, never on an assertion, so it flags the file even when other
// named failures in it are baseline-known.
const fileError = typeof file.message === 'string' && file.message.trim() !== ''
if (fileError || ((file.status === 'failed' || nameless > 0) && named === 0)) {

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

style (low): A file now enters unexplained because of a file-level message, even when it has completed named failures (the new hook-error case). The diagnostic at the unexplained branch still prints file(s) failed without any completed failed assertion, and the comment above the unexplained declaration says the same. For the hook-error case that text is wrong and can send a contributor looking for a load failure. Suggest wording like file(s) failed to run to completion (load/setup/hook error): and updating the comment to match. Non-blocking.

@hivecommons-hive hivecommons-hive Bot added stage/090-ready-for-human-merge Hive would merge this but trust/role blocks it; a maintainer's /lgtm /approve is all that is missing and removed stage/074-adopted-by-hive Contributor PR adopted: author unavailable after red CI; the hive is finishing it (takeover path) labels Nov 20, 2026
@hivecommons-hive

Copy link
Copy Markdown
Contributor

Hive status — updated in place; edits do not notify. Full history is in the Activity Monitor.

Ready for a human merge: verdict approved fa97461, 4/4 checks green, soak 30m0s elapsed. Auto-merge is off for this repo, so a person merges.

The hive never approves on your behalf: approve and merge it on GitHub if you accept it.

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

author/clanker A contributor's own hive authored this PR under their login dco All commits are DCO signed-off merge-method/merge Hive merges this PR with a merge commit (mixed authorship, each commit self-signed); never squash role/trusted Author is trusted for unattended merge (trusted-author role or registry/claim tier in require_tiers) stage/090-ready-for-human-merge Hive would merge this but trust/role blocks it; a maintainer's /lgtm /approve is all that is missing

Projects

None yet

Development

Successfully merging this pull request may close these issues.

[architect] Keep the website test check from passing when tests cannot run

1 participant