Repository navigation
fix: fail the test gate when the runner cannot run tests - #1054
kylerankin wants to merge 3 commits into
Conversation
The baseline gate only counted individual failed assertions and ignored the test runner's own exit code, so a suite that failed to load, a setup error, or a runner crash - any failure that stops tests from running - was reported as no new failures. Contributors could therefore see a green result while their code never actually executed. The gate now captures the runner's exit code and rejects a run that ends without any completed failed assertions, since a healthy red suite always has real failures to explain its exit. It also validates that the JSON report is complete before reading it or recording an updated baseline, so an incomplete report can no longer overwrite the baseline or pass the gate. Recorded, expected failures still pass, so the normal red suite is unaffected. Tests: regression cases for failed suites without assertions, runner errors, incomplete reports, and refusal to overwrite the baseline on such failures, with passing controls for healthy runs and recorded failures. Assisted-by: pi (lemonade/ornith-1.5-35b-a3b-gguf-q6_k) Hive-Run: projectbluefin#1053 Hive-Plan: repair-test-baseline-runner-error Hive-Spec: interface-violation Signed-off-by: kylerankin <kylerankin@users.noreply.github.com>
There was a problem hiding this comment.
Summary: This change makes the website's local test check fail when the tests never actually ran, instead of reporting success. It handles the cases it targets, but one common case still slips through: if one test file fails to load while some other test fails in an already-expected way, the check still says everything is fine. That case matters most once the list of expected failures is no longer empty, which is what this check exists for.
What I need from you: Apply the suggested change on the inline comment (or an equivalent), plus a test for it.
correctness (medium, in-scope) - see the inline comment on scripts/test-baseline.mjs:93. The new guard only fires when no named failure exists anywhere in the run, so a suite-load failure is hidden by any unrelated failure that is already in the baseline. I reproduced this against the head script with a stubbed npx (baseline src/tests/a.test.ts :: a fails; report with that failure plus broken.test.ts at status: "failed", assertionResults: []; child exit 1): output test-baseline: no new failures (1 known, baseline 1)., exit 0. Under --update the same run would rewrite the baseline without the broken file's tests. Issue #1053 asks to reject load/setup failures "not explained by completed failed assertions"; a failure in another file does not explain them.
Limit of this review: I did not run real vitest 5.1.0 to confirm the exact JSON shape it emits for an import failure (empty assertionResults vs. the nameless failed entry the PR describes). The suggested fix keys on the file-level status, so it covers both shapes. A runner-level unhandled error (not tied to one file) alongside known failures would still pass; that is worth a note in the code if you do not handle it.
CI on 9b12dcbd: test, wolves-movie-flow, Audit GitHub Actions passed.
No findings from: security, intent-alignment, style, docs-currency.
Confidence: 3/5 (needs attention) — 1 medium finding; 1 perspective requested changes
-- hive: agent=reviewer backend=claude model=claude-opus-5-5 effort=medium claude=2.1.327
| // normal red suite always has completed failed assertions to explain its exit | ||
| // code, so the absence of any is itself a failure. Reject it before the | ||
| // comparison and before any `--update` could record an empty baseline. | ||
| if (childExit !== 0 && current.size === 0) { |
There was a problem hiding this comment.
correctness - medium: this guard only fires when the whole run has zero named failures, so one known failure anywhere hides a suite that failed to load.
Reproduced at this head with a stubbed npx: baseline src/tests/a.test.ts :: a fails; report has that failure plus src/tests/broken.test.ts with "status": "failed" and "assertionResults": []; child exit 1. Result: test-baseline: no new failures (1 known, baseline 1)., exit 0. --update would also drop broken.test.ts's tests from the baseline.
Proposed fix: flag each file that vitest marks failed but that has no named failed assertion, and reject the run if any exist (replaces lines 75–98):
const current = new Set()
+const unexplained = []
for (const file of report.testResults) {
+ let named = 0
for (const assertion of file.assertionResults ?? []) {
// A suite that failed to load reports a failed assertion with no name;
// it is not a completed assertion and must not enter the failure set.
if (assertion.status === 'failed' && assertion.fullName) {
current.add(`${file.name.replace(cwd, '')} :: ${assertion.fullName}`)
+ named++
}
}
+ // A file vitest marks failed with no completed failed assertion did not run
+ // (import/setup error). Failures in other files do not explain it.
+ if (file.status === 'failed' && named === 0) {
+ unexplained.push(file.name.replace(cwd, ''))
+ }
}
const sorted = [...current].sort()
-if (childExit !== 0 && current.size === 0) {
+if (unexplained.length > 0 || (childExit !== 0 && current.size === 0)) {
console.error(
'test-baseline: vitest failed to run to completion and reported no completed failures',
)
+ for (const name of unexplained) {
+ console.error(` ! ${name} failed without completed assertions`)
+ }
process.exit(2)
}Plus a regression test: non-empty baseline with one known failure in a.test.ts, a second file with status: 'failed' and empty assertionResults, vitestRc: 1 → expect status 2 and the baseline unchanged under --update. The existing test fixtures in this PR omit the file-level status, so add status: 'failed' to the failed-suite reports there too.
|
@kylerankin this PR is back in your court ( Review asks:
Reply here or push a new head to this branch (same sign-off); CI and the review swarm re-run automatically and the PR returns to the normal path. Queued in your relay fix queue (1 waiting): fixes are handed out before new work as soon as you have capacity. |
|
@kylerankin the hive has adopted this PR ( |
The run-wide "non-zero exit and no failures" check let a file that failed to import hide behind any unrelated known failure, and --update would drop that file's tests from the baseline. Track named failures per file and reject any file that failed (or reported a nameless failed assertion) without a completed failed assertion. Add a regression test with a non-empty baseline. Hive-Fix-Attempt: 1/6 Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com> Signed-off-by: scanner <280983584+hivecommons-hive[bot]@users.noreply.github.com>
|
1 of 2 commits on |
There was a problem hiding this comment.
Summary: This change makes the project's local "did I break anything?" check report a failure when tests could not actually run. The latest update closes the gap found in the last review: a test file that fails to load is now caught even when other, already-known failures exist. The logic now looks right, but the project's automatic style check rejects two lines in the new test, so the automated checks on this pull request are failing.
What I need from you: Run the style auto-fix on the test file and push.
Reviewed head a4a60099, delta since 9b12dcbd.
Previous finding - suite-load failure masked by an unrelated known failure (scripts/test-baseline.mjs:93): fixed. Per-file unexplained check at scripts/test-baseline.mjs:95-116 rejects any file with status: 'failed' and no named failed assertion, before --update writes (:125). New regression test at scripts/tests/test-baseline.test.ts:431-458 covers both gate and --update paths.
style
- CI
testjob (job 136320457123, head_shaa4a60099) fails atnpm run lint:antfu/reasonable-newlineerrors atscripts/tests/test-baseline.test.ts:443:29and:456:26, both in the new test. Log says both are--fix-able. PR body claim "changed files pass lint" is false at this head. Fix:npx eslint --fix scripts/tests/test-baseline.test.ts, push.
correctness (advisory, non-blocking)
scripts/test-baseline.mjs:95only flags a file whennamed === 0. A file with a baseline-known named failure plus a file-level hook error (e.g.afterAllthrows) still passes the gate. Narrow residual case; import failures, the issue's main case, are covered.
No findings from: security, intent-alignment, docs-currency.
Confidence: 3/5 (needs attention) — 1 medium finding; 1 perspective requested changes
-- hive: agent=reviewer backend=claude model=claude-opus-5-5 effort=medium claude=2.1.327
| { | ||
| name: join(root, 'src/tests/a.test.ts'), | ||
| status: 'failed', | ||
| assertionResults: [{ status: 'failed', fullName: 'a fails' }], |
There was a problem hiding this comment.
CI lint fails here at head a4a6009: antfu/reasonable-newline (443:29). Put the nested object on its own line, or run npx eslint --fix scripts/tests/test-baseline.test.ts.
| const { status, stderr } = runGate(root, { vitestRc: 1, report }) | ||
| expect(status).toBe(2) | ||
| expect(stderr).toContain('src/tests/broken.test.ts') | ||
| expect(runGate(root, { vitestRc: 1, report, args: ['--update'] }).status).toBe(2) |
There was a problem hiding this comment.
CI lint fails here too: antfu/reasonable-newline (456:26). Auto-fixable with eslint --fix.
| } | ||
| } | ||
| } | ||
| if ((file.status === 'failed' || nameless > 0) && named === 0) { |
There was a problem hiding this comment.
Advisory: named === 0 means a file with a known named failure plus a file-level hook error (e.g. afterAll throw) is not flagged, so the gate still passes. Narrow case; import failures are covered. Consider also flagging files whose status === 'failed' carries a file-level message regardless of named.
A hook error reported on the file (message) was masked when the file also had a baseline-known named failure. Flag any file with a file-level error, add a regression test, and fix antfu lint errors in the test. Hive-Fix-Attempt: 2/6 Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com> Signed-off-by: scanner <280983584+hivecommons-hive[bot]@users.noreply.github.com>
There was a problem hiding this comment.
Summary: This change makes the project's local test check fail when tests did not actually run, instead of falsely reporting success. Since the last review, the author fixed both open problems: a test file whose setup or cleanup step crashes is now caught even when it also has an expected failure, and the new test now passes the project's formatting checks. The change is ready from every angle reviewed; one small wording note is left inline.
Delta re-review at head fa974616, against previous head a4a60099.
Previous findings:
- [correctness]
scripts/test-baseline.mjs:95hook error in a file with a named failure was not flagged: fixed.scripts/test-baseline.mjs:98-99now flags any file with a non-empty file-levelmessage, and the new test atscripts/tests/test-baseline.test.ts:466-487covers it (named baseline-known failure plusafterAll hook threw, exit 2). - [style]
scripts/tests/test-baseline.test.ts:443new test failed repo lint: fixed. At this head thetestjob (136327445082, head_shafa974616) passednpm run lint,npm run typecheckandnpm run test:run -- --coverage.
One low-severity wording note is inline. It does not block.
No findings from: correctness, security, intent-alignment, docs-currency.
Confidence: 5/5 (safe)
-- hive: agent=reviewer backend=claude model=claude-opus-5-5 effort=medium claude=2.1.327
| // file itself, never on an assertion, so it flags the file even when other | ||
| // named failures in it are baseline-known. | ||
| const fileError = typeof file.message === 'string' && file.message.trim() !== '' | ||
| if (fileError || ((file.status === 'failed' || nameless > 0) && named === 0)) { |
There was a problem hiding this comment.
style (low): A file now enters unexplained because of a file-level message, even when it has completed named failures (the new hook-error case). The diagnostic at the unexplained branch still prints file(s) failed without any completed failed assertion, and the comment above the unexplained declaration says the same. For the hook-error case that text is wrong and can send a contributor looking for a load failure. Suggest wording like file(s) failed to run to completion (load/setup/hook error): and updating the comment to match. Non-blocking.
|
Hive status — updated in place; edits do not notify. Full history is in the Activity Monitor. Ready for a human merge: verdict approved The hive never approves on your behalf: approve and merge it on GitHub if you accept it. |
Summary: This repository has a check that tells contributors whether their change broke anything, but it could report success even when the tests never actually ran — for example when a test file fails to load or the test runner crashes. That meant a contributor could be told everything was fine while the code they touched was never checked. This fix makes the check report a failure in those cases, so a green result now means the tests really did run. Normal results are unchanged: expected, previously-known failures still pass, and a healthy run still passes.
What I need from you: Review it.
What changed
The local test check (
npm run test:gate) compares the current set of failing tests against a recorded baseline and fails only on new failures. It shells out to the test runner, reads the JSON report, and counts individual failed assertions while ignoring the runner's own exit code.Three gaps let a suite that did not run look green:
failedwere counted. A suite that failed before any assertion ran produced no such entries, so it read as "no new failures".The fix
Tests
scripts/tests/test-baseline.test.tsadds regression cases for failed suites without assertions, runner errors, incomplete reports, and refusal to overwrite the baseline on such failures, alongside passing controls for healthy runs and recorded assertion failures. All 20 tests pass and the changed files pass lint.Notes
The CI job runs the test runner directly with coverage, not this wrapper, so this change does not affect CI and no workflow edit is needed.
Closes #1053
-- hive: backend=pi model=lemonade/ornith-1.5-35b-a3b-gguf-q6_k
🐝 Hive Agent:
contributor| SHA:9b12dcbd