Repository navigation
fix: fail the test gate when the runner cannot run tests #1054
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
base: main
Are you sure you want to change the base?
Changes from all commits
File filter
Filter by extension
Conversations
Jump to
Diff view
Diff view
There are no files selected for viewing
| Original file line number | Diff line number | Diff line change |
|---|---|---|
|
|
@@ -30,6 +30,11 @@ const shouldUpdate = process.argv.includes('--update') | |
|
|
||
| const reportPath = join(mkdtempSync(join(tmpdir(), 'wolves-baseline-')), 'vitest.json') | ||
|
|
||
| // Captured instead of swallowed: a non-zero exit is only acceptable when | ||
| // completed failed assertions account for it. A runner that failed to finish | ||
| // (suite-load error, setup error, import error) also exits non-zero, and that | ||
| // is exactly the case the gate used to report as passing. | ||
| let childExit = 0 | ||
| try { | ||
| execFileSync( | ||
| 'npx', | ||
|
|
@@ -44,8 +49,10 @@ try { | |
| }, | ||
| ) | ||
| } | ||
| catch { | ||
| // vitest exits non-zero whenever tests fail, which is the normal case here. | ||
| catch (error) { | ||
| childExit = Number.isInteger(error.status) | ||
| ? error.status | ||
| : 1 | ||
| } | ||
|
|
||
| if (!existsSync(reportPath)) { | ||
|
|
@@ -54,19 +61,71 @@ if (!existsSync(reportPath)) { | |
| } | ||
|
|
||
| const report = JSON.parse(readFileSync(reportPath, 'utf8')) | ||
|
|
||
| // Validate the report before interpreting it or writing an updated baseline. | ||
| // An incomplete report (no `testResults`) means nothing ran to completion, so | ||
| // it must never be read as "no new failures" nor recorded as a baseline. | ||
| if (!report || typeof report !== 'object' || !Array.isArray(report.testResults)) { | ||
| console.error('test-baseline: vitest report is incomplete or malformed') | ||
| process.exit(2) | ||
| } | ||
|
|
||
| const cwd = `${process.cwd()}/` | ||
|
|
||
| const current = new Set() | ||
| for (const file of report.testResults ?? []) { | ||
| // Files that failed (or reported a nameless failed assertion) without any | ||
| // completed failed assertion to explain it: suite-load / import / setup errors. | ||
| const unexplained = [] | ||
| for (const file of report.testResults) { | ||
| let named = 0 | ||
| let nameless = 0 | ||
| for (const assertion of file.assertionResults ?? []) { | ||
| // A suite that failed to load reports a failed assertion with no name; | ||
| // it is not a completed assertion and must not enter the failure set. | ||
| if (assertion.status === 'failed') { | ||
| current.add(`${file.name.replace(cwd, '')} :: ${assertion.fullName}`) | ||
| if (assertion.fullName) { | ||
| named++ | ||
| current.add(`${file.name.replace(cwd, '')} :: ${assertion.fullName}`) | ||
| } | ||
| else { | ||
| nameless++ | ||
| } | ||
| } | ||
| } | ||
| // A file-level error message (hook throw, import failure) is reported on the | ||
| // file itself, never on an assertion, so it flags the file even when other | ||
| // named failures in it are baseline-known. | ||
| const fileError = typeof file.message === 'string' && file.message.trim() !== '' | ||
| if (fileError || ((file.status === 'failed' || nameless > 0) && named === 0)) { | ||
| unexplained.push(String(file.name).replace(cwd, '')) | ||
| } | ||
| } | ||
|
|
||
| const sorted = [...current].sort() | ||
|
|
||
| // A non-zero child exit with no completed failed assertions is a run that did | ||
| // not finish: a suite failed to load, setup threw, or the runner errored. The | ||
| // normal red suite always has completed failed assertions to explain its exit | ||
| // code, so the absence of any is itself a failure. Reject it before the | ||
| // comparison and before any `--update` could record an empty baseline. | ||
| // The check is also per file: an unrelated known failure elsewhere must not | ||
| // mask a file that failed to load, so any such file rejects the run too. | ||
| if (unexplained.length > 0) { | ||
| console.error('test-baseline: file(s) failed without any completed failed assertion:') | ||
| for (const name of unexplained) { | ||
| console.error(` - ${name}`) | ||
| } | ||
| console.error('test-baseline: vitest failed to run to completion') | ||
| process.exit(2) | ||
| } | ||
|
|
||
| if (childExit !== 0 && current.size === 0) { | ||
|
Contributor
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. correctness - medium: this guard only fires when the whole run has zero named failures, so one known failure anywhere hides a suite that failed to load. Reproduced at this head with a stubbed Proposed fix: flag each file that vitest marks failed but that has no named failed assertion, and reject the run if any exist (replaces lines 75–98): const current = new Set()
+const unexplained = []
for (const file of report.testResults) {
+ let named = 0
for (const assertion of file.assertionResults ?? []) {
// A suite that failed to load reports a failed assertion with no name;
// it is not a completed assertion and must not enter the failure set.
if (assertion.status === 'failed' && assertion.fullName) {
current.add(`${file.name.replace(cwd, '')} :: ${assertion.fullName}`)
+ named++
}
}
+ // A file vitest marks failed with no completed failed assertion did not run
+ // (import/setup error). Failures in other files do not explain it.
+ if (file.status === 'failed' && named === 0) {
+ unexplained.push(file.name.replace(cwd, ''))
+ }
}
const sorted = [...current].sort()
-if (childExit !== 0 && current.size === 0) {
+if (unexplained.length > 0 || (childExit !== 0 && current.size === 0)) {
console.error(
'test-baseline: vitest failed to run to completion and reported no completed failures',
)
+ for (const name of unexplained) {
+ console.error(` ! ${name} failed without completed assertions`)
+ }
process.exit(2)
}Plus a regression test: non-empty baseline with one known failure in |
||
| console.error( | ||
| 'test-baseline: vitest failed to run to completion and reported no completed failures', | ||
| ) | ||
| process.exit(2) | ||
| } | ||
|
|
||
| if (shouldUpdate) { | ||
| writeFileSync(BASELINE, `${sorted.join('\n')}\n`) | ||
| console.info(`test-baseline: recorded ${sorted.length} known failures`) | ||
|
|
||
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
style (low): A file now enters
unexplainedbecause of a file-levelmessage, even when it has completed named failures (the new hook-error case). The diagnostic at theunexplainedbranch still printsfile(s) failed without any completed failed assertion, and the comment above theunexplaineddeclaration says the same. For the hook-error case that text is wrong and can send a contributor looking for a load failure. Suggest wording likefile(s) failed to run to completion (load/setup/hook error):and updating the comment to match. Non-blocking.