Skip to content

Commit 3924da9

Browse files
Huyvux12thymikeeclaude
authored
fix(coverage-changed): refuse stale or foreign reports via the coverage stamp (#4387)
* fix(coverage-changed): refuse stale or foreign reports via the coverage stamp The changed-line gate scored whatever lcov file it found, so a stale report or one written in another checkout read as a verdict — a foreign report even passed 0/0. It now refuses before scoring when the stamp beside the report is absent, names another commit or working tree, or records a narrowed run, and when lcov records files outside this checkout (the stamp cannot catch two checkouts of one commit). The refusal fires before the coverage-waiver label is consulted. Instrumentation now comes from the verified stamp's include/exclude, shared with the CRAP gate, and the duplicated SOURCE_ROOT policy is removed. Closes #4379 * fix(coverage-changed): address maintainer feedback on the stamp gate Refusal now fires before the waiver when the report has no stamp (run-level test pins it). The excluded-path tally is restored for stamp-excluded files an include root covers, so new logic under an exclude entry still shows in the job summary. Both gates share one outside-root scan via a predicate in stamp.ts, with no schema change. The COVERAGE_CONFIG fixture is shared between the model and run suites, and the CONTRIBUTING wording now names both the no-verdict and no-report messages. * fix(coverage-changed): widen test-material tally exclusion and share include matching The excluded tally exists to surface hidden product logic, not test code, but isTestFile missed test-kit helpers and fixtures files the stamp excludes, so their lines were tallied as hidden product logic. Widen the predicate to cover them, with a regression test that fails without the fix. Extract one include-match helper in stamp.ts so isInstrumentedPath and isIncludedPath cannot drift, and qualify the CONTRIBUTING sentence about the tally to code-like, non-test additions. * test(coverage-changed): mirror vitest test-material excludes in the shared fixture Adds **/*.fixtures.ts and packages/test-kit/** to COVERAGE_CONFIG, per maintainer feedback on #4387: the fixture should mirror the vitest config's test-material excludes instead of the tally test carrying its own extended copy. The test-kit/fixtures tally test now uses the shared fixture directly. * fix(coverage-changed): satisfy the fallow audit on the gate entrypoint and scoring Route the CLI entrypoint through the shared runEntrypoint owner instead of a copied try/catch block, and score each changed file in one helper so computeChangedCoverage stays under the complexity threshold. Behavior is unchanged. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> --------- Co-authored-by: Huyvux12 <Huyvux12@users.noreply.github.com> Co-authored-by: Michał Pierzchała <thymikee@gmail.com> Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
1 parent 9478a43 commit 3924da9

9 files changed

Lines changed: 438 additions & 84 deletions

File tree

‎CONTRIBUTING.md‎

Lines changed: 11 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -302,6 +302,17 @@ itself out of scope — the config's `exclude`, or a whole-file ignore comment.
302302
against 30 too, so a below-bar guess about new code is not reported at all; the stamp is what stops that
303303
guess from standing in for a measurement of code you changed.
304304

305+
The changed-line gate (`check:coverage-changed`) reads the same stamp: with no stamp, a stamp from
306+
another commit or working tree, a narrowed run, or an lcov report whose records name files outside this
307+
checkout, it prints `no trustworthy verdict` and names the reason instead of a PASS/FAIL table — once
308+
there is an lcov report to read; with no report at all it prints `no lcov report` and asks for a
309+
coverage run first. The fix is the same — re-run `pnpm test:coverage` on this commit — and the refusal
310+
fires before the `coverage-waiver` label is consulted, because a waiver skips a failure rather than
311+
blessing a report with no numbers to print. Which files owe coverage comes from the stamp's
312+
include/exclude: a file the config excludes is out of scope by declaration, but when an include root
313+
covers it, its code-like, non-test added lines are still tallied as excluded-path lines, as is a file the stamp instruments
314+
that lcov omits.
315+
305316
## Test app and Maestro compatibility
306317

307318
The Expo fixture app owns its setup, simulator/device, Metro, replay, and Maestro instructions in
Lines changed: 22 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,22 @@
1+
import type { CoverageConfig } from '../coverage-crap/stamp.ts';
2+
3+
/**
4+
* The stamp's include/exclude for these suites, mirroring the `test.coverage`
5+
* include/exclude in `vitest.config.ts` for the paths the tests exercise:
6+
* both coverage roots, test files out, test-kit helpers and fixtures files
7+
* out, `src/sdk/**` out. One copy, shared by the model and run suites, so a
8+
* config change cannot leave them disagreeing.
9+
*/
10+
export const COVERAGE_CONFIG: CoverageConfig = {
11+
include: ['src/**/*.ts', 'packages/*/src/**/*.ts'],
12+
exclude: [
13+
'**/*.test.ts',
14+
'**/__tests__/**',
15+
'**/*.fixtures.ts',
16+
'packages/test-kit/**',
17+
'**/*-types.ts',
18+
'**/types.ts',
19+
'src/sdk/**',
20+
'src/bin.ts',
21+
],
22+
};

‎scripts/coverage-changed/model.test.ts‎

Lines changed: 75 additions & 20 deletions
Original file line numberDiff line numberDiff line change
@@ -5,10 +5,10 @@ import {
55
CHANGED_LINE_COVERAGE_THRESHOLD,
66
computeChangedCoverage,
77
ignoredLineSet,
8-
isIncludableSource,
9-
isTestFile,
108
parseLcov,
119
} from './model.ts';
10+
import { isInstrumentedPath } from '../coverage-crap/stamp.ts';
11+
import { COVERAGE_CONFIG } from './fixtures.ts';
1212

1313
test('parseLcov reads DA hits and BRDA taken, treating "-" as not taken', () => {
1414
const lcov = [
@@ -34,23 +34,24 @@ test('parseLcov reads DA hits and BRDA taken, treating "-" as not taken', () =>
3434
assert.equal(index.get('src/b.ts')?.lineHits.get(10), 1);
3535
});
3636

37-
test('source/test classification matches the coverage include glob', () => {
38-
assert.equal(isIncludableSource('src/core/x.ts'), true);
39-
assert.equal(isIncludableSource('src/core/x.test.ts'), false);
40-
assert.equal(isIncludableSource('src/core/__tests__/x.ts'), false);
41-
assert.equal(isIncludableSource('scripts/y.ts'), false);
42-
assert.equal(isIncludableSource('src/core/x.tsx'), false);
37+
test('source/test classification matches the verified stamp selection', () => {
38+
assert.equal(isInstrumentedPath('src/core/x.ts', COVERAGE_CONFIG), true);
39+
assert.equal(isInstrumentedPath('src/core/x.test.ts', COVERAGE_CONFIG), false);
40+
assert.equal(isInstrumentedPath('src/core/__tests__/x.ts', COVERAGE_CONFIG), false);
41+
assert.equal(isInstrumentedPath('scripts/y.ts', COVERAGE_CONFIG), false);
42+
assert.equal(isInstrumentedPath('src/core/x.tsx', COVERAGE_CONFIG), false);
4343
// Workspace packages are the second coverage root.
44-
assert.equal(isIncludableSource('packages/contracts/src/x.ts'), true);
45-
assert.equal(isIncludableSource('packages/contracts/src/deep/x.ts'), true);
46-
assert.equal(isIncludableSource('packages/contracts/src/x.test.ts'), false);
47-
assert.equal(isIncludableSource('packages/contracts/src/__tests__/x.ts'), false);
48-
assert.equal(isIncludableSource('packages/contracts/src/x.tsx'), false);
49-
assert.equal(isIncludableSource('packages/contracts/test/x.ts'), false);
50-
assert.equal(isIncludableSource('packages/contracts/x.ts'), false);
51-
assert.equal(isTestFile('src/a.test.ts'), true);
52-
assert.equal(isTestFile('src/__tests__/a.ts'), true);
53-
assert.equal(isTestFile('src/a.ts'), false);
44+
assert.equal(isInstrumentedPath('packages/contracts/src/x.ts', COVERAGE_CONFIG), true);
45+
assert.equal(isInstrumentedPath('packages/contracts/src/deep/x.ts', COVERAGE_CONFIG), true);
46+
assert.equal(isInstrumentedPath('packages/contracts/src/x.test.ts', COVERAGE_CONFIG), false);
47+
assert.equal(isInstrumentedPath('packages/contracts/src/__tests__/x.ts', COVERAGE_CONFIG), false);
48+
assert.equal(isInstrumentedPath('packages/contracts/src/x.tsx', COVERAGE_CONFIG), false);
49+
assert.equal(isInstrumentedPath('packages/contracts/test/x.ts', COVERAGE_CONFIG), false);
50+
assert.equal(isInstrumentedPath('packages/contracts/x.ts', COVERAGE_CONFIG), false);
51+
// What the stamp excludes is out of scope by declaration, not merely unscored.
52+
assert.equal(isInstrumentedPath('src/sdk/thing.ts', COVERAGE_CONFIG), false);
53+
assert.equal(isInstrumentedPath('src/core/x-types.ts', COVERAGE_CONFIG), false);
54+
assert.equal(isInstrumentedPath('src/bin.ts', COVERAGE_CONFIG), false);
5455
});
5556

5657
test('ignoredLineSet handles next, next N and start/stop ranges', () => {
@@ -98,6 +99,7 @@ test('gate fails when changed-line coverage is below threshold and lists offende
9899
const result = computeChangedCoverage({
99100
diffs: [diff('src/covered.ts', [1, 2, 3])],
100101
coverage,
102+
stamp: COVERAGE_CONFIG,
101103
fileLines: () => null,
102104
});
103105
// lines 1 & 3 covered, line 2 uncovered -> 2/3 = 66.67% < 70.
@@ -118,6 +120,7 @@ test('workspace package source counts toward the gate like root source', () => {
118120
const result = computeChangedCoverage({
119121
diffs: [diff('packages/contracts/src/feature.ts', [1, 2])],
120122
coverage: packageCoverage,
123+
stamp: COVERAGE_CONFIG,
121124
fileLines: () => null,
122125
});
123126
// 1/2 = 50% < 70: a package change can now fail the gate on its own.
@@ -132,6 +135,7 @@ test('non-executable added lines (absent from DA) never enter the denominator',
132135
const result = computeChangedCoverage({
133136
diffs: [diff('src/covered.ts', [1, 3, 99])],
134137
coverage,
138+
stamp: COVERAGE_CONFIG,
135139
fileLines: () => null,
136140
});
137141
assert.equal(result.totalLines, 2);
@@ -144,6 +148,7 @@ test('deleted files and docs-only changes are a trivial pass with null pct', ()
144148
const result = computeChangedCoverage({
145149
diffs: [diff('src/covered.ts', [1], true), diff('README.md', [1, 2])],
146150
coverage,
151+
stamp: COVERAGE_CONFIG,
147152
fileLines: () => null,
148153
});
149154
assert.equal(result.pct, null);
@@ -155,6 +160,7 @@ test('waiver forces a pass but still reports the numbers', () => {
155160
const result = computeChangedCoverage({
156161
diffs: [diff('src/covered.ts', [1, 2, 3])],
157162
coverage,
163+
stamp: COVERAGE_CONFIG,
158164
waived: true,
159165
fileLines: () => null,
160166
});
@@ -163,7 +169,7 @@ test('waiver forces a pass but still reports the numbers', () => {
163169
assert.equal(result.pct, 66.67);
164170
});
165171

166-
test('an includable path whose new side is a symlink or gitlink is skipped, not read', () => {
172+
test('an instrumented path whose new side is a symlink or gitlink is skipped, not read', () => {
167173
// A committed symlink like `tools/helper-tests/Sources -> ../helper/Sources` shows in the
168174
// diff as added text (its target), so it reaches this loop by path shape. Its working-tree
169175
// entry resolves to a directory, so a content read dies on EISDIR; reading the
@@ -176,6 +182,7 @@ test('an includable path whose new side is a symlink or gitlink is skipped, not
176182
diff('src/covered.ts', [1, 2, 3]),
177183
],
178184
coverage,
185+
stamp: COVERAGE_CONFIG,
179186
fileLines: (path) => {
180187
readPaths.push(path);
181188
return null;
@@ -188,23 +195,70 @@ test('an includable path whose new side is a symlink or gitlink is skipped, not
188195
assert.deepEqual(result.excluded.files, []);
189196
});
190197

191-
test('includable source absent from lcov counts as excluded-path code lines', () => {
198+
test('instrumented source absent from lcov counts as excluded-path code lines', () => {
199+
const result = computeChangedCoverage({
200+
diffs: [diff('src/missing.ts', [1, 2, 3])],
201+
coverage,
202+
stamp: COVERAGE_CONFIG,
203+
fileLines: () => ['const kept = 1;', '', '// a comment'],
204+
});
205+
assert.equal(result.totalLines, 0);
206+
assert.equal(result.excluded.totalLines, 1);
207+
assert.equal(result.excluded.files[0]?.reason, 'excluded-path');
208+
assert.deepEqual(result.excluded.files[0]?.lines, [1]);
209+
});
210+
211+
test('a path the stamp excludes is tallied as excluded-path, not gated', () => {
192212
const result = computeChangedCoverage({
193213
diffs: [diff('src/sdk/thing.ts', [1, 2, 3])],
194214
coverage,
215+
stamp: COVERAGE_CONFIG,
195216
fileLines: () => ['const kept = 1;', '', '// a comment'],
196217
});
197218
assert.equal(result.totalLines, 0);
198219
assert.equal(result.excluded.totalLines, 1);
199220
assert.equal(result.excluded.files[0]?.reason, 'excluded-path');
200221
assert.deepEqual(result.excluded.files[0]?.lines, [1]);
222+
assert.equal(result.passed, true);
223+
});
224+
225+
test('a stamp-excluded test file or a path outside every include root is not tallied', () => {
226+
const result = computeChangedCoverage({
227+
diffs: [diff('src/sdk/thing.test.ts', [1, 2, 3]), diff('README.md', [1, 2])],
228+
coverage,
229+
stamp: COVERAGE_CONFIG,
230+
fileLines: () => ['const kept = 1;', '', '// a comment'],
231+
});
232+
assert.equal(result.totalLines, 0);
233+
assert.equal(result.excluded.totalLines, 0);
234+
assert.deepEqual(result.excluded.files, []);
235+
assert.equal(result.passed, true);
236+
});
237+
238+
test('stamp-excluded test-kit helpers and fixtures files are not tallied', () => {
239+
// The fixture mirrors the vitest config's test-material excludes, so they
240+
// are test material, not product logic hiding under an exclude entry.
241+
const result = computeChangedCoverage({
242+
diffs: [
243+
diff('packages/test-kit/src/helper.ts', [1, 2, 3]),
244+
diff('src/data.fixtures.ts', [1, 2]),
245+
],
246+
coverage,
247+
stamp: COVERAGE_CONFIG,
248+
fileLines: () => ['const kept = 1;', '', '// a comment'],
249+
});
250+
assert.equal(result.totalLines, 0);
251+
assert.equal(result.excluded.totalLines, 0);
252+
assert.deepEqual(result.excluded.files, []);
253+
assert.equal(result.passed, true);
201254
});
202255

203256
test('changed line suppressed by an ignore directive is tallied, not gated', () => {
204257
const cov = parseLcov(['SF:src/x.ts', 'DA:1,1', 'end_of_record'].join('\n'));
205258
const result = computeChangedCoverage({
206259
diffs: [diff('src/x.ts', [1, 3])],
207260
coverage: cov,
261+
stamp: COVERAGE_CONFIG,
208262
fileLines: () => ['const a = 1;', '/* v8 ignore next */', 'const suppressed = 3;'],
209263
});
210264
// Line 3 is code-like, dropped from DA by the directive on line 2, and inside
@@ -279,6 +333,7 @@ test('rename-only hunks owe no changed-line coverage; edited moves owe their edi
279333
uncoveredRecord('src/moved-pure.ts', 30),
280334
].join('\n'),
281335
),
336+
stamp: COVERAGE_CONFIG,
282337
fileLines: () => null,
283338
});
284339
// 2 ordinary edits + 3 edited move lines; the 30-line pure move owes nothing.

‎scripts/coverage-changed/model.ts‎

Lines changed: 74 additions & 50 deletions
Original file line numberDiff line numberDiff line change
@@ -3,15 +3,19 @@
33
// LINES ONLY. Only added (new-side) lines count — deleted lines never do — so
44
// the gate is deterministic against renames and deletes. run.ts owns all I/O.
55
//
6-
// The coverable universe is the lcov report itself, not a second copy of
7-
// vitest's exclude globs: coverage runs with `all` on, so every includable file
8-
// under either coverage root (`src/**/*.ts` and `packages/*/src/**/*.ts`)
9-
// appears in lcov even when untested. A changed includable file ABSENT from
10-
// lcov was dropped by an exclude glob — reported (non-gating) so exclusions
11-
// cannot silently absorb new logic.
6+
// The coverable universe is the verified stamp's include/exclude, not a second
7+
// copy of the coverage config: `isInstrumentedPath` against the stamp is what
8+
// decides whether a changed file owes coverage, the same selection the CRAP
9+
// gate reads from the same stamp. A changed instrumented file ABSENT from
10+
// lcov was dropped by the run — reported (non-gating) so the gap cannot
11+
// silently absorb new logic. A changed file the stamp excludes is out of scope
12+
// by declaration and owes no gating line — but when an include root covers it,
13+
// its code-like added lines still show in the non-gating excluded tally, so
14+
// new logic under an exclude entry cannot hide from the job summary either.
1215

1316
import type { ChangedFileDiff } from '../lib/repo-git.ts';
1417
import { normalizePath } from '../lib/repo-path.ts';
18+
import { isIncludedPath, isInstrumentedPath, type CoverageConfig } from '../coverage-crap/stamp.ts';
1519

1620
// Single source of truth for the gate. Changed-line coverage below this
1721
// percentage fails the Coverage CI job (unless waived).
@@ -73,22 +77,6 @@ export function parseLcov(text: string, rootDir?: string): CoverageIndex {
7377
return index;
7478
}
7579

76-
// Vitest coverage `include` spans two roots: `src/**/*.ts` and
77-
// `packages/*/src/**/*.ts`. Test files are excluded there and carry no product
78-
// logic, so they never count toward the gate or the excluded tally (which
79-
// exists to surface hidden logic, not test code).
80-
export function isTestFile(path: string): boolean {
81-
return /\.test\.ts$/.test(path) || /(^|\/)__tests__\//.test(path);
82-
}
83-
84-
// Both coverage roots, package-generic so a new workspace needs no edit here.
85-
// Anything outside them — scripts, package-level `test/`, `.tsx` — stays out.
86-
const SOURCE_ROOT = /^(?:packages\/[^/]+\/)?src\/.*\.ts$/;
87-
88-
export function isIncludableSource(path: string): boolean {
89-
return SOURCE_ROOT.test(path) && !isTestFile(path);
90-
}
91-
9280
// A changed line looks like product code (not blank, not a comment-only line).
9381
// Used only for the non-gating excluded tally, where no lcov signal exists.
9482
function isCodeLike(text: string | undefined): boolean {
@@ -205,6 +193,10 @@ function ignoredChangedLines(
205193
export type ComputeInput = {
206194
readonly diffs: readonly ChangedFileDiff[];
207195
readonly coverage: CoverageIndex;
196+
// The verified stamp's include/exclude: what the coverage run was told to
197+
// instrument. run.ts verifies the stamp before calling; a model test passes
198+
// the config the case is about.
199+
readonly stamp: CoverageConfig;
208200
// New-side content of a changed file (working tree at HEAD), or null when the
209201
// path cannot be read. Only consulted for the non-gating excluded tally.
210202
readonly fileLines: (path: string) => readonly string[] | null;
@@ -215,39 +207,71 @@ function sum<T>(items: readonly T[], pick: (item: T) => number): number {
215207
return items.reduce((acc, item) => acc + pick(item), 0);
216208
}
217209

210+
// Test code never counts toward the gate or the excluded tally, which exists
211+
// to surface hidden product logic, not test code. Test-kit helpers and
212+
// fixtures files are test material too — the coverage config excludes them
213+
// alongside test files — so their added lines stay out of the tally as well.
214+
function isTestFile(path: string): boolean {
215+
return (
216+
/\.test\.ts$/.test(path) ||
217+
/(^|\/)__tests__\//.test(path) ||
218+
/\.fixtures\.ts$/.test(path) ||
219+
path.startsWith('packages/test-kit/')
220+
);
221+
}
222+
223+
// Code-like added lines of a file the run did not measure, as the non-gating
224+
// excluded tally entry: a gap the summary must show, never a denominator line.
225+
function excludedPathReport(
226+
diff: ChangedFileDiff,
227+
lines: readonly string[] | null,
228+
): ExcludedFileReport | null {
229+
const excluded = diff.added.filter((n) => isCodeLike(lines?.[n - 1]));
230+
return excluded.length > 0 ? { path: diff.path, reason: 'excluded-path', lines: excluded } : null;
231+
}
232+
233+
type ChangedFileOutcome = {
234+
readonly file?: FileCoverageReport;
235+
readonly excluded?: ExcludedFileReport | null;
236+
};
237+
238+
// One changed file's contribution: a gated report, a non-gating excluded entry, both, or neither.
239+
function scoreChangedFile(diff: ChangedFileDiff, input: ComputeInput): ChangedFileOutcome {
240+
// A non-blob side (symlink, gitlink) is not source: its working-tree entry can
241+
// resolve to a directory, which no content read may attempt.
242+
if (diff.deleted || !diff.regularFile || diff.added.length === 0) return {};
243+
244+
if (!isInstrumentedPath(diff.path, input.stamp)) {
245+
// Out of scope by declaration — but a file an include root covers still
246+
// shows its added lines in the non-gating excluded tally, the way the
247+
// gate did before the stamp owned selection. A test file or a path
248+
// outside every include root owes not even that.
249+
const owesTally = isIncludedPath(diff.path, input.stamp) && !isTestFile(diff.path);
250+
return owesTally ? { excluded: excludedPathReport(diff, input.fileLines(diff.path)) } : {};
251+
}
252+
253+
const cov = input.coverage.get(diff.path);
254+
// Instrumented source absent from the lcov report: the run recorded no
255+
// counters for it. Count its code-like added lines so the gap is visible.
256+
if (!cov) return { excluded: excludedPathReport(diff, input.fileLines(diff.path)) };
257+
258+
const contentLines = input.fileLines(diff.path);
259+
const ignored = contentLines ? ignoredChangedLines(diff.added, cov, contentLines) : [];
260+
return {
261+
file: scoreGateFile(diff.path, diff.added, cov),
262+
excluded:
263+
ignored.length > 0 ? { path: diff.path, reason: 'ignored-lines', lines: ignored } : null,
264+
};
265+
}
266+
218267
export function computeChangedCoverage(input: ComputeInput): ChangedCoverageResult {
219268
const files: FileCoverageReport[] = [];
220269
const excludedFiles: ExcludedFileReport[] = [];
221270

222271
for (const diff of input.diffs) {
223-
// A non-blob side (symlink, gitlink) is not source: its working-tree entry can
224-
// resolve to a directory, which no content read may attempt.
225-
if (
226-
diff.deleted ||
227-
!diff.regularFile ||
228-
diff.added.length === 0 ||
229-
!isIncludableSource(diff.path)
230-
)
231-
continue;
232-
233-
const cov = input.coverage.get(diff.path);
234-
if (!cov) {
235-
// Includable source absent from the all-files lcov report: an exclude glob
236-
// dropped it. Count its code-like added lines so the exclusion is visible.
237-
const lines = input.fileLines(diff.path);
238-
const excluded = diff.added.filter((n) => isCodeLike(lines?.[n - 1]));
239-
if (excluded.length > 0) {
240-
excludedFiles.push({ path: diff.path, reason: 'excluded-path', lines: excluded });
241-
}
242-
continue;
243-
}
244-
245-
files.push(scoreGateFile(diff.path, diff.added, cov));
246-
const contentLines = input.fileLines(diff.path);
247-
const ignored = contentLines ? ignoredChangedLines(diff.added, cov, contentLines) : [];
248-
if (ignored.length > 0) {
249-
excludedFiles.push({ path: diff.path, reason: 'ignored-lines', lines: ignored });
250-
}
272+
const outcome = scoreChangedFile(diff, input);
273+
if (outcome.file) files.push(outcome.file);
274+
if (outcome.excluded) excludedFiles.push(outcome.excluded);
251275
}
252276

253277
const coveredLines = sum(files, (f) => f.coveredLines);

0 commit comments

Comments
 (0)