diff --git a/docs/plans/2026-07-27-observable-gates-plan.md b/docs/plans/2026-07-27-observable-gates-plan.md new file mode 100644 index 00000000..d72bae17 --- /dev/null +++ b/docs/plans/2026-07-27-observable-gates-plan.md @@ -0,0 +1,114 @@ +# Observable-Gates Plan — killing the "trust the proxy, not the outcome" false-green class + +Source: 2-round consultation with the 4-model panel (deepseek-pro, glm, grok, codex), +seeded with the Reddit/DeepSeek thesis and validated against the tsforge harness. All +four converged; disagreements adjudicated below. Two of my own code-verifications anchor it. + +## Diagnosis (unanimous) + +The dominant class of **runtime/product** false-greens (the jump from "well-formed code" +to "shipped behavior") is: the harness scores that an action **ran** (exit 0 / substring +present / tests green / judge said pass) instead of asserting the final **observable +state**. It's structural, not a few incidents — the same shape recurs at db:push (#200, +#204), reachability (#202), resume baseline (#198), hollow-UI/wiring, and the review panel +(META: the panel passed a fix by inheriting its framing; only a live build falsified it). +Corollary the panel corrected me on: **most per-cycle reds are honest** (lint/type/knip) — +the lie is specifically at the code→behavior boundary. So the fix is targeted, not a rewrite. + +## Verified facts (I checked the code) + +1. **`noE2eAcceptance` is real gate-relaxation.** In `build.ts runFinalAcceptance`, a failed + FULL gate (validate/build/size/root-drift) only flips to `stuck` when e2e is *enabled*; + with the flag set it emits a "stuck" event but `return result` keeps the passed status. + The `boringstack-final-acceptance.test.ts` "FIX 5" case codifies this as intended. +2. **`settleGate` does NOT green-cache** — it re-runs `evaluateGate` every cycle. The only + real caches are the panel verdict cache (`harness-review-mode.ts`) and the differential + baseline. (Kills a round-1 red herring.) +3. **Polish-recheck failure rolls back to the pre-polish green** (a genuine green), so it's + defensible; residual risk is only an incomplete rollback (files outside snapshot / DB). + +## Plan (ordered; per-cycle cheap, expensive stuff to CI/live-qualification) + +### P0a — Make the full gate terminal regardless of `noE2eAcceptance` (cheap, ~0 cost) +- Seam: `build.ts runFinalAcceptance` (the `!finalPassed && !e2eAcceptanceDisabled` branch, + ~line 873); invert `boringstack-final-acceptance.test.ts` "FIX 5". The flag skips only the + browser/chain e2e — a failed validate/build/size/root-drift is ALWAYS terminal. +- Observable: full gate red ⇒ status `stuck` and headless exits nonzero, flag or not. +- Proof: unit test both flag states against a failing full gate → both `stuck`; live repro: + set the flag, inject deterministic root-drift/size failure, assert exit 1 (was exit 0/done). + +### P0b — DB boundary oracle after every successful `db:push` (per-cycle, ~1–2s/entity) +- Seam: new `boringstack/db-oracle.ts`, invoked right after `dbPushForce` in + `gate-stages.ts` (thread the entity spec already derived in `build.ts`); query the same + `DATABASE_URL`. +- Expected columns are **harness-derived from the PLAN, never the model's schema file**: + table = `toCamelCase(entity.id)` in schema `app`; required set = + `{id, user_id, created_at, updated_at}` ∪ plan fields ∪ `{relEntity}_id` for each non-User + belongs-to (reuse the relationship derivation in `acceptance-spec.ts`; `belongs to User` → + existing `user_id`). **SUBSET, not exact** (scaffold/model may add extra columns — don't + fail on extras). Assert column **presence** (SQL type families are too coarse for the + current plan types → at most a soft check). Match names **tolerant of camel/snake** + (verified: the live DB uses `user_id`/`created_at`; domain cols use the field name as + written) — normalize (strip `_`, lowercase) both sides. Stub `name` required only if the + plan declares `name`. +- Two-phase: the pre-model scaffold push can't have domain fields yet → assert only the + fixed infra columns there; apply the full contract after model edits. +- Proof: exit-0 push whose DB lacks `title` for `Bookmark{title,url}` ⇒ `db-schema-mismatch` + BEFORE validate runs; extra columns pass; replay valbuild27/#204 + #200 → red same cycle. + +### P1 — Reachability = real HTTP + DB-derived sentinel (per-cycle, one fetch + one SELECT) +- Seam: replace the source-substring path in `reachability.ts` with an authenticated request + after the command gate (keep static as a cheap prefilter if useful). `fetch`, NOT Playwright. +- Observable: seed a unique sentinel row via the real create path, `GET` the collection/detail, + assert the response body carries that sentinel (tri-point: plan → seed → DB row → HTTP body). + A 200 with a template/stub body fails. +- Proof: route registered but handler returns `[]`/stub, or route unmounted, or API-only/DB-only + → each red. Live: hollow-API greens that pass today. + +### P2 — Sentinel identity in the per-slice e2e (finish the partial) (CI/final acceptance) +- Seam: `e2e-generator.ts` / `e2e-runner.ts`. The one harness identity must appear in the UI + row AND an authenticated API GET AND a direct DB SELECT (extend the existing unique-value row + check). Keep create/edit/delete→reload orderings. Add persisted-FK verification to the chain. +- Proof: valbuild19 hollow Contact/Deal, build49 dead hooks → red; a usable Company → green. + +### P2 — Adversarial orderings that are actually ours (CI/live-qualification only) +- Keep: **schema-evolve** (stub `name` → domain cols; the #204 trigger) and **reload/restart-persist** + (sentinel survives). **Drop generic concurrent-rewrite** — single-user CRUD has no conflict semantics. + +### P3 — Harness-honesty "disable-must-fail" suite (CI only, NOT per-cycle) +- Seam: a mutant runner under `packages/core/scripts/` wired into `core-ci.yml`. +- Observable: deliberately break each control (drop a column, unmount a route, strip a testid, + mock an exit-0 no-op db:push, disable the P0a terminal branch) → the suite MUST go red. +- Require this artifact only for PRs that touch behavioral-control seams (not every PR). +- Proof: mutants replay #204, the disabled-e2e/full-red repro, and 200-without-sentinel. + +### P4 — Panel cache + framing integrity (pre-push/CI) +- Seam: `harness-review-mode.ts`. Always run current `validate` before trusting a cached + verdict; key the cache by diff+base+intent+reviewer-roster+quick/full mode (agreed win). +- Framing control: adopt the cheap **fails-when-disabled artifact** for harness-behavioral PRs + now. Treat glm's **framing-free counterfactual reviewer** (one reviewer sees only the DB + schema diff + plan field list, never the narrative) as an EXPERIMENT — A/B it on planted + defects before making it blocking (codex/grok caution: could be bureaucratic/proxy). + +## Explicitly DROPPED (low-ROI / over-correction — unanimous) +- Generic concurrent-rewrite / race suites (not our failure mode). +- `settleGate` green-cache work (falsified — it re-runs every cycle). +- Per-cycle Playwright or full 4-model panel. +- Blanket secret/logging sink sweep — plan-gated only, if a plan ever declares logging/secrets. +- Auto-classifying Postgres error text as red/green (unwinnable, already a known tar-pit). +- Deriving oracle expected-columns from the model's migration/schema file. +- deepseek's reverse/exact-equality column check (majority: subset-presence only). +- Polish-rollback rework as a P0 (defensible; residual partly caught by P0b anyway). + +## Guardrails (unanimous) +Never relax the gate; every new check is ADDITIVE and turns the gate RED (never redefines +green). Deterministic cheap oracles (SQL/HTTP) run per-cycle; browser + mutation + adversarial +run at final acceptance / CI only (per-cycle Playwright latency+flakiness trains the model to +ignore reds). Boundary asserts derive from the PLAN, not the model's own output. Flakiness is +browser/env, not SQL — fix isolation, don't retreat to mocks. + +## Sequencing recommendation +P0a first (1-line-ish, restores gate integrity, ~0 cost), then P0b (the primary runtime +truth-source), each shipped as its own PR with an observable test + a live-build repro, panel- +reviewed. Then P1. P2–P4 as follow-ons. One boundary check at a time, each proven by making +the OLD behavior go red. diff --git a/packages/core/src/cli/harness-review-mode.ts b/packages/core/src/cli/harness-review-mode.ts index 68aaeff4..1ba7a216 100644 --- a/packages/core/src/cli/harness-review-mode.ts +++ b/packages/core/src/cli/harness-review-mode.ts @@ -1,6 +1,6 @@ import { mkdir, mkdtemp, readFile, writeFile, rm } from "node:fs/promises"; import { join } from "node:path"; -import { createHash, randomUUID } from "node:crypto"; +import { randomUUID } from "node:crypto"; import { tmpdir } from "node:os"; import { isRecord } from "../lib/guards"; import { OpenAICompatibleProvider, type IProvider } from "../inference"; @@ -11,18 +11,22 @@ import { type IModelEntry, type BinaryInputMode, } from "../models-config"; -import { resolvePanel } from "../reviewers/registry"; +import { resolvePanel, type IPanel } from "../reviewers/registry"; import { - runHarnessReview, + gatherChange, + reviewRequest, + runReviewFlow, + panelIdentityHash, DEFAULT_MAX_FILES, DEFAULT_MAX_CHARS, - verdictCacheKey, artifactBody, shouldCacheVerdict, honorCachedVerdict, - CACHE_VERSION, + type IReviewFlowDeps, + type IReviewDeps, + type IGitRunner, + type IValidateRunner, } from "../reviewers/harness-review"; -import { RUBRIC_VERSION } from "../reviewers/schema"; import { parseVerdict, type IVerdict } from "../reviewers/aggregate"; interface IArgs { @@ -200,10 +204,6 @@ async function validateRunner(): Promise<{ return { passed: code === 0, failCount: firstErrors.length, firstErrors }; } -function computePanelHash(panel: object): string { - return createHash("sha256").update(JSON.stringify(panel)).digest("hex"); -} - export const CACHE_DIR = join(".tsforge", "harness-review"); async function readCachedVerdict( @@ -278,6 +278,66 @@ export function formatVerdict(v: IVerdict): string { return lines.join("\n"); } +/** + * Wire the resolved CLI pieces (full panel + quick flag, args, git/validate, providers, cache + * seams) into the runReviewFlow deps — exported so the CLI's central wiring is unit-tested. + * The EFFECTIVE roster is derived HERE (quick mode → a 1-reviewer slice), so the cache key's + * rosterHash and the review both target that effective roster — a `quick` run can't reuse a + * full-panel verdict, or vice versa. mode/ci come from the args; gather reads args.base/intent. + * A miswire (cfg roster, hardcoded ci, wrong panel, un-sliced quick roster) is caught here. + */ +export function buildReviewFlowDeps(input: { + panel: IPanel; + identity: string; + quick: boolean; + ci: boolean; + base: string | undefined; + intent: string | undefined; + git: IGitRunner; + validate: IValidateRunner; + makeProvider: IReviewDeps["makeProvider"]; + runBinary: IReviewDeps["runBinary"]; + readCache: (key: string) => Promise; + persistArtifact: ( + verdict: IVerdict, + key: string, + rosterHash: string + ) => Promise; +}): IReviewFlowDeps { + // `quick` reviews with a REDUCED roster (the first reviewer only). The effective roster + // feeds BOTH the cache key and the review, so its verdict never satisfies a full review. + const effective: IPanel = input.quick + ? { ...input.panel, reviewers: input.panel.reviewers.slice(0, 1) } + : input.panel; + const rosterHash = panelIdentityHash(effective, input.identity); + + return { + gather: () => + gatherChange( + { git: input.git, validate: input.validate }, + { + base: input.base, + intent: input.intent, + maxFiles: DEFAULT_MAX_FILES, + maxChars: DEFAULT_MAX_CHARS, + } + ), + identity: input.identity, + rosterHash, + mode: input.quick ? "quick" : "full", + ci: input.ci, + readCache: input.readCache, + review: (request) => + reviewRequest(request, { + makeProvider: input.makeProvider, + runBinary: input.runBinary, + panel: effective, + identity: input.identity, + }), + persist: (v, key) => input.persistArtifact(v, key, rosterHash), + }; +} + export async function harnessReviewMode(argv: string[]): Promise { const args = parse(argv); @@ -297,70 +357,35 @@ export async function harnessReviewMode(argv: string[]): Promise { process.stdout.write(`skipped reviewer ${s.id}: ${s.reason}\n`); } - const effective = args.quick - ? { ...panel, reviewers: panel.reviewers.slice(0, 1) } - : panel; - const treeHashRes = await gitRunner(["write-tree"]); const treeHash = treeHashRes.stdout.trim(); - const panelHash = computePanelHash(cfg.reviewPanel ?? {}); - const cacheKey = verdictCacheKey({ - treeHash, - panelHash, - rubricVersion: RUBRIC_VERSION, - cacheVersion: CACHE_VERSION, - }); - - let verdict: IVerdict; - - if (!args.ci) { - const cached = await readCachedVerdict(cacheKey); - - if (cached !== null) { - process.stdout.write("harness-review: cache hit, reusing verdict\n"); - verdict = cached; - } else { - verdict = await runHarnessReview( - { - git: gitRunner, - validate: validateRunner, - makeProvider, - runBinary, - panel: effective, - identity: `${active.name}/${active.entry.model}`, - }, - { - base: args.base, - intent: args.intent, - maxFiles: DEFAULT_MAX_FILES, - maxChars: DEFAULT_MAX_CHARS, - } - ); - - // persistVerdict caches ONLY a real panel verdict. A pre-review gate block - // (validate flake, empty intent, diff too large) is transient — caching one - // poisons the tree-hash so a flaky validate under load blocks every future push. - await persistVerdict(verdict, cacheKey, treeHash, panelHash); - } - } else { - verdict = await runHarnessReview( - { - git: gitRunner, - validate: validateRunner, - makeProvider, - runBinary, - panel: effective, - identity: `${active.name}/${active.entry.model}`, - }, - { - base: args.base, - intent: args.intent, - maxFiles: DEFAULT_MAX_FILES, - maxChars: DEFAULT_MAX_CHARS, - } - ); - - await persistVerdict(verdict, cacheKey, treeHash, panelHash); + const identity = `${active.name}/${active.entry.model}`; + + // runReviewFlow enforces the wiring invariant: GATHER (validate runs fresh inside) BEFORE + // any cache access, and a gather block never touches the cache. The gathered request is + // keyed from its OWN bytes, so key and review can't diverge. --ci writes but never reads. + // buildReviewFlowDeps derives the EFFECTIVE roster (quick-slice) and the roster hash, and is + // unit-tested for that wiring. + const { verdict, cacheHit } = await runReviewFlow( + buildReviewFlowDeps({ + panel, + identity, + quick: args.quick, + ci: args.ci, + base: args.base, + intent: args.intent, + git: gitRunner, + validate: validateRunner, + makeProvider, + runBinary, + readCache: readCachedVerdict, + persistArtifact: (v, key, rosterHash) => + persistVerdict(v, key, treeHash, rosterHash), + }) + ); + + if (cacheHit) { + process.stdout.write("harness-review: cache hit, reusing verdict\n"); } process.stdout.write(`${formatVerdict(verdict)}\n`); diff --git a/packages/core/src/reviewers/harness-review.ts b/packages/core/src/reviewers/harness-review.ts index c42e9a59..8dcc93dd 100644 --- a/packages/core/src/reviewers/harness-review.ts +++ b/packages/core/src/reviewers/harness-review.ts @@ -1,5 +1,5 @@ import { createHash } from "node:crypto"; -import type { IPanel } from "./registry"; +import type { IPanel, ResolvedReviewer } from "./registry"; import { reviewerInvoke, type IInvokeDeps } from "./invoke"; import { aggregate, type IVerdict } from "./aggregate"; import { @@ -41,46 +41,78 @@ export type GatherResult = | { kind: "request"; request: IReviewRequest } | { kind: "block"; reason: string }; -async function resolveBase( +/** Resolve a commit-ish to an immutable SHA, or null if it can't be resolved. A pin must be a + * real SHA: rev-parse succeeding with EMPTY stdout (or a nonzero exit) is NOT a valid pin — + * returning the movable ref instead would re-open the mid-gather TOCTOU this exists to close. */ +async function resolveSha( git: IGitRunner, - explicit: string | undefined -): Promise { - if (explicit !== undefined && explicit.length > 0) { - return explicit; - } + ref: string +): Promise { + const res = await git(["rev-parse", ref]); + const sha = res.stdout.trim(); - const res = await git(["merge-base", "main", "HEAD"]); + return res.code === 0 && sha.length > 0 ? sha : null; +} + +async function resolveBase( + git: IGitRunner, + explicit: string | undefined, + head: string +): Promise { + const ref = explicit !== undefined && explicit.length > 0 ? explicit : "main"; + // Pin to the merge-base SHA against the PINNED head. The diff is `${base}...${head}` + // (three-dot), whose true origin is merge-base(ref, head); an immutable SHA means the bytes + // the fingerprint hashes and the bytes the review diffs are ONE snapshot even if `ref` moves. + const res = await git(["merge-base", ref, head]); const base = res.stdout.trim(); - return base.length > 0 ? base : "HEAD~1"; + if (res.code === 0 && base.length > 0) { + return base; // merge-base output IS a SHA + } + + // merge-base failed (e.g. shallow clone, no `main`): pin the fallback ref to a SHA too — + // NEVER diff against a movable ref, or the name-only / content diffs could observe different + // commits if the ref moves mid-gather. `head` is a SHA, so `${head}~1` is a stable target. + return resolveSha( + git, + explicit !== undefined && explicit.length > 0 ? explicit : `${head}~1` + ); } async function resolveIntent( git: IGitRunner, - explicit: string | undefined + explicit: string | undefined, + head: string ): Promise { if (explicit !== undefined && explicit.trim().length > 0) { return explicit.trim(); } - const subject = (await git(["log", "-1", "--format=%s"])).stdout.trim(); + const subject = (await git(["log", "-1", "--format=%s", head])).stdout.trim(); return GENERIC_INTENTS.has(subject.toLowerCase()) ? null : subject; } -async function changedFiles(git: IGitRunner, base: string): Promise { - const res = await git(["diff", "--name-only", `${base}...HEAD`]); - - return res.stdout - .split("\n") - .map((l) => l.trim()) - .filter((l) => l.length > 0); -} - export async function gatherChange( deps: IGatherDeps, opts: IGatherOptions ): Promise { + // Pin HEAD to an immutable SHA FIRST — before validate and every diff — so the whole gather + // references one snapshot (a HEAD move mid-gather can't mix file names, diff, and context + // from different commits into one request). A pin that isn't a real SHA BLOCKS: no soft + // fallback to the movable "HEAD" ref, which would re-open the TOCTOU. (Reviewing an arbitrary + // pushed OID — validating THAT commit's tree, not the working tree — is a separate + // review-targeting concern, tracked apart from cache integrity.) + const head = await resolveSha(deps.git, "HEAD"); + + if (head === null) { + return { + kind: "block", + reason: + "could not resolve HEAD to a commit (git rev-parse) — cannot review", + }; + } + const validateSummary = await deps.validate(); if (!validateSummary.passed) { @@ -90,8 +122,17 @@ export async function gatherChange( }; } - const base = await resolveBase(deps.git, opts.base); - const intent = await resolveIntent(deps.git, opts.intent); + const base = await resolveBase(deps.git, opts.base, head); + + if (base === null) { + return { + kind: "block", + reason: + "could not resolve the diff base to a commit (git merge-base/rev-parse) — cannot review", + }; + } + + const intent = await resolveIntent(deps.git, opts.intent, head); if (intent === null) { return { @@ -101,7 +142,29 @@ export async function gatherChange( }; } - const files = await changedFiles(deps.git, base); + // Read the changed-file list AND check git's exit code. Ignoring it lets a failed diff + // (empty stdout on error) build an EMPTY review that the panel green-lights, then caches + // under the real request's key — a false green. A failure blocks instead. + const namesRes = await deps.git(["diff", "--name-only", `${base}...${head}`]); + + if (namesRes.code !== 0) { + return { + kind: "block", + reason: `could not compute the changed-file list (git diff exited ${String(namesRes.code)}) — cannot review`, + }; + } + + const files = namesRes.stdout + .split("\n") + .map((l) => l.trim()) + .filter((l) => l.length > 0); + + if (files.length === 0) { + return { + kind: "block", + reason: `no changes between ${base} and ${head} to review`, + }; + } if (files.length > opts.maxFiles) { return { @@ -110,7 +173,27 @@ export async function gatherChange( }; } - const diff = (await deps.git(["diff", `${base}...HEAD`])).stdout; + const diffRes = await deps.git(["diff", `${base}...${head}`]); + + if (diffRes.code !== 0) { + return { + kind: "block", + reason: `could not compute the diff (git diff exited ${String(diffRes.code)}) — cannot review`, + }; + } + + const diff = diffRes.stdout; + + if (diff.length === 0) { + // Defense in depth: files were listed but the CONTENT diff came back empty. (A real + // rename/mode-only change is NOT empty — git emits `similarity index` / `old mode` etc. + // — so this is an anomaly: a git quirk, or a `--name-only`/`diff` disagreement.) There is + // nothing for the reviewers to judge, so block rather than build+cache a vacuous review. + return { + kind: "block", + reason: `the diff between ${base} and ${head} is empty despite listed changes — nothing to review`, + }; + } if (diff.length > opts.maxChars) { return { @@ -119,7 +202,12 @@ export async function gatherChange( }; } - const contextFiles = await gatherContext(deps.git, files, opts.maxChars); + const contextFiles = await gatherContext( + deps.git, + files, + head, + opts.maxChars + ); return { kind: "request", @@ -142,6 +230,7 @@ export async function gatherChange( async function gatherContext( git: IGatherDeps["git"], files: string[], + head: string, budget: number ): Promise { const blocks: string[] = []; @@ -149,10 +238,10 @@ async function gatherContext( let omitted = 0; for (const file of files) { - const res = await git(["show", `HEAD:${file}`]); + const res = await git(["show", `${head}:${file}`]); if (res.code !== 0) { - continue; // deleted/renamed/binary — not readable at HEAD, skip + continue; // deleted/renamed/binary — not readable at this commit, skip } const block = `=== ${file} ===\n${res.stdout}`; @@ -175,12 +264,11 @@ async function gatherContext( return blocks; } -export interface IRunDeps extends IGatherDeps, IInvokeDeps { - panel: IPanel; - identity: string; -} - -function blockedVerdict(reason: string, identity: string): IVerdict { +/** A pre-review gate/precondition block (validate red, empty intent, empty/oversized diff, + * git failure). The panel did NOT run. Marked `preReview` so it is never cached — a + * transient block must not poison the cache and reject every later push. Exported so the + * CLI produces it directly (it gathers the request itself, then reviews). */ +export function blockedVerdict(reason: string, identity: string): IVerdict { return { blocked: true, reason, @@ -188,24 +276,24 @@ function blockedVerdict(reason: string, identity: string): IVerdict { ranked: [], perReviewer: [], identity, - // Pre-review gate/precondition block — the panel did not run. Marked so the - // caller never caches it (a transient validate flake must not poison the - // tree-hash and block every later push). preReview: true, }; } -export async function runHarnessReview( - deps: IRunDeps, - opts: IGatherOptions -): Promise { - const gathered = await gatherChange(deps, opts); - - if (gathered.kind === "block") { - return blockedVerdict(gathered.reason, deps.identity); - } +export interface IReviewDeps extends IInvokeDeps { + panel: IPanel; + identity: string; +} - const outcomes = await reviewerInvoke(deps.panel, gathered.request, { +/** Run the panel on an ALREADY-GATHERED request (validate + diff + context already built). + * Split from the gather step so the CLI can fingerprint the exact request for the cache + * key BEFORE deciding whether to invoke the models — key and review then hash the same + * bytes by construction. */ +export async function reviewRequest( + request: IReviewRequest, + deps: IReviewDeps +): Promise { + const outcomes = await reviewerInvoke(deps.panel, request, { makeProvider: deps.makeProvider, runBinary: deps.runBinary, }); @@ -217,29 +305,150 @@ export async function runHarnessReview( } /** Verdict-cache schema version. Bump to invalidate ALL previously written cache - * artifacts in one shot. Bumped to "2" when pre-review gate blocks stopped being - * cached: legacy "1" artifacts can hold a poisoned "validate failed" block with no - * `preReview` marker, and without a version change readCachedVerdict would keep - * serving them and block every push of that tree. */ -export const CACHE_VERSION = "2"; - -export function verdictCacheKey(input: { - treeHash: string; - panelHash: string; - rubricVersion: string; - cacheVersion: string; -}): string { + * artifacts in one shot. Bumped to "3" when the key changed from a diff-hash to a full + * request fingerprint (below): legacy artifacts key on incomparable inputs, so the bump + * retires them rather than risk a stale-input collision. */ +export const CACHE_VERSION = "3"; + +/** Deterministic JSON: recursively sort object keys so equal CONTENT hashes equally + * regardless of construction/insertion order. A cache key must not thrash — or, if a + * second request-construction path ever appears, diverge — because two equal requests + * serialized their keys in a different order. Exported so the cache-key pin test can + * recompute the exact digest (and catch an accidental drop of CACHE_VERSION from the key). */ +export function canonicalJson(value: unknown): string { + const canon = (v: unknown): unknown => { + if (Array.isArray(v)) { + return v.map(canon); + } + + if (typeof v === "object" && v !== null) { + const out: Record = {}; + + for (const [k, val] of Object.entries(v).sort(([a], [b]) => + a < b ? -1 : a > b ? 1 : 0 + )) { + out[k] = canon(val); + } + + return out; + } + + return v; + }; + + return JSON.stringify(canon(value)); +} + +/** + * Fingerprint the EXACT review request the reviewers will judge, plus the roster identity + * and mode. This is the cache key. + * + * It hashes the WHOLE `request` object — the same object handed to `reviewRequest` — not a + * hand-picked subset. That makes key and review provably one input: any byte a reviewer + * sees (diff, the full `contextFiles`, intent, rubricVersion, AND the `validateSummary` + * including its `firstErrors`/`failCount`, which the panel reads) is in the key. Selecting a + * subset was unsound — e.g. `validateRunner` can return `passed:true` with a non-empty + * `firstErrors`, so two runs with an identical diff but different validate diagnostics feed + * the reviewers different bytes; omitting the summary would false-reuse across them. + * Serialized canonically (key-sorted) so equal content always yields the same digest. + */ +export function reviewRequestKey( + request: IReviewRequest, + opts: { rosterHash: string; mode: string } +): string { return createHash("sha256") - .update( - `${input.treeHash} ${input.panelHash} ${input.rubricVersion} ${input.cacheVersion}` - ) + .update(canonicalJson([request, opts.rosterHash, opts.mode, CACHE_VERSION])) .digest("hex"); } +/** + * Identity of the roster that ACTUALLY reviewed. Hashes the FULL resolved reviewer objects + * (kind, id, model `entry`, binary argv/input mode, endpoint, timeout — every field), not + * just their ids: retargeting the SAME id to a different model/endpoint/binary yields a + * different reviewer IMPLEMENTATION whose verdict must not be reused. Plus the quorum and + * the builder identity (independence differs per builder). Sorted by id so resolution order + * doesn't churn the key. This feeds the cache key; keying on ids alone (or the raw config) + * missed reviewer-implementation changes, which the panel flagged. + */ +export function panelIdentityHash( + panel: { reviewers: readonly ResolvedReviewer[]; minReviewers: number }, + builderIdentity: string +): string { + const roster = [...panel.reviewers].sort((a, b) => + a.id < b.id ? -1 : a.id > b.id ? 1 : 0 + ); + + return createHash("sha256") + .update(canonicalJson([roster, panel.minReviewers, builderIdentity])) + .digest("hex"); +} + +export interface IReviewFlowDeps { + /** Gather the review request — runs validate FRESH and builds the request (or blocks). */ + gather: () => Promise; + identity: string; + /** Fingerprint of the effective roster + builder (panelIdentityHash). */ + rosterHash: string; + /** "quick" | "full". */ + mode: string; + ci: boolean; + /** Read a cached verdict by key (already applies the pre-review guard); null = miss. */ + readCache: (key: string) => Promise; + /** Run the live panel on the gathered request (only reached on a miss or --ci). */ + review: (request: IReviewRequest) => Promise; + /** Persist the verdict under the key (the real impl guards against caching a pre-review block). */ + persist: (verdict: IVerdict, key: string) => Promise; +} + +/** + * The end-to-end review flow, injectable so the CLI wiring's CENTRAL INVARIANT is directly + * testable rather than only implied by the units: the request is GATHERED (validate runs + * inside gather) BEFORE any cache access, and a gather block (validate red / precondition) + * yields a blocked verdict WITHOUT reading OR writing the cache. Only a gathered request is + * keyed — from its own bytes — then reused-or-reviewed. A future reordering or bypass in the + * CLI that read the cache before validating, or reused a verdict across a red gate, breaks a + * test here (which unit tests for gather + the cache decision separately cannot catch). + * + * `--ci` never READS the cache (CI always re-reviews) but still WRITES it, seeding a later + * interactive run with the identical request. + */ +export async function runReviewFlow( + deps: IReviewFlowDeps +): Promise<{ verdict: IVerdict; cacheHit: boolean }> { + const gathered = await deps.gather(); + + if (gathered.kind === "block") { + // Validate red / precondition — never touch the cache. + return { + verdict: blockedVerdict(gathered.reason, deps.identity), + cacheHit: false, + }; + } + + const key = reviewRequestKey(gathered.request, { + rosterHash: deps.rosterHash, + mode: deps.mode, + }); + + if (!deps.ci) { + const cached = await deps.readCache(key); + + if (cached !== null) { + return { verdict: cached, cacheHit: true }; + } + } + + const verdict = await deps.review(gathered.request); + + await deps.persist(verdict, key); + + return { verdict, cacheHit: false }; +} + /** The caching decision, isolated so it is unit-testable without the filesystem. * ONLY a real panel verdict is cached. A pre-review gate/precondition block - * (validate failed, empty intent, diff too large) is transient — caching one - * poisons the tree-hash so a flaky validate under load blocks every later push. */ + * (validate failed, empty intent, empty/oversized diff) is transient — caching one + * would re-serve as a permanent block for that request, so it is skipped. */ export function shouldCacheVerdict(verdict: IVerdict): boolean { return verdict.preReview !== true; } diff --git a/packages/core/tests/harness-review-mode.test.ts b/packages/core/tests/harness-review-mode.test.ts new file mode 100644 index 00000000..a6fc1a12 --- /dev/null +++ b/packages/core/tests/harness-review-mode.test.ts @@ -0,0 +1,230 @@ +import { test, expect, describe } from "bun:test"; +import { buildReviewFlowDeps } from "../src/cli/harness-review-mode"; +import { panelIdentityHash } from "../src/reviewers/harness-review"; +import type { IReviewRequest } from "../src/reviewers/schema"; +import type { IPanel } from "../src/reviewers/registry"; +import type { IVerdict } from "../src/reviewers/aggregate"; + +/** + * The CLI wiring (harness-review-mode) is exercised through buildReviewFlowDeps: it must + * derive the cache key's rosterHash from the EFFECTIVE panel, target the review at the + * effective panel, map mode/ci from the args, gather with args.base/intent, and thread the + * effective rosterHash into persistence. A miswire (cfg roster, hardcoded ci, wrong panel) + * is caught here rather than only in production. + */ + +const effective: IPanel = { + reviewers: [ + { + kind: "model", + id: "r1", + entry: { baseUrl: "http://x/v1", model: "MODEL-X" }, + }, + ], + minReviewers: 1, + skipped: [], +}; + +const request: IReviewRequest = { + title: "t", + intent: "t", + diff: "d", + validateSummary: { passed: true, failCount: 0, firstErrors: [] }, + contextFiles: [], + rubricVersion: "1", +}; + +interface ILog { + validated: number; + gitArgs: string[][]; + providerModels: string[]; + readKeys: string[]; + persisted: { key: string; rosterHash: string }[]; +} + +function makeInput(over: { quick?: boolean; ci?: boolean }) { + const log: ILog = { + validated: 0, + gitArgs: [], + providerModels: [], + readKeys: [], + persisted: [], + }; + + return { + log, + input: { + panel: effective, + identity: "local/flash", + quick: over.quick ?? false, + ci: over.ci ?? false, + // Distinctive base/intent so a wiring that drops or overrides them is caught. + base: "customBase", + intent: "DISTINCT-INTENT", + git: async (args: string[]) => { + log.gitArgs.push(args); + + // Enough for gatherChange to build a request: rev-parse/merge-base SHAs, a file + + // a non-empty diff, readable context. merge-base ECHOES its ref arg so the diff range + // proves WHICH base was used (a dropped base would echo "main", not "customBase"). + if (args[0] === "rev-parse") { + return { stdout: "HEADSHA\n", code: 0 }; + } + + if (args[0] === "merge-base") { + return { stdout: `mb-${String(args[1])}\n`, code: 0 }; + } + + if (args[0] === "diff") { + return args.includes("--name-only") + ? { stdout: "x.ts", code: 0 } + : { stdout: "diff --git a/x b/x\n+code", code: 0 }; + } + + if (args[0] === "show") { + return { stdout: "content", code: 0 }; + } + + return { stdout: "", code: 0 }; + }, + validate: async () => { + log.validated += 1; + + return { passed: true, failCount: 0, firstErrors: [] }; + }, + makeProvider: (entry: { model: string }) => { + log.providerModels.push(entry.model); + + return { + async complete() { + return { + content: '{"decision":"approve","findings":[]}', + toolCalls: [], + }; + }, + }; + }, + runBinary: async () => ({ ok: true, stdout: "" }), + readCache: async (key: string) => { + log.readKeys.push(key); + + return null; + }, + persistArtifact: async ( + _v: IVerdict, + key: string, + rosterHash: string + ) => { + log.persisted.push({ key, rosterHash }); + }, + }, + }; +} + +describe("buildReviewFlowDeps (CLI wiring)", () => { + test("rosterHash is derived from the EFFECTIVE panel + builder (not the raw config)", () => { + const { input } = makeInput({}); + const deps = buildReviewFlowDeps(input); + + expect(deps.rosterHash).toBe(panelIdentityHash(effective, "local/flash")); + }); + + test("quick mode slices the roster to 1 reviewer — rosterHash uses the SLICED roster (a quick verdict can't satisfy a full review)", () => { + const twoReviewers: IPanel = { + reviewers: [ + { + kind: "model", + id: "r1", + entry: { baseUrl: "http://x/v1", model: "MODEL-X" }, + }, + { + kind: "model", + id: "r2", + entry: { baseUrl: "http://y/v1", model: "MODEL-Y" }, + }, + ], + minReviewers: 1, + skipped: [], + }; + const full = buildReviewFlowDeps({ + ...makeInput({}).input, + panel: twoReviewers, + quick: false, + }); + const quick = buildReviewFlowDeps({ + ...makeInput({}).input, + panel: twoReviewers, + quick: true, + }); + + expect(quick.rosterHash).not.toBe(full.rosterHash); + expect(quick.rosterHash).toBe( + panelIdentityHash( + { ...twoReviewers, reviewers: twoReviewers.reviewers.slice(0, 1) }, + "local/flash" + ) + ); + }); + + test("mode maps from --quick and ci maps from --ci", () => { + expect(buildReviewFlowDeps(makeInput({ quick: false }).input).mode).toBe( + "full" + ); + expect(buildReviewFlowDeps(makeInput({ quick: true }).input).mode).toBe( + "quick" + ); + expect(buildReviewFlowDeps(makeInput({ ci: true }).input).ci).toBe(true); + expect(buildReviewFlowDeps(makeInput({ ci: false }).input).ci).toBe(false); + }); + + test("gather runs validate FRESH and uses args.base + args.intent (a dropped base or frozen intent is caught)", async () => { + const { log, input } = makeInput({}); + const deps = buildReviewFlowDeps(input); + + const gathered = await deps.gather(); + + expect(gathered.kind).toBe("request"); + expect(log.validated).toBe(1); // validate ran inside gather + + if (gathered.kind === "request") { + // args.intent flowed through (not frozen to some other string). + expect(gathered.request.intent).toBe("DISTINCT-INTENT"); + } + + // The diff range proves the CONFIGURED base ("customBase") was used — merge-base echoes + // its ref, so a dropped base would show "mb-main...HEADSHA" and fail this. + expect( + log.gitArgs.some((a) => a.join(" ").includes("mb-customBase...HEADSHA")) + ).toBe(true); + }); + + test("review targets the EFFECTIVE panel (the effective reviewer's model provider is invoked)", async () => { + const { log, input } = makeInput({}); + const deps = buildReviewFlowDeps(input); + + await deps.review(request); + + expect(log.providerModels).toContain("MODEL-X"); + }); + + test("persist carries the EFFECTIVE rosterHash (the artifact records the roster that actually reviewed)", async () => { + const { log, input } = makeInput({}); + const deps = buildReviewFlowDeps(input); + const verdict: IVerdict = { + blocked: false, + reason: "ok", + reviewers: { ok: 1, errored: 0 }, + ranked: [], + perReviewer: [], + identity: "local/flash", + }; + + await deps.persist(verdict, "KEY"); + + expect(log.persisted).toHaveLength(1); + expect(log.persisted[0]?.key).toBe("KEY"); + expect(log.persisted[0]?.rosterHash).toBe( + panelIdentityHash(effective, "local/flash") + ); + }); +}); diff --git a/packages/core/tests/reviewers-artifact.test.ts b/packages/core/tests/reviewers-artifact.test.ts index 8064f132..343b50a4 100644 --- a/packages/core/tests/reviewers-artifact.test.ts +++ b/packages/core/tests/reviewers-artifact.test.ts @@ -1,9 +1,17 @@ import { test, expect, describe } from "bun:test"; +import { createHash } from "node:crypto"; import { - verdictCacheKey, + reviewRequestKey, + panelIdentityHash, + runReviewFlow, + canonicalJson, artifactBody, honorCachedVerdict, + CACHE_VERSION, } from "../src/reviewers/harness-review"; +import type { GatherResult } from "../src/reviewers/harness-review"; +import type { ResolvedReviewer } from "../src/reviewers/registry"; +import type { IReviewRequest } from "../src/reviewers/schema"; import type { IVerdict } from "../src/reviewers/aggregate"; const v: IVerdict = { @@ -15,46 +23,317 @@ const v: IVerdict = { identity: "local/flash", }; -describe("artifact + cache", () => { - test("cache key is stable for the same inputs and changes with the tree hash", () => { - const a = verdictCacheKey({ - treeHash: "t1", - panelHash: "p1", - rubricVersion: "1", - cacheVersion: "2", - }); - const b = verdictCacheKey({ - treeHash: "t1", - panelHash: "p1", - rubricVersion: "1", - cacheVersion: "2", - }); - const c = verdictCacheKey({ - treeHash: "t2", - panelHash: "p1", - rubricVersion: "1", - cacheVersion: "2", - }); +const request: IReviewRequest = { + title: "add the widget", + intent: "add the widget", + diff: "diff --git a/x b/x\n+code", + validateSummary: { passed: true, failCount: 0, firstErrors: [] }, + contextFiles: ["=== x.ts ===\nexport const x = 1;"], + rubricVersion: "1", +}; + +const rosterOpts = { rosterHash: "r1", mode: "full" }; - expect(a).toBe(b); - expect(a).not.toBe(c); +describe("reviewRequestKey (cache key = fingerprint of the ACTUAL review request)", () => { + test("stable for the same request + roster + mode", () => { + expect(reviewRequestKey(request, rosterOpts)).toBe( + reviewRequestKey(request, rosterOpts) + ); }); - test("cacheVersion is mixed into the key — bumping it retires ALL legacy artifacts", () => { - // CACHE_VERSION is the ONLY thing that retires already-on-disk poisoned v1 blocks - // (they carry no preReview flag, so the read-side guard can't reject them). If a - // regression dropped it from the hash, legacy poison would be re-served and every - // other test would still pass — so pin it here. - const base = { treeHash: "t1", panelHash: "p1", rubricVersion: "1" }; + test("every reviewer-visible dimension changes the key — diff, contextFiles, intent, rubric, validateSummary, roster, mode", () => { + const base = reviewRequestKey(request, rosterOpts); + + expect( + reviewRequestKey({ ...request, diff: "different" }, rosterOpts) + ).not.toBe(base); + // contextFiles: a rebase can yield an identical diff but different surrounding file + // content — the reviewers see this, so it MUST change the key. + expect( + reviewRequestKey( + { ...request, contextFiles: ["=== x.ts ===\nother"] }, + rosterOpts + ) + ).not.toBe(base); + expect( + reviewRequestKey({ ...request, intent: "different" }, rosterOpts) + ).not.toBe(base); + expect( + reviewRequestKey({ ...request, rubricVersion: "2" }, rosterOpts) + ).not.toBe(base); + // validateSummary is part of the request the reviewers read (firstErrors can differ even + // on a passing run), so a different summary MUST change the key — no false reuse. + expect( + reviewRequestKey( + { + ...request, + validateSummary: { + passed: true, + failCount: 0, + firstErrors: ["a stray 'error' line"], + }, + }, + rosterOpts + ) + ).not.toBe(base); + expect( + reviewRequestKey(request, { ...rosterOpts, rosterHash: "r2" }) + ).not.toBe(base); + expect( + reviewRequestKey(request, { ...rosterOpts, mode: "quick" }) + ).not.toBe(base); + }); - expect(verdictCacheKey({ ...base, cacheVersion: "1" })).not.toBe( - verdictCacheKey({ ...base, cacheVersion: "2" }) + test("unforgeable: a value sliding across a field boundary can't collide two distinct requests", () => { + // JSON serialization keeps ('a','b c') distinct from ('a b','c'). + expect( + reviewRequestKey({ ...request, diff: "a", intent: "b c" }, rosterOpts) + ).not.toBe( + reviewRequestKey({ ...request, diff: "a b", intent: "c" }, rosterOpts) + ); + }); + + test("canonical (key-sorted) serialization: SAME content in a different key insertion order hashes equally (no thrash / divergence)", () => { + // A second construction path (or a JS engine detail) that emits the request's keys in a + // different order must not change the digest, or the cache would thrash / a re-gather + // could miss. The recursive key-sort guarantees content-equality ⇒ digest-equality. + const reordered: IReviewRequest = { + rubricVersion: request.rubricVersion, + contextFiles: request.contextFiles, + validateSummary: request.validateSummary, + diff: request.diff, + intent: request.intent, + title: request.title, + }; + + expect(reviewRequestKey(reordered, rosterOpts)).toBe( + reviewRequestKey(request, rosterOpts) ); }); + test("CACHE_VERSION is mixed into the key — bumping it retires ALL legacy artifacts in one shot", () => { + // The ONLY lever that invalidates every already-on-disk artifact (e.g. legacy diff-hash + // keys, or a poisoned pre-review block). If a regression dropped CACHE_VERSION from the + // hashed array, one-shot invalidation would silently break while every other test stayed + // green — so pin it by recomputing the exact key WITH it and asserting equality. + const expected = createHash("sha256") + .update( + canonicalJson([ + request, + rosterOpts.rosterHash, + rosterOpts.mode, + CACHE_VERSION, + ]) + ) + .digest("hex"); + + expect(reviewRequestKey(request, rosterOpts)).toBe(expected); + + // And a different CACHE_VERSION would produce a different key (the invalidation itself). + const otherVersion = createHash("sha256") + .update( + canonicalJson([ + request, + rosterOpts.rosterHash, + rosterOpts.mode, + "OTHER", + ]) + ) + .digest("hex"); + + expect(reviewRequestKey(request, rosterOpts)).not.toBe(otherVersion); + }); +}); + +describe("panelIdentityHash (the roster that ACTUALLY reviewed keys the cache)", () => { + const grok: ResolvedReviewer = { + kind: "model", + id: "grok", + entry: { baseUrl: "http://a", model: "g1" }, + }; + const codex: ResolvedReviewer = { + kind: "binary", + id: "codex", + argv: ["codex"], + input: "stdin", + timeoutMs: 1000, + parse: "raw", + }; + const panel = { reviewers: [grok, codex], minReviewers: 2 }; + + test("stable and order-independent (roster is sorted by id before hashing)", () => { + expect(panelIdentityHash(panel, "local/flash")).toBe( + panelIdentityHash( + { reviewers: [codex, grok], minReviewers: 2 }, + "local/flash" + ) + ); + }); + + test("adding/dropping a reviewer, changing the quorum, or changing the builder all change the key", () => { + const base = panelIdentityHash(panel, "local/flash"); + + expect( + panelIdentityHash({ reviewers: [grok], minReviewers: 2 }, "local/flash") + ).not.toBe(base); // a dropped reviewer (effective roster ≠ configured) must not reuse + expect( + panelIdentityHash({ ...panel, minReviewers: 1 }, "local/flash") + ).not.toBe(base); + expect(panelIdentityHash(panel, "other/model")).not.toBe(base); // different builder + }); + + test("RETARGETING the same id to a different model/endpoint/binary changes the key (reviewer implementation is pinned, not just its id)", () => { + // The panel finding: hashing ids alone would reuse a verdict produced by a DIFFERENT + // reviewer implementation. Full config is hashed, so same-id-different-model differs. + const grokRetargeted: ResolvedReviewer = { + kind: "model", + id: "grok", + entry: { baseUrl: "http://a", model: "g2-DIFFERENT" }, + }; + + expect( + panelIdentityHash( + { reviewers: [grokRetargeted, codex], minReviewers: 2 }, + "local/flash" + ) + ).not.toBe(panelIdentityHash(panel, "local/flash")); + }); +}); + +describe("runReviewFlow (the CLI wiring invariant: gather-before-cache, block-never-caches)", () => { + const reviewVerdict: IVerdict = { ...v, reason: "fresh review" }; + const cachedVerdict: IVerdict = { ...v, reason: "from cache" }; + + interface ICalls { + gather: number; + read: number; + review: number; + persist: string[]; + /** Ordered log of every seam as it fires, so call ORDER (not just counts) is asserted. */ + order: string[]; + /** The key readCache was called with (null if never). */ + readKey: string | null; + } + + // A flow harness: `gathered` is what gather() returns (block or request); records the + // order/counts so the wiring invariant can be asserted, and captures the key each cache + // op received (to prove it derives from the gathered request). + const harness = (gathered: GatherResult, cached: IVerdict | null) => { + const calls: ICalls = { + gather: 0, + read: 0, + review: 0, + persist: [], + order: [], + readKey: null, + }; + const deps = { + gather: async () => { + calls.gather += 1; + calls.order.push("gather"); + + return gathered; + }, + identity: "local/flash", + rosterHash: "roster-1", + mode: "full", + ci: false, + readCache: async (key: string) => { + calls.read += 1; + calls.order.push("read"); + calls.readKey = key; + + return cached; + }, + review: async (_request: IReviewRequest) => { + calls.review += 1; + calls.order.push("review"); + + return reviewVerdict; + }, + persist: async (_verdict: IVerdict, key: string) => { + calls.order.push("persist"); + calls.persist.push(key); + }, + }; + + return { calls, deps }; + }; + + const requestResult: GatherResult = { kind: "request", request }; + + test("a gather BLOCK (validate red / precondition) returns a blocked verdict WITHOUT touching the cache", async () => { + // The central invariant: no cache read AND no write when the gate is red — a verdict is + // never reused across a failing validate, and a transient block is never persisted. + const { calls, deps } = harness( + { kind: "block", reason: "validate failed (3 errors)" }, + cachedVerdict + ); + + const r = await runReviewFlow(deps); + + expect(r.verdict.blocked).toBe(true); + expect(r.verdict.preReview).toBe(true); + expect(calls.gather).toBe(1); + expect(calls.read).toBe(0); + expect(calls.review).toBe(0); + expect(calls.persist).toHaveLength(0); + }); + + test("gather runs strictly BEFORE any cache access, and the READ + WRITE keys are exactly reviewRequestKey(gathered request)", async () => { + const { calls, deps } = harness(requestResult, null); + + await runReviewFlow(deps); + + const expectedKey = reviewRequestKey(request, { + rosterHash: "roster-1", + mode: "full", + }); + + // ORDER, not just counts: gather precedes the read, which precedes review+persist. + expect(calls.order).toEqual(["gather", "read", "review", "persist"]); + // Both cache ops keyed the gathered request (not the raw flags / a different base). + expect(calls.readKey).toBe(expectedKey); + expect(calls.persist).toEqual([expectedKey]); + }); + + test("cache HIT reuses the cached verdict — gather ran BEFORE the read, then no review, no write", async () => { + const { calls, deps } = harness(requestResult, cachedVerdict); + + const r = await runReviewFlow(deps); + + expect(r.cacheHit).toBe(true); + expect(r.verdict.reason).toBe("from cache"); + expect(calls.order).toEqual(["gather", "read"]); // gathered first, then read, then stop + expect(calls.review).toBe(0); + expect(calls.persist).toHaveLength(0); + }); + + test("cache MISS runs the review and writes under the derived key", async () => { + const { calls, deps } = harness(requestResult, null); + + const r = await runReviewFlow(deps); + + expect(r.cacheHit).toBe(false); + expect(r.verdict.reason).toBe("fresh review"); + expect(calls.review).toBe(1); + expect(calls.persist).toHaveLength(1); + }); + + test("--ci WRITES but never READS the cache (CI always re-reviews)", async () => { + const { calls, deps } = harness(requestResult, cachedVerdict); + + const r = await runReviewFlow({ ...deps, ci: true }); + + expect(calls.read).toBe(0); // never reads on CI, even though a cache entry exists + expect(calls.review).toBe(1); + expect(calls.persist).toHaveLength(1); + expect(r.cacheHit).toBe(false); + }); +}); + +describe("read-side + artifact", () => { test("honorCachedVerdict drops a cached pre-review block, passes a real verdict through", () => { - // The read-side defense in depth: a pre-review block that somehow reached disk - // must force a fresh live review (null), while a genuine panel verdict is honored. expect(honorCachedVerdict(null)).toBeNull(); expect( honorCachedVerdict({ ...v, blocked: true, preReview: true }) diff --git a/packages/core/tests/reviewers-harness-review.test.ts b/packages/core/tests/reviewers-harness-review.test.ts index 174194e0..5905fa17 100644 --- a/packages/core/tests/reviewers-harness-review.test.ts +++ b/packages/core/tests/reviewers-harness-review.test.ts @@ -1,10 +1,9 @@ import { test, expect, describe } from "bun:test"; import { gatherChange, - runHarnessReview, + reviewRequest, shouldCacheVerdict, type IGatherDeps, - type IRunDeps, } from "../src/reviewers/harness-review"; import type { IPanel } from "../src/reviewers/registry"; @@ -13,7 +12,21 @@ function git(map: Record): IGatherDeps["git"] { const key = args.join(" "); const hit = Object.entries(map).find(([k]) => key.includes(k)); - return { stdout: hit?.[1] ?? "", code: 0 }; + if (hit) { + return { stdout: hit[1], code: 0 }; + } + + // Defaults so the HEAD pin + base resolve to SHAs unless a test overrides them (the pin + // now REQUIRES a real SHA — an empty rev-parse would block). + if (args[0] === "rev-parse") { + return { stdout: "HEADSHA", code: 0 }; + } + + if (args[0] === "merge-base") { + return { stdout: "MBSHA", code: 0 }; + } + + return { stdout: "", code: 0 }; }; } @@ -95,12 +108,273 @@ describe("gatherChange", () => { } }); + test("a failing `git rev-parse HEAD` BLOCKS (no soft fallback to a moving HEAD ref)", async () => { + // The pin must be honest: on rev-parse failure, block — do NOT silently review the movable + // "HEAD" ref (which would re-open the TOCTOU the pin exists to close). + const failingHead: IGatherDeps["git"] = async (args) => + args[0] === "rev-parse" + ? { stdout: "", code: 128 } + : { stdout: "", code: 0 }; + const r = await gatherChange( + { git: failingHead, validate: cleanValidate }, + opts + ); + + expect(r.kind).toBe("block"); + + if (r.kind === "block") { + expect(r.reason).toMatch(/could not resolve HEAD/iu); + } + }); + + test("a failing `git diff --name-only` BLOCKS instead of building an empty review (exit code honored)", async () => { + // The false-green the panel flagged: if git errors and returns empty stdout, an + // unguarded gather would build a 0-file, empty-diff request that the panel green-lights + // and caches. Honoring the exit code turns that into a block. + const failingGit: IGatherDeps["git"] = async (args) => { + if (args[0] === "rev-parse") { + return { stdout: "HEADSHA", code: 0 }; + } + + if (args[0] === "merge-base") { + return { stdout: "MBSHA", code: 0 }; + } + + return args.includes("--name-only") + ? { stdout: "", code: 128 } + : { stdout: "", code: 0 }; + }; + + const r = await gatherChange( + { git: failingGit, validate: cleanValidate }, + opts + ); + + expect(r.kind).toBe("block"); + + if (r.kind === "block") { + expect(r.reason).toMatch(/could not compute the changed-file list/iu); + } + }); + + test("a failing `git diff` (content) BLOCKS even when the name list succeeded", async () => { + const git2: IGatherDeps["git"] = async (args) => { + if (args[0] === "rev-parse") { + return { stdout: "HEADSHA", code: 0 }; + } + + if (args[0] === "merge-base") { + return { stdout: "MBSHA", code: 0 }; + } + + if (args.includes("--name-only")) { + return { stdout: "x.ts", code: 0 }; + } + + if (args[0] === "diff") { + return { stdout: "", code: 129 }; // the content diff fails + } + + return { stdout: "", code: 0 }; + }; + + const r = await gatherChange({ git: git2, validate: cleanValidate }, opts); + + expect(r.kind).toBe("block"); + + if (r.kind === "block") { + expect(r.reason).toMatch(/could not compute the diff/iu); + } + }); + + test("files listed but an EMPTY content diff → block (defense in depth), not a vacuous cached green", async () => { + // `--name-only` reports a file, but the content `git diff` exits 0 with empty stdout — a + // git anomaly / name-only↔diff disagreement (a REAL rename/mode change is non-empty). The + // guard blocks it rather than build+cache a 0-byte review the panel would green-light. + const git2: IGatherDeps["git"] = async (args) => { + if (args[0] === "rev-parse") { + return { stdout: "HEADSHA", code: 0 }; + } + + if (args[0] === "merge-base") { + return { stdout: "MBSHA", code: 0 }; + } + + if (args.includes("--name-only")) { + return { stdout: "renamed.ts", code: 0 }; + } + + if (args[0] === "diff") { + return { stdout: "", code: 0 }; // exit 0, but empty content + } + + return { stdout: "", code: 0 }; + }; + + const r = await gatherChange({ git: git2, validate: cleanValidate }, opts); + + expect(r.kind).toBe("block"); + + if (r.kind === "block") { + expect(r.reason).toMatch(/empty/iu); + } + }); + + test("no changed files between base and HEAD → block (nothing to review, never a vacuous green)", async () => { + const deps: IGatherDeps = { + git: git({ "diff --name-only": "", diff: "" }), + validate: cleanValidate, + }; + const r = await gatherChange(deps, opts); + + expect(r.kind).toBe("block"); + + if (r.kind === "block") { + expect(r.reason).toMatch(/no changes/iu); + } + }); + + test("resolveBase SUCCESS path: diffs merge-base-SHA...HEAD-SHA (both pinned to immutable commits, not movable refs)", async () => { + // The happy path the panel flagged as untested: merge-base returns a real SHA and HEAD is + // pinned via rev-parse, so every read is against one snapshot. Deleting the pin would make + // this fail (the range would reference `main`/`HEAD` refs, not the SHAs). + const seen: string[] = []; + + const pinnedGit: IGatherDeps["git"] = async (args) => { + const key = args.join(" "); + + if (args[0] === "rev-parse") { + return { stdout: "HEADSHA\n", code: 0 }; + } + + if (args[0] === "merge-base") { + return { stdout: "MBSHA\n", code: 0 }; + } + + if (args[0] === "diff") { + seen.push(key); + + return key.includes("--name-only") + ? { stdout: "x.ts", code: 0 } + : { stdout: "diff --git a/x b/x\n+code", code: 0 }; + } + + if (args[0] === "show") { + seen.push(key); + + return { stdout: "file content", code: 0 }; + } + + return { stdout: "", code: 0 }; + }; + + const r = await gatherChange( + { git: pinnedGit, validate: cleanValidate }, + { ...opts, base: "main" } + ); + + expect(r.kind).toBe("request"); + // The diff range and the context `show` both reference the pinned SHAs. + expect(seen.some((k) => k.includes("MBSHA...HEADSHA"))).toBe(true); + expect(seen.some((k) => k.includes("show HEADSHA:x.ts"))).toBe(true); + }); + + test("resolveBase merge-base failure falls back to the ref resolved to a SHA (shallow/odd repos), still one snapshot", async () => { + // merge-base fails; the fallback base must be PINNED (rev-parse'd to a SHA), NOT the movable + // ref — else the name-only/content diffs could observe different commits mid-gather. Prove + // the range is ..., both immutable. + const seen: string[] = []; + + const fallbackGit: IGatherDeps["git"] = async (args) => { + const key = args.join(" "); + + if (args[0] === "rev-parse") { + // HEAD → HEADSHA; the fallback base ref "featureX" → FEATURESHA. + return args[1] === "HEAD" + ? { stdout: "HEADSHA", code: 0 } + : { stdout: "FEATURESHA", code: 0 }; + } + + if (args[0] === "merge-base") { + return { stdout: "", code: 1 }; // no merge-base + } + + if (args[0] === "diff") { + seen.push(key); + + return key.includes("--name-only") + ? { stdout: "x.ts", code: 0 } + : { stdout: "diff --git a/x b/x\n+code", code: 0 }; + } + + return { stdout: "", code: 0 }; + }; + + const r = await gatherChange( + { git: fallbackGit, validate: cleanValidate }, + { ...opts, base: "featureX" } + ); + + expect(r.kind).toBe("request"); + // Both ends are pinned SHAs — a moving ref would have shown "featureX...HEAD" (and would + // still pass on a broken pin), which this now rejects. + expect(seen.some((k) => k.includes("FEATURESHA...HEADSHA"))).toBe(true); + }); + + test("a merge-base failure whose fallback ref ALSO can't be pinned BLOCKS (never diff a movable ref)", async () => { + const unpinnable: IGatherDeps["git"] = async (args) => { + if (args[0] === "rev-parse") { + // HEAD pins, but the fallback base ref cannot be resolved. + return args[1] === "HEAD" + ? { stdout: "HEADSHA", code: 0 } + : { stdout: "", code: 128 }; + } + + if (args[0] === "merge-base") { + return { stdout: "", code: 1 }; + } + + return { stdout: "", code: 0 }; + }; + + const r = await gatherChange( + { git: unpinnable, validate: cleanValidate }, + { ...opts, base: "ghost" } + ); + + expect(r.kind).toBe("block"); + + if (r.kind === "block") { + expect(r.reason).toMatch(/could not resolve the diff base/iu); + } + }); + + test("rev-parse HEAD succeeds but returns EMPTY stdout → still BLOCKS (an empty SHA is not a valid pin)", async () => { + // The subtle soft-fallback the panel flagged: code 0 + empty output must NOT become the + // movable "HEAD" ref. It blocks like a hard rev-parse failure. + const emptyHead: IGatherDeps["git"] = async (args) => + args[0] === "rev-parse" + ? { stdout: "\n", code: 0 } + : { stdout: "", code: 0 }; + const r = await gatherChange( + { git: emptyHead, validate: cleanValidate }, + opts + ); + + expect(r.kind).toBe("block"); + + if (r.kind === "block") { + expect(r.reason).toMatch(/could not resolve HEAD/iu); + } + }); + test("attaches the changed files' current (HEAD) contents as review context", async () => { const deps: IGatherDeps = { git: git({ "diff --name-only": "src/x.ts", diff: "diff --git a/src/x.ts b/src/x.ts\n+code", - "show HEAD:src/x.ts": + // head is pinned to a SHA (default HEADSHA), so context reads `show :file`. + "show HEADSHA:src/x.ts": "export function realCode(): number {\n return 42;\n}", }), validate: cleanValidate, @@ -118,41 +392,50 @@ describe("gatherChange", () => { }); }); -describe("runHarnessReview", () => { - test("a blocked gather short-circuits to a blocked verdict without invoking reviewers", async () => { - let invoked = false; - const panel: IPanel = { reviewers: [], minReviewers: 2, skipped: [] }; - const deps: IRunDeps = { - git: git({ "diff --name-only": "x.ts", diff: "+x" }), - validate: async () => ({ - passed: false, - failCount: 1, - firstErrors: ["boom"], - }), +describe("reviewRequest (success path: invoke the panel on a gathered request → aggregate)", () => { + test("invokes the reviewers and returns their aggregated verdict", async () => { + const request = { + title: "add x", + intent: "add x", + diff: "diff --git a/x b/x\n+code", + validateSummary: { passed: true, failCount: 0, firstErrors: [] }, + contextFiles: [], + rubricVersion: "1", + }; + // A single approving model reviewer; aggregate should return an unblocked verdict. + const panel: IPanel = { + reviewers: [ + { + kind: "model", + id: "r1", + entry: { baseUrl: "http://x/v1", model: "m" }, + }, + ], + minReviewers: 1, + skipped: [], + }; + let invoked = 0; + const v = await reviewRequest(request, { makeProvider: () => { - invoked = true; + invoked += 1; return { async complete() { - return { content: "", toolCalls: [] }; + return { + content: '{"decision":"approve","findings":[]}', + toolCalls: [], + }; }, }; }, - runBinary: async () => { - invoked = true; - - return { ok: true, stdout: "" }; - }, + runBinary: async () => ({ ok: true, stdout: "" }), panel, identity: "local/flash", - }; - const v = await runHarnessReview(deps, opts); + }); - expect(v.blocked).toBe(true); - expect(invoked).toBe(false); - // Marked as a PRE-REVIEW gate block so the caller never caches it — a transient - // validate flake under load must not poison the tree-hash for every later push. - expect(v.preReview).toBe(true); + expect(invoked).toBe(1); // the panel actually ran (not short-circuited) + expect(v.identity).toBe("local/flash"); + expect(v.preReview).not.toBe(true); // a real panel verdict, cacheable }); });