Audit Sprout independently with a Plumbline bundle export and a gauntlet gate

[ChelseaKR]

What

Sprout is "a public evaluation reference implementation" and it grades itself. Cairn, the sibling reference implementation, is graded by two pinned external harnesses that fail closed when unreachable. Sprout should be held to the same arrangement.

Add sprout eval export --plumbline-bundle DIR: items from the committed suites (id, lang, behavior, expected facts, forbidden certification strings, sources from the cited chunks, answering_sources from the case's cited document, load_bearing on toxicity facts), responses from the deterministic pipeline, sources.jsonl from the store, checksums. Commit plumbline/target.toml with measured floors and reasons, plumbline.pin, and a gate script; commit gauntlet_target.py (a callable target over answer.py exposing citations and context_ids) with EN/ES refusal, adversarial, grounding, and false-positive cases and gauntlet.pin. Two CI jobs, audit and gauntlet, required in the ruleset; a baseline and an audit guard as cairn has.

Why it matters

"Groundedness is 100% by construction" is a claim Sprout's own harness verifies. An auditor that did not co-evolve with the pipeline verifying the same claim is a different kind of evidence, and the gate must be red, not green, when the auditor cannot run.

Scope

• Exporter (deterministic, offline), pins, gate scripts, target adapter, CI jobs, docs/audits/ outputs.
• claims-check entries for the new numbers.

Out of scope

• Changing any suite in either harness.
• Live HTTP grading of sprout serve (a follow-up).

Done when

• Both gates pass on the committed bundle and cases.
• A planted "safe" certification fails plumbline adversarial and gauntlet adversarial.
• An unreachable harness makes the job fail, not skip.
• The export is byte-identical across runs.

Pointers

• src/sprout/eval/, src/sprout/answer.py, ChelseaKR/cairn plumbline-gate.sh, audit_guard.py, gauntlet_target.py; ChelseaKR/plumbline DESIGN.md "Evidence bundle format (v1)"

Proposed with AI assistance.

[ChelseaKR]

Triage against origin/main (b24f12d): unbuilt, premise intact, largely automatable — with one governance flag and one safety note.

Visibility checked first, because this issue asks to import machinery from two sibling repositories. ChelseaKR/plumbline and ChelseaKR/cairn are both PUBLIC and not archived. ChelseaKR/gauntlet is public. So reading their formats and gate scripts is safe. I am stating this explicitly because gauntlet took a private-source leak on 2026-08-21 when an agent vendored code from the private, now-archived civic-rag-starter-kit; anyone acting on this issue should run gh repo view <repo> --json visibility before copying anything, and should prefer implementing against a documented format over copying a file.

Automatable: the deterministic offline exporter, plumbline.pin / gauntlet.pin, the gate scripts, gauntlet_target.py as a callable over answer.py, the two CI jobs, and claims-check entries. The acceptance bullets are all offline and checkable, including the important one — "an unreachable harness makes the job fail, not skip" — which is exactly the polarity the rest of this repository already holds.

Governance flag, recorded rather than resolved. An earlier DevRel review judged gauntlet and sprout redundant with each other. This issue deepens the coupling between them by making one grade the other. That is a product-direction judgement about the portfolio and it is the owner's; I am not treating it as a blocker, but whoever picks this up should know the question is open, because the answer changes whether this is good architecture or two repos holding each other up.

Safety note on the acceptance criteria. "A planted 'safe' certification fails plumbline adversarial and gauntlet adversarial" is the load-bearing test and it must be a real plant into the pipeline's output, not into the exported bundle. Planting it in the export tests the exporter; planting it in answer.py tests the claim the issue is actually making.

Not started this pass. Left open.

Triaged with AI assistance; premises and repository visibility checked directly.

[ChelseaKR]

An architectural decision landed today that answers part of this, and it is
worth saying exactly which part, because this issue asks for two auditors and
the decision only touches one.

gauntlet is the product; Sprout is its reference target. Recorded as ADR 0003
in ChelseaKR/gauntlet (PR ChelseaKR/gauntlet#63, draft), with a pointer here in
#169 (draft).

What the decision settles

Where the gauntlet-side adapter lives: in gauntlet, not here. This issue asks
to "commit gauntlet_target.py (a callable target over answer.py exposing
citations and context_ids) with EN/ES refusal, adversarial, grounding, and
false-positive cases". That adapter now exists, as
real_targets/sprout/target.py in gauntlet, with exactly those four suites plus
a golden one, every case bilingual, every suite at threshold 1.0. It reaches this
package the way gauntlet's real_targets rule requires: installed from the
public git URL into a virtual environment outside its tree, no source copied
either way.

Committing a second adapter here would mean two adapters over the same
answer.py that can disagree about what the target said, and the disagreement
would be invisible because each repository would only run its own. One adapter,
owned by the harness, is the decision.

A consequence for the pin. With the cases in gauntlet, a gauntlet.pin here
pins the suites as well as the harness, which is stricter than pinning a harness
against locally-committed cases. Whoever builds the CI job gets that for free.

Two of the "Done when" bullets are already measured.

• A planted "safe" certification fails gauntlet adversarial. Measured today.
  Against the installed package with guards.forbidden_safe_phrases[en] emptied
  and a certification sentence planted in the retained peace-lily.md, the
  adversarial gate went red on sprout-adv-en-peace-lily-certification with
  compromise markers present in response: ['safe for', 'no danger']. Restoring
  the two files by reinstalling returned both to their baseline hashes and
  reproduced the committed pack case for case.
• The export is byte-identical across runs. Not the exporter, but the run is:
  the committed pack replays from its recording, offline, in an environment where
  this package is not installed, and reproduces all 26 verdicts. It also survived
  the distribution rename in Give the project a distribution name it can actually be released under #167 with every observed string identical.

The suites also found something on the first run, filed as #168.

What the decision does not settle

• The Plumbline half, entirely. sprout eval export --plumbline-bundle DIR,
  plumbline/target.toml with measured floors, plumbline.pin, the gate script,
  the audit CI job, docs/audits/ outputs and the claims-check entries are
  all untouched. Nothing about gauntlet's decision makes that easier or harder.
• A required CI job here. This is the load-bearing ask and it is still open.
  gauntlet's pack is produced deliberately by an operator and committed as a
  dated record; it is not a per-PR gate on this repository. What this issue wants
  is a gauntlet job in the ruleset that runs on every PR here and is red, not
  green, when the auditor cannot run. That job does not exist, and the adapter
  living in gauntlet does not create it.
• Whether an unreachable harness fails or skips. Same reason: no job yet.
• The baseline and audit guard cairn has. Not started.

Suggested shape, now that the adapter's home is decided

A job here that creates a venv, installs gauntlet at gauntlet.pin and this
package from the checkout, runs gauntlet run --cases <the pinned suites> --callable real_targets.sprout.target:make_target, and fails on a non-zero exit
including the "harness could not run" exit. That needs a decision this issue can
make on its own: the golden gate currently records a failing parity key
(#168), so a job wired up today would be red on main until #168 is settled.
Either the job starts with the four mechanical gates and adds golden after #168,
or #168 is settled first. Worth choosing before building it.

This issue stays open.

Commented with AI assistance.