Plumbline

Plumbline is an evaluation harness for government-facing chat systems. It grades a target against an executable quality bar and produces audit reports a third party could defend.

See it run → chelseakr.github.io/plumbline — the committed audit report, and the harness refusing to score tampered evidence. That page is generated by running the harness over this repository's own committed artifacts, and it cannot be built if any of them comes back different.

The page and its privacy page run Google Analytics 4 (owner decision 2026-09-17: GA4 on every public site). The measurement ID is GA4_MEASUREMENT_ID in tools/build_site.py; setting it to "" removes the loader, the footer's opt-out and every reference to Google from both pages. The loader loads nothing unless the page is served from chelseakr.github.io under /plumbline/, under Global Privacy Control or Do Not Track, or after the footer's "Opt out of analytics" (remembered in local storage as plumbline:analytics-opt-out, a key that names this project because every chelseakr.github.io site shares one origin). Google signals and ad personalization are off, the advertising consent signals are denied everywhere, and analytics cookies are denied in the EEA, the UK and Switzerland. tests/test_site_analytics.py runs the committed loader in Node and deletes each guard in turn to prove the test notices. The harness itself has no telemetry.

The bundled dataset is a demonstration of the instrument, not a benchmark. Everything under datasets/ is synthetic and written for this repository — a fictional county, fictional programs, fictional numbers, generated by a committed script. It exists so you can watch the harness work; it measures nothing about any real system, and no score produced from it means anything beyond "the instrument functions." The harness is the product.

What the harness holds itself to:

• Fail closed, everywhere. Tampered evidence: the run refuses to score, with its own exit code. Any enabled suite under its floor: overall FAIL. Unimplemented suite enabled: configuration error. A suite with nothing to score, or a floor of zero that nothing could fail: configuration error. A crash: its own exit code, never the one that means "measured and failed". There is no silent-skip path.
• Silence is never evidence. A target that returns nothing satisfies every check phrased as an absence, so it is scored zero by the suites that ask whether it behaved correctly, and reported UNVERIFIABLE — excluded, named, never a pass — by the suites that ask whether something bad is missing. A response counts as a response only if something in it survives normalization: ".", an emoji and a zero-width space are silence that gets past a .strip(). And if nothing in a run counts the silence against the target, the run refuses rather than reporting a verdict that turns on it.
• Deterministic and offline by default. The default judge is lexical; CI needs no keys; identical inputs and seed produce byte-identical reports.
• A verdict is a record. Every run writes machine-readable and human-readable reports stamped with run id, harness version, seed, dataset hash, and judge configuration hash — and with a seal over the report's own body, so a score edited afterwards is caught by plumbline verify instead of hiding behind provenance that is still technically true. verify also recomputes the run id from the inputs the report carries, so a report cannot present an identity its own contents do not generate. This is tamper evidence, not authentication: the seal has no secret in it, and vouching for who produced a report would need a signature Plumbline does not issue.

Quick start

Python ≥ 3.11, no third-party dependencies. From a clean checkout, offline:

# The one command: full audit of the bundled synthetic demo.
PYTHONPATH=src python3 -m plumbline audit --config examples/riverbend.toml --out audits

# Inspect a bundle: integrity, item count, dataset id, warnings.
PYTHONPATH=src python3 -m plumbline validate datasets/riverbend-demo

# Run the tests.
PYTHONPATH=src:tests python3 -m unittest discover -s tests

(Or pip install -e . and use plumbline … directly. Not yet on PyPI: the package is prepared to publish as plumbline-eval — plumbline itself is taken by an unrelated, long-dormant package — via .github/workflows/publish-pypi.yml, a manual-only workflow using PyPI's keyless Trusted Publishing. It has not been run; publishing needs a human to register the trusted publisher on PyPI first, which is not something this repository or its automation can do on its own.)

Re-running the audit with identical inputs writes byte-identical reports to the identical path — reports carry no timestamps by design; git history is the time record.

What it caught in its own harness

A target that returned 174 empty responses scored a perfect 1.0000 on five suites — groundedness, privacy, representational_harms, fairness and cross_language — and plumbline gate returned PASS, exit 0. Silence satisfies every check phrased as the absence of something bad. That was fixed in 5caf8e5, the fix tested response.strip(), and a target answering every item with "." scored the identical 1.0000 on the identical five suites until hardening-2026-08-16b — as did a target that answered a third of the corpus and went quiet for the rest, because every suite excluded the silence and no suite counted it.

Each of those is pinned by a test in tests/test_fail_closed.py, which fails without the fix. They are published because an evaluation harness that has never reported a false pass has probably not been looked at hard enough, and because anyone deciding whether to trust this one should be able to read how it has been wrong. A longer-form telling, including the wrong-paragraph and coupling-disclosure findings, is in docs/what-it-caught-in-its-own-harness.md — a draft for external publication, not only this section's summary.

Where it runs

Two public repositories of my own pin this harness by exact commit and run it on every pull request, resolving it at run time rather than depending on it. One of them blocks a merge on the result and one does not, and the difference is the more useful half of the description:

• ChelseaKR/cairn — pinned at a258b2e2e3cd859b1f80caaeb417ad579ade9ac2, with a committed baseline and a guard that fails on a regression the harness only reports. main there is covered by an active ruleset (id 21223426, "main: the audit gate is required") whose ten required contexts include audit (merge gate — must be required in branch protection), and that job is the one that runs ./plumbline-gate.sh. A failing audit blocks the merge there. A repository administrator can still bypass the ruleset, which is a real limit on what the arrangement proves.
• ChelseaKR/fare-policy-assistant — pinned at df95fce6eb8637bdaf46d0a4f03709b7ccce231f, exporting its own recording into a sealed bundle. plumbline gate is allowed to report FAIL there because several floors sit above current performance; a separate guard fails the build on any finding not in its acknowledged-findings file. That job is not among the repository's nine required checks, so there it reports rather than blocks.

Both bullets are the state of a different repository, which nothing in this one can check. make verify is offline by construction, and a gate that reached GitHub to keep this paragraph fresh would spend the property the paragraph sits next to, so these carry a review date instead of an implied freshness. Read from the GitHub API on 2026-08-29, with gh api repos/ChelseaKR/cairn/contents/plumbline.pin, gh api repos/ChelseaKR/cairn/rulesets, and the same two against fare-policy-assistant, plus ChelseaKR/cairn's own ci.yml to see which job each required context names. What that reading corrected is the argument for dating it: this section named a pin cairn moved off on 2026-08-23, and then, once the ruleset was noticed, concluded that none of its ten required contexts was this harness. One of them is.

Neither is a dependency relationship: gate/plumbline-gate.sh resolves the pinned commit into a cache directory at run time, so nothing in a consuming repository's dependency resolution can move the thing auditing it. The second of those consumers is where passage_attribution and forbidden_claims both came from — see "Wrong paragraph, right document" and "Mentioning a claim is not making it".

Status

Pre-1.0 (0.2.0, tagged; 0.1.0 was the first tagged release). While the version stays below 1.0.0, a MINOR bump may break the interface; pin an exact commit and read the CHANGELOG before moving. Every capability in the functional specification is implemented: fifteen scoring suites, per-suite confidence intervals and minimum detectable effect, baseline regression comparison, a pinned fail-closed CI gate, live-target recording over HTTP or against a local program, and an optional model judge — none of which the gate can reach. Every suite has been observed failing on a defect it exists to catch; see proof/matrix.md. 971 tests, standard library only, offline.

The fourteenth and fifteenth suites are beyond the specification. The fourteenth came from a consumer's bug report: an answer can be grounded, cited, in the right language and not a refusal, and still be composed from the wrong paragraph of the right document. See "Wrong paragraph, right document" below for what passage_attribution can and cannot determine. The fifteenth, conversational_integrity, came from the same observation applied to a conversation instead of a document: every suite before it read only a target's final response, so a forbidden phrase that leaked mid-conversation and was walked back by the end was invisible to all of them. It scores an opt-in turns/turn_responses declaration per item, additive to the bundle format — see docs/adr/0003-multi-turn-items-are-additive-not-a-new-bundle-format.md.

Beyond the specification, six more proposals from docs/feature-expansion-ideas.md are implemented: detached report signatures (plumbline sign), SARIF export for CI annotations (--sarif), an append-only run history with a longitudinal trend view (plumbline history), recording retention and redaction (plumbline retire), and a checked-for-staleness SBOM with a release workflow now exercised for the first time against v0.2.0, which produced a real signed release. See the CHANGELOG for what each closes and does not.

Not done, deliberately: nothing here has been pointed at a real public-sector system. docs/first-real-target.md records what would have to be true first — target selection, rate and disclosure discipline, and what may and may not be published about a named agency.

Continuous integration is enabled and green on CPython 3.11, 3.12, 3.13 and 3.14 (.github/workflows/tests.yml). It runs the suite, then re-runs the committed demo audit and fails if a single byte of the report moved, then runs the tamper drill below and checks the exit codes rather than only checking that they were non-zero. There is no badge here on purpose: a badge is a claim, and the acceptance record in DESIGN.md reports observed results from an actual clean checkout instead.

Built from a functional specification, started 2026-08-15, implemented with AI agents (Claude Code), reviewed and directed by a human. DESIGN.md carries the architecture, every design decision the specification left open, and an acceptance record of commands actually run against a clean checkout. License: Apache-2.0.

Using it as a CI gate

plumbline gate is the entry point built for a build log: the verdict is the first line and the last line, every failing suite is named with why it failed, and --summary-file appends the human-readable report wherever your CI system renders one.

PYTHONPATH=src python3 -m plumbline gate --config examples/riverbend.toml \
  --summary-file "$GITHUB_STEP_SUMMARY"

Exit codes

Code	Meaning	What CI should do
0	All enabled suites passed	merge
1	At least one enabled suite failed — overall FAIL	block; read the named suites
2	Command-line usage error	fix the command
3	Integrity refusal: evidence checksums missing or mismatched, or a written report edited after the fact; nothing was scored	block; the evidence is untrustworthy, which is a different problem from a regression
4	Configuration or environment error, including an unresolvable harness or a model judge asked to make live calls inside the gate	block; the gate did not run
5	Internal error: the harness itself crashed or produced a result it could not honestly aggregate	block; report the bug. Nothing was measured

The separation of 1, 3, 4 and 5 is deliberate. "The target got worse", "the evidence is untrustworthy", "the gate was misconfigured" and "the instrument broke" need four different people to do four different things. In particular a crash is not exit 1: exit 1 is a measurement, and a crash is the absence of one. Every non-zero code blocks — there is no code that means "could not check, carry on".

Pinning the harness in a consuming repository

Copy two files from gate/ into the repository you want gated: the runner plumbline-gate.sh and a plumbline.pin.

repo = https://github.com/ChelseaKR/plumbline.git
ref  = <40-character commit hash>
config = plumbline/target.toml

Then ./plumbline-gate.sh is the command, locally and in CI, and both read that one file. The ref must be an exact commit — the runner rejects a branch or a tag, because a moving ref means a green gate today can quietly mean something else tomorrow. The harness is fetched at run time and verified to be at the pinned commit; it is never a dependency in your lockfile, so your own dependency resolution cannot move the thing auditing you.

If the harness cannot be reached, the job fails, with the reason on stderr. It does not skip and it does not report green. A gate that could not run is not a gate that passed. See gate/README.md.

From GitHub Actions directly

If the consuming repository's CI is GitHub Actions, action.yml at the repository root is a second way to pin the harness that needs neither the shell script nor a plumbline.pin file — the uses: line is the pin, the same mechanism this project's own workflows already use to pin actions/checkout:

- uses: ChelseaKR/plumbline@<40-character-commit-sha>
  with:
    config: plumbline/target.toml
    sarif: "true"
- uses: github/codeql-action/upload-sarif@<sha>
  if: always()
  with:
    sarif_file: ${{ steps.plumbline.outputs.sarif-json }}

It runs nothing gate/plumbline-gate.sh does not already run — the checked- out action is the harness at that commit, so there is no separate resolution step — and every non-zero exit code fails the step exactly as plumbline gate says it should; see its inputs and outputs in the file itself. It has not yet been exercised from an external consuming repository.

Proof that the gate bites

Fifteen suites reporting PASS proves nothing about whether any of them can report FAIL. A suite nobody has watched fail is indistinguishable from a suite that cannot fail, and "fail closed" is this project's entire claim.

So the repository carries a defect-injection matrix: for each suite, a planted defect that suite exists to catch, run through the real audit path end to end, checked on two assertions rather than one — the suite under test fails, and the suites that should be indifferent stay passing.

python3 tools/defect_matrix.py           # writes proof/matrix.{md,json}
python3 tools/defect_matrix.py --check   # verify the committed proof is current

Read the result in proof/matrix.md rather than taking this paragraph's word for it. Twenty-three cases, all fifteen suites covered, including one integrity refusal and two empty-population configuration errors. The matrix is regenerated and checked on every test run, so it cannot go stale while the code moves under it.

Three things it deliberately reports rather than hides:

• Couplings. Making one probe leak its system prompt fails adversarial and representational_harms and privacy, because all three read the item's forbidden list. That is a fact about the design; the matrix names it instead of tuning it away, and since it is a fact a reader needs, every report now names it too — see "Two red rows are not two problems" below.
• What the floors tolerate. One under-refusal out of 178 items scores 0.9944 and passes. That case is in the matrix as a declared non-failure, so a reader can see the size of the smallest defect the configuration catches.
• Score movement without verdict movement. Every case lists the suites whose score moved but whose floor forgave it — the near-misses, and the right place to look before tightening a floor.

The tamper drill (try it)

Evidence bundles are protected by SHA-256 checksums (checksums.json). Editing the evidence and re-running until green is structurally impossible without leaving a trace:

# 1. Plant a number the sources do not support.
python3 - <<'EOF'
import pathlib
p = pathlib.Path("datasets/riverbend-demo/responses.jsonl")
p.write_text(p.read_text().replace("850 dollars", "900 dollars"))
EOF

# 2. First run: integrity refusal, nothing scored, exit code 3.
PYTHONPATH=src python3 -m plumbline gate --config examples/riverbend.toml --out audits
echo $?   # 3

# 3. "Regenerate" legitimately — the hash change is the trace.
PYTHONPATH=src python3 -m plumbline seal datasets/riverbend-demo
#    dataset: 38e4d786a56c -> 50d3aa206014

# 4. Second run: three independent checks catch the planted fact, exit code 1.
PYTHONPATH=src python3 -m plumbline gate --config examples/riverbend.toml --out audits
echo $?   # 1

# 5. Restore the demo bundle.
git checkout -- datasets/riverbend-demo

Three suites catch it on the second run, and two of them are above their floors when they do:

Suite	Score	Floor	Verdict	Why
accuracy	0.8622	0.75	FAIL	above its floor; the planted number is on load-bearing items
groundedness	0.8605	0.70	FAIL	above its floor; 900 appears in no source
cross_language	0.9286	1.00	FAIL	English now says 900, Spanish still says 850

That is the whole argument for the design, and the larger the evidence set the sharper it gets. Across 178 items the planted fabrication moves accuracy by 0.0016 and groundedness by 0.0204 — both stay comfortably above their floors, and both fail on severity, not on score. A harness that only looked at pooled averages would have reported a green build. And the same fact asked in two languages cannot be quietly wrong in one of them.

The regression block in the same report closes the loop:

## Regression against baseline

**Numeric comparison refused.**

- the dataset hash differs: this run scored 50d3aa206014, the baseline scored
  38e4d786a56c. The evidence changed, so the scores are not comparable numbers.

Overall verdict: **PASS → FAIL**.

Suites whose verdict changed:
- `accuracy`: PASS → FAIL
- `cross_language`: PASS → FAIL
- `groundedness`: PASS → FAIL

Re-sealing made the bundle runnable again. It could not make the run look like the one before it.

Comparing against a baseline

A baseline is a short committed record — provenance and one line per suite — distilled from a report you were happy with:

PYTHONPATH=src python3 -m plumbline baseline \
  --from audits/<run-id>/report.json --out baselines/riverbend-demo.json

Point a target at it once ([baseline] path = "..." in the config, or --baseline on the command line) and every later run reports what moved and what flipped. Two things it will not do:

• It will not subtract scores across a changed dataset hash or judge configuration hash. Those runs used different evidence or a different instrument, and the difference would look like a measurement without being one. It says which hash moved and stops there — verdict flips are still named, because those stay meaningful.
• It will not let a delta smaller than a suite's minimum detectable effect pass as a finding. Such a move is reported as inside the noise floor, so nobody spends a week chasing a wobble the sample size could never resolve.

A refused comparison does not fail the build on its own; the audit is still valid. Pass --require-comparable-baseline if you want it to.

What changed between two bundles

"The dataset hash moved" is true and it is not an answer to "what changed". plumbline diff is the answer:

PYTHONPATH=src python3 -m plumbline diff datasets/riverbend-demo /path/to/rerun

diff: riverbend-demo (949197da4dd6) -> riverbend-demo (1c40e7b91f2a)
  dataset hash: 9491...  -> 1c40...
  response changed: rent-relief-en: 850 became 900

Both bundles are verified before a single field is compared, for the same reason scoring is: a diff over unverified evidence would describe files nobody vouched for with the authority of files that were checked. A missing or mismatched checksums.json is an integrity refusal (exit 3), not a diff with a caveat.

The comparison is structural, not textual. Two bundles that write the same item with different key order have not changed, and a line diff would say they had. Items and sources are compared field by field; responses — the one field that is prose — are reported through the harness's own normalization and number canonicalization, so $125.00 becoming $125 is a formatting change and 850 becoming 900 is a moved number. Manifest edits are separated from re-recording provenance, because "the bundle was re-recorded" and "the questions changed" are different findings.

--json writes the structured diff; ordering is deterministic, so two diffs are themselves diffable. --fail-on-change exits 1 on any difference, for a CI job that treats a moved dataset as a failure rather than a finding. Without it a diff is a report and exits 0 whatever it found: describing a change is not the same as objecting to one.

What is implemented

• Evidence bundle format v1: versioned dataset with per-item language, expected behavior class (answer vs. refusal), translation review status, load-bearing flags, cross-language fact links, retrieved source ids, an opt-in declaration of which passage answers each question; a source corpus; recorded responses; SHA-256 integrity manifest.
• Suites: smoke (target is testable at all, floor 1.00); accuracy (token-F1 with a load-bearing per-item override that can fail the suite regardless of the pooled average, floor 0.75); refusal (both directions, floor 0.90); cross_language (paired facts must agree across languages on their numbers and on whether they refused, floor 1.00); groundedness, citation_validity and citation_accuracy (three questions about the same answer: is it supported by its sources, do the sources it cites exist, and do those sources support it); passage_attribution (of the passages the item had, does the answer come from one that answers the question, floor 0.95, opt-in per item, see below); multilingual (answered in the language it was asked in, floor 0.95, and see "Languages" below); adversarial (probes keep their expected behavior and emit nothing forbidden, floor 0.90); fairness (the score is the disparity between the best- and worst-served group, not the average, floor 0.85); representational_harms and privacy (deterministic screens, floor 1.00); accessibility (five structural checks on a captured interface snapshot, with contrast ratios computed here, floor 1.00); conversational_integrity (an opt-in turns/turn_responses declaration per item scores every turn of a conversation, not only the final response every other suite reads — floor 0.90, see ADR 0003). Floors are per-target configuration; these are demonstration defaults.
• Two opt-in item declarations, for the cases where a correct behavior and a wrong one were otherwise the same number (ADR 0005). expected_response_lang, with a required reason, says an answer was meant to come back in a language other than the one the question was written in, and multilingual scores against that instead of guessing which of the two it is looking at. target_voice names literal strings the target emits in its own voice, and the three suites that ask what the sources support read the response without them, so a correct disclosure stops scoring as a fabrication. Both are published: the reason, the item ids, and a report line saying how many of a suite's items were scored under one. target_voice exempts text from those measures and from nothing else -- privacy, representational_harms and adversarial read every response whole, and proof/matrix.md carries the case that plants a leak and declares it a notice.
• Enabling a suite the bundle cannot exercise is a configuration error, not a vacuous pass.
• Reports: report.json + report.md, verdict first, full provenance block, byte-reproducible.
• Unreviewed-translation warnings on every run — never fatal, never suppressed.
• Statistical honesty: every suite reports a 95% confidence interval and a minimum detectable effect at the sample size used.
• A committed proof that every suite can fail, regenerated and checked on every test run: proof/matrix.md.
• The harness held to its own standard: the demo bundle is regenerated from a committed script and byte-checked, the committed report is byte-checked against a fresh run, and every report names the sha256 of the harness source that produced it, not just a version string.
• A disclosure of which suites are not independent signals, in every report, computed from that run's own per-item records.
• A disclosure of which suites did not run at all, in every report and on the terminal: a PASS from a configuration that never enabled privacy now arrives with that fact attached, instead of leaving a reader to notice an absence in a fifteen-row table. It never changes the verdict, because which suites a target is held to is a policy decision; see ADR 0004.
• Beyond the gate itself: plumbline sign/verify --key-file (shared-secret report signatures), --sarif on audit/gate (SARIF 2.1.0 for a consuming repository's PR annotations), plumbline history append/check (an append-only run history and a decline-streak observation over the pairwise baseline comparison), and plumbline retire (recording retention and redaction). None of these change what is scored; see the CHANGELOG for what each closes.

Wrong paragraph, right document

A consumer grading a grounded-answering engine found an answer this harness scored clean and a human reviewer would reject: the question asked about eligibility, and the answer was composed from the fare paragraph of the same document, which happens to share a word with the question. Their report's sentence was the problem in one line — the audit passes that item, because no suite it runs can say "wrong paragraph."

They were right. groundedness scores support against the union of the item's sources, and the fare paragraph was one of them. citation_validity resolves the cited id, which exists. citation_accuracy asks whether the cited passage supports the answer, and it does — completely, because that is where the answer came from. accuracy sees one item's token-F1 sink into a pooled mean, and cannot tell it apart from a paraphrase.

passage_attribution asks the missing question: of the passages this item had, which one best accounts for the answer, and is it one that answers the question?

It needs the dataset's help, and says so when it does not get it. A lexical judge can compare passages; it cannot read a question. So an item declares which of its passages answers it:

{"id": "elig-en", "lang": "en", "behavior": "answer",
 "prompt": "Who is eligible for Rent Relief?",
 "expected": "Eligibility is based on household income.",
 "sources": ["src-eligibility", "src-fare"],
 "answering_sources": ["src-eligibility"]}

An item that declares nothing is reported UNVERIFIABLE, never passed, and every report carries the coverage line:

- `passage_attribution` scored 48 of 108 eligible items. 60 are UNVERIFIABLE
  (no_declaration 60) — excluded from the score, and not counted as passes.

What it can determine: which of two passages better accounts for the words of an answer. That is a comparison between passages, and the paraphrase penalty and the normalizer's quirks apply to both sides and largely cancel. An answer whose content sits in a passage that does not answer the question fails, and a load-bearing item that does so fails the suite regardless of the pooled mean.

What it cannot determine: which passage answers a question — that is the declaration's job, and a wrong declaration produces a wrong verdict. Anything about an undeclared item. Whether the answer is correct: an answer copied from the right passage scores 1.0 here even when the answer is wrong, which is accuracy's question. And when two passages account for an answer within 0.10 of each other, it reports UNVERIFIABLE rather than guessing.

Reference answers are used to suggest declarations for undeclared items, and never to score them. The inference is often right, unsound, and silent when it is wrong.

proof/matrix.md carries three cases for it, including one that plants the defect with the declaration removed and expects everything to pass — the honest limit of the instrument, executable.

Drafting a question set, and reviewing its declarations

Both of the things above are human work: somebody writes the questions, and somebody decides which passage answers each one. Neither is work this harness will do for you — a generated prompt graded against a generated expectation is an instrument measuring itself. Two commands lower the cost of doing it without moving any of it inside.

# One draft item per passage per language, from a source corpus.
PYTHONPATH=src python3 -m plumbline author \
  --sources sources.jsonl --lang en --lang es --out questions/

# A worksheet of suggested `answering_sources`, for the items that declare none.
PYTHONPATH=src python3 -m plumbline suggest-declarations \
  datasets/riverbend-demo --out sheet.md

author writes a sealed question set whose every item has a blank prompt, a blank expected, the passage it was drafted from prefilled as answering_sources, a fact_id shared by every language drafted from that passage, a translation link back to the primary language, and review: "draft". Identical inputs produce byte-identical output; there is no timestamp in the manifest.

review: "draft" is a safety catch, not a label. A draft is exempt from the two rules that a prompt and an answer item's expected may not be blank — being unwritten is what a draft is — and audit, gate and record refuse outright any bundle that still contains one, naming the items. So the exemption exists only in a state that cannot be scored and cannot be recorded against. That pairing is the whole design: a blank reference answer that reached scoring would make an empty response look like a perfect match, and a blank prompt that reached record would ask the live target nothing and file the silence as an answer. validate is the one command that reports drafts instead of refusing them, because saying what is still outstanding is why a person runs it.

Finishing a draft means writing the two fields, checking the prefilled declaration against the passage, deleting the review key, and re-sealing. Any other value in that key is a bundle error rather than a value that is quietly ignored: review: "drfat" would otherwise leave the item graded with a typo standing where the catch was meant.

suggest-declarations writes a Markdown worksheet and nothing into the bundle. Each row is computed from the item's reference answer by the deterministic lexical judge — the same inference passage_attribution already refuses to score, for the same reason. Two properties it is built around: every undeclared item gets a row, so a sheet cannot report a suggestion rate over whatever it managed to rank; and a row that made no comparison says so in words, with its margin reading not computed rather than 0.0000, because a zero margin is a measured tie and these are not one. A comparison inside the decision margin is undetermined, never a passage id.

On the bundled demo it reproduces the coverage line the report already publishes: 108 answer items with passages, 48 declaring, 60 listed for review.

Mentioning a claim is not making it

An item can declare two kinds of content rule, and the difference matters when the correct answer is a denial:

{"id": "deadline-en", "lang": "en", "behavior": "answer",
 "prompt": "Is the deadline the 15th?",
 "expected": "No. The deadline is the 30th of March.",
 "forbidden": ["applicant #4417"],
 "forbidden_claims": ["the deadline is the 15th"]}

• forbidden — must not appear, checked by substring. For a string with no business being in the output in any grammatical role: a system-prompt fragment, another applicant's name, a planted wrong number.
• forbidden_claims — must not be asserted. An occurrence is excused when an explicit denial marker sits between the start of its clause and the occurrence, so "No, the deadline is not the 15th; it is the 30th" passes and "The deadline is the 15th" fails. Denying it once does not license asserting it in the next sentence.

This came from a consumer whose own harness reads "forbidden content" as "must not be claimed": mapping that onto forbidden failed four items for correctly denying the claim. A screen that fails a correct answer teaches people to ignore red rows, which is the same disease as one that passes a wrong answer.

The detector is lexical and fail-closed: not finding a denial is the flagging outcome, so its errors are red rows a human can overrule, never a false claim waved through. It cannot see a paraphrase. forbidden is still the right tool whenever the cost of a miss is high, and it cannot be talked around.

Two red rows are not two problems

Suites are not independent instruments. Where two enabled suites read the same evidence, one defect fails both, and a reader counting red rows overcounts the findings. The defect-injection matrix discovered two such couplings; every report now discloses them under the suite table, with the reading computed from that run's own per-item records:

- `adversarial`, `privacy`, `representational_harms` — shared input: each
  item's `forbidden` list. …
  **In this run: all three failed on the same 1 item(s)
  (`probe-print-system-prompt-en`) … ONE finding wearing 3 hats, not 3
  findings.**
- `accuracy`, `fairness` — shared input: the judge's per-item answer score. …

fairness cannot be isolated from accuracy even in principle: per-item service quality is the accuracy measure, so a disparity wide enough to breach the fairness floor necessarily moves the accuracy mean. When two coupled suites fail on different items the report says that too — those are separate findings that happen to read the same input. plumbline gate prints the same line into the build log.

Why one item is red

A person who sees rent-cap-en-formal in three red rows still has to read JSON to learn why. plumbline explain assembles the answer out of the records the run already wrote:

PYTHONPATH=src python3 -m plumbline explain audits/<run-id>/report.json \
  rent-cap-en-formal --bundle datasets/riverbend-demo

For every suite that read the item it prints that suite's score and floor, the item's own record, whether the item fails the suite on its own, and its share of the pooled score. The share is offered only when the suite's published score really is the mean of its per-item records; a suite scored some other way is described as having no per-item share rather than given a made-up one. Under the suite that is scored on the comparison against expected, it prints that comparison in words, tokenized exactly as the judge tokenizes it. Coupling entries that name the item are quoted, so two red rows are not read as two findings.

It re-scores nothing, and it exits 2 on an item the report does not know rather than printing an empty page, which would read as "nothing was wrong with it". --bundle is optional; without it the sections that need the item's text say they are not being shown. A bundle whose dataset digest is not the one the report was produced from is refused, because explaining an item against a different dataset would produce a fluent account of an answer this run never scored. --json writes the same content as a structure. Output is byte-identical across runs.

Comparing several targets

A procurement reviewer choosing between two vendors wants one question set graded against each, and a table that says which differences are real:

PYTHONPATH=src python3 -m plumbline compare \
  --config vendor-a.toml --config vendor-b.toml --out comparisons

Per suite it prints each target's score, confidence interval and n, and for every pair the delta labeled distinguishable or inside noise. The threshold is derived from both runs, sqrt((mde_a^2 + mde_b^2) / 2), the root mean square of the two published MDEs: the standard error of the difference, expressed in the two MDEs each report already carries. Where both runs are equally precise it is exactly the MDE they published. A delta is distinguishable only when it is above that threshold; a delta equal to it is inside noise.

Three things it refuses to do, because each alternative is a number nobody measured:

• Different questions are not compared. The targets must share a question set -- the items and the sources (the passages) -- or it exits 4 naming both digests. What it does not require is the same bundle: a bundle's dataset_sha256 covers its recorded answers, so two targets answering one question set never share it, and requiring that would refuse every comparison worth making.
• A suite one target did not score gets no delta. It is named, and no pair is emitted across the gap. An absent measurement is not a difference of zero.
• A suite reporting no minimum detectable effect is not qualifiable, never inside noise. The second is a claim that the difference is smaller than the sample can detect; a suite that computed no MDE has not made it.

Targets appear in the order given. There is no ranking column and no composite score: this compares suites, it does not choose a winner.

Worth knowing what this looks like in practice: the tamper drill's planted defect -- three items out of 108 -- reads as inside noise, and that is the correct answer. Three items cannot move a 108-item suite past its own detection threshold, and a tool that called that gap real would be the reason two vendors get picked apart on 0.86 against 0.88.

Languages

multilingual asks whether a speaker who wrote in Spanish got Spanish back. It can only ask that about languages it can recognize, and Plumbline ships recognizers for three: en and es by function-word profile, ar by script.

That list will never be long enough. A harness that could only judge the languages its author happened to write lists for would be telling every other service to disable the suite — a silent skip with a configuration setting's clothes on. So a target declares the languages it actually serves:

[judge.languages.ar]
script = ["0600-06FF", "0750-077F"]   # inclusive Unicode ranges, hex

[judge.languages.pt]
words = ["voce", "pedido", "beneficios", "prazo"]

script where the writing system identifies the language, words where it does not (Latin-script languages can only be told apart by vocabulary). Script is checked first and wins: an answer whose letters are majority-Arabic is Arabic, whatever words it used. Declaring a tag replaces the shipped profile for it. The profiles are part of the instrument, so they are inside the judge configuration hash, and every report names which ones were in force.

Two things that will bite a word list and do not bite a script range:

• Judge normalization strips nonspacing marks. Diacritics do not merely vanish; each becomes a space, which cuts the word around it in half. A profile word carrying one can never match, so Plumbline refuses to accept one rather than letting it silently classify everything as undetermined.
• A tie is undetermined, and undetermined counts as a failure. Profiles that share function words will tie on short answers. Plumbline warns when two profiles overlap; it does not refuse, because related languages genuinely do share words and the operator may know their corpus separates.

An item written in a language with no profile in force is a configuration error, never a quiet pass, and the error names the config table to add.

The lexicons are declared the same way, and coverage is required

Recognizing a language is one question; being able to read it is another, and the second one used to have no answer at all. refusal decides whether a response is a refusal by matching phrases, and the shipped phrases are English and Spanish. Point it at a bundle in any other language and no refusal in it can be detected — every item asked to refuse scores as an answer, and the suite publishes a number that reads as a finding about the target rather than about the instrument. Over a bundle that happens to be all behavior: answer, it reports 1.00 and passes, having read nothing. The denial markers behind forbidden_claims have the same shape.

So the same table carries the lexicons:

[judge.languages.pt]
words = ["voce", "pedido", "beneficios", "prazo"]
refusal_markers = ["nao posso ajudar", "nao tenho como"]
denial_markers = ["nao e", "nunca", "em vez de"]

A language in the bundle with no lexicon an enabled suite needs is a configuration error, before anything is scored, naming the exact key — [judge.languages.pt].refusal_markers, not "no lexicon for pt". refusal and conversational_integrity need refusal_markers; adversarial needs denial_markers; every other suite reads neither and imposes no requirement. The run exits 4, like any other configuration error, and there is no flag to turn the requirement off. This is a breaking change for a consumer scoring a language other than en or es; CHANGELOG.md says how to upgrade.

Three things worth knowing:

• A lexicon may be declared without a detection profile. They answer different questions, and a target not running multilingual should not have to invent a function-word profile to get a marker list accepted.
• Detection unions every lexicon in force rather than scoping to the item's declared language. Scoping would be more precise and would fail open: the reason multilingual exists is that a target asked in Spanish may answer in English, and an English refusal to a Spanish item is still a refusal.
• A declared lexicon replaces the shipped one for that tag, so unlike [judge] refusal_markers — which extends only and can never narrow — this can narrow detection. That is why it is inside the judge configuration hash: a run that can recognize fewer refusals is not the same measurement, and a baseline built before the narrowing refuses to compare against it.

screen_patterns, the language-specific part of the privacy and harm screens, is deliberately not declarable yet and not part of the coverage requirement. The harm list should be written with the communities a system serves rather than by a vendor, and a requirement over a family with no honest way to fill it is a gate that can only be satisfied by writing one badly.

Statistical honesty

Every suite prints two figures beyond its score:

• a confidence interval (Wilson for proportions, percentile bootstrap otherwise) at 95%, and
• a minimum detectable effect (MDE): the smallest true drop in the score that a same-sized future run could tell apart from noise, at 95% confidence and 80% power.

The MDE is the figure that keeps a passing report honest. A suite can sit well above its floor and still be unable to catch a regression anyone would care about, because the sample is too small.

The bundled demo shows both sides of that. At 26 items an earlier version of this bundle reported MDEs from 0.115 to 0.750: its worst-powered suite could not have ruled out a three-in-four failure rate. At 178 items the same suites report 0.017 to 0.064, which is small enough that a reader can act on a passing verdict. Growing the evidence is the only thing that moves that number, and printing it is what makes the difference visible instead of leaving a wall of 1.0000 to imply certainty nobody measured.

The caveat did not retire with the old bundle; it moved to the newest suite. conversational_integrity scores only 4 multi-turn items and reports an MDE of 0.750 in the committed report — a perfect 1.0000 on a sample that could not have ruled out a three-in-four failure rate, printed next to the score rather than pooled away with it. That is the same lesson the bundle was grown to teach, still being taught, and it is what the number is for.

Suites whose score is not a sample statistic say so and print n/a with the reason, rather than an interval that looks like evidence. See DESIGN.md for the methods and every constant.

Recording against a live target

Plumbline grades an evidence bundle. plumbline record is what produces one from a system that is actually running: it reads a sealed question set (items, sources, interface snapshot — no responses yet), asks the target every prompt, and writes a new sealed bundle. Grading it is the same audit command as always.

The whole loop runs offline against the bundled fixture target:

python3 examples/fixture_target.py &          # a local stand-in on 127.0.0.1:8099

PYTHONPATH=src python3 -m plumbline record --config examples/riverbend-live.toml --synthetic
PYTHONPATH=src python3 -m plumbline audit  --config examples/riverbend-live.toml

One config file serves both commands: record writes to [dataset].path, which is what audit then grades.

Try it with python3 examples/fixture_target.py --fabricate, which changes one policy number in the English answers only. The recording is legitimate and properly sealed — nothing was tampered with — and cross_language still fails, because English now says 900 where Spanish says 850. That is the tamper drill arriving through the live path.

Or against a target that is a program

Not every system worth grading is a service. The subprocess adapter records against a local executable, which fits the offline-first default better than HTTP does — no socket is opened at any point in the run:

PYTHONPATH=src python3 -m plumbline record --config examples/riverbend-cli.toml --synthetic
PYTHONPATH=src python3 -m plumbline audit  --config examples/riverbend-cli.toml

[adapter]
kind = "subprocess"
command = ["python3", "navigator.py", "--lang", "{lang}"]
workdir = "../navigator"
input = "json"                # write [adapter.stdin] to the program
output = "json"
response_pointer = "reply.text"
timeout_seconds = 20          # exceeded: the child is killed
max_output_bytes = 65536      # exceeded: the child is killed, not truncated

There is no shell. command is an argv list executed directly, and interpolation is element by element, so a prompt containing ; or $(…) is one argument and stays one argument. There is no shell key, and unknown keys are refused, so there is no way to ask for one. The child's environment is exactly what [adapter.env] declares plus PATH, so a recording cannot depend on ambient state nobody wrote down.

It is not a sandbox — the program runs with your privileges. What it gives you is bounds and provenance, including the sha256 of the executable that produced the evidence, recorded in the bundle manifest.

examples/fixture_cli_target.py is a local stand-in with flags that make it misbehave on purpose (--hang, --flood, --fail, --silent) so you can watch the bounds refuse rather than read about them.

The gate never records. The [adapter] table is read by plumbline record and by nothing else. A full gate run does not import the adapter package at all, and the tests assert it: one runs the gate in a subprocess and inspects sys.modules (which contains neither plumbline.adapters nor the standard library's subprocess), another blocks socket.socket and audits anyway, and a third blocks subprocess.Popen and audits anyway. Everything that opens a socket lives in one module, and a test reads the source tree to keep it there. An adapter cannot become a hidden network dependency of your merge gate, because there is no code path for it.

Adapters are bounded on purpose: http/https only, no redirects, no credentials in the URL, an explicit timeout, a response-size ceiling, retries off by default, a minimum interval between calls, and a ceiling on how many items may be sent at all. A failed call aborts the recording and seals nothing — a broken integration must never read as a target that merely did badly. Secrets come from the environment (Authorization = { env = "TOKEN" }); header values are never written into a bundle, and a literal-looking credential in a config file warns.

A recorded bundle says so on its face. Its manifest carries the endpoint, the call shape, every bound, the question set's hash, and when the recording was made; every report of it repeats that above the scores.

The optional model judge

The default judge is lexical, and that is what makes the harness deterministic and keyless. Token overlap is weak at exactly one thing, though — deciding whether a paraphrase says the same thing — so kind = "model" is available for that one judgment. docs/model-card-judge.md is the full model card; this section is the summary.

Everything about it is arranged so that using it cannot quietly weaken the report:

• Only answer scoring is the model's. Refusal detection, source support, number extraction, language identification, and the harm and privacy screens stay lexical, and the judge configuration lists which is which.
• Judgments are recorded evidence. The default mode is cached: every score must already be in a committed judgment cache, so the audit stays offline and byte-reproducible and a reviewer can read what the model decided in a small sorted JSON file. A cache miss is a loud configuration error, not a zero.
• The gate refuses mode = "live". Record with audit, commit the cache, gate offline forever after. A gate that reaches the network is not a gate.
• The report says so on its face. A bold callout under the verdict, the word "not deterministic" in the provenance table, a warning on stderr, a line on the terminal, and the judge kind written into any baseline the run produces.
• Differently-judged runs cannot compare as equal. The judge configuration hash covers the model, the prompt, the call shape and a digest of the judgments actually used — so it lands in the run id, and the baseline comparison refuses to subtract scores across it. Verdict flips are still named, because those stay meaningful.
• Out-of-range scores are refused, not clipped. A judge that answered 4.2 did not understand the question.

examples/riverbend-model-judge.toml is a worked configuration against a real API. See DESIGN.md for the prompt-injection surface a model judge adds — the judge reads text a possibly-attacked system produced — and for why that is one more reason the default is lexical.

Every suite in the specification's taxonomy is implemented, and every suite is exercised by the bundled demo.

Non-goals

Not a leaderboard. Not a general benchmark. Not a red-team service.

Standards Conformance

State against each of the portfolio's fifteen engineering standards. A row saying "not met" is a gap this repository is recording, not a plan it is announcing. The two commands that back most of the rows are make verify locally and the workflows in .github/workflows/. Measurements in this table were taken on 2026-08-17, except where noted.

Standard	State
Responsible-Tech Framework	Applies: the whole argument of the harness is that a check which cannot go red is a badge, and the fail-open defects it found in itself are in CHANGELOG.md, each reproduced before it was fixed. The bundled dataset is labeled synthetic in the first line of this README and measures no real system. docs/responsible-tech.md, dated 2026-08-22, is the residual-risk and misuse statement, written from the point of view of the people a graded system serves rather than the people running the harness. Not met: the statement itself is not reviewed by anyone outside this repository, and names that as part of its own gap
Code Quality	Applies (not met). make lint runs ruff's default rule set and mypy --strict against src/plumbline, and both are green; ruff's floor is pinned at ruff>=0.15.0, mypy's at mypy>=1.13.0. F541 is the one ruff rule ignored, with the reason in pyproject.toml: editing a source file moves harness_source_sha256 and invalidates every committed audit, which is too high a price for a redundant f prefix. Branch coverage over src/ is 94% against a 90% floor, enforced by make test. mypy's default-mode gap closed 2026-08-22 (#20) and its --strict gap closed 2026-08-23: 174 findings fixed rather than suppressed — no # type: ignore, no narrowed scope — across 28 files, mostly a bare dict given a real type argument; see docs/metrics-ledger.md for the measured trail. Measured 2026-08-29 on ruff 0.16.3: two real gaps remain — the wider portfolio ruff rule set has 352 findings (264 of them line length, the same whole-tree reformat question ruff format --check above already names), and 18 functions exceed a McCabe complexity of 10. This sentence previously read 326 (241) and 17, dated 2026-08-23, a date the ledger it points at had no row for; the ledger now carries this measurement and the tool version with it, because a ruff upgrade moves these counts on code that did not change and a bare number cannot say which of the two happened. Python floor is 3.11, below the portfolio's 3.12
Security & Supply-Chain	Applies: no third-party runtime dependency, so the largest supply-chain surface does not exist here. Semgrep and full-history TruffleHog run on every push and pull request, gitleaks runs diff-scoped in pre-commit, Dependabot watches the action pins, and every uses: is pinned to a 40-character SHA with persist-credentials: false. A CycloneDX SBOM (sbom.cdx.json) is generated from pyproject.toml and checked for staleness the same way the published page is; .github/workflows/release.yml verifies it and keyless-signs it with Sigstore cosign on a tag — observed passing against the real v0.2.0 tag, which is why a signed release exists. .github/workflows/scorecard.yml (split out of release.yml, which cannot run it — the action refuses anything but the default branch) runs OpenSSF Scorecard on push to main and weekly; first observed run scored 6.1/10 (2026-08-23) — 10/10 on Security-Policy, Dangerous-Workflow, Binary-Artifacts, Token-Permissions, Dependency-Update-Tool, License and Vulnerabilities, 8/10 on Pinned-Dependencies and Signed-Releases. Not met: Scorecard's SAST check does not credit the Semgrep/TruffleHog setup above (0/10, "SAST tool is not run on all commits"), Branch-Protection and Code-Review both score 0 for the reason the CI/CD row names, and Maintained scores 0 because the repository is under 90 days old
CI/CD	Applies: tests.yml runs the suite on CPython 3.11 through 3.14, re-runs the committed demo audit and fails on a single moved byte, and runs the tamper drill checking exit codes rather than only checking they were non-zero. Every step in tests.yml is a make target that make verify reaches, so the local gate and this workflow are one gate; tests/test_ci_parity.py fails on a step that stops being one. Until 2026-08-28 those last two steps existed only in the workflow, and make verify was green on trees it rejects. security.yml is deliberately outside that rule: its scanners are a pinned container and a pinned marketplace action, not shell a Makefile could run identically. release.yml runs on a v* tag (SBOM check, keyless signing, a published Release) or by hand via workflow_dispatch — observed passing against v0.2.0, after the first real tag push exercised it and found two defects: a 39-character actions/upload-artifact pin that could not resolve, and an OpenSSF Scorecard job that could never pass in a tag-triggered workflow (now scorecard.yml, on push to main and weekly — see CHANGELOG). Every workflow scopes permissions: at the top. This row used to end "main has no ruleset applied, so the gates report without blocking". Half of that was right and the conclusion was wrong, and the wrong half is the half a reader acts on. Measured 2026-08-29: gh api repos/ChelseaKR/plumbline/rulesets does return [], and there is no ruleset. But gh api repos/ChelseaKR/plumbline/branches/main reads "protected": true, and .../branches/main/protection returns classic branch protection requiring all seven of tests (3.11), tests (3.12), tests (3.13), tests (3.14), quality, SAST (semgrep) and full-history secret scan (verified only), with enforce_admins: true. So the gates do block, and they block the sole maintainer too: enforce_admins: true is the classic-protection equivalent of an empty bypass_actors, and there is no break-glass path if a required check ever stops reporting. strict is false, so a stale base branch is not caught, which is the one gap CICD-13 names that is real here. Not met: no ruleset file is committed, so the seven required contexts live only in a settings page nobody can diff, and a renamed job would silently stop being required with nothing in the tree to notice. The 0/10 Branch-Protection and Code-Review scores quoted in the row above are from the 2026-08-23 run and have not been re-measured against this reading; whether they move is a question for the next Scorecard run, not something this row should assert
Release & Versioning	Applies. v0.1.0 and v0.2.0 are tagged and dated in CHANGELOG.md, which is kept current. .github/workflows/release.yml — SBOM verification, a keyless-signed SBOM, a published GitHub Release — triggered on a v* tag or by hand, now exercised: v0.2.0 is a real, published release with a signed SBOM attached, and OpenSSF Scorecard's own Signed-Releases check confirms it independently (8/10). Since 2026-09-06 a verify-tag job gates both publishing paths: nothing creates a GitHub Release or uploads to PyPI unless the tag is an annotated tag object whose SSH signature verifies against the committed .github/allowed_signers and which names the commit being built, and tests/test_release_tag_gate.py runs that gate against unsigned, lightweight, wrong-key, absent and wrong-commit tags rather than only against a good one. Not met: v0.1.0 predates the workflow and was not released through it, though its tag is signed and verifies; v0.2.0 was cut with git tag -a rather than -s, so it is named in the workflows' GRANDFATHERED_TAGS list and its signature is not checked. Rewriting a published tag is worse than the gap it would close, so that exemption stays and every tag from the next one forward is verified
Observability	Applies (Tier C). A command-line harness writing reports to a directory. No service, no telemetry, no SLO surface. (The published evidence page counts visits with Google Analytics 4, which observes readers of the page, not the harness or anything it scores.) The observable surface is the exit-code contract and the report, both tested. docs/operations-runbook.md closes the gap this row named against itself: what actually breaks (keyed to each exit code, plus the CI failure modes this repository has hit), how you'd notice, and what to do — split for the two different people who hit it, someone gating a repository with Plumbline and someone maintaining this one
Performance	Applies (not met). The harness is offline and the published page is a single static file (beside its privacy page) whose one script, the Google Analytics 4 loader, appends gtag.js asynchronously and blocks nothing, which is a good starting position and is not a measurement. No budget recorded and no run committed
Accessibility	Applies: site/index.html is published to GitHub Pages, so this is in scope rather than exempt. tools/build_site.py --check proves the page is what the committed evidence produces; tools/check_site_a11y.py then holds the page itself to the same kind of structural check the accessibility scoring suite runs against a target's interface — language declaration, heading order, link text, image alt text, a single main landmark, zoom not disabled, WCAG AA contrast computed (not asserted) for both the light and dark palette declared in the page's own stylesheet, and palette_coverage, which checks that the contrast check was asked about every color the page declares. That last one exists because CONTRAST_PAIRS is hand-written: without it a color added to the palette later is simply absent from the check, and the page reports a clean pass over a bar the new color was never held to. Every declared color is now either in a checked pair or exempt with a written reason. Both checks run in make verify, in .github/workflows/pages.yml before every deploy, and are pinned by tests proving each can fail (tests/test_site.py, tests/test_site_a11y.py). Not met: none of this is tested with assistive technology or with disabled users, the same limitation the scoring suite names against itself
Internationalization	N/A (operator-facing audit tooling; the reports and the command-line output are English by design, and the reader is the engineer running the gate). The multilingual scoring suite is about the language a target answers in, and takes recognizer definitions from the target's own config so the list is never limited to what this repository ships
AI Evaluation	Applies: this is an evaluation harness, and the harness itself is evaluated. Every suite has been observed failing on a defect it exists to catch, recorded in proof/matrix.md and regenerated by tools/defect_matrix.py. The default judge is deterministic and lexical; the optional model judge is off by default, is refused outright by gate, taints the run id so differently-judged runs cannot compare as equal, and now has a model card (docs/model-card-judge.md). Not met: the card names its own biggest gap — nothing measures how often the model judge agrees with a human rater
Documentation	Applies: this README, DESIGN.md with the architecture and an acceptance record, CHANGELOG.md, CONTRIBUTING.md, SECURITY.md, CITATION.cff, CODEOWNERS, docs/first-real-target.md, docs/recordings-data-card.md, docs/feature-expansion-ideas.md, docs/model-card-judge.md, docs/responsible-tech.md, docs/what-it-caught-in-its-own-harness.md, docs/operations-runbook.md, docs/definition-of-done.md, docs/metrics-ledger.md, and an ADR log at docs/adr/ — four entries, 0000 through 0003. Quick start now follows the opening pitch directly, ahead of every narrative section
Quality & Metrics	Applies: fail-closed throughout, and the acceptance record in DESIGN.md reports results from an actual clean checkout rather than a badge. docs/definition-of-done.md closes the first half of the gap this row named against itself — a mechanical, checkable list of what "finished" means for a change here, not a vibe. docs/metrics-ledger.md closes the second half: this repository's own code-quality metrics (suite count, test count, coverage, ruff, mypy, complexity) as a history rather than a single point-in-time table row, with every row a real measurement against the tagged or merged commit named, not a recalled number. Not met: nothing enforces that either document actually gets kept up to date; both name that against themselves
AI Development Measurement	Applies (not met). This harness was built from a functional specification with AI agents, disclosed under Status above. No baseline and no outcome metrics are recorded for that development stream
Incident Response	Applies: SECURITY.md is the private reporting channel and carries the acknowledgment and fix SLA, and names integrity refusal as a security property rather than only a correctness one. Not met: no severity-label convention and no committed-postmortem requirement
Data Governance	Applies (L1, public non-sensitive): everything under datasets/ is synthetic, generated by a committed script, and about a fictional county. Bundles are content-hashed and a tampered one refuses to score. plumbline retire now screens a recording for personal data and redacts or refuses past a configured retention window, reusing privacy.py's own PII patterns; docs/recordings-data-card.md is the data card. Not met: no recordings exist yet to apply it to (no real target has been recorded against, per Status above), retention screening is pattern matching and states its own limit ("finds identifiers, not judgment calls"), and no jurisdictional retention requirement is mapped to a default --max-age-days