diff --git a/docs/Secure_AI_Development_Standards.md b/docs/Secure_AI_Development_Standards.md index c753e1c4..ba9c5717 100644 --- a/docs/Secure_AI_Development_Standards.md +++ b/docs/Secure_AI_Development_Standards.md @@ -351,20 +351,39 @@ The maintainer reviews **every** diff against the §6.2 plan. **Reject code you Work on a feature branch and open a PR (**direct `main` pushes are blocked**). One coherent layer per commit. -> ### ⛔ The `Co-Authored-By` trailer is NOT IN USE — and cannot be, as things stand +> ### NOT IN USE: the `Co-Authored-By` trailer is neither prescribed nor enforced > -> This section prescribed the trailer block below. **Measured 2026-07-29: the observed rate is zero** — -> `git log -n 300` contains **0** `Co-Authored-By` trailers and **0** `Tier:` lines — and **81 tracked -> files** under `docs/` instruct omitting it. That is not slippage; it is **structurally blocked**: -> [`cla.yml`](../.github/workflows/cla.yml) allowlists exactly three identities -> (`wshallwshall`, `dependabot[bot]`, `github-actions[bot]`), the CLA bot treats a trailer co-author as -> a contributor who must sign, and `cla` is a **required** status check. Adding the trailer therefore -> reds a required context and blocks the merge. +> This section prescribed the trailer block below. **Measured 2026-07-29: the observed rate was zero** — +> `git log -n 300` contained **0** `Co-Authored-By` trailers and **0** `Tier:` lines — and **81 tracked +> files** under `docs/` instruct omitting it. > -> So the trailer is recorded here as **designed, blocked, and not in use** (§9) — *not* as Built, and -> **not** as evidence for any claim in the A.4 register. A control with a measured adoption of zero -> cited as audit evidence is the same integrity failure the doc-drift test family exists to prevent, -> and this standard is published to adopters and auditors. +> **Re-measured 2026-08-13, and the count half has moved.** On `origin/main`, **15 commits carry at +> least one `Co-Authored-By` trailer** (77 trailer lines, because a squash merge concatenates the +> bodies it absorbs), every one of them dated **2026-07-30 to 2026-08-09** — the first landing the day +> *after* the original measurement. `Tier:` lines remain at **0**. So the 2026-07-29 count was correct +> when taken; nothing here was mis-measured. +> +> **What the re-measurement refutes is the prediction, and it is the load-bearing half.** This box +> claimed the trailer was `structurally blocked` (quoted as a token, per CLAUDE.md section 11's rule +> for naming a withdrawn term without adopting it) — that +> [`cla.yml`](../.github/workflows/cla.yml)'s three-identity allowlist plus the required `cla` context +> meant *"adding the trailer reds a required context and blocks the merge"*. Fifteen trailer-bearing +> commits reached `origin/main` through ordinary numbered PRs, starting one day later. Whatever the +> CLA action reads to decide who must sign, it is not this trailer, and the merge gate did not fire. +> **Do not restate "blocked" anywhere.** A compensating control resting on a false premise is the +> SDS-3.7 defect this document defines, and the premise was this document's own. +> +> **The honest state is a third one: unprescribed, unenforced, and intermittently present anyway.** +> The trailer arrived without a decision and stopped without a withdrawal, across an eleven-day window +> nobody recorded — the signature of a tooling default, not of a convention. That is *worse* evidence +> than a clean zero, not better: a provenance record that switches on and off for unknown reasons +> cannot answer "which model wrote this" for the commits that lack it, and cannot be trusted for the +> ones that have it. +> +> So the trailer stays recorded as **not prescribed, not enforced, and not in use** (§9) — *not* as +> Built, and **not** as evidence for any claim in the A.4 register. The conclusion is unchanged from +> 2026-07-29; only the reason is. This standard is published to adopters and auditors, and the +> integrity failure the doc-drift test family exists to prevent is citing a control nobody operates. > > **If it is ever adopted**, note the shipped Linux-kernel form is **`Assisted-by:`** — the kernel > explicitly **rejected** `Co-developed-by:` for AI attribution because it requires a paired @@ -607,7 +626,7 @@ The standard requires deviations be documented with a compensating control (SDS - **Single-maintainer review (PO.2 / PW.7) — the canonical deviation this doc owns.** *First accepted 2026-06-16 in SDS §A.6; expanded here 2026-06-18; owner: project maintainer.* The T3 "human second reviewer" control cannot mean an independent human today. **Compensating controls:** blocking SAST/SCA/secret-scan (bandit/semgrep/pip-audit/gitleaks), AI-assisted review (`/code-review`, `/security-review`), branch protection + required CI checks, no direct `main` pushes. **Build trigger:** *a second human maintainer joins ⇒ T3 review escalates to a true human second reviewer.* **Design record:** SDS §A.6 + this section (+ an ADR if cut). *Honesty:* wording forbids "AI-reviewed = independently audited" (§8). - **"Explain it with AI help" accepted in lieu of unaided comprehension — the anti-SOUP override.** *Risk accepted 2026-06-18; owner: project maintainer.* The floor's *reject code you cannot explain* (§4.4) is satisfied by an explanation reached **with AI assistance** — a deliberate relaxation of the stricter "the human understands it unaided" reading that most directly prevents **SOUP** (Software of Unknown Provenance, §10). **Why override:** the standard governs a solo maintainer who builds with AI; an unaided-comprehension bar would bar that workflow. **Compensating controls:** the maintainer must produce, verify, and **stand behind** the explanation (code opaque *even with AI help* is still rejected; rubber-stamping forbidden); the deterministic gates (tests + blocking SAST/SCA) still apply; at T3 the explanation is captured durably (comment/PR/ADR/test). **Build trigger:** a regulated-device / formal-IEC-62304 deployment requiring documented SOUP assessment or unaided comprehension ⇒ reinstate the strict bar. **Design record:** §4.4, §6.6, §10. -- **NO per-commit AI-authorship record at all.** *Risk accepted 2026-06-18 as "commit granularity by convention"; **corrected 2026-07-29** — owner: project maintainer.* The earlier wording claimed a commit-level AI-vs-human distinction via `Co-Authored-By`. **Measurement says otherwise:** 0 trailers and 0 `Tier:` lines across 300 commits, 81 tracked files instructing omission, and the required `cla` check structurally blocking the trailer (§6.7). So the honest deviation is broader than first accepted — there is **no** model/version record in the commit stream, at any granularity, and no retained transcript either, which means the project cannot answer "which model wrote this, under what policy" for any commit in its history. **Compensating controls:** the PR thread + the PR template's AI/tier declaration + this standard (a per-PR record, not per-commit). **Build trigger:** a CLA-compatible trailer (e.g. the kernel's `Assisted-by:`) enforced by a `commit-msg` hook **plus** an ungated CI backstop; separately, a hunk-attribution mechanism (§9). **Design record:** §6.7, §9. **Drift guard:** `tests/test_ai_provenance_claims.py` — it fails if the trailer is re-listed as Built or as A.4 evidence while the repo still instructs omitting it. +- **NO per-commit AI-authorship record at all.** *Risk accepted 2026-06-18 as "commit granularity by convention"; **corrected 2026-07-29** — owner: project maintainer.* The earlier wording claimed a commit-level AI-vs-human distinction via `Co-Authored-By`. **Measurement says otherwise:** 0 `Tier:` lines, 81 tracked files instructing omission, and a `Co-Authored-By` trailer that is neither prescribed nor enforced — present on 15 commits dated 2026-07-30 to 2026-08-09 and absent everywhere else, for reasons nobody recorded (§6.7, re-measured 2026-08-13). *The earlier "structurally blocked by the required `cla` check" reading is withdrawn: those 15 commits merged.* So the honest deviation is broader than first accepted — there is **no** model/version record in the commit stream, at any granularity, and no retained transcript either, which means the project cannot answer "which model wrote this, under what policy" for any commit in its history. **Compensating controls:** the PR thread + this standard (a per-PR record, not per-commit) — **and that is the whole list.** An earlier wording here also cited "the PR template's AI/tier declaration"; [`.github/PULL_REQUEST_TEMPLATE.md`](../.github/PULL_REQUEST_TEMPLATE.md) has no such field and never has, so the citation is withdrawn rather than repaired. Naming a control that does not exist is the SDS-3.7 failure this document defines, and it is more damaging in a deviation register than anywhere else, because a deviation register is read as the list of things that make the gap survivable. **Build trigger:** an AI/tier declaration actually added to the PR template (which would make the withdrawn control real); or a CLA-compatible trailer (e.g. the kernel's `Assisted-by:`) enforced by a `commit-msg` hook **plus** an ungated CI backstop; separately, a hunk-attribution mechanism (§9). **Design record:** §6.7, §9. **Drift guard:** `tests/test_ai_provenance_claims.py` — it fails if the trailer is re-listed as Built or as A.4 evidence while the repo still instructs omitting it. - **Automated requirement→design→test traceability not built (T3).** *Risk accepted 2026-06-18; owner: project maintainer.* **Compensating controls:** ADR ↔ test-name ↔ requirement links + the Plan artifact + the tier-declaration. **Build trigger:** the first regulated-release / adopter audit. **Design record:** §10, §9 (+ an ADR if cut). - **SPDX-header enforcement by convention.** *Risk accepted 2026-06-18; owner: project maintainer.* No dedicated test asserts header presence. **Compensating controls:** convention + AI memory + the leak/forbidden scan. **Build trigger:** a trivial pytest walking first-party `.py`. **Design record:** §9. diff --git a/tests/test_ai_provenance_claims.py b/tests/test_ai_provenance_claims.py index 770a9296..5e04fc21 100644 --- a/tests/test_ai_provenance_claims.py +++ b/tests/test_ai_provenance_claims.py @@ -7,11 +7,19 @@ §11's *retained, auditable evidence* set, and cited it as evidence for two claims in the A.4 register — a register whose stated audience is "adopters and auditors". -Measured 2026-07-29: **zero**. ``git log -n 300`` contains 0 ``Co-Authored-By`` trailers and 0 -``Tier:`` lines, while **81 tracked files** under ``docs/`` instruct omitting the trailer because the -CLA bot fails on it. And the blocker is structural rather than cultural: ``cla.yml`` allowlists exactly -three identities, the CLA bot reads a trailer co-author as a contributor who must sign, and ``cla`` is -a **required** status check — so adding the trailer reds the merge gate. +Measured 2026-07-29: **zero** ``Co-Authored-By`` trailers and zero ``Tier:`` lines across +``git log -n 300``, while **81 tracked files** under ``docs/`` instruct omitting the trailer. + +That count was right when taken. The *explanation* attached to it was not, and this module used to +encode the wrong one. The standard said the trailer was **structurally blocked** — ``cla.yml`` +allowlists three identities, ``cla`` is a required check, so adding the trailer supposedly reds the +merge gate — and the assertion below pinned that sentence, which made this test a guard REQUIRING the +document to keep asserting it. Re-measured 2026-08-13: 15 commits carrying the trailer are on +``origin/main``, dated 2026-07-30 to 2026-08-09, merged through ordinary PRs. Nothing blocked them. + +So the assertions here pin the *discipline* rather than any one causal story: the section must state +the status, must carry a DATED measurement, and must not restate the withdrawn "blocked" claim. A +drift guard that pins a specific cause becomes the thing keeping a refuted cause alive. Citing a control with a measured adoption of zero as audit evidence is the same integrity failure the doc-drift test family already guards elsewhere (``test_security_doc_drift.py``, @@ -121,8 +129,8 @@ def test_the_trailer_is_not_listed_as_a_built_guardrail() -> None: f"the {_TRAILER} trailer is listed as a BUILT guardrail:\n " + "\n ".join(offending) + "\n" - "Measured adoption is zero and the required `cla` check blocks it (§6.7). Move it to the " - "designed-but-deferred list, or actually adopt and enforce it." + "It is neither prescribed nor enforced, and its appearances are unexplained (§6.7). Move it " + "to the designed-but-deferred list, or actually adopt and enforce it." ) @@ -163,34 +171,56 @@ def test_the_claims_register_does_not_rest_a_live_claim_on_the_trailer() -> None ) -def test_section_6_7_says_the_trailer_is_not_in_use_and_why() -> None: - """A gap named without its cause gets 'fixed' by reinstating the thing that cannot work. +def test_section_6_7_states_the_status_and_dates_its_measurement() -> None: + """A gap named without a dated measurement gets 'fixed' from memory by the next session. - §6.7 is where a reader goes to learn the commit convention. If it still presents the trailer as - the convention, the next session re-adds it, watches the required `cla` context go red, and reaches - for the branch-protection settings. Both halves are asserted — the STATUS (not in use) and the - CAUSE (the required CLA check) — because the status alone reads as an oversight to correct. + §6.7 is where a reader goes to learn the commit convention. If it presents the trailer as the + convention, the next session re-adds it repo-wide. If it presents a *reason* the trailer cannot + work, the next session designs around a constraint that may no longer hold — which is exactly + what happened here, so the reason is no longer pinned. What is pinned is that the claim carries a + date, so a reader can tell a current measurement from a remembered one. """ section = _section("### 6.7 Commit / PR with provenance", until="### 6.8") low = section.lower() - assert any(marker in low for marker in ("not in use", "not currently used", "⛔")), ( - "§6.7 does not state that the Co-Authored-By trailer is NOT in use. Measured adoption is zero, " - "so presenting the trailer block as the commit convention makes this section fiction — and it " - "is the section a contributor actually follows." + assert any( + marker in low for marker in ("not in use", "not currently used", "not prescribed") + ), ( + "§6.7 does not state that the Co-Authored-By trailer is NOT in use. It is neither prescribed " + "nor enforced, so presenting the trailer block as the commit convention makes this section " + "fiction — and it is the section a contributor actually follows." ) - assert "cla" in low and re.search(r"block|reds\b|red the", low), ( - "§6.7 does not record WHY the trailer is unusable — that `cla` is a required status check and " - "the CLA bot treats a trailer co-author as a contributor who must sign. Without the cause, the " - "recorded gap invites exactly the change that wedges the merge gate for every PR." + assert re.search(r"re-?measured\s+20\d\d-\d\d-\d\d", low), ( + "§6.7 carries no DATED re-measurement. Trailer adoption here has already changed once without " + "anyone deciding it should, so an undated claim about the current rate is a claim about " + "whenever someone last looked. Re-measure, then write the date next to the number." ) + # Strip inline code before hunting the withdrawn phrase. A withdrawal has to NAME what it + # withdraws or no reader can follow it, so the section must be able to say the words without + # tripping this. Backticks are the repo's existing marker for "quoting a token, not adopting it" + # (CLAUDE.md section 11 makes exactly this distinction for the glyph rule), and + # scripts/docs/link_check.py already strips code spans for the same reason. + prose = re.sub(r"`[^`]*`", "", low) + assert not re.search(r"structurally blocked|cannot be, as things stand", prose), ( + "§6.7 has reinstated the withdrawn claim that the trailer is structurally blocked by the " + "required `cla` check. It is not: 15 trailer-bearing commits merged to origin/main between " + "2026-07-30 and 2026-08-09. Whatever the CLA action reads to decide who must sign, it is not " + "this trailer. Restating a refuted cause in a published standard is the SDS-3.7 defect." + ) + +def test_the_cla_allowlist_still_exists_and_names_the_maintainer() -> None: + """The allowlist is the CLA gate's shape. It is NOT what keeps the trailer out. -def test_the_cla_allowlist_is_still_what_makes_the_trailer_unusable() -> None: - """Pins the CAUSE in the workflow, so the doc claim and the mechanism cannot drift apart. + This test used to be named for the claim that the allowlist made the trailer unusable, and it + passed throughout the eleven days when trailer-bearing commits were merging — because what it + actually checks is that `cla.yml` declares an allowlist naming the maintainer, which was true the + whole time and says nothing about trailers. Keeping the old name meant a green test appeared to + corroborate a refuted claim. - If the allowlist mechanism changes, the standard's explanation needs revisiting — and this is the - test that says so, rather than the explanation quietly becoming fiction. + The assertions are worth keeping on their own terms: if the allowlist disappears or stops naming + the maintainer, fork PRs and the CLA gate behave differently and §6.7's adoption discussion needs + re-checking. That is the real invariant, and it is all this proves. """ yaml = pytest.importorskip("yaml") wf = yaml.safe_load(_CLA.read_text(encoding="utf-8")) @@ -202,11 +232,11 @@ def test_the_cla_allowlist_is_still_what_makes_the_trailer_unusable() -> None: if (step.get("with") or {}).get("allowlist") ] assert allowlists, ( - "cla.yml no longer declares an `allowlist`. That was the mechanism making a trailer co-author " - "read as an unsigned contributor — re-check whether the trailer is now usable and update " - "docs/Secure_AI_Development_Standards.md §6.7 either way." + "cla.yml no longer declares an `allowlist`. That list is what currently makes the CLA gate a " + "formality for the maintainer and a real check for a fork contributor — re-read " + "docs/Secure_AI_Development_Standards.md §6.7 and A.6 for anything that depended on it." ) assert "wshallwshall" in allowlists[0], ( - f"the CLA allowlist no longer names the maintainer: {allowlists[0]!r}. The trailer-blocking " - "explanation in §6.7 is derived from this list; re-verify it." + f"the CLA allowlist no longer names the maintainer: {allowlists[0]!r}. Every PR would now go " + "through signature collection; re-check §6.7 and A.6 before assuming either still holds." )