docs: bring the documentation set back in line with the code - #69
Merged
Conversation
A documentation accuracy pass over the public set: 191 corrections across 81 files, each verified against the code rather than against the prior prose. Content corrections - The retired PySide6 desktop console is swept out of FEATURE-MAP, DEPLOYMENT, SYSTEM-REQUIREMENTS, SERVICE, docker/, and ide/. The web console at /ui is the sole operator surface; PySide6 is harness-only. FEATURE-MAP's cluster/leader row moved to shipped rather than being deleted with the desktop console -- the capability lives in the web console (monitoring.py renders it from GET /cluster/status). - CONNECTIONS.md documented 12 of 17 registered connectors. Settings sections added for the missing ones from the registry and config models, and the SFTP/SMTP self-contradiction resolved against transports/remotefile.py -- those rows were marked "planned" while REMOTEFILE registers both and pyproject ships an [sftp] extra. - CONFIGURATION.md's "(proposed)" framing dropped where settings.py implements the mechanism, and the implemented vs accepted-but-ignored key lists corrected against the model. That table is load-bearing: an adopter setting an ignored key currently gets silence. - DEPLOYMENT.md's TLS/exposure guidance aligned to ADR 0148/0153. - SYSTEM-REQUIREMENTS.md no longer claims no published baseline exists; TUNING-BASELINE.md is published. Three prose capacity ceilings that asserted capability as fact were corrected; the labelled projection table was left alone and its caveat hardened. - MENTAL-MODEL.md off its v0.1.0 banner; USER-GUIDE.md walkthroughs checked against the real CLI surface. - The four throughput plans ADR 0107 closed now carry superseded banners. The bodies stay -- they are the record of why. Cross-references - ADR slug links repaired. Where a renamed ADR meant the citing sentence was quoting the superseded position, the sentence is flagged for review rather than silently repointed -- a working link to a reversed conclusion is worse than a dead one. - Links into the deleted messagefoundry/console/ tree and into the withheld docs/security/ tree converted to plain-text references, so they stop reading as rot. No public path was invented and no policy document was created; publishing docs/Secure_Development_Standards.md remains an owner decision, so its 44 citations are likewise plain text. adr/TEMPLATE.md fixed first, since it propagated the dead citation into every new ADR. - All 26 internal links in ASVS-L2-PHASE0-CHANGES.md repaired. What did NOT change, and why it matters 60 leads from the inventory were rejected after checking the code. Among them: docker/ never had a [console] extra or console entrypoint; the Dockerfile's PySide6-in-runtime warning is accurate (requirements.lock pins pyside6); FHIR-IN really is planned (only a destination is registered); and SYSTEM-REQUIREMENTS.md never carried the 45M/day figure, so it was deliberately not introduced -- ADR 0052 is a target, never quotable as capability. Verification: leak gate exit 0 with the real roster (names=7, estate=13, site_prefixes=1); 967 passed / 0 failed across the doc, ledger, ADR-index, link and connector guards, with test_no_links_into_private_paths and test_the_real_backlog_satisfies_the_invariant confirmed executed, not skipped. BACKLOG.md changes are link-to-text only -- no numbered heading added or removed, banner glyphs untouched. 43 questions that need an owner decision are recorded separately rather than guessed at.
Owner decision. The document licenses itself as publishable and adopter-facing,
messagefoundry.org already served it as both PDF and Markdown, and citations
across the repo -- including adr/TEMPLATE.md, which propagated the dead
reference into every new ADR -- pointed at a file that did not exist here.
Source and de-branding
Three copies existed. The newest and richest lived in the private vault as a
customer-branded fork, and its own revision history records that it was
"re-versioned and re-branded from the prior generic Secure Development
Standards (which had reached v2.0)". So the generic lineage is the original and
de-branding restores it rather than inventing it. That copy is the base: it
carries ten sections the un-branded copies lack -- SSDF PW practices,
spec-driven development, NIST SP 800-115 testing tiers, ASVS 5.0 L3 scope,
NIST SP 800-66 Rev. 2 safeguards, and References.
Four transformations, each auditable in scratch publish_sds.py: de-brand the
title and Document row; carry the public version forward to 2.1 from the v2.0
the history names; rewrite the two revision rows to describe the content
changes without naming a customer fork; de-link references into docs/security/,
which stays withheld.
Checked before publishing, not after
- Leak gate exit 0 with the real roster. Note it also passed on the BRANDED
copy -- the initialism is not in the token roster -- so the de-brand was
verified by direct inspection, not by the gate. Zero occurrences remain.
- Every other customer-shaped term run down by hand: the one Corepoint mention
is a competitor list ("Corepoint, Mirth Connect, Rhapsody, Cloverleaf"),
which is public positioning; no hostnames, IPs, or internal identifiers.
- docs/ is absent from the sdist allowlist (only-include = messagefoundry,
README, CHANGELOG, LICENSE, NOTICE), so this does NOT undo the #1020
packaging remediation. test_release_pipeline passes.
Citations
8 restored across .github/SECURITY.md, three docs/ standards companions, three
ADRs and adr/TEMPLATE.md -- all verified to resolve. The inventory's "44 links"
was an overcount; the real figure is 23 mentions, most by title rather than
filename, and linking every occurrence would be noise, so the convention is the
first mention per file plus any section-qualified citation.
653 passed / 0 failed across the doc, link, ADR, backlog and release-pipeline
guards.
97 changes across 11 file groups, each verified against code rather than against the prior prose. 60 further leads were rejected after checking; they are the reason to trust the 97. Corrections that changed what the docs claim - SYSTEM-REQUIREMENTS said "No native transport TLS" and "no MLLP-over-TLS". Both are false: settings.py:694-712 is an in-process API/WebSocket TLS block (WP-13a, ADR 0002) and mllp.py:505-524 builds per-connection MLLP SSL contexts with opt-in mTLS (WP-13b). The doc was understating shipped capability and telling adopters to buy a reverse proxy they may not need. - CONNECTIONS' ASVS 5.1.1 block claimed the file surface accepts "HL7 v2 text only". The code sniffs against the DECLARED content_type (file.py:514) and a non-HL7 drop bypasses HL7 handling entirely (file.py:644,673, ADR 0004). It now describes declared-type conformance. An overstated control is the kind an assessor finds and then distrusts the rest of the document over. - The eight sizing tiers had no measurement behind them. Re-derived from the published baseline with the arithmetic shown, and daily figures now state their duty-cycle assumption instead of implying 86400s of peak. - ~40 implemented settings keys had no row in CONFIGURATION. Added from ServiceSettings.model_fields. - NOT_DEPLOYED, a seventh disposition genuinely set by the finalizer (store.py:319, 912, 926, 955), was documented nowhere. Added to MENTAL-MODEL, USER-GUIDE and ARCHITECTURE. - MENTAL-MODEL never mentioned engine sharding, though it is the built default scaling axis, while asserting active-active scale-out was dropped -- together reading as "this does not scale". Both corrected, and the two are now distinguished rather than conflated. - SERVICE now documents that a custom -Environment name also requires [ai].data_class and [ai].production (install-service.ps1:66-71). Following the old text produced a service that would not start. Classification and structure - New docs/SECURITY-DOCS-POLICY.md states what is kept private and why, using the operative rule: private only if it discloses an OPEN, un-remediated weakness in exploitable detail, or names a customer. A closed audit is transparency. The ~145 plain-text references into the withheld tree now have something to point at instead of naming documents a reader cannot find. - Four docs self-marked "internal" while sitting in a public repo. Re-sorted under that same rule; the labels now match reality. - The five throughput plans ADR 0107 closed moved to docs/archive/throughput/ with a README. Bodies intact -- they are the record of why. ADRs, treated as historical records - ADR 0101 gains the estimand reporting rule it was already being cited for. ADR 0074:146 cited it; ADR 0101 did not contain it. The rule is real: the intake-vs-delivery conflation caused the Arm 0 "sustains 26" retraction. Added with a dated amendment note, since 0101 is Accepted. - ADR 0090's PySide6 residual cannot be built. Annotated with a dated note rather than repointed -- repointing would rewrite what was decided. Verification Leak gate exit 0 with the real roster (names=7, estate=13, site_prefixes=1). 972 passed / 0 failed across the doc, link, ledger, ADR, feature-map, connector and release-pipeline guards, with test_feature_map_claims and test_backlog_status_check confirmed EXECUTED (23 PASSED), not skipped. The archive move was the risk here, because moving a file breaks inbound links without any test going red. All 4,593 relative links across 414 markdown files were resolved: zero real breakages. (Two `command:` VS Code URIs in an IDE ADR are legitimate, not file paths.) 57 security-posture passages are held back for owner review before the PR.
Ten commits landed on main during this documentation pass. Two files conflicted; both resolutions take the other session's content and keep this branch's link fix. docs/ASVS-L2-PHASE0-CHANGES.md — the sibling ASVS-TLS session corrected a real error: `SSLContext.set_groups` is a Python 3.15 API, not 3.13, so the key-exchange pin currently pins NOTHING and those contexts inherit OpenSSL's default group list. That correction wins over this branch's older "pins the approved groups" text, which was wrong. Kept this branch's relative-path fix (../messagefoundry, not ../../, which escapes the repo root). docs/Secure_AI_Development_Standards.md — kept their added honesty note that CODEOWNERS is documentation rather than enforcement here, and kept this branch's de-linking of RELEASE-GATE.md, since docs/security/ stays withheld from the public repo. Verified after integrating: leak gate exit 0 with the real roster; 976 passed / 0 failed; 4,605 relative links across 414 files resolve with zero breakages.
CI went red on a documentation-only branch. The cause was a precision bug in
the guard, not rot in the docs.
`_PROSE` alternated on the bare string `private repo`, so it matched INSIDE
longer words. Two lines counted as present-tense mirror prose while being
nothing of the kind:
.github/workflows/release.yml:271 a quoted GitHub error about
"user-owned private REPOsitories"
docs/Secure_Development_Standards.md:284
"a private REPOrting channel"
The second arrived with the newly published standards doc and took the count to
56 against a ceiling of 55.
The fix is a word boundary on that one alternative. Measured before changing it,
because the temptation here is to widen a guard until your own change passes:
the boundary suppresses exactly those two lines and keeps every genuine
"private repo" hit -- about twenty, across the workflows, INSTALL-GUIDE,
VERSION-CONTROL, SECURITY and the ADRs. No true positive is lost.
The ceiling therefore moves 55 -> 54. That is the ratchet working as documented
("lower it when you fix some; NEVER raise it") -- the honest count fell because
two false positives were removed, so the ceiling follows it down. It also keeps
test_the_ratchet_is_not_slack satisfied, which fails if the gap exceeds 8.
Proven, not assumed: appending "This repository is the public mirror of the
private repo." to a tracked doc makes the guard go RED at 55 vs 54, and removing
it makes it green again. A guard changed without demonstrating it can still fail
is a guard nobody should trust.
Also worth recording: the module's failure message labels its tail `hits[-12:]`
as "New or changed", which it is not -- it lists the last twelve hits in scan
order. That is what made the failure look like it came from test files nobody
had touched. The delta had to be computed against origin/main by hand.
Six documentation findings today were all the same shape, and none was a false statement: true about the mechanism, misleading about the posture. A sentence that survives every spot-check while pointing the reader somewhere they cannot go. A correctness review terminates at "yes, that is accurate" and finds none of them. Secure_Development_Standards.md §3 gains "Reviewing security prose: ask what a reader would DO with it" as the governing instruction, with three rules under it and the evidence for each: - State a load-bearing fact ONCE and link to it. A repo that states a fact twice will eventually state it two ways, and the stale copy is the one that gets cited, because a reader who finds A statement stops looking for the other. Three instances in one day: harden_kex_groups described as pinning key-exchange groups across FIVE live documents when SSLContext.set_groups is a Python 3.15 API that pins nothing; a superseded Transit/unkeyed-audit-chain claim left in PHI.md after the code closed it, from where it reached a public page AND a security review; and PHI.md §5 asserting "no PHI is placed in a URL" while §7 documented the query parameters that carry it. Each time the repo held both the right and the wrong version. - A completeness claim is a liability. "Two configurations do X, and a reviewer should hear both" invites the check and survives it. Twice this set shipped such a sentence wrong in BOTH directions -- naming a case that no longer existed while omitting one that did. - A compensating control must not rest on a false premise. A Referrer-Policy relaxation was justified by "console URLs carry opaque ids only (never PHI)", untrue of the console's own search route. The control was sound; the reason was not, and the next person reasons from the comment. CLAUDE.md §11 carries these as four bare one-line imperatives pointing at that section. The duplication is deliberate and is the rule applied to itself: the FACT lives once, because evidence, dates and line numbers drift; the INSTRUCTION is repeated because it is one line, cannot meaningfully drift, and a pointer nobody follows mid-task changes no behaviour. If the two ever disagree it will be about the wording of an instruction, not about what is true. CONFIGURATION.md's audit_days row is aligned with docs/PHI.md §8, which is now named in the row as the source of record. Both say keep-forever rests on the retention requirement -- 45 CFR 164.316(b)(2)(i), against framework floors far below it -- and NOT on chain-breakage. That argument was inverted: deleting the OLDEST rows does break the walk, but an attacker hides by truncating the NEWEST, and the surviving prefix still verifies (store/store.py:7385). The anchor that would catch it is not reachable from the shipped CLI, so audit-verify returns clean after a truncation; that gap is filed separately. Provenance is recorded in the standards section because the provenance is the evidence: the state-it-once rule came from the parallel ASVS session, which also supplied two of the three instances. Verified: leak gate exit 0 with the real roster; 72 passed across the ratchet, feature-map, backlog and security-doc-drift guards; 4,606 relative links across 414 files resolve. CI was GREEN on all 35 checks for the preceding commit.
Reported by the messagefoundry-website session, which hit it for real: a peer's
PR merged mid-session and the gate kept blocking 31 files, a third of that site.
The repo squash-merges everything ("title (#NN)"). A squashed commit never
becomes an ancestor of the branch it came from, so the merge-base never advances
and `diff origin/main...HEAD` keeps reporting the branch's files after its work
has landed. Its session then blocks that whole file set until someone prunes the
worktree.
Neither diff form is right alone, and each fails in the opposite direction:
three-dot credits a landed branch forever; two-dot blames a merely-behind branch
for every file main moved underneath it. The intersection -- authored AND not yet
landed -- is correct, and self-clears on squash, rebase and merge-commit alike.
A two-dot failure falls back to the authored set, so a git hiccup over-blocks
rather than silently under-blocking.
Measured here before and after, on real worktrees:
landed, claiming phantoms asvs-15-2-4-name-claimed 3dot=8 intersect=0
gate-version-stamp 3dot=4 intersect=0
pins 3dot=16 intersect=0
genuine outstanding work documentation-inventory 101 -> 101
master-test-plan 21 -> 21
asvs-14-2-7-retention 11 -> 11
ci-suite-wiring 4 -> 4
prunefix 7 -> 7
`pins` was live: it held zero uncommitted files and zero outstanding committed
files, and was still blocking 16. After the fix it drops off the map entirely
while every branch with real work is unchanged. Verified with -Refresh, since the
cache is TTL-bounded and served the old map on the first run.
Landing here rather than in its own PR because this branch's merge is exactly the
event that triggers the bug: 101 files would have stayed blocked for every other
session until the worktree was pruned.
wshallwshall
enabled auto-merge (squash)
July 30, 2026 16:48
Seven commits landed on main. One file conflicted, .github/SECURITY.md, and both sides had independently fixed the SAME defect: two markdown links into docs/security/, a directory absent from the public repo. main (#78) removed the references outright. This branch converted them to plain text and pointed at docs/SECURITY-DOCS-POLICY.md, a page that exists only on this branch and that the owner ratified precisely so those ~145 withheld-path mentions have somewhere to point instead of naming documents a reader cannot find. Resolution takes both: main's tighter sentence structure (em-dash rather than a parenthetical), and this branch's named document plus the policy pointer. A reader now learns the analysis exists, that it is maintainer-internal, and what they can request -- which is the whole point of the policy page. main's other change to the file, a working security@ address replacing the dead "email the maintainer at the address on the GitHub profile" fallback (#77), came through untouched and is strictly better than what either side had. Note the five docs/ files a review workflow is currently editing were NOT touched by main and remain uncommitted and intact; this merge deliberately stages only the conflicted file. Verified: 0 conflict markers; the new security@ address present; the policy pointer present twice; 4,639 relative links across 414 files resolve.
wshallwshall
added a commit
that referenced
this pull request
Jul 30, 2026
…an path origin/main moved docs/throughput-build-plan.md under docs/archive/throughput/ (#69). The rename updated docs/AOAG-DEPLOYMENT.md but not these three ci.yml comments, and nothing catches it: no guard scans workflow COMMENTS for doc paths, so the drift is invisible. Comment-only, no behaviour change. Fixed here rather than left because one of the three sits directly above the file list this branch appends to -- shipping a stale path in a block I just rewrote is my drift, not inherited drift.
This was referenced Jul 30, 2026
wshallwshall
added a commit
that referenced
this pull request
Aug 3, 2026
…letion, and the legacy outbox PHI leak (#71) * test(14.2.7): land the no-DB guards FIRST, so the retirement cannot ship half-done Commit 1 of ASVS 14.2.7. No production code. Three guards that run on the plain leg with NO database, deliberately landed BEFORE the changes they police. TWO independent nine-agent workflows analysed this change (two Claude processes under one session id, neither aware of the other). They produced DIFFERENT plans -- 12 traps and 13 traps, 20 distinct. Both are preserved with their union in vault PR #1258. Two of the three guards here exist because of traps only ONE pass found, which is the argument for having run both. --- 1. tests/test_sqlserver_encrypt_pass_tables.py (AST, no DB) --------------- Every (table, column) literal driving a ciphered-cell sweep must name a table the module actually CREATEs. This is the ONLY mechanism in the repo that can catch the defect. Commit 6 drops the legacy `outbox` table; the pair ("outbox","payload") sits ~1200 lines away in _encrypt_existing_rows. That method early-returns on `if not self._cipher.encrypts` -- and EVERY SQL Server CI step runs KEYLESS, so the SELECT below that return never executes on any leg. Plan A's own stated mutation proof for this trap ("the keyed open must red") reds on no CI leg that exists. A keyed production open would crash-loop on Invalid object name; CI would stay green. The naive version over-matched on its first run and that was useful: both sweeps also build composite AAD tuples like ("body","detail") and ("event_type", "connection") whose first element is a COLUMN. Flagging those would have cried wolf on ~14 legitimate pairs, and a guard that always fails gets xfailed or deleted. So it anchors on the LOOP (`for table, <col> in (...)`), not on "any 2-string tuple in the function". Four mutations against the REAL source, not the fixture: M1 unknown table in _encrypt_existing_rows -> RED M2 unknown table in reencrypt_to_active -> RED (both sweeps walked) M3 ("outbox","payload") REMOVED -> GREEN (must not block Commit 6) M4 CREATE TABLE regex blinded -> RED (liveness receipt fires rather than passing vacuously) M3 and M4 matter as much as M1: a guard that blocks its own fix, or that passes when blinded, is worse than none. --- 2. tests/test_fixture_outbox_reset.py (AST, no DB) ------------------------ No fixture may issue an UNGUARDED `DELETE FROM outbox` once the table is retired. The existence-guarded form f"IF OBJECT_ID(N'{table}', N'U') IS NOT NULL DELETE FROM {table}" is a no-op afterwards and is accepted; the bare form is not. That distinction is structural, so the scanner reads the loop BODY rather than grepping for the word. MEASURED, independently reproducing plan B's trap 8: 538 test files scanned, 21 unguarded reset sites across 18 files, and NINE of those 18 run in no SQL Server CI step at all -- test_adr0071_dispatch_wiring_sqlserver, ..._fused_callables_..., test_adr0075_batch_sqlserver, test_adr0114_claim_proc_live, test_batch_completion, test_metadata_bag, test_outbound_batch, test_shard_recovery_sqlserver, test_sqlserver_sync_handoff. A mutation planted in any of those passes VACUOUSLY, so "the SS leg is green" was never evidence the reset lists were updated. Marked xfail(strict=True), not left red: the 21 sites are CORRECT today (the table exists). strict makes it SELF-CLEARING -- the moment the last reset list is fixed the test XPASSes, which strict turns into a failure, forcing the marker out in the same commit. A non-strict xfail would rot silently green forever. --- 3. _RETIRED_CLAIMS registrations (tests/test_phi_at_rest_inventory.py) ---- Three strings that are TRUE TODAY and become false in Commits 3 and 6, registered FIRST so the guard reds against the UNEDITED docs. Confirmed RED at registration: docs/PHI.md:123 '**No purge path at all**' docs/PHI.md:401 'is purged by nothing on any backend' docs/PHI.md:1050 'touched by no purge on any backend' That output is the evidence, and it is only obtainable BEFORE the edits. A registration made afterwards passes on day one whether or not the guard works. A FOURTH candidate was STRUCK, and this is the trap only plan B found. The obvious string "no retention purge" collides with a still-TRUE sentence at docs/security/ASVS-292-289-HANDOFF-2026-07-25.md:173 about pending_approvals.params. _retired_claim_hits() globs docs/*.md PLUS docs/security/*.md -- and docs/security/ is GIT-IGNORED. Registering it would red on a laptop and pass in CI, where the file does not exist. A guard strictly weaker in CI than locally is not a guard. Plan A instructed adding it, with no warning; I was minutes from doing so. Grep evidence recorded in-file (docs/*.md | docs/security/*.md): "no retention purge" 1 | 1 <- STRUCK "**No purge path at all**" 1 | 0 "touched by no purge on any backend" 1 | 0 "is purged by nothing on any backend" 1 | 0 DELIBERATELY RED IN THIS COMMIT: test_retired_false_claims_do_not_reappear fails until Commit 3 edits those three PHI.md lines. That is the forcing function, and CI gates the PR head, not each commit. * feat(retention): purge orphaned reference snapshots (ASVS 14.2.7 part 1) `reference.value` (ADR 0006 versioned lookup snapshots) is PL-2 in PHI.md §2 and can hold patient-keyed rows. It had NO purge path at all: the only thing that ever replaced a snapshot was the next sync's build-new-then-flip, which never comes for a set nobody declares any more. `grep purge_reference` returned zero production hits. Adds purge_reference_snapshots to the Store protocol and all three backends, the [retention].reference_snapshot_days window, and the runner phase. ORPHAN-SCOPED, and the limit is stated rather than glossed: a set that IS still declared is never touched however old its synced_at, because its snapshot is live data the engine serves. So the normal case -- a wired set holding live PHI -- is still purged by nothing. That is an honest residual, not a closed cell, and the classification table must not describe this window as covering reference.value generally (which would machine-bless a false claim). Three defences, each from an adversarial pass and each mutation-proven: 1. EMPTY `declared` RAISES, in all three stores AND at the runner call site. `declared` is the keep-set, so an empty one reads as "every set is abandoned" and would purge the whole store. `registry is None` does not cover it: a registry that LOADS FINE while declaring zero reference sets -- a subset --config, a per-team split, a harness redirect aimed at the real DB -- yields references == {}. Absence-based guards fail open, so this is positive-signal. Belt AND braces because one refactor dropping the runner check costs every snapshot in the store. Worse without it: ReferenceSyncRunner deliberately does NOT advance synced_at when a source fetch fails, so the rows that look most stale belong to a still-wired set whose source is merely DOWN. 2. ELIGIBILITY RE-ASSERTED INSIDE THE DELETE, on all three backends including SQLite. One analysis said SQLite "happens to be safe" via self._lock; its own adjudication corrected that and is right -- `declared` is computed in the RUNNER, outside any store lock, so the config-reload race is at the caller level everywhere. Do not simplify the EXISTS away because the SELECT already filtered. 3. THE POINTER SURVIVES BUT ITS VERSION IS BUMPED. Both analyses proposed bolting a removal arm onto converge_reference_cache in each server backend. Reading it gives a better fix: converge only reloads a set whose active version DIFFERS from the one a handle reflects. So leaving the version alone means a follower's populated cache is never revisited -- it serves purged PHI from RAM until restart -- and deleting the pointer is WORSE, because converge only ever adds/updates names present in a fresh read and would never notice. Bumping to 'purged:<version>' with row_count=0 routes the deletion through the mechanism that already exists, on both server backends, with no second removal path to keep in sync. Idempotent via `WHERE version NOT LIKE 'purged:%'`. Four mutations killed: M1 drop the `not in declared` filter -> a DECLARED set loses its rows M2 empty-declared raise -> return 0 -> the wipe path becomes a silent no-op M3 drop the version bump -> followers never converge M4 drop the EXISTS re-assert -> a concurrent re-sync is destroyed M4 SURVIVED THE FIRST ATTEMPT, and the reason is worth recording because it is the exact defect class this change set exists to prevent. The original test re-synced BEFORE calling the purge, so the internal SELECT already filtered the set out and the DELETE never ran -- it exercised the candidate filter, not the re-assert, and passed whether or not the control worked. The race only exists BETWEEN the SELECT and the DELETE, so the reload is now injected mid-method by wrapping _db.execute, with `assert fired` so a wrapper that stops matching can never let the test pass vacuously. SQLite keeps no per-set version map (single-node, sole writer, converge is a genuine no-op there) so it evicts its read-through cache instead; the server backends evict AFTER the commit, so a rolled-back purge cannot leave the cache claiming rows the store still holds. Expected reds until the docs commit: test_retired_false_claims_do_not_reappear and test_purge_surface_is_defined_on_every_backend_and_documented_per_backend. The second is NEWLY ARMED by this commit -- adding the method to the protocol makes the tree demand its PHI.md §8 row, which is the forcing function working. ruff + mypy(strict, 257 modules) clean; 219 passed across the touched suites. * docs(phi): reference snapshots gain a bounded window — and the limit is stated (14.2.7) Clears two of the four forcing functions landed in Commits 1-2. `test_purge_surface_is_defined_on_every_backend_and_documented_per_backend` was armed by Commit 2 the moment purge_reference_snapshots joined the Store protocol: the tree DEMANDED a §8 row rather than trusting anyone to remember one. That is the guards-first ordering doing its job. Added, naming the mechanism, the re-assert, and the pointer-survives-with-a-bumped-version behaviour that is what actually makes a cluster follower converge. Also retires the registered false claim at §2's reference.name/version/key row ("**No purge path at all**") and rewrites §2's reference.value row. THE ORPHAN-ONLY LIMIT IS WRITTEN INTO EVERY PLACE THE WINDOW IS NAMED, on purpose. purge_reference_snapshots bounds a set config has DROPPED; a set still declared is never purged whatever its age, because its snapshot is live data the engine serves. So the normal case -- a wired set holding live PHI -- remains bounded by nothing. An adversarial pass flagged that describing this as a plain `rides <window>` entry would machine-bless a FALSE claim and, worse, make the true statement unwritable once the retired-claim string was registered. So §8's honest-gaps table KEEPS a reference.value row rather than losing it: the row now scopes the gap to the declared-set case instead of deleting it. Two registered claims deliberately survive this commit, both about `outbox.payload` (docs/PHI.md:401 and :1051). They are Commit 6's to clear when the legacy SQL Server table is migrated and dropped. Registering all three in Commit 1 -- before any edit -- is what made the guard's RED output the work list; the count going 3 -> 2 here is that mechanism reporting progress. * docs(phi): the anchor that would catch tail-truncation is unreachable from the CLI Two more corrections from the sibling session's hostile review, both verified here before acting. The first sharpens a rewrite I made an hour ago and did not push far enough. 1. I WROTE 'only an out-of-band audit_anchor detects that' -- which implies an operator can reach it. They cannot. Verified: __main__.py:3418 return await store.verify_audit_chain() <- no anchor parser zero --expect-anchor / audit-anchor flags audit_anchor() implemented on all three backends; its ONLY callers are pipeline/dr.py:569,585 and rekey_audit_chain So the machinery exists and the shipped operator command cannot use it: an operator or compliance job running OK: verified 0 audit row(s) gets a CLEAN result after someone truncates the newest audit rows. As shipped, truncation detection depends on the off-box tee (sec-offbox-log #361/#363), not on anything runnable locally. My previous wording was true about the mechanism and misleading about the posture -- the same shape as the inverted argument it was correcting. Now states the gap plainly, with an explicit instruction not to upgrade it to 'the anchor detects it' without also shipping a way to pass one. 2. §5 AND §7 CONTRADICTED EACH OTHER, and §7 was right. §5 claimed 'no PHI is placed in a URL (so it cannot leak into history, bookmarks, or Referer)' while §7 documented, correctly, that '?content=1234567 or ?field_value=MRN12345 survives the whole filter chain unredacted'. The console's own search route takes both as query parameters. Narrowed §5 to what actually holds: no message BODY in a URL, plus Referrer-Policy: no-referrer. Also flagged in the same bullet: messagefoundry_webconsole/_security.py justifies degrading to no-referrer on the grounds that '/ui URLs carry opaque ids only (never PHI)'. The header is still the right call, but its stated REASON is false -- and a compensating control resting on a false premise is worse than an uncommented one, because the next person reasons from the comment. The route fix is filed separately by the sibling session; the comment would have outlived it. Third instance today of two live documents disagreeing while the stale/wrong half is the one a reviewer picks up (after harden_kex_groups across five documents, and the pre-#301 Transit claim). The pattern is worth naming: a repo that states a fact twice will eventually state it two ways, and the wrong copy is the one that gets cited. * docs(phi): classify all 38 at-rest tiers — and four of them are UNBOUNDED (14.2.7) Adds §2's seventh column: a retention verdict for EVERY at-rest tier, in a 七-form vocabulary the serve gate's tier list and the §2/§8 drift test are GENERATED from. A wider hand-typed tuple is the same defect with a longer literal; this is the artifact that replaces it. Produced by a nine-agent workflow (classify -> adversarially attack -> reconcile) because a wrong cell here does not fail a test, it gets MACHINE-BLESSED: a false 'rides <window>' asserts coverage that lives somewhere else, and the drift test would then defend it on every run. This project has retracted an ASVS score twice for that shape. DISTRIBUTION: 7 own-window, 13 rides, 2 orphan-only, 5 keep-forever, 6 not-PHI, 1 keep-N, and 4 UNBOUNDED. The F7 count being non-zero was a stated pass condition -- a classification of this codebase with no gaps in it would be flattering rather than complete. TWO UNBOUNDED PL-1 TIERS WERE IN NO GAP TABLE ANYWHERE, and both are now in §8: * at stage ingress/routed, when DEAD. This is a REAL DEFECT, not a documentation gap, and it is filed separately. dead_letter_now issues and never touches ; a router/handler content fault calls it on the claimed ingress/routed row (wiring_runner.py:4415, :4449); and BOTH purges are scoped Stage.OUTBOUND (store.py:8384, :8659). So blanks on its window and the message READS AS PURGED while a full raw PHI body survives here indefinitely. It was missed because these rows are documented as transient -- true on the happy path, and the dead-letter path is the exception nobody classified. * the backup staging dirs. A TemporaryDirectory unlinks on exit but not on a crash or SIGKILL, and verify_after_backup (default true) decrypts a full archive back out on EVERY run. ONE ATTACKER OVERTURNED A CLASSIFIER, and the correction matters beyond bookkeeping: at stage outbound was filed as riding the body window, but purge_dead_letters is a purge dedicated to that exact table+column on its OWN window. Filing it F2 would have dropped -- one of exactly two windows __main__.py enumerates for the auto-bound and the refusal -- out of a list generated from this column. MY OWN GUARD CAUGHT MY OWN WORDING while writing this: the new staging-dir row said 'point TMP/TMPDIR at an owner-only path', which test_owner_only_file_acl_is_always_qualified_to_the_sqlite_store correctly flagged as an unqualified ACL claim -- the engine applies no file ACL on a server-DB store, the deployed posture. Re-scoped to state the engine's no-ACL plainly and put the cover where it belongs, on the operator. WORKFLOW DEFECT WORTH RECORDING: was passed as a JSON array and arrived as a STRING, so the four intended row-batches sliced CHARACTERS. The agents recovered by reading the file themselves -- 68 classifications came back for 38 rows -- and the reconciler deduplicated. It worked by redundancy, not by design. Completeness was therefore verified independently (1..38 exactly once, no duplicates, every row 7 cells) rather than taken from the reconciler's own 'checked' claim. * feat(retention): generate the serve gate from the classification, not a literal (14.2.7) The startup retention gate checked a two-element literal. That literal is why this cell broke once: a new PHI tier landed and nobody widened it. Widening the literal is the same defect with more characters, so the tier list is now GENERATED from config/retention_classification.py, and tests/test_retention_classification_drift.py holds that constant equal -- in BOTH directions -- to docs/PHI.md 2's Retention column. Add a tier to the doc without adding it here, or here without the doc, and the suite reds. Nine windows across FIVE settings sections, not one. The first drift test checked every window against RetentionSettings and exempted [store]/[security] by prefix; adding [backup].retention_keep reded it and the tempting fix was to widen the exemption, which would have stopped verifying three of the five sections. Routing each window to its own model instead makes the check stronger than it was before it failed. MIN_PHI_RETENTION_WINDOWS is a FLOOR, not an emptiness check: `if not TUPLE` passes for a one-element tuple, so a bad merge would leave the gate checking one window while reporting success. The floor was itself corrected by its own drift test -- written as 7 by hand, the two-way equality immediately reported [backup].retention_keep and [retention].connection_event_retention_hours as documented-but-absent, before either had ever run. Behaviour change, deliberate and owner-approved (2026-07-30): the three PHI-BODY windows now DEFAULT to 30 days when UNSET, on both dials. Previously an unset window on the shipped `enforce` posture REFUSED TO START, which forced an operator to pick a number; it now starts bounded at 30. What survives is the fail-closed path for an EXPLICIT 0 -- typing 0 is a deliberate act and is still refused unless [security].allow_keeping_phi_indefinitely is set. So "unbounded by accident" is still prevented; "unbounded by inattention" becomes "30 days by inattention". test_cli.py carries both halves as a pair. state_max_age_days / search_preset_days / app_log_days / backup.retention_keep are classified and WARNED, never silently bounded -- also a ruling, not an omission. Their timestamps only move on a WRITE (state.set_at is never refreshed by state_get), so an auto-bound would delete a Handler's live MRN->surrogate crosswalk mid-stream, and the next message would transform WRONGLY, post-ACK, with no ERROR disposition. connection_event_retention_hours and uploads_retention_days are classified but never tested for unboundedness: on the first, 0 means INHERIT the body window, so a `days <= 0` predicate would refuse a start over a window that is doing its job; the second carries a ge=1 floor, so the clause is unfireable by construction. Both are recorded as zero_is_unbounded=False rather than dropped, so the generated list still matches 2. * fix(store): the legacy SQL Server `outbox` table held PHI nothing could purge (14.2.7) `outbox` was recreated by `_SCHEMA` on every open and read by NOTHING -- the staged pipeline and every delivery-side method use `queue`. So on a store upgraded from the pre-staged-pipeline layout, `outbox.payload` (a full transformed PHI body) was reached by no purge on any backend: `purge_old_messages` and `purge_dead_letters` both scope to `queue`. Meanwhile `messages.raw` blanked on its own window, so the message READ as purged while the body survived indefinitely. It also sat outside `reencrypt_to_active`, so a rotation that retired the old key left those bodies undecryptable. Both close the same way: a guarded `_SCHEMA` statement folds surviving rows into `queue` as stage='outbound' -- payload carried over VERBATIM, so encryption at rest is preserved -- and then DROPs the table. Migrated rows become ordinary outbound queue rows: bounded by [security].delete_message_bodies_after_days / [retention].dead_letter_days, swept by the on-open cipher migration, rotated by reencrypt_to_active. SQLite already did the equivalent (`_migrate_outbox_to_queue`); Postgres never had the table. The three `outbox` DDL statements are gone, so a fresh store never creates it again. THREE PLACEMENT DECISIONS, each load-bearing rather than stylistic: * It lives in `_SCHEMA`, not an open-path method like SQLite's. `_schema_hash()` is the ONLY thing deciding whether the DDL batch runs -- a current marker skips it entirely -- so migration code outside `_SCHEMA` would be invisible to that decision and would never fire on an upgrade. It also inherits the batch's `raw.timeout = 0`, so a large legacy backlog cannot be killed mid-INSERT into a rollback-and-re-fail crash-loop. * It sits AFTER the `queue` DDL, not where the `outbox` DDL was. SQL Server defers name resolution, so an earlier placement would parse fine and then fail at RUN time on a legacy DB old enough to have `outbox` but not `queue`. * Two filters guard the INSERT. EXISTS(messages) skips FK orphans -- unreplayable anyway, but one would abort the batch with an opaque FK error. NOT EXISTS(queue) makes a PK collision a no-op instead of a 2627 that rolls back and re-fails on every restart, and makes a partially-applied migration safely re-runnable. VERIFIED AGAINST A REAL SQL SERVER (2022 CU25 in Docker), not just asserted. The new live suite simulates an UPGRADE rather than a fresh open, because a fresh open never executes the migration at all (the existence guard is false) and a re-open with a current `schema_meta` marker skips the whole batch -- both would have passed while measuring nothing. Each test seeds a real legacy table and clears the marker, which is the state a genuine upgrade presents. All three filters were then mutation-proved against the live database: dropping the DROP reds 3 tests, removing EXISTS(messages) reds the orphan test with an IntegrityError, removing NOT EXISTS(queue) reds the collision test with an IntegrityError. FIFO on a migrated backlog was MEASURED, not assumed: rows INSERTed into `outbox` in the exact reverse of their created_at order came out with `seq` ascending by created_at. It is still not a documented SQL Server guarantee, so the comment says so and no test asserts it -- a flaky gate on a best-effort property is worse than none. `("outbox", "payload")` is removed from `_encrypt_existing_rows`; leaving it would have failed every KEYED open with *Invalid object name*. No CI leg would have caught that -- every SQL Server leg runs keyless and returns before the sweep -- which is why the AST guard from the previous commit exists. Mutation-proved: re-adding the entry reds it, naming the line. 27 `"outbox"` entries removed from reset lists across 24 files. Nine of those files run on NO SQL Server CI step, so "the SS leg is green" would not have been evidence; the source scanner is what covers them. Its `xfail(strict=True)` self-cleared on the last fix -- XPASS, which strict reports as a failure -- so removing the marker here was forced rather than remembered. The `test_no_engine_code_still_references_the_retired_table` sweep was rewritten mid-change after it correctly fired on BOTH migrations. A flat ban is unsatisfiable (a migration must name the table it migrates), but a plain file exemption would hide a stray read inside those same files forever. It now pins the reference COUNT per migration module and asserts each still contains the DROP, so a new `FROM outbox` in `store/sqlserver.py` reds on the count. Path keys are POSIX-normalised: native separators would have keyed on backslashes and missed on ubuntu -- a guard red on one OS and green on another. docs/PHI.md: the section 2 inventory row and the section 8 gap row are removed rather than reworded, because the at-rest location no longer exists. The per-backend cipher counts drop to 18/17/17 in both places they are stated -- and those are DERIVED from live code by test_phi_at_rest_inventory.py, which is what reported the change rather than my counting. The two registered retired claims ("touched by no purge on any backend", "is purged by nothing on any backend") now go green, which is what they were registered a commit early to prove. The bare "no purge on any backend" is still NOT registered, and the reason CHANGED without changing the answer: it used to collide with the legacy row, and now collides with this commit's own past-tense account of what the retirement fixed. A phrase whose collision merely MOVED is not a phrase that became safe to register. CI WIRING, without which the live suite above would have been decorative. CI's SQL Server steps run an explicit FILE LIST, not the whole suite, so a new MEFOR_TEST_SQLSERVER-gated file that nobody appends runs in NO environment: it skips on the plain legs, and a skip reads exactly like a pass. The new file is appended to the catch-all SQL Server step, and that step's comment now says it is the catch-all for ANY gated file rather than only throughput levers. Measuring that gap turned up a larger one, filed rather than fixed here: 17 test files are wholly skipped without a live SQL Server, and ELEVEN of them are named in no CI step at all. That is why several of the pre-existing failures below had rotted unnoticed -- nothing runs them anywhere. Unrelated pre-existing failures found and filed, NOT introduced here. A full local suite showed 29 failures; 25 were proven pre-existing by stashing this change out and re-running (the identical set failed). Of those, a large share are LOCAL-ONLY artifacts of exporting the SQL Server env vars across the WHOLE suite, which CI never does -- it sets them per step for a named file list. MEFOR_ALLOW_INSECURE_TLS=1 in particular defeats refusals that test_tls_trust_anchor and test_hop_refusal_rawtcp assert; test_audit_integrity.py goes 29/29 green on a clean env. Separately, 5 of 6 tests in test_adr0114_claim_proc_live.py fail against a live SQL Server (claim_proc_effective is False; the deployed proc body does not match the shipped definition, and the warning's own DELETE FROM schema_meta remedy does not fix it) -- also confirmed pre-existing the same way, and one of the eleven files that run nowhere. * docs(ci): three workflow comments still pointed at the pre-archive plan path origin/main moved docs/throughput-build-plan.md under docs/archive/throughput/ (#69). The rename updated docs/AOAG-DEPLOYMENT.md but not these three ci.yml comments, and nothing catches it: no guard scans workflow COMMENTS for doc paths, so the drift is invisible. Comment-only, no behaviour change. Fixed here rather than left because one of the three sits directly above the file list this branch appends to -- shipping a stale path in a block I just rewrote is my drift, not inherited drift.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What
A documentation accuracy pass over the public set, then the owner decisions it raised. Three commits:
10147b83203b4dd2docs/Secure_Development_Standards.mdabedf865Every claim was verified against code rather than against the prior prose, and 60 + 60 leads were rejected after checking. Those rejections matter as much as the fixes — in several cases the brief was wrong and the docs were already right.
Corrections that changed what the docs claim
SYSTEM-REQUIREMENTS.mdsaid "No native transport TLS" and "no MLLP-over-TLS". Both false, and both understated the product:settings.py:694-712is an in-process API/WebSocket TLS block (WP-13a, ADR 0002) andmllp.py:505-524builds per-connection MLLP SSL contexts with opt-in mTLS (WP-13b). It was telling adopters to add a reverse proxy they may not need.CONNECTIONS.md's ASVS 5.1.1 block claimed the file surface accepts "HL7 v2 text only". The code sniffs against the declaredcontent_type(file.py:514) and a non-HL7 drop bypasses HL7 handling entirely (file.py:644,673, ADR 0004). It advertised a narrower attack surface than exists — the kind of overclaim an assessor finds and then distrusts the rest of the document over. Now describes declared-type conformance.CONFIGURATION.mdwas missing ~40 implemented settings keys. An undocumented key is one an adopter cannot find — and two of them govern PHI encryption and single-factor admin.NOT_DEPLOYED, a seventh disposition genuinely set by the finalizer (store.py:319,912,926,955), was documented nowhere. Added toMENTAL-MODEL,USER-GUIDE,ARCHITECTURE.MENTAL-MODEL.mdnever mentioned engine sharding — the built default scaling axis — while asserting active-active was dropped. Together that read as "this does not scale". Both fixed, and the two are now distinguished rather than conflated.SERVICE.mdnow documents that a custom-Environmentname also requires[ai].data_classand[ai].production(install-service.ps1:66-71). Following the old text produced a service that would not start.Classification and structure
docs/SECURITY-DOCS-POLICY.mdstates what is kept private and why, using the operative rule: private only if it discloses an OPEN, un-remediated weakness in exploitable detail, or names a customer — a closed audit is transparency. The ~145 plain-text references into the withheld tree now have something to point at.docs/Secure_Development_Standards.mdis published (owner decision). The site already served it, it licenses itself as adopter-facing, and 44 citations pointed at a file that did not exist here. Published from the newest copy, de-branded back to the generic lineage its own revision history names.docs/is absent from the sdist allowlist, so this does not undo the #1020 packaging remediation.docs/archive/throughput/. Bodies intact — they are the record of why.ADRs, treated as historical records
Verification
names=7, estate=13, site_prefixes=1).test_feature_map_claimsandtest_backlog_status_checkconfirmed executed (23 PASSED), not skipped.command:VS Code URIs in an IDE ADR are legitimate, not paths.)BACKLOG.mdchanges are link-to-text only — no numbered heading added or removed, banner glyphs untouched.Three code defects this surfaced
Documented truthfully rather than described as intent, and filed separately:
min_age_secondsis a silent no-op onSftp()/Ftp()— accepted at wiring (wiring.py:2087,2147), never read (remotefile.py:715-748). It exists to avoid ingesting a file still being written.MEFOR_CERT_MONITOR_*/SECRET_ROTATION_*/UPDATE_CHECK_*can never work — the env parser splits the section at the first_(settings.py:3783);[sandbox]/[service]are missing from_SECTIONS.tls_hop_attestedrefusal messages instruct an impossible action — the field has no authoring surface, so an operator doing exactly what the error says hard-fails.For review before merge
57 security-posture passages are flagged for owner sign-off — chiefly the
[security]loosening pair, the[api]TLS/proxy keys,[store].cipher_provider, the ASVS 5.1.1 rewrite, and the web-console write-surface table.Correction to commit
5d60dfee's messageThat commit says the
overlap.ps1phantom "was biting" and thatMessageFoundry-pins"was stillblocking 16 files." Both overstate it, and the correction belongs where a reader meets the claim.
It was armed, not firing.
collision_gateonly blocks on a live session, and no sessionoccupies
MessageFoundry-pins— so nothing was ever denied.overlap.ps1's own comment says exactlythis: "dormant only: worth knowing, not worth blocking." The evidence supported "would bite", and I
wrote "was biting".
The diagnosis and the fix are unaffected: the phantom is real (16 authored, 14 differing,
intersection 0 — disjoint sets, so the work landed and
mainmoved elsewhere), and it would fire themoment anyone opened a session in that worktree. Caught by the messagefoundry-website session, which
also noted the mirror-image error on its own side: it had called the bug latent from an enumeration
that could only see
.claude/worktrees/*and missed the five<repo>-<name>siblings —git worktree listshows 10, not 3. Neither half was right alone.The commit message is left as-is deliberately. Force-pushing a wording fix would discard a verified
run across all 35 required checks — destroying real evidence to correct a description of it.
🤖 Generated with Claude Code