docs(testing): master test plan — 17 chapters, 1,186 rows, a three-tier release gate - #60
Merged
Conversation
…three-tier gate The umbrella test plan the two existing artifacts plug into. FEATURE-COVERAGE-PLAN.md owns the per-subsystem coverage-gap audit and WIN2025-TEST-PLAN.md owns what only a real Server 2025 host can prove; neither covers the product as a whole, and neither covers the IDE, the tray, the Steps editor or the publish path at all. This states what the product is, what has to be true before it ships, and who proves it — delegating rather than restating, with foreign IDs prefixed FCP:/W25: because both predecessors use IDs that collide with this plan's own (HA-20 and API-13 exist in both spaces). Three things are load-bearing: Row classes. Only class-T rows gate a release. A criterion of "Pass = report received" or "Pass = a published number" cannot fail, so it cannot be a gate — those are class C (characterisation, tracked) and class A (external assurance, gates production exposure only). The draft had a penetration-test procurement event sitting at P0. The gate is now 196 blocking T-rows + 9 campaign-gated + 3 C + 3 A = 211 P0-marked, reconciled row-for-row against an independent parse. The publishing boundary. docs/BACKLOG.md is a published baseline ending at #231 and docs/security/ is gitignored post-cutover, so a citation above the baseline or into a withheld tree is valid evidence, not a dangling reference. An earlier pass had this backwards and was filing withheld material as coverage defects, including one that reached a release-gate criterion. Two owner-named items were structurally unprovable. The tray had zero P0 rows — all eight sat on NSSM/service/install — so the icon-state mapping, action routing, Win32 message pump and the lying-tray case are now blocking. HA had no P0 that proved a failover works; six now do. Also folds in surfaces nothing owned: last_resort.py (the PHI-egress backstop), netaddr.py (the shared allow-list decision the co-location exists to keep consistent), pki.py, the shipped PySide6 harness GUI, disk-full/ENOSPC, licence/copyleft compliance, destructive operator errors and non-ASCII patient data end to end. Ships as an index plus 20 chapter files: the assembled document is 1.8 MB, past GitHub's markdown render limit. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
This was referenced Jul 30, 2026
wshallwshall
added a commit
that referenced
this pull request
Jul 30, 2026
PR #80 appended eight Steps-view items to docs/BACKLOG.md without the status banner every numbered item must carry, so tests/test_backlog_status_check.py::test_the_real_backlog_satisfies_the_invariant began failing on main itself (8 errors, first at line 6905). Because GitHub tests each PR merged into main, every open PR inherited the failure: #81, #74, #71, #66 and #60 were all blocked, three of them with auto-merge armed and unable to fire. Adds exactly one leading banner per item. Seven use the open/prioritized form; #239 uses the partial form, because its measurement ran and is recorded on PR #81 while the re-runnable scripts/quality/lens_coverage.py is still unmerged -- the number is not yet reproducible from main. The banners are deliberately minimal and do not re-litigate any item's verdict. #234 in particular is left explicitly unsettled rather than entrenched: it was filed as "revisit, not a bug" after the owner asked for a fix, and that framing is still open. Verified: scripts/docs/backlog_status_check.py exits 0 (237 items) and tests/test_backlog_status_check.py is 15 passed, was 14 passed 1 failed. Co-authored-by: Claude Opus 4.8 <noreply@anthropic.com>
wshallwshall
added a commit
that referenced
this pull request
Aug 2, 2026
That is the exact defect the backlog-hygiene gate exists for -- #60 shipped while its banner still read "PRE-RESERVED", and the stale banner was then repeated as fact in a merged PR. The gate did not fire here because it only inspects engine/IDE paths and this item is script-and-test work, which is precisely why the banner had to be corrected by hand rather than trusted to CI. Records what actually shipped (Half A) against what did not (Half B), and why Half B was backed out: claim.ps1 had three sessions in it at once, so the deferral is a coordination decision, not an estimate of difficulty. Someone reading this item needs that distinction to know it is pickup-able. The first draft of this named the contending BRANCH and the leak gate blocked it -- a worktree slug is internal naming and this ledger is public. Described by the code path it touches instead.
wshallwshall
added a commit
that referenced
this pull request
Aug 2, 2026
#345) (#141) * fix(coord): a pruned worktree stranded its work claims forever (BACKLOG #345) A claim (scripts/coord/claim.ps1) is a JSON file under <git-common-dir>/mefor-coord/claims/. It lives beside the SHARED object store, so it outlives the worktree that took it. prune-merged.ps1 removed worktrees and, across 1,108 lines, never touched claims -- and `claim.ps1 -Take` hard-blocks on any claim file that exists. So every prune could leave a key unclaimable by every future session until a human happened to run `-Release <key> -Force`. Nothing surfaced the condition and nothing could: the registry has no way to observe that a holder ceased to exist. Released on EVIDENCE, never on a timer. claim.ps1 is right that an auto-expiring claim silently re-opens the race it exists to prevent; this hooks only into the branch that has already PROVEN the directory is gone and deregistered ("OUTCOME, NOT EXIT CODE"), so there is no session left to collide with. A merely-quiet holder is never touched. Dry-run safety is inherited, not re-asserted: the whole apply loop is gated on -Apply. The false positive is worse than the bug. Releasing a claim held by a different, LIVING worktree hands its key away and invites the duplicate build the registry exists to stop, so the match is full normalised equality via ConvertTo-Norm -- no leaf name, no prefix, no StartsWith. Two tests pin it: a living sibling's claim survives, and the PRIMARY checkout's claim survives a sibling prune even though `<primary>` is a strict prefix of `<primary>-clean`. Both carry a positive control in the same invocation, so neither can pass on a run that released nothing. Two defects found by the tests rather than by review, both of the shape this file already guards against elsewhere: * an unreadable claim was counted once per REMOVED worktree, so one blocked key reported as 2. It belongs to the registry, not to any worktree -- not being able to read it is precisely not knowing whose it is -- so it is now surveyed once per run and reported at run level. * that survey ran only inside the removal branch, making it invisible to a dry run, and its Set-Exit sat AFTER the -Json block that emits the receipt and exits. The receipt would have carried exitCode 0 over a key nothing can claim -- the surface CI reads. `claims.scanned` distinguishes "read the registry, found it clean" from "never looked" (no git-common-dir, or no claims directory): an empty `unreadable` list is not a green light. Verified: the 8 new tests FAIL against the unpatched script before they pass against the patched one, so they can see the class. Full file green, 71 tests. * docs(backlog): #345's banner said "not started" on the PR that builds it That is the exact defect the backlog-hygiene gate exists for -- #60 shipped while its banner still read "PRE-RESERVED", and the stale banner was then repeated as fact in a merged PR. The gate did not fire here because it only inspects engine/IDE paths and this item is script-and-test work, which is precisely why the banner had to be corrected by hand rather than trusted to CI. Records what actually shipped (Half A) against what did not (Half B), and why Half B was backed out: claim.ps1 had three sessions in it at once, so the deferral is a coordination decision, not an estimate of difficulty. Someone reading this item needs that distinction to know it is pickup-able. The first draft of this named the contending BRANCH and the leak gate blocked it -- a worktree slug is internal naming and this ledger is public. Described by the code path it touches instead. * docs(worktrees): the prune receipt grew a claims surface and nothing described it WORKTREES.md is where the pruner's outcome contract is written down -- exit codes, which counts are subsets of which, why `git worktree prune` is never run. A new field in that receipt that only the source explains is the same drift this section already guards against elsewhere. States the property that makes the release safe rather than just the behaviour: it fires on PROVEN deletion, so a quiet holder is untouched and a dry run releases nothing, and the match is full path equality because freeing a LIVING worktree's key is worse than the orphan being cleaned up. * style(worktree): my #345 comments put 13 non-ASCII em-dashes in a BOM-less script prune-merged.ps1 had exactly ZERO non-ASCII bytes before this branch and no BOM -- every comment in it uses the ASCII `--`. A BOM-less file whose encoding is only inferred is a bad place to start introducing multi-byte characters: pwsh 7 assumes UTF-8 and would be fine, Windows PowerShell 5.1 assumes the ANSI codepage and would not, and nothing in the repo pins which one an operator runs it with. The file is back to 0 non-ASCII bytes. Comments only; CRLF line endings verified byte-identical in count before and after, and the claim tests re-run green.
wshallwshall
enabled auto-merge (squash)
August 3, 2026 13:06
wshallwshall
added a commit
that referenced
this pull request
Aug 3, 2026
…est plan (#161) * fix(docs): unbreak main — six new mirror-prose hits from the master test plan main has been red since 4ea1501 (#60): test_cutover_slug_rot's prose ratchet reports 58 hits against a ceiling of 54, so every PR current with main inherits the failure, #160 among them. Measured rather than inferred. The last green commit 8e6e7fa sits at 52 hits with 2 slots of slack, so #60 added SIX — not the four the ceiling arithmetic (58-54) suggests. c5ae1f8 (#96) added none. All six are in the new master-test-plan chapters, and the assertion's "New or changed" list is a tail slice of the sorted hits, not a diff, so it does not attribute them. Five are false positives in senses unrelated to the retired public mirror: three are `the mirror` matching INSIDE "the mirrorED" (the webview functions mirroring stepsModel, ADR 0108), one is a pip wheelhouse for air-gapped installs, and one is a detector spec quoting the very string it wants deleted — the _SELF collision this module already documents, reappearing in another file. The sixth is genuine rot. "the mirror-only gap" reincarnates the nightly cost lever that f7a337a (#25) swept from three files, leaving an explicit "do not re-add it" marker; windows-service-smoke is gated `github.repository == 'MEFORORG/MessageFoundry'`, i.e. ON this repo, which is the source. The line now names that gate instead of a mirror. The regex is deliberately NOT changed. Adding \b to `the mirror` clears three of the six and measures safe on the baseline (52 -> 52, suppressing only the three "mirrorED" lines) — but `private repo` -> "reporting" was a CROSS-LEMMA collision where \b removed noise, whereas `the mirror` -> "the mirrored" is the same lemma inflected, where it removes signal. Measured: \b blinds all five of "the mirrored workflows are rewritten by publish.ps1 on every push", "the mirroring job publishes ... nightly", "the mirrors are refreshed from the source repo each night", "CI runs on the source; the mirrored copy is read-only" and "the mirrored README is regenerated at release time", none of which carries a retrospective trigger — while both KEEP-class probes (the "mirror image of" idiom, a historical narrative) survive it untouched. Those verb forms are how a present-tense mirror claim actually gets written, so the boundary would preferentially blind the FIX class. Filing down the detector to clear a red is the failure this module exists to name. _PROSE_CEILING is unchanged at 54. The count returns to 52 — exactly the last-green posture — so no credit was earned to spend, and lowering it to 52 would set slack 0 as a new, stricter policy unrelated to this fix. Verified: 4/4 in the module; 52 hits / ceiling 54 / slack 2 across 1503 files; ruff check + ruff format clean; full suite 10222 passed, 836 skipped, with the sole failure test_gate_installed_parity — a local-machine test that skips on CI, whose installed-vs-source delta is one redacted comment line. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> * chore(ide): bump fast-uri/brace-expansion/undici to patched versions (3 HIGH advisories) npm-audit (a required gate) began failing on every fresh CI run: three HIGH advisories now cover the ide/ transitive deps at their locked versions — fast-uri (GHSA-7p8r-x3mc-p8w7), undici (GHSA-8xcm-r25x-g524 + siblings), and brace-expansion (GHSA-rgw5-rvv9-x895). Each has an in-major patched release, so this is a lockfile bump, not a breaking major: fast-uri ^3.1.3 -> ^3.1.5 brace-expansion ^5.0.8 -> ^5.0.9 undici (new override) ^7.29.0 npm audit --package-lock-only now reports 0 vulnerabilities. Folded onto the main-unblock branch so one merge clears both the slug-rot test-red and npm-audit (the two were a circular deadlock). * fix(deps): bump cryptography 49->50 + pyOpenSSL 26.3->26.4 (CVE-2026-69247) pip-audit (a required gate) flagged cryptography 49.0.0 for CVE-2026-69247 — a PKCS7 decrypt Bleichenbacher/timing oracle, fixed in 50.0.0. The straight bump was gated by pyOpenSSL's cryptography<50 cap; pyOpenSSL 26.4.0 lifts it to <51, so both move together, within the existing pyproject floors (cryptography>=48.0.1, pyOpenSSL>=26.3.0) — no constraint change. Re-locked via uv and re-exported all six DEP-1 artifacts (uv lock --check clean); the stale pyproject <50 note -> <51. The advisory's primary oracle is already mitigated on the OpenSSL 3.2+ wheels this project ships; this clears the flagged version regardless. Third and final red gate on the main-unblock branch: with the slug-rot ratchet and the ide npm advisories, one merge now clears all three. --------- Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
The umbrella test plan the two existing artifacts plug into.
FEATURE-COVERAGE-PLAN.mdowns the per-subsystemcoverage-gap audit;
WIN2025-TEST-PLAN.mdowns what only a realServer 2025 host under the NSSM service identity can prove. Neither covers the product as a whole,
and neither covers the IDE, the tray, the Steps editor or the publish path at all.
Ships as
docs/testing/MASTER-TEST-PLAN.md(index) plus 20chapter files under
docs/testing/master-test-plan/— the assembled document is 1.8 MB, pastGitHub's markdown render limit. Every chapter carries scope, what is already covered and must not
be re-tested, a risk analysis, the test matrix, detailed scenarios, an automation disposition,
prerequisites, exit criteria and open questions.
Foreign IDs are prefixed
FCP:/W25:throughout, because both predecessors use IDs that collidewith this plan's own —
HA-20andAPI-13exist in both spaces.What is load-bearing
Row classes. Only class-T rows gate a release. A criterion of "Pass = report received" or
"Pass = a published number" cannot fail, so it cannot be a gate — those are class C
(characterisation, tracked) and class A (external assurance, gates production exposure only).
The draft had a penetration-test procurement event sitting at P0. The gate is now 196 blocking
T-rows + 9 campaign-gated + 3 C + 3 A = 211 P0-marked, reconciled row-for-row against an
independent parse of all 17 matrices.
The publishing boundary.
docs/BACKLOG.mdis a published baseline ending at## 231.anddocs/security/is gitignored post-cutover, so a citation above the baseline or into a withheld treeis valid evidence, not a dangling reference. An earlier pass had this backwards and was filing
withheld material as coverage defects — including one that had reached a release-gate criterion.
Two owner-named items were structurally unprovable. The tray application held zero P0 rows
(all eight sat on NSSM/install) even though ADR 0113 names the untested Win32 layer as the design's
main risk; the state→icon map, action routing, message-pump dispatch and the "lying tray" are now
blocking. HA had no P0 proving a failover works — six now do, all on
container-CIwith aservice container, none needing a second host.
Surfaces nothing previously owned
last_resort.py(the process-level PHI-egress backstop),netaddr.py(the shared allow-listdecision its co-location exists to keep consistent),
pki.py, the shipped PySide6 harness GUI —which is both a distribution and the plan's only route to hostile-input injection — plus disk-full /
ENOSPC, licence and copyleft compliance, destructive operator errors, and non-ASCII patient data
end to end.
Review notes
whether
load_settingsshould refuse unknown keys, which transports are deliberately notauthorable from
connections.toml).messagefoundry checkis invoked by no workflow, and onemeta-test plus one
ci.ymlregex fix turns 36 already-written tests from dormant intoexecuting. That is items 1–2 of §24's prioritised ten.
🤖 Generated with Claude Code