ci(security): keep dependency-security.md true, and make expiry dates bite#266
Merged
Conversation
… bite docs/dependency-security.md was accurate and completely ungated. The scheduled audit runs `npm audit --omit=dev --audit-level=high`, so it structurally cannot see the advisories the document is about: they are every one dev-only and every one moderate. The document was true by luck, and nothing would have told anyone when it stopped being. Both halves of the fix are deliberately split by cost and blast radius. The tables are now generated from config/security/residual-advisories.json, and a deterministic offline check in the docs gate fails when the document and the manifest disagree, or when the manifest's override list and package.json's do. Prose stays hand-written; only the claims that must track reality are generated. A new scheduled job runs the full tree and compares it to the manifest, reporting advisories nobody approved and approved entries upstream has since fixed — the manifest can rot in both directions, and an entry implying live risk that no longer exists is its own kind of wrong. It is non-blocking on purpose: the advisory database changes without anyone touching this repo, and a required check must not go red because a third party published overnight. The production npm-audit gate is untouched — --omit=dev --audit-level=high remains the right question to ask about production risk, and is exactly why this needed its own. The manifest records the lockfile SHA it was verified against, so a lockfile change is what surfaces the need to re-verify. Expiry dates: coverage-thresholds.md carries `Expiry: 2026-09-30` on its ratchet exceptions. It is accurate today and becomes a defect on that date with nothing to flag it. The docs gate now fails on any documented Expiry that has passed, so the date announces itself instead of quietly going stale. markdownlint-cli2 moves 0.18.1 -> 0.23.0. Adding it pulled in js-yaml@4.1.0 and markdown-it@14.1.0, which carry four moderate advisories between them — the full tree went from 9 to 12. 0.23.0 clears all four and returns the tree to the 9 documented here, none of which are ours. Its new MD060 table-pipe rule is off: the repo mixes `|---|` and `| --- |`, and normalising 24 rows across four files is churn, not a fix.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
The gap
docs/dependency-security.mdis accurate — I re-verified every claim. It isalso completely ungated:
--omit=devmeans it cannot see the advisories the document is about — they areevery one dev-only.
--audit-level=highmeans it would not report them if itcould — they are every one moderate. The document's central claims had nothing
holding them up. It was true by luck, and nothing would have said when that
stopped.
Two halves, split by cost and blast radius
Generated tables (blocking, offline, deterministic). The override and
accepted-advisory tables now come from
config/security/residual-advisories.json. Thedocsgate fails when thedocument and the manifest disagree, or when the manifest's override list and
package.json's do. Prose stays hand-written — only the claims that must trackreality are generated.
Both failure directions are tested, and I checked they actually fire:
Full-tree audit (scheduled, non-blocking).
Security Audit / residual-advisoriesrunsnpm auditacross the whole tree and compares it tothe manifest:
It reports advisories nobody approved and approved entries upstream has
since fixed — the manifest rots in both directions, and an entry implying live
risk that no longer exists is its own kind of wrong.
Non-blocking on purpose, and this is the whole design point: the advisory
database changes without anyone touching this repo. A required check here would
go red because a third party published overnight — the same reason the audit
warned against making external-link liveness a required gate.
The production
npm-auditjob is untouched.--omit=dev --audit-level=highis the right question to ask about production risk. It is just not the question
this document answers, which is why the document needed its own.
The manifest records the lockfile SHA it was verified against, so a lockfile
change is what surfaces the need to re-verify.
I introduced advisories, and this caught them
Worth stating plainly. Adding
markdownlint-cli2@0.18.1(in #261) pulled injs-yaml@4.1.0andmarkdown-it@14.1.0— four moderate advisories betweenthem. The full tree went from the documented 9 to 12, and I did that.
dependency-reviewpassed it, because the production gate cannot see dev-onlymoderates. This is exactly the blind spot.
markdownlint-cli2@0.23.0clears all four and returns the tree to the 9documented here, none of which are ours:
Its new
MD060table-pipe rule is off: the repo mixes|---|and| --- |,and normalising 24 rows across four files — one of them
required-checks.md,whose content the audit verified as correct — is churn, not a fix.
Expiry dates
docs/coverage-thresholds.mdcarriesExpiry: 2026-09-30on its ratchetexceptions. Accurate today; a defect on that date, with nothing to flag it. The
docsgate now fails on any documentedExpiry:that has passed:Fenced code blocks are ignored, today-exactly is allowed, and one test asserts
the check is watching the real
coverage-thresholds.mddate rather than onlysynthetic strings.
Verification
lint · typecheck ·
docs:validate8/0 ·docs:lint0 ·security:docs·actionlint· 876/876 unit (104 suites). The script's lockfile hash agreeswith
shasum -a 256and with the manifest.