Skip to content

ci(security): keep dependency-security.md true, and make expiry dates bite#266

Merged
jsugg merged 2 commits into
mainfrom
ci/docs-freshness-and-expiry
Jul 15, 2026
Merged

ci(security): keep dependency-security.md true, and make expiry dates bite#266
jsugg merged 2 commits into
mainfrom
ci/docs-freshness-and-expiry

Conversation

@jsugg

@jsugg jsugg commented Jul 15, 2026

Copy link
Copy Markdown
Owner

The gap

docs/dependency-security.md is accurate — I re-verified every claim. It is
also completely ungated:

scripts/github/run-security-audit.sh:  npm audit --omit=dev --audit-level=high

--omit=dev means it cannot see the advisories the document is about — they are
every one dev-only. --audit-level=high means it would not report them if it
could — they are every one moderate. The document's central claims had nothing
holding them up. It was true by luck, and nothing would have said when that
stopped.

Two halves, split by cost and blast radius

Generated tables (blocking, offline, deterministic). The override and
accepted-advisory tables now come from
config/security/residual-advisories.json. The docs gate fails when the
document and the manifest disagree, or when the manifest's override list and
package.json's do. Prose stays hand-written — only the claims that must track
reality are generated.

$ npm run security:docs
security:docs OK (6 overrides, 2 accepted advisories, verified 2026-07-15)

Both failure directions are tested, and I checked they actually fire:

- docs/dependency-security.md does not match the manifest. Run: npm run security:docs -- --write
- package.json pins lodash ^4.0.0 but the manifest says ^4.18.1
- the manifest lists qs but package.json does not pin it

Full-tree audit (scheduled, non-blocking). Security Audit / residual-advisories runs npm audit across the whole tree and compares it to
the manifest:

security:residual full-tree audit: {"moderate":9,"high":0,"critical":0,"total":9}
  OK reality matches the approved manifest

It reports advisories nobody approved and approved entries upstream has
since fixed
— the manifest rots in both directions, and an entry implying live
risk that no longer exists is its own kind of wrong.

Non-blocking on purpose, and this is the whole design point: the advisory
database changes without anyone touching this repo. A required check here would
go red because a third party published overnight — the same reason the audit
warned against making external-link liveness a required gate.

The production npm-audit job is untouched. --omit=dev --audit-level=high
is the right question to ask about production risk. It is just not the question
this document answers, which is why the document needed its own.

The manifest records the lockfile SHA it was verified against, so a lockfile
change is what surfaces the need to re-verify.

I introduced advisories, and this caught them

Worth stating plainly. Adding markdownlint-cli2@0.18.1 (in #261) pulled in
js-yaml@4.1.0 and markdown-it@14.1.0 — four moderate advisories between
them. The full tree went from the documented 9 to 12, and I did that.
dependency-review passed it, because the production gate cannot see dev-only
moderates. This is exactly the blind spot.

markdownlint-cli2@0.23.0 clears all four and returns the tree to the 9
documented here, none of which are ours:

before: js-yaml, markdown-it, jest-junit, jose, newman, postman-*, serialised-error, uuid   (12)
after:  jest-junit, jose, newman, postman-*, serialised-error, uuid                          (9)

Its new MD060 table-pipe rule is off: the repo mixes |---| and | --- |,
and normalising 24 rows across four files — one of them required-checks.md,
whose content the audit verified as correct — is churn, not a fix.

Expiry dates

docs/coverage-thresholds.md carries Expiry: 2026-09-30 on its ratchet
exceptions. Accurate today; a defect on that date, with nothing to flag it. The
docs gate now fails on any documented Expiry: that has passed:

2026-09-30 as of 2026-07-15  -> ok
2026-07-14 as of 2026-07-15  -> Documented Expiry 2026-07-14 has passed — revisit or extend it
2026-13-45                   -> Expiry date is not a valid date

Fenced code blocks are ignored, today-exactly is allowed, and one test asserts
the check is watching the real coverage-thresholds.md date rather than only
synthetic strings.

Verification

lint · typecheck · docs:validate 8/0 · docs:lint 0 · security:docs ·
actionlint · 876/876 unit (104 suites). The script's lockfile hash agrees
with shasum -a 256 and with the manifest.

jsugg added 2 commits July 15, 2026 10:47
… bite

docs/dependency-security.md was accurate and completely ungated. The scheduled
audit runs `npm audit --omit=dev --audit-level=high`, so it structurally cannot
see the advisories the document is about: they are every one dev-only and every
one moderate. The document was true by luck, and nothing would have told anyone
when it stopped being.

Both halves of the fix are deliberately split by cost and blast radius.

The tables are now generated from config/security/residual-advisories.json, and
a deterministic offline check in the docs gate fails when the document and the
manifest disagree, or when the manifest's override list and package.json's do.
Prose stays hand-written; only the claims that must track reality are generated.

A new scheduled job runs the full tree and compares it to the manifest,
reporting advisories nobody approved and approved entries upstream has since
fixed — the manifest can rot in both directions, and an entry implying live risk
that no longer exists is its own kind of wrong. It is non-blocking on purpose:
the advisory database changes without anyone touching this repo, and a required
check must not go red because a third party published overnight. The production
npm-audit gate is untouched — --omit=dev --audit-level=high remains the right
question to ask about production risk, and is exactly why this needed its own.

The manifest records the lockfile SHA it was verified against, so a lockfile
change is what surfaces the need to re-verify.

Expiry dates: coverage-thresholds.md carries `Expiry: 2026-09-30` on its ratchet
exceptions. It is accurate today and becomes a defect on that date with nothing
to flag it. The docs gate now fails on any documented Expiry that has passed, so
the date announces itself instead of quietly going stale.

markdownlint-cli2 moves 0.18.1 -> 0.23.0. Adding it pulled in js-yaml@4.1.0 and
markdown-it@14.1.0, which carry four moderate advisories between them — the full
tree went from 9 to 12. 0.23.0 clears all four and returns the tree to the 9
documented here, none of which are ours. Its new MD060 table-pipe rule is off:
the repo mixes `|---|` and `| --- |`, and normalising 24 rows across four files
is churn, not a fix.
@jsugg
jsugg merged commit adf30f5 into main Jul 15, 2026
22 checks passed
@jsugg
jsugg deleted the ci/docs-freshness-and-expiry branch July 15, 2026 13:57
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant