Skip to content

Quote the pinned sentence behind each check - #70

Merged
adewale merged 2 commits into
mainfrom
spec/requirement-keywords
Nov 13, 2026
Merged

adewale merged 2 commits into
mainfrom
spec/requirement-keywords

Conversation

@adewale

@adewale adewale commented Nov 13, 2026 •

Copy link
Copy Markdown
Owner

What

Ties each normative validator message and mock check to the pinned sentence it enforces, so a change to a key word upstream fails npm run check:spec. It also fixes the two validator errors and the mock bases that the new check found stale.

Most of the 664 changed lines are quotes added to src/validator/messages.ts and test/mock-company/violations.ts. The checker verifies those mechanically. The parts that need a reviewer's judgement are the basis changes listed under How, scripts/check-requirements.mjs, and the scope check in src/validator/profiles/draft-0.4.ts.

Why

Fixes #71.

The live validator calls northbound.camp invalid, partly because its issuer lists openid, profile, email, phone and offline_access in scopes_supported without describing them in auth.scopes. That's scope-undescribed, a MUST error that makes our client refuse the Company. In 0.4 and 1.0, 4.7 says:

The Company SHOULD describe each scope it offers Personal Agents, so the Personal Agent can choose the ones a task needs

The root cause:

  • 28 October: the rule was written from 0.3's text of that day, which said "The Company MUST describe each scope in scopes_supported".
  • Between 29 October and 3 November: upstream changed the sentence to the SHOULD by editing an existing changelog entry in place.
  • 3 November: our refresh updated the ledger note but not the message. The note pointed at the client, not the validator.
  • 12 November: the 0.4 profile copied the rule unchanged, and 1.0 inherited it.

check:spec checked that each changelog entry had a disposition. Nothing compared a message's basis with the pinned text.

mediated-scope-unsupported has the same history. On 28 October the text said "all of them in scopes_supported"; by 3 November it said "The Company SHOULD also include them".

Northbound still has one real error after this change: its protected resource metadata says "resource": "https://www.northbound.camp/api", and 3.2 requires https://northbound.camp. Run on this branch, npm run validate -- northbound.camp now reports that error and two info notes, and nothing about scopes.

How

  • scripts/check-requirements.mjs, run by check:spec:

    • A validator message with a MUST, SHOULD or Rule basis must quote its sentence in a new requires field.
    • A mock check with a MUST, MUST NOT, SHOULD, SHOULD NOT, Rule or Guides basis must quote it in its basis string.
    • Each quote must appear in every runtime pin (0.4 and 1.0), or in 1.0 alone for a 1.0-only rule. The comparison strips links and code marks, and … skips text within a paragraph.
    • Each quote must carry its basis's key word: MUST, or a required member, for a MUST; SHOULD for a SHOULD; neither for a Rule or the Guides.
    • When upstream changes a key word, the sentence changes, so the quote stops matching. Quotes of RFCs and MCP aren't checked.
  • Quotes: 63 messages and 79 mock checks gained a quote or had theirs fixed. Every quote passed the checker, and I read the uncertain ones against the pins. Writing them moved bases. Each move below keeps its severity unless it says otherwise:

    • scope-undescribed: MUST → SHOULD. Now a warning, and the client no longer refuses over it.
    • mediated-scope-unsupported: MUST → SHOULD. Now a warning.
    • mediated-grant-missing, nothing-offered, issuer-ambiguous, mock session-jwt.not-jwt and feedback.fields: MUST → Rule. The pinned text states them with no key word.
    • not-a-url, domain-invalid: Rule → MUST. Each rests on a MUST sentence in 3 or 3.1.
    • json-duplicate-member: SHOULD → Guides. Still a warning that the client refuses.
    • jwks-kid-duplicate: its ref is now RFC 7517 §4.5, where its SHOULD comes from.
    • Mock version.missing: MUST → Rule. 3.4's sentence has no key word.
    • Mock account-token.rotated-reused: MUST → RFC 6749 §6. 4.4 no longer says MUST.
    • Mock handoff.body and close.body: they quoted an "empty JSON object". Under 0.4 these bodies carry a request ID, and the mock already accepted that.
  • Scope check: when the domain's protected resource metadata lists scopes_supported, scope-undescribed checks those scopes against that document, and skips the issuer's list. 3.2 says the domain's list "suits a Company whose issuer also serves other resources or clients", and 4.7 says either list "MAY leave out scopes the Company offers only to its other clients". Recorded as a reading.

  • RS256: the mock no longer accepts RS256 client assertions. The 0.4 changelog says "Companies are no longer told to accept RS256 client assertions", and the mock's own description already said ES256 or PS384.

  • Stale version text: the validator CLI's --protocol help and a validate.ts comment still called 0.4 the only version and the default. The default is 1.0.

  • Upgrade procedure: npm run spec:keywords -- <old snapshot> <new snapshot> lists the sentences whose key words changed. docs/upgrading.md step 3 runs it before a new pin. Across our consecutive pins it reports:

    • nine changes from 29 October to 3 November, including the scopes, rotation, PS384 and report changes;
    • one from 9 to 10 November;
    • three from 0.4 to 1.0, all PS384 becoming a MUST for Companies.

    Every one is either handled in this PR, already handled (metadata-assertion-algs for PS384 under 1.0), or a Company obligation that no check of ours enforces.

Not in this PR

The audit found checks whose behaviour, and not only their quote, is broader than their sentence. Each needs its own id, TAMPERS row and decision, so they're listed in TODO.md:

  • message.author under 1.0;
  • session-jwt.sub-changed: under 0.4 a known sid with another User ID starts a new Session, and the mock still refuses it;
  • the assertion half of authorize.secret-in-url;
  • the name-only half of http-non-public-host;
  • the grid's own company.* checks, which quote nothing yet.

Testing

  • scripts/check-requirements.test.mjs uses 0.3's two 4.7 sentences. The MUST quote matches the 28 October text and fails against the 3 November one, and a SHOULD quote fails under a MUST basis.
  • test/validator.test.ts:
    • the existing undescribed-scope case is now a warning, and the report stays valid;
    • a new case lists orders:* in the protected resource metadata and openid, profile and offline_access only in the issuer's list. It checks that only the undescribed protected-resource scope is reported, as a warning on that document.
  • Regression guard: with scope-undescribed put back to MUST and its 28 October quote, npm run check:spec fails: scope-undescribed: not in the 0.4 text.
  • Run locally, all passing: npm run check:spec, npm run check:docs, npm run typecheck, npm run test:tooling (22), npm test (1,756 passed, 1 skipped, after fixing the catalog test's format regex for quotes with parentheses), the interop unit tests (454), npm run test:properties (209) and npm run test:hypothesis (63).

No pixels change. The validator's messages page shows the new wording and severities through the existing template.

Risk

  • Validator results: reports change for real Companies. Two errors become warnings, and nothing that was a warning becomes an error. The other six validator basis changes keep their severity.
  • Mock: an agent that signs client assertions with RS256 is now refused by the mock. Our agent signs with ES256.
  • Maintenance: check:spec runs on every PR. A refreshed pin now fails until each changed sentence's check is reviewed, which is the point of the change and adds work to every refresh.

🤖 Generated with Claude Code

Ade Oshineye and others added 2 commits November 13, 2026 17:48
scope-undescribed was written as a MUST from 0.3's 28 October text and
stayed one after 4.7 became a SHOULD on 3 November, so the validator
called northbound.camp invalid and our client refused it. Nothing compared
a message's basis with later pins.

Each MUST, SHOULD or Rule validator message now quotes its sentence in
`requires`, and each such mock check quotes it in its basis.
check-requirements.mjs, run by check:spec, fails when a runtime pin lacks
the quote or the quote lacks its basis's key word. Writing the quotes
found mediated-scope-unsupported (also a SHOULD since 3 November), stale
0.3 quotes in the mock, and RS256 client assertions the 0.4 Guides
dropped. scope-undescribed checks the domain's own scopes_supported when
its protected resource metadata lists one (a reading). spec:keywords
reports key-word changes between snapshots for the upgrade procedure.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@adewale
adewale merged commit d9323da into main Nov 13, 2026
4 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Poppy scope out of compliance bug

1 participant