Skip to content

test: measure which endpoint families the gateway serves - #114

Open
viztor wants to merge 14 commits into
claude/vigilant-mendel-bctz9n-p-r1from
claude/vigilant-mendel-bctz9n-p-r2
Open

viztor wants to merge 14 commits into
claude/vigilant-mendel-bctz9n-p-r1from
claude/vigilant-mendel-bctz9n-p-r2

Conversation

@viztor

@viztor viztor commented Oct 24, 2026 •

Copy link
Copy Markdown
Owner

Phase P-R2 of docs/improvement-plan.md, stacked on #113.

P-R2 is a measurement phase. The default endpoint family stays modern, so the plugin's behaviour does not change and no S0_TODAY or characterization row flips. The title carries no ! and the commit has no breaking footer.

What changed

# Item Where
1 The default has a name. AUTO_ENDPOINT_FAMILY = "modern" is what auto resolves to. Before this, the choice sat inside a ternary. A later flip is a one-word edit, and the constant is pinned. src/utils/plane-model.ts, test/endpoint-family.test.ts
2 Wider keyless matrix. It now covers 3 URL shapes: legacy /zen(/go)/v1, a new root /inference(/go)/v1, and modern /inference(/go)/{openai,anthropic,google}/…. That is 24 cells over 4 protocols and 2 planes. It adds LEGACY_DISPATCH (the cells endpoint: legacy really sends to) and one keyless free-model probe with its own pure classifier. classifyKeylessProbe is unchanged. test/support/keyless-matrix.ts, test/e2e/keyless-matrix.e2e.ts
3 Minimal keyed probes. There is one probe per plane × protocol × shape. Each uses the plane's own key in the header its protocol reads, with tiny bodies (8/16/1/8 tokens) and a pure keyed classifier. The identity comes from one shared constant, PROBE_IDENTITY, and an offline test pins it against the identity a real turn sends. Only the cells the default sends to (TODAY_DISPATCH) are asserted. Every other cell is recorded. test/support/keyed-probes.ts, test/e2e/keyed-probes.e2e.ts, test/keyed-probes.test.ts
4 The live job is gated by tier. OPENCODE_E2E_KEYED takes none, probes or full. Unset means full, so local runs behave as before. CI picks the tier per event: probes for a pull request touching src/host/**, protocols.ts, plane-model.ts, identity.ts, request-shape.ts or the probes' own inputs; full nightly and on dispatch; none otherwise. Secrets are passed only when the tier is not none. The full keyed suites read their keys through armedKey(tier, "full", …). .github/workflows/ci.yml, test/e2e/*.e2e.ts
5 A dated record. Both live suites write a markdown table to $OPENCODE_E2E_RECORD_DIR. The job publishes it to the run summary and to an endpoint-families artifact. test/support/endpoint-record.ts, test/endpoint-record.test.ts
6 Docs. docs/redirect-planes.md §13 uses the same columns, and every cell reads "pending the first CI run". A unit test holds the doc's tables to the formatter's columns and row order. The tier scope, the skip reasons and the plan's status row are updated. docs/redirect-planes.md, docs/improvement-plan.md, CONTRIBUTING.md, AGENTS.md

Decision recorded in the plan: endpoint: legacy is recorded, not guarded, in the live job. Unit tests pin its mapping of every protocol on both planes. If a keyed modern cell fails in CI, that is the evidence for a later fix: that flips the default.

Behaviour changes

Plugin: none. The changes below are to the test suite and CI only:

  • The keyed probes take their identity from one shared constant, PROBE_IDENTITY in test/support/keyed-probes.ts. What they send live is unchanged: user-agent OPENCODE_UA, x-opencode-client, x-opencode-project, and the probe session id under both session headers. They deliberately leave out x-opencode-request, the per-request metric id.
  • CI arms the probes keyed tier on a pull request that touches src/utils/request-shape.ts, because every Messages probe sends its anthropic-version. This is in addition to src/host/**, protocols.ts, plane-model.ts, identity.ts and the probes' own inputs.
  • An earlier review round switched protocol-routing's PLUGIN_HEADERS to OPENCODE_UA. That user agent reaches the suite's keyless free-tier and unmetered cases on every tier, so on every pull request and every push to main. This PR's own e2e run is therefore the first live observation of the new user agent on keyless traffic, before the nightly run. docs/credentials.md records that this user agent did not change the free-tier verdict.

Review

Each round ran three reviewers (correctness, tests, rules) with none missing. Every upheld finding was fixed in the next round.

Round Found Upheld Fixed Headline findings
1 22 22 22 (round 2) (high) keyed probes sent no session header, so the Go half passed without measuring anything; (medium) a keyed probe passed green with no default-family cell measured; (medium) a rate-limit hang turned the probes red where protocol-routing skips
2 8 8 8 (round 3) (medium) probes sent the plugin's own-request user agent, not the one its turns carry, and the unit test checked against that same constant; 404 model-gone matched Anthropic's generic not_found_error; a pipefail hazard in the tier step
3 8 8 8 (round 4: 7 in the tree, 1 in wording) (medium) the live suite's identity was a second copy no offline test checked; upstream-busy clauses never decisive alone; the tier scope left out request-shape.ts

Round-4 mutation checks were each run, watched fail, then restored:

  • The e2e passing its own patch-UA identity: 1 test failed.
  • PROBE_IDENTITY bound to the patch UA: 4 failed.
  • Dropping the 529 clause: 1 failed.
  • Dropping the UNAVAILABLE and "model is overloaded" alternatives: 2 failed.
  • A case-insensitive UNAVAILABLE match: 1 failed.

Verification

Node Step Result
26 pnpm run check 0 errors, 11 warnings in 203 files (limit 21)
26 vp test run 94/94 files, 1454/1454 tests
26 vp test run --config vitest.mock.config.ts 4/4 files, 61/61 tests
26 pnpm run build exit 0
26 scripts/check.ts check passed, including "lib/ carries the current build, including the served client bundle"
26 shuffle, seeds 1, 42 and 1234 94/94 files, 1454/1454 tests each
26 vp test run --coverage statements 97.64 (ratchet 97.5), branches 94.31 (94), functions 97.51 (97.4), lines 97.63 (97.5); per-file thresholds pass
24 shuffle, seed 7 94/94 files, 1454/1454 tests
24 mock suite 4/4 files, 61/61 tests
— local-path grep over git diff, the 6 new files and lib/ 0 hits; a positive control matched 1
— credential-shape grep over the diff and new files only a pinned action SHA and the test fixture key sk-probe-0123456789abcdef

The first five Node 26 rows were re-run on the final tree right before the commit. The live keyed and keyless results are not in this PR: they come from this branch's CI run, and §13 stays "pending" until then.

Known red check: catalog freshness fails on every branch until the vendor-catalog refresh lands on main. It does not come from this change.

Restacked onto main 1.8.0

The branch now carries main 1.8.0 through its parent, #113 (P-R1), which merged main. P-R2 merged P-R1; it did not merge main itself.

Scope after the restack

test: measure the endpoint families against the live gateway, on both harness channels.

  • AUTO_ENDPOINT_FAMILY (modern) is the one value auto resolves to. test/endpoint-family.test.ts pins the bases of the default and of legacy for every plane × protocol, and the cells each setting reaches (TODAY_DISPATCH, LEGACY_DISPATCH). Since P-R1 these are the same on every host cell. No behaviour change, and no characterization row flips.
  • Keyless matrix: 3 shapes (legacy, root, modern) × 4 protocols × 2 planes, plus a keyless free-model probe and System One's three shapes (L13).
  • Keyed probes, tiered by OPENCODE_E2E_KEYED (none/probes/full). They add the keyed variants of the default completions cell: x-opencode-request (L4), x-opencode-project as global, folder or 40-hex (L5), and reasoning_effort: "none" (L10). They also record every 429's Retry-After (L9).
  • The usage cap is its own verdict. A 429 FreeUsageLimitError is now free-usage-limited and is never retried (L3). It is not a rate limit, and it is not free-tier-gated, the name already used for the 403 session gate.
  • The classifiers and outcome rules are pure and unit-tested offline. The mock serves every request that either endpoint setting dispatches and no root one. This is main's test(mock): the mock must serve every base the plugin can produce #142 mock-coverage gate, ported to work per request.
  • Both suites write dated tables to the run summary and to the endpoint-families artifact, for §13.
  • P-R2b (a conditional fix:) pastes the first CI run into §13. It flips the default only if that run shows flip evidence.

Dropped, because main or the parent now does it

  • ModelDeprecated as a P-R2 gone rule. P-R1 brought it with live-probe.ts's gone rule. After the merge, keyless-matrix.ts's isModelGoneMessage carries it, so there is nothing left for P-R2 to add.
  • "Probe the root family /inference/v1" (L3). P-R2 already probed the root shape as one of its three. The only new URL is System One's root.
  • The name free-tier-gated for the usage cap. That name already means a 403 FreeTierError (a pass). The cap is free-usage-limited (a skip).
  • D14' (pinned pi 1.3.0 with the harness patch) and the gate's patched alpha cell. The owner's no-patch, two-channel decision, which P-R1 carries, supersedes both.
  • The combined Google User-Agent re-measure. It was closed offline. The fetch patch replaces the User-Agent on every protocol's request, and test/mock/protocol-compat.mock.ts pins Google's wire value to OPENCODE_UA on both planes. docs/redirect-planes.md §9 now says so.

Decisions applied

  • D14'' / D22'' (supersede D14' / D22'): this repository carries no pi patch, and two channel records, rc and alpha, are named once. The dispatch set and the binary e2e cover both channels. patches/ is gone from the CI seam regex.
  • D26 (with D16 and D17 as it rewrites them): baseUrlFor takes no pi argument, and legacy reads the fixed table. Test fixtures restate none of pi-ai's defaults, so test/endpoint-family.test.ts now uses stand-in window and output values.
  • D24 and D25 do not touch this phase.

Live re-measures CI must confirm

  1. The first e2e run on this PR uses the probes tier, because the PR touches src/utils/plane-model.ts and ci.yml. Paste its keyless and keyed tables into §13 (P-R2b). If a modern cell shows flip evidence, flip AUTO_ENDPOINT_FAMILY, together with the copy, TODAY_DISPATCH and the re-tagged wire facets.
  2. From that run, record:
    • the free probe's verdict on the current gate (served, free-tier-gated or free-usage-limited);
    • the root /inference(/go)/v1 cells;
    • System One's three shapes;
    • x-opencode-request present vs absent (L4);
    • x-opencode-project as global, folder or 40-hex (L5);
    • Retry-After on Go's 429s (L9);
    • reasoning_effort none vs absent (L10).
  3. The nightly full run should measure whether the legacy /zen/v1 cells still answer for plane models under the current gate.
  4. These are inherited from P-R1 and were confirmed again locally. They belong to the alpha.3 bump and upstream, not to this PR:
    • On the alpha npm cell, the route claim does not engage (KNOWN_DEFECT).
    • On every cell, no config-editor save succeeds (HOST_LIMITATION).

Generated by Claude Code

A measurement phase: the default endpoint family stays `modern`, so no
plugin behaviour changes and no characterization row flips.

- Name the default in one constant, AUTO_ENDPOINT_FAMILY in
  src/utils/plane-model.ts, which is what `auto` resolves to, and pin
  it in test/endpoint-family.test.ts.
- Extend the keyless matrix to three URL shapes (legacy, root, modern)
  over four protocols and two planes, add LEGACY_DISPATCH, and add one
  keyless free-model probe with its own pure classifier.
- Add minimal keyed probes (pure test/support/keyed-probes.ts and its
  e2e): one tiny request per plane, protocol and shape, sent with the
  plane's own key and the plugin's identity from one shared constant,
  PROBE_IDENTITY. Only the cells the default sends to are asserted;
  the rest are recorded.
- Gate the live job by tier with OPENCODE_E2E_KEYED (none, probes,
  full; unset means full). CI arms `probes` on a pull request that
  touches what the probes send, `full` nightly and on dispatch, and
  passes the secrets only when the tier is not `none`.
- Write a dated endpoint-family record from both live suites to the
  run summary and an `endpoint-families` artifact, and add §13 to
  docs/redirect-planes.md with every cell pending the first CI run.

The live part runs in CI only; no live result is invented here. A
failing keyed modern cell in CI is the evidence for a later `fix:`
that flips the default.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_0141fKp5k6hiaJcP5ovhGtSD

viztor commented Oct 24, 2026

Copy link
Copy Markdown
Owner Author

catalog freshness fails on 4c491fe, in the Go limits half. The model catalog itself reports shim is current. The step that fails is limits:shim: it reports src/go-limits-data.ts as stale and notes a new row it resolves to gemini-4-argon, an id not in the bundled Go catalog.

This PR didn't cause it. scripts/generate-go-limits.ts and src/go-limits-data.ts are byte-identical to the base (#113), and the same check passed on #113 at 13:34 UTC. The vendor's Go limits page changed after that, so every branch, main included, fails this step on its next run.

No fix exists yet. Regenerating the table needs opencode.ai, which this machine can't reach, and the generated file is never edited by hand. The nightly scheduled run on main regenerates both tables into chore/generated-refresh (#111). Once it does, I'll port the new file down the stack the same way as 813e6f3. A re-run before then would fail the same way, so I haven't spent one.


Generated by Claude Code

What the review found, and what changed:

- The keyed flip rule read a default cell's retired-model answer as a
  retired model without checking the same model on the other shapes.
  planeOutcome now reads each default cell beside the legacy and root
  cells of the same plane and protocol.
- A refused key counted as flip evidence even when every shape refused
  it. It is now "check the CI secret" when a sibling refused the key
  and none accepted it, and "nothing else measured the key" only when
  no sibling refused it; a path-gone, rate-limited or busy sibling no
  longer blocks the CI-secret reason.
- Breaks no flip could fix (accepted on the root shape only, or Go's
  Google, which no flip moves) were called flip evidence, and the
  "protocol" and "gone" reasons named causes nothing measured. A
  wrong-protocol or path-gone cell is now decided by the cell that
  endpoint: legacy dispatches to: "refused" (a flip would not serve
  it) when that cell refused readably, and always for Go's Google;
  "unmeasured" (re-run) when it measured nothing. Neither is flip
  evidence.
- An account answer counted as the flip target accepting a protocol or
  a model, though the account's gate answers before either is read. It
  now counts only beside a refused key or a gone path, and a retired
  model next to an account answer no longer fails the plane.
- The keyless free probe failed on a busy upstream (503/529) that the
  keyed classifier treats as routed. It now moves on past a busy
  candidate, and its dated record no longer pairs a derived
  upstream-busy with the last retired candidate's model and status.
- A busy upstream followed by no answer failed the keyed plane, against
  the verdict definitions in redirect-planes.md section 13. Section 13
  now says what the code does (no answer on the last attempt is
  transport-error unless an earlier attempt was a 429), and a test pins
  that it fails as unreadable, not as flip evidence.
- Comments and docs said P-R2 had already measured the families while
  section 13 is pending its first CI run; said models.opencode.ai
  declares the root shape, which the generated catalog contradicts;
  said legacy cells are asserted nowhere live; and said a keyed
  auth-refused means the plugin's own turns would be refused, though a
  default Go turn sends a different key. All corrected.
- CONTRIBUTING's key and mirror bullets now cover the keyed probes.
- The plan no longer claims P-R2 opens with a green pin, gives the
  conditional flip a phase, and says the keyed probes decide a plane
  from the default cells read beside their siblings, not alone, and
  that an unreadable answer is never flip evidence.
- Tests now cover the tier wiring of the four keyed live suites, no
  probe following a redirect, the push-to-none arm and the publish and
  upload steps of the CI wiring, the free probe moving on past a busy
  upstream, and the record being rewritten after every cell.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_0141fKp5k6hiaJcP5ovhGtSD

viztor commented Oct 27, 2026

Copy link
Copy Markdown
Owner Author

This phase's first reviews ran on the general model because the dedicated reviewer was unavailable at the time. It has now had the dedicated review: three rounds through three lenses (correctness, repository rules, test matrix).

Rounds

Round Found Upheld Fixed Upheld by lens
1 16 15 15 correctness 5, rules 6, test matrix 4
2 7 6 6 correctness 3, test matrix 3
3 6 6 6 correctness 3, test matrix 3

Four pairs of upheld findings were the same defect reported by two lenses. Rounds 2 and 3 mostly reviewed the previous round's classifier fixes.

Fixed

  • test/support/keyed-probes.ts: planeOutcome now reads each default cell beside the legacy and root cells of the same plane and protocol. A model that the other shapes serve is no longer passed as retired.
  • test/support/keyed-probes.ts: a refused key reads "check the CI secret" when a sibling that read the key refused it and none accepted it. It reads "nothing else measured the key" only when no sibling refused it.
  • test/support/keyed-probes.ts: a break that no flip could fix (accepted on the root shape only, or Go's Google) still fails, but is not counted as flip evidence.
  • test/support/keyed-probes.ts: wrong-protocol and path-gone are now decided by the cell that endpoint: legacy dispatches to. They read refused ("a flip would not serve it") or unmeasured ("re-run"). These replace the protocol and gone reasons, which named causes nothing had measured.
  • test/support/keyed-probes.ts: an account answer counts as acceptance only beside a refused key or a gone path, not beside a refused protocol or a retired model.
  • test/support/keyed-probes.ts: the keyed auth-refused comment now says that a Go entry without apiKeyEnv sends the Zen key, which the probe does not measure.
  • test/support/keyless-matrix.ts: the keyless free probe moves past a busy upstream (503/529) instead of failing the pull request.
  • test/support/keyless-matrix.ts: the free probe's dated record names the model and status of the candidate that decided it.
  • docs/redirect-planes.md: §13 now defines a busy upstream followed by no answer as transport-error, which is what the code does.
  • docs/redirect-planes.md: the root shape is now described as the base the catalog declared until its 2026-10-16 regeneration. The same correction is in test/support/keyless-matrix.ts and CONTRIBUTING.md.
  • test/endpoint-family.test.ts: comments no longer say P-R2 already measured the families. §13 is pending the first CI run, and test/support/keyless-matrix.ts now says so too.
  • docs/improvement-plan.md: legacy (†) cells are "asserted nowhere live, except Go's Google".
  • CONTRIBUTING.md: the key and mirror bullets now cover the keyed probes.
  • docs/improvement-plan.md: the claim that P-R2 opens with a green pin is replaced with how its pins were checked on the base.
  • docs/improvement-plan.md: the conditional flip of the default is now its own phase, P-R2b.
  • docs/improvement-plan.md: the keyed probes decide a plane from the default cells read beside their siblings, not from those cells "alone". An unreadable answer is never flip evidence.
  • test/keyed-probes.test.ts: a test for the tier wiring of the four keyed live suites.
  • test/keyed-probes.test.ts: a test that no probe follows a redirect.
  • test/keyed-probes.test.ts: a test for the CI wiring's push-to-none arm and the steps that publish and upload the dated record.
  • test/keyed-probes.test.ts: a test that each suite rewrites its record after every cell.
  • test/keyless-matrix.test.ts: tests for the free probe moving past a busy upstream.

Declined

  • No upheld finding was declined. One finding in round 1 and one in round 2 were not upheld when verified.
  • Two narrower choices within the round 3 fixes:
    • Every account answer is treated alike, with no special case for "Model access is disabled".
    • key-unmeasured stays a kind of its own instead of merging into unmeasured, because it is computed from every sibling that read the key, not from the flip target.

Check, unit suite, mock suite, build and bundle gate all pass on the commit.

Commit: 3ccd70c


Generated by Claude Code

claude added 10 commits October 27, 2026 13:21
Ported from the stack's base (d13cbe2 on #105) so this pull request's
own `catalog freshness` goes green now, rather than waiting for the base
to be merged up. It no-ops once the merge from P-R1 brings the same
change.

It takes `src/catalog-data.ts` exactly as the nightly refresh produced
it (#111). The counts that restate the catalog's size follow it, on this
branch's own wording of those passages:

- the Zen shim count in catalog.test.ts (82 -> 81);
- redirect-planes.md: Zen served 82 -> 81, the no-SDK Zen cell 25 -> 24,
  and "0 of 82" -> "0 of 81";
- protocol-routing-and-merge.md: the per-protocol shares now read "of 81",
  with completions at 24;
- both READMEs: the no-SDK row is 24, and the active total is 81.

"127 listed" is unchanged, because a model the vendor retires stays
listed as `status: "deprecated"`.

(cherry picked from commit d13cbe2)

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_0141fKp5k6hiaJcP5ovhGtSD
…r1' into claude/vigilant-mendel-bctz9n-p-r2

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_0141fKp5k6hiaJcP5ovhGtSD
…t-mendel-bctz9n-p-r2

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_0141fKp5k6hiaJcP5ovhGtSD
…r1' into claude/vigilant-mendel-bctz9n-p-r2

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_0141fKp5k6hiaJcP5ovhGtSD
main's coverage gate (#142) checks bases; this adds its per-request form
over the cells either endpoint setting dispatches to, with the bundled
catalog's models and each protocol's own path. The root shape is the
converse: nothing dispatches there, so a mock that routed it could hide a
turn sent to the wrong family, and every root cell must answer no_route.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_0141fKp5k6hiaJcP5ovhGtSD
…ins report

P-R2 measures before any code moves, so each lesson becomes a record, not
an assertion:

- a 429 FreeUsageLimitError is the free tier's usage cap, its own verdict:
  never retried, never read as served, a skip (L3); main's live-run
  verdicts read it as the tier closing, not a busy gateway;
- the System One family's three shapes on the Zen plane, keyless (L13);
- keyed variants of each plane's default completions cell: a minted
  x-opencode-request (L4), x-opencode-project as global, a folder name and
  a 40-hex digest (L5), and reasoning_effort "none" beside no effort (L10);
- every keyed 429's Retry-After (L9).

The record formatter, section 13 and the docs follow; P6 and P-R3 decide.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_0141fKp5k6hiaJcP5ovhGtSD
P-R1's derivation said the matrix did not probe the root family yet; on
P-R2 it does, so a default turn reaching a root cell would match it rather
than fall through as unmatched. The case now says so and asserts that no
default cell is a root one.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_0141fKp5k6hiaJcP5ovhGtSD
The pins handed the route the rc channel's own pi-ai default window and
output tokens, which reads as the plugin restating them. P-R1 removed that
(D17, D26) and its plane-model tests use stand-in values; the family a turn
is sent to does not depend on them, so these pins follow.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_0141fKp5k6hiaJcP5ovhGtSD
…h the P-R2 restack

The probe classifiers now read a refusal of type ModelDeprecated as a
withdrawn model at any status. The per-request mock-coverage port
requires each dispatched cell to reach its own protocol's endpoint. The
usage-cap verdict change now reaches the walk and the prose that
describe it.

- both-kept-1-F1 / resolutions-1-PR2-1 (fixed): one shared predicate,
  isModelDeprecated (status >= 400 and a word-bounded ModelDeprecated),
  read right after the redirect arm and before the 404, 410 and 400
  arms in classifyKeylessProbe, classifyFreeProbe and
  classifyKeyedProbe. A retired free model now moves the free probe on
  instead of failing it. A keyed 404 or 400 of that type is no longer
  path-gone or path-exists. Classifier rows (400, 404 and 410 gone, a
  ModelDeprecatedish negative, a 2xx naming the word) and a runFreeProbe
  move-on row were added. Each arm, the status guard and the word
  boundary were mutation-checked, and each mutant failed its rows.
- both-kept-1-F2 / resolutions-1-PR2-2 (fixed): a dispatched cell must
  be recorded with its own protocol and plane, and not as a method
  refusal; the root converse is unchanged. Three mutants turn the case
  red: narrowing the Google pattern, dropping the Messages pattern, and
  relabelling the completions endpoint. Under the old form the Google
  mutant left the case green.
- both-kept-1-F4 / rules-docs-1-R2-RD-1 (3) (fixed): verdict.ts gains
  freeUsageCapped and freeWalkGap. The keyed free-tier walk skips with
  a reason that names the usage cap, instead of reporting a session
  gate it did not observe. probe-verdict rows were added, and the cap
  and busy arms were mutation-checked.
- resolutions-1-PR2-3 / rules-docs-1-R2-RD-1 (1, 2) (fixed): the
  accessDisabled JSDoc reads a 429 as load or the usage cap.
  docs/testing.md rule 1 names the cap as a closed tier and a busy
  gateway as any other 429.
- both-kept-1-F3 / rules-docs-1-R2-RD-2 (fixed): the plan's tier table
  and matrix line and ci.yml's tier comment name the System One record,
  the six keyed variants and Retry-After, and the free probe's usage-cap
  skip.
- rules-docs-1-R2-RD-3 (fixed): the plan credits P-R1 with the turn
  pre-filter and P-R2 with the probe classifiers, in both its record
  and its restack row. The section 13 legends and CONTRIBUTING name the
  type.
- rules-docs-1-R2-RD-4, rules-docs-1-R2-RD-5 (declined here): both are
  inherited from main and touch only the README pair, which this
  test-only phase does not change. They are recorded as a follow-up for
  a docs PR to main or the lowest open branch.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_0141fKp5k6hiaJcP5ovhGtSD
…rose findings

- both-kept-2-PR2-R3-1, rules-docs-2-R2-RD-N1: the plan's restack bullet now
  states the tested rule, that the free walk skips on the cap when ANY
  candidate met it and none met a busy gateway (busy, then capped, then
  closed), not when every candidate did.
- resolutions-2-R2-1: freeWalkGap keeps its "any" rule; the capped skip's
  reason now comes from freeWalkCappedReason, which also names how many
  candidates answered 403 FreeTierError, so a mixed walk still prints the
  session gate. The probe-verdict row for it was written first and failed
  without the helper. The freeUsageCapped JSDoc and the test comment say
  "a walk that met the cap", matching the rule.
- both-kept-2-PR2-R3-2: the combined Google User-Agent item is closed
  offline, not left open: the fetch patch replaces the User-Agent on the
  request every protocol sends, and the mock pins Google's wire value to
  OPENCODE_UA on both planes. redirect-planes.md section 9 says so, and the
  plan's restack bullet records it, reopening only if P10a moves the agent
  into call options.
- both-kept-2-PR2-R3-3, rules-docs-2-R2-RD-N2: the stray "* *" in the
  keyed-probes module doc is a paragraph break again.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_0141fKp5k6hiaJcP5ovhGtSD

viztor commented Nov 4, 2026

Copy link
Copy Markdown
Owner Author

Restacked onto main 1.8.0. This branch now carries main 1.8.0 through its parent, #113 (P-R1), which merged main. P-R2 merged P-R1. Pushed head: 24fcf009b840d126dbe9dc7f0b489c68939550e0.

Conflicts and how they were resolved

Source

  • src/utils/plane-model.ts: kept P-R1's D16/D26 body. baseUrlFor takes no pi argument, and legacy reads the fixed table. Added P-R2's AUTO_ENDPOINT_FAMILY on top, and auto resolves through it. No pi base or protocol came back.

Test support

  • test/support/host.ts: kept P-R1's per-channel PI_AI_DEFAULTS and S0_TODAY_BY_CELL/HostCell wording. Reworded the BASELINE exception to P-R2b flipping AUTO_ENDPOINT_FAMILY on §13's evidence.
  • test/support/src.ts: kept P-R1's removal of usesCatalogThinkingLevels/ReasoningSpec. Added AUTO_ENDPOINT_FAMILY and EndpointChoice to the door.
  • test/support/keyless-matrix.ts: TODAY_DISPATCH is now documented as holding on every host cell, built from the plugin's own endpoint table (D26). P-R2's "LEGACY_DISPATCH recorded, not guarded" decision replaces P-R1's "P-R2 decides" note.
  • test/support/live-probe.ts: kept P-R1's ModelDeprecated gone rule. The identity user agent is OPENCODE_UA, the turn's own.

Tests and e2e

  • test/keyless-matrix.test.ts: merged P-R1's per-host-cell derivation with P-R2's three-shape MATRIX_FAMILIES. A follow-up asserts that no default cell is a root cell.
  • test/e2e/keyless-matrix.e2e.ts: kept P-R2's helpers and free-model probe, with P-R1's failure message. The timeout covers 24 cells.
  • test/e2e/protocol-routing.e2e.ts and test/e2e/dispatch-levels.e2e.ts: kept P-R1's imports and its feature-detected level case, and added P-R2's full-tier key arming.

CI

  • .github/workflows/ci.yml: kept P-R2's tier step, record publishing and 30-minute timeout. Kept P-R1's two-channel e2e-host job, with its patch-free seam regex written as P-R2's here-string grep.

Docs

  • AGENTS.md: kept P-R1's two-channel host-versions line and its utils line. Added P-R2's keyless-matrix, keyed-probes and endpoint-record lines.
  • CONTRIBUTING.md: kept P-R1's text. Added P-R2's tiers, record directory, keyed-probe text and three-shape matrix.
  • docs/improvement-plan.md: kept P-R1's D14''/D22''/D26 and its two-channel status. Added P-R2's phase text and status row, the P-R2b row and the restack bullet with the L3/L4/L5/L9/L10/L13 records.
  • docs/redirect-planes.md: kept P-R2's §13 against P-R1's routing prose. Added the free-usage-limited legend, plus the System One, keyed-variant and Retry-After records.

Follow-ups added in this restack

  • The mock-coverage gate, per request. This is main's test(mock): the mock must serve every base the plugin can produce #142 gate, ported over TODAY_DISPATCH ∪ LEGACY_DISPATCH with the bundled catalog's models. Every root cell must answer no_route.
  • The step-9 records:
    • L3: a 429 FreeUsageLimitError is its own verdict, free-usage-limited, and is never retried.
    • L13: System One's three shapes.
    • L4/L5/L10: the keyed variants.
    • L9: Retry-After on every keyed 429.
  • No default cell is a root cell. This is now asserted.
  • test/endpoint-family.test.ts uses stand-in window and output values. It no longer uses the rc channel's pi-ai defaults (D17, D26).

Dropped

  • ModelDeprecated as a P-R2 gone rule. P-R1 already brought it.
  • "Probe the root /inference/v1" (L3). P-R2 already had the root shape. Only System One's root URL is new.
  • The verdict name free-tier-gated for the usage cap. That name already means the 403 session gate, so the cap is free-usage-limited.
  • D14' (patched pi 1.3.0) and the patched alpha gate cell. The no-patch, two-channel decision supersedes both.
  • The combined Google User-Agent re-measure. It was closed offline: the fetch patch replaces the User-Agent on every protocol's request, and the mock suite pins Google's wire value to OPENCODE_UA on both planes.

Review rounds

Round Found Upheld Fixed
1 12 12 12 (round 2)
2 6 6 6 (round 3)
3 0 0 none needed

Round 1 found:

  • classifiers missing the ModelDeprecated rule;
  • the per-request mock gate counting unknown_endpoint as served;
  • docs and report text still treating every 429 as a busy gateway;
  • a misattributed restack row;
  • README-pair gaps inherited from main.

Round 2 found:

  • the plan's "every candidate" wording contradicting the tested "any candidate" usage-cap rule;
  • the Google User-Agent item having no record;
  • a stray * * in a module doc.

For the round-2 fixes:

  • freeWalkGap keeps its rule.
  • A new pure helper, freeWalkCappedReason, still prints the session gate when a walk met both the gate and the cap. Its test was written first and failed without the helper.

Live re-measures still owed

  1. The first CI e2e run on this PR (probes tier). Paste its keyless and keyed tables into §13 (P-R2b). Flip AUTO_ENDPOINT_FAMILY only if a modern cell shows flip evidence.
  2. Record from that run:
    • the free probe on the current gate (served, free-tier-gated or free-usage-limited);
    • the root /inference(/go)/v1 cells;
    • System One's three shapes;
    • x-opencode-request present vs absent (L4);
    • x-opencode-project as global, folder or 40-hex (L5);
    • Retry-After on Go's 429s (L9);
    • reasoning_effort none vs absent (L10).
  3. For the nightly full run: whether the legacy /zen/v1 cells still answer for plane models under the current gate.
  4. Inherited from P-R1, for the alpha.3 bump and upstream, not this PR:
    • On the alpha npm cell, the route claim does not engage (KNOWN_DEFECT).
    • On every cell, no config-editor save succeeds (HOST_LIMITATION).

The dedicated reviewer was unavailable, so every review and verify pass in this run used the general model. That covers all three rounds of the resolutions, both-kept and rules-docs reviews, and both verify passes.


Generated by Claude Code

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants