Skip to content

bench(v5): measure the Flash disclosure tiers with the full denominator - #1024

Merged
Nicholas023 merged 10 commits into
mainfrom
card/v5-flash-comparison
Nov 20, 2026
Merged

Nicholas023 merged 10 commits into
mainfrom
card/v5-flash-comparison

Conversation

@Thatgfsj

Copy link
Copy Markdown
Collaborator

Closes #994 (parent #980).

What was wrong

#994 is a measurement card: the OPS-PTC disclosure tiers (on-demand vs full, implemented by #986/#987 and verified by the V3 matrix) had prompt-cost baselines but no behavioral evidence on a real model — the card asks whether the shorter first-round prompt trades for more calls or more failures, and demands the complete denominator (planned/completed/timed out/missing usage), never a bare pass rate.

A blocking finding shaped the driver: the host's one-shot headless runner cannot compose agent presets (its adoption check literally refuses: "session … runs under agent preset …, which the one-shot runner does not compose"), and the plugin treats a registry-less host realm as Standard — so a bench ab sweep would measure Standard and be invalid per #980. A smoke session on the shipped bench profiles confirmed it (session record agentPreset.val = null, native write/bash tools).

The change

  • bench/v5-run.mjs — the V5 driver: one fresh composed OPS-PTC group per session (the V1 isolated-entry composition: real AgentLoop, real PTC executor, real staged worker, the shipped dsh-ops row) with the real llm-pi-ai adapter on the frozen route opencode-go/deepseek-v4.1-flash/high. Arms differ only in the row's disclosurePolicy; AB/BA interleaved per (task, repeat); fresh seeded workspace per session; per-session archives (events, judge-ready trajectory, folded metrics, verified answer) plus per-arm dump-config.txt. The pi-ai adapter is vendored into node_modules at run time (gitignored, no manifest change) so one cordis instance serves the whole composition.
  • bench/v5-collect.mjs — the collector: judges archived trajectories with the V4 shared rules (scanReissues, countOpportunities) plus the two generalizing task verifiers, and prints the denominator, per-task tables, first-round vs post-load deltas and the A/A noise verdict. The scripted-only story rules (catalog's mandatory wrong call, load-order's mandatory not-loaded rejection) are deliberately not required of real runs.
  • bench/lib/tracks.mjs — export-only change: the two shared rules become importable (the V4 header already promised "V5 will feed real captured trajectories through the same functions").
  • docs/flash-comparison-v5.md + docs/Flash对照-V5.md — the bilingual report; docs/CHANGELOG.md — the Unreleased entry.

Stack note: this branch is main@58f6e28 plus the approved, unmerged prerequisites merged in (V1 entry + V4 tracks card/v4-bench-tasks@ec4d25c via PR #1014's stack; the tested D2/D3 disclosure implementation card/v3-two-tier-matrix@c68fe3d via PR #1020's stack). The V5-specific delta is the single commit bench(v5): measure the Flash disclosure tiers with the full denominator; the measurement ran at plugin commit 28c0cf7 (the merge tip), recorded in every environment.json.

Validation

Real model runs (all on linux-x64, host 0.2.2-rc.1, staged worker sha256 2b5f988d9115…, preset ops-ptc, sandbox danger-full-access):

sweep planned completed timed out missing usage verify pass
comparison (on-demand vs full) 12 12 0 0 11
A/A (on-demand twice) 12 12 0 0 12
total 24 24 0 0 23

Per-task key numbers (medians over 3 sessions per arm; full tables in the docs):

task metric on-demand full A/A band
ptc-catalog-recover verify pass 3/3 3/3 3/3–3/3
ptc-catalog-recover correct next steps 26 17 23–24
ptc-catalog-recover correction opportunities 3 0 0–1
ptc-catalog-recover no-progress retries 5 4 5–7
ptc-catalog-recover first-round prompt 5,271 29,946 5,271–5,271
ptc-catalog-recover prompt total 86,136 185,465 66,881–88,078
ptc-wait-dispatch verify pass 2/3 3/3 3/3–3/3
ptc-wait-dispatch correct next steps 8 8 14–27
ptc-wait-dispatch no-progress retries 7 6 7–9
ptc-wait-dispatch first-round prompt 5,306 29,981 5,306–5,306
ptc-wait-dispatch prompt total 137,419 236,416 85,813–103,238

First round vs after the completed load (on-demand): 5,271 → 12,820 (catalog) and 5,306 → 15,574 (wait-dispatch); the full arm has no load step by design. Cache: every step reported cacheReadTokens, cacheWriteTokens 0 throughout, missing-usage rows 0, reasoningTokens 0 throughout (recorded as-is).

Noise verdict: 未能区分 / not distinguished. Pass rates, retries and opportunities all sit inside the A/A bands at n=3; the only differences far outside the A/A band are the tier's own input-side prompt cost (first round 5.3k vs 30.0k, totals ~2x, cacheRead 2.4x) and the mechanically-invited loader correction on the on-demand arm (3 opportunities, all recovered, 3/3 pass). No model-quality attribution is made; one failing session (wrong config layer picked) is a reasoning miss within n=3 noise.

Reproduction:

npm run build && npm run build:ptc
node bench/v5-run.mjs --task ptc-catalog-recover --task ptc-wait-dispatch --repeat 3 --out bench/runs/v5-comparison --dsh-home <home with OPENCODE_GO_API_KEY>
node bench/v5-run.mjs --task ptc-catalog-recover --task ptc-wait-dispatch --repeat 3 --aa --out bench/runs/v5-aa --dsh-home <same>
node bench/v5-collect.mjs --run bench/runs/v5-comparison --run bench/runs/v5-aa

Command output from this run:

node bench/cli.mjs tracks
recovered 4 of 9 tracks that had a correction opportunity
tracks: 10 judged, 10 matched the declared expectation

DSH_HOME=/tmp/v5-gate-home node test/run.mjs bench-tracks
  0.09s bench-tracks.test.mjs
dsh-ops test: 1 suite(s) passed in 0s

npm run check:size
check-file-size: 832 code file(s) checked, all at or under 1000 lines

node bench/v5-run.mjs --task ptc-catalog-recover --task ptc-wait-dispatch --repeat 3 --out bench/runs/v5-comparison ...
[v5] sweep complete: 12 sessions, 0 with session errors
node bench/v5-run.mjs ... --aa --out bench/runs/v5-aa
[v5] sweep complete: 12 sessions, 0 with session errors

Raw artifacts: bench/runs/v5-comparison, bench/runs/v5-aa, smoke at bench/runs/v5-smoke (gitignored paths, listed in the docs).

CI is deliberately not triggered on this PR per the operator's convention (CI runs at the build stage, not per PR). The size gate and the touched suites are green locally as shown above.

Lock defaults, global dynamicTools:false, row dynamicTools:false, that row plus alwaysLoaded:[stat] and residentTools:false as exact first-round sets, with the old-off-versus-full counterexample and input-sensitivity checks (#985).
Add the bilingual 2026-11-19 design decision pair with the five-input truth table, the target full-selection row marked unimplemented, and the result exception, global switch and alwaysLoaded-cap conflicts recorded as maintainer sign-off items; cross-link from the configuration reference and record the Unreleased entry (#985).
bootEntryGroup mounts the shipped single row on the real agent loop inside each suite's throwaway DSH_HOME and records the environment a measurement must name: plugin commit, host version, platform, the worker sha256 with its staged receipt, sandbox mode, the resolved row switches and the actual agentPreset. enterPtc runs one scripted provider request, proves the top-level transport is run_code before anything executes, and reports a group that lands on Standard, on a native executor or without an executable worker as invalid with its own reason and no comparable output. progressive-loop asserts all of this beside its existing loop checks.
Record four verified facts alongside the delivered truth table, changing no row or existing assertion: the custom integration path rejects alwaysLoaded outside allowedTools (options.ts:100) while built-in rows accept it silently; the settings-page discovery-entry copy versus the typed ops_load_sdk the PTC row actually publishes; the legal custom ptc-profile integration misdiagnosed PUBLICATION_UNAVAILABLE: ops_result (fix owned by #988); and the loader 1-5-per-call cap with the uncapped resident startup list. Add the two matching assertions to test/ptc-config-truth.test.mjs (13 checks).
Add the YAML-only disclosurePolicy key ('on-demand' | 'full', default
'on-demand' = the legacy path) to the ptc and ops-ptc rows: PTC config
validation, mode-row routing with a readable rejection on non-PTC rows, and
settings serialization that preserves an explicit choice and never writes
the key for the user. Resolve the D1 sign-off item on precedence
conservatively per the workflow contract: while global.dynamicTools is off
the full tier is not applied and the saved choice stays untouched.
Publication of the full tier remains the D3 slice; no publication
behaviour or settings UI changes.
Four bench/tasks samples whose prompts state only the business goal: paged
cursor continuation, catalog-error recovery, load-before-use and
wait/multi-dispatch. A new pagedLog fixture family puts the target on page 2
and derives its answer from the bytes. bench/lib/tracks.mjs judges a scripted
trajectory against each task's fixed facts (cursor advance in the next program
on the same path, complete-page stop, advertised-set recovery, load order,
revision-awaited edits) and names the fact that did not hold; verbatim re-issues
after a delivered or input-refused envelope are no-progress retries while
bounded-transient pauses count as real waits. The tracks CLI runs ten
should-pass / should-fail trajectories and exits non-zero on any disagreement;
the recovery rate counts only tracks that had a correction opportunity and says
no denominator instead of 100%. bench-tracks pins both states red first (#993).
Consume the D2-parsed disclosurePolicy in the builtin plan slice and the
PTC publication: an explicit 'full' row publishes its whole permitted
allowedTools pool plus result in the first round with no ops_load_sdk
(#987, #979 SS2). Resolve the D1 sign-off item 1 conservatively per
#979 SS3 -- and per the unified acceptance rule, which on-demand loading
of result forces anyway: the full first round publishes ops_result so
refs spilled this step stay consumable. Read-only still removes every
mutating API; an empty cap stays at resident + result; residentTools
false drops the resident surface but not the tier; duplicated names
register once; global.dynamicTools:false keeps its D2 veto end to end
and a row that also turns its loader off keeps the tier. On-demand
(default) behaviour is letter-identical. PtcConfig.disclosurePolicy
stays optional because custom integrations keep the legacy default
until #988; the builtin plan carries the parsed value.
@Nicholas023
Nicholas023 merged commit e928001 into main Nov 20, 2026
8 of 10 checks passed
@Thatgfsj
Thatgfsj deleted the card/v5-flash-comparison branch November 20, 2026 15:46
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

[OPS-PTC V5] 执行 Flash 小规模对照并报告完整分母

2 participants