Repository navigation
bench(v5): measure the Flash disclosure tiers with the full denominator - #1024
Merged
Merged
Conversation
Lock defaults, global dynamicTools:false, row dynamicTools:false, that row plus alwaysLoaded:[stat] and residentTools:false as exact first-round sets, with the old-off-versus-full counterexample and input-sensitivity checks (#985).
Add the bilingual 2026-11-19 design decision pair with the five-input truth table, the target full-selection row marked unimplemented, and the result exception, global switch and alwaysLoaded-cap conflicts recorded as maintainer sign-off items; cross-link from the configuration reference and record the Unreleased entry (#985).
bootEntryGroup mounts the shipped single row on the real agent loop inside each suite's throwaway DSH_HOME and records the environment a measurement must name: plugin commit, host version, platform, the worker sha256 with its staged receipt, sandbox mode, the resolved row switches and the actual agentPreset. enterPtc runs one scripted provider request, proves the top-level transport is run_code before anything executes, and reports a group that lands on Standard, on a native executor or without an executable worker as invalid with its own reason and no comparable output. progressive-loop asserts all of this beside its existing loop checks.
Record four verified facts alongside the delivered truth table, changing no row or existing assertion: the custom integration path rejects alwaysLoaded outside allowedTools (options.ts:100) while built-in rows accept it silently; the settings-page discovery-entry copy versus the typed ops_load_sdk the PTC row actually publishes; the legal custom ptc-profile integration misdiagnosed PUBLICATION_UNAVAILABLE: ops_result (fix owned by #988); and the loader 1-5-per-call cap with the uncapped resident startup list. Add the two matching assertions to test/ptc-config-truth.test.mjs (13 checks).
Add the YAML-only disclosurePolicy key ('on-demand' | 'full', default
'on-demand' = the legacy path) to the ptc and ops-ptc rows: PTC config
validation, mode-row routing with a readable rejection on non-PTC rows, and
settings serialization that preserves an explicit choice and never writes
the key for the user. Resolve the D1 sign-off item on precedence
conservatively per the workflow contract: while global.dynamicTools is off
the full tier is not applied and the saved choice stays untouched.
Publication of the full tier remains the D3 slice; no publication
behaviour or settings UI changes.
Four bench/tasks samples whose prompts state only the business goal: paged cursor continuation, catalog-error recovery, load-before-use and wait/multi-dispatch. A new pagedLog fixture family puts the target on page 2 and derives its answer from the bytes. bench/lib/tracks.mjs judges a scripted trajectory against each task's fixed facts (cursor advance in the next program on the same path, complete-page stop, advertised-set recovery, load order, revision-awaited edits) and names the fact that did not hold; verbatim re-issues after a delivered or input-refused envelope are no-progress retries while bounded-transient pauses count as real waits. The tracks CLI runs ten should-pass / should-fail trajectories and exits non-zero on any disagreement; the recovery rate counts only tracks that had a correction opportunity and says no denominator instead of 100%. bench-tracks pins both states red first (#993).
Consume the D2-parsed disclosurePolicy in the builtin plan slice and the PTC publication: an explicit 'full' row publishes its whole permitted allowedTools pool plus result in the first round with no ops_load_sdk (#987, #979 SS2). Resolve the D1 sign-off item 1 conservatively per #979 SS3 -- and per the unified acceptance rule, which on-demand loading of result forces anyway: the full first round publishes ops_result so refs spilled this step stay consumable. Read-only still removes every mutating API; an empty cap stays at resident + result; residentTools false drops the resident surface but not the tier; duplicated names register once; global.dynamicTools:false keeps its D2 veto end to end and a row that also turns its loader off keeps the tier. On-demand (default) behaviour is letter-identical. PtcConfig.disclosurePolicy stays optional because custom integrations keep the legacy default until #988; the builtin plan carries the parsed value.
…rd/v5-flash-comparison
This was referenced Nov 19, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Closes #994 (parent #980).
What was wrong
#994 is a measurement card: the OPS-PTC disclosure tiers (on-demand vs full, implemented by #986/#987 and verified by the V3 matrix) had prompt-cost baselines but no behavioral evidence on a real model — the card asks whether the shorter first-round prompt trades for more calls or more failures, and demands the complete denominator (planned/completed/timed out/missing usage), never a bare pass rate.
A blocking finding shaped the driver: the host's one-shot headless runner cannot compose agent presets (its adoption check literally refuses: "session … runs under agent preset …, which the one-shot runner does not compose"), and the plugin treats a registry-less host realm as Standard — so a
bench absweep would measure Standard and be invalid per #980. A smoke session on the shipped bench profiles confirmed it (session recordagentPreset.val = null, nativewrite/bashtools).The change
bench/v5-run.mjs— the V5 driver: one fresh composed OPS-PTC group per session (the V1 isolated-entry composition: real AgentLoop, real PTC executor, real staged worker, the shippeddsh-opsrow) with the realllm-pi-aiadapter on the frozen routeopencode-go/deepseek-v4.1-flash/high. Arms differ only in the row'sdisclosurePolicy; AB/BA interleaved per (task, repeat); fresh seeded workspace per session; per-session archives (events, judge-ready trajectory, folded metrics, verified answer) plus per-armdump-config.txt. The pi-ai adapter is vendored intonode_modulesat run time (gitignored, no manifest change) so one cordis instance serves the whole composition.bench/v5-collect.mjs— the collector: judges archived trajectories with the V4 shared rules (scanReissues,countOpportunities) plus the two generalizing task verifiers, and prints the denominator, per-task tables, first-round vs post-load deltas and the A/A noise verdict. The scripted-only story rules (catalog's mandatory wrong call, load-order's mandatory not-loaded rejection) are deliberately not required of real runs.bench/lib/tracks.mjs— export-only change: the two shared rules become importable (the V4 header already promised "V5 will feed real captured trajectories through the same functions").docs/flash-comparison-v5.md+docs/Flash对照-V5.md— the bilingual report;docs/CHANGELOG.md— the Unreleased entry.Stack note: this branch is
main@58f6e28plus the approved, unmerged prerequisites merged in (V1 entry + V4 trackscard/v4-bench-tasks@ec4d25cvia PR #1014's stack; the tested D2/D3 disclosure implementationcard/v3-two-tier-matrix@c68fe3dvia PR #1020's stack). The V5-specific delta is the single commitbench(v5): measure the Flash disclosure tiers with the full denominator; the measurement ran at plugin commit28c0cf7(the merge tip), recorded in everyenvironment.json.Validation
Real model runs (all on linux-x64, host 0.2.2-rc.1, staged worker sha256
2b5f988d9115…, presetops-ptc, sandbox danger-full-access):Per-task key numbers (medians over 3 sessions per arm; full tables in the docs):
First round vs after the completed load (on-demand): 5,271 → 12,820 (catalog) and 5,306 → 15,574 (wait-dispatch); the full arm has no load step by design. Cache: every step reported
cacheReadTokens,cacheWriteTokens0 throughout, missing-usage rows 0,reasoningTokens0 throughout (recorded as-is).Noise verdict: 未能区分 / not distinguished. Pass rates, retries and opportunities all sit inside the A/A bands at n=3; the only differences far outside the A/A band are the tier's own input-side prompt cost (first round 5.3k vs 30.0k, totals ~2x, cacheRead 2.4x) and the mechanically-invited loader correction on the on-demand arm (3 opportunities, all recovered, 3/3 pass). No model-quality attribution is made; one failing session (wrong config layer picked) is a reasoning miss within n=3 noise.
Reproduction:
Command output from this run:
Raw artifacts:
bench/runs/v5-comparison,bench/runs/v5-aa, smoke atbench/runs/v5-smoke(gitignored paths, listed in the docs).CI is deliberately not triggered on this PR per the operator's convention (CI runs at the build stage, not per PR). The size gate and the touched suites are green locally as shown above.