Skip to content
Closed
Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
52 commits
Select commit Hold shift + click to select a range
995d753
fix(ptc): teach only the tool surface a scope publishes
Nicholas023 Nov 13, 2026
520695f
fix(ptc): stop calling wait_agent and multi-dispatch programs a singl…
Y-fyan Nov 14, 2026
29db05c
fix(ptc): hide replaced official file tools under PTC presets
Nicholas023 Nov 15, 2026
61f28e8
test(ptc): assert the five legacy config first-round OPS name sets
Thatgfsj Nov 19, 2026
4f1952f
docs(presets): record the legacy OPS-PTC truth table and sign-off items
Thatgfsj Nov 19, 2026
205cb94
test(ptc): add the isolated OPS-PTC entry and routing counter-examples
Thatgfsj Nov 19, 2026
d1d31c2
test(ptc): lock the PTC official-tool hiding fallback boundaries
Thatgfsj Nov 19, 2026
2ad0985
docs: record the PTC replacement fallback verification
Thatgfsj Nov 19, 2026
c7e5e34
fix(ptc): state exact paths and load order on narrow surfaces
Thatgfsj Nov 19, 2026
72bd6d9
test(ptc): pin the read-only loader-less lesson and notice live
Thatgfsj Nov 19, 2026
12e3ac5
docs(presets): append the D1 truth-table supplementary audit
Thatgfsj Nov 19, 2026
2fc5540
docs(reference): correct the PTC replacement scope statement
Thatgfsj Nov 19, 2026
5c40cb6
fix(ptc): render the SDK-loading prose only on a loader surface
Thatgfsj Nov 19, 2026
a277691
docs: state that loadable passes the loader's own gate
Thatgfsj Nov 19, 2026
c3f0fdd
feat(presets): parse an explicit PTC disclosure policy key
Thatgfsj Nov 19, 2026
9286d1d
test(ptc): add the four-account cost ledger over the isolated entry
Thatgfsj Nov 19, 2026
29f9cb8
test(ptc): pin cursor pagination and the wait latch cycle against the…
Thatgfsj Nov 19, 2026
ec4d25c
bench(tracks): script and judge the four OPS-PTC behavior tasks
Thatgfsj Nov 19, 2026
490a5e3
docs(cost): fix a stray parenthesis in the accounts entry
Thatgfsj Nov 19, 2026
4e0fcb7
docs(presets): mark the disclosure gating decision as revisitable
Thatgfsj Nov 19, 2026
1d15083
fix(ptc): retire guidance that names hidden official file tools
Thatgfsj Nov 19, 2026
b78fe65
docs(ptc): trim the first-round batching lesson to its rules and four…
Thatgfsj Nov 19, 2026
d0fcd6f
feat(publication): publish the explicit full PTC disclosure tier
Thatgfsj Nov 19, 2026
61744b1
fix(diagnostics): derive PTC expectations from the disclosure plan
Thatgfsj Nov 19, 2026
24f0111
test(ptc): assert executable next steps for resumable reads
Thatgfsj Nov 19, 2026
27af626
docs(diagnostics): record the D4 plan-aligned custom PTC integration
Thatgfsj Nov 19, 2026
52ca1fa
test(prompt-injection): reproduce resume duplication and compaction miss
Thatgfsj Nov 19, 2026
d78a387
feat(ptc): recommend executable next steps for resumable reads
Thatgfsj Nov 19, 2026
a1b4a9b
docs(ptc): describe executable nextAction recommendations
Thatgfsj Nov 19, 2026
c68fe3d
test(ptc): drive the two-tier publication rule as a data-driven matrix
Thatgfsj Nov 19, 2026
23e0397
fix(publication): classify unloaded, forbidden and revoked admissions…
Thatgfsj Nov 19, 2026
28c0cf7
Merge remote-tracking branch 'origin/card/v3-two-tier-matrix' into ca…
Thatgfsj Nov 19, 2026
22b4fbf
feat(prompt): distinguish a replayed lesson from a compacted one (#998)
Thatgfsj Nov 19, 2026
39afb0e
Merge commit 'a277691' into release/ops-ptc-18-cards
Thatgfsj Nov 19, 2026
3dd873d
Merge remote-tracking branch 'origin/card/p2-teaching-trim' into rele…
Thatgfsj Nov 19, 2026
573b067
Merge remote-tracking branch 'origin/card/v1-isolated-entry' into rel…
Thatgfsj Nov 19, 2026
c95a887
Merge remote-tracking branch 'origin/card/v2-cost-accounts' into rele…
Thatgfsj Nov 19, 2026
6105be6
Merge remote-tracking branch 'origin/card/v4-bench-tasks' into releas…
Thatgfsj Nov 19, 2026
943c962
Merge remote-tracking branch 'origin/card/v3-two-tier-matrix' into re…
Thatgfsj Nov 19, 2026
baacde9
Merge remote-tracking branch 'origin/card/d2-policy-parsing' into rel…
Thatgfsj Nov 19, 2026
e2ac1e4
Merge remote-tracking branch 'origin/card/d4-integration-diagnostics'…
Thatgfsj Nov 19, 2026
066270a
Merge commit '1d15083' into release/ops-ptc-18-cards
Thatgfsj Nov 19, 2026
3f00cfa
Merge commit '29f9cb8' into release/ops-ptc-18-cards
Thatgfsj Nov 19, 2026
5d56262
Merge remote-tracking branch 'origin/card/p6-outcome-nextsteps' into …
Thatgfsj Nov 19, 2026
22fda78
Merge remote-tracking branch 'origin/card/p5-declaration-admission' i…
Thatgfsj Nov 19, 2026
cfd993e
Merge remote-tracking branch 'origin/card/p7-teaching-dedupe' into re…
Thatgfsj Nov 19, 2026
e916d4e
feat(settings): let the user pick the PTC disclosure tier
Thatgfsj Nov 19, 2026
884bead
fix(integration): reconcile P7 teaching dedupe, P5 verdicts and the l…
Thatgfsj Nov 19, 2026
84a2f22
Merge remote-tracking branch 'origin/card/d5-settings-entry' into rel…
Thatgfsj Nov 19, 2026
316a16d
test(ptc): update the drift pins now that #961 hides the official trio
Thatgfsj Nov 19, 2026
8ce07c0
bench(v5): measure the Flash disclosure tiers with the full denominator
Thatgfsj Nov 19, 2026
e928001
Merge remote-tracking branch 'origin/card/v5-flash-comparison' into r…
Thatgfsj Nov 19, 2026
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
20 changes: 19 additions & 1 deletion bench/README.md
Original file line number Diff line number Diff line change
Expand Up @@ -108,7 +108,7 @@ The `index-ab` command compares the walk with the opt-in `cache_dir` path on one

## 任务一览

六个任务各自压一种能力,`ops_*` 工具名来自 `publicToolName()`。每个任务的期望值都由夹具生成器现算(`derive*`),判卷在 `bench/lib/verify.mjs`。
十个任务各自压一种能力,`ops_*` 工具名来自 `publicToolName()`。前六个只判 `answer.txt` 这类最终工件;后四个是 V4 行为任务(#993),除最终工件外还由轨迹判定器验证达成方式(见下节)。每个任务的期望值都由夹具生成器现算(`derive*`),工作区判卷在 `bench/lib/verify.mjs`。

| 任务 | 工作量 | 主要测什么 | 相关工具 |
| --- | --- | --- | --- |
Expand All @@ -118,3 +118,21 @@ The `index-ab` command compares the walk with the opt-in `cache_dir` path on one
| `csv-sum` | 8000 行 CSV,汇总 EU 区 `amount` 列 | 上下文控制:把聚合交给 shell 而不是把 8000 行读进上下文,`answer.txt` 只写整数和 | `ops_bash` 聚合;`ops_read_files` 只看表头确认列;`ops_write_file` 写答案 |
| `gate-repair-loop` | 12 个 stage 模块(每个约 70 行)+ `check.js` | 迭代修复循环:门禁停在第一个坏 stage,每个 stage 只错一处绑定常量、契约写在模块自己的注释里,跑→修→再跑直到退 0;`check.js` 字节不变是 invariant | `ops_bash` 跑门禁;`ops_read_files` 读 stage;`ops_edit_file` 按契约修常量;`ops_write_file` 写 `answer.txt` |
| `noisy-test-triage` | 2400 个测试、恰好一行 `not ok` | 精确提取:跑测试重定向到文件,在日志里只捞那一行失败测试名写入 `failure.txt`(必须精确匹配),且 runner 仍退 1 是 invariant | `ops_bash` 跑并重定向;`ops_grep` 只捞那一行(如 `^not ok`,可配 `outputMode: summary/content`);必要时 `ops_read_files`;`ops_write_file` 写结果 |
| `ptc-paged-lookup` | 600 行单文件 `logs/stream.log`,目标行在第 2 页 | 分页游标续读:partial 带 `nextOffset` 后必须在下一程序同路径续读,末页 complete 即停;原样重读、提前收尾都判 FAIL | `ops_inspect_local_file` 续读;`ops_write_file` 写答案 |
| `ptc-catalog-recover` | grepTree 语料,5 个文件含 `NEEDLE_CATALOG` | 目录错误纠偏:叫错工具名被拒后,下一步必须用当时可用集合里的替代工具,不许再追不存在的名字 | `ops_grep`(或 `ops_glob`/`ops_inspect_local_file`);`ops_write_file` 写答案 |
| `ptc-sdk-load-order` | grepTree 语料,3 个文件含 `NEEDLE_SDKLOAD` | 未加载工具:识别需先 `ops_load_sdk`,下一程序才可调;原样重试不加载判 FAIL | `ops_load_sdk` 加载 `grep`;下一程序 `ops_grep`;`ops_write_file` 写答案 |
| `ptc-wait-dispatch` | configTree 三层配置 + 8 个包级诱饵 | 等待/多派发:层文件并行读取一次派发、编辑要等待读取返回的 revision;锁忙(`bounded-transient`)后的原样重发是合理等待不是无进展重试;正常完成无须重试的反例必须保留 | `ops_inspect_local_file` 并读;`ops_edit_file` 带期望 revision;`ops_write_file` 写答案 |

## 行为轨迹判定 / Behavior tracks

`node bench/cli.mjs tracks`(V4,#993)不调用模型、不派生 worker:它把 `bench/lib/track-corpus.mjs` 里的十段脚本化轨迹(四个 V4 任务各配应过与应败)逐条交给 `bench/lib/tracks.mjs` 的判定器,应过轨迹必须判 PASS、应败轨迹必须判 FAIL,且失败输出指明是哪项事实不符;任何一条判定与声明不符,命令退 1。

恢复率只统计真的出现过纠偏机会(被拒 / 出错 / 冲突 / 带 `nextOffset` 的 partial)的轨迹:全量语料报 `4/9`;选中的轨迹全无纠偏机会时输出「no denominator」而不是 100%(可用 `--track ptc-wait-immediate-pass` 复现)。

复跑:`node bench/cli.mjs tracks`;按任务收窄 `--task <id>`;按轨迹收窄 `--track <id>`。红绿两步如实记录:先写 `test/bench-tracks.test.mjs` 断言并跑红(判定器与语料缺失,`ERR_MODULE_NOT_FOUND`),实现落地后 `node test/run.mjs bench-tracks` 全绿。V5 会用同一套判定器接真实模型的轨迹。

The `tracks` command (V4, #993) sends no model request and spawns no worker: it feeds the ten scripted trajectories in `bench/lib/track-corpus.mjs` (each V4 task carries a should-pass and a should-fail track) to the verifier in `bench/lib/tracks.mjs`; a should-pass track must judge PASS, a should-fail one must judge FAIL naming the fact that did not hold, and any disagreement exits 1.

The recovery rate counts only trajectories that actually had a correction opportunity (a rejection, an error, a conflict, or a partial page carrying `nextOffset`): the full corpus reports `4/9`; a selection with no opportunity at all prints an explicit no-denominator line instead of 100% (reproduce with `--track ptc-wait-immediate-pass`).

Rerun with `node bench/cli.mjs tracks`, narrow with `--task <id>` or `--track <id>`. The red step was recorded honestly first — the suite failed at the import because no verifier existed — and `node test/run.mjs bench-tracks` is green now. V5 feeds real captured trajectories through the same verifiers.
1 change: 1 addition & 0 deletions bench/cli.mjs
Original file line number Diff line number Diff line change
Expand Up @@ -34,6 +34,7 @@ export const COMMANDS = [
{ name: 'tools', slice: 'P13', target: 'bench/tools/index.mjs', summary: 'model-free deterministic tool-level comparison' },
{ name: 'calls', slice: 'P14', target: 'bench/calls/index.mjs', summary: 'host-level timing over a real corpus: official versus OPS, or two OPS builds' },
{ name: 'index-ab', slice: 'G4 (#614)', target: 'bench/index-ab/index.mjs', summary: 'index versus walk A/B on one corpus with the same worker binary' },
{ name: 'tracks', slice: 'V4 (#993)', target: 'bench/tracks.mjs', summary: 'judge the scripted OPS-PTC behavior tracks: paging, catalog, load order, wait' },
]

const AB_FLAGS_WITH_VALUE = ['--task', '--repeat', '--folder', '--base', '--cand']
Expand Down
8 changes: 5 additions & 3 deletions bench/lib/fixtures/index.mjs
Original file line number Diff line number Diff line change
Expand Up @@ -3,8 +3,8 @@
* name -> builder registry, and the derivation functions that recompute each
* builder's expectation from the bytes it wrote.
*
* The catalogue is split by family — `csv` / `log` / `tree` / `symbol` / `corpus`
* / `deep` / `repair` — but re-exported from this one path so the tasks, `run.mjs`
* The catalogue is split by family — `csv` / `log` / `tree` / `symbol` / `paged`
* / `corpus` / `deep` / `repair` — but re-exported from this one path so the tasks
* and the suites keep a single import. The scaffold itself lives in
* `./internal.mjs`, which imports nothing, so the graph is a DAG: barrel ->
* family -> scaffold. The shared names are still re-exported from here, because
Expand All @@ -20,6 +20,7 @@ import { symbolTree } from './symbol.mjs'
import { deepModule } from './deep.mjs'
import { deepCorpus } from './corpus.mjs'
import { brokenProject, gatePipeline } from './repair.mjs'
import { pagedLog } from './paged.mjs'

export { deriveLargeCsv, largeCsv } from './csv.mjs'
export { deriveLogStats, deriveNoisySuite, logStats, noisySuite } from './log.mjs'
Expand All @@ -28,6 +29,7 @@ export { deriveSymbolTree, symbolTree } from './symbol.mjs'
export { deepModule, deriveDeepModule } from './deep.mjs'
export { deepCorpus } from './corpus.mjs'
export { brokenProject, gatePipeline } from './repair.mjs'
export { derivePagedLog, pagedLog } from './paged.mjs'
export { drawIndices, hexRun, mulberry32, pad, pick, randInt, WORDS } from './internal.mjs'

/**
Expand Down Expand Up @@ -58,5 +60,5 @@ export function generate(name, params = {}) {
*/
export const GENERATORS = {
grepTree, renameTree, largeCsv, brokenProject, logStats, configTree,
deepModule, deepCorpus, noisySuite, symbolTree, gatePipeline,
deepModule, deepCorpus, noisySuite, symbolTree, gatePipeline, pagedLog,
}
63 changes: 63 additions & 0 deletions bench/lib/fixtures/paged.mjs
Original file line number Diff line number Diff line change
@@ -0,0 +1,63 @@
/**
* The `paged` fixture family: one long log file whose answer line sits beyond the
* first page, for the V4 paged-cursor behavior task (#993).
*
* The shape is the point, not the content: a 600-line file read at the default
* 200-line page size puts the marked line on page 2, so a trajectory must continue
* at the envelope's nextOffset instead of re-reading page 1 or stopping early.
* The marker token appears exactly once and every other payload is lowercase words
* plus lowercase hex, so a wrong guess cannot pass and the exported answer is
* recomputable from the bytes alone.
*
* @module dsh-ops/bench/fixtures/paged
*/

import { hexRun, mulberry32, pad, pick, WORDS } from './internal.mjs'

/**
* `pagedLog` — one log file with exactly one marked line at a chosen position.
* @param {{seed?: number, lines?: number, markerAt?: number}} params - the shape of the log.
* @returns {{files: {path: string, content: string}[], exports: {answer: string, markerLine: number, markerText: string, totalLines: number}}} the fixture.
*/
export function pagedLog(params = {}) {
const seed = params.seed ?? 41
const lines = Math.max(120, params.lines ?? 600)
const markerAt = params.markerAt ?? 350
if (!Number.isInteger(markerAt) || markerAt < 2 || markerAt > lines) {
throw new RangeError('markerAt must be an integer line index between 2 and lines')
}
const next = mulberry32(seed)
const levels = ['INFO', 'INFO', 'DEBUG', 'INFO', 'WARN']
const rows = []
const stamp = (index) => '2025-03-' + pad((index % 28) + 1, 2) + 'T' + pad(index % 24, 2) + ':' + pad(index % 60, 2) + ':' + pad((index * 7) % 60, 2) + 'Z'
for (let index = 1; index <= lines; index += 1) {
const payload = pick(next, WORDS) + ' ' + pick(next, WORDS) + ' ' + hexRun(next, 8).toLowerCase()
rows.push(pick(next, levels) + '|' + stamp(index) + '|' + payload)
}
const markerText = 'TARGET-' + hexRun(next, 8).toUpperCase() + ' ' + pick(next, WORDS) + ' ' + pick(next, WORDS)
rows[markerAt - 1] = 'WARN|' + stamp(markerAt) + '|' + markerText
const content = rows.join('\n') + '\n'
// The token must be unique in the bytes, or the exported answer would be a lie.
if (content.split('TARGET-').length !== 2) throw new Error('the TARGET marker must appear exactly once in the log')
return {
files: [{ path: 'logs/stream.log', content }],
exports: { answer: markerAt + ' ' + markerText, markerLine: markerAt, markerText, totalLines: lines },
}
}

/**
* Recompute `pagedLog`'s exports from the generated bytes.
* @param {{path: string, content: string}[]} files - the generated tree.
* @returns {object} the recomputed expectation.
* @throws {Error} when the log file is missing or the marker is not unique.
*/
export function derivePagedLog(files) {
const file = files.find((entry) => entry.path === 'logs/stream.log')
if (file === undefined) throw new Error('derivePagedLog: logs/stream.log is missing')
const rows = file.content.replace(/\n$/, '').split('\n')
const marked = rows.map((row, index) => (row.includes('TARGET-') ? index : -1)).filter((index) => index >= 0)
if (marked.length !== 1) throw new Error('derivePagedLog: the TARGET marker must appear exactly once, found ' + marked.length)
const markerLine = marked[0] + 1
const markerText = rows[marked[0]].split('|')[2]
return { answer: markerLine + ' ' + markerText, markerLine, markerText, totalLines: rows.length }
}
Loading
Loading