From c68fe3d0e62e0e89b238c121899a30e60fd66231 Mon Sep 17 00:00:00 2001 From: Thatgfsj <171755367+Thatgfsj@users.noreply.github.com> Date: Thu, 19 Nov 2026 14:13:01 +0800 Subject: [PATCH] test(ptc): drive the two-tier publication rule as a data-driven matrix --- docs/CHANGELOG.md | 2 + docs/design-decisions-20261119.md | 77 ++++ ...6\241\345\206\263\347\255\226-20261119.md" | 59 +++ test/ptc-two-tier-matrix.test.mjs | 357 ++++++++++++++++++ 4 files changed, 495 insertions(+) create mode 100644 test/ptc-two-tier-matrix.test.mjs diff --git a/docs/CHANGELOG.md b/docs/CHANGELOG.md index ea2d24f3..10bc0aaa 100644 --- a/docs/CHANGELOG.md +++ b/docs/CHANGELOG.md @@ -7,6 +7,8 @@ this package adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0.htm ## [Unreleased] +- Turn the two-tier OPS-PTC publication rule into a data-driven acceptance matrix (#992): `test/ptc-two-tier-matrix.test.mjs` runs eight canonical rows (legacy default, the four legacy-off variants, on-demand first step, on-demand after loading, the explicit full first round, read-only, a narrow pool and a custom `profile: 'ptc'` integration) through one shared comparator across three views — the SDK text the model reads, the step admission the loop resolves and the `run_code` program bindings — plus a separately collected no-toolSet diagnostic view that is a registration superset, never an admission proof. Every row executes a real no-side-effect sentinel through the real worker (the read-only row's inspect runs under read-only) and write authority is asserted with a safe rejection naming the refused API. The drift property is itself asserted: removing any single expected name — or reading the diagnostic view where a step view is required — fails at least one cell with the missing/extra names printed, and the unified acceptance holds: the loaded business set equals the full first round on both publication paths. The official `read`/`glob`/`grep` trio is pinned as visible under both `hideOfficialTools` states while #961 is out of baseline, so its landing flips a reviewable pin. No publication behaviour changes. +- 把两档 OPS-PTC 发布规则做成数据驱动验收矩阵(#992):`test/ptc-two-tier-matrix.test.mjs` 用同一比较器跑八个规范行(旧缺省、四个旧关闭变体、按需首步、按需加载后、显式全量首轮、只读、窄池、自定义 `profile: 'ptc'` integration)的三个视角——模型读到的 SDK 文本、循环解析的当步准入、`run_code` 程序绑定——外加单列采集的无 toolSet 诊断视图(注册超集,绝非准入证据)。每行都经真实 worker 执行无副作用哨兵(只读行的 inspect 在只读沙箱下真跑),写权限以点名被拒 API 的安全拒绝断言。漂移性质本身被断言:删任一期望名或错读诊断视角,至少一格失败并打印缺失/多余名字;统一口径成立:按需加载完毕的业务集合与全量首轮在两条发布路径上一致。官方 `read`/`glob`/`grep` 三件在 #961 未入基线期间钉为两种 `hideOfficialTools` 状态下均可见,其合入将翻转可评审的钉子。发布行为不变。 - Publish the explicit full PTC disclosure tier (#987): with `disclosurePolicy: 'full'` the `ptc` and `ops-ptc` rows' first round now carries the whole permitted `allowedTools` pool plus `ops_result` (the D1 sign-off item 1 decision, taken conservatively per #979 §3 so refs spilled this step stay consumable — and forced by the unified rule, since on-demand can always load `result`) with no `ops_load_sdk`. The first program can call the permitted pool directly; read-only sessions still exclude every write API, an empty cap stays at resident + `result`, `residentTools:false` drops the resident surface but not the tier, duplicated names register once, `global.dynamicTools:false` keeps its veto (the row publishes its legacy surface letter for letter) and a row that also turns its loader off keeps the tier. `PtcConfig.disclosurePolicy` stays optional because custom integrations keep the legacy on-demand default until #988; the builtin plan carries the parsed value. On-demand (default) behaviour is letter-identical; the settings-page entry is #989. - 发布显式 PTC 全量披露档位(#987):`disclosurePolicy: 'full'` 的 `ptc` 与 `ops-ptc` 行首轮现在携带整个获准 `allowedTools` 池外加 `ops_result`(D1 签字项 1 的保守决定,依 #979 §3 让本步落盘的 ref 仍可消费——统一口径也逼出同一结论,因为按需档永远可加载 `result`),且无 `ops_load_sdk`。第一个程序即可直接调用获准池;只读会话仍排除全部写 API,空上限停在 常驻 + `result`,`residentTools:false` 丢常驻面但不丢档位,重复名只注册一次,`global.dynamicTools:false` 保持否决(该行逐字发布旧面),同时关掉 loader 的行保留档位。`PtcConfig.disclosurePolicy` 保持可选,因为自定义 integration 在 #988 之前维持旧 on-demand 缺省;内置计划携带解析值。按需档(缺省)行为逐字不变;设置页入口归 #989。 - Parse an explicit PTC disclosure policy without changing any legacy behaviour: the `ptc` and `ops-ptc` rows accept a YAML-only `disclosurePolicy` key, `'on-demand'` (default, the legacy progressive path) or `'full'`. Illegal values are errors naming the key and the received value or type, and non-PTC rows reject the key with a readable diagnostic. The choice survives read → save-unrelated-setting → read through the settings serializer, and no upgrade, settings save or `global.dynamicTools` flip ever writes or erases it. The D1 sign-off item on precedence is resolved conservatively: while `global.dynamicTools` is off the full tier is not applied — the row keeps its legacy resident/alwaysLoaded surface — and the saved choice re-applies when the switch returns on; the row-level `dynamicTools` keeps its loader-only meaning because the full tier publishes no loader. Publishing the full tier remains the D3 slice; no publication behaviour or settings UI changes (#986). diff --git a/docs/design-decisions-20261119.md b/docs/design-decisions-20261119.md index 4d6f3983..25770708 100644 --- a/docs/design-decisions-20261119.md +++ b/docs/design-decisions-20261119.md @@ -264,3 +264,80 @@ unchanged — ptc-config-truth 13/0/0, resident-switch 24/0/0, ptc-disclosure-po note, not a regression: `ptc.test.mjs`'s installed-runtime check asserts precedence over `DSH_OPS_WORKER_BIN`, so it runs without that env override (it fails identically on the base commit with the override set). + +## V3 addendum — the data-driven acceptance matrix (#992, 2026-11-19) + +Card [#992](https://github.com/T-Auto/dsh-ops/issues/992). This slice adds tests and +assertion helpers only: no publication behaviour changes. One new suite, +`test/ptc-two-tier-matrix.test.mjs` (16 checks), drives every configuration as data. + +**The matrix.** Eight canonical rows — legacy default, legacy off (the +`global.dynamicTools:false`, row-level `dynamicTools:false`, that row plus +`alwaysLoaded:[stat]`, and `residentTools:false` variants), on-demand first step, +on-demand after loading, the explicit full first round, read-only, a narrow pool, +and a custom `profile: 'ptc'` integration — each carrying its config inputs and the +exact business names it must publish. Every row is measured in three views with one +shared comparator: the SDK text the model reads (declared), the step admission the +loop resolves (admitted), and the `run_code` program bindings (callable). A fourth, +clearly separate view collects the no-toolSet diagnostic assembly: a host assembly +that omits the step tool set reads every registered definition, so it is a +registration superset, never an admission proof — the default row's diagnostic shows +all sixteen pool APIs plus inspect and the loader while its step admits exactly +inspect + loader. + +**Sentinels and safe rejections.** Every row executes a real no-side-effect sentinel +through the real worker (`ops_inspect_local_file`, `ops_stat`, or `ops_grep` over a +fixture file); the read-only row's inspect runs with the sandbox in read-only mode. +Write authority is asserted with a safe rejection: under read-only, +`ops_load_sdk({tools:['write_file']})` answers `INVALID_ARGUMENT` naming the refused +API with `effectState: 'none'` — no worker spawn, no side effect — and the +`ops_write_file` binding stays absent in all three step views while remaining visible +in the registration diagnostic. + +**The drift property is itself asserted.** Removing any single expected name from a +row's collected set fails exactly its cell and prints the culprit; inventing a name +fails as extra; and reading the diagnostic registry view where a step view is +required fails with the pool named. Demonstrated live for the PR: corrupting the +global-off row's expected set (as if the dynamic gate had been removed) fails with +`global-off:first/declared(ops) drifted: missing [ops_load_sdk]`, and restoring the +row returns the suite to green. + +**Unified acceptance.** With the same permission tier, the on-demand loaded business +set equals the full first round on both publication paths (preset and custom +integration) — the load set must include `result`, forced by the D3 sign-off item 1 +rule, so the comparison is exact including `ops_result`. A loaded API becomes +callable only in the next program; an unloaded or permission-revoked one never +pretends to be executable (`typeof` stays `'undefined'` in every row's probe). + +**#961 pinning.** [#961](https://github.com/T-Auto/dsh-ops/issues/961) (hide replaced +official file tools under PTC presets) is not in this baseline, so the matrix pins +today's truth: the official `read`/`glob`/`grep` trio is declared (in the `: ` +SDK shape, next to the `: {` ops shape) and admitted beside the ops surface under +both `global.hideOfficialTools` states — the switch is inert for PTC rows today, and +the `official-swap-off` row records it letter for letter. When #961 lands, these pins +go red for its on state and must be updated together with it; they exist so that flip +is a reviewable diff, not a silent drift. + +**Platform note.** The matrix is measured on Linux with the real worker, no Windows +simulation; POSIX rows keep the host shell, so `ops_bash` is absent from every +expected set. The Windows surface keeps its own suites on real Windows machines. + +**Current-state facts recorded along the way.** The custom integration path admits +its own `ops_status` diagnostics reader — admitted and callable beside the pool, but +never declared (it has no SDK projection) and not part of the two-tier kernel, so the +unified comparison excludes it. The loader description carries the configured cap +(`Configured raw names (current authority may narrow this pool): …`), not the +sandbox-narrowed one; a narrow pool narrows it correctly. The directory +recommendation text disappears once the configured pool is exhausted. Byte budgets +are V2 and real-model rounds are V5; the loader-less declaration half owned by +`test/ptc-declaration-surface.test.mjs` belongs to P5 and is untouched. + +**Verification.** `npm run build`; red first (the honest sequence): the first run +failed 5 rows on boot mechanics and 3 on wrong expectations — every correction +matched shipped behaviour or fixed the suite's own probe (the raw-name `tools.glob` +vs `ops_glob` binding typo, the lossless-JSON sentinel, the loader-less host set). +Green after: `DSH_OPS_WORKER_BIN=runtime/linux-x64/dsh-ops-worker` (sha256 2b5f988d) +`node test/run.mjs ptc-two-tier-matrix` → all 16 checks passed in ~11 s; regressions +unchanged: ptc-full-disclosure 8/0/0, ptc-declaration-surface 5/0/0, +ptc-config-truth 13/0/0, resident-switch 24/0/0; `npm run check:size` clean. + diff --git "a/docs/\350\256\276\350\256\241\345\206\263\347\255\226-20261119.md" "b/docs/\350\256\276\350\256\241\345\206\263\347\255\226-20261119.md" index 9662dd23..6ebfb9ea 100644 --- "a/docs/\350\256\276\350\256\241\345\206\263\347\255\226-20261119.md" +++ "b/docs/\350\256\276\350\256\241\345\206\263\347\255\226-20261119.md" @@ -219,3 +219,62 @@ ptc-disclosure-policy 8/0/0、settings-rows 14/0/0;worker 用 `DSH_OPS_WORKER_BIN=runtime/linux-x64/dsh-ops-worker`(sha256 2b5f988d)。环境备注, 非回归:`ptc.test.mjs` 的安装包解析检查断言优先于 `DSH_OPS_WORKER_BIN`,因此该项 在不设该环境变量时运行(在基准 commit 上设同样的变量,失败方式相同)。 + +## V3 附录——数据驱动验收矩阵(#992,2026-11-19) + +卡 [#992](https://github.com/T-Auto/dsh-ops/issues/992)。本切片只加测试与断言 +helper,发布行为不变。新套件 `test/ptc-two-tier-matrix.test.mjs`(16 项检查)把 +每行配置当作数据驱动。 + +**矩阵。** 八个规范行——旧缺省、旧关闭(`global.dynamicTools:false`、行级 +`dynamicTools:false`、该行加 `alwaysLoaded:[stat]`、`residentTools:false` 四个变体)、 +按需首步、按需加载后、显式全量首轮、只读、窄池、自定义 `profile: 'ptc'` +integration——每行写明输入与必须发布的精确业务名。每行用同一比较器测三个视角: +模型读到的 SDK 文本(声明)、循环解析的当步准入(准入)、`run_code` 程序绑定 +(可调)。第四个明确分列的视角采集无 toolSet 的诊断装配:省略当步工具集的宿主 +装配读出全部已注册定义,它是注册超集,绝非准入证据——缺省行的诊断视图显示十六 +个池 API 加 inspect 与 loader,而其当步只准入 inspect + loader。 + +**哨兵与安全拒绝。** 每行都通过真实 worker 执行一个无副作用哨兵(对固定文件跑 +`ops_inspect_local_file`、`ops_stat` 或 `ops_grep`);只读行的 inspect 在只读沙箱下 +真跑。写权限用安全拒绝断言:只读下 `ops_load_sdk({tools:['write_file']})` 回 +`INVALID_ARGUMENT` 并点名被拒 API,`effectState: 'none'`——不 spawn worker、无副作用 +——且 `ops_write_file` 绑定在三个当步视角全部缺席,仅在注册诊断视图中可见。 + +**漂移性质本身被断言。** 从任一行已采集的集合里删掉任何一个期望名,恰好那一格 +失败并打印缺失名;凭空捏造名以多余报红;在该读当步视角处读注册诊断视角,则以池 +名报红。为 PR 现场演示过:把 global-off 行的期望集改坏(如同动态门被删), +失败信息为 `global-off:first/declared(ops) drifted: missing [ops_load_sdk]`, +还原后套件恢复全绿。 + +**统一口径。** 同权限档下,按需加载完毕的业务集合与全量首轮在两条发布路径 +(预设与自定义 integration)上都一致——加载集必须包含 `result`(D3 签字项 1 规则 +逼出),因此比较含 `ops_result` 逐名精确。加载的 API 只在下一个程序可调;未加载 +或被撤销权限的 API 绝不假装可执行(每行探针里 `typeof` 恒为 `'undefined'`)。 + +**#961 现状钉法。** [#961](https://github.com/T-Auto/dsh-ops/issues/961)(PTC 预设下 +隐藏被替代的官方文件工具)不在本基线,矩阵钉住今天的实况:官方 +`read`/`glob`/`grep` 三件以 `: ` 的 SDK 形状(并列于 ops 的 `: {` 形状)声明并 +准入,与 ops 面同在,且两种 `global.hideOfficialTools` 状态完全一致——该开关今天 +对 PTC 行无效,`official-swap-off` 行逐字记录。#961 合入时其开状态的钉子变红, +必须与它一并更新;钉子存在的意义就是让这次翻转成为可评审的 diff,而非静默漂移。 + +**平台说明。** 矩阵在 Linux 上以真实 worker 实测,不做 Windows 模拟;POSIX 行保留 +宿主 shell,所有期望集都不含 `ops_bash`。Windows 面由真实 Windows 机器上的独立 +套件负责。 + +**顺带钉住的现状事实。** 自定义 integration 路径额外准入自带的 `ops_status` 诊断 +读取器——与池并列地被准入且可调,但从不被声明(无 SDK 投影),不属两档内核, +统一比较将其排除。loader 描述携带配置上限(`Configured raw names (current +authority may narrow this pool): …`)而非沙箱收窄后的集合;窄池时正确收窄。配置池 +耗尽后目录推荐文本正确消失。字节分账归 V2、真实模型轮次归 V5; +`test/ptc-declaration-surface.test.mjs` 的无 loader 声明半边归 P5,未触碰。 + +**验证。** `npm run build`;先红(如实记录):首跑 5 行因引导机制失败、3 行因期望 +集错误——每处修正要么对齐已发布行为,要么修套件自身的探针(原始名 `tools.glob` +与 `ops_glob` 绑定笔误、无损 JSON 哨兵、无 loader 行的宿主集合)。后绿: +`DSH_OPS_WORKER_BIN=runtime/linux-x64/dsh-ops-worker`(sha256 2b5f988d) +`node test/run.mjs ptc-two-tier-matrix` → 16 项全过(约 11 秒);回归不变: +ptc-full-disclosure 8/0/0、ptc-declaration-surface 5/0/0、ptc-config-truth 13/0/0、 +resident-switch 24/0/0;`npm run check:size` 干净。 + diff --git a/test/ptc-two-tier-matrix.test.mjs b/test/ptc-two-tier-matrix.test.mjs new file mode 100644 index 00000000..a87857a5 --- /dev/null +++ b/test/ptc-two-tier-matrix.test.mjs @@ -0,0 +1,357 @@ +/** The #992 data-driven acceptance matrix for the two-tier publication rule. + * + * One kernel, three projections, one comparison. Every row is data: the config + * inputs and the exact business names the row must publish in each view — the + * SDK text the model reads (declared), the step admission the loop resolves + * (admitted), and the run_code program bindings (callable). A fourth, clearly + * separate view collects the no-toolSet diagnostic assembly: a host assembly + * that omits the step tool set reads every registered definition, so it is a + * registration superset, never an admission proof. All four go through one + * comparator that prints the missing and the extra names, and the drift + * property is itself asserted: removing any single expected name — or reading + * the diagnostic view where a step view is required — fails at least one cell + * with the culprit named. + * + * Every row executes a real sentinel through the real worker (the read-only + * row's inspect runs with the sandbox in read-only mode), and write authority + * is asserted with a safe rejection: ops_load_sdk never spawns the worker for + * a pool-outside or permission-denied name, it answers INVALID_ARGUMENT. + * + * The official read/glob/grep trio is pinned as *visible* next to the ops + * surface: #961 (hide replaced official file tools under PTC presets) is not + * in this baseline, and its hiding implementation is deliberately not + * simulated here. The official-swap-off row records that the shipped + * global.hideOfficialTools switch is inert for PTC rows today; when #961 lands + * these two rows flip together with it. + * + * Platform note: this matrix is measured on Linux with the real worker, no + * Windows simulation. POSIX rows keep the host shell, so ops_bash is out of + * every expected set; the Windows surface has its own suites on real Windows. + * Out of scope by card: byte accounting (V2) and real-model rounds (V5); the + * loader-less declaration half owned by test/ptc-declaration-surface.test.mjs + * belongs to P5 and is not touched here. + */ +import { createScope, scopeOf, scopeParentOf } from '@deepseek-ai/dsh-scope' +import Subprocess from '@deepseek-ai/dsh-subprocess-local' +import Sandbox from '@deepseek-ai/dsh-sandbox-local' +import FileSystem from '@deepseek-ai/dsh-fs-local' +import NodePtc from '@deepseek-ai/dsh-ptc-runtime-node' +import fs from 'node:fs' +import path from 'node:path' +import { apply } from '../lib/index.js' +import { integration } from '../lib/integration/index.js' +import { bootHost, shutdown } from './lib/host.mjs' +import { prepareStep, executeCall, selectExecutor } from './lib/executor.mjs' +import { assert, test, report, wait, waitFor, withTempDir } from './lib/harness.mjs' +import { WORKER, requireWorker } from './lib/boundary.mjs' + +requireWorker('ptc-two-tier-matrix') + +const OFFICIAL_TRIO = ['glob', 'grep', 'read'] +/** run_code is the executor's own tool: it is not part of the registry views this matrix compares. */ +const HOST_STEP_TOOLS = [...OFFICIAL_TRIO] +const I = 'ops_inspect_local_file' +const L = 'ops_load_sdk' +/** The sixteen-API default pool: every file API but the resident inspect, plus batch and result. */ +const POOL16 = ['ops_batch', 'ops_copy', 'ops_diff', 'ops_edit_file', 'ops_glob', 'ops_grep', 'ops_move', + 'ops_read_docx', 'ops_read_files', 'ops_read_pdf', 'ops_read_table', 'ops_remove', 'ops_replace', + 'ops_result', 'ops_stat', 'ops_write_file'] + +let sequence = 0 + +/** One comparison for every cell of the matrix: exact set equality, culprits named. */ +function expectCell(rowId, view, expected, actual) { + const missing = expected.filter(name => !actual.includes(name)) + const extra = actual.filter(name => !expected.includes(name)) + if (missing.length || extra.length) + throw new Error(`matrix cell ${rowId}/${view} drifted: missing [${missing.join(', ')}], extra [${extra.join(', ')}]`) +} + +const sdkTextOf = assembly => assembly.sections.find(section => section.name === 'tools:sdk')?.text ?? '' +/** The ops_* names one assembly declares in tools:sdk (the ": {" shape). */ +const declaredOps = assembly => [...new Set([...sdkTextOf(assembly).matchAll(/^ {2}(ops_[a-z_]+): \{/gm)].map(match => match[1]))].sort() +/** The official trio declared in tools:sdk (the ": " shape; #961 is not in this baseline). */ +const declaredOfficial = assembly => OFFICIAL_TRIO.filter(name => new RegExp(`^ {2}${name}: `, 'm').test(sdkTextOf(assembly))) +/** A host assembly without the step tool set reads every registered definition; never an admission proof. */ +const diagnosticOps = async (ctx, agent) => { + const assembly = await ctx.get('systemPrompt').assemble({ scope: agent }) + return [...new Set([...sdkTextOf(assembly).matchAll(/^ {2}(ops_[a-z_]+): \{/gm)].map(match => match[1]))].sort() +} +const admittedNames = (ctx, agent) => ctx.get('tools').definitions(agent, agent.prepared.toolSet).map(tool => tool.name) +const admittedOps = names => names.filter(name => name.startsWith('ops_')).sort() +const admittedHost = names => names.filter(name => !name.startsWith('ops_')).sort() + +async function scoped(parent, key, parentKey) { + let result + await parent.plugin({ inject: ['tools'], apply(inner) { + result = createScope(inner, key, parentKey ? { parent: parentKey } : undefined).ctx + } }) + return result +} + +/** Boot one ops-ptc preset row exactly as the shipped settings compose it. */ +async function bootPresetRow(cwd, row) { + const sandboxMode = row.sandboxMode ?? 'danger-full-access' + const { ctx } = await bootHost({ withWorkingDirectory: true }) + await ctx.plugin(Subprocess); await ctx.plugin(Sandbox); await ctx.plugin(FileSystem); await ctx.plugin(NodePtc) + ctx.provide('sandboxPolicy', { resolve: ({ session } = {}) => ({ mode: session?.mode ?? sandboxMode, workspaceRoot: cwd, sessionId: session?.id }) }) + const presets = new Map(), agents = [] + ctx.provide('agents', { list: () => [...agents] }) + ctx.provide('agentPresets', { composedPreset: inner => presets.get(scopeParentOf(scopeOf(inner))) }) + const config = { global: row.global ?? {}, discovery: { binaryPath: WORKER, required: true } } + for (const id of ['minimal', 'standard', 'ptc', 'ops-ptc', 'cordis']) config[id] = { enabled: id === 'ops-ptc', + binaryPath: WORKER, required: true, ...(id === 'ops-ptc' ? row.switches : {}) } + await ctx.plugin({ name: 'matrix-preset-fixture', apply: inner => apply(inner, config) }) + const id = 'matrix-' + row.id + '-' + ++sequence, preset = await scoped(ctx, { preset: id }) + presets.set(scopeOf(preset), 'ops-ptc') + for (const name of OFFICIAL_TRIO) preset.tools.register({ name, description: name, + parameters: { type: 'object', properties: {} }, output: { schema: { type: 'object' }, render: () => [] }, execute: async () => ({}) }) + const agent = { id, session: { id, mode: sandboxMode, header: { cwd }, append() {}, read() { return [] } } } + agent.ctx = await scoped(preset, agent, scopeOf(preset)) + await agent.ctx.plugin({ inject: ['tools', 'systemPrompt'], apply: inner => { selectExecutor(inner, 'ptc') } }) + agents.push(agent) + await ctx.serial('agent/created', { agent }) + await advanceStep(ctx, agent) + return { ctx, agent } +} + +/** Boot one custom integration row: the plan slice integrations compose themselves (#988 keeps it on-demand). */ +async function bootIntegrationRow(cwd, row) { + const sandboxMode = row.sandboxMode ?? 'danger-full-access' + const { ctx } = await bootHost({ withWorkingDirectory: true }) + await ctx.plugin(Subprocess); await ctx.plugin(Sandbox); await ctx.plugin(FileSystem); await ctx.plugin(NodePtc) + ctx.provide('sandboxPolicy', { resolve: ({ session } = {}) => ({ mode: session?.mode ?? sandboxMode, workspaceRoot: cwd, sessionId: session?.id }) }) + ctx.provide('agentPresets', { composedPreset: () => undefined }) + const agents = [] + ctx.provide('agents', { list: () => [...agents] }) + await ctx.plugin(integration({ preset: row.preset, profile: 'ptc', binaryPath: WORKER, required: true, ...row.switches })) + const id = row.preset + '-' + ++sequence + const agent = { id, session: { id, mode: sandboxMode, header: { cwd, agentPreset: row.preset }, append() {}, read() { return [] } } } + await ctx.plugin({ inject: ['tools'], apply(inner) { agent.ctx = createScope(inner, agent).ctx } }) + await agent.ctx.plugin({ inject: ['tools', 'systemPrompt'], apply: inner => { selectExecutor(inner, 'ptc') } }) + agents.push(agent) + await ctx.serial('agent/created', { agent }) + await advanceStep(ctx, agent) + return { ctx, agent } +} + +async function advanceStep(ctx, agent) { + ctx.emit('session/event', agent.session, { type: 'step/end', turn: ++sequence, step: ++sequence }) + agent.prepared?.dispose() + agent.prepared = await prepareStep(ctx, agent) +} + +/** One real run_code program through the prepared step, as the loop issues it. */ +async function program(ctx, agent, code, label) { + const result = await executeCall(ctx, { agent, toolSet: agent.prepared.toolSet, name: 'run_code', + arguments: { description: label, code }, callId: 'matrix-' + ++sequence, signal: new AbortController().signal }) + assert.equal(result.isError, false, JSON.stringify(result)) + return result.value.result +} + +/** The no-side-effect sentinel each row really executes through the worker. */ +function sentinelCode(spec) { + if (spec === 'inspect') return { code: "const sentinel = await tools.ops_inspect_local_file({ path: 'fixture.txt', limit: 2 })", + checks: value => { assert.equal(value.sentinelStatus, 'completed', JSON.stringify(value)) + assert.equal(value.sentinelText, true, 'the real worker answered the inspect call') } } + if (spec === 'grep') return { code: "const sentinel = await tools.ops_grep({ pattern: 'needle', path: '.' })", + checks: value => { assert.equal(value.sentinelStatus, 'completed', JSON.stringify(value)) + assert.equal(value.sentinelText, true, 'the real worker answered the grep call') } } + if (spec === 'stat') return { code: "const sentinel = await tools.ops_stat({ path: 'fixture.txt' })", + checks: value => { assert.equal(value.sentinelStatus, 'completed', JSON.stringify(value)) } } + if (spec === 'none') return { code: 'const sentinel = null', checks: () => {} } + throw new Error('unknown sentinel: ' + spec) +} + +/** Collect the four views of one settled state and drive them all through expectCell. */ +async function collectViews(ctx, agent, row, phase, evidence) { + const spec = row.expect[phase] + const names = admittedNames(ctx, agent) + const assembly = agent.prepared.assembly + expectCell(row.id + ':' + phase, 'declared(ops)', spec.ops, declaredOps(assembly)) + expectCell(row.id + ':' + phase, 'declared(official)', spec.official, declaredOfficial(assembly)) + expectCell(row.id + ':' + phase, 'admitted(ops)', [...spec.ops, ...(spec.admittedExtra ?? [])], admittedOps(names)) + expectCell(row.id + ':' + phase, 'admitted(host)', spec.host, admittedHost(names)) + // The diagnostic view is registration state: a load changes selection, not registration. + if (spec.diagnostic) expectCell(row.id + ':' + phase, 'diagnostic(ops)', spec.diagnostic, await diagnosticOps(ctx, agent)) + assert.equal(assembly.sections.some(section => section.text.includes('For on-demand APIs, use tools.ops_load_sdk')), + spec.directory, `${row.id}:${phase} directory recommendation text`) + if (spec.loaderDescription) { + const loader = ctx.get('tools').definitions(agent, agent.prepared.toolSet).find(tool => tool.name === L) + for (const word of spec.loaderDescription.includes) assert.ok(loader?.description.includes(word), `${row.id} loader description names ${word}`) + for (const word of spec.loaderDescription.excludes) assert.equal(loader?.description.includes(word), false, `${row.id} loader description must stay narrowed (${word})`) + } + // The callable view: one real program reports typeof for every probed name plus the sentinel. + const probed = [...new Set([...spec.callable, ...spec.absent])] + const sentinel = sentinelCode(row.sentinel[phase]) + const noSentinel = row.sentinel[phase] === 'none' + const reject = row.reject && phase === 'first' + ? `const rejection = await tools.ops_load_sdk({ tools: ${JSON.stringify([row.reject.tool])} })` : '' + const value = await program(ctx, agent, sentinel.code + '\n' + reject + '\nreturn { ' + + probed.map(name => `${name}: typeof tools.${name}, `).join('') + + (noSentinel ? '' : 'sentinelStatus: sentinel?.status, sentinelText: JSON.stringify(sentinel).includes(\'needle\'), ') + + (reject ? 'rejectStatus: rejection?.status, rejectCode: rejection?.error?.code, ' + + 'rejectMessage: rejection?.error?.message, rejectEffect: rejection?.error?.effectState, ' : '') + + ' }', 'Matrix probe ' + row.id + ':' + phase) + const callable = probed.filter(name => value[name] === 'function').sort() + expectCell(row.id + ':' + phase, 'callable', spec.callable, callable) + for (const name of spec.absent) assert.equal(value[name], 'undefined', `${row.id}:${phase} ${name} must not pretend to be executable`) + if (!noSentinel) sentinel.checks(value) + if (reject) { + assert.equal(value.rejectStatus, 'error', 'the safe rejection is a returned value, not a throw') + assert.equal(value.rejectCode, row.reject.code, JSON.stringify(value)) + assert.ok(value.rejectMessage.includes(row.reject.tool), 'the rejection names the refused API: ' + value.rejectMessage) + assert.equal(value.rejectEffect, 'none', 'a refused load has no effect') + } + evidence[row.id] = { ...(evidence[row.id] ?? {}), [phase]: { ops: admittedOps(names), diagnostic: await diagnosticOps(ctx, agent), + official: declaredOfficial(assembly), callable } } +} + +/** Run one matrix row end to end and stash its collected views for the synthesis checks. */ +async function runRow(row, evidence) { + return withTempDir('matrix-' + row.id, async cwd => { + fs.writeFileSync(path.join(cwd, 'fixture.txt'), 'needle in haystack\nplain second line\n') + const { ctx, agent } = row.kind === 'integration' ? await bootIntegrationRow(cwd, row) : await bootPresetRow(cwd, row) + try { + await waitFor(() => admittedOps(admittedNames(ctx, agent)).includes(row.anchor), { what: row.anchor + ' admitted', timeoutMs: 20_000 }) + await wait(120) + agent.prepared.dispose(); agent.prepared = await prepareStep(ctx, agent) + await collectViews(ctx, agent, row, 'first', evidence) + if (row.load) { + const value = await program(ctx, agent, `const receipt = await tools.ops_load_sdk({ tools: ${JSON.stringify(row.load)} })\n` + + `return { receiptStatus: receipt.status, early: typeof tools.ops_${row.load[0]} }`, 'Matrix load ' + row.id) + assert.equal(value.receiptStatus, 'completed', JSON.stringify(value)) + assert.equal(value.early, 'undefined', row.id + ': a loaded API is callable only in the next program') + await advanceStep(ctx, agent) + await wait(120) + await collectViews(ctx, agent, row, 'loaded', evidence) + } + } finally { await shutdown(ctx) } + }) +} + +// ---- The matrix rows. Eight canonical configurations (issue #992) with the +// legacy-off variants spelled out, each carrying inputs and expected names. ---- + +const ON_DEMAND_CAP = { ops: [I, L], host: [...HOST_STEP_TOOLS], official: [...OFFICIAL_TRIO], + diagnostic: [I, L, 'ops_glob', 'ops_grep', 'ops_result'], callable: [I, L], + absent: ['ops_glob', 'ops_grep', 'ops_result', 'ops_stat', 'ops_write_file', 'ops_batch'], directory: true } + +const ROWS = [ + { id: 'legacy-default', anchor: L, sentinel: { first: 'inspect' }, switches: {}, expect: { first: { + ...ON_DEMAND_CAP, diagnostic: [I, L, ...POOL16] } } }, + // #961 is out of baseline: the shipped hideOfficialTools switch is inert for PTC + // rows, so its off state publishes the identical surface. #961 flips this row. + { id: 'official-swap-off', anchor: L, sentinel: { first: 'inspect' }, global: { hideOfficialTools: false }, + expect: { first: { ...ON_DEMAND_CAP, diagnostic: [I, L, ...POOL16] } } }, + { id: 'global-off', anchor: I, sentinel: { first: 'inspect' }, global: { dynamicTools: false }, expect: { first: { + ops: [I], host: [...HOST_STEP_TOOLS], official: [...OFFICIAL_TRIO], diagnostic: [I], callable: [I], + absent: [L, ...POOL16], directory: false } } }, + { id: 'row-off', anchor: I, sentinel: { first: 'inspect' }, switches: { dynamicTools: false }, expect: { first: { + ops: [I], host: [...HOST_STEP_TOOLS], official: [...OFFICIAL_TRIO], diagnostic: [I], callable: [I], + absent: [L, ...POOL16], directory: false } } }, + { id: 'row-off-always', anchor: 'ops_stat', sentinel: { first: 'stat' }, switches: { dynamicTools: false, alwaysLoaded: ['stat'] }, expect: { first: { + ops: [I, 'ops_stat'], host: [...HOST_STEP_TOOLS], official: [...OFFICIAL_TRIO], diagnostic: [I, 'ops_stat'], + callable: [I, 'ops_stat'], absent: [L, ...POOL16.filter(name => name !== 'ops_stat')], directory: false } } }, + { id: 'resident-off', anchor: L, load: ['grep'], reject: { tool: 'inspect_local_file', code: 'INVALID_ARGUMENT' }, + sentinel: { first: 'none', loaded: 'grep' }, switches: { residentTools: false }, expect: { + first: { ops: [L], host: [...HOST_STEP_TOOLS], official: [...OFFICIAL_TRIO], diagnostic: [L, ...POOL16], + callable: [L], absent: [I, ...POOL16], directory: true }, + loaded: { ops: [L, 'ops_grep'], host: [...HOST_STEP_TOOLS], official: [...OFFICIAL_TRIO], + callable: [L, 'ops_grep'], absent: [I, 'ops_result', 'ops_stat', 'ops_glob'], directory: true } } }, + { id: 'on-demand-first', anchor: L, sentinel: { first: 'inspect' }, switches: { allowedTools: ['glob', 'grep'] }, + expect: { first: ON_DEMAND_CAP } }, + // The unified acceptance rule (#979 §3) forces result into the load set: the + // full first round publishes it, so the loaded business set equals it only + // when on-demand loads it explicitly, like any other pool member. + { id: 'on-demand-loaded', anchor: L, load: ['glob', 'grep', 'result'], sentinel: { first: 'inspect', loaded: 'grep' }, + switches: { allowedTools: ['glob', 'grep'] }, expect: { first: ON_DEMAND_CAP, + // The pool is exhausted after the load: the directory recommendation + // correctly disappears when no allowed name is left to load. + loaded: { ops: [I, L, 'ops_glob', 'ops_grep', 'ops_result'], host: [...HOST_STEP_TOOLS], official: [...OFFICIAL_TRIO], + callable: [I, L, 'ops_glob', 'ops_grep', 'ops_result'], absent: ['ops_stat', 'ops_write_file', 'ops_batch'], directory: false } } }, + { id: 'full-direct', anchor: 'ops_glob', sentinel: { first: 'grep' }, + switches: { disclosurePolicy: 'full', allowedTools: ['glob', 'grep'] }, expect: { first: { + ops: [I, 'ops_glob', 'ops_grep', 'ops_result'], host: [...HOST_STEP_TOOLS], official: [...OFFICIAL_TRIO], + diagnostic: [I, 'ops_glob', 'ops_grep', 'ops_result'], callable: [I, 'ops_glob', 'ops_grep', 'ops_result'], + absent: [L, 'ops_stat', 'ops_write_file', 'ops_batch'], directory: false } } }, + { id: 'read-only', anchor: L, sandboxMode: 'read-only', reject: { tool: 'write_file', code: 'INVALID_ARGUMENT' }, + sentinel: { first: 'inspect' }, switches: { allowedTools: ['write_file', 'grep'] }, expect: { first: { + ops: [I, L], host: [...HOST_STEP_TOOLS], official: [...OFFICIAL_TRIO], + diagnostic: [I, L, 'ops_grep', 'ops_result', 'ops_write_file'], callable: [I, L], + absent: ['ops_write_file', 'ops_grep', 'ops_result', 'ops_stat'], directory: true, + loaderDescription: { includes: ['write_file, grep, result'], excludes: [] } } } }, + { id: 'narrow-pool', anchor: L, reject: { tool: 'stat', code: 'INVALID_ARGUMENT' }, sentinel: { first: 'inspect' }, + switches: { allowedTools: ['grep'] }, expect: { first: { ...ON_DEMAND_CAP, + diagnostic: [I, L, 'ops_grep', 'ops_result'], + loaderDescription: { includes: ['grep, result'], excludes: ['stat', 'glob'] } } } }, + // The custom integration path also admits its own ops_status diagnostics reader, + // which has no SDK projection: it is admitted and callable but never declared. + { id: 'custom-preset', kind: 'integration', preset: 'matrix-custom', anchor: L, load: ['glob', 'grep', 'result'], + sentinel: { first: 'inspect', loaded: 'grep' }, switches: { allowedTools: ['glob', 'grep'] }, expect: { + first: { ops: [I, L], admittedExtra: ['ops_status'], host: [], official: [], diagnostic: [I, L, 'ops_glob', 'ops_grep', 'ops_result'], + callable: [I, L, 'ops_status'], absent: ['ops_glob', 'ops_grep', 'ops_result', 'ops_stat'], directory: true }, + loaded: { ops: [I, L, 'ops_glob', 'ops_grep', 'ops_result'], admittedExtra: ['ops_status'], host: [], official: [], + callable: [I, L, 'ops_glob', 'ops_grep', 'ops_result', 'ops_status'], absent: ['ops_stat', 'ops_write_file'], directory: false } } }, +] + +/** Collected views of every row, reused by the synthesis checks below. */ +const evidence = {} + +for (const row of ROWS) { + await test('matrix row ' + row.id + ': three views and the diagnostic view agree with the row', () => runRow(row, evidence)) +} + +await test('unified acceptance: the on-demand loaded business set equals the full first round on both publication paths', () => { + const business = evidence['on-demand-loaded'].loaded.ops.filter(name => name !== L) + const full = evidence['full-direct'].first.ops + // ops_status is the custom integration's own diagnostics reader, admitted + // beside the pool; it is not a member of the two-tier publication kernel. + const custom = evidence['custom-preset'].loaded.ops.filter(name => name !== L && name !== 'ops_status') + expectCell('unified-acceptance', 'preset-loaded-vs-full', full, business) + expectCell('unified-acceptance', 'custom-loaded-vs-full', full, custom) +}) + +await test('drift property: removing any one expected name fails exactly its cell and prints the culprit', () => { + const stepOps = evidence['on-demand-first'].first.ops + for (const name of stepOps) { + const corrupted = stepOps.filter(candidate => candidate !== name) + assert.throws(() => expectCell('on-demand-first', 'declared(ops)', corrupted, stepOps), + new RegExp('missing \\[.*' + name), name + ' removal must fail with the name printed') + } + assert.throws(() => expectCell('on-demand-first', 'declared(ops)', stepOps, [...stepOps, 'ops_nope']), + /extra \[.*ops_nope/, 'an invented name must fail as extra with the name printed') + assert.throws(() => expectCell('on-demand-first', 'declared(ops)', [...stepOps, 'ops_nope'], stepOps), + /missing \[.*ops_nope/, 'a lost expected name must fail as missing with the name printed') + const callable = evidence['on-demand-first'].first.callable + assert.throws(() => expectCell('on-demand-first', 'callable', callable.filter(name => name !== L), callable), + new RegExp('missing \\[.*' + L), 'deleting the loader admission fails the callable cell') +}) + +await test('drift property: reading the diagnostic registry view cannot satisfy a step cell', () => { + // The diagnostic assembly registers the whole pool while the step admits + // only inspect + loader, so a wrong-view read must fail with the pool named. + const step = evidence['on-demand-first'].first.ops + const diagnostic = evidence['legacy-default'].first.diagnostic + assert.throws(() => expectCell('on-demand-first', 'declared(ops)', step, diagnostic), + /extra \[.*ops_grep/, 'the pool registration must be reported as extra names') + assert.throws(() => expectCell('legacy-default', 'diagnostic(ops)', diagnostic, step), + /missing \[.*ops_grep/, 'the step view must be reported as missing names') +}) + +await test('the official trio stays visible next to the ops surface while #961 is out of baseline', () => { + // #961 (hide replaced official file tools under PTC presets) is not in this + // baseline, so today the trio is declared and admitted next to the ops + // surface under both hideOfficialTools states — the switch is inert for PTC + // rows. When #961 lands, these pins go red for its on state and must be + // updated together with it; they exist so that flip is a reviewable diff. + for (const id of ['legacy-default', 'official-swap-off']) { + const first = evidence[id].first + expectCell(id, 'declared(official)', OFFICIAL_TRIO, OFFICIAL_TRIO) + assert.deepEqual(first.ops, [I, L], id + ' publishes the resident surface plus the loader') + } + assert.deepEqual(evidence['legacy-default'].first.ops, evidence['official-swap-off'].first.ops, + 'the shipped hideOfficialTools switch is inert for PTC rows today') +}) + +report('ptc-two-tier-matrix')