Skip to content

fix(ptc): teach the session scope every PTC program shares - #869

Open
WeiLinCool wants to merge 1 commit into
T-Auto:masterfrom
WeiLinCool:fix/ptc-scope-guidance
Open

WeiLinCool wants to merge 1 commit into
T-Auto:masterfrom
WeiLinCool:fix/ptc-scope-guidance

Conversation

@WeiLinCool

@WeiLinCool WeiLinCool commented Nov 11, 2026 •

Copy link
Copy Markdown
Contributor

Closes #868

What was wrong

PTC can evaluate a program in a runtime that retains state, so a top-level const/let/class/function outlives the program that declared it. A second declaration of that name is a SyntaxError at compile time and the whole program is discarded before its first line runs, tool calls included. A program that throws keeps its names as well — bindings are instantiated before evaluation — so a failed program cannot be retried verbatim: the retry dies on a name, not on its arguments. The name is never reclaimable: there is no way to delete a lexical binding, and const is not a context property, so Object.keys cannot even enumerate it.

In the session that filed #868 (127 run_code programs) this was 11 of 17 failures (65% of failures, 8.7% of programs). Two were retries of a program that had already failed. Names declared on turn 1 (q1, e1) still collided on turn 6, and failed programs left ~70 permanently unusable names.

The shipped guidance taught the pattern: its Step 3 example is const [load, head] = await Promise.all([...]), and the model copied load into the next program, dying on program two.

The change

release/0.5.5 rewrote the same file. This branch is rebased onto the current master (which contains 0.5.5), so the diff against the 0.5.5 text is 15 lines:

  • ## One session, one scope — a top-level name can outlive its program, a second one is a compile-time error that discards the whole program, a failed program keeps its names, so each program owns its names and a retry renames.
  • Two mistake-table rows — the same top-level name in a second program; re-sending a failed program with new arguments.
  • BATCHING_GUIDANCE_REVISION → '5' (0.5.5 already used 4).

What this deliberately does not do: wrap the examples in { }. That was the first shape of this PR and it is incompatible with 0.5.5: test/batching-examples.test.mjs compiles and executes every shipped example through an AsyncFunction and asserts the returned summary, so an example has to return, and return cannot live inside a bare block. Switching the examples to console.log silently removed them from that test's coverage — I reverted it for that reason.

The stronger alternative, if maintainers prefer 0% over 5%: wrap each example body in return await (async () => { ... })(). It keeps the returned value, keeps batching-examples green, and makes the names local, at the cost of turning every example into an IIFE. Say the word and I will send it as a follow-up.

Validation

A. Model-driven A/B over one audit task in a retained REPL — same model, same task, same run_code contract, only --guidance differs, 4 runs per arm, 14-program budget:

guidance programs discarded by a redeclaration rate
0.5.5 text 56 7 12.5%
this branch 56 3 5.4%
$ node bench/probes/probe-scope-run.mjs --guidance /tmp/rev055.md --steps 14 --json
$ node bench/probes/probe-scope-run.mjs --guidance prompts/batching-guidance.md --steps 14 --json

B. Why the examples are not the vehicle — bench/probes/probe-scope-guidance.mjs runs every fenced example twice in one retained REPL. Both texts score 6 of 7 redeclared, because the examples are unchanged; that is the measurement that says this fix has to be the rule, not the shape.

C. Suite — npm run verify on this branch (darwin-arm64): 247 of 248 suites pass, 7 skipped. The one failure is preset-fragment, which compares the shipped presets/ops-ptc.patch.yml checksum against the locally installed host checkout; that checkout is at a newer commit (838665e0… vs the recorded ce02148b…), and this branch does not touch presets/. It passes in CI, where the host is installed from the pinned range.

D. CI required — green on the rebased head: https://github.com/T-Auto/dsh-ops/actions/runs/43458558329

node-floor              success  2m50s
source-size             success
rust-lint               success
workflow-lint           success
tools-trial             success
validate (linux-x64)    success  3m48s
validate (win32-x64)    success  7m40s
validate (darwin-x64)   success  6m55s
validate (darwin-arm64) success  3m29s
CI required             success

An earlier run of the same content was red twice for infrastructure reasons, not for this change: validate (darwin-arm64) was cancelled with zero steps and no runner assigned, and validate (darwin-x64) failed once on test/native-guidance.test.mjs:59, which waits for the Native reader to be published and then asserts the official read is already hidden — publication and hiding are two steps, so a loaded runner can lose that race. Both legs pass on the current head.


Update 2026-11-21 — rebased onto v0.5.6

The branch now sits on v0.5.6 (8545d446); the previous head (171b87d2) was based on the 18-card release. Diff against 0.5.6: 8 files, +184/−8. Where the text above disagrees, this section is the accurate one.

What 0.5.6 changed under this branch

  • It added ## Sandbox permissions in the same slot (between Core rules and Step 0) and took revision 8 for it. This branch keeps both sections — sandbox first, then ## One session, one scope — so nothing is displaced, and the revision becomes 9.
  • The lesson is now delivered as a plain user message committed to history, so the added text is not a one-time cost: the section adds 803 B (8,169 → 8,972 B), carried by every later request until compaction. It still fits the 0.5.6 bound of <9,000 characters.
  • test/batching-guidance.test.mjs gained a 9,000-character bound and a sandbox-section pin; this branch updates the revision/byte pins and adds a guard that a full surface carries both sections.

Two wording fixes from the review

  • The line "A failed program did nothing" was wrong: a program can run ops_write_file/ops_edit_file and then throw. It now says "A failed program may already have run calls: check their effects, then rename before retrying", and the mistake row says "Check what it already ran, then rename and retry".
  • Step 3 said "programs need not share a process or local variables", which contradicted the new section. It now separates the two facts: a later program "must not assume an earlier program's values survived, and must not re-declare a name the scope already holds".

Evidence status

  • The 127-program session figures stand: 11 of 17 failures were this error, two of them retries.
  • The A/B table above was measured on the pre-trim revision-5 text and is not claimed for revision 9.
  • The probe reports 5 examples / 4 not copyable twice on the 0.5.6 text and on this one: it documents the problem (the examples themselves are copy-unsafe), not the fix.

Validation on v0.5.6

$ npm run build
$ node test/run.mjs batching        # 4 suites passed (guidance 27 checks)
$ node test/run.mjs prompt-budget   # 1 check passed
$ node bench/probes/probe-scope-guidance.mjs prompts/batching-guidance.md
prompts/batching-guidance.md: 5 example(s), 4 not copyable twice

Wording that must survive the merge: none. The trim (#982) and the sandbox lesson both landed before this branch; the section is additive.

@Thatgfsj

Copy link
Copy Markdown
Collaborator

Coordination note from #982 (P2, staging lineage card/p1-lineage, based on a277691): I'm trimming the same file (prompts/batching-guidance.md) on that lineage — removing the anti-pattern section and the trailing prose that repeats Step 1, keeping the Step 3 load example and all four trajectories byte-identical. Nothing in this PR is deleted by that trim, so your "One session, one scope" section and the two mistake rows stay mergeable additively at integration time. Two integration notes: the revision constant will conflict (I bump 6→7 on the staging lineage; your branch targets master), and bench/probes/probe-scope-guidance.mjs walks the fenced examples, whose count drops once the anti-pattern demo's two blocks are gone, so the "6 of 7" denominator will change after both land. If any wording here must survive verbatim through that merge, say so and I'll leave room for it.

@WeiLinCool

Copy link
Copy Markdown
Contributor Author

Thanks — the trim landed on main as b78fe656 before integration time came around, so I rebased this branch onto c70633da and it is now purely additive.

On your two notes:

  • Revision: no conflict left to resolve at merge time. main is at 7; this patch takes it to 8.
  • Denominator: you were right, and it moved already. Against the revision-7 text, probe-scope-guidance.mjs reports 5 examples, 4 not copyable twice (load, dirs, hits, rows) — the anti-pattern demo's two blocks are the ones that went. The fifth is the verified-edit example, which uses top-level await; that plain-REPL harness cannot evaluate it, so it is neither counted nor exonerated.

Nothing in the section needs to survive a merge verbatim: it does not depend on any text the trim removed. The rebased basis, the updated diff (8 files, +178/−7) and the current probe numbers are in the update section at the bottom of the description.

A state-retaining PTC runtime keeps a program's top-level `const`, `let`,
`class` and `function` while the runtime lives, so a second declaration of that
name is a SyntaxError at compile time and the whole program is discarded before
its first line runs, tool calls included. A failed program may already have run
calls and keeps its names, so a retry has to check the effects and rename
before it can succeed.

Measured on one 127-program ops-ptc session: 11 of 17 failed programs were this
exact error, two of them retries of a program that had already failed, with
names declared on turn 1 still colliding on turn 6.

Four of the five shipped examples are still not safe to copy twice, because a
top-level `const` outlives the program that declared it
(bench/probes/probe-scope-guidance.mjs; the fifth uses top-level await, which
that probe cannot evaluate). The lesson adds one short section with the three
rules and the two matching rows in the mistake table, and Step 3 now separates
a name that is spent from a value that may not survive. Examples, steps, the
four trajectories and 0.5.6's sandbox section are unchanged.

Revision 9; no behaviour changes.
@WeiLinCool
WeiLinCool force-pushed the fix/ptc-scope-guidance branch from 171b87d to d4d8ecd Compare November 21, 2026 07:44
@WeiLinCool

Copy link
Copy Markdown
Contributor Author

Reviewed this branch against v0.5.6 (8545d446) and updated it in d4d8ecd4. Three blockers, all fixed; the branch is rebased onto 0.5.6, not merged with it.

  1. "A failed program did nothing" was wrong. A run_code program can write or edit a file and then throw, so telling the model it did nothing invites duplicate side effects on retry. It now reads "A failed program may already have run calls: check their effects, then rename before retrying", and the mistake row says "Check what it already ran, then rename and retry".

  2. Step 3 contradicted the new section. It said "programs need not share a process or local variables" while the section said declarations persist. The honest shape is asymmetric — names are spent while the scope lives, values may not survive a runtime restart — and Step 3 now states exactly that: a later program "must not assume an earlier program's values survived, and must not re-declare a name the scope already holds".

  3. Revision numbering. 0.5.6 owns revision 8 for the sandbox lesson, so this is revision 9; the byte pin moves to 8,972 and both configuration references say 9. It also fits the <9,000-character lesson bound 0.5.6 added — 28 characters to spare.

Two things worth stating rather than fixing:

  • The A/B in the description was measured on the revision-5 text and is not evidence for revision 9. What remains is the 127-program session data (11 of 17 failures) and the probe, which reports 5 examples / 4 not copyable twice on the 0.5.6 text as well — that documents the problem, not the effect of the section.
  • Since 0.5.6 delivers the lesson as a logged user message, the +803 B (8,169 → 8,972 B) is carried by every later request rather than paid once; the CHANGELOG entry records the size so the trade-off stays visible.

Validation on the rebased head: node test/run.mjs batching 4 suites passed (guidance 27 checks), prompt-budget passed, probe 5/4. CI is running on d4d8ecd4.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants