Skip to content

Add opt-in code-mode tool executor (hybrid run_code + compact SDK) - #1818

Open
Fei2411 wants to merge 13 commits into
devfrom
feat/code-mode-executor
Open

Fei2411 wants to merge 13 commits into
devfrom
feat/code-mode-executor

Conversation

@Fei2411

@Fei2411 Fei2411 commented Nov 18, 2026 •

Copy link
Copy Markdown
Owner

Issue

No issue. Follows lulu's PTC benchmark and the Claude Code Code Mode doc (https://code.claude.com/docs/en/code-mode).

What changed / 改了什么

New plugin entry ds-bot/code-mode (ds-bot/src/host/code-mode.js). It registers a tool executor built only from public DSH APIs: it composes NativeToolExecutor and PtcExecutor and does not patch DSH.

  • mode: hybrid (default): the model gets every normal tool plus run_code and picks one itself. Program bindings cover only fs, search, shell, web and mcp__*. Team, group, agent and goal tools stay native-only.
  • mode: code-only: only run_code, like the PTC preset.
  • Compact SDK: no output-schema dump, one-line descriptions, and a short shape for common tools. In hybrid it doesn't repeat argument schemas. On the fixture's 23 tools: stock 20,120 bytes, hybrid 1,568, code-only 6,611.
  • maxResultChars (default 8000) caps the model-visible run_code result with head and tail kept, plus an explicit "N characters omitted, filter in code" marker.
  • Guidance modeled on Code Mode: do a whole multi-file change in one program, return summaries rather than raw dumps, and use the normal tool for a single call. Pair it with tool-fs.applyPatch: true.
  • Switch: bot.config.codeMode.enabled in ds-bot/cordis.patch.yml, off by default. When on, the executor mounts in the Bot and group agent scopes; restart the engine after changing it. When off, Bot behavior is unchanged.
  • The DSH packages are optional peerDependencies, loaded only when the switch is on. On hosts older than DSH 0.2.2-rc.1, enabling it fails with a clear error naming the minimum version and the missing packages.
  • Design note: docs/plans/2026-11-18-code-mode.md.

Benchmark on headless DSH 0.2.2-rc.3 with deepseek-v4.1-flash: 6 tasks × 3 runs, compared against the earlier baseline. Costs are list-price estimates.

Group Success Steps Est. cost (off-peak)
standard (same day) 16/18 93 $0.02244
code-only (new) 18/18 95 $0.02657
hybrid (new) 18/18 101 $0.02408
PTC (baseline) 18/18 100 $0.03049

Compared with the baseline PTC, hybrid costs 21% less and code-only 13% less, both with 18/18 success. Both still cost more than same-day standard: hybrid +7%, code-only +18%. Standard's two failures were cheap aborts caused by malformed native tool-call JSON, the issue deferred to #1754. Hybrid chose run_code in 10 steps and native tools in 73. Many-file edits are still the expensive case: code-only spent 51% more than the baseline PTC there.

Changelog

none (opt-in, off by default)

How I checked it / 怎么验证的

  • npm test (in ds-bot/): 2959 pass, 0 fail, 3 skipped. Under legacy DSH 0.2.0-rc.2, the minimum-host error tests run and pass.
  • node scripts/build-client.mjs --check
  • npm run lint
  • Smoke test (dev/bench.sh on dsh-v0.2.2-rc.1): all 52 steps passed
  • Screenshot attached, for interface changes (no UI change). Desktop engine (app/src/host.mjs, mock model): with the switch on, run_code appears next to the normal tools and one call worked; with it off, run_code is absent.

Checklist / 检查

  • One issue, one subject. / 只做一件事。
  • src/client/ changed → npm run build ran and client.js is committed. (not changed)
  • No keys, tokens or personal data, also in screenshots.
  • No AI tool as author or Co-authored-by.

Product Engineer opened this PR for Lulu
Implemented in session Compare DSH PTC and standard presets on DeepSeek
Review with Devin · Open in Devin Desktop


Compose the public native and PTC executors, shrink the program SDK and bound model-visible program output without changing default Bot behavior. Cover hybrid schemas, scoped bindings, policy dispatch, capped durable results and relay compatibility.
Keep DSH executors and schemastery host-provided instead of bundling pinned copies. Exercise scoped plugin prompt assembly in both modes so stock PTC instructions and SDK sections cannot silently return.
Hybrid already supplies native tool declarations, so keep its program SDK to guidance, tool names and compact result shapes. Preserve the complete compact code-only declarations.
@devin-ai-integration

Copy link
Copy Markdown
Contributor

I'll fix CI failures and address comments from users with write access that start with 'Devin'.

  • Disable automatic comment, CI, and merge conflict monitoring

@chatgpt-codex-connector

Copy link
Copy Markdown
Codex Review Summary: ❌ Failed

This comment shows the latest Codex review activity on this pull request.

Review Status Commit Review trigger
📝 Code Review ❌ Failed 2026-11-18T15:42:34.738395Z
We couldn’t complete this review. Comment @codex review to try again.
7d79442 PR opened
ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review" or "@codex security review".

Codex reacts with 👀 while any review is running, comments if it has suggestions, and reacts with 👍 once all reviews finish with no findings.

@chatgpt-codex-connector

Copy link
Copy Markdown

You have reached your Codex usage limits for code reviews. You can see your limits in the Codex usage dashboard.
To continue using code reviews, you can upgrade your account or add credits to your account and enable them for code reviews in your settings.

Union explicit executor knownNames and retain the registry fallback when native omits them. Supply program names through a scoped public prompt provider without exposing restricted or inactive schemas.
Keep the public TypeScript renderer's formatting instead of collapsing whitespace across string literals and quoted keys. Cover repeated and padded spaces in keys, enums, constants and nested schemas.
Code-mode requires DSH 0.2.2-rc.1 while the main plugin still supports older hosts. Skip the opt-in executor tests only when executor packages are absent, preserving failures from packages that are present, and document the minimum host.
The 0.2.0 testkit lacks provideWorkingDirectoryFixture. Load it only after executor capability checks so unsupported hosts can register their skips before any feature-only named exports are linked.

@devin-ai-integration devin-ai-integration Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Devin Review found 2 findings. Devin automatically fixed all 2 findings.

Fixed by Devin (2)
  • Hybrid tool ordering blocks affected chats — Reproduced the toolOrder rejection; knownNames now unions executor names with the registry fallback.
  • Altered string literals in generated SDK — Reproduced the literal collapse; removed the blanket whitespace collapse and added tests.

View findings in Devin Review · Follow Devin's work

Devin Review

@Fei2411 Fei2411 left a comment

Copy link
Copy Markdown
Owner Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Review of PR 1818 (code-mode executor) at 8ecce1e7. Verdict: the design is correct and safe, and it changes nothing at runtime for Bots that do not opt in. Three things should be fixed or verified before merge (items 1–3). CI is 16/16 green. Locally npm test gives 2955 pass, 0 fail, 3 skipped (Windows-only), and build-client --check and lint pass.

Should fix / verify before merge

  1. New peers on the main package also reach users who don't opt in. ds-bot/package.json now peers on dsh-tool-executor-native, dsh-tool-executor-ptc, dsh-ptc-runtime and schemastery (all "*"), for every install of ds-bot and not just ds-bot/code-mode. The two executor packages first exist in 0.2.2-alpha.2. On DSH 0.2.0-rc.2, which CI still supports, npm with legacy-peer-deps=false would auto-install them at their latest tag (0.2.2-alpha.2), and that tag pins its own dsh-tools. The result is a second, mismatched DSH copy next to the host's. Suggest marking these four as peerDependenciesMeta: { ...: { optional: true } }.
  2. The DSH ≥ 0.2.2-rc.1 requirement is documented but not enforced. On a 0.2.0-rc.2 harness, import('./code-mode.js') throws a bare ERR_MODULE_NOT_FOUND: Cannot find package '@deepseek-ai/dsh-tool-executor-native'. The 0.2.0-rc.2 CI job passes only because all 14 code-mode tests skip, so nothing checks that path. Suggest a clear load-time error, for example a dynamic import that rethrows "code-mode requires DSH >= 0.2.2-rc.1". Also add one test on the old matrix entry that the main plugin.js still loads.
  3. AGENTS.md smoke test not run for a src/host/ change. The PR body admits this and lists headless benchmark runs instead. The gap matters because the tools.js change is on the default path: the early return for late team-tool schemas used to fire when any run_code was in the list, and now fires only when the list is exactly [run_code]. Stock PTC is unaffected, since PtcExecutor.describe returns only run_code, but please run dev/bench.sh + dev/smoke.mjs on the branch. Related: the app opt-in path (code-mode.disabled: false in app/engine/preset-standard.patch.yml) was never tried in the app. code-mode injects ptcRuntime, and I could not confirm that a TypeScript PTC runtime is mounted in Bot Session scopes there. If it is not, flipping the switch does nothing and nothing reports it. One enabled smoke run, or a note in the design doc, would settle this.

Security (checked against DSH 0.2.2-rc.1 sources) — no bypass found

  • run_code and every nested call go through the public PtcExecutor's ToolExecutionPipeline. Guards, tools/pre-execute / post-execute, approvals and sandbox-escalation approval (approveEscalation) all still apply. The tests check guard denial and post-policy replacement.
  • Hybrid narrows bindings through programToolSet().view(), and nested dispatch resolves through tools.executable(name, scope, toolSet). So team, group, agent, goal and other native-only tools can't be reached from code. The view returns empty for other scopes and respects step admission (when). Code-only exposes everything, the same as stock PTC.
  • The PR adds no eval or sandbox of its own. Isolation, timeoutMs and cancellation come from the host runtime. Inner tool/ptc-dispatch audit events are still written, because they don't depend on the log: undefined passed to the PTC executor.
  • The cap applies only to the model-visible outer text. Nested values and images are not capped, as documented. capResultText's marker loop converges, and the cap is ≥256 > marker length.

Minor

  • execute() writes its own tool/call / tool/result events for program calls, copying the shape of DSH's internal BatchSettlement. If DSH changes that event shape, these logs will drift. A comment naming the dependency would help.
  • A run_code in a batch splits it into separate segments, so native calls on either side of it no longer share one parallel pool. This is fine as a barrier, but worth one line in the doc.

Overlap with other open PRs

  • #1765 and #1298 merge cleanly with dev but conflict with this PR in ds-bot/package-lock.json (the peer block). The conflict is mechanical: whichever lands second runs scripts/merge-dev.sh and regenerates the lock. Suggest landing #1765 first, since it is narrower, then this PR.
  • #1302 and #524 already conflict with dev and gain package.json / lock conflicts from this PR.
  • The other tools.js / scoped-tools-creation.test.js PRs (#1453, #1163, #985, #954, #1168) conflict with dev already and gain no new conflicts.
  • No open PR bumps the DSH pin.

Written by Devin

Do not auto-install executor or runtime copies into older hosts when the independent code-mode entry is unused. Keep wildcard peers host-provided and mirror optional metadata in the lockfile.
Use the public dsh-tools package metadata to reject pre-rc.1 hosts and name absent optional packages before importing the implementation. Keep main-plugin import and the unsupported-host error covered by the legacy CI job.
Keep the active ds-bot bundle switch off by default and import the optional executor only when enabled. Scope it to Bot and group agents without reviving the retired app engine bundle or changing DSH presets. Document the applyPatch prerequisite and engine restart, and cover scope cleanup plus the scripted desktop smoke command.

@Fei2411 Fei2411 left a comment

Copy link
Copy Markdown
Owner Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Re-review at 4f4b171 (comments only). The three items from my earlier review are fixed. I found no blockers; this looks ready to merge once the lockfile conflict below is handled.

Earlier items

  1. Optional peers: fixed. The 4 DSH packages are now peerDependenciesMeta.optional in both package.json and the lock. With the switch off, plugin.js never imports code-mode.js, so they are not loaded.
  2. DSH minimum: fixed. code-mode.js checks @deepseek-ai/dsh-tools against 0.2.2-rc.1 and the 4 packages before it loads the implementation, and throws one message naming the minimum version and what is missing. The new test asserts that message, and that plugin.apply({ codeMode: { enabled: true } }) rejects with it, so the legacy 0.2.0-rc.2 CI job now runs the old-host path instead of only skipping. semver is already a dependency, and dsh-tools exports ./package.json.
  3. Smoke test: reported as passing (52/52 on dsh-v0.2.2-rc.1), which covers the index.js, tools.js and new src/host/ changes. I did not re-run it.

New switch (bot.config.codeMode.enabled)

  • Off by default: ds-bot/cordis.patch.yml sets enabled: false. With it off, createRuntime gets no codeMode argument, rt.scopeCodeMode is never defined, and the new hideBotUnusedTools line does nothing.
  • It does reach the desktop app. app/src/profile.mjs puts the ds-bot bundle in the app profile, so its bot row and this config load there. @deepseek-ai/dsh-base (also in the app profile) mounts ptc-runtime (dsh-ptc-runtime-node) and leaves tool-executor-ptc disabled, so the ptcRuntime dependency is met with no executor clash. A user override in <DSH_HOME>/profiles/ds-bot/cordis.patch.yml survives import, because writeModelRows keeps rows it does not own. Moving the switch out of the retired app/engine/ preset was right. I did not drive the desktop app myself; the PR reports a host.mjs run with the switch on and off.
  • Scoping: the executor mounts only in owned Bot and group Session scopes and is disposed when a Session leaves, an Agent is disposed or the plugin unloads. This is tested. Ordinary DSH Agents keep the fallback executor.

Non-blocking notes

  • When the switch is on, the whole Bot team runtime runs inside a child plugin that injects ptcRuntime. If someone disables the ptc-runtime row, the team waits silently instead of showing an error. That is fine for an opt-in experiment, but worth a line in the design note.
  • There is no in-app UI for the switch: users edit the profile patch file and restart the engine, as documented.

CI: 16/16 green, including the DSH 0.2.0-rc.2 job.

Overlaps: #1765 and #1298 still conflict with this PR only in ds-bot/package-lock.json, and the PR merges cleanly into dev. Same order as before: land #1765, then run scripts/merge-dev.sh here and regenerate the lock.

Written by Devin

Avoid waiting on an unmet child injection that silently suppresses the Bot team. Start the normal runtime and report the missing provider and recovery options through the existing logger and Feedback diagnostics. Cover API startup, diagnostics visibility and logging in a regression test.

@devin-ai-integration devin-ai-integration Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The opt-in executor is separate from #1754's native tool-call parsing issue, but removing it breaks tool execution for live Sessions. Please keep ordinary DSH execution working when a Session leaves Bot ownership or the plugin unloads.

const remove = id => {
const held = scopes.get(id)
scopes.delete(id)
return held?.scope.dispose()

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Disposing this scope leaves the DSH executor slot unavailable rather than restoring the inherited executor. The new real-harness test currently asserts that selectExecutor(agent) throws after the Session loses ownership and after plugin unload. A former Bot Session can be adopted as a plain DSH Session, so leaving it without any executor is a user-visible regression. Use a supported lifecycle that restores inherited execution (or constrain activation to a lifecycle that can safely replace the Agent), and change the test to assert that native tool execution works on the same live Session afterward.

@Fei2411 Fei2411 added review/changes-requested Independent review found issues to fix reviewed-by/antigravity Reviewed by Antigravity reviewed-by/devin Reviewed by Devin labels Nov 21, 2026
@Fei2411 Fei2411 added this to the v0.4.0 milestone Nov 21, 2026
Keep both DeepSeek Harness attribution entries and rebuild the client and link bundles from the merged sources. Preserve the code-mode optional peers alongside dev's additions.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

review/changes-requested Independent review found issues to fix reviewed-by/antigravity Reviewed by Antigravity reviewed-by/devin Reviewed by Devin

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant