Experimental speculative execution for programmatic tool calls

[acoliver]

Problem and evidence

A generated orchestration program can contain independent tool calls whose arguments become complete while the model is still generating later code. Speculative programmatic tool calling begins eligible calls during generation and claims results if the final program invokes those same calls. The reference implementation restricts speculation to registered pure tools. Its stream parser skips unresolved inputs and uncertain control flow. The author reports roughly 1–1.2× throughput/runtime improvement in RLM OOLONG/OOLONG-Pairs experiments using Qwen3-30B-A3B on eight H100s with five repeats and four/eight concurrent runs. The fixed-program suite shows larger synthetic overlaps. Neither result establishes a speedup for LLxprt.

Code Puppy optionally exposes run_code through pydantic-ai-harness. It speculates list_files, read_file, and grep, while leaving file-writing tools native. It also enables eager execution of completed statements, including non-speculated operations. Its sandbox mounts the workspace read-write, so its read-only speculative tool list does not establish that scripts cannot write. Its published status example sums hidden tool durations instead of measuring paired end-to-end wall-clock improvement. No controlled Code Puppy speedup for LLxprt-like coding work was found.

OpenCode's code mode offers a precedent for a confined interpreter with per-child permissions, hooks and cancellation. The inspected implementation does not establish speculative execution. The reference project's OpenCode adapter is sample daemon wiring, not proof OpenCode ships this feature. An earlier OpenCode PTC PR #9833 closed unmerged. Hosted Anthropic and OpenAI programmatic calling are separate execution paths; their documentation does not establish client-side early execution during argument streaming.

LLxprt currently collects completed tool requests and schedules them after the model stream in AgenticLoop.ts. turn.ts creates requests from completed arguments. Anthropic accumulates argument deltas in its stream processor, but these are not currently an early-execution API. Provider behavior needs testing before assuming useful incremental streaming.

Proposed scope

1. Introduce an opt-in, off-by-default programmatic tool interface with a restricted script interpreter. Programs call explicitly exposed tools and return bounded JSON/text results, without direct host filesystem, process, environment, network or module access. Validate isolation before selecting the interpreter; Node vm alone is not a security boundary. Keep native tool behavior unchanged when disabled. Do not add eager side-effectful execution.
2. Expose provisional program-argument streaming only for providers that support it. Consider only fully parsed calls with complete, demonstrably safe inputs. Skip uncertain branches, dynamic/unsafe dependencies, malformed code and incomplete JSON. Match turn, retry attempt, program-call identity, tool, canonical arguments, occurrence, permission revision and snapshot revision; identical arguments alone cannot identify repeated nondeterministic calls.
3. Admit only explicitly registered pure, stable, bounded, policy-allowed operations. Select the initial eligible set from tests and benchmarks, such as a controlled snapshot-backed read capability. Names like read_file and grep do not prove eligibility: files change, paths/symlinks affect authorization and hooks can alter calls. Do not speculate writes, shell, GitHub/MCP/network, subagents, arbitrary code or approval-required actions. When authorization, hooks or snapshot equivalence cannot be established before launch, run through the normal path at claim time.
4. Preserve the authority of ToolDispatcher, CoreToolScheduler, ConfirmationCoordinator and ToolExecutor hooks. Never bypass policy, confirmations, BeforeTool modification/veto, AfterTool decoration, deduplication, cancellation or result ordering. Initially decline speculation where a relevant hook can affect calls/results. Bound concurrency, memory, output, elapsed time and extra calls. Abort/discard on retry, cancellation, failed generation, changed revisions and turn disposal. Do not publish discarded results.

Test-first plan

Follow dev-docs/RULES.md: failing behavioral tests before minimal TypeScript/Bun production code. Mock transport boundaries rather than the orchestration component under test.

1. Interpreter boundary: disabled-mode parity, denied child tools, no host access, independent/dependent calls, errors, ordering and bounded output. Prove filesystem/process/network isolation before accepting generated programs.
2. Streaming boundary: fragmented arguments, quoted braces and escapes, interleaved tool IDs, buffered providers, truncated/malformed generations, retry and abort. Introduce provisional events without changing completed ToolCallRequestInfo semantics.
3. Admission and claim: deny/ask/disabled policy, hook veto/modification/decoration, duplicate IDs, mismatched arguments, repeated nondeterministic occurrences, stale files/snapshots and untaken branches. Admit only proved-safe calls; run misses normally. Actual child calls yield exactly one ordered model-visible result.
4. Lifecycle and measurement: concurrent turns, retries, cancellation, quotas and discarded results. Count attempts, claims, misses, discards and extra calls without storing sensitive code/output. Simulated delayed calls demonstrate overlap with identical visible behavior. Measure paired on/off whole-turn wall-clock distributions, stream runway, tokens, extra calls/cost and correctness across providers and concurrency configurations. Keep experimental unless measured benefit outweighs overhead and risk.

Acceptance criteria

• Disabled mode and providers without incremental argument streaming retain current behavior and permissions; buffered providers report zero speculative overlap.
• A complete eligible call starts before generation ends, is claimed only by its matching authorized final-program occurrence, and produces one ordered result. A deterministic delayed-tool test proves overlap.
• Denied, approval-needed, disabled, hook-sensitive, mutating, externally connected, stale-snapshot, malformed, abandoned and cancelled calls never run speculatively. Repeated nondeterministic calls cannot share results by arguments alone.
• Scripts cannot access host capabilities outside their approved tool bridge. Byte, time and concurrency budgets and cancellation are enforced.
• Provider, scheduler, tool and end-to-end Bun behavior tests cover these cases. Benchmark notes report paired wall-clock and cost measurements without assuming published speedups transfer.

Related work and open questions

#662 discusses Anthropic tool search and hosted programmatic calls. #245 covers async subagents. #2483 expressly excluded programmatic calling from Codex-model support. This issue covers an opt-in client-side execution path. Unknowns include actual provider streaming runway, safe/valuable LLxprt workloads, gains under shared rate limits, and confined interpreter cost.

[github-actions]

LLxprt Issue Planner — infrastructure failure

The autonomous planner failed with exit code 1. Please inspect the workflow logs (Run planner agent section) and re-run once resolved.