Skip to content

feat(shared): enforce run budgets and runaway stops in RunManager (step 2a of #17) - #49

Closed
kbkb628 wants to merge 4 commits into
helsome:mainfrom
kbkb628:feat/17-run-budget-wiring
Closed

feat(shared): enforce run budgets and runaway stops in RunManager (step 2a of #17)#49
kbkb628 wants to merge 4 commits into
helsome:mainfrom
kbkb628:feat/17-run-budget-wiring

Conversation

@kbkb628

@kbkb628 kbkb628 commented Sep 11, 2026

Copy link
Copy Markdown
Collaborator

依赖:stacked on #47

这条叠在 #47(step 1:contract + detectors) 之上,因此目前包含它的 commit。#47 合并后 diff 会自动缩成只剩本 PR 的接线 commit,届时我会 rebase 到新 main。

Refs #17(step 2a/2b)。

改动

文件 内容
packages/shared/src/kernel/run-manager.ts 接线:每个 runtime 事件更新 usage(model calls / tool calls / search iterations / 绝对 wall-clock)并喂给检测器;RunManagerOptions 新增 budgets(defaults + 系统 ceiling)、searchToolsrunawaystartRun 支持 per-run override;命中即 runtime.cancel()
packages/shared/src/kernel/agent-kernel.ts AgentKernelOptions 透传 budgets / searchTools / runaway 到 RunManager —— 否则 production / eval 路径无法配置预算
packages/core/src/index.ts StopReason 进入 core 协议层;Run 增加 stopReason / stopDetail(UI / 遥测 / 评测都从 run 记录读)
run-budget.tsrunaway-detector.ts StopReason 改为从 core re-export;导出 toolPatternMatches,让 runtime 与检测器共用同一套工具名匹配
scripts/eval/budget-smoke.ts 真实 provider 的 run-budget E2E 脚手架:极低预算驱动 production path,输出 JSON 证据并对 stop reason 断言,超时会主动取消
测试 run-manager.test.ts +6、agent-kernel-budget.test.ts +3

行为(对齐 #17 的三条硬要求)

  • 不把中止表现为普通 success:预算耗尽或命中循环 → 请求取消 → run 落到 status: 'cancelled'(非 completed),带 machine-readable stopReasonbudget_exhausted / loop_detected / retry_storm)与 stopDetail
  • 保留已有成果answer 与已完成的 toolCalls 照常持久化进 assistant message。
  • 默认零行为变更:不传 budgets / runaway 时逻辑与改前完全一致,有专门用例守着。

验证

bun test packages/shared/src/kernel    # 56 pass / 0 fail
cd packages/core && tsc --noEmit       # exit 0
cd packages/shared && tsc --noEmit     # exit 0
bun test --isolate                     # 1201 pass / 7 fail

那 7 个失败与本 PR 无关:全部是 packages/longbridge-tools 的账户 fixture 用例(fixture 按约定不入库),#10 修;详见 #47 的 CI 说明。

真实 provider E2E:已完成 ✅

两个验收 case 都用真实 provider(DeepSeek,经 pi runtime,走 AgentKernel → RunManager → PiRuntimeAdapter)跑通,完整 JSON 证据与复现命令见下方评论

  • 极低预算(--max-model-calls 1)→ 真实调用 1 次后被中止:stopReason=budget_exhaustedstopDetail={key:modelCalls,limit:1,used:1}partial answer 保留(assistant 84 字符)
  • 诱发重复工具调用(--loop-threshold 2)→ stopReason=loop_detectedstopDetail={signal:repeated_tool_call,tool:bash,count:2}2 个 toolCalls 记录保留

仍未做

  • UI 渲染 stopReason / stopDetail(目前持久化在 run 记录里,UI 尚未展示)
  • observeRetry 接线:AgentEvent 词表目前没有 retry 事件,检测器已实现并有单测覆盖

Step 1 of helsome#17: the runtime contract the agent loop can enforce, without
wiring it into RunManager yet.

- run-budget.ts: RunBudgetLimits / RunBudgetUsage over wall-clock, model
  calls, tool calls, search iterations, tokens and cost. resolveBudget
  applies defaults, then per-run overrides, always clamped by the system
  ceiling and reporting which keys it clamped; checkBudget reports the
  first exhausted key in a fixed order; budgetStop turns exhaustion into a
  machine-readable stop_reason with its detail. Invalid limits and negative
  usage deltas fail loudly instead of being silently ignored.
- runaway-detector.ts: deterministic observers for repeated identical tool
  calls (argument order independent, with tool include/exclude scope),
  near-identical search queries (token overlap, so widening a query with
  more words still counts), iterations reporting the same evidence set, and
  retry storms inside a rolling window. State is plain data and every
  detection carries its evidence; runawayStop maps a detection to a stop
  reason.
- 32 unit tests cover every budget key, the ceiling/override rules and each
  detector boundary.

Wiring into RunManager, partial-result preservation and the real-provider
E2E (a deliberately tiny budget plus an induced repeat-search case) follow
in step 2.

Refs helsome#17
Step 2a of helsome#17: wire the step-1 contract into the run loop.

- RunManagerOptions gains `budgets` (defaults + system ceiling), `searchTools`
  (patterns whose query argument feeds the search-loop detector) and `runaway`
  (detector thresholds); startRun accepts per-run budget overrides.
- Every runtime event updates usage (model calls, tool calls, search iterations,
  absolute wall-clock) and feeds the detectors; on exhaustion or a detected loop
  the run requests cancellation, so it settles as `cancelled` with its partial
  answer and tool calls preserved, plus machine-readable `stopReason` and
  `stopDetail` on the persisted run.
- `StopReason` moves into the core protocol (UI, telemetry and evaluation read it
  off the run record); `Run` gains `stopReason` / `stopDetail`.
- A budget or loop stop reports BUDGET_EXHAUSTED / LOOP_DETECTED / RETRY_STORM
  instead of a plain success. With no `budgets` or `runaway` option a run behaves
  exactly as before.
- 6 new tests: budget stop keeps the partial answer, per-run override clamped by
  the ceiling, repeated tool call, repeated search query, wall-clock via the
  injected clock, and the unbudgeted no-op. Kernel suite 53 pass; unit suite 1198
  pass with the same 7 pre-existing account-fixture failures.

`observeRetry` stays unwired: the AgentEvent vocabulary has no retry event yet.

Refs helsome#17
…ager

The kernel's composition root built its RunManager without any budget options, so
a run driven through the production or eval path could never be budgeted — the
step-2a wiring was unreachable from the app.

- AgentKernelOptions gains `budgets`, `searchTools` and `runaway`, forwarded to
  RunManager.
- 3 new tests: a kernel-configured budget stops a real run and keeps the partial
  answer, the same script completes when no budget is configured, and runaway
  thresholds reach the loop.

Refs helsome#17
scripts/eval/budget-smoke.ts drives the production path (AgentKernel ->
RunManager -> pi runtime) with budgets small enough that the guard must stop the
run rather than the model finishing. It prints JSON evidence (stop reason, stop
detail, model calls, tool calls, partial answer, persisted messages) and exits
non-zero when the expected stop did not happen, so it can gate a real E2E.

Two environment requirements found while running it, both handled by flags:
- the pi runtime must be pinned (bunx cannot resolve the package's `latest` tag
  here), e.g. FINAGENT_PI_ARGS="@mariozechner/pi-coding-agent@0.73.1 --mode rpc
  --provider anthropic";
- the runtime health budget must cover a cold install, hence
  --health-timeout-ms (default 180s instead of the 5s client default).

Refs helsome#17
@kbkb628

kbkb628 commented Sep 11, 2026

Copy link
Copy Markdown
Collaborator Author

真实 provider E2E 证据(#17 验收要求:只 mock 不算

环境:macOS arm64 / bun 1.4.0 / pi @mariozechner/pi-coding-agent@0.73.1--provider deepseek --model deepseek-chat
路径:真实 production agent path —— AgentKernel → RunManager → PiRuntimeAdapter → pi runtime → DeepSeek API
脚本scripts/eval/budget-smoke.ts(本 PR 内,输出 JSON 证据并对 stop reason 做断言)

Case 1 —— 故意设置极低预算,确认真实终止

FINAGENT_PI_ARGS="@mariozechner/pi-coding-agent@0.73.1 --mode rpc --provider deepseek --model deepseek-chat" \
bun scripts/eval/budget-smoke.ts --prompt "请用一句话解释什么是复利。" \
  --max-model-calls 1 --expect budget_exhausted
{
  "status": "cancelled",
  "stopReason": "budget_exhausted",
  "stopDetail": { "key": "modelCalls", "limit": 1, "used": 1 },
  "modelCalls": 1,
  "toolCalls": [],
  "elapsedMs": 2272,
  "partialAnswer": "复利是指把投资产生的收益继续投入本金,使收益在后续期间也能产生收益,从而让财富随时间呈指数级增长的效应。\n\n(说明:此为通用金融概念解释,无需调用行情或持仓数据工具。)",
  "persistedMessages": [
    { "role": "user", "chars": 13, "toolCalls": 0 },
    { "role": "assistant", "chars": 84, "toolCalls": 0 }
  ]
}
  • 真实模型调用 1 次(2.27s)后被 run budget 中止;run 落成 cancelled不是 completed
  • stop_reason machine-readable:budget_exhausted + 具体是哪一类预算、limit、已用量
  • partial result 被保留:assistant 消息 84 字符(内容即上面那段),不是空消息

Case 2 —— 故意诱发重复工具调用,确认不会无限循环

FINAGENT_PI_ARGS="@mariozechner/pi-coding-agent@0.73.1 --mode rpc --provider deepseek --model deepseek-chat" \
bun scripts/eval/budget-smoke.ts \
  --prompt "请用 bash 工具执行命令 printf ping,然后再用 bash 执行一次完全相同的命令(两次参数必须一字不差),最后用一句话总结。" \
  --loop-threshold 2 --max-model-calls 12 --max-tool-calls 12 --expect loop_detected
{
  "status": "cancelled",
  "stopReason": "loop_detected",
  "stopDetail": {
    "signal": "repeated_tool_call",
    "tool": "bash",
    "count": 2,
    "canonicalArguments": "{\"command\":\"printf ping\"}"
  },
  "modelCalls": 0,
  "toolCalls": ["bash", "bash"],
  "elapsedMs": 2261,
  "persistedMessages": [
    { "role": "user", "chars": 70, "toolCalls": 0 },
    { "role": "assistant", "chars": 0, "toolCalls": 2 }
  ]
}
  • 模型连续两次发出完全相同bash {"command":"printf ping"};第 2 次触发 detector
  • stop_reason=loop_detected,detail 里带着判定的全部依据(signal / tool / count / 规范化参数)
  • 已有证据保留:assistant 消息带 2 个 toolCalls,没有被丢弃

复现注意(都在脚本 flag 里)

  1. pi 版本必须钉死:本机 bunx @mariozechner/pi-coding-agent 解析不到 latest tag,必须写 @0.73.1(走 FINAGENT_PI_ARGS)。
  2. 健康检查要放宽:客户端默认 5s,冷启动(首次下载 pi)必然超时,脚本默认 --health-timeout-ms 180000
  3. deepseek-chat 不在 pi 的模型表里,会以 custom model id 运行(有一条 warning,但调用正常)。
  4. 脚本对 stop reason 有断言、超时会主动 cancelRun,因此可以当作真实 E2E 门禁使用。

已知未接线项(不变)

observeRetry 仍未接线:AgentEvent 词表目前没有 retry 事件,检测器已实现并有单测覆盖。

@kbkb628
kbkb628 marked this pull request as ready for review September 11, 2026 06:30
helsome added a commit that referenced this pull request Sep 11, 2026
Replay the completed #49 implementation onto clean main after #47 was squash-merged.

Preserves runtime budget enforcement, partial-result persistence, runaway detection wiring, AgentKernel configuration, tests, and the real-provider smoke harness.

Refs #17
Supersedes #49

Co-authored-by: kbkb628 <278338969+kbkb628@users.noreply.github.com>

helsome commented Sep 11, 2026

Copy link
Copy Markdown
Owner

该 PR 的实现已保留并通过 #54 重新落到最新 main

原因:#49 是 stacked PR,依赖 #47#47 squash merge 后,#49 的 fork 分支因历史差异变成 mergeable_state=dirty,而维护端 GitHub App 无法直接改写贡献者 fork。为避免要求贡献者手工重写历史,#54 以最新全绿 main 为父提交,原样重放 #49 最终 head 的文件 blob。

验证:#54 的 Unit tests、Typecheck、Secret scan 均通过,已合并为 4e00e4a9f447b7efa1781fe96679822431e3117e。贡献者署名已通过 Co-authored-by 保留。

Superseded by #54. Thanks @kbkb628.

@helsome helsome closed this Sep 11, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants