feat(shared): enforce run budgets and runaway stops in RunManager (step 2a of #17) - #49
Closed
kbkb628 wants to merge 4 commits into
Closed
feat(shared): enforce run budgets and runaway stops in RunManager (step 2a of #17)#49kbkb628 wants to merge 4 commits into
kbkb628 wants to merge 4 commits into
Conversation
Step 1 of helsome#17: the runtime contract the agent loop can enforce, without wiring it into RunManager yet. - run-budget.ts: RunBudgetLimits / RunBudgetUsage over wall-clock, model calls, tool calls, search iterations, tokens and cost. resolveBudget applies defaults, then per-run overrides, always clamped by the system ceiling and reporting which keys it clamped; checkBudget reports the first exhausted key in a fixed order; budgetStop turns exhaustion into a machine-readable stop_reason with its detail. Invalid limits and negative usage deltas fail loudly instead of being silently ignored. - runaway-detector.ts: deterministic observers for repeated identical tool calls (argument order independent, with tool include/exclude scope), near-identical search queries (token overlap, so widening a query with more words still counts), iterations reporting the same evidence set, and retry storms inside a rolling window. State is plain data and every detection carries its evidence; runawayStop maps a detection to a stop reason. - 32 unit tests cover every budget key, the ceiling/override rules and each detector boundary. Wiring into RunManager, partial-result preservation and the real-provider E2E (a deliberately tiny budget plus an induced repeat-search case) follow in step 2. Refs helsome#17
Step 2a of helsome#17: wire the step-1 contract into the run loop. - RunManagerOptions gains `budgets` (defaults + system ceiling), `searchTools` (patterns whose query argument feeds the search-loop detector) and `runaway` (detector thresholds); startRun accepts per-run budget overrides. - Every runtime event updates usage (model calls, tool calls, search iterations, absolute wall-clock) and feeds the detectors; on exhaustion or a detected loop the run requests cancellation, so it settles as `cancelled` with its partial answer and tool calls preserved, plus machine-readable `stopReason` and `stopDetail` on the persisted run. - `StopReason` moves into the core protocol (UI, telemetry and evaluation read it off the run record); `Run` gains `stopReason` / `stopDetail`. - A budget or loop stop reports BUDGET_EXHAUSTED / LOOP_DETECTED / RETRY_STORM instead of a plain success. With no `budgets` or `runaway` option a run behaves exactly as before. - 6 new tests: budget stop keeps the partial answer, per-run override clamped by the ceiling, repeated tool call, repeated search query, wall-clock via the injected clock, and the unbudgeted no-op. Kernel suite 53 pass; unit suite 1198 pass with the same 7 pre-existing account-fixture failures. `observeRetry` stays unwired: the AgentEvent vocabulary has no retry event yet. Refs helsome#17
…ager The kernel's composition root built its RunManager without any budget options, so a run driven through the production or eval path could never be budgeted — the step-2a wiring was unreachable from the app. - AgentKernelOptions gains `budgets`, `searchTools` and `runaway`, forwarded to RunManager. - 3 new tests: a kernel-configured budget stops a real run and keeps the partial answer, the same script completes when no budget is configured, and runaway thresholds reach the loop. Refs helsome#17
scripts/eval/budget-smoke.ts drives the production path (AgentKernel -> RunManager -> pi runtime) with budgets small enough that the guard must stop the run rather than the model finishing. It prints JSON evidence (stop reason, stop detail, model calls, tool calls, partial answer, persisted messages) and exits non-zero when the expected stop did not happen, so it can gate a real E2E. Two environment requirements found while running it, both handled by flags: - the pi runtime must be pinned (bunx cannot resolve the package's `latest` tag here), e.g. FINAGENT_PI_ARGS="@mariozechner/pi-coding-agent@0.73.1 --mode rpc --provider anthropic"; - the runtime health budget must cover a cold install, hence --health-timeout-ms (default 180s instead of the 5s client default). Refs helsome#17
Collaborator
Author
真实 provider E2E 证据(#17 验收要求:只 mock 不算)环境:macOS arm64 / bun 1.4.0 / pi Case 1 —— 故意设置极低预算,确认真实终止FINAGENT_PI_ARGS="@mariozechner/pi-coding-agent@0.73.1 --mode rpc --provider deepseek --model deepseek-chat" \
bun scripts/eval/budget-smoke.ts --prompt "请用一句话解释什么是复利。" \
--max-model-calls 1 --expect budget_exhausted{
"status": "cancelled",
"stopReason": "budget_exhausted",
"stopDetail": { "key": "modelCalls", "limit": 1, "used": 1 },
"modelCalls": 1,
"toolCalls": [],
"elapsedMs": 2272,
"partialAnswer": "复利是指把投资产生的收益继续投入本金,使收益在后续期间也能产生收益,从而让财富随时间呈指数级增长的效应。\n\n(说明:此为通用金融概念解释,无需调用行情或持仓数据工具。)",
"persistedMessages": [
{ "role": "user", "chars": 13, "toolCalls": 0 },
{ "role": "assistant", "chars": 84, "toolCalls": 0 }
]
}
Case 2 —— 故意诱发重复工具调用,确认不会无限循环FINAGENT_PI_ARGS="@mariozechner/pi-coding-agent@0.73.1 --mode rpc --provider deepseek --model deepseek-chat" \
bun scripts/eval/budget-smoke.ts \
--prompt "请用 bash 工具执行命令 printf ping,然后再用 bash 执行一次完全相同的命令(两次参数必须一字不差),最后用一句话总结。" \
--loop-threshold 2 --max-model-calls 12 --max-tool-calls 12 --expect loop_detected{
"status": "cancelled",
"stopReason": "loop_detected",
"stopDetail": {
"signal": "repeated_tool_call",
"tool": "bash",
"count": 2,
"canonicalArguments": "{\"command\":\"printf ping\"}"
},
"modelCalls": 0,
"toolCalls": ["bash", "bash"],
"elapsedMs": 2261,
"persistedMessages": [
{ "role": "user", "chars": 70, "toolCalls": 0 },
{ "role": "assistant", "chars": 0, "toolCalls": 2 }
]
}
复现注意(都在脚本 flag 里)
已知未接线项(不变)
|
kbkb628
marked this pull request as ready for review
September 11, 2026 06:30
9 tasks
helsome
added a commit
that referenced
this pull request
Sep 11, 2026
Replay the completed #49 implementation onto clean main after #47 was squash-merged. Preserves runtime budget enforcement, partial-result persistence, runaway detection wiring, AgentKernel configuration, tests, and the real-provider smoke harness. Refs #17 Supersedes #49 Co-authored-by: kbkb628 <278338969+kbkb628@users.noreply.github.com>
Owner
|
该 PR 的实现已保留并通过 #54 重新落到最新 原因:#49 是 stacked PR,依赖 #47;#47 squash merge 后,#49 的 fork 分支因历史差异变成 验证:#54 的 Unit tests、Typecheck、Secret scan 均通过,已合并为 |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
依赖:stacked on #47
这条叠在 #47(step 1:contract + detectors) 之上,因此目前包含它的 commit。#47 合并后 diff 会自动缩成只剩本 PR 的接线 commit,届时我会 rebase 到新 main。
Refs #17(step 2a/2b)。改动
packages/shared/src/kernel/run-manager.tsRunManagerOptions新增budgets(defaults + 系统 ceiling)、searchTools、runaway;startRun支持 per-run override;命中即runtime.cancel()packages/shared/src/kernel/agent-kernel.tsAgentKernelOptions透传budgets/searchTools/runaway到 RunManager —— 否则 production / eval 路径无法配置预算packages/core/src/index.tsStopReason进入 core 协议层;Run增加stopReason/stopDetail(UI / 遥测 / 评测都从 run 记录读)run-budget.ts、runaway-detector.tsStopReason改为从 core re-export;导出toolPatternMatches,让 runtime 与检测器共用同一套工具名匹配scripts/eval/budget-smoke.tsrun-manager.test.ts+6、agent-kernel-budget.test.ts+3行为(对齐 #17 的三条硬要求)
status: 'cancelled'(非completed),带 machine-readablestopReason(budget_exhausted/loop_detected/retry_storm)与stopDetail。answer与已完成的toolCalls照常持久化进 assistant message。budgets/runaway时逻辑与改前完全一致,有专门用例守着。验证
那 7 个失败与本 PR 无关:全部是
packages/longbridge-tools的账户 fixture 用例(fixture 按约定不入库),#10 修;详见 #47 的 CI 说明。真实 provider E2E:已完成 ✅
两个验收 case 都用真实 provider(DeepSeek,经 pi runtime,走
AgentKernel → RunManager → PiRuntimeAdapter)跑通,完整 JSON 证据与复现命令见下方评论:--max-model-calls 1)→ 真实调用 1 次后被中止:stopReason=budget_exhausted、stopDetail={key:modelCalls,limit:1,used:1}、partial answer 保留(assistant 84 字符)--loop-threshold 2)→stopReason=loop_detected、stopDetail={signal:repeated_tool_call,tool:bash,count:2}、2 个 toolCalls 记录保留仍未做
stopReason/stopDetail(目前持久化在 run 记录里,UI 尚未展示)observeRetry接线:AgentEvent词表目前没有 retry 事件,检测器已实现并有单测覆盖