Skip to content

Add budget and checkpoint guardrails for runaway review trajectories #306

Description

@yxlyx

A PR review trajectory on harness 0.0.213 ran for roughly 3 active hours before returning, despite ordinary file/tool operations being fast. The trace recorded approximately 1,829 model calls in the initial recipe outcome, including 713 main-agent calls overall, 16 subagent invocations, 3 workflows, and 143 judge calls. It performed repeated audit/fix/audit loops, 210 edits, and several full-suite reruns without surfacing an interim checkpoint. Only 19 transport retries occurred, so infrastructure retries were not the main cause.

Suggested guardrails:

  • configurable wall-time/model-call/tool-call budgets for review tasks
  • require user confirmation before a review transitions into a large multi-file implementation
  • warn or stop on repeated review/fix cycles after validation passes
  • cap or clearly preview workflow/fleet judge expansion
  • automatically emit a progress checkpoint after a fixed time or call count
  • distinguish independent workflow tasks from prompt variants so judging does not add unnecessary overhead

The relevant local trace is fa5e10c4f10541c7.jsonl; the initial failed review recipe reported latency_ms=10840375, model_calls=1829, and tool_calls=1003. This is a limitation/guardrail request rather than a claim that shell/file tools were slow.

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions