Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
52 commits
Select commit Hold shift + click to select a range
ce68f35
fix(llm): key streamed tool calls by id when the provider sends no index
plombeer31 Sep 1, 2026
53b54c1
fix(cli): default NODE_ENV to production so the TUI stops leaking to OOM
plombeer31 Sep 1, 2026
4309b95
fix(install): say why an unpublished platform has no binary
plombeer31 Sep 1, 2026
6a5691c
fix(tools): route source files from read_document to os.fs.read
Sep 2, 2026
315cf82
fix(openai): retry a stream that dies before its first chunk
plombeer31 Sep 2, 2026
84b6537
feat(loop-detector): detect overlapping re-reads of an unchanged file
plombeer31 Sep 2, 2026
99a23b9
fix(tools): close the format enum and stop the read_document summary …
plombeer31 Sep 2, 2026
8946ad5
feat(providers): probe streaming native-tool contract at setup
plombeer31 Sep 2, 2026
9ebdc4f
fix(openai): make the stream retry budget actually shared
plombeer31 Sep 2, 2026
4ae2d01
fix(loop-detector): stop read_repeat misreading two honest reads
plombeer31 Sep 2, 2026
dd35ddc
test(loop-detector): pin the read_repeat event, trace and notice wiring
plombeer31 Sep 2, 2026
b070c39
fix(providers): stop the probe reporting our own limits as route defects
plombeer31 Sep 2, 2026
6322a0d
test(providers): cover the probe's wiring, its gate and its redaction
plombeer31 Sep 2, 2026
4fcad17
tui: stop button in the composer while a turn is running
plombeer31 Sep 2, 2026
b657a65
tui: persistent update banner in the status bar's top-right corner
plombeer31 Sep 2, 2026
57a485d
llm: report the whole prompt in timing.promptTokens, cached prefix in…
plombeer31 Sep 2, 2026
3a87513
tui: the composer context chip recalculates live
plombeer31 Sep 2, 2026
0f6b8c5
session: persist the context gauge and restore it on session switch
plombeer31 Sep 2, 2026
38789fb
session: pin the provider/model per session and restore it on switch
plombeer31 Sep 2, 2026
2f0a9b4
tui: a stopped turn says who stopped it, and offers a retry
plombeer31 Sep 2, 2026
cc175ae
tui: update banner clickable through the modal floor; lifecycle strip
plombeer31 Sep 2, 2026
1b677b8
fix(llm): provider-availability failures no longer block fallover
plombeer31 Sep 2, 2026
ca4b1c3
fix(llm): a streaming llama response is bounded by idle time, not tot…
plombeer31 Sep 2, 2026
2a32511
fix(runtime): stop probing the local llama backend while a cloud prov…
plombeer31 Sep 2, 2026
33af1f6
fix(agent): route the llama status split through classifyFailure
plombeer31 Sep 2, 2026
778f2ee
test(tui): drive the llama chat-error assertion through the real pipe…
plombeer31 Sep 2, 2026
1aa7141
docs(fallback): say what AdvanceDecision.immediate actually governs
plombeer31 Sep 2, 2026
9d2bb40
fix(llm): put an upper bound back on a single streaming response
plombeer31 Sep 2, 2026
f3f103b
fix(llm): tell a pre-first-token stall apart from a mid-reply stall
plombeer31 Sep 2, 2026
d343831
fix(llm): latch the deferred probes on a sync throw, and signal a ser…
plombeer31 Sep 3, 2026
f618af9
fix(agent): keep the local profile live through a sustained cloud->lo…
plombeer31 Sep 3, 2026
2b7ae94
test(tui): cover the resolveWindow local-route guard
plombeer31 Sep 3, 2026
a247b9f
fix(tui): gate the poller's updateUrl emit on the active route
plombeer31 Sep 3, 2026
b5f86f0
docs: correct two claims the review found overstated
plombeer31 Sep 3, 2026
34a9fe2
test(runtime): re-assert the zero-request criterion across cloud TURNS
plombeer31 Sep 3, 2026
6debbbf
refactor(llm): make the seam's local-link preparer a testable unit, a…
plombeer31 Sep 3, 2026
6a08b47
rc: #309
plombeer31 Sep 3, 2026
7057c30
rc: #310
plombeer31 Sep 3, 2026
8b5568e
rc: #311
plombeer31 Sep 3, 2026
99b6bc1
rc: #312
plombeer31 Sep 3, 2026
c659a2a
rc: #313
plombeer31 Sep 3, 2026
2dc6619
rc: #314
plombeer31 Sep 3, 2026
4e80015
rc: #315
plombeer31 Sep 3, 2026
9536859
rc: #316
plombeer31 Sep 3, 2026
aeda7dc
rc: #317
plombeer31 Sep 3, 2026
70e3a07
rc: #318
plombeer31 Sep 3, 2026
9f7495d
rc: #319
plombeer31 Sep 3, 2026
28efd37
rc: #320
plombeer31 Sep 3, 2026
fc6f917
rc: #321
plombeer31 Sep 3, 2026
e9408d5
rc: #322
plombeer31 Sep 3, 2026
841505d
rc: #323
plombeer31 Sep 3, 2026
944d905
rc: #324
plombeer31 Sep 3, 2026
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
19 changes: 16 additions & 3 deletions AGENTS.md
Original file line number Diff line number Diff line change
Expand Up @@ -124,13 +124,17 @@ The runtime guards against "stuck" turns where the model re-emits the same tool

**Wandering detector (distinct-spread).** `getRepeatCount` / `getNoProgressStreak` only catch the *same* signature repeating. A model probing endless **distinct** URLs / queries / pages on one tool (e.g. guessing 8 different `os.web.fetch` URLs, or firing 21 search POSTs that differ only in volatile result fields) is a different failure mode. For wandering-prone tools (`isWanderingProneTool`: `os.web.fetch`, `os.http.request`, `browser.*`), `check()` computes `effectiveSpread` — the count of distinct completed `argsHash`es for that tool in the window, plus one when the prospective call introduces a new signature. Crossing `loopWanderingThreshold` ⇒ a `wandering` warn whose notice is an **actionable redirect** (`formatWanderingRedirect`: "stop probing URLs, run a web search or reply best-effort") rather than the repeat advisory. Crossing `loopWanderingEscalation` ⇒ `isWanderingEscalated()` returns true and the gate raises a `breaker` signal — the unique call **is** vetoed and the turn ends gracefully. Bulk reads over distinct files (`os.fs.read`) are deliberately **not** wandering-prone — scanning many files is legitimate work.

**Read-coverage detector (semantic progress on `os.fs.read`).** Argument hashing cannot see that `offset: 40, limit: 30` and `offset: 90, limit: 30` returned text the model already has, and reads are (correctly) not wandering-prone, so an overlapping re-read of one unchanged file used to be invisible. `os.fs.read` therefore publishes a `details.readCoverage` block — canonical (symlink-resolved) path, a hash of the bytes it actually read, and the line range it actually returned (see [src/tools/os/fs-read-coverage.ts](src/tools/os/fs-read-coverage.ts)) — and after each call the gate folds that range into a per-file coverage set (`checkReadRepeat` / `recordRead`, [src/agent/read-coverage.ts](src/agent/read-coverage.ts)). A read whose returned range is already fully covered at the same *version* makes no progress; newly covered lines do. Version is `(contentHash, numbered)`: a changed hash discards the file's coverage, so a same-size in-place edit with an untouched mtime resets it, and so does a switch to or from `lineNumbers: true` — the same lines with `LINE_NUMBER|` prefixes are text the model did not have, and calling that a repeat would be a false positive on the ordinary read-then-re-read-numbered-to-edit workflow. Crossing `READ_REPEAT_WARNING_THRESHOLD` (2 consecutive no-progress reads) ⇒ a `read_repeat` **warn** (`formatReadRepeatNotice`, line numbers only — never file content). Warn-only by construction: the read has already executed, so there is nothing to veto. Failed reads record nothing, a truncated read banks only the prefix it returned, and a scan over many distinct files never signals.

The notice picks its remediation from three cases, because a repeat that returned *nothing* is not the same failure as one that returned covered lines. An empty return means the requested range does not exist — either past the end of the file, or (when the coverage detail's `truncated` flag is set) behind this read's `maxBytes` budget, which no `offset` can reach. Telling that model to "read a range you have not covered" would point it straight back at the request that just came back empty, so it is told the reachable window instead, and the byte cap is named where it is the cause.

**Volatile-stripping in `hashToolOutcome`.** Before hashing a generic (non-shell) result's `details`, `stripVolatile` recursively drops `VOLATILE_RESULT_KEYS` (`timestamp`, `ts`, `date`, `time`, `timeTotal`, `timeTotalSeconds`, `durationMs`, `sizeDownload`, `requestId`/`request_id`, `id`, `traceId`/`trace_id`, `sentAt`, `createdAt`, `deliveredAt`). Without this, per-call timings/sizes (e.g. `timeTotalSeconds`, `sizeDownload` on `os.http.request`) make every result hash unique, so a repeated dead/identical endpoint never registers as a no-progress streak. Mirrors OpenClaw's `stripVolatileSendIds`.

**Breaker → graceful reply.** When `breakerVetoStreak` reaches `loopBreakerVetoStreak` (consecutive-veto path) **or** a wandering loop crosses `loopWanderingEscalation`, the gate raises a `breaker` signal. `AgentLoop` then ends the turn with a forced synthetic `reply` (`formatForcedLoopReply`) recorded as a normal `assistant_reply` turn — `reason: "reply"`, session stays `pending`. **No `loop_failed`, no `ModelError`.** `hashToolOutcome` keys results on error details / shell exit codes / volatile-stripped summary+details so two genuinely different results break the streak.

**Trace.** `loop_detected` events carry `level` (`warn` | `critical` | `breaker`) and `detector` (`generic_repeat` | `no_progress` | `wandering`) — see [src/tracing/trace/trace-event.ts](src/tracing/trace/trace-event.ts).
**Trace.** `loop_detected` events carry `level` (`warn` | `critical` | `breaker`) and `detector` (`generic_repeat` | `no_progress` | `wandering` | `test_repeat` | `read_repeat`) — see [src/tracing/trace/trace-event.ts](src/tracing/trace/trace-event.ts). A `read_repeat` event also carries `read` (resolved path, returned range, and the fingerprint on either side of the read), which is enough to audit why it fired without recording a line of the file.

Pinned by [src/agent/loop-detector.test.ts](src/agent/loop-detector.test.ts), [src/agent/batch-executor.test.ts](src/agent/batch-executor.test.ts) (veto single call / siblings survive / terminal never vetoed / breaker escalation), and [src/agent/agent-loop.test.ts](src/agent/agent-loop.test.ts) ("ends the turn with a graceful reply (not loop_failed) when the breaker trips").
Pinned by [src/agent/loop-detector.test.ts](src/agent/loop-detector.test.ts), [src/agent/read-coverage.test.ts](src/agent/read-coverage.test.ts) (containment / partial overlap / pagination / symlink identity / same-size replacement / rendering switch / byte-cap wording / warn-bucket keying / truncation / multi-file scan), [src/cli/trace-formatter.test.ts](src/cli/trace-formatter.test.ts) (`loop_detected` rendering, including traces that predate `detector`/`read`), [src/agent/batch-executor.test.ts](src/agent/batch-executor.test.ts) (veto single call / siblings survive / terminal never vetoed / breaker escalation), and [src/agent/agent-loop.test.ts](src/agent/agent-loop.test.ts) ("ends the turn with a graceful reply (not loop_failed) when the breaker trips", "warns on the second no-progress re-read of one unchanged file" — the latter drives the real `os.fs.read` through the production path and pins the detector's own threshold, its notice and the event payload).

### Out of scope (deferred)

Expand Down Expand Up @@ -1787,10 +1791,19 @@ Deliberately out of scope: an opt-in whole-disk / drive index (the issue sketche

## LLM reliability policy

Two narrow retry layers sit between the agent loop and `llama-server`. Both are deliberately bounded and never replay already-executed tool calls:
Three narrow retry layers sit between the agent loop and the model server. All are deliberately bounded and never replay already-executed tool calls, and none of them ever replays output a caller has already seen:

1. **Parser retry (step-executor).** If the first `parseToolCall` on a completion throws, the executor calls the unary `llmComplete` exactly once more with the same prompt/slot and re-parses. A `parse_retry` event is emitted for observability. If the second attempt also fails, the original error (with a raw-output preview) is thrown. The streaming path always falls back to unary for the retry so partial SSE deltas are not double-emitted.
2. **Transport retry (LlamaServerClient).** `complete()` and the initial pre-body fetch of `completeStream()` are wrapped in a bounded retry governed by `llama.completionRetries` (default 3) and `llama.completionRetryBackoffMs` (default 150ms, exponential with ±20% jitter). Retries fire **only** for network errors (`LlamaServerError.status === null`) and HTTP 5xx. Grammar/validation 4xx and abort signals short-circuit immediately. Once the SSE body starts streaming, no further retries happen — the conversation state on the server is considered indeterminate.
3. **Pre-first-chunk stream retry (`OpenAiProvider.completeStream`).** Closes the one window nobody owned: a cloud provider answers 2xx, thinks for a long time (reasoning models, cold routes), then drops the socket **before emitting a single delta**. `openAiStartStream` has already returned by then, and undici surfaces the death as a bare `Error: terminated` from the body reader, so the whole turn used to fail instantly with `Turn failed [transport]: terminated`. `completeStream` reopens the stream while `committed` is still false — i.e. while not one chunk has been yielded to its caller, which is exactly when a replay is unobservable.

Cancellation is handled before the guards run and *throws* rather than returning a verdict: `signal.reason` is thrown so the failure classifies as `cancelled`, because both shapes that otherwise reach the classifier here (`Error: terminated`, and the `OpenAiHttpError` an abort produces inside `runOpenAiWithRetry`) are filed as `transport`, which makes `shouldAdvance` switch providers and restart the very turn the user stopped. The signal is re-checked after the backoff, since `sleep()` resolves on abort instead of rejecting.

The guards, in order: **committed** (any yielded chunk, reasoning or even a bare `role` preamble, ends retrying forever — but *not* a completion that streamed only tool-call arguments, which yields nothing and is therefore replayed; safe, because nothing reached the user and tools are dispatched only after the completion returns), **`OpenAiHttpError`** (came from the *open*, where this client's own retry policy has already judged it; reopening would replace that policy with `isNetworkError`'s looser one), **shape** (`isNetworkError`, so a consumer bug is not replayed three times), and **budget**.

The budget is genuinely shared, not shared by assertion: `OpenAiAttemptBudget` is a mutable counter created once per `completeStream` and passed into every `openAiStartStream`, so opens and reopens add up to `OPENAI_MAX_ATTEMPTS` instead of nesting into `3 x 3 = 9` requests for one turn. Pacing is `openAiRetryBackoff`, indexed by requests already spent so the delay keeps growing across the open/reopen seam. Nothing is done to the dead response body: the only way to reach the reopen with a response in hand is a body-read failure, so the stream is already errored, `cancel()` on it rejects, and undici has already destroyed the socket.

Known costs, accepted: this layer emits **no observability event** (unlike layer 1's `parse_retry`) and the body-read phase is untimed (`openAiFetch`'s timeout controller is cleared once headers arrive), so a provider that thinks for a minute and then drops now freezes the UI for up to three of those minutes before the same message appears. Bounded by the shared budget, but a wall-clock cap and a `stream_reopen` event are the obvious follow-ups. Resuming a stream that has *already* emitted output is deliberately out of scope — it would either duplicate text or need prefill continuation, and non-deterministic sampling rules out a prefix dedupe. Pinned by [src/llm/provider/openai/openai-stream-retry.test.ts](src/llm/provider/openai/openai-stream-retry.test.ts).

### Failure taxonomy

Expand Down
42 changes: 42 additions & 0 deletions scripts/install.sh
Original file line number Diff line number Diff line change
Expand Up @@ -375,6 +375,48 @@ else
SHA_URL="${BASE}/releases/latest/download/${TAR_NAME}.sha256"
fi

# Preflight: is there a build for this platform at all?
#
# Not every slug the installer can *name* is one the release matrix
# *publishes* — darwin-x64 is commented out of .github/workflows/release.yml,
# so an Intel Mac used to reach `download` and die on a bare
# "download failed: <url>" with a 404 behind it and no hint that the binary
# had never existed. Say so instead, and say what to do about it.
#
# Only a literal 404 counts as "not published". Everything else — offline
# (000), a proxy, a 5xx, no curl at all — falls through to the real
# download, which reports failures as it always has. curl's own exit code
# is no good here: a 404 after -L follows the release redirect surfaces as
# 56, not 22, so read the status directly.
asset_missing() {
have curl || return 1
_am_code="$(curl -sIL --retry 2 -o /dev/null -w '%{http_code}' "$1" 2>/dev/null || echo 000)"
[ "$_am_code" = "404" ]
}

no_build_published() {
echo "no published build for ${SLUG}." >&2
echo >&2
if [ "$SLUG" = "darwin-x64" ]; then
echo "atomic-agent does not publish a macOS Intel binary yet, and the Apple" >&2
echo "Silicon build will not run on this machine. Tracking:" >&2
echo " https://github.com/${REPO}/issues/300" >&2
else
echo "${TAR_NAME} is not attached to this release. See what is published:" >&2
echo " https://github.com/${REPO}/releases" >&2
fi
echo >&2
echo "to run atomic-agent here, build it from source (needs Node 25.7+):" >&2
echo " git clone https://github.com/${REPO}.git" >&2
echo " cd atomic-agent && npm install && npm run build" >&2
echo " node dist/cli/index.js" >&2
exit 1
}

if asset_missing "$TAR_URL"; then
no_build_published
fi

TMPDIR="${TMPDIR:-/tmp}"
WORK="$(mktemp -d "$TMPDIR/atomic-agent-install.XXXXXX")"
TMP_BIN=""
Expand Down
Loading
Loading