Skip to content

feat(observability): make node logs reviewable and fix Codex thread dispatch - #564

Merged
harryisfish merged 8 commits into
mainfrom
feat/node-observability
Nov 13, 2026
Merged

harryisfish merged 8 commits into
mainfrom
feat/node-observability

Conversation

@harryisfish

@harryisfish harryisfish commented Nov 13, 2026 •

Copy link
Copy Markdown
Collaborator

Problem

The Node log was hard to use for reviewing a test session. On one development Node, 81% of nine days of records were warnings, half of them one repeated triage wake failure. Lifecycle steps such as start and stop, input receipt and peer changes were mostly logged only on failure. Web request failures kept only the user-facing text. Agent dispatches to Codex threads also failed with thread not loaded, and Agent thread/read calls failed with missing field threadId.

Changes

Log merging. Sink::write counts a record identical to one written in the last 60 seconds instead of writing it again, then writes one line with repeated, first_ms and last_ms. Counts are flushed every 5 seconds, right after node.stopping, and at shutdown. At most 256 records are tracked at once. State-change events (peer.*, mesh.neighbor_*, sync.write_state, work.state) are always written as they come. Submitted records cannot carry the count fields.

Lifecycle events. New events: node.started (version, data epoch, platform), node.stopping, node.stopped, node.start_failed, input.*, schedule.fired, triage.queued, triage.concluded, peer.online, peer.offline, mesh.neighbor_down, net.accept_failed and update.*. The updater writes its results through the Node's log endpoint.

Diagnosability. web.rpc_failed records the error code and the raw cause.

Library noise. iroh connection handling and gossip networking log errors only by default. iroh_gossip::net::util keeps its warnings, and RUST_LOG still overrides the default.

Codex threads. When thread/read says the App Server has no such thread, dispatch_work refuses before registering or sending anything. It tells the bot to leave delivery to the user in that session. Node reconciliation treats such a thread as missing, so the work becomes unverified rather than stuck. The bot rules and codex call --help show App Server parameters in camelCase.

Validation

  • CI / Core on d153d7b1: all jobs succeeded, including native packages on four platforms and the chat-agent protocol acceptance.
  • Locally: the delivery sandbox (20 passed), the delivery probe build and the www tests passed.
  • A real Node process in a temporary data directory logged node.started, node.stopping and node.stopped, and merged five identical records into one record and one count line.
  • New regression tests cover repeat merging, state flips, forged counts, the default filter and the missing-thread refusal. The flip and missing-thread tests fail on the previous code.
  • CI / Windows native: core and bridge passed. windows rust (cli) fails in manage::tests::recovery_review_needs_no_replacement_file_and_keeps_selection_explicit. That test comes from feat(cli): add a complete client for Bots and Core operations #556, this PR does not touch it, and fix(update): keep release discovery and package size bounds ahead of growth #562 fixes it.

Not covered: delivery into a real Codex App Server session and the phone log page on a device.

Owner: @harryisfish.
Status: ready for review.

Sink::write now counts a record that repeats one written in the last 60 s
and writes one line with fields.repeated, first_ms and last_ms when the
window ends, on eviction or when the Node stops. Every source goes through
it. The default filter keeps iroh::protocol and iroh_gossip::net at error;
inbound handler failures of our own are logged as net.accept_failed.

New info events: node.started/stopping/stopped, node.start_failed,
input.accepted/rejected/admitted/delivered, schedule.fired,
triage.queued/concluded, peer.online/offline, mesh.neighbor_down;
mesh.neighbor_up moves from debug to info.
web.rpc_failed now carries the device error code and the raw cause
beside the worded message. The web and phone log pages title the new
lifecycle, peer, triage and update events and show repeat counts, and
the troubleshooting guide describes merging, the iroh filter and the
new events.
update check and update apply post update.checked, update.deferred,
update.applied or update.failed through the running Node the way
vibeface log does, so the Node redacts and merges them. Without a
reachable Node the result stays in the updater's own output.
Codex 0.167 reads a saved thread even when it is not loaded, so
`thread/read` answering "thread not loaded" means the App Server has
no such thread at all, typically a session of the user's own Codex
folder while the Node runs its own App Server. `thread/resume` then
fails with "no rollout found" and retrying cannot help.

- Recognize that answer in the bridge (`CodexError::missing_thread`),
  only for `thread/read`, since turn methods say "thread not found"
  for a saved thread that is merely not loaded.
- `dispatch_work` with `thread` refuses before registering or sending
  and tells the bot to leave delivery to the user in that session
  instead of resuming, retrying or linking its files.
- The Node's probe settles such a thread as missing instead of an
  error, so the work becomes unverified rather than stuck.
- The bot rules and `codex call` help give App Server parameters in
  camelCase (`threadId`), not the Node's `thread_id`.
- A record about the same event, source and ids that says something
  else ends the held repeat first, so a peer that goes offline and back
  is written in order instead of held for a window.
- Flush repeat counts right after node.stopping, before shutdown steps
  that could hang until the service manager kills the Node.
- Submitted records cannot carry repeated, first_ms or last_ms.
- Keep iroh_gossip::net::util warnings such as dropped oversized
  messages.
- Align the troubleshooting and protocol docs with the code.
…them

Peer and mesh neighbor changes use separate online and offline events,
so the per-subject rule did not cover them and also let unrelated
warnings of one event end each other's windows. Events already logged
only on change now bypass merging; everything else merges by content
as before. The troubleshooting doc names them and the full iroh_gossip
scope that shows errors only.
A repeated NeighborUp for a known neighbor no longer writes
mesh.neighbor_up, and only the Node's own state events bypass merging,
not records a browser, CLI or Agent submits under those names.
@harryisfish
harryisfish marked this pull request as ready for review November 13, 2026 14:51
@harryisfish
harryisfish merged commit 053a767 into main Nov 13, 2026
25 of 27 checks passed
@harryisfish
harryisfish deleted the feat/node-observability branch November 13, 2026 15:10
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant