Repository navigation
feat(observability): make node logs reviewable and fix Codex thread dispatch - #564
Merged
Merged
Conversation
Sink::write now counts a record that repeats one written in the last 60 s and writes one line with fields.repeated, first_ms and last_ms when the window ends, on eviction or when the Node stops. Every source goes through it. The default filter keeps iroh::protocol and iroh_gossip::net at error; inbound handler failures of our own are logged as net.accept_failed. New info events: node.started/stopping/stopped, node.start_failed, input.accepted/rejected/admitted/delivered, schedule.fired, triage.queued/concluded, peer.online/offline, mesh.neighbor_down; mesh.neighbor_up moves from debug to info.
web.rpc_failed now carries the device error code and the raw cause beside the worded message. The web and phone log pages title the new lifecycle, peer, triage and update events and show repeat counts, and the troubleshooting guide describes merging, the iroh filter and the new events.
update check and update apply post update.checked, update.deferred, update.applied or update.failed through the running Node the way vibeface log does, so the Node redacts and merges them. Without a reachable Node the result stays in the updater's own output.
Codex 0.167 reads a saved thread even when it is not loaded, so `thread/read` answering "thread not loaded" means the App Server has no such thread at all, typically a session of the user's own Codex folder while the Node runs its own App Server. `thread/resume` then fails with "no rollout found" and retrying cannot help. - Recognize that answer in the bridge (`CodexError::missing_thread`), only for `thread/read`, since turn methods say "thread not found" for a saved thread that is merely not loaded. - `dispatch_work` with `thread` refuses before registering or sending and tells the bot to leave delivery to the user in that session instead of resuming, retrying or linking its files. - The Node's probe settles such a thread as missing instead of an error, so the work becomes unverified rather than stuck. - The bot rules and `codex call` help give App Server parameters in camelCase (`threadId`), not the Node's `thread_id`.
- A record about the same event, source and ids that says something else ends the held repeat first, so a peer that goes offline and back is written in order instead of held for a window. - Flush repeat counts right after node.stopping, before shutdown steps that could hang until the service manager kills the Node. - Submitted records cannot carry repeated, first_ms or last_ms. - Keep iroh_gossip::net::util warnings such as dropped oversized messages. - Align the troubleshooting and protocol docs with the code.
…them Peer and mesh neighbor changes use separate online and offline events, so the per-subject rule did not cover them and also let unrelated warnings of one event end each other's windows. Events already logged only on change now bypass merging; everything else merges by content as before. The troubleshooting doc names them and the full iroh_gossip scope that shows errors only.
A repeated NeighborUp for a known neighbor no longer writes mesh.neighbor_up, and only the Node's own state events bypass merging, not records a browser, CLI or Agent submits under those names.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Problem
The Node log was hard to use for reviewing a test session. On one development Node, 81% of nine days of records were warnings, half of them one repeated triage wake failure. Lifecycle steps such as start and stop, input receipt and peer changes were mostly logged only on failure. Web request failures kept only the user-facing text. Agent dispatches to Codex threads also failed with
thread not loaded, and Agentthread/readcalls failed withmissing field threadId.Changes
Log merging.
Sink::writecounts a record identical to one written in the last 60 seconds instead of writing it again, then writes one line withrepeated,first_msandlast_ms. Counts are flushed every 5 seconds, right afternode.stopping, and at shutdown. At most 256 records are tracked at once. State-change events (peer.*,mesh.neighbor_*,sync.write_state,work.state) are always written as they come. Submitted records cannot carry the count fields.Lifecycle events. New events:
node.started(version, data epoch, platform),node.stopping,node.stopped,node.start_failed,input.*,schedule.fired,triage.queued,triage.concluded,peer.online,peer.offline,mesh.neighbor_down,net.accept_failedandupdate.*. The updater writes its results through the Node's log endpoint.Diagnosability.
web.rpc_failedrecords the error code and the raw cause.Library noise. iroh connection handling and gossip networking log errors only by default.
iroh_gossip::net::utilkeeps its warnings, andRUST_LOGstill overrides the default.Codex threads. When
thread/readsays the App Server has no such thread,dispatch_workrefuses before registering or sending anything. It tells the bot to leave delivery to the user in that session. Node reconciliation treats such a thread as missing, so the work becomes unverified rather than stuck. The bot rules andcodex call --helpshow App Server parameters in camelCase.Validation
d153d7b1: all jobs succeeded, including native packages on four platforms and the chat-agent protocol acceptance.node.started,node.stoppingandnode.stopped, and merged five identical records into one record and one count line.windows rust (cli)fails inmanage::tests::recovery_review_needs_no_replacement_file_and_keeps_selection_explicit. That test comes from feat(cli): add a complete client for Bots and Core operations #556, this PR does not touch it, and fix(update): keep release discovery and package size bounds ahead of growth #562 fixes it.Not covered: delivery into a real Codex App Server session and the phone log page on a device.
Owner: @harryisfish.
Status: ready for review.