fix(daemon): a fresh daemon converges its backlog on a live host - #4683
Conversation
The browser-capture spool probe constructed a cursor store every tick, and initializing the ops tier rewrote its schema-state row on every open. Under the writer lease that write raised 'database is locked' and killed the fresh daemon mid catch-up (rehearsal 2026-09-05, 207 sessions in). Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01DNGJ3awfNrsLaMdHgQZvid
A hot file whose accepted prefix and captured tail are both recorded is a typed safe state, not an authority gap. Counting it as a gap made the fail-closed gate refuse the entire catch-up whenever any session file was being appended, which on a live host is always: the fresh-start rehearsal ingested 207 of 45,490 files in 2.6 hours. Deferred cursors are now counted separately and reported, and only genuinely incomparable rows block. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01DNGJ3awfNrsLaMdHgQZvid
|
Important
This repository does not receive automatic reviews because it has fewer than 10 stars. ⚙️ Run configurationConfiguration used: Organization UI Review profile: ASSERTIVE Plan: Team Run ID: Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out. Comment |
Codex Review SummaryThis comment shows the latest Codex review activity on this pull request.
ℹ️ About Codex in GitHubYour team has set up Codex to review pull requests in this repo. Reviews are triggered when you
Codex reacts with 👀 while any review is running, comments if it has suggestions, and reacts with 👍 once all reviews finish with no findings. |
…he backlog The frontier proof attributes every violation and authority gap to a source path. A live batch now drops exactly those paths and ingests the rest; a refusal no path explains still blocks everything. Rehearsal-6 showed one gemini-cli file with no accepted head refusing all 45,490 files of the catch-up. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01DNGJ3awfNrsLaMdHgQZvid
![chatgpt-codex-connector[bot]](https://avatars.githubusercontent.com/in/1144995?s=60&v=4){kind=small}
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: 163a59bfbd
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
| # path explains still blocks everything. | ||
| blocked = self._blocked_source_paths() | ||
| if blocked.unattributed_reason is None: | ||
| if paths is None: |
There was a problem hiding this comment.
When catch-up or a debounce flush contains only a cursor-ahead path, both watcher routes first call require_cursor_authority() without paths. This branch now returns successfully for any path-attributed violation, so cursor initialization and _plan_catch_up/_needs_work_from_state run before the later path-specific check rejects ingestion; those routines can reconcile, revive, or rebase the forbidden cursor. The unchanged test_live_watcher_catch_up_refuses_ahead_cursor_before_cursor_planning and flush equivalent also fail because they explicitly verify that no initialization or filtering occurs for this input. Pass the candidate paths into the preflight, or otherwise stop before planning when every candidate is refused.
Useful? React with 👍 / 👎.
| cursor_ahead_samples: list[dict[str, object]] = Field(default_factory=list) | ||
| cursor_authority_gap_count: int = 0 | ||
| cursor_authority_gap_samples: list[dict[str, object]] = Field(default_factory=list) | ||
| cursor_authority_deferred_count: int = 0 |
There was a problem hiding this comment.
Adding this serialized field changes both raw_frontier_integrity and the component-readiness counts object emitted by polylogue ops status --format json, but tests/unit/cli/__snapshots__/test_plain_cli_snapshots.ambr still pins the old shapes without cursor_authority_deferred_count. Consequently test_json_status_snapshot fails on every healthy seeded archive, so the required affected-pytest verification cannot pass until the reviewed new field is added to the snapshot baseline.
AGENTS.md reference: AGENTS.md:L172-L173
Useful? React with 👍 / 👎.
| projection = raw_frontier_integrity_projection( | ||
| archive_root, | ||
| raw_materialization_readiness, | ||
| sample_limit=1_000_000, | ||
| ) |
There was a problem hiding this comment.
If an archive has more than 1,000,000 cursor-ahead or broken-head violations, the integrity projection still reports the exact larger count but retains only this many samples. raw_frontier_blocked_source_paths then treats the sampled paths as the complete refusal set and leaves unattributed_reason unset, so a live batch containing an unsampled violating path passes the per-path gate and ingests against a frontier already known to be invalid. Detect count > len(samples) and retain a global refusal, or enumerate the complete path set without using a bounded diagnostic sample.
Useful? React with 👍 / 👎.
| refused = frozenset( | ||
| path | ||
| for path in selected | ||
| if str(path) in blocked.source_paths or str(path.resolve()) in blocked.source_paths | ||
| ) |
There was a problem hiding this comment.
When a cursor-ahead violation was recorded through a symlinked watch root and the daemon is restarted with the same directory configured through its real path, the projection contains the stored symlink spelling while the selected candidate uses the real spelling. This comparison resolves only the selected path, not the strings in blocked.source_paths, so refused is empty and the newly added per-path gate admits the physically identical file despite its known violation. Canonicalize the blocked paths as well, or compare a stable filesystem identity.
Useful? React with 👍 / 👎.
| authorization = self.require_cursor_authority(paths) | ||
| refused_paths = self._refused_paths | ||
| self._refused_paths = frozenset() | ||
| if refused_paths: | ||
| paths = [path for path in paths if path not in refused_paths] | ||
| skipped_file_count += len(refused_paths) |
There was a problem hiding this comment.
For a direct ingest_files([blocked, healthy]) call with the default queued_file_count=None, filtering changes paths to one element and increments skipped_file_count to one, after which the attempt and returned metrics derive queued_file_count from the filtered length. The observable result claims one queued, one needed, and one skipped file for an original two-file batch, breaking the counter accounting in ingest events and attempt telemetry. Capture the original length before filtering and use it as the default queued count.
Useful? React with 👍 / 👎.
Rehearsal-8 died at 11:41Z when a live catch-up chunk's source write lost a lock race with the Drive acquisition that held source.db for 112 s. Lock contention is a retryable chunk failure; a malformed database still propagates. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01DNGJ3awfNrsLaMdHgQZvid
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: 57c6f4f2d6
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
| self._defer_unaccounted_failed_retries(needed) | ||
| return |
There was a problem hiding this comment.
When watcher.live_batch processes a newly observed file with no cursor row and _ingest_files raises database is locked, this inner handler calls _defer_unaccounted_failed_retries, which skips paths whose cursor record is None, and then returns before the outer handler can restore the already-cleared _pending_paths. The file therefore remains uningested until the low-duty periodic catch-up, whose delay grows from 5 minutes to an hour, rather than entering the explicit retry path required for retryable backlog failures. Requeue needed here or durably create retry state before returning.
AGENTS.md reference: AGENTS.md:L109-L115
Useful? React with 👍 / 👎.
| if not projection.available: | ||
| unattributed.append(projection.summary) |
There was a problem hiding this comment.
For a cursor whose source raw exists but has no accepted byte head, the projection is unavailable because cursor_ahead_status is unknown, even though its cursor_authority_gap_sample names the exact source path. This branch immediately adds the whole summary as an unattributed reason, so a batch containing that anomalous path plus a healthy new file is rejected globally instead of filtering only the named path as this API promises. Reserve the unattributed reason for genuinely unreadable or pathless authority failures.
Useful? React with 👍 / 👎.
| def _is_retryable_lock_error(exc: sqlite3.OperationalError) -> bool: | ||
| """SQLite lock contention, as opposed to a broken database.""" | ||
| message = str(exc).lower() | ||
| return "database is locked" in message or "database table is locked" in message or "busy" in message |
There was a problem hiding this comment.
When SQLite reports transient SQLITE_LOCKED contention as database schema is locked: main, this helper returns false because it recognizes only the database/table spellings, so both newly added handlers re-raise the exception and can still terminate the watcher instead of deferring work. The existing polylogue.core.sqlite_locking.is_transient_sqlite_lock also checks SQLite result codes and explicitly recognizes schema locks; use that shared classifier here so all retryable lock variants preserve the daemon's retryable backlog contract.
AGENTS.md reference: AGENTS.md:L109-L115
Useful? React with 👍 / 👎.
The flush and the catch-up planner gate their own path set, dropping the refused paths before cursor reconciliation touches them; a batch whose every path is refused, or a refusal no path explains, still raises. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01DNGJ3awfNrsLaMdHgQZvid
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: feb577a7ce
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
| if cursor.is_deferred: | ||
| gap_count += 1 | ||
| if len(gaps) < sample_limit: | ||
| gaps.append( | ||
| CursorAuthorityGapSample( | ||
| state="deferred", | ||
| source_path=path, | ||
| logical_source_key=None, | ||
| cursor_byte_offset=cursor_offset, | ||
| reason=( | ||
| "ingest cursor has durably captured material awaiting authority resolution " | ||
| f"through byte {cursor.deferred_end_offset}" | ||
| ), | ||
| ) | ||
| ) | ||
| # A deferred cursor is the positive proof of a safe incomplete | ||
| # tail: the prefix is accepted, the captured range is recorded, | ||
| # and the quiet window resolves it. Counting it as a gap made | ||
| # every live host refuse its whole backlog while one file was hot. | ||
| deferred_count += 1 | ||
| continue |
There was a problem hiding this comment.
When an archive has an accepted byte head at offset 10 and an ingest cursor with byte_offset=20 and deferred_end_offset=30, is_deferred is true solely because the deferred endpoint exceeds the cursor, so this branch skips the accepted-frontier comparison and reports the cursor check—and potentially the entire projection—as healthy. A deferred tail does not itself prove that its prefix was accepted; compare byte_offset against the matching head first so an already-ahead cursor remains an explicit violation rather than allowing further live ingestion.
AGENTS.md reference: AGENTS.md:L113-L115
Useful? React with 👍 / 👎.
| # lifecycle actuators, so the source-selection proof must be | ||
| # consumed before initialization or any stateful decision. | ||
| self._batch_processor.require_cursor_authority() | ||
| paths = self._batch_processor.admit_paths(paths) |
There was a problem hiding this comment.
When a Claude tool-result event enqueues an unchanged owner transcript that is currently a path-attributed refusal together with an admitted sidecar, forced_paths is removed from _forced_reparse_paths before this assignment drops the refused owner from paths. The admitted sidecar then completes without entering either exception handler that restores forced_paths; after frontier authority is repaired, the unchanged owner has neither pending nor forced state, so it may never be reparsed and its sidecar-derived tool result remains stale. Retain refused forced paths as explicit retryable work rather than consuming their one-shot reparse signal.
AGENTS.md reference: AGENTS.md:L113-L115
Useful? React with 👍 / 👎.
…d refusal The frontier projection reports available=False whenever one check is unknown; the refused paths are still named. Only rows no path explains block everything. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01DNGJ3awfNrsLaMdHgQZvid
…rsor probe (#4695) ## Summary Four tests encoded behaviour changed by merged PRs: the queue runner's name in the slot refusal (#4684), the descriptor's `verify` exec without a width override (#4682), and the cursor-store stub's signature after the read-only spool probe (#4683). The descriptor's `verify_all` entry still pinned `POLYLOGUE_PYTEST_WORKERS=2`; it now follows the same constant as the affected tier. ## Verification `devtools test tests/unit/devtools/test_verify.py tests/unit/devtools/test_dev_loop_service.py tests/unit/devtools/test_pytest_slot.py tests/unit/daemon/test_daemon_cli.py::test_spool_pending_check_ignores_terminal_cursor_states` green (result line in the pre-push run). From the 2026-09-05 corpus on 754553b (85 failed / 20,522 passed). 🤖 Generated with [Claude Code](https://claude.com/claude-code) https://claude.ai/code/session_01DNGJ3awfNrsLaMdHgQZvid Co-authored-by: Claude Fable 5.1 <noreply@anthropic.com>
1 participant
Summary
Two defects that stopped the fresh-start rehearsal (2026-09-05, task 1547) at 207 of 45,490 files:
deferredcursor was counted as a cursor-authority gap, so the fail-closed source-selection gate refused the entire catch-up whenever any session file was being appended. On a live host that is always. Deferred cursors are now a separate, reported count; only genuinely incomparable rows block.database is lockedand killed the daemon.Problem
Rehearsal evidence:
/realm/tmp/work/rehearsal-4/prev-rehearsal-5/daemon.log(61refused by cursor authoritylines; the integrity snapshot's single gap sample isstate=deferredfor a transcript being appended), and the terminal traceback in_record_ops_schema_statefrom_browser_capture_spool_has_pending_files.Solution
raw_retention: deferred cursors leave the gap count, addcursor_authority_deferred_countto the snapshot, projection, status model and readiness breakdown; the reason string names them.ops_write._record_ops_schema_state: read first, write only on change.cli._browser_capture_spool_has_pending_files:initialize=False.Verification
devtools verify --quick: all gates ok.test_deferred_cursor_never_blocks_source_selection(red if deferred counts as a gap again),test_reopening_a_current_ops_db_writes_nothing(red if the unconditional write returns; observesPRAGMA data_version). Existing deferred-classification test updated to the new law.Residual risk
Hand-built status payloads that omit the new count validate (it defaults to 0). A deferred cursor that never resolves is now invisible to the gate; it remains visible in status and in the readiness breakdown.
Beads: polylogue-6q16u, polylogue-74kj3
🤖 Generated with Claude Code
https://claude.ai/code/session_01DNGJ3awfNrsLaMdHgQZvid