fix(ops): prove the deployed archive is coherent after rebuild and repair

[Sinity]

Outcome

The deployed Polylogue archive, daemon, CLI, browser-capture intake, and derived read models agree on one current state after the large rebuild/repair campaign. A restart does not regress authority, convergence, or queryability, and the operator can prove that with one bounded receipt set.

This is the public live closure gate, not a container for every historical deployment bug.

Current reality

Many original implementation gaps are already fixed: version probes, deployment smoke, browser-capture spool discovery/materialization, receiver lifecycle states, route availability, package closure, source-row-missing recovery, bounded health probing, replay/repair machinery, and substantial scale hardening.

The remaining question is empirical and whole-system:

Does the current deployed archive converge to a trustworthy fixed point, survive restart, and report the same truth through every operator surface?

Historical June measurements and one-off repairs remain evidence, but they are not current acceptance criteria unless reproduced on current code.

Required receipt set

Capture one coherent run containing:

• expected and deployed commit/package identity;
• source/index/user/ops/embedding schema identity;
• raw-authority reconciliation result and unresolved conflicts/debt;
• latest per-origin acquisition and index convergence;
• browser-capture spool → raw → indexed identity/provenance;
• bounded query/read smoke against the live archive;
• daemon restart and post-restart convergence;
• stale/interrupted attempt handling;
• resource/time summary for startup, replay, derived rebuild, FTS finalization, and steady state;
• explicit caveats for any deferred or intentionally unresolved debt.

Acceptance criteria

• The canonical raw-authority reconciler reaches fixed point or emits a finite named set of unresolved conflicts requiring operator judgment.
• Re-running the reconciliation is idempotent and does not replace accepted authority with a weaker/stale candidate.
• Source, index, and derived materialization debt converge or remain explicitly bounded and explained.
• A new browser capture is traced from receiver receipt through raw evidence to a nonzero-message indexed session with provenance identifying provider-native versus fallback capture where applicable.
• polylogue, polylogued, daemon HTTP status/health, and deployment smoke agree on deployed version, archive root, schema state, browser-capture state, and current debt.
• Stale running attempts are interrupted/recovered after process death; no phantom running work survives restart.
• Restart followed by the same receipt set remains green without manual cursor edits, forged raw rows, or ad hoc SQL repair.
• Representative live reads succeed under bounded resource/deadline controls; failures identify the exact stage and retained recovery state.
• The final receipt names any remaining non-blocking debt and states why it does not invalidate archive trust.

Tracker authority

• GitHub: public live closure outcome.
• Bead implementation owner: polylogue-lkrc, one raw-authority reconciler (implements).
• Bead executor: polylogue-hjpx, execute accepted replay plans to fixed point (implements).
• Bead live gate: polylogue-yla8, operator-authorized replay/closure proof (implements).
• Bead attestation slice: polylogue-s8q, deployed version/schema/origin attestation (implements, not a mirror of the whole issue).
• Incident and repair Beads may close independently while this issue remains open until the aggregate receipt is coherent.

Non-goals

• No fresh archive reset merely to obtain a green report.
• No forged source rows or silent conflict resolution.
• No reopening completed historical subproblems unless current evidence reproduces them.
• No unbounded optimization campaign; performance work belongs here only when it blocks the live closure receipt or perf(archive): prove current full-corpus ingest and rebuild bounds #2391's current benchmark.
• No closure based solely on unit tests or branch-local smoke.

[Sinity]

Update from PR #2303 / deployed branch head dcbf559:

• Live system override is on 0.1.0+dcbf559 for both polylogue and polylogued.
• devtools workspace deployment-smoke --json --timeout-s 8 --browser passes against the installed daemon: ok=true, failures=[], DOM bytes 221899, screenshot bytes 154869, latest ChatGPT browser capture indexed with 78 messages.
• /api/status and polylogued status --format json now agree on browser capture: nested browser_capture.active=true, flattened browser_capture_active=true, component_state.browser_capture=running, and component_readiness.browser_capture.state=ready.
• The original raw/spool/index browser-capture materialization failure is repaired for the current live artifact.

Still open here: provider-native ChatGPT capture fidelity vs DOM fallback, explicit archive-maintenance cleanup for the stale zero-message placeholder, receiver archive-state lifecycle states beyond boolean captured, and daemon resource/backpressure behavior on large restored archives.

[Sinity]

Final deployed-state update from PR #2303 head 0d3885461 plus Sinnix commit 4b5930f:

What changed since the previous deployed note:

• Deployed Polylogue through Sinnix override at 0.1.0+0d38854 for both polylogue and polylogued.
• Fixed the Sinnix activation issue where a successful switch updated polylogued.service but left the old store-path process running. Sinnix commit 4b5930f forces Home Manager user service activation for the Polylogue service and adds a config assertion. The second switch proved it: before switch ActiveEnterTimestamp=17:00:30 and ExecStart pointed at ...0.1.0+0363ce9; after switch the service restarted automatically at 17:16:44 and ExecStart points at the 0d38854 package.
• devtools workspace deployment-smoke --json --daemon-url http://127.0.0.1:8766 --receiver-url http://127.0.0.1:8765 --archive-root /home/sinity/.local/share/polylogue now reports ok=true, failures=[], installed versions 0.1.0+0d38854, completion probes all passing, and browser capture archive materialized with latest indexed ChatGPT message count 78.
• Live /api/sessions?query=polylogue&limit=1 now includes top-level query-result action_affordances, closing the split-archive HTTP search gap found during probing.
• Live /healthz/live and /healthz/ready return 200. Root /healthz intentionally 404s; if generic operator expectations matter, that should be a separate ergonomics follow-up, not a readiness failure.
• The restored archive is the expected large split archive: /api/health reports db_size_bytes=20251570176 and quick_check=pass.
• The switch reset about 3924 MiB of stale zram residue, which confirms the machine was under real memory pressure during the earlier daemon/package probing.

Still-open evidence from this pass:

• polylogued can still produce high short-term startup/read pressure on the live archive. After the first restart it reached the 4 GiB cgroup peak; after the final restart it read about 1.1 GiB within ~20s and then catch-up processed only the two changed files. This is much better than the original stuck state but still needs a daemon resource/backpressure issue or subtask.
• Startup still logged a recoverable OperationalError: database is locked against ops.db during daemon event DB initialization in the previous restart. The daemon recovered, but journaling a full Rich traceback for a recoverable lock race is poor operator UX and should be tracked.
• Web UI screenshot evidence (/realm/tmp/polylogue-web-root.png) showed the shell header reading the real archive (DB: 18.9 GB) while the default visible state said 0 convs / No sessions in archive. Direct /api/sessions?limit=5 was healthy, so this is a web-shell loading/default-state issue rather than missing data.
• Broad live FTS search remains slow enough to shape UX: /api/sessions?query=polylogue&limit=1 took roughly 15-20s in repeated probes. The web workbench needs loading/progress/cancel behavior and the search backend needs latency budgeting.
• The deployment smoke report still shows /api/facets can return very large live payloads. Facets should be bounded/lazy in the web workbench.
• Provider-native ChatGPT capture fidelity remains open: current captured artifact is still the DOM/fallback shape, while earlier probing showed the authenticated page can expose much richer /backend-api/conversation/<id> JSON.

Verification artifacts:

• /realm/tmp/polylogue-deployment-smoke-final-20260622.json
• /realm/tmp/polylogue-deployment-smoke-browser-20260622.json
• /realm/tmp/polylogue-web-root.png

[Sinity]

Native ChatGPT capture deployed-state update from PR #2303 head 2c8e83af6:

• Deployed through Sinnix override at 0.1.0+2c8e83a for both polylogue and polylogued.
• Live ChatGPT backend payload evidence: /backend-api/conversation/6a2ea930-87a8-83eb-adb7-4cf38ccaa6a3 exposed ~28.4 MB of provider-native JSON with 5,056 mapping nodes and 3,375 textual turns.
• The receiver now accepts the native capture envelope size, and the extension/source parser path prefers raw_provider_payload over DOM fallback.
• Fixed the daemon live-batch path that was marking the 47.9 MB native capture artifact as unknown-export, excluded=1, record_count=0 before full-payload detection could see the browser-capture/native ChatGPT shape.
• Fixed follow-on cursor issues: successful full ingest preserves detected source names; archive-source cursor reconciliation maps from raw_sessions.origin; non-JSONL full-ingest cursors use full-file byte offsets.
• After clearing the two stale cursor rows and restarting, the canonical browser-capture spool artifact is healthy: source.db.raw_sessions has origin=chatgpt-export, native_id=6a2ea930-87a8-83eb-adb7-4cf38ccaa6a3, blob_size=47976748; index.db.sessions has title AAA Provisional central planning center for polylogue and sinex, message_count=3375, user=96, assistant=1636, tool=1643; ops.db.ingest_cursor for the browser-capture path is chatgpt-export, excluded=0, byte_offset=stat_size=47976748.
• Fresh deployment smoke: /realm/tmp/polylogue-deployment-smoke-native-20260622.json reports ok=true, failures=[], deployed versions 0.1.0+2c8e83a, latest browser-capture turn count 3375, latest indexed message count 3375, session chatgpt-export:6a2ea930-87a8-83eb-adb7-4cf38ccaa6a3.

Remaining papercut: the duplicate copy previously scheduled by polylogue import <artifact> into /home/sinity/.local/share/polylogue/inbox/ is still an excluded unknown-export cursor. The real browser-capture source path is repaired and queryable; large browser-capture envelopes entering through generic inbox/import scheduling should be a separate import-routing cleanup unless we decide to delete that duplicate inbox artifact.

[Sinity]

2026-06-22 current deployed state after CLI probe fixes

The workstation is currently switched through the Sinnix override to PR #2303 branch HEAD dac27caf:

• /etc/profiles/per-user/sinity/bin/polylogue --version -> polylogue, version 0.1.0+dac27ca.
• /etc/profiles/per-user/sinity/bin/polylogued --version -> polylogued, version 0.1.0+dac27ca.
• polylogued.service is active/running, NRestarts=0; immediately after the latest switch it reported MemoryCurrent=1155436544, MemoryPeak=1158815744.

The native ChatGPT capture ingestion evidence from the previous comment remains the important archive-state result: the large browser-capture artifact materialized as chatgpt-export:6a2ea930-87a8-83eb-adb7-4cf38ccaa6a3 with 3375 indexed messages.

Still open here: broad daemon resource behavior and the generic inbox duplicate/cursor papercut. The canonical browser-capture spool path is healthy; the copied inbox artifact still lands as an excluded unknown-export row, which is less urgent but confusing enough to keep as ops trust/backlog under this issue.

[Sinity]

2026-06-22 deployed-state update\n\nDeployment has been advanced again through the Sinnix override to PR #2303 HEAD eb27e5e96 (systemwide polylogue --version and polylogued --version report 0.1.0+eb27e5e).\n\nLive checks after switch:\n\n- polylogued.service: active/running, NRestarts=0, current memory observed ~1.1 GiB immediately after switch.\n- GET /api/status: ok=True, component_state.api=running, component_state.browser_capture=running, component_readiness.daemon_api.state=ready.\n- Receiver status now includes browser_capture.spool_path=/home/sinity/.local/share/polylogue/browser-capture.\n- Selector no-match/hit probes against the live 20 GB archive are bounded and correct.\n\nBrowser-capture snapshot handling was tightened in eb27e5e96: browser-capture .json artifacts are snapshots, not append-only streams, so they now bypass append planning and force full parse on replacement/growth. This avoids the stale cursor case where a larger replacement with no newline could be recorded as deferred append and then skipped.\n\nOperational note: during the previous Sinnix switch, activation reset zram from fully used (4 GiB/4 GiB) to empty. That is likely relevant to the earlier machine-choking symptom, but daemon memory behavior still deserves longer observation rather than declaring it solved from one post-switch sample.

[Sinity]

2026-06-22 final deployment probe update

After the review-follow-up commit, the live Sinnix deployment is on 0.1.0+1ec3d48 from PR #2303.

Deployment evidence:

• polylogue --version -> polylogue, version 0.1.0+1ec3d48.
• polylogued --version -> polylogued, version 0.1.0+1ec3d48.
• polylogued.service: active/running, NRestarts=0.
• Shortly after catch-up completed: current memory ~420 MiB, peak ~2.0 GiB; service CPU ~15.8s; cgroup limits still MemoryHigh=4G, MemoryMax=6G.
• The catch-up scan covered 13,100 files, ingested 2 changed files totaling ~337 MiB, then entered watch mode.
• GET /api/status: ok=True, component_state watcher/api/browser_capture all running, browser_capture_active=True, browser spool /home/sinity/.local/share/polylogue/browser-capture, readiness ready.
• Selector no-match probe: polylogue find id:abc then select --json exits 2 with No sessions matched. within 8s.
• Selector hit probe: polylogue find id:claude-code-session:971ddc00-0dea-4f67-9bac-d5756bdea57e then select --json returns the expected JSON row within 8s.

The Sinnix switch again reset stale zram residue (3.8 GiB -> 0B). That is now a repeated deployment observation and remains a strong candidate contributor to the earlier “system choking” symptom, separate from the daemon's own current RSS.

[Sinity]

2026-06-22 deployment-smoke browser/web evidence

Ran the repo deployment smoke command against the systemwide deployment at 0.1.0+1ec3d48.

Non-browser smoke:

• devtools workspace deployment-smoke --json -> ok=True, failures=[].
• Completion probes passed for query then, query action, read views, and message formats.
• Browser-capture archive probe: latest ChatGPT capture 6a2ea930-87a8-83eb-adb7-4cf38ccaa6a3, 3,375 turns, 3,375 indexed messages, latest indexed session chatgpt-export:6a2ea930-87a8-83eb-adb7-4cf38ccaa6a3.
• Receiver archive-state probe returned HTTP 200 with captured=True for the same provider session.
• /api/status, /api/read-view-profiles, /api/facets, and receiver /v1/status routes all probed successfully.

Browser-render smoke:

• devtools workspace deployment-smoke --browser --browser-executable /etc/profiles/per-user/sinity/bin/google-chrome --json -> ok=True, failures=[].
• Browser render produced dom_bytes=222321 and screenshot_bytes=158105 for http://127.0.0.1:8766/.
• The browser probe reports browser_timeout_after_capture as its internal shutdown error, but ok=True; the harness captured the DOM and screenshot before Chrome exceeded the graceful timeout.

[Sinity]

2026-06-22 persistent Sinnix deployment verified

Persistent deployment is now anchored in Sinnix rather than the temporary SINNIX_POLYLOGUE_OVERRIDE path.

Evidence:

• Polylogue PR fix: make deployed Polylogue smoke trustworthy #2303 is merged at 94a8bfb949508b01abd85d10d94dbaf2c3eecc3a.
• /realm/project/polylogue fast-forwarded to 94a8bfb94.
• /realm/project/sinnix/flake.lock now pins polylogue to 94a8bfb949508b01abd85d10d94dbaf2c3eecc3a with nar hash sha256-TrTxE9noV7Q+h6hPOJxgdbTbV8uPYaOShNLAAwKZiYA=.
• nix develop --command switch completed and activated /nix/store/ax081yazpg4spjxb5ds1k6vlc6d0fzqw-nixos-system-sinnix-prime-26.11.20260616.567a49d.
• Installed binaries now report:
  • polylogue, version 0.1.0+94a8bfb
  • polylogued, version 0.1.0+94a8bfb
• polylogued.service restarted from the packaged unit and is active.
• devtools workspace deployment-smoke --json wrote /realm/tmp/polylogue-deployment-smoke-after-sinnix-switch.json and reported ok=true, failures=[].
• devtools workspace deployment-smoke --browser --browser-executable /etc/profiles/per-user/sinity/bin/google-chrome --json wrote /realm/tmp/polylogue-deployment-smoke-after-sinnix-switch-browser.json and reported ok=true, failures=[]; browser render captured dom_bytes=222320, screenshot_bytes=155608 with the known non-fatal browser_timeout_after_capture marker.
• Live status used the real archive: /home/sinity/.local/share/polylogue/index.db, size 20297080832, FTS ready over 3633598 messages.
• Browser-capture receiver status and latest archive-state probe were OK; latest capture was indexed from the browser-capture spool.
• Post-smoke polylogued sample: about 16.8% CPU, 262444 KiB RSS by ps; systemd showed 352.9M memory, peak 1.4G shortly after catch-up. This is not the previous >100% CPU/choking failure shape, but it should keep being watched under normal dogfooding.

Sinnix commit: f5cae37 build(polylogue): deploy merged smoke fixes.