From ad90d6badc767d6a14b13644583254f79dc76446 Mon Sep 17 00:00:00 2001 From: Sinity Date: Mon, 10 Aug 2026 09:18:30 +0200 Subject: [PATCH 1/2] chore(beads): close satisfied merged implementation work --- .beads/issues.jsonl | 6 +++--- 1 file changed, 3 insertions(+), 3 deletions(-) diff --git a/.beads/issues.jsonl b/.beads/issues.jsonl index 8550560882..aac4b981d1 100644 --- a/.beads/issues.jsonl +++ b/.beads/issues.jsonl @@ -1,5 +1,5 @@ -{"_type":"issue","id":"polylogue-1mfxh","title":"raw-authority: persist paginated artifact census receipts","description":"Add a durable source-tier raw-authority artifact census and receipt route. It must use the canonical byte-duplicate authority planner, exclude parser-failed rows from the authoritative candidate universe, support resumable bounded pages, validate backup evidence before checkpointing, and persist apply receipts in source.db.","acceptance_criteria":"1. Census candidates are derived through the canonical duplicate-supersession planner and include only the declared accepted parser universe; parser-failed rows and alternate authority logic cannot enter the applied population. 2. Bounded apply supports an exclusive continuation cursor and durable receipt/checkpoint so successive pages cannot repeat or skip rows. 3. Backup manifest and source ownership are validated before checkpoint or mutation, and any invalid or changed evidence refuses without mutation. 4. Each apply page persists an immutable source-tier receipt bound to census, cursor, plan digest, before/after inventory, and command identity. 5. Real temporary-archive tests exercise two pages, stale/invalid backup refusal, parser-failure exclusion, receipt persistence, and red mutations; quick verification passes. 6. No live production apply is claimed by implementation closure.","status":"open","priority":0,"issue_type":"task","owner":"ezo.dev@gmail.com","created_at":"2026-08-10T00:51:50Z","created_by":"Sinity","updated_at":"2026-08-10T00:51:50Z","dependencies":[{"issue_id":"polylogue-1mfxh","depends_on_id":"polylogue-fbkr","type":"discovered-from","created_at":"2026-08-10T00:52:00Z","created_by":"Sinity","metadata":"{}"}],"dependency_count":0,"dependent_count":0,"comment_count":0} -{"_type":"issue","id":"polylogue-mupq0","title":"reindex: bind message-owner backfill to durable candidate authority","description":"Make message-owner scope backfill a shared, daemon-safe prerequisite for candidate creation and promotion. The route must reject unscoped assertions, bind receipts to current durable state and candidate ownership, and recover safely after a durable commit before receipt publication.","acceptance_criteria":"1. Direct rebuild and daemon bulk-rebuild paths invoke the same message-owner gate before candidate creation, after archive ownership acquisition, and before promotion. 2. The receipt binds exact durable assertion fingerprints, candidate generation/ownership, and source/index authority; missing, drifted, duplicated, or mismatched state refuses without candidate mutation. 3. A prepared marker supports restart-safe completion after commit-before-receipt interruption and rejects altered post-commit state. 4. Real SQLite route tests and mutation twins cover bypass, durable drift, candidate-owner mismatch, and recovery. 5. Implementation closure does not claim a live candidate or promotion; those remain under the reindex phase receipts.","status":"open","priority":0,"issue_type":"task","owner":"ezo.dev@gmail.com","created_at":"2026-08-10T00:43:21Z","created_by":"Sinity","updated_at":"2026-08-10T00:43:21Z","dependency_count":0,"dependent_count":0,"comment_count":0} +{"_type":"issue","id":"polylogue-1mfxh","title":"raw-authority: persist paginated artifact census receipts","description":"Add a durable source-tier raw-authority artifact census and receipt route. It must use the canonical byte-duplicate authority planner, exclude parser-failed rows from the authoritative candidate universe, support resumable bounded pages, validate backup evidence before checkpointing, and persist apply receipts in source.db.","acceptance_criteria":"1. Census candidates are derived through the canonical duplicate-supersession planner and include only the declared accepted parser universe; parser-failed rows and alternate authority logic cannot enter the applied population. 2. Bounded apply supports an exclusive continuation cursor and durable receipt/checkpoint so successive pages cannot repeat or skip rows. 3. Backup manifest and source ownership are validated before checkpoint or mutation, and any invalid or changed evidence refuses without mutation. 4. Each apply page persists an immutable source-tier receipt bound to census, cursor, plan digest, before/after inventory, and command identity. 5. Real temporary-archive tests exercise two pages, stale/invalid backup refusal, parser-failure exclusion, receipt persistence, and red mutations; quick verification passes. 6. No live production apply is claimed by implementation closure.","status":"closed","priority":0,"issue_type":"task","owner":"ezo.dev@gmail.com","created_at":"2026-08-10T00:51:50Z","created_by":"Sinity","updated_at":"2026-08-10T07:17:24Z","closed_at":"2026-08-10T07:17:24Z","close_reason":"Satisfied by merged PR #3911 (9f8a0a4e2). Paginated raw-authority census receipts use the canonical planner, parser-failure exclusion, resumable cursor/checkpoint, backup/ownership validation, immutable receipt binding, and red-mutation coverage. Verification: focused census suite and quick gate passed on exact head e2e80dfb; no live production apply claimed.","dependencies":[{"issue_id":"polylogue-1mfxh","depends_on_id":"polylogue-fbkr","type":"discovered-from","created_at":"2026-08-10T00:52:00Z","created_by":"Sinity","metadata":"{}"}],"dependency_count":0,"dependent_count":0,"comment_count":0} +{"_type":"issue","id":"polylogue-mupq0","title":"reindex: bind message-owner backfill to durable candidate authority","description":"Make message-owner scope backfill a shared, daemon-safe prerequisite for candidate creation and promotion. The route must reject unscoped assertions, bind receipts to current durable state and candidate ownership, and recover safely after a durable commit before receipt publication.","acceptance_criteria":"1. Direct rebuild and daemon bulk-rebuild paths invoke the same message-owner gate before candidate creation, after archive ownership acquisition, and before promotion. 2. The receipt binds exact durable assertion fingerprints, candidate generation/ownership, and source/index authority; missing, drifted, duplicated, or mismatched state refuses without candidate mutation. 3. A prepared marker supports restart-safe completion after commit-before-receipt interruption and rejects altered post-commit state. 4. Real SQLite route tests and mutation twins cover bypass, durable drift, candidate-owner mismatch, and recovery. 5. Implementation closure does not claim a live candidate or promotion; those remain under the reindex phase receipts.","status":"closed","priority":0,"issue_type":"task","owner":"ezo.dev@gmail.com","created_at":"2026-08-10T00:43:21Z","created_by":"Sinity","updated_at":"2026-08-10T07:17:24Z","closed_at":"2026-08-10T07:17:24Z","close_reason":"Satisfied by merged PR #3909 (f33e859f9). Direct and daemon rebuild routes share the owner gate; durable assertion/source/index/candidate-owner bindings and prepared-marker recovery are implemented. Verification: 54 focused route tests passed on exact rebased head 68d54b736; merge-gate passed; implementation closure does not claim live candidate or promotion.","dependency_count":0,"dependent_count":0,"comment_count":0} {"_type":"issue","id":"polylogue-d96ta","title":"testmon: run and publish a bounded fresh seed receipt","description":"Execute a fresh seed-testmon run after the concurrency cap, retain the complete selection denominator and resource receipt, and decide whether the result is a complete red baseline, release-eligible baseline, or typed incomplete/resource-timeout outcome.","acceptance_criteria":"1. The fresh seed runs on a clean selected code tree and records the complete expected node universe, selection digest, harness and dependency identity, and actual worker/resource evidence. 2. The run reaches a typed terminal outcome or records an explicit timeout/incomplete receipt; no partial selection can be promoted. 3. The resulting receipt is independently replayable and accepted only by the typed testmon promotion gate. 4. The exact command, resource envelope, result, and residual failure attribution are published for the release ledger.","status":"open","priority":0,"issue_type":"task","owner":"ezo.dev@gmail.com","created_at":"2026-08-10T00:40:19Z","created_by":"Sinity","updated_at":"2026-08-10T00:40:19Z","dependencies":[{"issue_id":"polylogue-d96ta","depends_on_id":"polylogue-817er","type":"discovered-from","created_at":"2026-08-10T00:40:29Z","created_by":"Sinity","metadata":"{}"}],"dependency_count":0,"dependent_count":0,"comment_count":0} {"_type":"issue","id":"polylogue-817er","title":"testmon: complete bounded fresh seed and release-baseline admission","description":"Finish the testmon seed-harness residual after bounding seed-only concurrency. Prove a fresh seed can complete within the declared resource envelope, retain a complete denominator and provenance, and distinguish a complete red baseline from release eligibility.","acceptance_criteria":"1. A fresh-worktree seed selects the complete expected node universe with a declared denominator and does not silently shrink selection. 2. Seed-testmon uses the bounded worker policy and records actual process/resource/timeout evidence; ordinary adaptive test lanes remain unchanged. 3. A completed seed receipt is bound to exact code tree, dependency/testmon graph, harness version, selection digest, and outcome; incomplete or stale receipts cannot promote. 4. Red baseline, green release baseline, incomplete run, and resource-timeout outcomes are distinct typed states. 5. Focused harness mutation tests and devtools verify --quick pass; a fresh seed attempt is run before closure and its exact result is recorded.","status":"open","priority":0,"issue_type":"task","owner":"ezo.dev@gmail.com","created_at":"2026-08-10T00:39:28Z","created_by":"Sinity","updated_at":"2026-08-10T00:39:28Z","dependencies":[{"issue_id":"polylogue-817er","depends_on_id":"polylogue-mq4vx","type":"discovered-from","created_at":"2026-08-10T00:39:37Z","created_by":"Sinity","metadata":"{}"}],"dependency_count":0,"dependent_count":0,"comment_count":0} {"_type":"issue","id":"polylogue-ox2iz","title":"reindex: execute and classify the canary changelog report","description":"Execute the repaired daemon-owned canary differ route against the selected deployed package and inactive generation, classify every observed row difference against the declared schema/parser deltas, and attach the reviewed report to the candidate-build phase.","acceptance_criteria":"1. A fresh canary run uses a frozen source snapshot, selected deployed package SHA, inactive no-promote generation, and exact comparator version. 2. The sampled corpus includes each origin, zoo/pathology fixtures, and a declared denominator; selection cannot be shrunk after observation. 3. Every sessions/messages/blocks/session_links/derived difference is classified as expected with a cited delta or unexpected with a named Bead, with zero unclassified rows. 4. The report includes receipts, digests, command identity, and reviewer disposition and is consumed by candidate-build preflight. 5. A red mutation removing a diff or changing the authority binding fails the report gate; no pointer promotion occurs.","status":"open","priority":0,"issue_type":"task","owner":"ezo.dev@gmail.com","created_at":"2026-08-10T00:37:28Z","created_by":"Sinity","updated_at":"2026-08-10T00:37:28Z","dependencies":[{"issue_id":"polylogue-ox2iz","depends_on_id":"polylogue-0x7nh","type":"discovered-from","created_at":"2026-08-10T00:37:38Z","created_by":"Sinity","metadata":"{}"}],"dependency_count":0,"dependent_count":0,"comment_count":0} @@ -213,7 +213,7 @@ {"_type":"issue","id":"polylogue-tfzw0","title":"storage: hook-event blob_refs born orphaned (del raw_id) - 73,427 refs / ~1.94 GiB unreclaimable and invisible to GC","description":"Structural audit H3 (/realm/data/derived/reports/polylogue-structural-audit-2026-08-03.html). _insert_hook_event (archive_tiers/source_write.py:1176-1177) starts with 'del raw_id' while its caller writes blob_refs(ref_type=raw_payload, ref_id=raw_id) for the payload; nothing joins the ref to raw_hook_events (which has NO blob_hash column - payload lives inline as payload_json, so the blob is a duplicate copy nothing reads back). Live: 73,427/116,149 raw_payload refs resolve to no raw_sessions row; ~69,249 hashes / 1.94 GiB with no live referent, growing since 06-29. blob_gc._still_referenced (blob_gc.py:150-221) is a MEMBERSHIP test on blob_refs so these are retained forever, uncounted. Design fork (hook blobs were retained deliberately in the de-inflation, PR #3265 era): (a) first-class retained ref class: new ref_type hook_payload + blob_hash column on raw_hook_events, GC liveness becomes a per-ref_type JOIN; or (b) declare payload_json the record, delete orphan refs, reclaim 1.94 GiB. Either way: make GC liveness a join not membership, add a standing blob_refs-liveness census metric. Related: polylogue-feu0 (same cross-tier reference class gap).","notes":"2026-08-03 invariant I3 run: reference-liveness violation is not hook-events-only - 1,336 blob_refs rows with ref_type='attachment' have no matching raw_artifacts row either. Strengthens the join-not-membership GC redesign: the census must cover every ref_type. Also fold: invariant I8 found 1 session with drifted sessions.message_count vs actual messages count (projection drift, likely lineage tail-extraction related) - investigate while in the area or split out if unrelated.","status":"closed","priority":1,"issue_type":"bug","owner":"ezo.dev@gmail.com","created_at":"2026-08-03T05:07:18Z","created_by":"Sinity","updated_at":"2026-08-06T19:24:35Z","closed_at":"2026-08-06T19:24:35Z","close_reason":"Closed after current-master audit: PR #3847 landed the unified production hook, attachment, sidecar, and unknown-evidence blob-reference liveness map and focused production-route tests. Remaining live blob reconciliation is owned by the reindex proof graph.","dependency_count":0,"dependent_count":0,"comment_count":0} {"_type":"issue","id":"polylogue-qhk8z","title":"PR #3574 duplicate-chain links trip the pre-existing ActiveByteRevisionChainError membership-census guard","description":"Discovered during polylogue-id4n's fresh triage pass. 2 tests fail:\n\n- tests/unit/sources/test_revision_backfill.py::test_backfill_content_cache_across_pages_reduces_parses_and_matches_uncached_archive\n- tests/unit/storage/test_rebuild_paging_content_order.py::test_rebuild_content_order_paging_dedups_first_time_classification_via_content_cache\n\nBoth fail with:\n\n polylogue.storage.sqlite.archive_tiers.revision_governance.ActiveByteRevisionChainError:\n an active byte-revision chain cannot move to membership governance\n\nRoot cause: a genuine cross-feature interaction between two independent,\nindividually-correct changes.\n\n1. PR #3574 (fix(storage): collapse byte-equal duplicates before revision-chain\n proof) added a \"duplicate\" relation to HistoricalRevisionDecision: two\n byte-identical raws (same content, different acquisition path -- the ordinary\n \"re-exported the same conversation\" shape both failing tests construct)\n now get one classified as representative and the other linked to it via\n predecessor_raw_id/baseline_raw_id, mirroring the representative's verdict.\n This is correct and intentional (fixes 50GB of over-quarantined content on\n the live archive).\n\n2. The pre-existing (#3406, long-standing) membership-census guard in\n revision_governance.py's _replace_full_revision_governance requires that a\n raw being promoted to membership governance have NO other raw pointing at\n it via predecessor_raw_id/baseline_raw_id (\"an active byte-revision chain\n cannot move to membership governance\") -- this guard was written assuming\n only genuine incremental append chains create such links.\n\n#3574 now also creates predecessor/baseline links for the DUPLICATE case, which\nthe membership-census guard was never designed to distinguish from a genuine\nin-progress append chain. A backfill that re-parses a raw touched by this\nguard after #3574's dedup linking now hits ActiveByteRevisionChainError where\nit previously succeeded.\n\nConfirmed via git log -S \"ActiveByteRevisionChainError\" that the guard's own\ncode is unchanged since #3406 -- the trigger is #3574's newly-created links,\nnot the guard itself. Confirmed via git show 31614661f that #3574's own\nverification section did not exercise this specific backfill-then-membership-\ncensus interaction (it ran tests/unit/storage/test_raw_revision_authority.py\nand a `-k \"raw_revision or revision_governance or raw_authority\"` selection,\nwhich apparently does not include these two files).\n\nNeeds design judgment: should the membership-census guard learn to\ndistinguish \"duplicate\" relation links (safe to promote past) from genuine\nincremental chain links (unsafe), or should #3574's duplicate-linking be\nscoped to skip cohorts that would trip this guard? Not attempted as a quick\nfix given the sensitivity of this subsystem (raw-authority correctness,\nquarantine-as-absorbing-state history) -- reproduction is solid, fix\ndirection needs an operator/maintainer decision.\n\nReproduction: devtools test tests/unit/sources/test_revision_backfill.py::test_backfill_content_cache_across_pages_reduces_parses_and_matches_uncached_archive tests/unit/storage/test_rebuild_paging_content_order.py::test_rebuild_content_order_paging_dedups_first_time_classification_via_content_cache","status":"closed","priority":1,"issue_type":"bug","owner":"ezo.dev@gmail.com","created_at":"2026-08-03T00:08:35Z","created_by":"Sinity","updated_at":"2026-08-03T10:30:25Z","closed_at":"2026-08-03T10:30:25Z","close_reason":"Fixed: PR #3616 (exclude byte-identical duplicates from revision baseline tie-break). Both named regression tests pass.","dependency_count":0,"dependent_count":1,"comment_count":0} {"_type":"issue","id":"polylogue-t73c2","title":"Fix the dogfood loop: polylogue's own archive can't answer 'what did agents do' questions","description":"Polylogue's whole thesis is that the archive answers 'what did agents do'. The fanout-operations report (2026-08-02) had to grep 700MB of raw session/subagent JSONL by hand because polylogue itself could not answer these questions: live index.db is at schema v46 with v53 code deployed, recent days of sessions are not ingested, and the daemon has been deliberately held off mid-merge-train.\n\nThis is the SAME root cause as polylogue-9qnzy (P0, schema-currency gap blocking the planned reindex) -- not a separate bug, a direct consequence of it. This bead exists to make explicit the SECOND reason 9qnzy matters: it's not just blocking a planned reindex, it's actively preventing polylogue from dogfooding its own coordination data right now.\n\nOnce 9qnzy resolves and the reindex/daemon-restart sequence completes: make the coordinator dashboard (output-token ratio, dispatch counts, per-lane outcomes, model distribution) a standing polylogue query instead of a bespoke mining pass every time someone wants to know how a fanout session went. Depends on polylogue-9qnzy.","status":"open","priority":1,"issue_type":"task","owner":"ezo.dev@gmail.com","created_at":"2026-08-02T23:40:08Z","created_by":"Sinity","updated_at":"2026-08-02T23:40:08Z","dependencies":[{"issue_id":"polylogue-t73c2","depends_on_id":"polylogue-3bsrp","type":"relates-to","created_at":"2026-08-03T07:01:30Z","created_by":"Sinity","metadata":"{}"},{"issue_id":"polylogue-t73c2","depends_on_id":"polylogue-9qnzy","type":"blocks","created_at":"2026-08-03T01:40:20Z","created_by":"Sinity","metadata":"{}"},{"issue_id":"polylogue-t73c2","depends_on_id":"polylogue-ltfj9","type":"parent-child","created_at":"2026-08-03T01:40:09Z","created_by":"Sinity","metadata":"{}"}],"dependency_count":1,"dependent_count":0,"comment_count":0} -{"_type":"issue","id":"polylogue-tw4ar","title":"Raw-authority verdict: persist a cache table + wire daemon convergence (Phase 2 follow-up)","description":"Follow-up from polylogue-w6hql (PR #3593): project_raw_authority_verdicts (polylogue/storage/raw_authority_verdict_projection.py) currently recomputes verdicts on demand by re-running classify_historical_full_revision_streams against live blob storage every call -- correct but not cheap at scale (761K+ census_plans-era cohort sizes). This bead is to design and land a persisted raw_authority_verdicts cache table (additive migration, numbered under storage/sqlite/migrations/source/) plus wiring into DaemonConverger so the cache tracks new/reclassified cohorts without a full rescan each read. Needed before polylogue-ds4b4 item 4 (blob-GC invariant verification) can cheaply check verdicts at scale rather than via the on-demand read path.","design":"DESIGN (2026-08-03): remaining half only — the cache table + invalidation shipped in PR #3628 (migration 024). Build the DaemonConverger warm-keeping stage: a ConvergenceStage in daemon/convergence_stages.py with check (are there cohorts whose logical_source_key changed since their cached cohort_fingerprint, or never-cached cohorts?) and execute (recompute via project_raw_authority_verdicts and upsert the cache in bounded batches). Use false_means_pending to push remaining backlog into convergence_debt rather than blocking; main process is the sole writer — no worker-process computation of the byte-proof classifier. Invalidation is already content-keyed (cohort_fingerprint over (raw_id, revision_kind, blob_hash) rows, storage/raw_authority_verdict_cache.py) — the stage only needs to FIND stale/missing cohorts cheaply (e.g. join raw_sessions cohort fingerprints against cache rows), never trust elapsed time. Pitfall: append-kind cohorts raise NotImplementedError in the projection — the stage must skip them typed-visibly (count reported), not crash, until w6hql stage-3 extends coverage. Consumer readiness: ds4b4 item 4 reads through the cache once this stage keeps it warm.\n","acceptance_criteria":"1. A DaemonConverger stage exists (daemon/convergence_stages.py) that finds never-cached and fingerprint-stale cohorts and upserts raw_authority_verdicts in bounded batches, deferring backlog via false_means_pending; unit test through the real stage interface.\n2. Append-kind cohorts are skipped typed-visibly (reported count), not crashed on, until w6hql extends coverage.\n3. A repeated read (e.g. ds4b4-style GC invariant check) hits the cache (no classify_historical_full_revision_streams recompute) — proven by a test asserting call counts or receipts.\n4. Cache staleness is content-keyed only (cohort_fingerprint); no time-based trust. Verify: devtools test -k verdict_cache; devtools test -k convergence.","notes":"2026-08-03: PR #3628 shipped the persisted raw_authority_verdicts cache table + cohort-fingerprint invalidation (SOURCE_SCHEMA_VERSION 24, migration 024). Remaining scope: wiring a DaemonConverger stage to keep the cache warm proactively -- deliberately deferred per that PR's own body. Bead stays open for that remaining half.","status":"open","priority":1,"issue_type":"task","owner":"ezo.dev@gmail.com","created_at":"2026-08-02T22:08:58Z","created_by":"Sinity","updated_at":"2026-08-03T11:09:58Z","dependencies":[{"issue_id":"polylogue-tw4ar","depends_on_id":"polylogue-fbkr","type":"discovered-from","created_at":"2026-08-10T00:18:54Z","created_by":"Sinity","metadata":"{}"}],"dependency_count":0,"dependent_count":3,"comment_count":0} +{"_type":"issue","id":"polylogue-tw4ar","title":"Raw-authority verdict: persist a cache table + wire daemon convergence (Phase 2 follow-up)","description":"Follow-up from polylogue-w6hql (PR #3593): project_raw_authority_verdicts (polylogue/storage/raw_authority_verdict_projection.py) currently recomputes verdicts on demand by re-running classify_historical_full_revision_streams against live blob storage every call -- correct but not cheap at scale (761K+ census_plans-era cohort sizes). This bead is to design and land a persisted raw_authority_verdicts cache table (additive migration, numbered under storage/sqlite/migrations/source/) plus wiring into DaemonConverger so the cache tracks new/reclassified cohorts without a full rescan each read. Needed before polylogue-ds4b4 item 4 (blob-GC invariant verification) can cheaply check verdicts at scale rather than via the on-demand read path.","design":"DESIGN (2026-08-03): remaining half only — the cache table + invalidation shipped in PR #3628 (migration 024). Build the DaemonConverger warm-keeping stage: a ConvergenceStage in daemon/convergence_stages.py with check (are there cohorts whose logical_source_key changed since their cached cohort_fingerprint, or never-cached cohorts?) and execute (recompute via project_raw_authority_verdicts and upsert the cache in bounded batches). Use false_means_pending to push remaining backlog into convergence_debt rather than blocking; main process is the sole writer — no worker-process computation of the byte-proof classifier. Invalidation is already content-keyed (cohort_fingerprint over (raw_id, revision_kind, blob_hash) rows, storage/raw_authority_verdict_cache.py) — the stage only needs to FIND stale/missing cohorts cheaply (e.g. join raw_sessions cohort fingerprints against cache rows), never trust elapsed time. Pitfall: append-kind cohorts raise NotImplementedError in the projection — the stage must skip them typed-visibly (count reported), not crash, until w6hql stage-3 extends coverage. Consumer readiness: ds4b4 item 4 reads through the cache once this stage keeps it warm.\n","acceptance_criteria":"1. A DaemonConverger stage exists (daemon/convergence_stages.py) that finds never-cached and fingerprint-stale cohorts and upserts raw_authority_verdicts in bounded batches, deferring backlog via false_means_pending; unit test through the real stage interface.\n2. Append-kind cohorts are skipped typed-visibly (reported count), not crashed on, until w6hql extends coverage.\n3. A repeated read (e.g. ds4b4-style GC invariant check) hits the cache (no classify_historical_full_revision_streams recompute) — proven by a test asserting call counts or receipts.\n4. Cache staleness is content-keyed only (cohort_fingerprint); no time-based trust. Verify: devtools test -k verdict_cache; devtools test -k convergence.","notes":"2026-08-03: PR #3628 shipped the persisted raw_authority_verdicts cache table + cohort-fingerprint invalidation (SOURCE_SCHEMA_VERSION 24, migration 024). Remaining scope: wiring a DaemonConverger stage to keep the cache warm proactively -- deliberately deferred per that PR's own body. Bead stays open for that remaining half.","status":"open","priority":1,"issue_type":"task","owner":"ezo.dev@gmail.com","created_at":"2026-08-02T22:08:58Z","created_by":"Sinity","updated_at":"2026-08-03T11:09:58Z","dependency_count":0,"dependent_count":3,"comment_count":0} {"_type":"issue","id":"polylogue-gysk3","title":"message_identity_hash reads position-derived provider_message_id fallback (attachment-class bug, unfixed for messages)","description":"Found during polylogue-ds4b4 item 3 investigation (position-derived synthetic\nidentity audit). polylogue-hith/qkuq already found and fixed exactly this\nclass of bug for attachments: a synthetic id seeded partly by array index\n(`att-\u003chash(message_id:name:index)\u003e`) is unstable across export vintages that\nreorder/insert array entries, causing false divergence in revision-authority\nmembership comparison. The fix there was NOT to remove the synthetic\ngenerator (still used as a last-resort storage/display id, seed no longer\nincludes index) but to make `attachment_identity_hash`\n(polylogue/pipeline/ids.py:254) stop reading either the real or synthetic\nattachment id at all -- it hashes only (message_id, name, mime_type).\n\nThe message-level sibling, `message_identity_hash` (polylogue/pipeline/ids.py:212),\nhas the analogous doc claim (\"A provider's own message id is stable across\nre-exports even when the export's array ordering is not\") but no such\nexclusion mechanism -- it hashes the message's `id` directly, and that id\nIS `provider_message_id`, which multiple parsers construct as\n`f\"msg-{index}\"`/`f\"{record_type}-{index}\"` when the raw record carries no\nnative id of its own:\n\n - polylogue/sources/parsers/claude/common.py:912-914 -- `f\"msg-{index}\"`\n - polylogue/sources/parsers/claude/code_parser.py:1620 -- `str(record_uuid or f\"msg-{index}\")`\n - polylogue/sources/parsers/codex.py:1592,1895,1931,2010,2200,2385 --\n `f\"function-call-{index}\"`, `f\"function-call-output-{index}\"`,\n `f\"reasoning-{index}\"`, `f\"{record_type}-{index}\"`,\n `f\"compaction-summary-{idx}\"`\n - polylogue/sources/parsers/local_agent.py:205,252 -- `f\"msg-{index}\"`\n - polylogue/sources/parsers/grok.py:139 -- `f\"{fallback_id}:{index}\"` (unconditional, no real id ever present)\n - polylogue/sources/parsers/drive.py:345 -- `f\"chunk-{idx}\"`\n - polylogue/sources/parsers/chatgpt.py:604 -- `f\"msg-{idx}\"`\n - polylogue/sources/parsers/base_support.py:360 -- `f\"msg-{idx}\"` (shared segment-message builder)\n - polylogue/sources/parsers/antigravity.py:414 -- `f\"{cascade_id}:{index}:{_message_kind(heading)}\"`\n\nUnlike attachments, there is no separate field to fall back to for messages\n-- `provider_message_id` IS the sole identity input by construction\n(`message_identity_hash(*, id: str)`'s fixed keyword-only signature), so the\n\"exclude both real and synthetic id\" fix pattern used for attachments\ndoesn't directly transplant. This needs its own design: likely a\ncomparison-identity axis that anchors on structural position (index within\nthe message array) ONLY when no provider-native id exists, combined with a\ncontent-similarity fallback, or an explicit typed\n\"positionally-anchored, not identity-anchored\" marker threaded through\nsession_revision_membership.py's comparison so a reorder is detected as\n\"can't prove sameness\" rather than silently comparing wrong pairs as if\nmessage ids matched.\n\nNot fixed in polylogue-ds4b4's session: this is genuinely new-discovered\ndebt, and reworking a `pipeline/ids.py` core identity function used\narchive-wide is a substantial, high-risk change (touches every provider's\ncontent-hash/revision-membership comparison) that deserves its own\ndedicated, unhurried session with its own regression-test design -- not a\nrushed fix bundled into an unrelated raw-authority-Phase-3 PR. ds4b4's own\nscope was a preventive LINT for this pattern (shipped separately, flags\nfuture occurrences of position-derived identity construction), not fixing\nevery existing instance.\n\nRef polylogue-ds4b4 (raw-authority redesign Phase 3, item 3)","status":"closed","priority":1,"issue_type":"bug","owner":"ezo.dev@gmail.com","created_at":"2026-08-02T19:36:47Z","created_by":"Sinity","updated_at":"2026-08-03T08:14:39Z","closed_at":"2026-08-03T08:14:39Z","close_reason":"Fixed (in-scope instance): PR #3604 (00e40ef30). _message_comparison_id prefers real provider_message_id, falls back to role+timestamp content anchor instead of position index; only falls back to position when neither exists. Red-first test. NOTE: root cause (parsers baking position-derived ids directly into provider_message_id) is tracked separately - all 18 call sites already acked in docs/plans/position-derived-identity-acks.json referencing this bead.","dependency_count":0,"dependent_count":2,"comment_count":0} {"_type":"issue","id":"polylogue-4zqh3","title":"Acquire the hermes-comparison recovery packet: shared-page decode + sole-copy attachment payloads","description":"A recovery bundle now lives at /realm/data/exports/chatlog/raw/recovery/hermes-project-comparison-2026-07/ (moved from /realm/inbox 2026-08-02; README + SHA256SUMS inside). Two capture gaps, verified against the live archive read-only on 2026-08-02: (1) the ChatGPT shared-page decode chatgpt-shared-decode-6a4ac87b/ (conversation 6a4ac87b, title 'Project and Codebase Analysis', 948 messages, decoded from the share-page React Router stream into messages.json + md + raw html) matches NO raw_sessions row by native_id — the session is entirely absent from the archive and the decode format has no parser; (2) the Claude.ai session claude-ai-export:2c2eab57-fc6c-4c61-99fa-f61af3b7ac57 IS acquired+indexed (raw 8e622747..., quarantined) but 70 of its 83 attachment refs are unfetched with 0 bytes, and the actual payload bytes sit only in this packet (hermes-agent-main.zip 57,853,455 B sha256 31267de3..., hermes-agent-all.tar.gz 257,839,520 B sha256 1b4eba44..., full ChatGPT temporary transcript 309,149 B sha256 db526f41... — none of the three hashes exist in the blob store). Wanted: an ingest path for the decode (or a one-off import), and attachment-byte acquisition from local packet files so the unfetched refs become acquired blobs. The packet is the sole copy of these bytes; exclude from any prune.","design":"DESIGN (2026-08-03): two independent acquisitions from the sole-copy recovery packet (/realm/data/exports/chatlog/raw/recovery/hermes-project-comparison-2026-07/; README + SHA256SUMS; EXCLUDE FROM ANY PRUNE — sole copy):\n1. ChatGPT shared-page decode (conversation 6a4ac87b, 948 messages, messages.json + md + raw html): the decode format has no parser. Options: (a) a narrow detector + parser for the decoded messages.json shape at the document-tightness level in sources/dispatch.py (durable: future share-page decodes ingest too); (b) one-off import via a conversion script mapping the decode into an existing admitted shape. Prefer (a) if the messages.json shape is close to chatgpt-export's mapping node structure (likely, it was decoded from the share-page React Router stream); the parser reuses chatgpt.py's message lowering. Origin question: it is a chatgpt conversation — admit as chatgpt-export with acquisition evidence marking the share-page provenance, not a new origin.\n2. Attachment-byte backfill for claude-ai-export:2c2eab57... (70/83 refs unfetched, 0 bytes; the three packet payloads' sha256 absent from the blob store): an acquisition path that matches local packet files to unfetched attachment refs (by declared hash where the export carries one, else by operator-asserted mapping recorded as evidence) and publishes blobs + flips acquisition_status to acquired. Reuses the attachment blob-write path from #2469 (_acquire_attachment_blob/_write_attachments); the raw row is quarantined — attachment acquisition must not depend on the session's authority state.\nBoth feed f1vg's attachment-fidelity and absence buckets; record before/after bucket counts.\n","acceptance_criteria":"1. The shared-page decode conversation (6a4ac87b, 948 messages) is ingested and queryable (as chatgpt-export with share-page provenance evidence), via parser or documented one-off import; raw bytes + provenance recorded in source.db.\n2. The claude-ai session's 70 unfetched attachment refs become acquired blobs with true SHA-256s from the packet payloads (the three named hashes present in the blob store); acquisition does not depend on the raw row's authority state.\n3. f1vg's absence and attachment-fidelity buckets drop accordingly (before/after recorded).\n4. The packet directory is protected from prune/cleanup (noted in its README or the owning inventory).\n5. Verify: devtools test -k chatgpt or -k attachments for new paths; read-only live queries for ref status.","notes":"Footprint: polylogue/sources/dispatch.py, polylogue/sources/parsers/hermes_spans.py, polylogue/sources/parsers/chatgpt.py (recovery-packet ingestion: ChatGPT shared-page decode + hermes sole-copy attachment payloads).","status":"open","priority":1,"issue_type":"task","owner":"ezo.dev@gmail.com","created_at":"2026-08-02T18:27:09Z","created_by":"Sinity","updated_at":"2026-08-03T11:12:52Z","dependency_count":0,"dependent_count":1,"comment_count":0} {"_type":"issue","id":"polylogue-rn5jh","title":"Wire blob-reference-replace-from-source + embeddings-rescue into daemon automation (mislabeled as 'needs judgment')","notes":"2026-08-02 correction after operator pushback: sharper findings than the original AC framing.\n\nembeddings-rescue: polylogue-04kl (the bead this command was built for) is CLOSED -- the actual rescue already happened (187,888 vectors recovered from the one specific 2026-07-10 retired tier, real production win, done). The command remains in the CLI as generic '--source \u003cany path\u003e' product surface for a scenario that occurred exactly once and is finished. Revised AC: DELETE this command (not 'automate it') unless investigation finds embeddings-tier retirement is a routine recurring event (check EMBEDDINGS_SCHEMA_VERSION bump history/frequency) that would need it again -- if genuinely recurring, THEN build the registry+automation described below; if it was truly one-time, it should simply be removed as dead product surface once 04kl's specific job is confirmed fully done (528 partial sessions + ~8949 non-fully-rescuable sessions were noted as still needing real API embedding in 04kl's own notes -- confirm those don't still need this exact command before deleting).\n\nblob-reference-replace-from-source: checked live production archive directly (2026-08-02): 'polylogue ops maintenance blob-reference-debt' reports 160,609 references / 104,026 distinct blobs / 0 missing / status=ok. There is currently ZERO active debt for this command to repair -- the acquire-blob/commit-row safety invariants (leases + snapshot reference check, per CLAUDE.md) appear to be holding in practice right now, not just in theory. This changes the framing: this isn't a live ongoing backlog needing automation urgently -- it's a rare/historical-scenario tool sitting at zero. Revised AC: (a) confirm whether missing-blob-ref debt has EVER been nonzero on this archive historically (check gc_generations/blob GC pass history, or absence of evidence either way), (b) if it's genuinely rare/historical, a read-only periodic check (fails loud if debt ever appears, rather than silent accumulation) may be sufficient instead of full automation -- don't over-build automation for a zero-occurrence problem, (c) if investigation finds real recurring instances, THEN wire the deterministic replace-from-source logic into daemon convergence as originally scoped.\n2026-08-02 RESOLVED via PR #3585 (branch feature/refactor/retire-embeddings-rescue-guard-blob-debt).\n\nembeddings-rescue: DELETED as dead product surface, confirmed one-time. No mechanism in the codebase ever preserves a retired embeddings.db (reset --database hard-deletes it; storage/sqlite/archive_tiers/embeddings.py has had 4 EMBEDDINGS_SCHEMA_VERSION bumps since inception, none of which route through a \"retired file\" path) -- the embeddings.db.v2-retired-20260710 file the command read was a one-off manual operator rename during the single 2026-07-10 incident, not routine behavior. polylogue-04kl (the bead this served) is closed with 187,888 vectors recovered live 2026-07-28; its own notes explicitly say the remaining 528 partial + 8,949 non-fully-rescuable sessions \"still need real API embedding (separate from this rescue path)\" -- i.e. this exact command does not serve them. Removed polylogue/cli/commands/maintenance/_embeddings_rescue.py, polylogue/storage/embeddings/rescue.py, CLI registration, 3 test files, and the docs/maintenance.md section.\n\nblob-reference-replace-from-source: KEPT as a manual CLI command; did NOT wire into daemon automation. Live archive check (2026-08-02, /realm/db/polylogue): 0 missing / 160,609 references / 104,026 distinct blobs -- confirms rn5jh's earlier zero reading. But git history shows this class of debt is not purely theoretical: 2026-06-26, a verified production backup found 39,586 missing referenced blobs (PR #2422 \"report missing referenced blob debt\"), diagnosed and repaired same-day via classify/direct-restore/replace-from-source (#2423, #2425, #2426, #2427) -- the exact deterministic function this bead asked about. It has held at 0 for 5+ weeks since that incident. Read as \"rare, real, not currently recurring\" rather than \"never happened\" or \"actively ongoing\" -- built the lightweight always-on detector the notes asked for rather than full automation: added _check_blob_reference_debt_expensive to polylogue/daemon/health.py's EXPENSIVE tier (reuses the same read-only scan_blob_reference_debt scanner `polylogue ops backup` already runs), OK at 0 missing, ERROR otherwise with a pointer to the existing classify command. This closes the actual gap (nothing previously alerted on recurrence outside a manual backup run) without building daemon-convergence wiring for a problem that hasn't recurred in 5+ weeks of live operation.\n\nVerification: devtools test (55 passed, includes 2 new OK/ERROR-path tests + anti-vacuity check that removing the health-check wiring makes test_expensive_tier_inventory_pinned fail); devtools verify --quick exit 0; 2 pre-existing unrelated test_archive_maintenance_cli.py failures confirmed via git stash to reproduce on origin/master.\n\nPR: https://github.com/Sinity/polylogue/pull/3585","status":"closed","priority":1,"issue_type":"task","owner":"ezo.dev@gmail.com","created_at":"2026-08-02T16:41:01Z","created_by":"Sinity","updated_at":"2026-08-02T20:19:19Z","closed_at":"2026-08-02T20:19:19Z","close_reason":"PR #3585 merged: embeddings-rescue confirmed one-time migration (job done, table deleted); blob-reference debt guarded loudly via new EXPENSIVE health-check tier (_check_blob_reference_debt_expensive) reusing scan_blob_reference_debt — fails loud if debt reappears, currently zero live. No daemon automation wiring needed since both were confirmed non-recurring/already-zero rather than needing ongoing automation.","dependency_count":0,"dependent_count":0,"comment_count":0} From 7fa6b2309a9bbc0476fc4d0866370a40ab90fcd1 Mon Sep 17 00:00:00 2001 From: Sinity Date: Mon, 10 Aug 2026 09:26:34 +0200 Subject: [PATCH 2/2] chore(beads): preserve raw-authority successor provenance --- .beads/issues.jsonl | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/.beads/issues.jsonl b/.beads/issues.jsonl index aac4b981d1..7d10a8f129 100644 --- a/.beads/issues.jsonl +++ b/.beads/issues.jsonl @@ -213,7 +213,7 @@ {"_type":"issue","id":"polylogue-tfzw0","title":"storage: hook-event blob_refs born orphaned (del raw_id) - 73,427 refs / ~1.94 GiB unreclaimable and invisible to GC","description":"Structural audit H3 (/realm/data/derived/reports/polylogue-structural-audit-2026-08-03.html). _insert_hook_event (archive_tiers/source_write.py:1176-1177) starts with 'del raw_id' while its caller writes blob_refs(ref_type=raw_payload, ref_id=raw_id) for the payload; nothing joins the ref to raw_hook_events (which has NO blob_hash column - payload lives inline as payload_json, so the blob is a duplicate copy nothing reads back). Live: 73,427/116,149 raw_payload refs resolve to no raw_sessions row; ~69,249 hashes / 1.94 GiB with no live referent, growing since 06-29. blob_gc._still_referenced (blob_gc.py:150-221) is a MEMBERSHIP test on blob_refs so these are retained forever, uncounted. Design fork (hook blobs were retained deliberately in the de-inflation, PR #3265 era): (a) first-class retained ref class: new ref_type hook_payload + blob_hash column on raw_hook_events, GC liveness becomes a per-ref_type JOIN; or (b) declare payload_json the record, delete orphan refs, reclaim 1.94 GiB. Either way: make GC liveness a join not membership, add a standing blob_refs-liveness census metric. Related: polylogue-feu0 (same cross-tier reference class gap).","notes":"2026-08-03 invariant I3 run: reference-liveness violation is not hook-events-only - 1,336 blob_refs rows with ref_type='attachment' have no matching raw_artifacts row either. Strengthens the join-not-membership GC redesign: the census must cover every ref_type. Also fold: invariant I8 found 1 session with drifted sessions.message_count vs actual messages count (projection drift, likely lineage tail-extraction related) - investigate while in the area or split out if unrelated.","status":"closed","priority":1,"issue_type":"bug","owner":"ezo.dev@gmail.com","created_at":"2026-08-03T05:07:18Z","created_by":"Sinity","updated_at":"2026-08-06T19:24:35Z","closed_at":"2026-08-06T19:24:35Z","close_reason":"Closed after current-master audit: PR #3847 landed the unified production hook, attachment, sidecar, and unknown-evidence blob-reference liveness map and focused production-route tests. Remaining live blob reconciliation is owned by the reindex proof graph.","dependency_count":0,"dependent_count":0,"comment_count":0} {"_type":"issue","id":"polylogue-qhk8z","title":"PR #3574 duplicate-chain links trip the pre-existing ActiveByteRevisionChainError membership-census guard","description":"Discovered during polylogue-id4n's fresh triage pass. 2 tests fail:\n\n- tests/unit/sources/test_revision_backfill.py::test_backfill_content_cache_across_pages_reduces_parses_and_matches_uncached_archive\n- tests/unit/storage/test_rebuild_paging_content_order.py::test_rebuild_content_order_paging_dedups_first_time_classification_via_content_cache\n\nBoth fail with:\n\n polylogue.storage.sqlite.archive_tiers.revision_governance.ActiveByteRevisionChainError:\n an active byte-revision chain cannot move to membership governance\n\nRoot cause: a genuine cross-feature interaction between two independent,\nindividually-correct changes.\n\n1. PR #3574 (fix(storage): collapse byte-equal duplicates before revision-chain\n proof) added a \"duplicate\" relation to HistoricalRevisionDecision: two\n byte-identical raws (same content, different acquisition path -- the ordinary\n \"re-exported the same conversation\" shape both failing tests construct)\n now get one classified as representative and the other linked to it via\n predecessor_raw_id/baseline_raw_id, mirroring the representative's verdict.\n This is correct and intentional (fixes 50GB of over-quarantined content on\n the live archive).\n\n2. The pre-existing (#3406, long-standing) membership-census guard in\n revision_governance.py's _replace_full_revision_governance requires that a\n raw being promoted to membership governance have NO other raw pointing at\n it via predecessor_raw_id/baseline_raw_id (\"an active byte-revision chain\n cannot move to membership governance\") -- this guard was written assuming\n only genuine incremental append chains create such links.\n\n#3574 now also creates predecessor/baseline links for the DUPLICATE case, which\nthe membership-census guard was never designed to distinguish from a genuine\nin-progress append chain. A backfill that re-parses a raw touched by this\nguard after #3574's dedup linking now hits ActiveByteRevisionChainError where\nit previously succeeded.\n\nConfirmed via git log -S \"ActiveByteRevisionChainError\" that the guard's own\ncode is unchanged since #3406 -- the trigger is #3574's newly-created links,\nnot the guard itself. Confirmed via git show 31614661f that #3574's own\nverification section did not exercise this specific backfill-then-membership-\ncensus interaction (it ran tests/unit/storage/test_raw_revision_authority.py\nand a `-k \"raw_revision or revision_governance or raw_authority\"` selection,\nwhich apparently does not include these two files).\n\nNeeds design judgment: should the membership-census guard learn to\ndistinguish \"duplicate\" relation links (safe to promote past) from genuine\nincremental chain links (unsafe), or should #3574's duplicate-linking be\nscoped to skip cohorts that would trip this guard? Not attempted as a quick\nfix given the sensitivity of this subsystem (raw-authority correctness,\nquarantine-as-absorbing-state history) -- reproduction is solid, fix\ndirection needs an operator/maintainer decision.\n\nReproduction: devtools test tests/unit/sources/test_revision_backfill.py::test_backfill_content_cache_across_pages_reduces_parses_and_matches_uncached_archive tests/unit/storage/test_rebuild_paging_content_order.py::test_rebuild_content_order_paging_dedups_first_time_classification_via_content_cache","status":"closed","priority":1,"issue_type":"bug","owner":"ezo.dev@gmail.com","created_at":"2026-08-03T00:08:35Z","created_by":"Sinity","updated_at":"2026-08-03T10:30:25Z","closed_at":"2026-08-03T10:30:25Z","close_reason":"Fixed: PR #3616 (exclude byte-identical duplicates from revision baseline tie-break). Both named regression tests pass.","dependency_count":0,"dependent_count":1,"comment_count":0} {"_type":"issue","id":"polylogue-t73c2","title":"Fix the dogfood loop: polylogue's own archive can't answer 'what did agents do' questions","description":"Polylogue's whole thesis is that the archive answers 'what did agents do'. The fanout-operations report (2026-08-02) had to grep 700MB of raw session/subagent JSONL by hand because polylogue itself could not answer these questions: live index.db is at schema v46 with v53 code deployed, recent days of sessions are not ingested, and the daemon has been deliberately held off mid-merge-train.\n\nThis is the SAME root cause as polylogue-9qnzy (P0, schema-currency gap blocking the planned reindex) -- not a separate bug, a direct consequence of it. This bead exists to make explicit the SECOND reason 9qnzy matters: it's not just blocking a planned reindex, it's actively preventing polylogue from dogfooding its own coordination data right now.\n\nOnce 9qnzy resolves and the reindex/daemon-restart sequence completes: make the coordinator dashboard (output-token ratio, dispatch counts, per-lane outcomes, model distribution) a standing polylogue query instead of a bespoke mining pass every time someone wants to know how a fanout session went. Depends on polylogue-9qnzy.","status":"open","priority":1,"issue_type":"task","owner":"ezo.dev@gmail.com","created_at":"2026-08-02T23:40:08Z","created_by":"Sinity","updated_at":"2026-08-02T23:40:08Z","dependencies":[{"issue_id":"polylogue-t73c2","depends_on_id":"polylogue-3bsrp","type":"relates-to","created_at":"2026-08-03T07:01:30Z","created_by":"Sinity","metadata":"{}"},{"issue_id":"polylogue-t73c2","depends_on_id":"polylogue-9qnzy","type":"blocks","created_at":"2026-08-03T01:40:20Z","created_by":"Sinity","metadata":"{}"},{"issue_id":"polylogue-t73c2","depends_on_id":"polylogue-ltfj9","type":"parent-child","created_at":"2026-08-03T01:40:09Z","created_by":"Sinity","metadata":"{}"}],"dependency_count":1,"dependent_count":0,"comment_count":0} -{"_type":"issue","id":"polylogue-tw4ar","title":"Raw-authority verdict: persist a cache table + wire daemon convergence (Phase 2 follow-up)","description":"Follow-up from polylogue-w6hql (PR #3593): project_raw_authority_verdicts (polylogue/storage/raw_authority_verdict_projection.py) currently recomputes verdicts on demand by re-running classify_historical_full_revision_streams against live blob storage every call -- correct but not cheap at scale (761K+ census_plans-era cohort sizes). This bead is to design and land a persisted raw_authority_verdicts cache table (additive migration, numbered under storage/sqlite/migrations/source/) plus wiring into DaemonConverger so the cache tracks new/reclassified cohorts without a full rescan each read. Needed before polylogue-ds4b4 item 4 (blob-GC invariant verification) can cheaply check verdicts at scale rather than via the on-demand read path.","design":"DESIGN (2026-08-03): remaining half only — the cache table + invalidation shipped in PR #3628 (migration 024). Build the DaemonConverger warm-keeping stage: a ConvergenceStage in daemon/convergence_stages.py with check (are there cohorts whose logical_source_key changed since their cached cohort_fingerprint, or never-cached cohorts?) and execute (recompute via project_raw_authority_verdicts and upsert the cache in bounded batches). Use false_means_pending to push remaining backlog into convergence_debt rather than blocking; main process is the sole writer — no worker-process computation of the byte-proof classifier. Invalidation is already content-keyed (cohort_fingerprint over (raw_id, revision_kind, blob_hash) rows, storage/raw_authority_verdict_cache.py) — the stage only needs to FIND stale/missing cohorts cheaply (e.g. join raw_sessions cohort fingerprints against cache rows), never trust elapsed time. Pitfall: append-kind cohorts raise NotImplementedError in the projection — the stage must skip them typed-visibly (count reported), not crash, until w6hql stage-3 extends coverage. Consumer readiness: ds4b4 item 4 reads through the cache once this stage keeps it warm.\n","acceptance_criteria":"1. A DaemonConverger stage exists (daemon/convergence_stages.py) that finds never-cached and fingerprint-stale cohorts and upserts raw_authority_verdicts in bounded batches, deferring backlog via false_means_pending; unit test through the real stage interface.\n2. Append-kind cohorts are skipped typed-visibly (reported count), not crashed on, until w6hql extends coverage.\n3. A repeated read (e.g. ds4b4-style GC invariant check) hits the cache (no classify_historical_full_revision_streams recompute) — proven by a test asserting call counts or receipts.\n4. Cache staleness is content-keyed only (cohort_fingerprint); no time-based trust. Verify: devtools test -k verdict_cache; devtools test -k convergence.","notes":"2026-08-03: PR #3628 shipped the persisted raw_authority_verdicts cache table + cohort-fingerprint invalidation (SOURCE_SCHEMA_VERSION 24, migration 024). Remaining scope: wiring a DaemonConverger stage to keep the cache warm proactively -- deliberately deferred per that PR's own body. Bead stays open for that remaining half.","status":"open","priority":1,"issue_type":"task","owner":"ezo.dev@gmail.com","created_at":"2026-08-02T22:08:58Z","created_by":"Sinity","updated_at":"2026-08-03T11:09:58Z","dependency_count":0,"dependent_count":3,"comment_count":0} +{"_type":"issue","id":"polylogue-tw4ar","title":"Raw-authority verdict: persist a cache table + wire daemon convergence (Phase 2 follow-up)","description":"Follow-up from polylogue-w6hql (PR #3593): project_raw_authority_verdicts (polylogue/storage/raw_authority_verdict_projection.py) currently recomputes verdicts on demand by re-running classify_historical_full_revision_streams against live blob storage every call -- correct but not cheap at scale (761K+ census_plans-era cohort sizes). This bead is to design and land a persisted raw_authority_verdicts cache table (additive migration, numbered under storage/sqlite/migrations/source/) plus wiring into DaemonConverger so the cache tracks new/reclassified cohorts without a full rescan each read. Needed before polylogue-ds4b4 item 4 (blob-GC invariant verification) can cheaply check verdicts at scale rather than via the on-demand read path.","design":"DESIGN (2026-08-03): remaining half only — the cache table + invalidation shipped in PR #3628 (migration 024). Build the DaemonConverger warm-keeping stage: a ConvergenceStage in daemon/convergence_stages.py with check (are there cohorts whose logical_source_key changed since their cached cohort_fingerprint, or never-cached cohorts?) and execute (recompute via project_raw_authority_verdicts and upsert the cache in bounded batches). Use false_means_pending to push remaining backlog into convergence_debt rather than blocking; main process is the sole writer — no worker-process computation of the byte-proof classifier. Invalidation is already content-keyed (cohort_fingerprint over (raw_id, revision_kind, blob_hash) rows, storage/raw_authority_verdict_cache.py) — the stage only needs to FIND stale/missing cohorts cheaply (e.g. join raw_sessions cohort fingerprints against cache rows), never trust elapsed time. Pitfall: append-kind cohorts raise NotImplementedError in the projection — the stage must skip them typed-visibly (count reported), not crash, until w6hql stage-3 extends coverage. Consumer readiness: ds4b4 item 4 reads through the cache once this stage keeps it warm.\n","acceptance_criteria":"1. A DaemonConverger stage exists (daemon/convergence_stages.py) that finds never-cached and fingerprint-stale cohorts and upserts raw_authority_verdicts in bounded batches, deferring backlog via false_means_pending; unit test through the real stage interface.\n2. Append-kind cohorts are skipped typed-visibly (reported count), not crashed on, until w6hql extends coverage.\n3. A repeated read (e.g. ds4b4-style GC invariant check) hits the cache (no classify_historical_full_revision_streams recompute) — proven by a test asserting call counts or receipts.\n4. Cache staleness is content-keyed only (cohort_fingerprint); no time-based trust. Verify: devtools test -k verdict_cache; devtools test -k convergence.","notes":"2026-08-03: PR #3628 shipped the persisted raw_authority_verdicts cache table + cohort-fingerprint invalidation (SOURCE_SCHEMA_VERSION 24, migration 024). Remaining scope: wiring a DaemonConverger stage to keep the cache warm proactively -- deliberately deferred per that PR's own body. Bead stays open for that remaining half.","status":"open","priority":1,"issue_type":"task","owner":"ezo.dev@gmail.com","created_at":"2026-08-02T22:08:58Z","created_by":"Sinity","updated_at":"2026-08-03T11:09:58Z","dependencies":[{"issue_id":"polylogue-tw4ar","depends_on_id":"polylogue-fbkr","type":"discovered-from","created_at":"2026-08-10T07:26:19Z","created_by":"Sinity","metadata":"{}"}],"dependency_count":0,"dependent_count":3,"comment_count":0} {"_type":"issue","id":"polylogue-gysk3","title":"message_identity_hash reads position-derived provider_message_id fallback (attachment-class bug, unfixed for messages)","description":"Found during polylogue-ds4b4 item 3 investigation (position-derived synthetic\nidentity audit). polylogue-hith/qkuq already found and fixed exactly this\nclass of bug for attachments: a synthetic id seeded partly by array index\n(`att-\u003chash(message_id:name:index)\u003e`) is unstable across export vintages that\nreorder/insert array entries, causing false divergence in revision-authority\nmembership comparison. The fix there was NOT to remove the synthetic\ngenerator (still used as a last-resort storage/display id, seed no longer\nincludes index) but to make `attachment_identity_hash`\n(polylogue/pipeline/ids.py:254) stop reading either the real or synthetic\nattachment id at all -- it hashes only (message_id, name, mime_type).\n\nThe message-level sibling, `message_identity_hash` (polylogue/pipeline/ids.py:212),\nhas the analogous doc claim (\"A provider's own message id is stable across\nre-exports even when the export's array ordering is not\") but no such\nexclusion mechanism -- it hashes the message's `id` directly, and that id\nIS `provider_message_id`, which multiple parsers construct as\n`f\"msg-{index}\"`/`f\"{record_type}-{index}\"` when the raw record carries no\nnative id of its own:\n\n - polylogue/sources/parsers/claude/common.py:912-914 -- `f\"msg-{index}\"`\n - polylogue/sources/parsers/claude/code_parser.py:1620 -- `str(record_uuid or f\"msg-{index}\")`\n - polylogue/sources/parsers/codex.py:1592,1895,1931,2010,2200,2385 --\n `f\"function-call-{index}\"`, `f\"function-call-output-{index}\"`,\n `f\"reasoning-{index}\"`, `f\"{record_type}-{index}\"`,\n `f\"compaction-summary-{idx}\"`\n - polylogue/sources/parsers/local_agent.py:205,252 -- `f\"msg-{index}\"`\n - polylogue/sources/parsers/grok.py:139 -- `f\"{fallback_id}:{index}\"` (unconditional, no real id ever present)\n - polylogue/sources/parsers/drive.py:345 -- `f\"chunk-{idx}\"`\n - polylogue/sources/parsers/chatgpt.py:604 -- `f\"msg-{idx}\"`\n - polylogue/sources/parsers/base_support.py:360 -- `f\"msg-{idx}\"` (shared segment-message builder)\n - polylogue/sources/parsers/antigravity.py:414 -- `f\"{cascade_id}:{index}:{_message_kind(heading)}\"`\n\nUnlike attachments, there is no separate field to fall back to for messages\n-- `provider_message_id` IS the sole identity input by construction\n(`message_identity_hash(*, id: str)`'s fixed keyword-only signature), so the\n\"exclude both real and synthetic id\" fix pattern used for attachments\ndoesn't directly transplant. This needs its own design: likely a\ncomparison-identity axis that anchors on structural position (index within\nthe message array) ONLY when no provider-native id exists, combined with a\ncontent-similarity fallback, or an explicit typed\n\"positionally-anchored, not identity-anchored\" marker threaded through\nsession_revision_membership.py's comparison so a reorder is detected as\n\"can't prove sameness\" rather than silently comparing wrong pairs as if\nmessage ids matched.\n\nNot fixed in polylogue-ds4b4's session: this is genuinely new-discovered\ndebt, and reworking a `pipeline/ids.py` core identity function used\narchive-wide is a substantial, high-risk change (touches every provider's\ncontent-hash/revision-membership comparison) that deserves its own\ndedicated, unhurried session with its own regression-test design -- not a\nrushed fix bundled into an unrelated raw-authority-Phase-3 PR. ds4b4's own\nscope was a preventive LINT for this pattern (shipped separately, flags\nfuture occurrences of position-derived identity construction), not fixing\nevery existing instance.\n\nRef polylogue-ds4b4 (raw-authority redesign Phase 3, item 3)","status":"closed","priority":1,"issue_type":"bug","owner":"ezo.dev@gmail.com","created_at":"2026-08-02T19:36:47Z","created_by":"Sinity","updated_at":"2026-08-03T08:14:39Z","closed_at":"2026-08-03T08:14:39Z","close_reason":"Fixed (in-scope instance): PR #3604 (00e40ef30). _message_comparison_id prefers real provider_message_id, falls back to role+timestamp content anchor instead of position index; only falls back to position when neither exists. Red-first test. NOTE: root cause (parsers baking position-derived ids directly into provider_message_id) is tracked separately - all 18 call sites already acked in docs/plans/position-derived-identity-acks.json referencing this bead.","dependency_count":0,"dependent_count":2,"comment_count":0} {"_type":"issue","id":"polylogue-4zqh3","title":"Acquire the hermes-comparison recovery packet: shared-page decode + sole-copy attachment payloads","description":"A recovery bundle now lives at /realm/data/exports/chatlog/raw/recovery/hermes-project-comparison-2026-07/ (moved from /realm/inbox 2026-08-02; README + SHA256SUMS inside). Two capture gaps, verified against the live archive read-only on 2026-08-02: (1) the ChatGPT shared-page decode chatgpt-shared-decode-6a4ac87b/ (conversation 6a4ac87b, title 'Project and Codebase Analysis', 948 messages, decoded from the share-page React Router stream into messages.json + md + raw html) matches NO raw_sessions row by native_id — the session is entirely absent from the archive and the decode format has no parser; (2) the Claude.ai session claude-ai-export:2c2eab57-fc6c-4c61-99fa-f61af3b7ac57 IS acquired+indexed (raw 8e622747..., quarantined) but 70 of its 83 attachment refs are unfetched with 0 bytes, and the actual payload bytes sit only in this packet (hermes-agent-main.zip 57,853,455 B sha256 31267de3..., hermes-agent-all.tar.gz 257,839,520 B sha256 1b4eba44..., full ChatGPT temporary transcript 309,149 B sha256 db526f41... — none of the three hashes exist in the blob store). Wanted: an ingest path for the decode (or a one-off import), and attachment-byte acquisition from local packet files so the unfetched refs become acquired blobs. The packet is the sole copy of these bytes; exclude from any prune.","design":"DESIGN (2026-08-03): two independent acquisitions from the sole-copy recovery packet (/realm/data/exports/chatlog/raw/recovery/hermes-project-comparison-2026-07/; README + SHA256SUMS; EXCLUDE FROM ANY PRUNE — sole copy):\n1. ChatGPT shared-page decode (conversation 6a4ac87b, 948 messages, messages.json + md + raw html): the decode format has no parser. Options: (a) a narrow detector + parser for the decoded messages.json shape at the document-tightness level in sources/dispatch.py (durable: future share-page decodes ingest too); (b) one-off import via a conversion script mapping the decode into an existing admitted shape. Prefer (a) if the messages.json shape is close to chatgpt-export's mapping node structure (likely, it was decoded from the share-page React Router stream); the parser reuses chatgpt.py's message lowering. Origin question: it is a chatgpt conversation — admit as chatgpt-export with acquisition evidence marking the share-page provenance, not a new origin.\n2. Attachment-byte backfill for claude-ai-export:2c2eab57... (70/83 refs unfetched, 0 bytes; the three packet payloads' sha256 absent from the blob store): an acquisition path that matches local packet files to unfetched attachment refs (by declared hash where the export carries one, else by operator-asserted mapping recorded as evidence) and publishes blobs + flips acquisition_status to acquired. Reuses the attachment blob-write path from #2469 (_acquire_attachment_blob/_write_attachments); the raw row is quarantined — attachment acquisition must not depend on the session's authority state.\nBoth feed f1vg's attachment-fidelity and absence buckets; record before/after bucket counts.\n","acceptance_criteria":"1. The shared-page decode conversation (6a4ac87b, 948 messages) is ingested and queryable (as chatgpt-export with share-page provenance evidence), via parser or documented one-off import; raw bytes + provenance recorded in source.db.\n2. The claude-ai session's 70 unfetched attachment refs become acquired blobs with true SHA-256s from the packet payloads (the three named hashes present in the blob store); acquisition does not depend on the raw row's authority state.\n3. f1vg's absence and attachment-fidelity buckets drop accordingly (before/after recorded).\n4. The packet directory is protected from prune/cleanup (noted in its README or the owning inventory).\n5. Verify: devtools test -k chatgpt or -k attachments for new paths; read-only live queries for ref status.","notes":"Footprint: polylogue/sources/dispatch.py, polylogue/sources/parsers/hermes_spans.py, polylogue/sources/parsers/chatgpt.py (recovery-packet ingestion: ChatGPT shared-page decode + hermes sole-copy attachment payloads).","status":"open","priority":1,"issue_type":"task","owner":"ezo.dev@gmail.com","created_at":"2026-08-02T18:27:09Z","created_by":"Sinity","updated_at":"2026-08-03T11:12:52Z","dependency_count":0,"dependent_count":1,"comment_count":0} {"_type":"issue","id":"polylogue-rn5jh","title":"Wire blob-reference-replace-from-source + embeddings-rescue into daemon automation (mislabeled as 'needs judgment')","notes":"2026-08-02 correction after operator pushback: sharper findings than the original AC framing.\n\nembeddings-rescue: polylogue-04kl (the bead this command was built for) is CLOSED -- the actual rescue already happened (187,888 vectors recovered from the one specific 2026-07-10 retired tier, real production win, done). The command remains in the CLI as generic '--source \u003cany path\u003e' product surface for a scenario that occurred exactly once and is finished. Revised AC: DELETE this command (not 'automate it') unless investigation finds embeddings-tier retirement is a routine recurring event (check EMBEDDINGS_SCHEMA_VERSION bump history/frequency) that would need it again -- if genuinely recurring, THEN build the registry+automation described below; if it was truly one-time, it should simply be removed as dead product surface once 04kl's specific job is confirmed fully done (528 partial sessions + ~8949 non-fully-rescuable sessions were noted as still needing real API embedding in 04kl's own notes -- confirm those don't still need this exact command before deleting).\n\nblob-reference-replace-from-source: checked live production archive directly (2026-08-02): 'polylogue ops maintenance blob-reference-debt' reports 160,609 references / 104,026 distinct blobs / 0 missing / status=ok. There is currently ZERO active debt for this command to repair -- the acquire-blob/commit-row safety invariants (leases + snapshot reference check, per CLAUDE.md) appear to be holding in practice right now, not just in theory. This changes the framing: this isn't a live ongoing backlog needing automation urgently -- it's a rare/historical-scenario tool sitting at zero. Revised AC: (a) confirm whether missing-blob-ref debt has EVER been nonzero on this archive historically (check gc_generations/blob GC pass history, or absence of evidence either way), (b) if it's genuinely rare/historical, a read-only periodic check (fails loud if debt ever appears, rather than silent accumulation) may be sufficient instead of full automation -- don't over-build automation for a zero-occurrence problem, (c) if investigation finds real recurring instances, THEN wire the deterministic replace-from-source logic into daemon convergence as originally scoped.\n2026-08-02 RESOLVED via PR #3585 (branch feature/refactor/retire-embeddings-rescue-guard-blob-debt).\n\nembeddings-rescue: DELETED as dead product surface, confirmed one-time. No mechanism in the codebase ever preserves a retired embeddings.db (reset --database hard-deletes it; storage/sqlite/archive_tiers/embeddings.py has had 4 EMBEDDINGS_SCHEMA_VERSION bumps since inception, none of which route through a \"retired file\" path) -- the embeddings.db.v2-retired-20260710 file the command read was a one-off manual operator rename during the single 2026-07-10 incident, not routine behavior. polylogue-04kl (the bead this served) is closed with 187,888 vectors recovered live 2026-07-28; its own notes explicitly say the remaining 528 partial + 8,949 non-fully-rescuable sessions \"still need real API embedding (separate from this rescue path)\" -- i.e. this exact command does not serve them. Removed polylogue/cli/commands/maintenance/_embeddings_rescue.py, polylogue/storage/embeddings/rescue.py, CLI registration, 3 test files, and the docs/maintenance.md section.\n\nblob-reference-replace-from-source: KEPT as a manual CLI command; did NOT wire into daemon automation. Live archive check (2026-08-02, /realm/db/polylogue): 0 missing / 160,609 references / 104,026 distinct blobs -- confirms rn5jh's earlier zero reading. But git history shows this class of debt is not purely theoretical: 2026-06-26, a verified production backup found 39,586 missing referenced blobs (PR #2422 \"report missing referenced blob debt\"), diagnosed and repaired same-day via classify/direct-restore/replace-from-source (#2423, #2425, #2426, #2427) -- the exact deterministic function this bead asked about. It has held at 0 for 5+ weeks since that incident. Read as \"rare, real, not currently recurring\" rather than \"never happened\" or \"actively ongoing\" -- built the lightweight always-on detector the notes asked for rather than full automation: added _check_blob_reference_debt_expensive to polylogue/daemon/health.py's EXPENSIVE tier (reuses the same read-only scan_blob_reference_debt scanner `polylogue ops backup` already runs), OK at 0 missing, ERROR otherwise with a pointer to the existing classify command. This closes the actual gap (nothing previously alerted on recurrence outside a manual backup run) without building daemon-convergence wiring for a problem that hasn't recurred in 5+ weeks of live operation.\n\nVerification: devtools test (55 passed, includes 2 new OK/ERROR-path tests + anti-vacuity check that removing the health-check wiring makes test_expensive_tier_inventory_pinned fail); devtools verify --quick exit 0; 2 pre-existing unrelated test_archive_maintenance_cli.py failures confirmed via git stash to reproduce on origin/master.\n\nPR: https://github.com/Sinity/polylogue/pull/3585","status":"closed","priority":1,"issue_type":"task","owner":"ezo.dev@gmail.com","created_at":"2026-08-02T16:41:01Z","created_by":"Sinity","updated_at":"2026-08-02T20:19:19Z","closed_at":"2026-08-02T20:19:19Z","close_reason":"PR #3585 merged: embeddings-rescue confirmed one-time migration (job done, table deleted); blob-reference debt guarded loudly via new EXPENSIVE health-check tier (_check_blob_reference_debt_expensive) reusing scan_blob_reference_debt — fails loud if debt reappears, currently zero live. No daemon automation wiring needed since both were confirmed non-recurring/already-zero rather than needing ongoing automation.","dependency_count":0,"dependent_count":0,"comment_count":0}