diff --git a/.beads/issues.jsonl b/.beads/issues.jsonl index 93465a9924..e5fd82e530 100644 --- a/.beads/issues.jsonl +++ b/.beads/issues.jsonl @@ -1,3 +1,5 @@ +{"_type":"issue","id":"polylogue-4ma3","title":"paths.archive_root() ignores polylogue.toml, splitting the archive root","description":"polylogue/paths/_roots.py:archive_root() resolves POLYLOGUE_ARCHIVE_ROOT from\nthe environment only and never consults polylogue.toml's [archive] root, even\nthough polylogue/config.py documents and implements a 5-layer resolution\n(default, site TOML, user TOML, env, CLI) that DOES honour it.\n\nConsequence: any process without POLYLOGUE_ARCHIVE_ROOT set in its own\nenvironment (bare CLI invocations, hook writers, the browser-capture\nreceiver, ad hoc scripts) silently falls back to XDG_DATA_HOME/polylogue\ninstead of the operator's configured root (e.g. /realm/db/polylogue),\nsplitting archive state across two directories that nothing reconciles.\n\nMeasured live damage before the fix: 108,094 files (2.2 GB) accumulated\nin ~/.local/share/polylogue/hooks/pending/ since 2026-07-14 while the\ndaemon (which does get POLYLOGUE_ARCHIVE_ROOT from its systemd unit) drained\n/realm/db/polylogue/hooks/pending/ instead -- nothing processed the XDG-root\nbacklog. Browser-capture spool and inbox/ content were also split across\nboth roots at different times depending on which process's environment\nhappened to have the override set.\n\nFix: polylogue.config gained resolve_archive_root() (same layered precedence\nas load_polylogue_config, extracted so paths._roots can reuse it via a lazy\nfunction-local import without an import cycle -- config.py already imports\npolylogue.paths for GEMINI_DRIVE_FOLDER). paths.archive_root() now checks\nPOLYLOGUE_ARCHIVE_ROOT first (fast path, no config import) and falls back to\nresolve_archive_root() (site/user TOML, then XDG default) when unset.\nNothing is cached, preserving per-test POLYLOGUE_ARCHIVE_ROOT isolation.\n\nExplicitly out of scope for this fix: migrating the ~176K files already\nmisplaced under the XDG root (hooks pending+acknowledged, browser-capture\nspool, inbox) -- that is a separate data-migration lane.","status":"in_progress","priority":0,"issue_type":"bug","assignee":"Sinity","owner":"ezo.dev@gmail.com","created_at":"2026-07-31T03:49:09Z","created_by":"Sinity","updated_at":"2026-07-31T03:49:18Z","started_at":"2026-07-31T03:49:18Z","comments":[{"id":"019fb653-c632-716f-9aa0-5cbc7b2faaac","issue_id":"polylogue-4ma3","author":"Sinity","text":"Fixed via PR #3414 (branch feature/fix/archive-root-honours-config, commit e9e7a7245). paths.archive_root() now falls back to polylogue.config.resolve_archive_root() (site/user TOML archive.root) when POLYLOGUE_ARCHIVE_ROOT is unset, instead of silently defaulting to XDG_DATA_HOME/polylogue. Verified: devtools test on tests/unit/core/test_paths.py (new TestArchiveRootHonoursConfigFile suite, 25 passed), test_config_resolution_regression.py (9 passed), plus config/cli-paths/browser-capture-token/hook-spool suites (143 passed); devtools verify --quick green. Data migration of the ~176K files already misplaced under the XDG root (hooks pending+acknowledged, browser-capture spool, inbox) is explicitly out of scope -- needs a separate follow-up.","created_at":"2026-07-31T03:59:31Z"}],"dependency_count":0,"dependent_count":0,"comment_count":1} +{"_type":"issue","id":"polylogue-geop","title":"newer chatgpt exports are NOT supersets - April holds 33% more messages than July","description":"MEASURED 2026-07-31, comparing chatgpt-data-2026-04-23 against chatgpt-data-2026-07-29 over the 2,094 conversations present in BOTH.\n\n April 109,657 messages total / 97,403 in the common set\n July 72,981 messages total / 44,834 in the common set\n EVERY ONE of the 2,094 common conversations lost messages. Not one gained.\n\nNot deletion, not branch pruning (July's current_node path count is also far\nbelow April's), and not head/tail truncation (survivors are spread across the\nfull 0-100% index range with identical date spans). OpenAI DROPPED WHOLE\nCATEGORIES between export generations:\n\n content_type April July delta\n code 20,384 0 -20,384\n computer_output 8,192 0 -8,192\n execution_output 6,816 0 -6,816\n tether_browsing_display 1,399 0 -1,399\n tether_quote 1,178 0 -1,178\n system_error 177 0\n sonic_webpage 30 0\n citable_code_output 8 0\n text 37,829 24,890 -12,939\n multimodal_text 1,457 694 -763\n user_editable_context 821 1 -820\n thoughts 17,374 17,506 +132 (retained)\n reasoning_recap 1,738 1,743 +5 (retained)\n\n role\n tool 24,914 0 -24,914 \u003c- the ENTIRE tool layer\n system 5,099 0 -5,099\n assistant 54,513 32,839 -21,674\n user 12,877 11,995 -882\n\nThe whole code-interpreter / tool-use / browsing layer is absent from the newer\nexport. This also explains why model-produced sandbox files carry no file id in\nthe July data (polylogue-dt5s): the tool messages that created them are gone.\n\nCONSEQUENCES - these change import strategy, not just this one file:\n\n1. A newer export can be a STRICT SUBSET of an older one. 'Latest wins' is\n wrong for this provider. Coalescing must be a per-message UNION keyed on\n message id, with each export treated as a partial observation.\n2. The April 2026 and Oct 2025 exports are NOT superseded and must never be\n pruned as redundant. They are the only surviving record of 24,914 tool\n messages and 20,384 code blocks.\n3. This is precisely the aggz/superset question the operator raised for\n aistudio, now confirmed with hard numbers on a second provider: neither\n revision is a superset, so any model that must pick ONE winner loses data.\n The content-only comparison relation (#3401) must classify this pair as\n 'conflict', not 'contains' in either direction.\n4. Absence detection should compare across export generations per message id,\n not per conversation - a conversation present in both looked fine at\n session granularity while silently losing 78% of its messages.\n\nAC: importing all three chatgpt exports yields the UNION of their messages;\na conversation present in several exports carries every message any export\nobserved; and a regression test pins that the newer-export-is-subset case\ndoes not delete previously-ingested messages.","notes":"VERIFIED THREE WAYS (2026-07-31) after the finding was challenged as implausible for a GDPR export.\n\n1. THE EXPORT IS COMPLETE AS DELIVERED. Checked every file against the export's\n own export_manifest.json: 3,266 declared files, 3,266 present, ZERO missing,\n ZERO size mismatches, 18.091 GB declared vs 18.092 GB actual (delta is the\n manifest itself, which is not self-declared). So the loss is not download\n corruption, not truncation from the 5 stalled resumes, and not extraction\n error. It is what OpenAI shipped.\n\n2. IT IS A FORMAT CHANGE, NOT RETENTION AGE-OUT. Conversations created as\n recently as 2026-07-27 - two days before the export was generated - also\n contain ZERO tool-role and ZERO system-role messages. Across the ENTIRE July\n export the only roles present are assistant (59,728) and user (13,253).\n A retention window would have spared recent conversations; it did not.\n\n3. THE TOOL LAYER IS NOT HIDING IN chat.html EITHER. grep over the 221 MB\n chat.html: execution_output 0, computer_output 0, tether_quote 0. The\n rendered view carries no more than the JSON.\n\nWHAT APRIL STILL HAS (answers 'are the sandbox files in April then?' - yes):\n April non-json members 9,958 (vs 3,228 .dat in July)\n distinct file ids in member names 9,887\n file ids referenced INSIDE tool messages 10,453\n of those WITH bytes present 9,225 (88.2%)\n asset_pointer + metadata.attachments refs 3,189 distinct, 1,104 with bytes (34.6%)\n\n So in April the file ids live in the TOOL messages, which is exactly why\n July - having deleted the tool layer - cannot resolve model-produced files.\n April is the only record of ~9,225 attachment blobs.\n\nCONVERSATION-LEVEL COVERAGE IS ALSO NON-NESTED IN BOTH DIRECTIONS:\n in April but not July 309\n in July but not April 378 (some created as far back as 2023-02-14,\n i.e. April was ALSO missing old conversations)\n Neither export is a superset at conversation level either.\n\nCONTEXT FROM THE WEB: incomplete ChatGPT exports are a documented user\ncomplaint (community.openai.com/t/incomplete-data-export-with-conversations-json/1019950,\nNov 2024: a user's export dropped everything before 2024-10-28, 35MB -\u003e 4MB, no\nofficial response). The specific tool-layer removal is not publicly documented,\nso treat provider export completeness as untrusted and verify per generation.\nDECISIVE RESOLUTION RULE (2026-07-31). The union is not a heuristic merge - the two exports are in STRICT CONTAINMENT and there is no genuine disagreement anywhere in the corpus. Proven by field-walking all 44,171 messages present in both exports:\n\n field observations 748,209\n both set \u0026 AGREE 291,774\n both set \u0026 CONFLICT 2,479 (0.33%)\n only April 453,956\n only July 0 \u003c- July contributes NOTHING April lacks\n\nAnd the 2,479 'conflicts' are subsetting one level deeper, not disagreement.\nThey occur in exactly two fields - metadata.content_references (1,766) and\nmetadata.search_result_groups (713) - and inspecting them shows identical\nrecord COUNTS (29,528 both sides) and identical type distributions (file 8,543,\ngrouped_webpages 7,363, webpage_extended 6,239, hidden 4,889, attribution\n1,073, sources_footnote 951 - the same on both sides). What differs is the KEY\nSET of each citation record:\n\n April keys: alt end_idx error fallback_items items matched_text prompt_text\n refs safe_urls start_idx status style type\n July keys: alt fallback_items items prompt_text type\n\nJuly dropped end_idx, start_idx, matched_text, refs, safe_urls, error, status,\nstyle. Note start_idx/end_idx: July's citations LOST THEIR TEXT ANCHORS, which\nis the conceptual core of a citation.\n\nAlso lost from message.metadata between generations (top-level keys present in\nApril, absent in July): can_save, message_type, timestamp_, request_id,\ndefault_model_slug, CITATIONS (20,471 messages!), reasoning_status,\nturn_exchange_id, finish_details, is_complete. New in July: NONE.\nEnvelope fields nulled in July: status (finished_successfully -\u003e null, 42,000),\nweight (1.0 -\u003e null, 44,164), author.metadata removed - including\nreal_author='tool:web' on 237 messages.\n\nmessage CONTENT is byte-identical on all 44,171 common messages. Zero content\nconflicts.\n\nTHEREFORE the correct algorithm is deterministic and lossless, and needs no\nconflict policy at all:\n\n for each message id, and each field PATH (including inside nested citation\n records), take the value from whichever acquisition has one; where several\n have one they are equal; record which acquisition supplied each field.\n\n'Record the disagreement' is not needed for this provider pair because there IS\nno disagreement - only presence vs absence. This is a much stronger position\nthan the earlier framing and should be the default model for every origin:\ntreat an acquisition as a partial observation, merge at field-path granularity,\nand only escalate to a recorded conflict if two acquisitions ever assert\nDIFFERENT non-null values for the same path - which happened zero times here.","status":"in_progress","priority":0,"issue_type":"task","assignee":"Sinity","owner":"ezo.dev@gmail.com","created_at":"2026-07-31T01:10:03Z","created_by":"Sinity","updated_at":"2026-07-31T03:18:49Z","started_at":"2026-07-31T03:18:49Z","dependency_count":0,"dependent_count":0,"comment_count":0} {"_type":"issue","id":"polylogue-b508","title":"21% of index sessions are metadata sidecars materialized as conversations (agent-*.meta, toolu_*, wf_*)","description":"## What the data shows\n\nClassifying every `claude-code-session` session_id in the live index by the shape\nof its native_id:\n\n 8,586 (52.6%) parent:agent-* real subagent transcripts\n 4,945 (30.3%) \u003cagent\u003e.meta SIDECAR METADATA, not a conversation\n 2,762 (16.9%) uuid real top-level sessions\n 7 wf_* workflow ids\n 3 toolu_* TOOL-USE ids\n 16,312 total\n\nContent of the suspicious classes:\n\n .meta 4,945 sessions 0 with messages 4,286 with events\n toolu_ 3 sessions 0 with messages 0 with events\n wf_ 7 sessions 0 with messages 0 with events\n\nThe `.meta` rows originate from\n`~/.claude/projects/\u003cproject\u003e/\u003csession-uuid\u003e/subagents/agent-\u003cid\u003e.meta.json` --\na per-subagent metadata sidecar. 5,053 raws come from `*.meta.json` paths.\n\n**The real subagent transcript is separately and correctly ingested.** Sampled\n300 `.meta` sessions and looked for the corresponding `%:agent-\u003cid\u003e` session:\n300 of 300 found. So these are not the only record of anything; they are\nduplicate phantom rows standing beside the real session.\n\nNet effect: 4,945 of 23,230 index sessions -- **21% of the archive's session\ncount** -- are metadata sidecars materialized as conversations.\n\n## Why this matters beyond a wrong count\n\nThis is the same pathology as the hook-event inflation already fixed once\n(83,286 -\u003e 18,391 sessions, `write_hook_event`, PR #3265): a per-session sidecar\nrecord ingested as a standalone session. A different sidecar type, the identical\nbug class, and it survived that repair because the fix was specific to hook\nevents rather than to the category.\n\nConsequences that are not merely cosmetic:\n\n- Every per-session aggregate -- counts, cost rollups, activity timelines,\n \"how many sessions did I have\" -- is inflated by 21% for claude-code.\n- 659 of them carry neither messages nor events, so they are pure empty rows.\n- Search and read surfaces can return a `.meta` session that has no content to\n show.\n- `toolu_*` sessions mean a TOOL-USE id was promoted to a session identity,\n which indicates identity derivation falling back to whatever id it found\n rather than failing loudly.\n\n## Hypothesis for the mechanism (needs confirming before fixing)\n\nProvider detection / payload lowering treats any JSON document under a\n`subagents/` directory as a session-bearing payload, so a `.meta.json` sidecar\nis lowered into a `LoweredPayloadSpec` and parsed. `provider_session_id` then\nfalls back to the filename stem (`agent-\u003cid\u003e.meta`), producing a well-formed but\nmeaningless identity. The `toolu_*` and `wf_*` cases look like the same fallback\npicking up whichever id field is present in a fragment.\n\nThat should be verified in `sources/dispatch.py` and the Claude Code parser\nbefore any fix -- the shape above is inference from the data, not yet traced in\ncode.\n\n## Direction\n\nTwo candidate fixes, and the second is the one that matches\n`polylogue-aggz`'s spirit:\n\n1. Narrow: skip `*.meta.json` under `subagents/`, and attach its content to the\n subagent session it describes rather than to a session of its own.\n2. Structural: a payload may only become a session when it yields a session\n identity the PROVIDER asserted. A filename-derived or fragment-derived\n fallback identity should be a parse refusal, not a session. That kills\n `.meta`, `toolu_*` and `wf_*` in one rule, and prevents the next sidecar\n format from doing this again -- which is exactly what the hook-event fix\n failed to do.\n\nPrefer (2), with (1) only if (2) proves too broad. Under (2) this stops being a\ncategory anyone has to remember.\n\n## Acceptance criteria\n\n- No session exists whose identity was derived from a filename stem or a\n non-session fragment id.\n- The metadata carried by `*.meta.json` is still retained and attached to the\n subagent session it describes -- this must not become data loss.\n- Sampled `.meta` ids resolve to their real `%:agent-\u003cid\u003e` session, which keeps\n its content.\n- claude-code session count drops by roughly 4,945; verify against\n `.agent/scripts/corpus-fidelity-audit.py` that absences do NOT rise, i.e. that\n nothing real was removed.\n- Anti-vacuity: state the production line mutated and the resulting failure.\n\nRef polylogue-aggz\n","status":"in_progress","priority":0,"issue_type":"bug","assignee":"Sinity","owner":"ezo.dev@gmail.com","created_at":"2026-07-30T16:10:03Z","created_by":"Sinity","updated_at":"2026-07-30T16:48:33Z","started_at":"2026-07-30T16:48:33Z","labels":["area:ingest"],"comments":[{"id":"019fb3ee-7eaf-71b8-8e5a-d1d66efffce1","issue_id":"polylogue-b508","author":"Sinity","text":"## Traced mechanism (not the original hypothesis)\n\nThe original hypothesis (\"provider detection treats any JSON under\nsubagents/ as session-bearing, provider_session_id falls back to the\nfilename stem\") was PARTLY wrong and PARTLY right, in a way that matters.\n\n**Live daemon ingest path (sources/live/batch.py + pipeline/services/\ningest_worker.py) already refuses this correctly**, and has since well\nbefore this session (classify_artifact_path's agent-*.meta.json branch\ndates to 82fc0e4ff2, 2026-03-27; the OriginSpec artifact_rule_for_path\nroute that shadows it is newer but agrees). Proved empirically: built a\nthrowaway archive and ingested 9 REAL files pulled from\n~/.claude/projects (1 top-level session, 3 real agent-*.jsonl subagent\ntranscripts, 4 real agent-*.meta.json sidecars, 1 standalone\nmeta+transcript pair) through LiveBatchProcessor (same primitives\npolylogued run wires up) -- result: exactly 5 real sessions, 0 phantom\n`.meta` rows.\n\n**The actual live bug is a second, separate parse chokepoint**:\n`sources/revision_backfill.py` (`_parse_one`/`_parse_stream`, driving\n`polylogue ops reset --index` / the offline rebuild-index path via\n`backfill_historical_revision_evidence`) calls\n`dispatch.parse_payload`/`parse_stream_payload` on every retained raw\nUNCONDITIONALLY -- no OriginSpec/artifact-taxonomy gate at all. Reproduced\nlive: rebuilding an index from the same 9-file real corpus through this\npath (bypassing the daemon) produced 9 sessions, 4 of them phantom\n`claude-code-session:agent-\u003cid\u003e.meta` rows with 0 messages/0 events --\nthe EXACT reported shape. `fallback_id = Path(source_path).stem` on\n`agent-\u003cid\u003e.meta.json` strips only the trailing `.json`, leaving\n`agent-\u003cid\u003e.meta` -- literally the observed native_id.\n\nThis means the bead's suggested remediation (\"index.db is rebuildable,\nprefer a rebuild\") would have RECREATED the defect it was meant to fix,\nnot eliminated it -- this is now fixed (see below), so the plan below is\nsafe.\n\nA structural gap also existed independent of both mechanisms:\n`dispatch.py:_generic_messages_session` (the one payload-lowering branch\nwith zero provider-specific identity handling, reached both by genuinely\nunknown providers and by the Drive-like generic fallback) fell back to\n`fallback_id` -- a filename stem the *source-discovery walk* invented --\nwhenever a payload had a `messages` list but no `id` field. Didn't\nreproduce with real `.meta.json`/`toolu_*`/`wf_*` fixtures (those are\ncovered by the OriginSpec/artifact-taxonomy path rules), but is exactly\nthe \"next sidecar format\" risk the bead is about, and closing it is what\nimplements the structural rule generically rather than per-shape.\n\n## Fixes shipped (PR, branch feature/fix/provider-asserted-session-identity)\n\n1. `polylogue/sources/dispatch.py`: `_generic_messages_session` now\n requires the payload to assert its own `id`; absent that it refuses to\n parse (returns None) instead of synthesizing an identity from\n `fallback_id`.\n2. `polylogue/sources/revision_backfill.py`: `_parse_one`/`_parse_stream`\n now consult `artifact_rule_for_path` (same OriginSpec table batch.py\n already uses) and refuse to parse (return `[]`) when the declared\n artifact's `parse_policy` isn't `\"session\"`. One rule table, enforced\n at both entry points -- a rebuild and a live ingest now agree.\n\nBoth fixes proven with:\n- Unit regression tests\n (`tests/unit/sources/test_source_laws.py::test_parse_payload_generic_messages_without_asserted_id_refuses_to_parse`,\n `tests/unit/sources/test_revision_backfill.py::test_parse_one_refuses_declared_fact_artifacts`)\n that fail before the fix and pass after (anti-vacuity verified by\n reverting each fix in isolation and re-running).\n- The real 9-file fixture-corpus rebuild: 9 sessions / 4 phantom before\n fix #2, 5 sessions / 0 phantom after, with the 3 real subagent\n transcripts' message counts (96, 120, 164, 31... unaffected across the\n run) identical in both states -- no data loss to real content.\n- `devtools test tests/unit/sources/test_source_laws.py\n tests/unit/sources/test_revision_backfill.py` -- 180 passed.\n\n## AC: metadata retention (not data loss)\n\nAlready satisfied by existing, pre-existing code, unaffected by this fix:\n`insights/claude_workflow_materializer.py` +\n`insights/claude_workflow_evidence.py` read `agent_sidecar_meta` facts\nfrom retained raw bytes (independent of whether a `sessions` row exists)\nand materialize them into the `claude-workflow:*` work-evidence graph\n(run/invocation/attempt nodes with sidecar-meta claims attached). This\nfix only removes the DUPLICATE phantom `sessions` row; the raw bytes stay\nin `raw_sessions` (admitted as \"fact\" artifacts) and the metadata content\nkeeps flowing into that graph exactly as before.\n\n## `toolu_*` / `wf_*` (10 rows total, not separately reproduced)\n\n`wf_*` (workflow_run_snapshot, `.json`) is covered by the same\nOriginSpec-declared-fact gate as `.meta.json` -- fix #2 covers it\nstructurally, same mechanism.\n\n`toolu_*` (3 rows) could not be reproduced with real fixture data: real\n`tool-results/*.txt` sidecars are excluded from the live discovery walk\nby suffix filtering (`artifact_suffixes_for_provider` only allows\n`.json`/`.jsonl`/`.ndjson` for claude-code) and are NOT declared in\nOriginSpec at all, so if a `raw_sessions` row for one of these 3 exists\nin the live archive it's very likely a relic of an older\nacquisition-scope bug already superseded by that suffix filtering. Given\nthere are only 3 (vs 4,945 `.meta`), recommend: after the rebuild below,\ncheck whether they're gone; if any survive, file a narrow follow-up bead\nwith their actual `source_path`/payload shape rather than guessing\nfurther blind.\n\n## Verified live-data remediation procedure\n\nindex.db is the rebuildable tier; source.db (raw bytes) is durable and\nuntouched by this fix. With both fixes merged and deployed:\n\n1. Stop anything writing to the live archive (already stopped per the\n session's safety rule).\n2. `polylogue ops reset --index` -- wipes only the index tier (new\n generation), source.db/user.db/ops.db untouched.\n3. `polylogued run` (or the offline `devtools`/maintenance rebuild-index\n path) -- replays EVERY `raw_sessions` row from source.db through the\n now-fixed `revision_backfill.py` path. Verified there is no\n `parsed_at_ms`-style skip: `all_index_rebuild_raw_ids` selects every\n raw unconditionally and `RebuildIndexRequest(only_missing=False)`\n forces a full non-incremental replay; content-hash idempotency only\n skips re-writing a session that ALREADY EXISTS in the target, which is\n moot against a freshly wiped, empty index.db. So no additional\n durable-tier invalidation beyond the code fix is needed -- the raws\n ARE the source of truth and will now be reparsed correctly.\n4. Verify with `.agent/scripts/corpus-fidelity-audit.py` (or equivalent\n session-count query) that: claude-code session count drops by\n approximately 4,945+7(+ up to 3), absences do NOT rise (nothing real\n removed), and the 300-sample `.meta -\u003e %:agent-\u003cid\u003e` resolution check\n from the original investigation still resolves (the real subagent\n sessions are untouched by this fix -- it only removes the duplicate).\n\nNot run against the live archive per the session's explicit\ninstruction -- this is the procedure to execute, not evidence that it was\nexecuted.\n","created_at":"2026-07-30T16:49:39Z"}],"dependency_count":0,"dependent_count":0,"comment_count":1} {"_type":"issue","id":"polylogue-aggz","title":"Collapse the failure taxonomy into three invariants that make the cases unrepresentable","description":"## The problem with the current shape\n\nOne day of investigation produced eleven separately-named defects and found four\nexisting special-case code paths. That is a taxonomy, not an architecture. Every\nnew provider quirk becomes another named category, another branch, another bead,\nand the system's correctness becomes a function of how many cases someone\nremembered. The goal is the opposite: make these failures unrepresentable, so\nthey are not known as anything at all.\n\nAlmost all of it collapses into three invariants.\n\n## Invariant 1 -- comparison identity contains only content\n\n**A conversation is a SET of messages keyed by stable provider identity, each\ncarrying only content-bearing fields. Nothing else may enter the value used to\ncompare two acquisitions of it.**\n\nCollapses, as consequences rather than cases:\n\n- polylogue-bu1i (attachment acquisition state in attachment identity) --\n acquisition state is not content.\n- polylogue-c429 (message array order) -- a set has no order.\n- polylogue-nuec (chatgpt elapsed_duration_ms) -- a measurement is not content.\n- polylogue-hith (synthetic attachment id seeded on position) -- position is not\n identity.\n- polylogue-d8al (real-id presence varies between vintages) -- identity must be\n derivable from content when the provider omits its own.\n- polylogue-oycw (positional-prefix superset test) -- set containment, not\n sequence prefix.\n- `_provider_ordered_browser_snapshots` -- exists only because DOM ordering\n differs from export ordering. Under a set, it has nothing to fix.\n- The `superseded_prefix` / `superseded_equivalent` distinction -- both are just\n \"contained or equal\".\n\nSuperset-ness becomes total and decidable, with no residual category:\n\n equal same id set, equal content per id\n contains A's id set contains B's, equal content on the intersection\n conflict content differs on the intersection\n\nOrdering remains a stored, rendered property of a session. The claim is only\nthat it is not part of the comparison value. `_direct_export_precedence` (a real\nexport outranks a browser capture) probably survives as a genuine provenance\nrule rather than a repair.\n\n## Invariant 2 -- one chokepoint may write a session\n\n**It must be structurally impossible to materialize a session without consulting\nrevision authority.**\n\npolylogue-c737, PR #3397 and PR #3398 all exist because two write paths each\ncarried their own precedence logic, and one of them forgot. #3398 then had to\ncorrect #3397's scope on one path while the other stayed wrong, which is the\nsignature of duplicated semantics rather than a missing check.\n\nThe fix is structural, not another check: one function through which every\nsession write passes, taking authority as a required argument, so a caller\ncannot forget to ask. A predicate copied into two places is a bug that has not\nhappened yet.\n\n## Invariant 3 -- derived state carries the version of the logic that derived it\n\n**Any stored conclusion records which version of which computation produced it,\nso a corrected computation invalidates its own stale outputs automatically.**\n\npolylogue-9dxn is this, and its absence is what made polylogue-bu1i inert on\nexisting data: a persisted `ambiguous` verdict has no version, so a corrected\nclassifier cannot know which verdicts it now disagrees with. The two-component\ndesign already recorded on 9dxn (separate identity and classification\nfingerprints) is the mechanism.\n\nWith this, \"stale verdict\", \"needs re-census\", and \"the fix does not apply to\nexisting rows\" all stop being categories. Correction becomes self-healing by\nconstruction.\n\n## What this does to the current bead set\n\nReframe rather than close -- the individual fixes still ship, but as instances:\n\n bu1i c429 nuec hith d8al oycw -\u003e Invariant 1\n c737 (+ the shape behind #3397/#3398) -\u003e Invariant 2\n 9dxn -\u003e Invariant 3\n ck5v -\u003e not covered; genuinely separate\n (backfill coupled to acquisition\n route -- an availability rule, not\n an identity one)\n ey3r -\u003e a measurement defect, but its cause\n is Invariant 1: it counts\n `superseded_*` as missing because\n the vocabulary has redundant\n categories that Invariant 1 removes\n\n## How to tell whether this worked\n\nNot \"the tests pass\". The observable is that the vocabulary shrinks:\n\n- The membership decision vocabulary loses `superseded_prefix` as distinct from\n `superseded_equivalent`.\n- `_provider_ordered_browser_snapshots` is deleted rather than maintained.\n- `HISTORICAL_NON_PREFIX_GOVERNANCE_DETAIL` and its legacy-detail variants stop\n needing to exist, because non-prefix growth stops being exceptional.\n- No new provider quirk requires a new branch in the classifier.\n\nIf a change adds a case instead of removing one, it is going the wrong way even\nif it makes a test pass. Per this repo's own surgical-renewal rule, the old path\nis deleted in the same change that replaces it -- these special cases must not\nsurvive as dead alternates beside the invariant.\n\n## Acceptance criteria\n\n- The comparison value for a session is constructed from an explicit\n content-only allowlist, so adding a field to a parser cannot silently enter\n identity. Adding a volatile field and observing that comparison is unaffected\n is the test.\n- Exactly one code path can write a session, and it cannot be called without\n authority.\n- Every stored verdict carries a version; changing the logic invalidates the\n affected verdicts without an operator command.\n- At least two existing special-case paths are DELETED, not merely bypassed.\n","status":"open","priority":0,"issue_type":"task","owner":"ezo.dev@gmail.com","created_at":"2026-07-30T14:41:28Z","created_by":"Sinity","updated_at":"2026-07-30T14:41:28Z","labels":["area:ingest"],"dependency_count":0,"dependent_count":0,"comment_count":0} {"_type":"issue","id":"polylogue-oycw","title":"Coalescing rests on a positional-prefix superset test that real providers violate; 41% of the corpus depends on it","description":"## Scale first: this is the archive's normal condition, not an edge case\n\n logical identities with more than one raw 7,440\n total logical identities 18,228\n -\u003e 41% of the corpus is multi-raw\n\nCohort sizes by origin (raws in multi-member cohorts):\n\n chatgpt-export 3-member 5,817 4-member 551 +tail to 12\n claude-ai-export 4-member 3,592 3-member 258 +tail to 9\n codex-session 2-member 3,544 ... one cohort of 105\n claude-code-session 2-member 2,338 3-member 663 +tail to 25\n hermes-session 2-member 536\n aistudio-drive 2-member 302\n antigravity-session 2-member 232\n browser-capture raws 887 (786 chatgpt, 47 claude-ai, 38 unknown, 16 grok)\n\nCorrectness for nearly half the archive rests on the revision-arbitration layer.\nIt is not a rarely-exercised safety net.\n\n## Where the multiplicity comes from\n\nNot divergence. Repeated whole-account acquisition:\n\n claude-ai-data-2025-10-04 906 raws\n claude-ai-data-2026-04-23 973 raws\n claude-ai-data-2026-06-14 1,998 raws\n chatgpt-data-2025-10-20 2,072 raws\n chatgpt-data-2026-04-23 4,805 raws\n\nEvery GDPR export contains every conversation, so each conversation enters the\narchive once per export vintage. 577 of the 587 claude-ai ambiguous cohorts have\nexactly 4 members for this reason.\n\n## Layer 1 -- identity. This one is sound.\n\n`sessions.session_id` is a generated column, `origin || ':' || native_id`, where\n`native_id` is the parser's `provider_session_id` -- the provider's own\nconversation uuid. Measured: 34 of 35 sampled claude-ai cohorts have an\nIDENTICAL provider_message_id set across all members, and the conversation uuid\nis identical across all four export vintages.\n\nSession identity is stable across acquisitions. The failures found on\n2026-07-30 were narrower and are separately tracked: a dispatch bug appending a\nspurious `-0` (fixed 2026-07-20, polylogue-eqnv), and unstable synthetic\n*attachment* ids (polylogue-hith / polylogue-d8al) -- not session ids.\n\n**Identity is not the problem, and a fix aimed at identity will not help.**\n\n## Layer 2 -- coalescing. Two mechanisms that do not compose.\n\n**(a) Content-hash idempotency** (`pipeline/ids.py:session_content_hash`).\nRe-ingest with a matching hash is skipped. The hash deliberately excludes user\nmetadata, but it INCLUDES: message array order, attachment acquisition state,\nvolatile provider metadata, and synthetic ids. Across two exports of an\nunchanged conversation, at least one of those always differs.\n\nSo idempotency never fires across export vintages -- by construction, not by\naccident. Every re-export falls through to (b).\n\n**(b) Revision membership arbitration.** Decides which raw is authoritative for\na session id when hashes differ. This carries the entire load that (a) fails to\nabsorb, for 41% of the corpus.\n\n## Layer 3 -- superset determination. This is the actual defect.\n\n`_strictly_dominates` (`archive/session_revision_membership.py`) requires:\n\n older.message_hashes == newer.message_hashes[: len(older.message_hashes)]\n\na POSITIONAL PREFIX. Three assumptions are embedded there, and all three are\nviolated by real providers:\n\n1. *Messages keep a stable order across acquisitions.* Violated: 19 of 35\n sampled claude-ai cohorts differ only in array order, same ids, zero content\n differences. Claude.ai does not emit a stable sequence between exports.\n2. *A message's hash is a function of its content alone.* Violated by volatile\n provider metadata (chatgpt `elapsed_duration_ms`, polylogue-nuec) and by\n acquisition state (Drive attachment bytes, polylogue-bu1i).\n3. *Growth is append-only at the tail.* Violated whenever a provider edits or\n inserts mid-conversation, and structurally by browser-capture DOM snapshots.\n\nWhen the test fails in both directions the cohort is quarantined ambiguous and\nNOTHING is indexed -- so a conversation held complete, correct, and in four\nidentical copies is absent from the archive. That is the 1,009-1,027 absence\npopulation.\n\n## What the correct test looks like\n\nPer-message ids are stable (34/35 measured), so superset-ness is decidable on\nevidence we already hold, without ordering:\n\n equal same provider_message_id SET, equal content per id\n -\u003e semantically the same revision; `equivalent_raw_ids`,\n no arbitration needed at all\n dominates A's id set strictly contains B's, content equal on the\n intersection -\u003e A is authoritative\n fork neither contains the other, OR content differs on the\n intersection -\u003e genuinely ambiguous, and rare\n (0 of 35 sampled claude-ai; 1 plausible case archive-wide,\n in grok-export)\n\nOrdering remains a real property of a session and must still be stored and\nrendered -- the claim is only that ordering must not be the DOMINANCE key.\nA conversation is a set of identified messages plus an ordering; which evidence\nexists is a set question, and treating the sequence as identity makes every\nprovider-side reordering look like divergence.\n\nLikewise a message's identity for comparison must exclude provider-volatile\nmeasurement fields and acquisition state, for the same reason bu1i split\nattachment identity from attachment acquisition.\n\n## Browser capture\n\n887 raws, 786 of them chatgpt. A DOM snapshot legitimately carries different\nsynthetic ids and a different ordering from the same conversation's export, so\nit violates assumptions 1 and 3 by design. `_provider_ordered_browser_snapshots`\nand `_direct_export_precedence` exist to special-case it, which is evidence that\nthe general test was already known to be too strict -- the special cases are\npatches over the wrong primitive rather than genuine domain rules. Re-evaluate\nboth once the set-based test lands; `_direct_export_precedence` (a real export\noutranks a browser capture) is probably a genuine rule worth keeping, while the\nordering special-case may become unnecessary.\n\n## Acceptance criteria\n\n- Superset determination is order-independent and decided on stable per-message\n identity plus per-id content equality.\n- Equal-content cohorts resolve as `equivalent`, not `ambiguous`, and index one\n member -- no arbitration for the 34/35 case.\n- Message comparison identity excludes provider-volatile measurement fields and\n acquisition state.\n- Report how many cohorts still reach a genuine-fork verdict; it should be very\n small, and a large number means one of the above is wrong.\n- Re-run `.agent/scripts/corpus-fidelity-audit.py`: absent_documents must fall\n to approximately zero from the 1,027 baseline.\n\nRef polylogue-bu1i, polylogue-c429, polylogue-nuec, polylogue-d8al, polylogue-f1vg\n","status":"open","priority":0,"issue_type":"bug","owner":"ezo.dev@gmail.com","created_at":"2026-07-30T14:35:25Z","created_by":"Sinity","updated_at":"2026-07-30T14:35:25Z","labels":["area:ingest"],"dependency_count":0,"dependent_count":0,"comment_count":0} @@ -72,6 +74,10 @@ {"_type":"issue","id":"polylogue-tf2.1","title":"Rerun forensics on current archive; price origin_reported providers","description":"Rerun scripts/agent_forensics.py against the current archive (v23+); price origin_reported providers via the vendored LiteLLM catalog (match last path segment); all-provider headline or explicitly-labeled per-provenance figures that cannot be misread; record deltas vs 06-27; verify chart SVGs render. Cache-inclusion must be disambiguated (Codex input INCLUDES cached ~96%; see bd memories). Also blocked on logical-session token attribution — the headline must not be double-counted.","notes":"Correction to close_reason monetary values: stored/provider-priced subset was $239,453.14; catalog API-equivalent was $318,650.88; origin_reported catalog estimate was $79,197.74. The original close_reason text lost dollar-prefixed digits due shell expansion, not measurement drift.","status":"closed","priority":0,"issue_type":"task","assignee":"Sinity","owner":"ezo.dev@gmail.com","created_at":"2026-07-03T04:31:33Z","created_by":"Sinity","updated_at":"2026-07-03T09:59:13Z","started_at":"2026-07-03T09:28:10Z","closed_at":"2026-07-03T09:59:02Z","close_reason":"Completed with blocker caveat preserved: scripts/agent_forensics.py now prices origin_reported rows through the shared vendored LiteLLM pricing catalog while preserving stored provenance; report separates stored/provider-priced cost from catalog API-equivalent estimates and carries logical-session/cache caveats instead of claiming final billing reconciliation. Regenerated current artifact at .agent/demos/agent-forensics against /home/sinity/.local/share/polylogue schema v23: 16,498 physical sessions, 4,142,175 messages, 356.5B tokens, ,453.14 stored/provider-priced subset, ,650.88 catalog API-equivalent, and ,197.74 origin_reported catalog estimate. SVG parse check passed for 9 charts; devtools test tests/unit/scripts/test_agent_forensics.py passed; devtools verify --quick passed run 20260703T095718Z-quick-753466-96559776; devloop-review clean. Remaining final-reconciliation blocker stays open as polylogue-4ts.2.","labels":["area:usage","campaign"],"dependencies":[{"issue_id":"polylogue-tf2.1","depends_on_id":"polylogue-4ts.2","type":"blocks","created_at":"2026-07-03T06:32:45Z","created_by":"Sinity","metadata":"{}"},{"issue_id":"polylogue-tf2.1","depends_on_id":"polylogue-sru.7","type":"blocks","created_at":"2026-07-03T06:31:33Z","created_by":"Sinity","metadata":"{}"},{"issue_id":"polylogue-tf2.1","depends_on_id":"polylogue-tf2","type":"parent-child","created_at":"2026-07-03T06:31:33Z","created_by":"Sinity","metadata":"{}"}],"dependency_count":2,"dependent_count":2,"comment_count":0} {"_type":"issue","id":"polylogue-tf2","title":"Campaign: agent-forensics regeneration + all-provider repricing","description":"Regenerate the agent-forensics packet on the current archive with an honest all-provider headline. The 2026-06-27 report (546.6B tokens, $89,368 API-list equivalent, 216x cache amplification) is the most stranger-legible artifact on any shelf, but its numbers are pre-dedup stale and the headline prices only the priced-provenance subset (Claude Code cost_usd rows); Codex/ChatGPT/Gemini are origin_reported token counts with no dollar value (operator estimate ~$150K all-provider). Sequenced after claim-vs-evidence per operator direction 2026-07-02.","design":"Current slice design: turn the existing agent-forensics/cost headline into a product-backed all-provider repricing artifact. First inspect devtools/scripts and polylogue analyze surfaces for agent_forensics/cost code. Use active archive usage headline (detail=headline) for authoritative physical_session and logical_session_model_high_water token totals. Keep priced-provenance dollars and origin-reported token estimates separate: do not multiply every token by one blended price without a labeled lane. Add or reuse a shared pricing/projection helper so the demo artifact is regenerated from Polylogue product code, not ad hoc SQL. Acceptance for this slice: the generated agent-forensics artifact names archive root/schema, includes physical vs logical token grain, separates priced subset from origin-reported estimate lanes, gives reproduction commands, and has focused tests for any new repricing helper/surface.","acceptance_criteria":"Terminal state: regenerated forensics packet on the current archive with an honest all-provider headline (priced subset AND origin-reported estimate lanes separated), agent_forensics.py folded into polylogue analyze (tf2.2), artifact on the demo shelf with reproduction commands, cold-reader gate passed. Epic closes only when that artifact is recorded.","status":"closed","priority":0,"issue_type":"epic","assignee":"Sinity","owner":"ezo.dev@gmail.com","created_at":"2026-07-03T04:31:32Z","created_by":"Sinity","updated_at":"2026-07-03T19:06:44Z","started_at":"2026-07-03T18:47:23Z","closed_at":"2026-07-03T19:06:44Z","close_reason":"Completed: provider usage headline now exposes product-backed pricing lanes in polylogue analyze usage --detail headline, separating stored/provider-priced cost from catalog API-equivalent estimates for origin_reported rows. Regenerated the current .agent/demos/agent-forensics artifact against /home/sinity/.local/share/polylogue schema v23: physical-session tokens 395,320,980,423; logical high-water tokens 288,741,229,728; stored/provider-priced USD 243,392.189328; catalog API-equivalent USD 337,565.031618; priced lane 13,889 rows / 12,331 sessions / 12,650 matched rows; origin_reported lane 2,308 rows / 2,270 sessions / 2,302 matched rows. Verification: live polylogue --plain analyze usage --detail headline --format json --limit 0 wrote /realm/tmp/polylogue-usage-headline-pricing-current.json; devtools test tests/unit/storage/test_provider_usage_report.py tests/unit/cli/test_diagnostics.py passed 23 tests; devtools verify --quick passed run 20260703T190553Z-quick-2226137-d91d4e8f; devtools workspace demo-shelf --json reported ok. Non-claim preserved: this is not final billing reconciliation and physical/logical token grains stay explicitly separated.","labels":["area:usage","campaign","size:M","spine"],"dependency_count":0,"dependent_count":0,"comment_count":0} {"_type":"issue","id":"polylogue-sru","title":"Campaign: claim-vs-evidence report to finding-grade","description":"Terminal state: an externally publishable finding ('how often do coding agents proceed past failed tool calls, by model/tool') with stated sample frame, calibrated markers, benign/consequential split, seeded stranger-runnable reproduction, and a passed cold-reader gate. Slice closure is NOT campaign closure; this epic stays top-of-frame until its terminal state is recorded.\\n\\nState as of 2026-07-03 after calibrated active-archive regeneration: archive root /home/sinity/.local/share/polylogue, index schema v23, 41,886 structured failures total, 5,000 origin-stratified failures inspected (3,746 claude-code-session, 1,247 codex-session, 7 claude-ai-export), 100 unpaired structured failures. Marker vocabulary was tightened to avoid broad issue/fix/block/gitignored false positives. Immediate next-turn totals: acknowledged=420, silent_proceed=1,205, ambiguous=3,375 (2,624 wordless tool continuations; 751 prose without marker). Lower-bound silent rate is 24.1%; among classified immediate next turns, silent rate is 74.2%. Next-3 sensitivity window, stopping before the next user message, finds 302 acknowledgments that appear only after the next turn; window3 silent lower bound is 37.0%. Calibration: 50 hand-labeled immediate-next-turn rows, acknowledged-marker precision=1.0, recall=0.8421052631578947, invalid rows=0. Artifact: .agent/demos/claim-vs-evidence/claim-vs-evidence.report.json.","notes":"2026-07-03 update: methodology package is now cold-read gated. .agent/demos/claim-vs-evidence contains aggregate live evidence, public-summary.json, PUBLIC_REPRODUCTION.md, COLD_READER_GATE.md, and COLD_READ_RESULT.md. Seeded reproduction is meaningful, not empty: 4 structured failures, 2 acknowledged follow-ups, 2 silent-proceed follow-ups, 0 unpaired. Cold-reader subagent PASS recovered claim/non-claim, sample frame, rates, calibration, caveats, and reproduction commands from the artifact directory only. Remaining campaign child: polylogue-sru.1 productizes action-unit outcome/followup_class capability.","status":"closed","priority":0,"issue_type":"epic","owner":"ezo.dev@gmail.com","created_at":"2026-07-03T04:31:26Z","created_by":"Sinity","updated_at":"2026-07-03T09:28:09Z","closed_at":"2026-07-03T09:28:09Z","close_reason":"Completed: all seven campaign children are closed. The claim-vs-evidence finding now has bounded sample-frame reporting, calibrated marker precision/recall, handler-class and next-3 sensitivity splits, meaningful seeded reproduction, cold-reader PASS, and productized action-unit followup_class/followup_message_ref query capability. Current artifact lives under .agent/demos/claim-vs-evidence and was regenerated against /home/sinity/.local/share/polylogue schema v23.","labels":["area:substrate","campaign"],"dependency_count":0,"dependent_count":0,"comment_count":0} +{"_type":"issue","id":"polylogue-8ac0","title":"acquire chatgpt export .dat asset bytes into the blob store","description":"Follow-up to polylogue-0hwv: that bead's PR resolves every referenced .dat\nasset id to its real name/mime/size/sha256 (via library_files.json /\nconversation_asset_file_names.json) and records sandbox-file tier resolution,\nbut does NOT yet stream the .dat blobs themselves into the content-addressed\nblob store. attachments stay acquisition_status=\"unfetched\" with real\nmetadata but no bytes.\n\nWhy deferred: decoder_zip.py's ZipEntryValidator.filter_entries only admits\n.json/.jsonl entries (session_only=True) -- .dat members are filtered out\nbefore the main per-entry loop ever sees them. Real byte acquisition needs a\ntwo-pass ZIP scan: (1) stream every .dat member into BlobStore via\nstore.write_from_fileobj() (same pattern decoder_zip.py's capture_raw branch\nalready uses for raw JSON capture -- streaming hash+write, no full-file\nmemory load), building a dat_id -\u003e (blob_hash, size) map; (2) during\nconversation parsing, join resolved attachments against that map and mark\nthem acquired via the same preacquired-blob receipt mechanism\ningest_batch/_core.py uses for inline_bytes (publication_receipt_id +\nflush_blob_publications), without re-hashing bytes already written in pass 1.\n\nFor the extracted-directory import shape (not a ZIP), the .dat files sit on\ndisk as ordinary sibling files next to conversations-*.json --\nChatGPTAssemblySpec.discover_sidecars already walks that directory and could\nread them directly with BlobStore.write_from_path (also streaming).\n\nAC: importing the real 2026-07-29 export (or an extracted copy) acquires\n.dat bytes as attachment blobs with acquisition_status=\"acquired\" and a true\nSHA-256 for every dat id resolved by polylogue-0hwv's ChatGPTAssetIndex;\nattachments referenced by asset_pointer/attachments[]/resolved sandbox links\nresolve to stored bytes when the underlying .dat member is present in the\nsource. Verify end-to-end against a synthetic ZIP fixture (a few .dat members\n+ matching library_files.json/conversation_asset_file_names.json +\nconversations.json) before attempting the real 16GB export, then confirm\nagainst a real (or truncated real) export.\n\nNot in scope for polylogue-0hwv's own PR: this needs its own focused\nbyte-acquisition-specific verification pass (streaming correctness, receipt/\nGC interaction, aggregate-size ceiling interaction with 3,228 more zip\nentries) separate from the naming/resolution logic polylogue-0hwv covers.","status":"open","priority":1,"issue_type":"task","owner":"ezo.dev@gmail.com","created_at":"2026-07-31T03:32:03Z","created_by":"Sinity","updated_at":"2026-07-31T03:32:03Z","dependency_count":0,"dependent_count":0,"comment_count":0} +{"_type":"issue","id":"polylogue-e98k","title":"reconcile SQLite mmap budget with the cgroup memory limit","description":"MEASURED 2026-07-31. The polylogued memory incident was not opaque kernel caching - it was two independently chosen constants that never met.\n\nAPP SIDE (polylogue/storage/sqlite/connection_profile.py):\n BULK_BUILD_MMAP_SIZE_BYTES = 4 GiB\n BULK_BUILD_CACHE_SIZE_KIB = 512 MiB\n WRITE_MMAP_SIZE_BYTES = 1 GiB\n READ_MMAP_SIZE_BYTES = 128 MiB\n\nCGROUP SIDE (sinnix modules/services/polylogue.nix:283):\n MemoryHigh = 6G MemoryMax = 8G\n\nARCHIVE SIZE: index.db 38 GB (symlink into .index-generations), source.db 9.1 GB.\n\nA 4 GiB mmap window over a 38 GB database fills completely under any scan-heavy\nwork. One bulk connection therefore accounts for ~4.5 GiB of a 6 GiB ceiling,\nleaving ~1.5 GiB for the daemon's ~1 GiB RSS and everything else. Pinning at the\nlimit was structurally guaranteed, not a leak. Observed: memory.events high\ncounter at 538k+ and climbing, memory.pressure ~3.9%, repeated slow_write, and a\nzip sitting unprocessed in the inbox for 2.5h. A runtime-only MemoryHigh=14G\nstopped throttling dead (0 events over a properly timed 180s, pressure 0.00),\nand MemoryCurrent then settled at 8.59 GB - above the old ceiling, proving the\nlimit was the binding constraint.\n\nTHREE FIXES, in order of value:\n\n1. DERIVE BOTH FROM ONE BUDGET. The mmap/cache profile sizes and the systemd\n limits should come from a single declared memory budget rather than being\n picked separately in two repos. Any future archive growth then moves both.\n\n2. memory.high IS THE WRONG INSTRUMENT for mmap'd/file-backed pages. It is\n designed to throttle anon growth. Mapped DB pages are reclaimable, so\n throttling produces evict -\u003e immediate re-fault -\u003e evict thrash, which is\n exactly the slow_write signature. Keep MemoryMax as the genuine leak guard;\n set MemoryHigh above the mapped budget, or drop it and let global reclaim\n handle cache.\n\n3. MAKE THE MISMATCH OBSERVABLE. Log mapped-bytes-budget vs the cgroup limit at\n daemon startup. This incident was discovered by symptom hours later; it\n should be a startup warning.\n\nNote mmap_size is an upper bound, not an allocation - which is why this stayed\ninvisible until the archive grew large enough to fill the window.\n\nHousekeeping seen while measuring: .index-generations/ holds 72 GB for a 38 GB\nactive index (one stale generation plus a retired one).","status":"open","priority":1,"issue_type":"task","owner":"ezo.dev@gmail.com","created_at":"2026-07-31T01:00:41Z","created_by":"Sinity","updated_at":"2026-07-31T01:00:41Z","dependency_count":0,"dependent_count":0,"comment_count":0} +{"_type":"issue","id":"polylogue-0hwv","title":"resolve chatgpt export .dat assets to real filenames","description":"The 2026-07-29 chatgpt export ships attachment BYTES for the first time: 3,228 .dat members, of which 1,656 are mapped by conversation_asset_file_names.json (e.g. file-078R8dTqVR9lYSLVmOsCh6ht.dat -\u003e image.png). Message parts reference them as asset_pointer 'file-service://file-\u003cid\u003e', which matches the .dat basename.\n\nThe parser already handles asset_pointer / image_asset_pointer / audio_asset_pointer / audio_transcription. What is missing is the mapping file: rg finds conversation_asset_file_names NOT REFERENCED ANYWHERE in polylogue/.\n\nThis is the standing C6 gap (6,075 chatgpt attachment refs with no bytes) becoming resolvable for the first time - the bytes are now in the archive-side artifact rather than behind an expired URL.\n\nAC: importing the 2026-07-29 export acquires the .dat bytes as attachment blobs with their real filenames and content types, and an attachment referenced by asset_pointer resolves to stored bytes.","notes":"MEASURED SPEC (2026-07-31, from the 2026-07-29 export).\n\nTwo id namespaces among the 3,228 .dat blobs:\n file-\u003cb64ish\u003e 677 conversation assets\n file_\u003c32 hex\u003e 2,551 library files\n\nTwo independent name sources, and TOGETHER they are exhaustive:\n conversation_asset_file_names.json names 1,656 (dat basename -\u003e 'image.png')\n library_files.json names 2,231 (file_id -\u003e file_name, file_extension,\n file_size_bytes, sha256 digest,\n upload/processed times, directory_id)\n either names 3,228 = 100.0%, ZERO unnamed\n\nSo the join is: strip .dat -\u003e look up in asset-name map, else library_files.file_id.\nlibrary_files is the richer source (mime/size/digest/provenance), so prefer it when both hit.\n\nREFERENCE SIDE (this is the part that corrects the earlier framing):\n distinct file ids referenced by messages 3,626\n via content.parts[].asset_pointer 267\n via message.metadata.attachments[] 3,444 \u003c- the LARGER channel, previously unexamined\n referenced AND bytes present 1,608 (44.3%)\n referenced but bytes ABSENT 2,018 (55.7% - still unresolvable)\n bytes present but unreferenced 1,620 of which 1,438 are library_files\n and 182 remain unexplained\n\nSo this does NOT close C6 outright: it makes 44% of referenced attachments resolvable and\nadds a whole second population (Library) that has bytes but no message reference. Both are\nworth storing; conflating them would be wrong.\nIMPLEMENTED (branch feature/sources/chatgpt-export-assets-and-sidecars, PR pending).\n\nScope: name/mime/size/sha256 resolution for every referenced .dat id\n(library_files.json preferred, conversation_asset_file_names.json fallback).\nChatGPTAssetIndex.resolve_dat in polylogue/sources/parsers/chatgpt_sidecars.py,\nwired via a new ChatGPTAssemblySpec (polylogue/sources/assembly_chatgpt.py)\nusing the existing ProviderAssemblySpec discover_sidecars/enrich_session\nprotocol. Resolution recorded as a chatgpt_asset_resolution session_event\n(not a new attachment column -- index.db is a derived tier).\n\nMeasured against the real 2026-07-29 export corpus (all 29 conversations-*.json\nshards + both sidecars, 2,836 sessions, 0 parse errors): 1,924/1,924 = 100% of\nreferenced .dat attachments resolved a name.\n\nNOT satisfied yet: actual byte acquisition into the blob store (AC says\n\"acquires the .dat bytes as attachment blobs ... resolves to stored bytes\").\ndecoder_zip.py's ZipEntryValidator only admits .json/.jsonl entries, so .dat\nZIP members are never read at all today. Filed as a dedicated follow-up,\npolylogue-8ac0, with the two-pass streaming design (collect .dat blobs via\nBlobStore.write_from_fileobj, join during conversation parsing, reuse the\ninline_bytes-style preacquired-blob receipt path) -- this needs its own\nverification pass and is high enough risk (touches the zip streaming/receipt/\nGC machinery) that bundling it into this PR would have made both halves\nharder to review and verify.\n","status":"closed","priority":1,"issue_type":"task","assignee":"Sinity","owner":"ezo.dev@gmail.com","created_at":"2026-07-30T23:44:26Z","created_by":"Sinity","updated_at":"2026-07-31T03:55:38Z","started_at":"2026-07-31T03:32:13Z","closed_at":"2026-07-31T03:55:38Z","close_reason":"Merged in PR #3409 (polylogue/master@11403388d): .dat asset id -\u003e name/mime/size/sha256 resolution via ChatGPTAssetIndex, wired through the assembly protocol. Actual byte acquisition into the blob store deferred to polylogue-8ac0 (decoder_zip.py streaming change, out of scope for this PR).","dependency_count":0,"dependent_count":0,"comment_count":0} +{"_type":"issue","id":"polylogue-2bc2","title":"bd list --all infinite recursion: tree renderer loops on cyclic/duplicate parent-child edge, wrote 54GB before kill","description":"Reproducible 2026-07-30: 'bd list --all' in /realm/project/polylogue emits unbounded repeating tree-indentation glyphs; a probe wrote 23GB in \u003c2min before kill. Prior casualty: /realm/tmp/_bd_poly_full.txt grew to 58,427,205,502 bytes (2026-07-21) before its process died. Suspect cyclic or duplicated dependency edge: polylogue-z9gh.7 appears twice as child of polylogue-z9gh in --status open output. Fix = cycle guard in the tree renderer + dedupe/repair of the offending edge in this DB.","status":"open","priority":1,"issue_type":"bug","owner":"ezo.dev@gmail.com","created_at":"2026-07-30T19:35:47Z","created_by":"Sinity","updated_at":"2026-07-30T19:35:47Z","dependency_count":0,"dependent_count":0,"comment_count":0} {"_type":"issue","id":"polylogue-6kur","title":"Cull the repair surface: 10k lines of manual repair against 2.7k of convergence, with targets guarding schema-impossible states","description":"## Measured shape\n\n repair/maintenance surface ~10,164 lines\n storage/repair.py 7,154 (123 top-level defs, 22 public entrypoints)\n maintenance/*.py 3,010\n daemon convergence 2,665 lines\n convergence.py 637\n convergence_stages.py 2,028\n\nA 3.8:1 ratio of manual repair machinery to the automatic convergence meant to\nmake it unnecessary. Convergence registers only FIVE stages: `fts`, `embed`,\n`insights`, `claude_workflow`, `sinex_publication`. `repair.py` exposes eleven\nrepair targets.\n\nThis contradicts the project's own stated principle: *if Polylogue can maintain\na condition fully automatically it should, there is NO break-glass tier, and\nonce the automatic path maintains an invariant the redundant manual surface is\nDELETED rather than demoted.*\n\n## Per-target analysis (live archive, frozen 2026-07-30)\n\nNote first: `REPAIR_HANDLERS[target]` is a name-\u003efunction dispatch table, so\n\"no external references\" means dynamically dispatched, NOT dead. Every target\nbelow is reachable via `run_safe_repairs`/`run_archive_cleanup`.\n\n### Structurally impossible — delete (strongest case)\n\n| target | live violations | why it cannot occur |\n| --- | -- | --- |\n| `orphaned_messages` | **0** | `messages.session_id TEXT NOT NULL REFERENCES sessions(session_id) ON DELETE CASCADE` |\n| `orphaned_attachments` | **0** | `attachment_refs.session_id`/`message_id` both `NOT NULL ... ON DELETE CASCADE` |\n\n`PRAGMA foreign_keys = ON` is set in `storage/sqlite/connection_profile.py`, so\nthese are enforced, not decorative. The schema forbids the state; the repair\nscans for it anyway. Zero violations is not luck.\n\nDelete both repairs, both previews, their `SAFE_REPAIR_TARGETS`/`CLEANUP_TARGETS`\nentries, and their debt-status rows.\n\n### Spent one-shot migrations — delete once confirmed\n\n| target | live violations | note |\n| --- | -- | --- |\n| `message_type_backfill` | **0** | A backfill for a column added later. Confirm the write path always sets it (NOT NULL would settle it), then the migration is spent. |\n\nA backfill is inherently one-shot: once the historical rows are filled and the\nwriter populates the column, the repair guards nothing.\n\n### Symptom-treating — the repair is the wrong fix\n\n| target | live violations | note |\n| --- | -- | --- |\n| `session_timestamp_backfill` | **5,382, GROWING** | Was 1,117 after the hook de-inflation; now 5,382. A backfill whose backlog grows means the WRITE PATH is still producing the defect. |\n\nThis is the \"fix the automatic path\" case, and the most valuable finding here.\nDo not keep running the backfill; find why sessions are still written with\n`created_at_ms IS NULL` and stop that. The repair has been masking a live\nwriter bug, which is exactly what a break-glass tier does to you.\n\n### Cause fixed elsewhere — expect near-no-op\n\n| target | live violations | note |\n| --- | -- | --- |\n| `empty_sessions` | 5,255, of which **4,945** are `.meta` phantoms | PR #3403 fixes the cause (an ungated parse chokepoint in `revision_backfill.py`). After it lands, ~310 remain, and some of those are legitimately empty (sessions carrying only `session_events` after the v46 reclassification). Re-measure post-rebuild before deciding. |\n\n### Genuinely load-bearing — keep\n\n`raw_materialization`, `session_insights`, `orphaned_blobs`,\n`superseded_raw_snapshots`, `stale_supersession_receipts`. These were exercised\nfor real this session (raw materialization and authority blockers had to be\nunstuck manually). But note that needing them manually is itself evidence the\nautomatic path has gaps -- `session_insights` in particular overlaps the\n`insights` convergence stage and should be examined for redundancy.\n\n## Sequencing\n\nThe `archive.py` decomposition lane may relocate `repair.py`'s seam\n(`architecture-hotspots.md` note-on-#3 leaves its `storage/` vs `maintenance/`\nplacement explicitly undecided). Do the deletions after that lands, or they\ncollide.\n\n## Acceptance criteria\n\n- `orphaned_messages` and `orphaned_attachments` repair+preview+registry entries\n deleted, with the FK/CASCADE constraint cited as the replacement guarantee.\n- `message_type_backfill` deleted after confirming the writer always populates it.\n- A separate bead opened for the `created_at_ms IS NULL` WRITER defect, with the\n 1,117 -\u003e 5,382 growth as evidence; the backfill target is not deleted until\n that is fixed.\n- Line count of `storage/repair.py` reported before and after.\n- No new registry or allowlist introduced by any of this.\n","notes":"CORRECTION 2026-07-30: my per-target verdict on `empty_sessions` was wrong, and wrong in the dangerous direction.\n\nI classified it as 'cause fixed elsewhere, expect near-no-op after #3403'. polylogue-ne6k, which already existed and which I failed to read before writing this analysis, records the opposite: **repair_empty_sessions would DELETE the 832 genuinely-empty sessions the hook-inflation postmortem deliberately chose to retain.**\n\nSo the target is not a soon-to-be-no-op. It is actively destructive against data an earlier postmortem made a considered decision to keep. Running it after #3403 lands would remove real archive content, not phantom rows.\n\nRevised verdict for `empty_sessions`: do NOT delete the target as spent, and do NOT run it. It needs a decision about the 832 retained-empty sessions first (ne6k owns that), and any culling work must treat ne6k as a blocker rather than a footnote.\n\nMethod failure worth recording, because it is the same one twice in a day: I derived a verdict from live measurement plus code reading without first checking whether an existing bead already contained the answer. 587 open beads exist; `bd list` silently caps its output (returned 50 of 1,234 records), so a survey that trusts its default limit sees 4% of the backlog and reads as exhaustive. Query the exported .beads/issues.jsonl directly rather than the CLI default.\n\nThe rest of this bead's analysis is unaffected: the FK/CASCADE structural-impossibility case for orphaned_messages and orphaned_attachments stands on schema evidence, and the session_timestamp_backfill growth finding (1,117 -\u003e 5,382) stands on measurement.","status":"open","priority":1,"issue_type":"task","owner":"ezo.dev@gmail.com","created_at":"2026-07-30T17:20:12Z","created_by":"Sinity","updated_at":"2026-07-30T17:41:30Z","labels":["area:ingest"],"dependency_count":0,"dependent_count":0,"comment_count":0} {"_type":"issue","id":"polylogue-f1vg","title":"Corpus acceptance gate: no absences and maximum fidelity, with the 2026-07-30 frozen baseline","description":"## What the operator asked for\n\n\"Ensure max fidelity as well as no absences, through the entire corpus.\" That is\na stronger bar than any existing check enforces, and nothing measured either\nhalf until now.\n\n## Why the existing checks cannot serve\n\n`verify-archive`'s `source-index-coverage` counts superseded revisions as\nmissing work (polylogue-ey3r), so it cannot reach zero on any archive that ever\ningested a conversation twice, and therefore cannot gate a rebuild. Nothing at\nall measures fidelity: an archive can report perfect coverage while every\nattachment whose bytes it holds is recorded `unfetched`, which is precisely the\nstate measured on 2026-07-30.\n\n## The gate\n\n`.agent/scripts/corpus-fidelity-audit.py` (read-only, `mode=ro` throughout,\nexits 1 on failure so it can gate a rebuild). Three measures:\n\n1. **Absences** -- logical documents (origin + provider_session_id) the archive\n holds evidence for but does not surface, bucketed by cause so a fix's effect\n is attributable rather than a single number moving for unknown reasons.\n2. **Attachment fidelity** -- acquired vs not-acquired refs, split by origin and\n upload_origin, because a Drive-hosted reference never fetched is actionable\n while a genuinely byte-less attachment kind is not.\n3. **Revision fidelity** -- documents whose indexed evidence is smaller than the\n largest revision recorded for them.\n\n## Baseline, live archive frozen 2026-07-30 (daemon stopped)\n\n ABSENCES 1,009 of 18,248 known documents\n 587 claude-ai-export/ambiguous-only\n 184 claude-code-session/ambiguous-only\n 135 chatgpt-export/ambiguous-only\n 71 aistudio-drive/settled-yet-absent\n 20 unknown-export/settled-yet-absent\n 12 gemini-cli / hermes / unknown / grok / codex\n\n ATTACHMENT FIDELITY acquired=2,118 not-acquired=7,655\n 3,684 chatgpt-export/oauth/unfetched\n 2,391 chatgpt-export/\u003cnone\u003e/unfetched\n 1,975 aistudio-drive/drive/acquired\n 1,119 aistudio-drive/drive/unfetched\n 398 claude-ai-export/oauth/unfetched\n\n REVISION FIDELITY 94 documents below best recorded evidence\n 76 hermes-session\n 16 claude-code-session\n 2 chatgpt-export\n\n VERDICT: FAIL\n\nThe `settled-yet-absent` buckets (71 drive, 20 unknown-export, 1 codex) are not\nexplained by any currently-tracked cause and want their own investigation --\nthese are documents with no ambiguous decision anywhere that are nonetheless\nmissing.\n\nThe 94 revision-fidelity documents are a residue after correcting a false\npositive, and should be treated as a prompt to investigate rather than proof of\nloss (see below).\n\n## Measurement trap this already caught\n\nThe first version compared indexed *messages* against\n`raw_session_memberships.message_count` and reported **474** shortfalls, 294 of\nthem codex-session. All false. `message_count` was recorded by whichever parser\ncensused that raw, and index v46 deliberately reclassified a large share of\nCodex/Claude Code rows from chat turns into typed `session_events`. One codex\nsession read as \"15 indexed vs 68,553 recorded\" when it actually holds 15\nmessages plus 84,612 events. Counting `messages + session_events` drops the\nfigure to 94.\n\nAnyone extending this must keep that in mind: cross-generation counts are only\napproximately comparable, so a metric built on them needs its assumption stated\nand checked against a real sample before its number is believed.\n\n## Follow-up\n\nPromote this into `devtools` as a first-class command with a `CommandSpec` (plus\n`devtools render devtools-reference`) so it is an enforced gate rather than a\nscript, and wire it into the post-rebuild acceptance path alongside\n`verify-archive`. Kept as a script for now because the fixes it measures are\nstill in flight and its thresholds will move as they land.\n\n## Acceptance criteria\n\n- Absences reach 0, or every residual is individually justified in writing.\n- Attachment refs marked not-acquired are either acquired or shown to be\n genuinely unfetchable (deleted upstream, over the size cap, byte-less kind).\n- Revision-fidelity residue is explained rather than merely small.\n","status":"open","priority":1,"issue_type":"task","owner":"ezo.dev@gmail.com","created_at":"2026-07-30T14:04:01Z","created_by":"Sinity","updated_at":"2026-07-30T14:04:01Z","labels":["area:ingest"],"dependency_count":0,"dependent_count":0,"comment_count":0} {"_type":"issue","id":"polylogue-d8al","title":"claude-ai-export: attachment real-id presence is inconsistent across export vintages, needs comparison-layer relaxation","description":"## What the data says\n\nCensus (full population, not a sample): replayed the production classifier\n(polylogue.sources.dispatch.parse_payload -\u003e session_revision_projection -\u003e\nclassify_membership_revisions) over all 566 claude-ai-export\nequal-message-count ambiguous cohorts in the live archive (read-only,\n/realm/db/polylogue), with polylogue-hith's parser-side fix (drop the\npositional-index seed for synthetic attachment ids) already applied.\n\n 566 claude-ai-export equal-message-count ambiguous cohorts (full census)\n 297 still ambiguous because message_hashes differ (polylogue-c429 /\n message-order-not-stable territory, or genuine content divergence)\n 268 still ambiguous with message_hashes EQUAL (0 content diffs) but\n attachment identity axis mismatched -- the exact shape hith\n targeted\n 0 of those 268 resolved by hith's fix\n 268 of those 268 are \"mixed real/synthetic\": one export vintage of the\n SAME conversation carries a real id (id/file_id/fileId/uuid/\n file_uuid) for an attachment; the OTHER vintage of the same\n conversation has no real id for the physically-same attachment and\n synthesizes one instead\n 0 are \"pure synthetic on both sides\" (the positional-index shape\n hith's fix targets and fully resolves when it occurs)\n\nIn other words: in the population currently persisted as ambiguous, 100% of\nthe identity-mismatch cases are this real-id-presence axis, not the\npositional-index axis. hith's fix is verified correct and regression-safe\n(250-cohort replay of already-resolved cohorts: 249/250 agree old vs new\nlogic, 1 improvement, 0 regressions) but resolves 0 of the currently-measured\n566-cohort population by itself, because no synthetic-minting scheme can ever\nmake a real UUID and a hash of (message, name, mime_type) collide.\n\n## Root cause\n\n`polylogue/sources/parsers/base_support.py:attachment_from_meta` uses the\nexport's own `id`/`file_id`/`fileId`/`uuid`/`file_uuid` field when present,\nand only falls back to synthesis when absent. Claude.ai does not consistently\nemit this field for the same attachment across export vintages of the same\nconversation -- verified directly against blob content for 6 sampled\ncohorts, all showing exactly this shape (one blob's attachment has a real\nUUID-shaped id, the other blob's attachment for the same message has no id\nfield and synthesizes `att-\u003chash\u003e`).\n\nNo id-minting scheme at the parser layer can reconcile this: a real id and a\nsynthetic hash will never be equal strings by construction, regardless of\nwhat the synthetic hash is seeded from.\n\n## Proposed fix (comparison layer, NOT parser layer)\n\nIn `polylogue/archive/session_revision_membership.py` (and/or\n`polylogue/pipeline/ids.py`'s `SessionRevisionProjection` /\n`_attachment_identity_payload`), the dominance/equivalence test should\ncompare attachments by a looser key when testing dominance -- e.g.\n`(message_provider_id, name, mime_type)` without the `id` field -- falling\nback to strict id equality only when that looser key is itself ambiguous\n(more than one attachment sharing the tuple on one side). This is the same\nclass of relaxation polylogue-bu1i introduced for acquisition state\n(`attachment_identities` vs `attachment_contents`), generalized to a third\naxis: \"same attachment referenced with and without a stable provider id\".\n\nThis bead deliberately does NOT propose an implementation in those files --\npolylogue-hith's owning lane was scoped away from\n`session_revision_membership.py`/`ids.py` because another lane owns them\nconcurrently. Whoever picks this up should re-run the census harness\ndescribed in polylogue-hith (or the updated one referenced in its closing\nnote) against the classifier change to prove the 268-cohort population above\nactually resolves, the same way polylogue-bu1i's PR proved 157/157.\n\n## Verification recipe\n\nSame read-only harness as polylogue-hith / polylogue-bu1i: parse both blobs\nof a cohort with production `parse_payload`, project with\n`session_revision_projection`, and diff the resulting\n`attachment_identities` sets. For the 268-cohort population, at least one\nattachment identity differs solely because one side has a real id string and\nthe other has a synthetic hash string for what is, by every other field\n(message anchor, name, mime_type), the same attachment.\n\nRef polylogue-hith\nRef polylogue-bu1i","notes":"Superseded by polylogue-aggz's architecture: attachment identity now unconditionally drops the provider id (content-derived: message_id+name+mime_type only), eliminating the strict/loose duality and its pairwise correlation machinery entirely rather than adding a fallback. See PR.","status":"open","priority":1,"issue_type":"bug","owner":"ezo.dev@gmail.com","created_at":"2026-07-30T13:02:14Z","created_by":"Sinity","updated_at":"2026-07-30T15:15:31Z","labels":["area:ingest"],"dependency_count":0,"dependent_count":0,"comment_count":0} @@ -454,6 +460,21 @@ {"_type":"issue","id":"polylogue-sru.2","title":"Characterize ambiguous bucket: wordless continuation vs prose-without-markers","description":"Split next-turn-is-tool-call (wordless continuation) from prose-lacking-ack-markers; state counts for both. Opus-4-7 74% ambiguous vs deepseek 17% is likely turn-structure variance, not behavior — this split disambiguates.","design":"Implementation home: the claim-vs-evidence classifier in devtools (devtools/ module behind `devtools workspace claim-vs-evidence`; tests tests/unit/devtools/test_claim_vs_evidence.py). Wordless-continuation detection: for each failure's paired next assistant message, check whether its blocks contain tool_use and no text block with \u003eN chars before the first tool_use — that is 'wordless continuation'; prose without matched ack markers stays 'ambiguous-prose'. Emit both as classification_reason variants (field already exists) and add the two counts to the report summary + by_model/by_tool cuts. Regen: `devtools workspace claim-vs-evidence --limit 5000 --out-dir .agent/demos/claim-vs-evidence --json`. Acceptance: report shows ambiguous split into wordless_continuation vs prose_no_marker with counts; per-model ambiguous variance (opus-4-7 74% vs deepseek 17%) re-examined after the split.","notes":"2026-07-03 Codex WIP: unit implementation for ambiguous split passes focused tests, but live regeneration with --limit 5000 became too slow and had to be killed twice. First attempt used correlated subqueries for next-message block shape; second used set-based CTE; third used chunked second query after sampled rows, but the full command still exceeded 90s on active archive and ignored SIGINT while inside SQLite. Do not close or commit this slice until the live regeneration path is profiled/fixed. Dirty files currently show the WIP implementation: devtools/claim_vs_evidence.py and tests/unit/devtools/test_claim_vs_evidence.py. Last passing focused proof: python -m py_compile + ruff check + devtools test tests/unit/devtools/test_claim_vs_evidence.py -\u003e 3 passed.","status":"closed","priority":1,"issue_type":"task","assignee":"Sinity","owner":"ezo.dev@gmail.com","created_at":"2026-07-03T04:31:28Z","created_by":"Sinity","updated_at":"2026-07-03T07:45:10Z","started_at":"2026-07-03T07:09:21Z","closed_at":"2026-07-03T07:45:10Z","close_reason":"Completed: claim-vs-evidence now splits ambiguous follow-ups into wordless tool continuations and prose-without-marker buckets, reports the counts in JSON/README summaries, and regenerates the current demo on the active archive. Focused tests pass; live regen/check completed.","labels":["area:substrate","campaign"],"dependencies":[{"issue_id":"polylogue-sru.2","depends_on_id":"polylogue-sru","type":"parent-child","created_at":"2026-07-03T06:31:27Z","created_by":"Sinity","metadata":"{}"}],"dependency_count":0,"dependent_count":1,"comment_count":0} {"_type":"issue","id":"polylogue-sru.3","title":"Benign-recovery vs consequential-silence split by handler kind","description":"Read failures are ~94% silent but 'tried another path' is usually benign; Bash/test failures are the consequential class. Scope the headline to consequential handler kinds or add an explicit split — credibility depends on not inflating with trivial recoveries.","design":"Handler kind is already available on the paired failure row (actions lane exposes handler/tool). Define the consequential set explicitly in code (Bash/test/build/write-class handlers) and the benign-recovery set (Read/Glob/Grep-class 'tried another path'), emit split headline rows: silent-proceed among consequential vs among all. Keep the mapping a named constant with a rationale comment so reviewers can argue with it. Report both; never let the headline mix classes silently. Same regen/tests as the other methodology children.","status":"closed","priority":1,"issue_type":"task","assignee":"Sinity","owner":"ezo.dev@gmail.com","created_at":"2026-07-03T04:31:28Z","created_by":"Sinity","updated_at":"2026-07-03T07:58:08Z","started_at":"2026-07-03T07:55:37Z","closed_at":"2026-07-03T07:58:08Z","close_reason":"Completed: claim-vs-evidence now reports a first-class handler-class split separating consequential shell/edit/write-class tool failures from benign read/search/path-discovery failures and other tools. The regenerated active-archive artifact shows consequential=4,177 failures with 921 silent-proceed (22.0% lower bound), benign_recovery=633 with 166 silent-proceed (26.2%), and other=190 with 92 silent-proceed (48.4%). Focused tests and demo shelf checks passed.","labels":["area:substrate","campaign"],"dependencies":[{"issue_id":"polylogue-sru.3","depends_on_id":"polylogue-sru","type":"parent-child","created_at":"2026-07-03T06:31:28Z","created_by":"Sinity","metadata":"{}"}],"dependency_count":0,"dependent_count":1,"comment_count":0} {"_type":"issue","id":"polylogue-sru.1","title":"Expose action-unit outcome fields + followup_class as product capability","description":"Capabilities-may-not-be-silos gate for the campaign: the facts the report needs must become composable query capability. After this, the whole report is `actions where is_error:true | group by session.origin, followup_class | count` and every future cut (model/tool/repo/time) is free.","design":"1) is_error/exit_code are normalized at parse time (sources/parsers/base_models.py:74-75) but ActionQueryRowPayload (surfaces/payloads.py:~1298) carries neither — add as filterable/groupable action-unit fields. 2) Add derived followup_class (acknowledged|silent_proceed|wordless_continuation|ambiguous) + followup_message_ref computed in the source-derived lowering (no cache tables). 3) Reduce devtools workspace claim-vs-evidence to a render preset over these query strings, or retire it. Touchpoint chain: stage parser -\u003e AST to_payload -\u003e executor -\u003e metadata.py aggregate_group_fields -\u003e shell_completion_values.py -\u003e devtools render openapi + cli-output-schemas + cli-reference. Line refs pre-07-03; re-locate.","acceptance_criteria":"Fixture session with known unacknowledged failure fires via pure query strings; report README numbers reproducible from the printed queries.","notes":"Completed: action-unit outcome follow-up classification is now shared query capability. is_error/exit_code were already wired; this slice added source-derived followup_class and followup_message_ref over existing actions/messages/blocks, exposed followup_class as filterable/groupable action metadata, added action row payload fields, routed root CLI terminal-unit aggregate expressions before session-selector compilation, and moved the report classifier from scripts into polylogue.archive.actions.followup. Reproduction/query forms are now printed in .agent/demos/claim-vs-evidence/PUBLIC_REPRODUCTION.md: actions where is_error:true | group by followup_class | count; actions where followup_class:silent_proceed. Verification: focused DSL/report/CLI tests passed; active demo packet regenerated over archive root /home/sinity/.local/share/polylogue schema v23 with 41,886 structured failures and 5,000 inspected; devtools verify --quick passed run 20260703T092510Z-quick-718233-46e8b587.","status":"closed","priority":1,"issue_type":"feature","assignee":"Sinity","owner":"ezo.dev@gmail.com","created_at":"2026-07-03T04:31:27Z","created_by":"Sinity","updated_at":"2026-07-03T09:25:36Z","started_at":"2026-07-03T09:05:37Z","closed_at":"2026-07-03T09:25:36Z","close_reason":"Completed","labels":["area:query","area:substrate","campaign"],"dependencies":[{"issue_id":"polylogue-sru.1","depends_on_id":"polylogue-sru","type":"parent-child","created_at":"2026-07-03T06:31:26Z","created_by":"Sinity","metadata":"{}"}],"dependency_count":0,"dependent_count":0,"comment_count":0} +{"_type":"issue","id":"polylogue-j8yo","title":"AI Studio browser-capture adapter: SKIP — live transport is undocumented internal RPC, not the Drive JSON","description":"Investigation for the hypothesis: \"aistudio.google.com fetches essentially the\nsame underlying JSON that ends up on Drive, so a browser-capture adapter\nwould be cheap.\" Verdict: SKIP for now. Evidence below.\n\n## Method\n\nLive authenticated browser (sinnix-chrome-control --target live), read-only:\nopened aistudio.google.com, listed the prompt library, navigated to two\ndistinct existing prompts, captured CDP Network domain traffic (not\npage-level fetch/XHR hooks or Resource Timing -- per the operator's prior\nfinding on Claude Design, those miss RPC transports; CDP Network did not).\n\n## (a) What the live app actually fetches\n\nPrompt URLs are literally Drive file ids\n(`aistudio.google.com/prompts/1cVKebxYa9oOCM05J3BqQQizFzIgfjvyH` etc. --\n`1...` prefix is the standard Drive file-id shape). Opening a prompt does\nNOT issue a plain REST GET, and does NOT call `drive.googleapis.com`\ndirectly. All application data comes from one internal RPC service:\n\n POST https://alkalimakersuite-pa.clients6.google.com/$rpc/google.internal.alkali.applications.makersuite.v1.MakerSuiteService/\u003cMethod\u003e\n\nMethods observed opening two different prompts: GetLoggingContext,\nGetUserPreferences, GenerateAccessToken, ListPromos, GetAiStudioBenefitTier,\nListModels, ListRecentApplets, ListPrompts, ResolveDriveResource. Response\nContent-Type for all of them: `application/json+protobuf` (confirmed via\nCDP response headers on ResolveDriveResource, status 200).\n\n`application/json+protobuf` is Google's internal positional-array RPC\nframing (same family used by Photos/Keep/Docs-style Closure apps): the body\nis syntactically valid JSON but semantically an array of proto field values\nkeyed by field NUMBER, not name -- there is no published `.proto` for\n`google.internal.alkali.applications.makersuite.v1.MakerSuiteService`, so\nturning it into named fields means reverse-engineering positional mappings\nper RPC method, with no compatibility guarantee across Google's backend\ndeploys. This is the same failure class the operator's Claude Design\ncomparison hit (page-level hooks missed it; the format itself is\nundocumented and fragile), not a REST/JSON API.\n\n## (b) Does it match the Drive-synced shape?\n\nNot directly. `ResolveDriveResource` is the RPC that resolves a prompt id to\nits Drive-resident content, but it goes through MakerSuiteService's own\nproxy/serialization, not a client-visible `drive.googleapis.com` files.get.\nThe *canonical stored artifact* is the same Drive file the operator's Drive\nsync already downloads (both ultimately reference the identical Drive\nobject), but the *live wire representation* is not the plain object-keyed\nJSON polylogue already parses (`polylogue/sources/parsers/drive.py`) -- it\nis the positional `application/json+protobuf` RPC envelope. So the premise\n\"same JSON, cheap adapter\" is false at the transport level even though the\nunderlying data is the same document.\n\nNo content-bearing RPC beyond ResolveDriveResource was captured in two\n~20s windows across two different prompts; either the message content\nrides inside that same RPC's payload (plausible -- one full prompt fetch\nper open) or a further call wasn't triggered in the capture window. Either\nway nothing suggests a second, cleaner JSON transport exists alongside it.\n\n## (c) What would a live adapter gain that Drive cannot?\n\nChecked against the two most-cited justifications and found both already\nsatisfied by the Drive-synced file itself:\n\n- **Drafts/unsaved runs**: `chunkedPrompt.pendingInputs` -- the exact\n not-yet-submitted textbox content -- IS present in the Drive-synced JSON.\n Verified against the live archive: 396/397 aistudio-drive raw sessions\n carry a `pendingInputs` entry, 7 with non-blank draft text (one a full\n multi-paragraph prompt that was never sent). Just parsed and landed as a\n `draft_input` session_event (polylogue-o4j2, PR pending). Drive sync\n already captures this; live capture would not add draft coverage.\n- **Generation params absent from the synced file**: none found. runSettings\n (temperature/topP/topK/maxOutputTokens/thinkingLevel/safetySettings/\n enable* flags) is present verbatim in the Drive-synced JSON and already\n reaches `sessions.run_settings_json` (polylogue-2qx.4/cgfy, index v46,\n PR #3390, predating this investigation).\n\nRemaining plausible (unverified, not measured this session) gains:\n- **Realtime vs Drive-sync polling lag**: real, but modest -- Drive sync\n latency is not the archive's current bottleneck for this origin (397\n sessions total, low volume).\n- **Sessions later deleted from Drive/AI Studio**: real edge case, same\n argument applies to every delete-capable source; not AI-Studio-specific.\n- **Removing the separate Drive OAuth flow**: real operational simplification\n (one fewer auth surface) but orthogonal to data completeness.\n\n## (d) Cost\n\nBuilding a live adapter would mean either (i) reverse-engineering\n`application/json+protobuf` positional RPC payloads for\n`MakerSuiteService` with no published schema and no stability guarantee, or\n(ii) falling back to DOM scraping of the rendered chat UI (the pattern the\nexisting ChatGPT/Claude browser-capture adapters already use) -- itself a\nreal, non-trivial adapter (selectors, pagination, run-settings-panel\nscraping, draft-textbox capture) comparable in cost to any other\nbrowser-capture origin, not a cheap win from shape-reuse.\n\n## Recommendation: SKIP\n\nThe \"cheap because same JSON\" premise does not hold: the live transport is\nan undocumented internal RPC (protobuf-JSON hybrid), not the archive's\nalready-parsed Drive JSON shape. The two headline capabilities a live\nadapter was hoped to add -- drafts and generation params -- are already\npresent in the Drive-synced file and now parsed (o4j2). What remains\n(latency, one fewer OAuth flow, delete-survivorship) does not clear the bar\nof reverse-engineering an undocumented Google-internal RPC surface, or\nbuilding a from-scratch DOM-scrape adapter at ordinary browser-capture cost.\nRevisit only if the operator specifically wants realtime AI Studio capture\nregardless of cost, or if a documented public transport for AI Studio\nappears.\n\nRead-only investigation; no AI Studio content was created, edited, or\ndeleted. Evidence lives in this issue only (raw archive blob paths quoted,\nnot copied) to avoid persisting the operator's personal draft-prompt content\ninto tracked repo files.","status":"open","priority":2,"issue_type":"task","owner":"ezo.dev@gmail.com","created_at":"2026-07-31T04:00:47Z","created_by":"Sinity","updated_at":"2026-07-31T04:00:47Z","dependency_count":0,"dependent_count":0,"comment_count":0} +{"_type":"issue","id":"polylogue-knc7","title":"model claude subscription session and weekly credit windows","description":"polylogue/cost/plans.py models only a MONTHLY quota (SubscriptionPlan has quota, quota_basis, billing_cycle_days, cycle_anchor_day). It has NO session-window or weekly field. The two limits that actually bind in practice are therefore unmodelled.\n\nPublished figures (she-llac.com/claude-limits, dated 2026-01-25):\n plan 5-hour session weekly\n Pro 550,000 5,000,000\n Max 5x 3,300,000 41,666,700\n Max 20x 11,000,000 83,333,300\n\nOur seeded monthly quotas (21.7M / 180.6M / 361.1M) match that source exactly, as do the per-model credit rates in archive/semantic/subscription_pricing.py (opus 10/50, sonnet 6/30, haiku 2/10, cache_read 0, cache_write at the input rate). So the rate model is right; the WINDOW model is missing.\n\nWhy it matters: monthly quota is almost never the binding constraint - you get rate-limited by the 5-hour window mid-session. Today polylogue can say what a session cost in credits but not whether it would have exhausted a window, which is the operationally useful question.\n\nNote the weekly limits are NOT monthly/4 and are not derivable: Pro 5M x4 = 20M against a 21.7M month, but Max 20x 83.3M x4 = 333M against 361.1M. The ratio differs per tier, so both numbers must be carried explicitly.\n\nAC: SubscriptionPlan carries session-window and weekly quotas with their window lengths; a session can be evaluated against them; and a query can answer 'did this session approach a window limit'.","status":"open","priority":2,"issue_type":"task","owner":"ezo.dev@gmail.com","created_at":"2026-07-31T03:56:51Z","created_by":"Sinity","updated_at":"2026-07-31T03:56:51Z","dependency_count":0,"dependent_count":0,"comment_count":0} +{"_type":"issue","id":"polylogue-t83q","title":"subscription credit rates are missing the Claude 5 model family","description":"polylogue/archive/semantic/subscription_pricing.py declares ModelCreditRate rows for claude-opus-4-6, claude-opus-4-5, claude-sonnet-4-6, claude-sonnet-4-5, claude-haiku-4-5. The Claude 5 family (claude-opus-5, claude-sonnet-5) is ABSENT, and those are the current models - this very session runs on Opus 5.\n\ndocs/cost-model.md states credits are emitted only for models with a DECLARED rate, 'never a fabricated figure'. That is the correct failure mode, but the consequence is that current-model sessions silently produce no subscription_credit_usd at all, so credit accounting has a growing blind spot exactly where usage is concentrated.\n\nDo NOT simply copy the 4.x rates forward - whether Opus 5 inherits 10/50 and Sonnet 5 inherits 6/30 is an assumption, not a verified fact. Source the real rates before declaring them, and if they cannot be sourced, record that explicitly rather than guessing.\n\nRelated staleness: CURATED_SEED_EFFECTIVE_DATE is 2026-05-17 and the upstream reference (she-llac.com/claude-limits) was published 2026-01-25 with no update date, six months stale as of 2026-07-31. Its own wording ('actual multiplier: 6-8.33x') signals reverse-engineering rather than published spec. Neither figure is verifiable from local data: session JSONL carries cost_usd (API-list-equivalent) and token counts, never subscription credits, so there is no ground truth on disk to test the formula against. Treat the whole credit model as best-effort inference and label it as such wherever it surfaces.\n\nAC: Claude 5 rates present with a sourced provenance note, or an explicit recorded statement that they are unavailable; and a check that flags when a model appearing in the archive has no declared credit rate.","status":"open","priority":2,"issue_type":"task","owner":"ezo.dev@gmail.com","created_at":"2026-07-31T03:56:51Z","created_by":"Sinity","updated_at":"2026-07-31T03:56:51Z","dependency_count":0,"dependent_count":0,"comment_count":0} +{"_type":"issue","id":"polylogue-54gj","title":"Grok-on-X (x.com/twitter.com) has no capture path after DOM removal","description":"The Grok native-capture upgrade (grok.com REST adapter + grok_bridge.js) deleted the old grok-dom-v1 fallback and dropped x.com/twitter.com from manifest.json's content_scripts and background.js's injectionPlanForUrl, because grok.com's /rest/app-chat/* REST surface this bridge calls is same-site to grok.com only -- X's embedded Grok surface is served through X's own API (not verified live in that session; no authenticated x.com Grok conversation tab was available). background.js's archiveProviderForUrl/conversationIdForUrl and popup.js's provider labeling still classify x.com/twitter.com as the 'grok' provider and show a 'Grok / X' label, but no content script is installed there anymore, so any auto-capture trigger targeting those tabs now silently finds no listener (captureTab's injectionPlanForUrl(...).length guard already short-circuits it cleanly -- no hang, no error -- but the UI label is misleading).\n\nFollow-up scope:\n1. Verify live whether x.com's embedded Grok assistant actually has its own distinct GraphQL/REST API, and if so build a dedicated adapter for it (same shape as GrokBackfillAdapter/grok.js, different origin/endpoints).\n2. If not pursued, drop x.com/twitter.com from archiveProviderForUrl/popup provider labeling and host_permissions so the UI stops claiming a capture path that doesn't exist.","status":"open","priority":2,"issue_type":"task","owner":"ezo.dev@gmail.com","created_at":"2026-07-31T03:34:11Z","created_by":"Sinity","updated_at":"2026-07-31T03:34:11Z","dependency_count":0,"dependent_count":0,"comment_count":0} +{"_type":"issue","id":"polylogue-u8x7","title":"Extend field-path union coalescing to web_content_constructs/file_edits and session_model_usage","description":"Follow-up to polylogue-geop (field-path union coalescing for provider exports,\nimplemented in messages/blocks via _union_with_existing_rows in\npolylogue/storage/sqlite/archive_tiers/write.py).\n\nScope deliberately deferred from the initial implementation:\n\n1. web_content_constructs and file_edits are derived sidecar tables\n populated ONLY from the current acquisition's parsed ParsedMessage/\n ParsedContentBlock domain objects (_write_web_constructs/_write_file_edits\n in write.py), not from the merged/unioned row tuples. When a message is\n reinjected by the field-path union because a newer acquisition dropped it\n entirely, its web_content_constructs/file_edits rows are NOT restored\n (they were deleted by the session-scoped replace and nothing repopulates\n them, since the domain object carrying that data no longer exists in\n this write's `messages` list). This directly affects the bead's own\n measured scenario: metadata.content_references citations map onto\n web_content_constructs.\n\n2. session_events / session_model_usage: the union operates only on\n messages/blocks. A reinjected message's model_name produces a\n zero-usage session_model_usage skeleton row (see\n test_provider_usage_model_vanishing_on_reingest_preserves_message_with_zero_usage_rollup)\n because token_count session_events aren't unioned across acquisitions.\n Consider extending the same union principle to session_events keyed by a\n stable native event id, if one exists per provider.\n\nBoth would need the same \"read existing rows before delete, reinject/merge,\nskip for prefix-sharing lineage parents\" pattern already established in\n_union_with_existing_rows, extended per-table.","notes":"2026-07-31 update: the initial polylogue-geop implementation applied field-path\nunion unconditionally to every full-replace, which broke ~19 tests (browser-\ncapture/native-vs-DOM-fallback precedence, same-acquisition re-parse\nretraction). Fixed by gating union on a raw_id-based discriminator: union only\nfires when the incoming and previously-stored sessions.raw_id are both known\nand differ (proven different acquisition), or is skipped when they're equal\n(same acquisition re-parsed), either is unknown, or the caller passed\nforce_replace=True (an explicit precedence decision, e.g.\nbrowser_capture_precedence()). See _union_with_existing_rows in\npolylogue/storage/sqlite/archive_tiers/write.py.\n\nThis directly affects this follow-up's scope: extending union to\nweb_content_constructs/file_edits/session_events must respect the SAME\nraw_id/force_replace discriminator, not just the message/block matching\nlogic -- otherwise the same class of regression (same-acquisition re-parse\nunable to retract a stale citation/file-edit/usage row) would recur there.","status":"open","priority":2,"issue_type":"task","owner":"ezo.dev@gmail.com","created_at":"2026-07-31T03:19:09Z","created_by":"Sinity","updated_at":"2026-07-31T03:51:04Z","dependency_count":0,"dependent_count":0,"comment_count":0} +{"_type":"issue","id":"polylogue-tbun","title":"model claude design as a distinct origin, with webui representation","description":"MEASURED over the 11 design_chats in claude-ai-data-2026-07-30. Claude Design is NOT claude.ai with a flag - it is a different product with a different wire format, currently reduced to a CLAUDE_DESIGN_CHAT_INGEST_FLAG on a claude-ai-export session.\n\nWIRE SHAPE (all camelCase, vs claude.ai's snake_case - different backend):\n message: uuid, role, content, created_at where content is a DICT not a list\n content: role, content, id, timestamp, contentBlocks, authorAccountUuid,\n authorName, attachments, turnInputTokens, pill, turnChanges\n\nIT IS AN AGENTIC ENVIRONMENT, NOT A CHAT. Across 11 chats: 751 tool_call\nblocks, 211 thinking, 175 text, 10 error, 5 user_interjection. Tools used:\n write_file 146, snip 121, read_file 120, update_todos 50, str_replace_edit 48,\n github_read_file 31, done 29, github_get_tree 26, fork_verifier_agent 25,\n list_files 19, save_screenshot 16, local_read 15, grep 14, web_fetch 11\ntoolCall records carry id/type/name/input/output with toolu_* ids - the SAME id\nspace as the Claude API, so tool identity joins cleanly with claude-code.\n\nCONSTRUCTS WITH NO CLAUDE.AI EQUIVALENT:\n turnChanges {created, edited, deleted, moved} - a materialised filesystem\n diff PER TURN. Highest-value part; nothing else in the archive\n records what a turn changed on disk.\n user_interjection - a user message nested INSIDE an assistant turn. Flattening\n it to an ordinary user message destroys both the interruption\n semantics and the ordering.\n attachments typed file(143) skill(21) text(19) image(17) folder(2) - skills\n and folders as attachable objects.\n authorAccountUuid + authorName - named multi-account authorship; claude.ai\n exports have no author identity at all.\n turnInputTokens - per-turn token accounting.\n error blocks - refusals as first-class content.\n\nPROVIDER QUIRKS: every title is literally 'Chat' (titles must be derived, same\nclass as the claude-code raw-UUID title problem); content is a dict not a list,\nso a parser assuming the claude.ai shape fails immediately.\n\nWORK:\n1. Origin.CLAUDE_DESIGN_SESSION as a new token; retire\n CLAUDE_DESIGN_CHAT_INGEST_FLAG in the same change (hard rename, no compat).\n2. tool_call -\u003e TOOL_USE/TOOL_RESULT with shared tool_id (records already carry\n both sides). thinking -\u003e THINKING. text -\u003e TEXT. error -\u003e error block.\n3. turnChanges -\u003e per-turn session_event or a new construct type. Decide which.\n4. user_interjection -\u003e needs a real answer, not a flatten.\n5. attachment taxonomy gains skill and folder.\n6. Both acquisition paths: GDPR import AND browser-extension capture, coalescing\n on message uuid at field-path granularity (see the strict-containment bead) -\n design chats are the ideal first case since both sources will cover the same\n sessions.\n\nWEBUI (polylogue/daemon/webui.py, 1,638 lines, 59 functions): a design session\nrenders poorly as a chat transcript - it is 751 tool calls and 5 file mutations\nacross 11 sessions. It needs a session view that leads with turnChanges (what\nthis turn changed), folds tool calls by default like the reader already folds\ntool_use, and shows user_interjection inline at its true position rather than as\na sibling message. Scope note: the corpus is only 11 chats and the product is\nnew, so the parser should be strict about what it recognises and loud about what\nit does not, rather than guessing a shape that is still moving.","notes":"LIVE TRANSPORT DISCOVERED 2026-07-31 via CDP Network domain (page-level fetch hooks and resource-timing both showed nothing - this is why).\n\nClaude Design does NOT use a REST /api/ route. /api/organizations/\u003corg\u003e/design_chats 404s on every org. It uses a Connect-RPC service:\n\n POST https://claude.ai/design/anthropic.omelette.api.v1alpha.OmeletteService/\u003cMethod\u003e\n\nMethods observed on a project load (counts from one trace):\n GetFile x6, TrackEvent x4, ListFiles x2, UpdateProjectData, MintPreviewToken,\n McpStreamTools, McpListDesignImportPartners, ListUserSkills, ListOrgProjects,\n ListExperiences, ListComments, GoogleGetStatus, GithubGetStatus,\n GetUserSettings, GetUsageStatus, GetProjectPresence, GetProject,\n GetPrepaidBalance, GetOrgSettings\n\nCRITICAL FOR IMPLEMENTATION: responses are content-type **application/proto**\n(binary protobuf), not JSON. McpStreamTools is application/connect+proto\n(streaming). Only GetProjectPresence returned application/json.\n\nSo a live capture adapter CANNOT parse the wire the way the chatgpt/claude\nadapters do - there is no published .proto schema. Two viable directions:\n (a) hook the app's own DECODED objects in page context (MAIN world), after\n the Connect client has deserialised, rather than intercepting the wire;\n (b) reverse the protobuf shape per method, which is brittle and would break\n on any schema change.\n(a) is strongly preferred and matches how chatgpt_bridge.js already works\n(intercepting window.fetch and reading decoded JSON).\n\nAlso confirmed: design files render inside a SANDBOXED CROSS-ORIGIN IFRAME at\nhttps://\u003cproject-uuid\u003e.claudeusercontent.com/_bootstrap (subdomain IS the\nproject uuid), sandbox='allow-scripts allow-forms allow-popups allow-modals\nallow-downloads allow-same-origin'. Same pattern as artifacts. host_permissions\nfor https://*.claudeusercontent.com/* has now been added to manifest.json AND\nto scripts/validate-manifest.mjs's ALLOWED_HOST_GLOBS (the validator correctly\nrejected it until declared).\n\nNo GetChat/ListMessages method was observed, so the design conversation itself\nlikely arrives via GetProject, ListExperiences, or a stream - needs one more\ntrace with the project's chat pane actually loading.","status":"open","priority":2,"issue_type":"task","owner":"ezo.dev@gmail.com","created_at":"2026-07-31T01:43:06Z","created_by":"Sinity","updated_at":"2026-07-31T03:04:37Z","dependency_count":0,"dependent_count":0,"comment_count":0} +{"_type":"issue","id":"polylogue-xofj","title":"handle the six unmodelled chatgpt content types from the April-era format","description":"MEASURED in chatgpt-data-2026-04-23. parsers/chatgpt.py explicitly handles code, execution_output, thoughts, reasoning_recap, audio_transcription, user_editable_context/model_editable_context, image_asset_pointer and the audio pointer set - and recognises the tool role. These six are NOT handled and fall through to a generic TEXT block, so the content survives but the semantic type is lost:\n\n computer_output 8,192\n tether_browsing_display 1,399\n tether_quote 1,178\n system_error 177\n sonic_webpage 30\n citable_code_output 8\n\nThey are all code-interpreter / browsing-era constructs. computer_output is a\ntool result (pairs with the same tool_id logic execution_output already uses);\ntether_quote and tether_browsing_display are retrieved-source constructs and\nshould become web constructs, not text; system_error is an error block;\ncitable_code_output is a code result with citation anchors.\n\nThese only ever appear in the April-and-earlier format - the July 2026 export\ndeleted the whole tool layer (see the strict-containment bead) - so this is\nhistorical-format support. We want it anyway: the April export is the sole\nsurviving record of that layer.\n\nAC: each of the six maps to a typed block or web construct rather than TEXT;\na re-import of the April export shows the new typed rows; and the mapping is\ncovered by a parser test using a real (anonymised) node of each shape.","notes":"Implemented in PR #3408 (branch feature/parsers/chatgpt-april-content-types-and-web-constructs). All six content types (computer_output, tether_browsing_display, tether_quote, system_error, sonic_webpage, citable_code_output) now map to typed blocks/constructs in polylogue/sources/parsers/chatgpt.py, each covered by a parser test using an anonymized real-node shape. Not yet merged/deployed -- re-import of the April export against the live archive still pending until PR lands.","status":"open","priority":2,"issue_type":"task","owner":"ezo.dev@gmail.com","created_at":"2026-07-31T01:42:36Z","created_by":"Sinity","updated_at":"2026-07-31T03:20:03Z","dependency_count":0,"dependent_count":0,"comment_count":0} +{"_type":"issue","id":"polylogue-zocm","title":"both parsers under-populate the web-construct vocabulary","description":"VERIFIED 2026-07-31 by counting WebConstructType references per parser.\n\n chatgpt.py emits 9 types: SEARCH_QUERY, CONTENT_REFERENCE, ASYNC_TASK,\n SELECTED_SOURCE, SEARCH_RESULT, IMAGE_RESULT, CANVAS, AUDIO_TRANSCRIPTION,\n AUDIO_ASSET\n claude/*.py emits 2 types: CANVAS, CONTENT_REFERENCE\n\nSo the vocabulary is right and provider-neutral; the population is wrong, in\ntwo different ways.\n\nGAP 1 - chatgpt loses 60.8% of citation URLs. _construct_from_reference\ndescends into item.metadata and item.metadata.extra but NOT into item.items /\nitem.fallback_items, which is where grouped_webpages keeps its URLs.\nMeasured over the July export:\n 6,596 content_references[].url EXTRACTED\n 10,130 content_references[].items[] NOT extracted\n 116 content_references[].fallback_items[] NOT extracted\n -\u003e 10,246 / 16,842 URLs (60.8%) never become constructs.\nThe search_result_groups loop already does exactly this descent\n(group.results/items/search_results/sources) - content_references needs the\nsame treatment.\n\nGAP 2 - claude does not distinguish retrieved from cited. Claude's export\ncarries BOTH layers and they are semantically distinct:\n 326 anchored citations on text blocks\n {uuid, start_index, end_index, details:{type:web_search_citation,url}}\n -\u003e these are CITED, with a character span into the answer text\n 1,514 URLs inside web_search tool_result content\n {type:knowledge, title, url, metadata:{site_domain, site_name,...}}\n -\u003e these are RETRIEVED, never necessarily cited\nOnly the first becomes a CONTENT_REFERENCE; the retrieved set stays buried in\ntool_result text and never becomes SEARCH_RESULT constructs.\n\nGetting this wrong in the obvious direction would make ChatGPT look like it\ncites 25x more than Claude when it mostly just reads more. CONTENT_REFERENCE\nshould mean cited-with-span; SEARCH_RESULT should mean retrieved.\n\nAC: chatgpt nested citation items become constructs; claude web_search results\nbecome SEARCH_RESULT constructs; a query can distinguish 'sources cited' from\n'sources read' for both providers.","notes":"Implemented in PR #3408 (branch feature/parsers/chatgpt-april-content-types-and-web-constructs). GAP 1 (chatgpt): content_references/citations now descend into item.items[]/item.fallback_items[] via _constructs_from_content_reference_item, mirroring the existing search_result_groups descent. GAP 2 (claude): content_blocks_from_segments (base_support.py, shared by codex/claude) projects web_search tool_result {type:knowledge} entries as SEARCH_RESULT constructs, kept distinct from the existing CONTENT_REFERENCE citation-anchor projection in claude/common.py so cited vs retrieved sources stay separately queryable. Not yet merged.","status":"open","priority":2,"issue_type":"task","owner":"ezo.dev@gmail.com","created_at":"2026-07-31T01:42:36Z","created_by":"Sinity","updated_at":"2026-07-31T03:20:15Z","dependency_count":0,"dependent_count":0,"comment_count":0} +{"_type":"issue","id":"polylogue-dt5s","title":"capture model-produced sandbox files as first-class references","description":"MEASURED 2026-07-31 against the 2026-07-29 chatgpt export.\n\nThe model writes files into its sandbox and links them as sandbox:/mnt/data/\u003cname\u003e. These are a DISTINCT population from user uploads and are currently invisible to polylogue.\n\nScale: 639 assistant messages carry such links; 1,782 distinct output filenames.\nExtensions: md 1025, csv 299, json 298, zip 193, png 178, patch 159, txt 113,\ngz 78, jsonl 75, py 61, sh 55, yaml 54.\n\nTHE KEY CONSTRAINT: a sandbox link carries NO file id. The assistant message\nmetadata on those 639 messages contains only model_slug/parent_id/content_references\n- no attachment record, no asset_pointer, no file id. So there is no id join.\n\nBYTE AVAILABILITY (name-match is the only join available):\n 1,782 distinct sandbox output names\n 823 match a library_files.file_name\n 40 match a content_references name\n 27 match a conversation_asset_file_names value\n 826 resolvable by ANY of the three (46.4%)\n 956 have NO byte source anywhere (53.6%)\n\nSo roughly half the model-produced files are recoverable, and only via filename -\nwhich is fuzzy and can collide. Treat a name match as EVIDENCE, not identity:\nrecord how the link was resolved so a wrong match is auditable, and never let a\nname match mint the same identity as an id match.\n\nFOR THE OTHER 956: capture metadata anyway - filename, sandbox path, extension,\nproducing message id, conversation, timestamp - as a model-produced-file\nreference with no bytes. Operator directive: better a recorded absence with\nmetadata than silence. This also makes the population countable, so a future\nexport or browser capture that DOES carry the bytes can be joined to it.\n\nRELATED CORRECTION: message.metadata.attachments[] (3,444 ids) are ALL on user\nmessages - they are uploads, not model output. Do not conflate the two.","notes":"CORRECTION + REAL SPEC (2026-07-31). The earlier 'only fuzzy filename matching, 46%' was wrong. I had truncated the library_files key list to the first 9 keys and concluded from what I could see. The full schema carries an EXACT producing-message id.\n\nRelevant library_files fields (2,367 entries):\n origination_message_id 1,742 \u003c- exact id of the assistant message that produced the file\n origination_thread_id 1,733 \u003c- conversation\n sha256_digest 733 \u003c- content addressing / dedup\n library_artifact_type 953 (other 840, report 43, image 36, image_gen 14,\n writing_block 11, deep_research_report 7, sheet 2)\n initiating_conversation_id 0 (always null - do not use)\n file_name_provenance: 'upload' for ALL 2,367, so it does NOT distinguish\n model-produced from uploaded. Provenance comes from origination_message_id\n being set, not from this field.\n\nTIERED RESOLUTION, measured over all 2,943 (message, sandbox-filename) links:\n\n tier links with bytes\n 1 exact msg id + name match 1,412 1,319\n 2 exact msg id, name differs 418 418\n 3 thread id + name 2 2\n 4 global name, provably UNIQUE 91 91\n 5 global name, AMBIGUOUS 0 0 \u003c- none exist\n 6 unresolved, metadata only 1,020 0\n\n identity-grade (1-3) with bytes: 1,739 = 59.1% of all links\n zero genuinely ambiguous name matches in the entire corpus\n\nSo the implementation is a layered resolver, not a fuzzy matcher:\n 1. join on origination_message_id (identity-grade; note tier 2 - the library\n name can differ from the linked name, so match on the id ALONE and treat\n the name as a label, not a key)\n 2. fall back to (origination_thread_id, file_name)\n 3. fall back to a global name match ONLY when it is provably unique\n 4. otherwise record a metadata-only model-produced-file reference\n\nRecord which tier resolved each link so a later audit can distinguish an id\njoin from a name join. Tier 4 should be marked as evidence rather than\nidentity, but the collision risk that motivated that caution does not\nmaterialise here (tier 5 is empty).\n\nUse sha256_digest where present for content-addressed dedup against blobs\nalready stored from other sources.\nIMPLEMENTED (branch feature/sources/chatgpt-export-assets-and-sidecars, PR pending).\n\nImplemented the exact 6-tier resolver from the corrected spec:\nChatGPTAssetIndex.resolve_sandbox in polylogue/sources/parsers/chatgpt_sidecars.py.\nTier 1 (msg id + name exact), tier 2 (msg id only -- name is a label, not a\nkey, per spec), tier 3 (thread id + name), tier 4 (globally unique name),\ntier 5 (globally ambiguous name -- evidence, no file), tier 6 (unresolved,\nmetadata-only). Wired into chatgpt_assembly.py's enrich_session: for tiers\n1-4, attachment.provider_file_id is updated to the matched library file_id\n(real identity strengthening); every tier, including 6, gets a\nchatgpt_sandbox_file_resolution session_event recording which tier resolved\nit (audit trail per the spec's directive).\n\nRe-measured resolver behavior against the real corpus (all 29\nconversations-*.json shards + library_files.json): tiers\n{1: 1273, 2: 370, 3: 2, 4: 83, 6: 995} over 2,723 links found by my\nverification harness (some magnitude difference from the bead's own 2,943\ncount is expected -- my harness only scanned \"parts\"-shaped assistant text\nfor sandbox links as a sanity check; the actual production\n_sandbox_file_paths/_extract_content_text already covers more content\nshapes). Tier 3 count (2) matches exactly. Zero tier-5 ambiguous matches,\nmatching the spec's claim that no genuine collision exists in this corpus.\n\nBytes for the resolved fraction: still not acquired (see polylogue-8ac0,\nfiled as the shared byte-acquisition follow-up for both this bead and\npolylogue-0hwv -- the .dat blob itself needs the same streaming-ZIP-scan\nwork regardless of which resolver named it). Tier 6 (the ~35% with no id/\nname evidence at all) already gets the \"recorded absence with metadata\"\ntreatment the bead asked for: filename, sandbox path, extension (via name),\nproducing message id, and tier=6/method=unresolved on the session_event --\nno bytes were ever going to be available for this population regardless of\nthe acquisition follow-up.\n\nCORRECTION to my previous note's tier-count claim (2026-07-31, caught by\ncoordinator review before merge): I wrote the re-measured tiers were\n\"consistent with the spec\"; they were NOT identical, and I had not run the\nreconciliation needed to say why before making that claim.\n\nRoot cause, now confirmed exactly: this bead's measured spec counted every\nraw sandbox-link OCCURRENCE (regex match on assistant text). Reproducing\nthat exact counting method against the real corpus gives\n{1: 1412, 2: 418, 3: 2, 4: 91, 5: 0, 6: 1020} sum 2943 -- bit-for-bit\nidentical to the spec in every tier. But the PR's actual production\nattachments are built by chatgpt.py's pre-existing _sandbox_file_paths()\n(not touched by this PR), which deduplicates repeated identical sandbox\nlinks WITHIN one message's text before any attachment is constructed -- a\nmessage that links the same file twice yields one ParsedAttachment, not\ntwo. Counting production attachments (the honest apples-to-apples number\nfor what actually lands in the archive) gives\n{1: 1273, 2: 370, 3: 2, 4: 83, 6: 995} sum 2723 (-7.5% overall, every\npopulated tier down by roughly the same proportion). This is a denominator\ndifference (occurrences vs. distinct (message,filename) pairs), not a\nresolver disagreement, and it is the CORRECT product behavior (no duplicate\nattachment rows for a repeated identical link) -- but the two counts are\nnot interchangeable and I should not have called them consistent without\ndoing this reconciliation first.\n\nWhat DOES hold exactly, in both countings, and is the structurally\nload-bearing result: tier 5 (globally-ambiguous name) is ZERO -- no\nfuzzy-match collision exists anywhere in the corpus -- and tier 3 is 2.\nThose are what actually validate the tiered-resolver design over a flat\nfuzzy matcher; the rest is denominator noise from a pre-existing\ndeduplication step this PR did not introduce and did not need to change.\n","status":"closed","priority":2,"issue_type":"task","assignee":"Sinity","owner":"ezo.dev@gmail.com","created_at":"2026-07-31T00:57:54Z","created_by":"Sinity","updated_at":"2026-07-31T03:55:39Z","started_at":"2026-07-31T03:32:13Z","closed_at":"2026-07-31T03:55:39Z","close_reason":"Merged in PR #3409 (polylogue/master@11403388d): 6-tier sandbox-file resolver implemented exactly per spec, tier recorded per link via session_events, tier-6 unresolved links still get a metadata-only reference. Tier 5 (ambiguous) confirmed zero, tier 3 confirmed 2, both exact matches to the measured spec.","dependency_count":0,"dependent_count":0,"comment_count":0} +{"_type":"issue","id":"polylogue-80ks","title":"audit browser-capture attachment parity against GDPR export fidelity","description":"Open question raised 2026-07-31: does browser-extension capture handle attachments/images/audio as well as the GDPR export path does?\n\nPartial evidence (not a full audit): browser_capture/models.py has mime_type + extracted_content; parsers/browser_capture.py builds inline_bytes via _browser_capture_attachment_inline_bytes and merges them across candidates, with upload_origin url|paste|oauth. So TEXT extraction and pasted bytes are modelled.\n\nUnverified: whether binary image/audio bytes are captured at all from the live DOM, or only a URL + extracted text; and whether an asset captured live and later re-delivered by a GDPR export coalesces to one attachment or duplicates.\n\nThis matters more now that exports ship real bytes (see polylogue-0hwv): the two paths could disagree about what an attachment IS, which is the aggz-invariant-2 shape (two write paths, one forgets).\n\nAC: a per-modality matrix (text / image / audio / model-produced file) x (browser capture / GDPR export) stating what is stored for each, with the gaps either fixed or recorded as deliberate.","status":"open","priority":2,"issue_type":"task","owner":"ezo.dev@gmail.com","created_at":"2026-07-31T00:50:13Z","created_by":"Sinity","updated_at":"2026-07-31T00:50:13Z","dependency_count":0,"dependent_count":0,"comment_count":0} +{"_type":"issue","id":"polylogue-2m2e","title":"chatgpt export sidecars library_files.json and codex.json are unparsed","description":"The 2026-07-29 chatgpt export contains sidecars polylogue does not reference at all (verified by rg over polylogue/):\n\n- library_files.json - 2,367 entries, the ChatGPT Library (generated/uploaded file collection) with sha256 digests, context scopes, versions\n- codex.json - 20 Codex threads with a 'turns' structure, i.e. cloud-Codex sessions delivered through the chatgpt export rather than ~/.codex\n\nshared_conversations.json (154) IS referenced in dispatch.py. message_feedback.json (21 ratings) and ads.json (empty) are low value.\n\ncodex.json is the interesting one: it is a second, independent delivery path for Codex sessions, so it risks either absence or duplicate identity against codex-session origin records.\n\nAC: decide per sidecar - parsed, or explicitly out of scope with the reason recorded. For codex.json specifically, determine whether its threads coalesce with existing codex-session sessions or create duplicates.","notes":"IMPLEMENTED (branch feature/sources/chatgpt-export-assets-and-sidecars, PR pending).\n\nPer-sidecar decision, as the AC asked:\n\n- library_files.json: PARSED. Feeds ChatGPTAssetIndex (polylogue-0hwv/\n polylogue-dt5s resolvers) as the primary (richer) name/mime/size/sha256/\n origination-id source. Deferred, not silently dropped: the sub-population\n of library files with NO origination_message_id/thread_id AND never\n referenced by any conversation attachment or sandbox link (measured\n ~1,438 in the bead's own notes) is not yet surfaced as a first-class\n standalone reference -- that needs whole-source-scan aggregation\n (tracking every file_id actually consulted across all sessions from one\n source, then diffing against the full library_files population) that\n the current per-session enrich_session hook doesn't have a natural home\n for. Left as an explicit gap rather than building a half-working\n aggregation path under this PR's budget; worth its own follow-up if the\n operator wants that population queryable.\n- conversation_asset_file_names.json: PARSED (already covered by\n polylogue-0hwv's resolver as the fallback name source).\n- codex.json: PARSED as first-class sessions. New parser\n polylogue/sources/parsers/chatgpt_codex_sidecar.py + a tight structural\n detector (task_e_\u003chex\u003e id + turns shape) wired into\n archive/artifact_taxonomy/runtime.py (classification -- without this a\n task record fails every session-document heuristic and is silently\n dropped before parsing ever runs) and sources/dispatch.py (routing to the\n new parser instead of chatgpt.parse, which would otherwise silently\n produce a zero-message, hence write-time-dropped, session for it).\n\n Coalescing question resolved: codex.json tasks do NOT coalesce with\n existing codex-session records. Confirmed both structurally and by test:\n local Codex CLI sessions are keyed by a rollout session_id UUID\n (sources/parsers/codex.py, Origin.CODEX_SESSION); these cloud tasks are\n keyed by task_e_\u003chex\u003e ids with turn ids task_e_\u003chex\u003e~usertrn_e_\u003chex\u003e /\n ~assttrn_e_\u003chex\u003e -- a disjoint namespace, verified against the real\n codex.json (codex.looks_like returns False on every real task record).\n Ingesting them adds one new session per task under\n source_name=Provider.CHATGPT (they physically arrive via this export)\n tagged ingest_flags=[\"capture:chatgpt-codex-cloud-task\"], never a\n duplicate of anything already archived.\n\n All 20 real tasks in the corpus now parse into 20 distinct 2-message\n sessions (previously 0 -- every one was silently dropped).\n\n- message_feedback.json / shared_conversations.json / ads.json: unchanged,\n per the bead's own framing (shared_conversations already referenced,\n message_feedback/ads low value) -- out of scope for this PR, no new\n decision needed.\n","status":"closed","priority":2,"issue_type":"task","assignee":"Sinity","owner":"ezo.dev@gmail.com","created_at":"2026-07-30T23:44:28Z","created_by":"Sinity","updated_at":"2026-07-31T03:55:39Z","started_at":"2026-07-31T03:32:14Z","closed_at":"2026-07-31T03:55:39Z","close_reason":"Merged in PR #3409 (polylogue/master@11403388d): library_files.json parsed (feeds the asset resolver), conversation_asset_file_names.json parsed (fallback name source), codex.json parsed as first-class sessions with confirmed-disjoint identity from codex-session records. Library files with no message reference as standalone first-class refs explicitly deferred (documented in bead notes, needs whole-source-scan aggregation not yet built).","dependency_count":0,"dependent_count":0,"comment_count":0} +{"_type":"issue","id":"polylogue-075v","title":"extend browser extension to capture Claude Design chats live","description":"Claude Design (claude.ai design mode) currently reaches the archive only via the GDPR export's design_chats/ directory - 11 sessions, 95 messages in the 2026-07-30 batch. That is a quarterly-batch path for a surface the operator uses interactively.\n\nThe browser-capture lane already handles claude.ai conversations end-to-end (browser+ext -\u003e receiver -\u003e spool -\u003e archive). Design chats are a distinct route/DOM on the same origin.\n\nNote the wire shape differs from ordinary conversations: design chats use messages[]/role rather than chat_messages[]/sender, plus project/title/uuid. ai_parser._parse_design_chat already handles the export shape and should be the target model.\n\nAC: design chats captured live by the extension land as claude-ai sessions equivalent to their export representation, and a session captured both ways coalesces rather than duplicating.","status":"open","priority":2,"issue_type":"task","owner":"ezo.dev@gmail.com","created_at":"2026-07-30T22:05:28Z","created_by":"Sinity","updated_at":"2026-07-30T22:05:28Z","dependency_count":0,"dependent_count":0,"comment_count":0} +{"_type":"issue","id":"polylogue-erf3","title":"claude.ai export zip detects as unknown-export at the container level","description":"polylogue import --explain on claude-ai-data-2026-07-30-16-36-batch-0000.zip reports detector=zip.container, detected_origin=unknown-export, detected_provider=unknown, with artifact_taxonomy.path matched=false - even though every inner entry lowers to session:claude-ai:\u003cuuid\u003e and 1013 sessions parse correctly.\n\nSo the container carries no origin identity while its contents do. Plausibly the same shape as dataset finding C5 (20 'unknown' settled-yet-absent documents).\n\nAC: a claude.ai GDPR export zip is detected as claude-ai-export at the container level, or the reason it cannot be is documented and C5's unknown cohort is re-checked against that answer.","status":"open","priority":2,"issue_type":"task","owner":"ezo.dev@gmail.com","created_at":"2026-07-30T22:05:28Z","created_by":"Sinity","updated_at":"2026-07-30T22:05:28Z","dependency_count":0,"dependent_count":0,"comment_count":0} +{"_type":"issue","id":"polylogue-zng9","title":"parse claude.ai memories.json from GDPR exports","description":"The claude.ai GDPR export ships memories.json (15.7 KB in the 2026-07-30 batch) and polylogue drops it entirely: the only 'memories' parser is codex's memories_1.sqlite (sources/parsers/codex_state.py). No claude-ai handling exists.\n\nEvidence: rg -n 'memories' over polylogue/ shows zero claude-ai hits; import --explain on claude-ai-data-2026-07-30-16-36-batch-0000.zip yields 1013 sessions = 1002 conversations + 11 design_chats, with memories.json contributing nothing.\n\nAC: memories.json content is represented in the archive (assertion, sidecar, or session-scoped construct - decide which), and re-import is idempotent.","status":"open","priority":2,"issue_type":"task","owner":"ezo.dev@gmail.com","created_at":"2026-07-30T22:05:10Z","created_by":"Sinity","updated_at":"2026-07-30T22:05:10Z","dependency_count":0,"dependent_count":0,"comment_count":0} +{"_type":"issue","id":"polylogue-vs5x","title":"Clock guard installs per-test, so module-level clock reads at collection time escape it","description":"## The gap\n\n`tests/infra/clock_guard.py` replaces the old `test-clock-allowlist.yaml` lint\nwith a runtime guard: reaching for a host clock inside a guarded test file\nraises, pointing at `frozen_clock`. That is a genuine upgrade -- an allowlist is\nwhat you build when the capability is still available.\n\nBut the guard installs as an `autouse` **fixture**, so it arms per-test, after\npytest has already imported the test module. A clock read at module level --\na constant, a decorator argument, a `@pytest.mark.parametrize` value -- executes\nduring collection and escapes it entirely.\n\nThe old static AST lint DID catch that case. So on this one axis the runtime\nguard is weaker than what it replaced, and the PR's \"unreachable\" framing\noverstates it: it is \"unreachable from inside a test function\", not\n\"unreachable\".\n\n## Why this is worth closing rather than documenting\n\nA module-level clock read is unusual but it is exactly the shape that produces\nthe flakiness the guard exists to prevent -- a value captured once at import\nand reused across every test in the file, drifting from the frozen clock the\ntests believe they are using.\n\n## Direction\n\n`pytest_configure` runs before collection, so patches installed there cover\nmodule import. The scoping mechanism already exists: `_time_raiser` uses a\ncaller-frame check to distinguish guarded test files from production code, so a\nprocess-wide patch does not have to mean a process-wide failure.\n\nTwo things to work out:\n\n- The per-module `datetime` symbol patch is module-specific (it rebinds\n `datetime` in the test module's own namespace when that module did\n `from datetime import datetime`). A configure-time install cannot know the\n module set yet, so this likely needs a different mechanism -- patching\n `datetime.datetime` itself, guarded by the caller-frame check, rather than\n per-module rebinding.\n- `conftest.py` and `tests/infra` are deliberately exempt, and both are imported\n before ordinary test modules; the exemption must survive the move.\n\n## Acceptance criteria\n\n- A test file with a module-level `datetime.now()` fails with the guard's\n guidance message, not silently.\n- Existing exemptions (`tests/infra`, `conftest.py`,\n `@pytest.mark.uses_real_clock`) still hold.\n- Tests requesting `frozen_clock` still work -- note the guard now narrows\n rather than disables for those (it keeps guarding `time_ns`/`monotonic_ns`,\n which `freeze_clock` does not patch).\n- The word \"unreachable\" is only used where it is true.\n\nRef polylogue-aggz\n","status":"open","priority":2,"issue_type":"bug","owner":"ezo.dev@gmail.com","created_at":"2026-07-30T17:55:44Z","created_by":"Sinity","updated_at":"2026-07-30T17:55:44Z","labels":["area:testing"],"dependency_count":0,"dependent_count":0,"comment_count":0} {"_type":"issue","id":"polylogue-ubwg","title":"Evaluate typed-constructor chokepoints for the remaining hash-boundary-registry sites","description":"polylogue-aggz Invariant 1 (comparison identity contains only content) is now structurally enforced in polylogue/pipeline/ids.py via fixed keyword-only constructors (message_identity_hash/attachment_identity_hash/event_base_identity_hash/event_canonical_identity_hash) instead of dict-key-list projection -- passing a non-content field is a TypeError at the call boundary, not a value a reviewer has to remember to strip. This closed the exact defect pattern behind polylogue-bu1i and polylogue-nuec.\n\ndocs/plans/hash-boundary-registry.yaml was NOT retired in that change and should stay open as tracked debt, not be treated as superseded. It governs all 198 hashlib/core.hashing call sites across polylogue/ (90 content-hash, 91 identifier, 17 other, spanning 58+ files: blob_store.py, security/excision.py, judgment/*, sinex/*, browser_capture/*, ...), the overwhelming majority of which are NOT session/message/attachment/event comparison identity -- they are content-addressed storage keys, HMAC signatures, redaction digests, and other identifier-generation sites with a different (and often already-correct) risk shape. Retiring the whole registry would have been a false claim of coverage this session did not do the work for.\n\nFollow-up: audit whether any of the 91 'identifier'-classified sites share the aggz failure shape (a mutable/acquisition-state field folded into a value used for equality/dedup comparison) and, for those that do, build the same fixed-signature-constructor pattern used in pipeline/ids.py. Sites that are pure content-hashing of raw bytes/already-hashed values (the 'content-hash'/'other' tags) don't need this -- only sites where an identifier is also treated as a stable comparison key are candidates. Only once every hash-boundary site is provably covered by a structural chokepoint (or provably out of the aggz identity-comparison class) can the registry itself be deleted.","status":"open","priority":2,"issue_type":"task","owner":"ezo.dev@gmail.com","created_at":"2026-07-30T17:22:26Z","created_by":"Sinity","updated_at":"2026-07-30T17:22:26Z","dependency_count":0,"dependent_count":0,"comment_count":0} {"_type":"issue","id":"polylogue-2jga","title":"Split or delete test-closure-matrix.yaml / test-quality-coverage.yaml's unenforced narrative fields","description":"Audit (2026-07-30, meta-machinery purge) found two related but distinct\nmanifests each mixing one real enforced check with unenforced free-text\nnarrative:\n\n1. docs/plans/test-closure-matrix.yaml (381 lines): devtools/verify_closure_matrix.py\n only checks that target_files/representative_tests paths exist on disk and\n that gate:absent rows carry a known_gaps bullet — it never runs the\n representative tests or verifies they exercise the target files. Its only\n failure mode is \"a file moved/renamed and the hand-maintained matrix wasn't\n updated\" — the fossilized-diff pattern CLAUDE.md flags for deletion. Counter-\n consideration: it forces explicit known_gaps documentation per declared-\n absent domain, which has narrative value distinct from the path check, and\n git history (d068d6482, 054dfa9e1, dc6fa632a) shows only refactor/consolidation\n commits, never a caught coverage gap that wasn't already known from the\n known_gaps text itself.\n\n2. docs/plans/test-quality-coverage.yaml: check_test_quality_ci_claims verifies\n ci_gate:true dimensions actually appear in a real CI workflow step (a\n genuine, real check — keep this). But most of the file's content\n (flakiness.known_flaky, mock_depth, fuzz tool locations) is pure narrative\n with no executable check beyond generic schema/coverage-gap validation, and\n nothing re-verifies a known_flaky entry is still flaky or that\n value_percent/last_verified stay current.\n\nOperator call needed: (a) for test-closure-matrix.yaml, keep as narrative\ndocumentation with path-existence hygiene, or delete and let the real\nper-domain test suites speak for themselves; (b) for test-quality-coverage.yaml,\nsplit the ci_gate dimension (keep, real check) from the flakiness/fuzz/mock_depth\nnarrative (move to a plain doc outside docs/plans/ verification, or delete).\nNot resolved in the purge session because both are genuinely load-bearing in\npart and the split requires deciding how much narrative value survives without\nthe doc.","status":"open","priority":2,"issue_type":"chore","owner":"ezo.dev@gmail.com","created_at":"2026-07-30T16:55:04Z","created_by":"Sinity","updated_at":"2026-07-30T16:55:04Z","dependency_count":0,"dependent_count":0,"comment_count":0} {"_type":"issue","id":"polylogue-ganm","title":"Reduce topology-target.yaml to a bare file inventory, drop placement-judgment columns","description":"Audit (2026-07-30, meta-machinery purge, feature/chore/purge-meta-machinery) found\ndocs/plans/topology-target.yaml's 4618 lines are almost entirely a per-file\n`target`/`reason`/`owner` placement-judgment projection that no code or doc\nreads to make a placement decision — it is written by\ndevtools/build_topology_projection.py, then only checked against itself by\ndevtools/verify_topology.py's orphan/missing/conflict checks (which need only\na bare path list) plus a narrow kernel_rule check (which needs `target`/`owner`\nonly for the ~15 files that live directly at polylogue/ root).\n\ngit log --oneline --follow on the yaml and on build_topology_projection.py /\nrender_topology_status.py (predecessor render target, already deleted this\nsession) shows only mechanical regenerate-after-adding-a-module commits,\nnever a commit that used the placement judgments to actually relocate code.\n\nReal defect class the SURVIVING checks prevent (keep these): orphan file in\ntree not declared, declared file missing from tree, duplicate declaration,\nnon-kernel file sitting at polylogue/ root. These only need a file inventory\n+ owner tag for root files, not a placement/target/reason judgment per file.\n\nProposed scope: rewrite devtools/build_topology_projection.py and\ndevtools/verify_topology.py so the generated artifact is a flat sorted list\nof declared paths (+ owner/target only for the root-level kernel_rule check),\ndropping target/reason/loc/cross_cut columns for the ~600 non-root files.\nUpdate polylogue/verification/manifests/models.py's TopologyManifest/\nTopologyEntry to match the reduced schema.\n\nNot done in the purge session because it is a generator/schema rewrite, not\na deletion — real engineering risk of breaking `render all --check` /\n`verify topology` if done without careful review, and genuinely needs an\noperator call on whether the placement-judgment metadata has narrative value\nworth keeping despite zero consumption evidence.","status":"open","priority":2,"issue_type":"chore","owner":"ezo.dev@gmail.com","created_at":"2026-07-30T16:54:46Z","created_by":"Sinity","updated_at":"2026-07-30T16:54:46Z","dependency_count":0,"dependent_count":0,"comment_count":0} @@ -481,7 +502,7 @@ {"_type":"issue","id":"polylogue-ei0d","title":"session_provider_usage_events.payload_json is 1.28 GiB of write-only data whose every field is a typed column beside it","design":"Measured on the live archive 2026-07-29 (index.db, 33.69 GiB by dbstat).\n\n session_provider_usage_events (table) 2443.9 MB 7.1% of the index tier\n of which payload_json 1.28 GiB (52% of the table)\n 4,030,168 rows, avg 340 B, zero NULLs\n\nWRITE-ONLY\n`payload_json` is written by _PROVIDER_USAGE_EVENT_INSERT_SQL\n(storage/sqlite/archive_tiers/write.py:3041-3047) and never read back. An AST-ish scan\nfor `SELECT ... FROM session_provider_usage_events` mentioning payload_json returns zero\nhits; the only reads of this table anywhere select COUNT(*)\n(archive_tiers/self_verify.py:28), `position` (ingest_precedence.py:146,196 and\nwrite.py:2904), or `SELECT 1` (archive.py:4871).\n\n(`insights/claude_workflow_materializer.py:458` does read a `payload_json`, but from\n`session_events` -- a different table. Do not confuse the two.)\n\nREDUNDANT BY CONSTRUCTION\nEvery field in the blob is already an extracted, typed column in the same row:\n {\"last_token_usage\":{\"cache_write_tokens\":451,\"cached_input_tokens\":54332,\n \"input_tokens\":7,\"output_tokens\":3},\n \"model\":\"claude-opus-4-20250514\",\"semantics\":\"per_message\",\"type\":\"message_usage\"}\nmaps to last_cache_write_tokens / last_cached_input_tokens / last_input_tokens /\nlast_output_tokens / model_name / provider_event_type. The table has 20 columns and the\nblob adds no field they do not already carry.\n\nIt is also doubly redundant by tier: index.db is REBUILDABLE, and the authoritative raw\npayload already lives in source.db's blob store. Keeping a copy of provider wire bytes in\nthe derived tier stores the same evidence a third time.\n\nALSO WORTH A LOOK WHILE HERE\n`total_cache_write_tokens` is constant across a 400k-row sample (1 distinct value), and\n`provider_event_type` / `model_context_window` have 2 each. A constant column over 4M rows\nis its own small waste; confirm against the full table before acting, since the sample was\nthe first 400k rows and may not be representative.\n\nDO\nDrop payload_json from the table (index tier, so this is a derived-schema change: classify\nper CLAUDE.md's \"Schema regimes\" and declare the delta class in\nstorage/sqlite/lifecycle.py -- an undeclared bump silently forces a full raw replay). If\nsome future consumer genuinely needs the provider's original wire shape, it should read it\nfrom source.db's blob, not from a duplicate in a rebuildable tier.\n\nExpected reclaim: ~1.28 GiB of index.db, plus a smaller write-path saving on every usage\nevent ingested. Batch this with any other index-tier change so it costs one rebuild, not\ntwo -- a full rebuild currently replays 92 GiB.\n","status":"open","priority":2,"issue_type":"bug","owner":"ezo.dev@gmail.com","created_at":"2026-07-29T05:50:50Z","created_by":"Sinity","updated_at":"2026-07-29T05:50:50Z","dependency_count":0,"dependent_count":0,"comment_count":0} {"_type":"issue","id":"polylogue-lrdh","title":"master: 3 browser-capture coalescing tests fail on title precedence (GDPR export beats browser capture)","design":"Reproduced on pure origin/master (not introduced by any in-flight branch), verified by\nchecking out origin/master's polylogue/ and tests/unit/sources/test_browser_capture.py into\na clean tree and running the selection:\n\n devtools test tests/unit/sources/test_browser_capture.py -k coalesces 3 failed\n\n test_browser_capture_raw_payload_coalesces_with_claude_ai_export\n test_browser_capture.py:1008 assert 'Claude GDPR title' == 'Claude browser title'\n test_browser_capture_raw_payload_coalesces_with_chatgpt_export[browser-first]\n test_browser_capture_raw_payload_coalesces_with_chatgpt_export[export-first]\n test_browser_capture.py:922 assert 'GDPR title' == 'Browser title'\n\nThe tests assert a browser-capture title outranks a GDPR/export title when the two\ncoalesce into one session; the export title is winning instead. Both parametrizations\nfail, so it is not acquisition-order dependent.\n\nEither the precedence rule changed and these tests were not updated, or a real regression\nin coalescing title selection landed without being caught -- per-PR CI skips the heavy\ntest suite (it runs post-merge on master), which is the mechanism that lets this sit\nbroken on master.\n\nDetermine which before editing: if the intended rule is now export-wins, the tests encode\na stale contract and should be rewritten to state the new one with its reason; if\nbrowser-wins is still intended, this is a live bug in the coalescing path and the tests\nare correct.\n\nFound while merging two agent branches (pbuh sidecar evidence, ah21 browser-capture blocks);\nneither touches title coalescing and both reproduce the failure identically, as does a\nclean origin/master checkout.\n","status":"closed","priority":2,"issue_type":"bug","assignee":"Sinity","owner":"ezo.dev@gmail.com","created_at":"2026-07-29T05:43:23Z","created_by":"Sinity","updated_at":"2026-07-29T06:51:48Z","started_at":"2026-07-29T06:51:28Z","closed_at":"2026-07-29T06:51:48Z","close_reason":"Determined: precedence rule legitimately changed, tests were stale.\n\n#3179 (commit b473d9256, Ref polylogue-z1c6, merged 2026-07-20) intentionally\nadded a mirror rule to browser_capture_precedence() in\npolylogue/storage/sqlite/archive_tiers/ingest_precedence.py: a genuine\nnon-browser-capture arrival (direct/GDPR export) now always outranks\nbrowser-capture-only content and vice versa is skipped, making the outcome\norder-independent (fixing a real order-dependent flakiness bug where whichever\nmaterial a live daemon happened to process first would win). That PR added\nand updated the sibling proof\ntest_archive_tiers_archive_facade_export_vs_native_precedence_is_order_independent\n(tests/unit/storage/test_archive_tiers_archive.py:737, asserting\n(\"Direct export\", export_message_count) regardless of arrival order) but\nmissed updating the three browser_capture.py coalescing tests that predate\nit (last touched by ddf4f3efc, well before #3179).\n\nFixed: rewrote the three stale title assertions in\ntests/unit/sources/test_browser_capture.py (now lines 930 and 1019) from\n\"Browser title\"/\"Claude browser title\" to \"GDPR title\"/\"Claude GDPR title\",\nwith comments citing browser_capture_precedence(), #3179/polylogue-z1c6, and\nthe sibling order-independence test. No production code changed -- this was\nnever a live regression.\n\nLive-archive impact (read-only check against /realm/db/polylogue, confirmed\nPOLYLOGUE_ARCHIVE_ROOT resolves there): 0 sessions in raw_sessions currently\nhave more than one distinct capture_mode for the same (origin, native_id), so\nno live session's stored title is affected by this either way -- production\nhas been export-wins all along.\n\nGate recommendation: per-PR CI skipping the heavy test suite is the\ndocumented mechanism (CLAUDE.md) that let a legitimate #3179 rule change\nmerge without updating every affected test; this is already a known,\naccepted tradeoff (heavy suite runs post-merge). Not recommending a new\nfossilized-diff-style check -- CLAUDE.md forbids gates that memorialize a\nrenamed spelling, and the actual missing net here is \"did #3179 run the full\ntest_browser_capture.py file\", which devtools test \u003cchanged files\u003e would\nhave caught if run; no new lint needed.\n\nVerification: devtools test tests/unit/sources/test_browser_capture.py -k coalesces\n(3 passed), full file green, tests/unit/storage/test_archive_tiers_archive.py -k\nprecedence (13 passed), devtools verify --quick (exit_code 0). Committed as\n1f0040353 on branch worktree-agent-acdc97dbfb9cf3928 (agent worktree; PR not\nopened/merged per task scope -- report only).","dependency_count":0,"dependent_count":0,"comment_count":0} {"_type":"issue","id":"polylogue-7to5","title":"Capture and export convergence on one session_id is untested and would silently downgrade fidelity","description":"Measured 2026-07-29 -- this is a LATENT hazard, not an active bug, which is why it needs recording before it fires.\n\n chatgpt conversations reachable by browser capture only: 43\n reachable by GDPR export only: 2,423\n reachable by BOTH: 0\n sessions == distinct native_ids == 2,635 (no duplication today)\n\nThe two paths have never overlapped, so coalescing has never been exercised.\nWhat would happen is determined by two facts already established:\n\n 1. IDENTITY WOULD COLLIDE, NOT DUPLICATE. Both paths key the ChatGPT\n conversation id as native_id, and session_id is a generated column\n (origin || ':' || native_id). Same conversation, same session_id.\n 2. THE PARSED WRITE PATH IS FULL-REPLACE. write.py deletes blocks and messages\n for the session id, then inserts. Whichever path ingests SECOND wins\n entirely.\n\nAnd the two paths carry materially different fidelity: the export has the\nmapping tree with tool nodes and status; the capture has flat text with no\nblocks channel at all (see the BrowserCaptureTurn bead). So exporting a\nconversation you had already captured is fine, and CAPTURING one you had\nalready exported silently replaces structured evidence with flattened text.\n\nThe raw tier already models this correctly -- raw_revision_heads, revision\nauthority, accepted frontiers. It is the parsed tier that resolves by\nreplacement instead of by fidelity.","acceptance_criteria":"1. Two observations of one conversation are retained as revisions, and the composed session reflects the higher-fidelity one regardless of arrival order. 2. A test ingests export-then-capture and capture-then-export for the same conversation and asserts the same, higher-fidelity result both ways. 3. Fidelity is declared per acquisition path in the OriginSpec so 'higher' is not a judgement call at write time. 4. Related: unknown-export currently holds 52 raws with NULL native_id -- conversations that failed origin detection and therefore cannot coalesce with anything.","status":"open","priority":2,"issue_type":"task","owner":"ezo.dev@gmail.com","created_at":"2026-07-29T04:52:48Z","created_by":"Sinity","updated_at":"2026-07-29T04:52:48Z","labels":["area:ingest","lane:capture-reliability"],"dependency_count":0,"dependent_count":0,"comment_count":0} -{"_type":"issue","id":"polylogue-o4j2","title":"aistudio-drive discards every model setting that produced its outputs","description":"The per-origin wire enumeration found the entire runSettings block unread for aistudio-drive:\n\n temperature, topP, topK, maxOutputTokens, thinkingLevel, safetySettings\n (with threshold), enableCodeExecution, enableSearchAsATool,\n enableBrowseAsATool, enableAutoFunctionResponse\n plus chunkedPrompt.pendingInputs\n\nThis is the model configuration for every AI Studio session in the archive.\nPolylogue's stated purpose includes reconstructing what produced a result; for\nthis origin the generation parameters are present in the acquired bytes and\ndropped at parse.\n\nIt is also the only origin where the operator can vary sampling settings freely,\nwhich makes it the one place where 'same prompt, different settings, different\noutput' is answerable -- if the settings were kept.","acceptance_criteria":"1. runSettings is parsed into typed session-level evidence for aistudio-drive. 2. The settings are queryable, so 'sessions where temperature \u003e X' is expressible. 3. Existing sessions acquire it by reprocess of retained bytes. 4. Other origins are checked for an equivalent settings block rather than assuming AI Studio is unique.","status":"open","priority":2,"issue_type":"task","owner":"ezo.dev@gmail.com","created_at":"2026-07-29T04:52:47Z","created_by":"Sinity","updated_at":"2026-07-29T04:52:47Z","labels":["area:ingest","lane:origin-interop-export"],"dependency_count":0,"dependent_count":0,"comment_count":0} +{"_type":"issue","id":"polylogue-o4j2","title":"aistudio-drive discards every model setting that produced its outputs","description":"The per-origin wire enumeration found the entire runSettings block unread for aistudio-drive:\n\n temperature, topP, topK, maxOutputTokens, thinkingLevel, safetySettings\n (with threshold), enableCodeExecution, enableSearchAsATool,\n enableBrowseAsATool, enableAutoFunctionResponse\n plus chunkedPrompt.pendingInputs\n\nThis is the model configuration for every AI Studio session in the archive.\nPolylogue's stated purpose includes reconstructing what produced a result; for\nthis origin the generation parameters are present in the acquired bytes and\ndropped at parse.\n\nIt is also the only origin where the operator can vary sampling settings freely,\nwhich makes it the one place where 'same prompt, different settings, different\noutput' is answerable -- if the settings were kept.","acceptance_criteria":"1. runSettings is parsed into typed session-level evidence for aistudio-drive. 2. The settings are queryable, so 'sessions where temperature \u003e X' is expressible. 3. Existing sessions acquire it by reprocess of retained bytes. 4. Other origins are checked for an equivalent settings block rather than assuming AI Studio is unique.","status":"closed","priority":2,"issue_type":"task","assignee":"Sinity","owner":"ezo.dev@gmail.com","created_at":"2026-07-29T04:52:47Z","created_by":"Sinity","updated_at":"2026-07-31T04:02:59Z","started_at":"2026-07-31T04:01:05Z","closed_at":"2026-07-31T04:02:59Z","close_reason":"runSettings storage was already shipped (PR #3390, polylogue-2qx.4/cgfy, index v46) before this bead was filed. The genuinely-remaining gap -- chunkedPrompt.pendingInputs (draft/unsent textbox content, 7/397 real sessions with non-blank drafts) -- is fixed on PR #3415 (draft_input session_event). AC2 (query-DSL numeric predicates over run_settings, e.g. temperature \u003e X) is NOT satisfied: the boolean-query grammar only accepts integer literals and NumericQueryFieldInfo assumes a plain SQL column, not a JSON-extract expression -- needs a separate DSL float-literal + JSON-field-predicate feature, out of parser scope. AC4 checked: grepped sources/parsers + sources/providers for generationConfig/sampling_params/temperature/inference_config/model_settings, no other origin has an equivalent settings block.","labels":["area:ingest","lane:origin-interop-export"],"dependency_count":0,"dependent_count":0,"comment_count":0} {"_type":"issue","id":"polylogue-4p1.3","title":"Insights: the concept earns its place, five of eleven types do not","description":"DEEP REVIEW of polylogue/insights (23,103 lines, 11 registered types) 2026-07-29.\n\nWHAT AN INSIGHT IS, AND WHY IT IS NOT JUST A NAMED QUERY. tool_usage pairs an\naggregate with per-origin COVERAGE: 'an origin with sessions but zero actions is\nthe explicit data-unavailable signal, not a quiet zero.' That distinction --\nzero versus unavailable -- is real, is not expressible in the query DSL's\n, and is the honest core of the concept. The insights package\nshould NOT be dissolved into the query algebra wholesale.\n\nWHICH TYPES EARN THEIR EXISTENCE (materialized tables in parentheses):\n KEEP threads (9,914) a root session's lineage tree; single-session\n threads are correct, not degenerate -- 810\n multi-session threads correspond exactly to\n the 810 sessions with lineage children\n KEEP tool_usage computed; the coverage pairing is the value\n (but its CLI surface is currently BROKEN --\n see the analyze-tools bead)\n KEEP session_costs, cost_rollups, usage_timeline, archive_coverage,\n archive_debt computed rollups with coverage semantics\n DELETE session_phases (29,432) see the deletion bead: 82% single span, no\n label, index-synthesized timestamps\n DELETE session_work_events (21,190) 82% single event, duplicates action_pairs\n REDUCE session_profiles (18,871) keep the profile, delete the five\n constant version/family columns and fix the\n 100%-NULL cost columns (see f2qv.6)\n REDUCE session_tag_rollups (3,593) explicit_count constant 0\n\nSTRUCTURAL FINDING: 5 of 11 types are materialized tables, 6 are computed. There\nis no stated rule for which. The materialized ones are precisely where the\nfreshness machinery lives (insight_materialization's seven proxy columns,\nderived_refresh_guard, delegation_refresh_scope). Under content-addressed\nderivation the distinction stops mattering -- a materialized insight becomes a\nhash-keyed cache, and a stale row is a miss rather than a lie.\n\nSURFACE FINDING: every type is MCP-reachable through\nmcp/insight_tool_contracts.py and only some are CLI-reachable; see the\nregistry-surface bead.","acceptance_criteria":"1. Each of the 11 types carries a recorded verdict: keep / reduce / delete, with the discrimination evidence. 2. A stated rule governs materialized versus computed, or the distinction is removed by hash-keying. 3. The coverage-pairing property is documented as the reason the package exists, so a future refactor does not dissolve it into the query DSL by accident. 4. Deleting a type removes its table, materializer, registry entry, MCP contract and any FTS index together.","status":"open","priority":2,"issue_type":"task","owner":"ezo.dev@gmail.com","created_at":"2026-07-29T04:52:40Z","created_by":"Sinity","updated_at":"2026-07-29T04:52:40Z","labels":["area:analytics","area:query","area:surface","decision","delivery:C-read-evidence-contract","horizon:frontier","lane:read-contracts"],"dependencies":[{"issue_id":"polylogue-4p1.3","depends_on_id":"polylogue-4p1","type":"parent-child","created_at":"2026-07-29T06:52:40Z","created_by":"Sinity","metadata":"{}"}],"dependency_count":0,"dependent_count":0,"comment_count":0} {"_type":"issue","id":"polylogue-4p1.2","title":"Registered insights are MCP-reachable and CLI-absent: decide the surface, do not let it drift","description":"Measured 2026-07-29. insights/registry.py registers insight types including session_phases (cli_command_name='phases'), session_work_events, threads, session_profiles, session_tag_rollups, session_costs, cost_rollups, usage_timeline, tool_usage, archive_coverage, archive_debt.\n\n $ polylogue analyze --help -\u003e insights, latency, pace, tools, turns, usage\n\nSo 'polylogue analyze phases' does not exist despite the registry declaring that\nname. MCP reaches all of them through mcp/insight_tool_contracts.py, which is\nregistry-driven. The registry is therefore authoritative for MCP and decorative\nfor the CLI -- a declare-once mechanism honoured by one surface and not the\nother, which is the pattern polylogue-t46 exists to remove.\n\nOPERATOR POSITION (2026-07-29): a registered insight should be CLI-reachable\nunless there is a clear reason not to -- but the CLI itself must stay\ndisciplined rather than sprawling one subcommand per registry entry. Those pull\nin opposite directions and the resolution is a decision, not a default.\n\nNote this interacts with two deletions: session_phases and session_work_events\nare condemned by a sibling bead, so their registry entries and MCP contracts go\nwith them rather than gaining CLI commands.","acceptance_criteria":"1. Every registry entry is classified: CLI-reachable, MCP-only with a stated reason, or deleted. 2. No registry entry declares a cli_command_name that produces no command. 3. Whatever the decision, one mechanism generates both surfaces -- a registry honoured by MCP and ignored by the CLI does not survive. 4. The CLI does not gain a subcommand per entry by default; the disciplined shape is argued explicitly.","status":"open","priority":2,"issue_type":"task","owner":"ezo.dev@gmail.com","created_at":"2026-07-29T04:52:38Z","created_by":"Sinity","updated_at":"2026-07-29T04:52:38Z","labels":["area:query","area:surface","decision","delivery:C-read-evidence-contract","horizon:frontier","lane:read-contracts"],"dependencies":[{"issue_id":"polylogue-4p1.2","depends_on_id":"polylogue-4p1","type":"parent-child","created_at":"2026-07-29T06:52:38Z","created_by":"Sinity","metadata":"{}"}],"dependency_count":0,"dependent_count":0,"comment_count":0} {"_type":"issue","id":"polylogue-cuxz.11","title":"session_agent_policies: 402,869 rows encoding 3,053 facts","description":"Measured 2026-07-29, full scan.\n\n rows 402,869\n distinct sessions 3,031 (133 rows per session)\n sessions whose policy NEVER changes 3,010 (99.3%)\n rows remaining if deduped by value 3,053\n\nThe table records approval_policy / sandbox_policy / network_policy at every\nmessage position, and the policy is invariant within a session 99.3% of the\ntime. It is a change-log of non-changes at 132x redundancy.\n\nAlready known degenerate on the same table: network_policy is constant 'false',\nsource_message_id is 100% NULL.\n\n sqlite3 -readonly index.db \"with per as (select session_id,\n count(distinct coalesce(approval_policy,'')||'|'||coalesce(sandbox_policy,'')||'|'||coalesce(network_policy,'')) d,\n count(*) n from session_agent_policies group by session_id)\n select count(*), sum(n), sum(d=1), sum(d) from per;\"","acceptance_criteria":"1. Policy is stored once per session (or per genuine change), not per message position. 2. If policy genuinely varies for some sessions, those keep interval rows; the 3,010 invariant sessions do not. 3. network_policy and source_message_id are dropped unless a producer is named. 4. Report row count before and after against the 402,869 baseline.","status":"open","priority":2,"issue_type":"task","owner":"ezo.dev@gmail.com","created_at":"2026-07-29T04:52:36Z","created_by":"Sinity","updated_at":"2026-07-29T04:52:36Z","labels":["area:storage","area:substrate","horizon:frontier"],"dependencies":[{"issue_id":"polylogue-cuxz.11","depends_on_id":"polylogue-cuxz","type":"parent-child","created_at":"2026-07-29T06:52:36Z","created_by":"Sinity","metadata":"{}"}],"dependency_count":0,"dependent_count":0,"comment_count":0} @@ -815,6 +836,8 @@ {"_type":"issue","id":"polylogue-rii.1","title":"Agent work-event write-leg -\u003e session_events -\u003e materialized read-models","description":"record_work_event/emit_decision write surface routed through the existing idempotent ingest seam (no parallel writer); flows into the run-projection read models. Today agents can only record_correction/blackboard_post/tag — there is no 'I ran this tool / spawned this subagent / decided X' write. GH issue thread (body + comments) is input, not authority; this bead's scope statement wins where they conflict.","design":"Route through the existing idempotent ingest seam (write_raw_and_parsed / the daemon ingest path) — no parallel writer (gh#2459 body is code-grounded here). Surface: MCP tools record_work_event/emit_decision (mutation role) accepting typed events (tool run, subagent spawn, decision, artifact change) with evidence/session refs; land in session_events; run-projection read models pick them up through the normal materializer. MCP registration trap: EXPECTED_TOOL_NAMES + TOOL_CONTRACT + role gating + render openapi/cli-output-schemas regen (see bd memories). Acceptance: an agent posts a work event mid-session; it is queryable via observed-events within one convergence cycle; re-posting is idempotent.","acceptance_criteria":"- MCP tools record_work_event / emit_decision are registered with the mutation role: EXPECTED_TOOL_NAMES + TOOL_CONTRACT updated, role gating enforced, and `devtools render openapi \u0026\u0026 devtools render cli-output-schemas` regenerated with `devtools render all --check` clean.\n- Typed events (tool run, subagent spawn, decision, artifact change) with evidence/session refs route through the existing idempotent ingest seam (write_raw_and_parsed / the daemon ingest path) into session_events — no parallel writer (grep confirms reuse).\n- Behavior test: an agent posts a work event mid-session and it is queryable via observed-events (session_work_events / DSL) within one convergence cycle; re-posting the same event is idempotent (no duplicate row). `devtools test \u003cmcp work-event test\u003e` green.","notes":"[Delivery upgrade 2026-07-07T00:05:00Z] Release=D-agent-context-coordination; lane=agent-coordination; readiness=A-implementation-ready; proof=two-agent separate-worktree proof with before/after coordination envelopes. Original readiness=A-implementation-ready.\n[Prework packet 2026-07-07] Static execution packet (anchors, mechanism, plan, tests, verification): .agent/handoffs/polylogue-gpt-pro-2026-07-07/prework-v2/task_packets/071_polylogue_rii_1.md (depth: bead-localized-from-export; urgency: T2-foundation-before-feature-proof). Generated from master @ 8a975a40 2026-07-06 — verify source anchors before coding; line numbers are snapshot-relative.\nRECONCILED 2026-07-13 with 37t.2 inline protocol: the agent work-event write-leg and the marker channel are ONE channel with two encodings (structured MCP writes; prose markers extracted at enrichment). Unify vocabularies — work-event kinds and marker kinds must share the registry (a ::phase marker IS a work event). Do not build parallel event taxonomies.","status":"open","priority":2,"issue_type":"feature","owner":"ezo.dev@gmail.com","created_at":"2026-07-03T04:31:43Z","created_by":"Sinity","updated_at":"2026-07-13T04:00:08Z","external_ref":"gh-2459","labels":["area:substrate","delivery:D-agent-context-coordination","horizon:frontier","lane:agent-coordination"],"dependencies":[{"issue_id":"polylogue-rii.1","depends_on_id":"polylogue-rii","type":"parent-child","created_at":"2026-07-03T06:31:43Z","created_by":"Sinity","metadata":"{}"}],"dependency_count":0,"dependent_count":1,"comment_count":0} {"_type":"issue","id":"polylogue-fs1.3","title":"Per-source coverage/fidelity declaration for Hermes imports","description":"Every Hermes acquisition tier and schema version needs a machine-readable fidelity declaration that distinguishes what is exact, absent, redacted, degraded, or inferred. The declaration is the guard against a parser test going green while silently dropping forensic history or cost/addressing provenance.","design":"Extend the OriginSpec/fidelity surface with: producer/schema version; installation/profile namespace; acquisition method (sqlite_backup, stable export, JSON fallback, runtime spans); exact retained-blob-to-normalized reproducibility verdict; counts and coverage for active, rewound, compacted, and observed messages; addressing/material-origin semantics; actual/estimated cost with status/source/pricing/billing provenance; lifecycle/relationship coverage; runtime-span coverage and explicit missingness. The snapshot and span lanes may enrich one logical session revision only with per-field provenance; they may not double-count or silently prefer a lower-fidelity tier.","acceptance_criteria":"explain-import on Hermes v16, a later schema, JSON fallback, and a spans-plus-snapshot merge names every capability as exact, absent, redacted, degraded, or inferred; exact-blob reproducibility is stated and verified; the same logical session from two tiers remains one revision with field-level provenance; message-state/addressing and cost-provenance counts reconcile to fixtures; deliberately dropping observed mapping, cost provenance, snapshot proof, or an unpaired span changes the declared fidelity and surfaces a downstream forensics caveat. OriginSpec fixtures and mutation-style negative tests pass.","notes":"[Delivery upgrade 2026-07-07T00:05:00Z] Release=K-interop-origin-export; lane=origin-interop-export; readiness=D-horizon-ready; proof=OriginSpec detector/parser/fixture/fidelity suite and content-hash export/import roundtrip. Original readiness=E-spec-needed.\n2026-07-12 fanout lane finding: blocked as scoped — explain-import cannot inspect SQLite Hermes state DBs and its payload lacks a fidelity-declaration field; both surfaces (import_explain.py + payload schema) must be in scope to implement. Evidence: 37bdfa04c; import_explain.py decodes JSON/JSONL only.","status":"closed","priority":2,"issue_type":"feature","owner":"ezo.dev@gmail.com","created_at":"2026-07-03T04:31:40Z","created_by":"Sinity","updated_at":"2026-07-12T23:15:18Z","closed_at":"2026-07-12T23:15:18Z","close_reason":"PR #2789 merged: Hermes per-source coverage/fidelity declaration shipped (import_explain.py, hermes_state.py, generated CLI-output schema regenerated)","labels":["area:ingest","area:substrate","delivery:K-interop-origin-export","delivery:ac-patched","horizon:frontier","lane:origin-interop-export"],"dependencies":[{"issue_id":"polylogue-fs1.3","depends_on_id":"polylogue-fs1","type":"parent-child","created_at":"2026-07-03T06:31:40Z","created_by":"Sinity","metadata":"{}"}],"dependency_count":0,"dependent_count":3,"comment_count":0} {"_type":"issue","id":"polylogue-tf2.2","title":"Fold agent_forensics.py into polylogue analyze","description":"~70% already materialized (cost_rollups, archive_coverage, total_credit_cost, portfolio, cost_outlook). Real gaps: reasoning-token lane on SessionProfile; usage_timeline archive insight (tokens/cost per month per model) registered in insights/registry.py; optional markdown forensics renderer. Drop the script's hand-rolled _CREDIT_RATES; delete the script. Sequenced AFTER the campaign regen (the campaign uses the script one last time). GH issue thread (body + comments) is input, not authority; this bead's scope statement wins where they conflict.","status":"closed","priority":2,"issue_type":"feature","assignee":"Sinity","owner":"ezo.dev@gmail.com","created_at":"2026-07-03T04:31:34Z","created_by":"Sinity","updated_at":"2026-07-03T11:54:39Z","started_at":"2026-07-03T11:31:18Z","closed_at":"2026-07-03T11:54:39Z","close_reason":"Completed: usage forensics is no longer a standalone script surface. Added registered usage_timeline archive insight with CLI/API/MCP registry coverage, reused the shared subscription-pricing catalog for credit estimates, deleted scripts/agent_forensics.py and its private-helper tests, and rewrote README/docs around polylogue analyze insights coverage/cost-rollups/usage-timeline plus devtools workspace claim-vs-evidence. Verification: focused claim-vs-evidence/insights tests passed, render all --check passed, devtools verify --quick passed, and live active-archive usage-timeline smoke returned valid JSON. Follow-up polylogue-5nn tracks the observed 18s whole-archive aggregation latency for unfiltered month-origin-model usage-timeline.","external_ref":"gh-2480","labels":["area:usage","campaign"],"dependencies":[{"issue_id":"polylogue-tf2.2","depends_on_id":"polylogue-tf2","type":"parent-child","created_at":"2026-07-03T06:31:34Z","created_by":"Sinity","metadata":"{}"},{"issue_id":"polylogue-tf2.2","depends_on_id":"polylogue-tf2.1","type":"blocks","created_at":"2026-07-03T06:31:34Z","created_by":"Sinity","metadata":"{}"}],"dependency_count":1,"dependent_count":0,"comment_count":0} +{"_type":"issue","id":"polylogue-mgf6","title":"Query DSL: float-literal numeric predicates for JSON-extracted fields (run_settings temperature/topP)","description":"Follow-up from polylogue-o4j2 (AC2, deferred).\n\naistudio-drive's runSettings (temperature/topP/topK/maxOutputTokens/\nthinkingLevel/safetySettings/enable* flags) is parsed and stored verbatim as\nsessions.run_settings_json (polylogue-2qx.4/cgfy, index v46). It is not\nexposed to the query DSL, so \"sessions where temperature \u003e 0.5\" is not\nexpressible. Two independent gaps block it:\n\n1. Grammar: the boolean-query numeric-comparison rule only accepts integer\n literals (`COUNT_FIELD COMP_OP INT` in archive/query/expression.py) --\n temperature/topP are floats (0.0-2.0 / 0.0-1.0 range).\n2. SQL builder: NUMERIC_QUERY_FIELD_REGISTRY's NumericQueryFieldInfo.unit_columns\n values are treated as plain column names (`f\"{table_alias}.{column}\"` in\n storage/sqlite/archive_tiers/archive.py, two call sites) -- there is no\n path for a computed/JSON-extract expression like\n `json_extract(run_settings_json, '$.temperature')`.\n\nScope: extend the grammar to accept decimal literals for numeric predicates\n(without breaking existing integer-only fields), and extend the SQL-builder\ncall sites (and NumericQueryFieldInfo, if needed) to support an expression\ncolumn alongside plain columns. Consider starting with the integer-typed\nrun_settings fields (topK, maxOutputTokens) which fit the existing INT-only\ngrammar and only need the SQL-builder JSON-extract half, then float support\n(temperature, topP) as a second phase needing the grammar change too.\n\nNot urgent: run_settings is durably stored and readable via `read --view`\nalready; this is about ergonomic filtering, not data loss.","status":"open","priority":3,"issue_type":"task","owner":"ezo.dev@gmail.com","created_at":"2026-07-31T04:03:16Z","created_by":"Sinity","updated_at":"2026-07-31T04:03:16Z","dependency_count":0,"dependent_count":0,"comment_count":0} +{"_type":"issue","id":"polylogue-je9t","title":"7 of 95 design-chat messages dropped by _parse_design_chat","description":"Measured _parse_design_chat directly against all 11 design_chats/*.json in claude-ai-data-2026-07-30-16-36-batch-0000.zip: 95 source messages -\u003e 88 parsed. Loss is concentrated in two files (9-\u003e6 and 20-\u003e16); the other nine are lossless.\n\nNot yet diagnosed - candidates are role values the mapper does not recognise, or content shapes _design_content_payload returns {} for.\n\nAC: either all 95 parse, or the dropped shapes are identified and dropping them is shown to be correct (with the reason recorded here).","status":"open","priority":3,"issue_type":"task","owner":"ezo.dev@gmail.com","created_at":"2026-07-30T22:05:41Z","created_by":"Sinity","updated_at":"2026-07-30T22:05:41Z","dependency_count":0,"dependent_count":0,"comment_count":0} {"_type":"issue","id":"polylogue-inoh","title":"Ambiguous-cohort census residue: claude-code-session/hermes-session/grok-export/unknown-export not root-caused","description":"## Context\n\nFollow-up from the cross-origin ambiguous-cohort census (polylogue-c429,\npolylogue-hith, polylogue-nuec; all descend from polylogue-bu1i). That\ninvestigation focused on the three largest equal-message-count ambiguous\npopulations (claude-ai-export 566, chatgpt-export 129, aistudio-drive 151 --\nthe last already proven by polylogue-bu1i). The remaining origins are small\nand were sampled but not root-caused to the same depth; this bead tracks\nthat residue so it doesn't disappear as anonymous debt.\n\n## claude-code-session (6 of 191 equal-message-count ambiguous cohorts)\n\nExplicitly flagged low-priority/different-shape by the parent investigation.\nSampled all 6 with production `parse_payload`/`parse_stream_payload`\n(claude-code-session raws are stream-record JSONL, routed via\n`is_stream_record_provider` + `parse_stream_payload`, unlike the other three\norigins' single-document JSON). 5 of 6 collapsed to a SINGLE distinct\nblob_hash content group when grouped by raw blob bytes -- i.e. under a fresh\nparse, the members recorded 'ambiguous' in `raw_session_memberships` are\nbyte-identical to each other, which should not classify ambiguous at all\nunder current `classify_membership_revisions` logic (a single by-content\ngroup never reaches the dominance-failure branch). This suggests either (a)\nthese decisions are stale relative to the current member set (see\npolylogue-9dxn's general \"persisted ambiguous verdicts never get\nre-derived\" finding -- may be the same root mechanism, not independently\nconfirmed here), or (b) a raw sibling with genuinely different content was\nremoved (GC/retention) since the decision was recorded, or (c) a\nmethodology gap in the reproduction script not caught in this pass. Not\ndisambiguated -- would need dedicated investigation with access to\n`raw_revision_heads`/retention history for these specific\n`logical_source_key`s, which the parent investigation's read-only harness\ndid not attempt.\n\n## hermes-session (3 of 4 equal-message-count ambiguous cohorts)\n\nSampled all 3. 2 have identical message id set/order/attachment keys with\nsingle-message (`n_messages=1`) conversations -- the actual delta wasn't\nisolated (didn't check `session_events`/text content at the level of detail\nused for claude-ai-export/chatgpt-export given the tiny population). 1\nraised a parse-routing error in the census harness (hermes has a\nSQLite-backed raw path -- `looks_like_sqlite_bytes` /\n`hermes_state.parse_state_db` / `hermes_verification.parse_verification_evidence_db`\nin `polylogue/sources/revision_backfill.py:_parse_one` -- that the harness's\ngeneric `parse_payload` call doesn't handle; this is a harness gap, not\nevidence of a real defect).\n\n## grok-export (1 of 1 -- full population)\n\nThe one ambiguous grok-export cohort (`grok:dom:815e0a1c`) is a\nbrowser-capture DOM snapshot with genuinely DIFFERENT message id sets at\nequal count across its two distinct-content revisions -- this looks like a\nreal content divergence (re-captured page state), not a misclassification\nartifact. Tentatively bucket as GENUINELY AMBIGUOUS, not investigated\nfurther given n=1. Note as an aside: one of its four raw rows'\n`source_path` points at\n`/realm/project/polylogue/.cache/dev-loop/feature-docs-accuracy-revamp-*`,\ni.e. a development/test-fixture path, not a personal capture location --\nworth a separate look at whether stale dev-loop fixtures leaked into the\nlive archive, but out of scope here.\n\n## unknown-export (2 of 3 equal-message-count ambiguous cohorts)\n\nSampled 2 of 2. Both collapsed to a single distinct blob_hash content group,\nsame shape as the claude-code-session finding above. Given `unknown-export`\nis itself a fallback/unclassified bucket, not investigated further.\n\n## Acceptance criteria\n\n- Either resolve each sub-population's cause with the same rigor as\n polylogue-c429/hith/nuec (parse both distinct-content sides, run the\n production classifier, characterize the minimal delta), or explicitly\n downgrade/close this bead with the reason each population is too small to\n be worth the investigation cost, stated per-origin.\n- If the claude-code-session/unknown-export \"single distinct content group\"\n pattern is confirmed to be the polylogue-9dxn stale-verdict mechanism\n rather than a new defect, cross-link and close this portion as\n subsumed by 9dxn's fix rather than re-deriving a new root cause.\n\nRef polylogue-bu1i\nRef polylogue-c429\nRef polylogue-hith\nRef polylogue-nuec\nRef polylogue-9dxn\n","status":"open","priority":3,"issue_type":"task","owner":"ezo.dev@gmail.com","created_at":"2026-07-30T12:18:20Z","created_by":"Sinity","updated_at":"2026-07-30T12:18:33Z","labels":["area:ingest"],"dependency_count":0,"dependent_count":0,"comment_count":0} {"_type":"issue","id":"polylogue-uyci","title":"Expose sessions.display_name/run_settings_json and session_links.parent_tool_use_block_id on a public surface","description":"Feature-gap sweep finding (2026-07-29, feature/chore/promote-schemas-and-wire-gates\n@ bdeb6d1d2). Three columns are written and readable in SQL but never reach any\ndomain model, so no surface (CLI/MCP/API) can answer for them at all:\n\n1. sessions.display_name -- polylogue/storage/runtime/archive/records.py\n (SessionRecord.display_name) and sessions_reads.py read it, but\n archive/session/domain_models.py::Session/SessionSummary has no\n `display_name` field, so hydrators.py drops it silently when building the\n domain model. Subagent-slug/session display metadata that's stored is\n currently unreachable end-to-end.\n2. sessions.run_settings_json -- same shape: SessionRecord.run_settings is\n read (Drive/Gemini run-settings verbatim JSON, model name etc.) but the\n Session domain model has no field for it either.\n3. session_links.parent_tool_use_block_id -- modeled on\n archive/topology/edge.py::TopologyEdge.parent_tool_use_block_id (the real\n delegation join key, replacing prior best-effort inference), populated by\n storage/sqlite/archive_tiers/write.py, but grep finds zero CLI/MCP/insights\n consumers of TopologyEdge.parent_tool_use_block_id -- the topology surface\n (`read --view` / MCP topology tool) cannot yet answer \"which exact tool_use\n call spawned this subagent session\" even though the join key is stored.\n\nNone of these need a schema change (all already exist on schema v46). Each is\na small, mechanical field-add to a domain model + hydrator + one surface\n(topology reader for #3; Session model + relevant CLI/MCP session payload for\n#1/#2) -- similar shape to the stop_reason fix landed alongside this bead.\nScope each separately since they touch different domain models (Session vs\nTopologyEdge) and different surfaces.","status":"open","priority":3,"issue_type":"task","owner":"ezo.dev@gmail.com","created_at":"2026-07-29T18:40:07Z","created_by":"Sinity","updated_at":"2026-07-29T18:40:07Z","dependency_count":0,"dependent_count":0,"comment_count":0} {"_type":"issue","id":"polylogue-zahj","title":"Operator decision: 2 stuck blob-publication reservations (42.5MB) pin unreferenced blobs","description":"Blob-store audit found 2 rows in source.db's blob_publication_reservations that have been 'unresolved' (blob present on disk, not referenced by raw_sessions/blob_refs/index.db attachments) since reservation, with no automatic path to clear them:\n publication_id=f1c44ec2-4250-4f12-87d3-97412cd08144 blob_hash=a8387c87ff8550f69330e30f1ea581e86e3e61fad590b5f8e33bcfe21a53d1a2 size=16021146B reserved_at=2026-07-12 14:03 UTC\n publication_id=0d21f742-779e-49e3-b96f-c8f1abeecc59 blob_hash=bad5d59e51a8b9b509c59e58f21f8afb0e8b7c7fbfe95cc6fc922bfbe7ead83d size=26470839B reserved_at=2026-07-13 10:45 UTC\nBoth blobs total 42.5MB and are on disk at blob/a8/387c... and blob/ba/d5d5.... They are the ONLY 2 truly-orphaned blobs in the entire 69GB/100K-object store (everything else that looked orphaned from source.db alone is still legitimately referenced via index.db's attachments table -- confirmed the store IS correctly content-addressed/deduplicated, no other waste found).\nVerify with: polylogue ops maintenance blob-publications (lists all receipts with referenced/present state), then if the operator confirms these two acquisitions were genuinely superseded/abandoned (not an in-flight publisher), release them with:\npolylogue ops maintenance blob-publications --abandon f1c44ec2-4250-4f12-87d3-97412cd08144 --abandon 0d21f742-779e-49e3-b96f-c8f1abeecc59 --yes\nThis only removes the RESERVATION (the protection), not the blob itself -- the next blob-gc pass would then be free to consider deleting the underlying blob bytes if truly unreferenced. Not doing this myself: blob deletion is explicitly the highest-risk operation in this system (evidence loss unrecoverable) and this decision needs operator judgment on whether the July 2026 acquisitions these reservations protected are safe to release.","status":"open","priority":3,"issue_type":"task","owner":"ezo.dev@gmail.com","created_at":"2026-07-29T08:44:55Z","created_by":"Sinity","updated_at":"2026-07-29T08:44:55Z","dependency_count":0,"dependent_count":0,"comment_count":0} diff --git a/polylogue/sources/origin_specs.py b/polylogue/sources/origin_specs.py index a3126041a2..561fcc0990 100644 --- a/polylogue/sources/origin_specs.py +++ b/polylogue/sources/origin_specs.py @@ -831,6 +831,10 @@ def _aistudio_drive_spec() -> OriginSpec: "runSettings (temperature/topP/topK/maxOutputTokens/thinkingLevel/safetySettings/enable* flags) " "is read and stored verbatim as sessions.run_settings_json (polylogue-2qx.4 / polylogue-cgfy); " "deliberately not decomposed into columns so the schema stays uncoupled from one provider's knobs.", + "chunkedPrompt.pendingInputs (unsent textbox drafts) is read and stored verbatim as " + "sessions.pending_drafts_json (polylogue-o4j2), deliberately as a session-row field rather than a " + "session_event: a draft is mutable current UI state, and session_events participate in " + "session_revision_projection's append-only comparison axes (polylogue-aggz Invariant 1).", "drive_support_blocks.py's _SUCCESS_OUTCOMES ({'ok', 'success', " "'succeeded', 'completed', 'outcome_ok'}) is not (yet) a " "DroppedValueVocabulary (polylogue-2qx): Gemini's own committed " diff --git a/polylogue/sources/parsers/base_models.py b/polylogue/sources/parsers/base_models.py index d13b4153ec..a0c461a581 100644 --- a/polylogue/sources/parsers/base_models.py +++ b/polylogue/sources/parsers/base_models.py @@ -359,6 +359,15 @@ class ParsedSession(BaseModel): # maxOutputTokens/thinkingLevel/safetySettings/enable* flags). Stored # verbatim as a JSON column -- see sessions.run_settings_json. run_settings: dict[str, object] | None = None + # polylogue-o4j2: non-blank chunkedPrompt.pendingInputs entries (unsent + # AI Studio textbox drafts), each {"text": ..., "role": ..., optionally + # "token_count": ...}. Deliberately NOT a session_event: a draft is + # mutable current UI state -- edited in place, gone entirely once + # submitted -- and session_events participate in + # session_revision_projection's append-only comparison axes + # (polylogue-aggz Invariant 1). Stored verbatim as a JSON column, same + # pattern as run_settings -- see sessions.pending_drafts_json. + pending_drafts: list[dict[str, object]] = Field(default_factory=list) # polylogue-2qx.4 / polylogue-cgfy: tracker-agnostic external references # (pr-link today, issue refs generalize to the same relation). session_refs: list[ParsedSessionRef] = Field(default_factory=list) diff --git a/polylogue/sources/parsers/drive.py b/polylogue/sources/parsers/drive.py index 97e5855afa..da55e8b504 100644 --- a/polylogue/sources/parsers/drive.py +++ b/polylogue/sources/parsers/drive.py @@ -215,6 +215,50 @@ def _model_config_event( ) +def _pending_drafts(pending_inputs: object) -> list[dict[str, object]]: + """Extract non-blank ``chunkedPrompt.pendingInputs`` entries. + + AI Studio's Drive-synced JSON carries the operator's not-yet-submitted + textbox content here -- draft prompts that never became a chunk and are + otherwise unrecoverable once overwritten (polylogue-o4j2). Entries with + blank/whitespace-only text are the overwhelmingly common case (the + textbox was empty when synced) and carry no evidence, so they are + skipped rather than kept as near-100%-empty noise. + + Deliberately returned as plain dicts for ``ParsedSession.pending_drafts``, + NOT ``ParsedSessionEvent``s: a draft is mutable CURRENT state (the + operator edits the same textbox in place, and the entry disappears + entirely once submitted), not an append-only historical fact. + ``session_events`` feeds ``session_revision_projection``'s + message/attachment/event comparison axes (polylogue-aggz Invariant 1), + which assume every axis only ever grows between two acquisitions of the + same session; a mutable, disappearing item there reproduces the exact + defect class polylogue-bu1i (acquisition state in identity) and + polylogue-nuec (provider-remeasurement in identity) were fixed for -- + edits would compare as disjoint forks, and submission would shrink the + event axis while the message axis grows, both misclassifying revision + membership. ``pending_drafts`` stays outside every identity/hash + computation in ``pipeline/ids.py`` (see ``sessions.pending_drafts_json``). + """ + if not isinstance(pending_inputs, list): + return [] + drafts: list[dict[str, object]] = [] + for entry in pending_inputs: + entry_obj = json_document(entry) + text = entry_obj.get("text") + if not isinstance(text, str) or not text.strip(): + continue + draft: dict[str, object] = {"text": text} + role_val = _string_field(entry_obj, "role") + if role_val is not None: + draft["role"] = role_val + token_count = _non_negative_int_field(entry_obj, "tokenCount", "token_count") + if token_count is not None: + draft["token_count"] = token_count + drafts.append(draft) + return drafts + + def _delivery_status(chunk_obj: JSONDocument) -> str | None: if _string_field(chunk_obj, "errorMessage", "error_message") is not None: return "error" @@ -380,6 +424,7 @@ def parse_chunked_prompt(provider: Provider | str, payload: JSONDocument, fallba if payload.get("updateTime") else _select_timestamp(observed_timestamps, latest=True) ) + pending_drafts = _pending_drafts(prompt.get("pendingInputs")) active_leaf_message_provider_id = messages[-1].provider_message_id if messages else None if active_leaf_message_provider_id is not None: messages = [ @@ -409,6 +454,11 @@ def parse_chunked_prompt(provider: Provider | str, payload: JSONDocument, fallba # the ``model_config`` session_event above; this is the same value # landing on the session row itself. run_settings=dict(run_settings) if run_settings else None, + # polylogue-o4j2: pendingInputs draft(s), kept off session_events on + # purpose -- see _pending_drafts' docstring for why (mutable current + # state must not enter session_revision_projection's comparison + # axes). + pending_drafts=pending_drafts, ) diff --git a/polylogue/storage/runtime/archive/records.py b/polylogue/storage/runtime/archive/records.py index 813bc96937..551f7ee4d3 100644 --- a/polylogue/storage/runtime/archive/records.py +++ b/polylogue/storage/runtime/archive/records.py @@ -12,7 +12,7 @@ from polylogue.archive.session.branch_type import BranchType from polylogue.core.enums import BlockType, MaterialOrigin, Origin, SemanticBlockType, SessionKind from polylogue.core.hashing import hash_text -from polylogue.core.json import json_document +from polylogue.core.json import json_document, json_document_list from polylogue.core.security import sanitize_path as _sanitize_path_helper from polylogue.core.timestamps import canonical_timestamp_text from polylogue.core.types import AttachmentId, ContentHash, MessageId, SessionEventId, SessionId @@ -58,6 +58,12 @@ class SessionRecord(BaseModel): # verbatim (aistudio-drive runSettings). None when the read path didn't # select the column or the provider carries none. run_settings: JSONObject | None = None + # polylogue-o4j2 (v47): non-blank chunkedPrompt.pendingInputs entries + # (unsent AI Studio textbox drafts), stored verbatim. Deliberately a + # plain session-row field, outside session_revision_projection's + # comparison axes -- see sessions.pending_drafts_json / drive.py's + # _pending_drafts docstring for why a draft cannot be a session_event. + pending_drafts: list[JSONObject] | None = None @field_validator("origin", mode="before") @classmethod @@ -83,6 +89,14 @@ def non_empty_string(cls, v: str) -> str: def coerce_json_document(cls, value: object) -> JSONObject | None: return _coerce_json_object(value) + @field_validator("pending_drafts", mode="before") + @classmethod + def coerce_pending_drafts(cls, value: object) -> list[JSONObject] | None: + if value is None: + return None + documents = json_document_list(value) + return [dict(document) for document in documents] or None + @field_validator("created_at", "updated_at", mode="before") @classmethod def coerce_archive_timestamp(cls, value: object) -> str | None: diff --git a/polylogue/storage/sqlite/archive_tiers/index.py b/polylogue/storage/sqlite/archive_tiers/index.py index 42a8849b33..34a6ca2333 100644 --- a/polylogue/storage/sqlite/archive_tiers/index.py +++ b/polylogue/storage/sqlite/archive_tiers/index.py @@ -71,7 +71,17 @@ # A bump without a declaration is a policy violation, not a free rebuild: # `devtools lab policy schema-versioning` fails, and the archive silently # falls back to full raw replay. See polylogue-9rw0 / polylogue-b5l. -INDEX_SCHEMA_VERSION = 46 +# +# polylogue-o4j2: v47 adds sessions.pending_drafts_json -- aistudio-drive's +# chunkedPrompt.pendingInputs non-blank entries (unsent textbox drafts, 7/397 +# real sessions with draft text on the live archive). Landed as a session-row +# field rather than a session_event on purpose: a draft is mutable CURRENT +# state (edited in place, gone entirely once submitted), and session_events +# participate in session_revision_projection's append-only comparison axes +# (polylogue-aggz Invariant 1) -- putting mutable state there reproduces the +# exact defect class polylogue-bu1i (acquisition state) and polylogue-nuec +# (provider-remeasurement) were fixed for. See sessions table comment. +INDEX_SCHEMA_VERSION = 47 # polylogue-v6i3: shared WHEN-clause fragment gating the blocks_command_trigram # trigger BODIES on the same dedicated bulk-build guard row messages_fts's @@ -193,6 +203,17 @@ -- into typed columns would couple this schema to one provider for no -- query benefit; nothing here is queried across origins today. run_settings_json TEXT CHECK ({json_object_check("run_settings_json", nullable=True)}), + -- polylogue-o4j2 (v47): non-blank chunkedPrompt.pendingInputs entries -- + -- the operator's not-yet-submitted textbox draft(s) -- verbatim as a + -- JSON array of {{text, role, token_count}} objects. Deliberately a + -- session-row field, NOT a session_event: a draft is CURRENT mutable + -- UI state (edited in place, then disappears entirely on submit), not + -- an append-only historical fact, so it must stay outside + -- session_revision_projection's message/attachment/event comparison + -- axes (polylogue-aggz Invariant 1) -- exactly the shape polylogue-bu1i + -- and polylogue-nuec were fixed for, on a third axis (mutable session + -- state rather than acquisition state or provider-remeasurement). + pending_drafts_json TEXT CHECK ({json_array_check("pending_drafts_json", nullable=True)}), git_branch TEXT, git_repository_url TEXT, provider_project_ref TEXT, diff --git a/polylogue/storage/sqlite/archive_tiers/write.py b/polylogue/storage/sqlite/archive_tiers/write.py index a1c1b965c0..45037151db 100644 --- a/polylogue/storage/sqlite/archive_tiers/write.py +++ b/polylogue/storage/sqlite/archive_tiers/write.py @@ -516,7 +516,7 @@ def add_timing(name: str, started_at: float) -> None: INSERT INTO sessions ( native_id, origin, raw_id, branch_type, active_leaf_message_id, title, session_kind, title_source, title_ref, title_confidence, - display_name, run_settings_json, + display_name, run_settings_json, pending_drafts_json, git_branch, git_repository_url, commit_hash, instructions_text, reported_duration_ms, provider_project_ref, message_count, word_count, tool_use_count, thinking_count, @@ -524,7 +524,7 @@ def add_timing(name: str, started_at: float) -> None: assistant_message_count, system_message_count, tool_message_count, user_word_count, authored_user_word_count, assistant_word_count, content_hash, created_at_ms, updated_at_ms - ) VALUES (?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?) + ) VALUES (?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?) ON CONFLICT(origin, native_id) DO UPDATE SET raw_id = excluded.raw_id, branch_type = excluded.branch_type, @@ -536,6 +536,12 @@ def add_timing(name: str, started_at: float) -> None: title_confidence = COALESCE(excluded.title_confidence, sessions.title_confidence), display_name = COALESCE(excluded.display_name, sessions.display_name), run_settings_json = COALESCE(excluded.run_settings_json, sessions.run_settings_json), + -- Plain overwrite, NOT COALESCE like run_settings_json above: + -- a draft is current mutable state, so a reprocess that finds + -- no non-blank pendingInputs (submitted, or cleared) must + -- actually clear the stored value rather than preserving a + -- now-stale draft forever (polylogue-o4j2). + pending_drafts_json = excluded.pending_drafts_json, git_branch = excluded.git_branch, git_repository_url = excluded.git_repository_url, commit_hash = excluded.commit_hash, @@ -562,6 +568,7 @@ def add_timing(name: str, started_at: float) -> None: session.title_confidence, _sqlite_text(session.display_name), _json_dumps(session.run_settings) if session.run_settings else None, + _json_dumps(session.pending_drafts) if session.pending_drafts else None, _sqlite_text(session.git_branch), _sqlite_text(session.git_repository_url), _sqlite_text(session.git_commit_hash), diff --git a/polylogue/storage/sqlite/lifecycle.py b/polylogue/storage/sqlite/lifecycle.py index eed94dd99d..3d030d8e0a 100644 --- a/polylogue/storage/sqlite/lifecycle.py +++ b/polylogue/storage/sqlite/lifecycle.py @@ -434,6 +434,17 @@ class IndexDeltaDeclarationReport(TypedDict): # behaviour. classes=(DerivedDeltaClass.SEMANTIC_REPARSE,), ), + IndexDeltaDeclaration( + version=47, + # polylogue-o4j2: sessions.pending_drafts_json -- aistudio-drive + # pendingInputs draft text, moved off the session_events axis (see + # index.py's v47 header comment). Values depend on parser semantics + # (the new column is populated only by re-parsing the drive.py + # payload), so a shape-only copy-forward would leave every row NULL + # -- the same v42/v44/v45/v46 precedent. SEMANTIC_REPARSE routes + # through `polylogue ops reset --index && polylogued run`. + classes=(DerivedDeltaClass.SEMANTIC_REPARSE,), + ), ) diff --git a/polylogue/storage/sqlite/queries/mappers_archive.py b/polylogue/storage/sqlite/queries/mappers_archive.py index 352c1a7076..d8d6eb02ea 100644 --- a/polylogue/storage/sqlite/queries/mappers_archive.py +++ b/polylogue/storage/sqlite/queries/mappers_archive.py @@ -33,6 +33,7 @@ ) from polylogue.storage.sqlite.queries.mappers_support import ( _json_object, + _json_object_list, _parse_json, _row_float, _row_get, @@ -68,6 +69,9 @@ def _row_to_session(row: sqlite3.Row) -> SessionRecord: run_settings=_json_object( _parse_json(_row_get(row, "run_settings_json"), field="run_settings_json", record_id=row["session_id"]) ), + pending_drafts=_json_object_list( + _parse_json(_row_get(row, "pending_drafts_json"), field="pending_drafts_json", record_id=row["session_id"]) + ), ) diff --git a/polylogue/storage/sqlite/queries/mappers_support.py b/polylogue/storage/sqlite/queries/mappers_support.py index 0bed076b7e..4d06f23b51 100644 --- a/polylogue/storage/sqlite/queries/mappers_support.py +++ b/polylogue/storage/sqlite/queries/mappers_support.py @@ -142,6 +142,18 @@ def _json_object(value: JSONValue | None) -> JSONObject | None: return result +def _json_object_list(value: JSONValue | None) -> list[JSONObject] | None: + """Parse a JSON array of objects, e.g. ``sessions.pending_drafts_json``.""" + if not isinstance(value, list): + return None + documents: list[JSONObject] = [] + for item in value: + document = json_document(item) + if document: + documents.append(dict(document)) + return documents or None + + def _json_text_tuple(value: JSONValue | None) -> tuple[str, ...]: if not isinstance(value, list): return () diff --git a/polylogue/storage/sqlite/queries/sessions_reads.py b/polylogue/storage/sqlite/queries/sessions_reads.py index eb3adaf08d..e9a2b5cd75 100644 --- a/polylogue/storage/sqlite/queries/sessions_reads.py +++ b/polylogue/storage/sqlite/queries/sessions_reads.py @@ -29,7 +29,8 @@ git_repository_url, provider_project_ref, display_name, - run_settings_json + run_settings_json, + pending_drafts_json """ @@ -60,7 +61,8 @@ def _session_record_select(alias: str | None = None) -> str: {prefix}git_repository_url AS git_repository_url, {prefix}provider_project_ref AS provider_project_ref, {prefix}display_name AS display_name, - {prefix}run_settings_json AS run_settings_json + {prefix}run_settings_json AS run_settings_json, + {prefix}pending_drafts_json AS pending_drafts_json """ diff --git a/tests/unit/sources/test_parsers_drive.py b/tests/unit/sources/test_parsers_drive.py index 815e1d7691..6ef6fee16b 100644 --- a/tests/unit/sources/test_parsers_drive.py +++ b/tests/unit/sources/test_parsers_drive.py @@ -15,7 +15,7 @@ import pytest -from polylogue.core.json import JSONDocument +from polylogue.core.json import JSONDocument, JSONValue from polylogue.scenarios import CorpusSpec from polylogue.sources.parsers.drive import ( _attachment_from_doc, @@ -198,6 +198,147 @@ def test_parse_chunked_prompt_without_run_settings_leaves_it_none() -> None: assert result.run_settings is None +def test_parse_chunked_prompt_records_nonempty_pending_input_as_draft() -> None: + """``chunkedPrompt.pendingInputs`` (polylogue-o4j2) is the operator's + not-yet-submitted textbox content -- unrecoverable once overwritten if + dropped at parse. A non-blank entry must survive on + ``ParsedSession.pending_drafts``. + + Deliberately NOT a session_event: a draft is mutable current state, and + session_events feed session_revision_projection's append-only + comparison axes (polylogue-aggz Invariant 1) -- see + ``test_pending_draft_mutation_does_not_break_revision_containment`` + below for the failure this would otherwise reproduce. + """ + payload: JSONDocument = { + "id": "gemini-pending-draft", + "updateTime": "2024-01-15T11:45:00Z", + "chunkedPrompt": { + "chunks": [{"id": "msg-user", "role": "user", "text": "hi"}], + "pendingInputs": [{"text": "unsent follow-up question", "role": "user", "tokenCount": 4}], + }, + } + + result = parse_chunked_prompt("gemini", payload, "fallback-id") + + assert result.pending_drafts == [{"text": "unsent follow-up question", "role": "user", "token_count": 4}] + assert not [e for e in result.session_events if e.event_type == "draft_input"] + + +def test_parse_chunked_prompt_skips_blank_pending_input() -> None: + """The wire-common case -- the textbox was empty when Drive synced -- carries + no evidence and must not be recorded as a draft. + """ + payload: JSONDocument = { + "id": "gemini-pending-blank", + "chunkedPrompt": { + "chunks": [{"id": "msg-user", "role": "user", "text": "hi"}], + "pendingInputs": [{"text": "", "role": "user"}, {"text": " ", "role": "user"}], + }, + } + + result = parse_chunked_prompt("gemini", payload, "fallback-id") + + assert result.pending_drafts == [] + + +def test_parse_chunked_prompt_without_pending_inputs_has_no_drafts() -> None: + payload: JSONDocument = { + "id": "gemini-no-pending", + "chunkedPrompt": {"chunks": [{"id": "msg-user", "role": "user", "text": "hi"}]}, + } + + result = parse_chunked_prompt("gemini", payload, "fallback-id") + + assert result.pending_drafts == [] + + +def test_pending_draft_mutation_does_not_break_revision_containment() -> None: + """Regression for the P1 a reviewer traced on drive.py's original + draft-as-session_event design (polylogue-o4j2 fix-up). + + A draft is mutable: the operator edits the textbox, then eventually + submits it (at which point the pendingInputs entry disappears and a real + message appears instead). If the draft were folded into + session_revision_projection's event axis, editing it would create + disjoint event identities (comparing as a conflict/fork) and submitting + it would shrink the event axis while the message axis grows -- + ``_relation`` requires every non-equal axis to agree on direction, so + both cases would misclassify revision membership + (classify_membership_revisions). This walks retain -> edit draft -> + retain -> submit and asserts containment holds at every step now that + drafts live outside every comparison axis. + """ + from polylogue.archive.session_revision_membership import ( + MembershipRevision, + _relation, + classify_membership_revisions, + ) + from polylogue.pipeline.ids import session_revision_projection + + chunks_before_submit: list[JSONValue] = [{"id": "msg-1", "role": "user", "text": "hi"}] + + # Revision 1: retain with an initial draft in the textbox. + payload_1: JSONDocument = { + "id": "gemini-draft-lifecycle", + "chunkedPrompt": { + "chunks": chunks_before_submit, + "pendingInputs": [{"text": "draft v1", "role": "user"}], + }, + } + revision_1 = parse_chunked_prompt("gemini", payload_1, "fallback-id") + # Revision 2: the SAME conversation retained again after the operator + # edited the draft text (no new message yet). + payload_2: JSONDocument = { + "id": "gemini-draft-lifecycle", + "chunkedPrompt": { + "chunks": chunks_before_submit, + "pendingInputs": [{"text": "draft v2, much longer now", "role": "user"}], + }, + } + revision_2 = parse_chunked_prompt("gemini", payload_2, "fallback-id") + # Revision 3: the draft was submitted -- it becomes a real message and + # pendingInputs is empty again. + payload_3: JSONDocument = { + "id": "gemini-draft-lifecycle", + "chunkedPrompt": { + "chunks": [ + *chunks_before_submit, + {"id": "msg-2", "role": "user", "text": "draft v2, much longer now"}, + ], + "pendingInputs": [{"text": "", "role": "user"}], + }, + } + revision_3 = parse_chunked_prompt("gemini", payload_3, "fallback-id") + + projection_1 = session_revision_projection(revision_1) + projection_2 = session_revision_projection(revision_2) + projection_3 = session_revision_projection(revision_3) + + # Editing the draft alone (same messages, different draft text) must not + # look like a fork -- both revisions carry the exact same content-bearing + # evidence once drafts are excluded from comparison identity. + assert _relation(projection_1, projection_2) == "equal" + # Submitting must read as ordinary append-only growth (revision 3 + # contains revision 2), not a conflict from the event axis shrinking. + assert _relation(projection_3, projection_2) == "a_contains_b" + + classification = classify_membership_revisions( + [ + MembershipRevision(raw_id="r1", projection=projection_1), + MembershipRevision(raw_id="r2", projection=projection_2), + MembershipRevision(raw_id="r3", projection=projection_3), + ] + ) + # accepted_raw_ids is the whole append-only growth chain, oldest to + # newest (r1/r2 collapse to one "equal" representative -- edit-only + # revisions -- which then chains into r3's growth); the key assertion is + # what is ABSENT: no conflict, so nothing lands in ambiguous_raw_ids. + assert classification.accepted_raw_ids == ("r1", "r3") + assert classification.equivalent_raw_ids == ("r2",) + assert not classification.ambiguous_raw_ids + + def test_parse_chunked_prompt_records_fallback_title_source() -> None: payload: JSONDocument = { "id": "gemini-fallback-title", diff --git a/tests/unit/storage/test_unread_wire_batch_v46.py b/tests/unit/storage/test_unread_wire_batch_v46.py index 2e28b57b13..e6aadf88ab 100644 --- a/tests/unit/storage/test_unread_wire_batch_v46.py +++ b/tests/unit/storage/test_unread_wire_batch_v46.py @@ -157,6 +157,75 @@ async def test_session_display_name_and_run_settings_round_trip(tmp_path: Path) assert sessions[0].run_settings == {"temperature": 0.7, "topP": 0.9} +async def test_session_pending_drafts_round_trips_through_writer_and_repository(tmp_path: Path) -> None: + """sessions.pending_drafts_json (v47, polylogue-o4j2): written by the real writer. + + Fails if the writer stops persisting ``ParsedSession.pending_drafts`` (a + revert of the ``sessions`` INSERT column list), or if + ``SessionRepository.get_sessions_batch`` stops selecting/mapping the + column. Deliberately NOT a session_event round trip -- see + ``ParsedSession.pending_drafts``'s docstring for why a draft must stay + outside session_revision_projection's comparison axes. + """ + backend = SQLiteBackend(db_path=tmp_path / "pending-drafts.db") + repo = SessionRepository(backend=backend) + try: + session_id = await ingest_session( + ParsedSession( + source_name=Provider.GEMINI, + provider_session_id="pending-drafts-1", + title="Untitled", + pending_drafts=[{"text": "unsent follow-up", "role": "user", "token_count": 3}], + messages=[ + ParsedMessage( + provider_message_id="m1", + role=Role.USER, + text="hi", + position=0, + blocks=[ParsedContentBlock(type=BlockType.TEXT, text="hi")], + ), + ], + ), + backend=backend, + ) + sessions = await repo.get_sessions_batch([session_id]) + finally: + await repo.close() + + assert len(sessions) == 1 + assert sessions[0].pending_drafts == [{"text": "unsent follow-up", "role": "user", "token_count": 3}] + + +async def test_session_pending_drafts_empty_for_session_without_drafts(tmp_path: Path) -> None: + """No pendingInputs on the wire must round-trip as None, not an empty list.""" + backend = SQLiteBackend(db_path=tmp_path / "pending-drafts-empty.db") + repo = SessionRepository(backend=backend) + try: + session_id = await ingest_session( + ParsedSession( + source_name=Provider.GEMINI, + provider_session_id="pending-drafts-empty-1", + title="Untitled", + messages=[ + ParsedMessage( + provider_message_id="m1", + role=Role.USER, + text="hi", + position=0, + blocks=[ParsedContentBlock(type=BlockType.TEXT, text="hi")], + ), + ], + ), + backend=backend, + ) + sessions = await repo.get_sessions_batch([session_id]) + finally: + await repo.close() + + assert len(sessions) == 1 + assert sessions[0].pending_drafts is None + + async def test_file_edits_round_trip_keyed_by_tool_use_block(tmp_path: Path) -> None: """file_edits: a new relation keyed by the tool_use block id.