Skip to content

refactor(storage): route canonical writes through Bun - #1346

Open
mariusvniekerk wants to merge 13 commits into
t3code/bun-search-unificationfrom
t3code/bun-write-unification
Open

refactor(storage): route canonical writes through Bun#1346
mariusvniekerk wants to merge 13 commits into
t3code/bun-search-unificationfrom
t3code/bun-write-unification

Conversation

@mariusvniekerk

@mariusvniekerk mariusvniekerk commented Aug 5, 2026

Copy link
Copy Markdown
Collaborator

Independent archive, PostgreSQL replication, and DuckDB mirror writers manually projected overlapping schemas, which made column ownership and transaction semantics vulnerable to drift. This layer routes canonical session, pricing, cursor, identity, and dependent writes through Bun transactions and the shared model registry.

PostgreSQL ownership serialization uses collision-resistant full-digest keys, SQLite pricing uses the canonical Bun writer and pricing contract, and usage-event snapshots order mixed-offset timestamps chronologically. Operational synchronization metadata and backend lifecycle remain adapter-owned. Stack 4 of 5.

generated by a clanker

@roborev-ci

roborev-ci Bot commented Aug 5, 2026

Copy link
Copy Markdown

roborev: Combined Review (18e86ea)

Storage refactor needs changes for two medium-severity data integrity regressions.

Medium

  • internal/db/bun_write.go:141 — Batch ingestion no longer performs prior tool-result normalization, omitting positional event indexes, inferred content lengths, and inherited parent tool_use_id/subagent_session_id values. Parsers such as Poolside can therefore persist incomplete result events, while multiple zero-valued indexes can collide on canonical keys. Normalize result events at the archive-ingestion boundary before canonical conversion, preserve the previous defaults, and add coverage for parent-ID inheritance and multiple unindexed events.

  • internal/db/bun_usage.go:224 — Revision normalization incorrectly applies to underscore-prefixed pricing metadata whose updated_at values are arbitrary versions such as "v42" or "2". Fresh PostgreSQL or DuckDB replication replaces these values with timestamps, and later pushes may not repair the corruption. Bypass timestamp revision handling for metadata patterns, preserve sanitized UpdatedAt values verbatim, and add target contract tests with non-timestamp metadata values.


Reviewers: 2 done | Synthesis: codex, 14s | Total: 33m22s

@mariusvniekerk
mariusvniekerk force-pushed the t3code/bun-write-unification branch from 18e86ea to 7a4225f Compare August 7, 2026 03:46
@roborev-ci

roborev-ci Bot commented Aug 7, 2026

Copy link
Copy Markdown

roborev: Combined Review (7a4225f)

Medium-severity issue found: one malformed session can block an entire incremental PostgreSQL push.

Medium

  • internal/postgres/push.go:462 — Incremental PostgreSQL preparation aborts the entire push when a snapshot contains a legacy unsupported timestamp. Because this occurs before batching, it bypasses per-session retry/error isolation and prevents valid sessions from being pushed.
    • Fix: Handle bunmodel.ErrUnsupportedTimestamp like the DuckDB path: force that session’s fingerprint to mismatch, then let pushBatch isolate the failure. Add coverage for an incremental push containing both valid and malformed sessions.

Reviewers: 2 done | Synthesis: codex, 7s | Total: 25m41s

@mariusvniekerk
mariusvniekerk force-pushed the t3code/bun-write-unification branch from 7a4225f to 2ef9863 Compare August 7, 2026 14:10
@roborev-ci

roborev-ci Bot commented Aug 7, 2026

Copy link
Copy Markdown

roborev: Combined Review (2ef9863)

Medium-severity concurrency issue found; no critical or high-severity findings.

Medium

  • internal/db/bun_usage.go:172 — Pricing revisions are calculated before the upsert without locking. Concurrent pushers can derive revisions from the same stale row, allowing a later write to store an equal or older revision despite changed pricing. Compute the revision atomically against the current target row during the upsert to preserve monotonic revisions.

Reviewers: 2 done | Synthesis: codex, 5s | Total: 24m13s

@mariusvniekerk
mariusvniekerk force-pushed the t3code/bun-write-unification branch from 2ef9863 to bb1a3c8 Compare August 7, 2026 16:34
@roborev-ci

roborev-ci Bot commented Aug 7, 2026

Copy link
Copy Markdown

roborev: Combined Review (bb1a3c8)

Storage review found two medium-severity concurrency and replication parity issues.

Medium

  • internal/db/bun_usage.go:194 — PostgreSQL pricing revisions use an unlocked read, allowing concurrent pushers to overwrite newer revisions with stale timestamps. Serialize pricing updates or restore an atomic monotonic revision calculation in the conflict clause.

  • internal/postgres/worktree_mappings_push.go:158 — PostgreSQL mapping writes omit the canonical id and created_at fields, causing incorrect defaults and breaking DuckDB/PostgreSQL parity. Include and update both fields from canonical source mapping rows.


Reviewers: 2 done | Synthesis: codex, 12s | Total: 12m37s

@mariusvniekerk
mariusvniekerk force-pushed the t3code/bun-write-unification branch from bb1a3c8 to 66ea17d Compare August 9, 2026 03:46
@roborev-ci

roborev-ci Bot commented Aug 9, 2026

Copy link
Copy Markdown

roborev: Combined Review (66ea17d)

Code changes are clean with no Medium, High, or Critical findings.


Reviewers: 2 done | Synthesis: codex, 9s | Total: 34m12s

Secret listing and reveal behavior are canonical storage policy, but separate SQLite, PostgreSQL, and DuckDB implementations duplicated filtering and reconstructed entire sessions to resolve one finding. That made parity fragile and turned a targeted reveal into an unbounded payload read.

Route all adapters through the shared Bun store and resolve sources by their persisted message, call, and event coordinates. Keep tool-result fallback semantics intact while removing the obsolete full-session reconstruction path.
Once BunStore owns validation, hydration, pagination, and capability reporting, backend forwarding methods and the old PostgreSQL content pipeline only preserve duplicate ownership and invite drift. Zero-value fixture fallbacks also hide invalid construction instead of exercising the real common-store boundary.

Rely on promoted BunStore methods across SQLite, PostgreSQL, and DuckDB, retain only the PostgreSQL hybrid lexical helpers at the dialect edge, and make test fixtures initialize or record the actual boundary they exercise.
Session ingestion and mirror replication need one canonical conversion and transaction boundary so message, tool, usage, and secret rows cannot drift between adapters.\n\nKeep SQLite's FTS5 bulk-delete optimization and DuckDB's source-ID/unique-index requirements at narrow adapter seams while sharing row validation, batching, logical-key upserts, and atomic replacement behavior.
Mirror replication must never publish a session row and dependents observed from different archive revisions, and shared search must not let stale anchors or SQLite's ASCII-only LIKE behavior consume canonical result slots.\n\nUse one archive read snapshot per replicated session, compare canonical PostgreSQL rows before rewriting, validate portable tool relationships, and remove the superseded PostgreSQL comparison pipeline. This keeps dialect differences at storage seams while preserving stable no-op pushes.
Pricing bands and Cursor accounting have the same portable shape across adapters, so separate SQL builders only multiply conflict, batching, and sanitization behavior.\n\nRoute PostgreSQL and DuckDB through the canonical Bun row converters and transactional writers while retaining fallback catalog selection, changed-row filtering, project scope, and DuckDB's source high-water publication.
A target skip marker is only safe when it fingerprints the exact source snapshot committed beside it. Preflight hashes assembled from separate reads could label newer rows with an older revision and permanently suppress repair.\n\nDerive one canonical fingerprint from the raw session and all replicated dependents, recompute it at write time, preserve target-owned PostgreSQL curation, and use the full Bun session row for DuckDB. Keep pricing revisions monotonic and historical Cursor rows replicable while removing the superseded fingerprint pipelines.
Session replication still carried a PostgreSQL-only upsert with its own column list, sanitization, timestamp precision, and no-op policy. That made the canonical schema incomplete at the most important parent-row boundary and allowed adapter behavior to drift.\n\nApply PostgreSQL ownership and target-curation policy around the shared Bun session row, compare the complete portable row before publishing a revision, and normalize text, defaults, and microsecond timestamps once for every adapter.
Canonical session writes must not weaken target ownership, local curation, or incremental push scalability. Serialize first ownership, fingerprint only the projection PostgreSQL commits, and force one canonical-row backfill when that projection changes.\n\nKeep full pushes from materializing transcripts twice, preserve unchanged pricing revisions, and restore real PostgreSQL coverage for aliases, exclusions, and no-op target state.
Archive ingestion still depended on a hand-maintained session column list, leaving parser batches, identity updates, and recall/eval placeholders outside the canonical Bun row contract. That preserved a second schema path and allowed timestamp and default behavior to drift.\n\nRoute those writes through canonical rows while preserving archive-owned curation and parser bookkeeping, placeholder no-overwrite semantics, and trash/exclusion guards. Canonical timestamp output now consistently uses normalized microsecond precision.
Canonical writes need explicit ownership boundaries so malformed legacy source data can be repaired without allowing ingestion or replication to overwrite archive and target curation. The PostgreSQL projection also needs a portable first-owner lock and a versioned completion fence so retries and older binaries cannot silently retain partial rows.\n\nMake session ownership exhaustive, keep pricing revisions stable and repairable across engines, and route DuckDB worktree publication through the shared row contract while preserving source identities and timestamps.
DuckDB's Bun dialect omitted default-tagged columns from an entire batch when its first row used the default. This erased later message models, system flags, and authoritative usage-cost provenance, changing usage totals and analytics results.

Enable per-row DEFAULT placeholders, normalize colliding tool-result coordinates, and retain call/event ordering in shared content search.
SQLite pricing writes still bypassed the shared Bun contract, so revision behavior could diverge across adapters. Route the public writer through the canonical transaction and pin SQLite to the same pricing contract.

Ownership locks must identify a complete session key, and source usage events must preserve chronological order even when valid RFC3339 offsets have different textual forms. These fixes keep replication behavior stable without widening dialect-specific storage seams.
The write cutover no longer maintains a separate usage-event fingerprint loader. Keep exact microdollar coverage on the production skip-check boundary by comparing canonical dependent rows and proving a one-microdollar difference is detected.
@mariusvniekerk
mariusvniekerk force-pushed the t3code/bun-write-unification branch from 66ea17d to a560de7 Compare August 10, 2026 15:39
@roborev-ci

roborev-ci Bot commented Aug 10, 2026

Copy link
Copy Markdown

roborev: Combined Review (a560de7)

Changes requested: two medium-severity replication issues could block progress or leave historical DuckDB data incomplete.

Medium

  • internal/postgres/push.go:462 — Incremental preparation aborts the entire PostgreSQL push when any candidate has a legacy unsupported timestamp, bypassing batch fallback and potentially preventing all subsequent incremental progress when the watermark remains unchanged. Handle bunmodel.ErrUnsupportedTimestamp like the DuckDB path: assign a guaranteed-nonmatching marker and retain the session as a candidate so pushBatch can isolate its failure while other sessions continue.

  • internal/duckdb/push.go:949, internal/duckdb/schema.go:28 — The canonical upsert now writes previously omitted fields such as parser_parent_session_id, but the mirror schema/write-contract version was not changed. Existing version-11 mirrors only repair sessions within the incremental watermark window, leaving unchanged historical sessions without provenance indefinitely. Bump the DuckDB schema version or add a mirror write-contract version that forces a complete session backfill when the projection changes.


Reviewers: 2 done | Synthesis: codex, 18s | Total: 22m29s

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Development

Successfully merging this pull request may close these issues.

1 participant