refactor(storage): route canonical writes through Bun - #1346
refactor(storage): route canonical writes through Bun#1346mariusvniekerk wants to merge 13 commits into
Conversation
roborev: Combined Review (
|
18e86ea to
7a4225f
Compare
roborev: Combined Review (
|
7a4225f to
2ef9863
Compare
roborev: Combined Review (
|
2ef9863 to
bb1a3c8
Compare
roborev: Combined Review (
|
bb1a3c8 to
66ea17d
Compare
roborev: Combined Review (
|
Secret listing and reveal behavior are canonical storage policy, but separate SQLite, PostgreSQL, and DuckDB implementations duplicated filtering and reconstructed entire sessions to resolve one finding. That made parity fragile and turned a targeted reveal into an unbounded payload read. Route all adapters through the shared Bun store and resolve sources by their persisted message, call, and event coordinates. Keep tool-result fallback semantics intact while removing the obsolete full-session reconstruction path.
Once BunStore owns validation, hydration, pagination, and capability reporting, backend forwarding methods and the old PostgreSQL content pipeline only preserve duplicate ownership and invite drift. Zero-value fixture fallbacks also hide invalid construction instead of exercising the real common-store boundary. Rely on promoted BunStore methods across SQLite, PostgreSQL, and DuckDB, retain only the PostgreSQL hybrid lexical helpers at the dialect edge, and make test fixtures initialize or record the actual boundary they exercise.
Session ingestion and mirror replication need one canonical conversion and transaction boundary so message, tool, usage, and secret rows cannot drift between adapters.\n\nKeep SQLite's FTS5 bulk-delete optimization and DuckDB's source-ID/unique-index requirements at narrow adapter seams while sharing row validation, batching, logical-key upserts, and atomic replacement behavior.
Mirror replication must never publish a session row and dependents observed from different archive revisions, and shared search must not let stale anchors or SQLite's ASCII-only LIKE behavior consume canonical result slots.\n\nUse one archive read snapshot per replicated session, compare canonical PostgreSQL rows before rewriting, validate portable tool relationships, and remove the superseded PostgreSQL comparison pipeline. This keeps dialect differences at storage seams while preserving stable no-op pushes.
Pricing bands and Cursor accounting have the same portable shape across adapters, so separate SQL builders only multiply conflict, batching, and sanitization behavior.\n\nRoute PostgreSQL and DuckDB through the canonical Bun row converters and transactional writers while retaining fallback catalog selection, changed-row filtering, project scope, and DuckDB's source high-water publication.
A target skip marker is only safe when it fingerprints the exact source snapshot committed beside it. Preflight hashes assembled from separate reads could label newer rows with an older revision and permanently suppress repair.\n\nDerive one canonical fingerprint from the raw session and all replicated dependents, recompute it at write time, preserve target-owned PostgreSQL curation, and use the full Bun session row for DuckDB. Keep pricing revisions monotonic and historical Cursor rows replicable while removing the superseded fingerprint pipelines.
Session replication still carried a PostgreSQL-only upsert with its own column list, sanitization, timestamp precision, and no-op policy. That made the canonical schema incomplete at the most important parent-row boundary and allowed adapter behavior to drift.\n\nApply PostgreSQL ownership and target-curation policy around the shared Bun session row, compare the complete portable row before publishing a revision, and normalize text, defaults, and microsecond timestamps once for every adapter.
Canonical session writes must not weaken target ownership, local curation, or incremental push scalability. Serialize first ownership, fingerprint only the projection PostgreSQL commits, and force one canonical-row backfill when that projection changes.\n\nKeep full pushes from materializing transcripts twice, preserve unchanged pricing revisions, and restore real PostgreSQL coverage for aliases, exclusions, and no-op target state.
Archive ingestion still depended on a hand-maintained session column list, leaving parser batches, identity updates, and recall/eval placeholders outside the canonical Bun row contract. That preserved a second schema path and allowed timestamp and default behavior to drift.\n\nRoute those writes through canonical rows while preserving archive-owned curation and parser bookkeeping, placeholder no-overwrite semantics, and trash/exclusion guards. Canonical timestamp output now consistently uses normalized microsecond precision.
Canonical writes need explicit ownership boundaries so malformed legacy source data can be repaired without allowing ingestion or replication to overwrite archive and target curation. The PostgreSQL projection also needs a portable first-owner lock and a versioned completion fence so retries and older binaries cannot silently retain partial rows.\n\nMake session ownership exhaustive, keep pricing revisions stable and repairable across engines, and route DuckDB worktree publication through the shared row contract while preserving source identities and timestamps.
DuckDB's Bun dialect omitted default-tagged columns from an entire batch when its first row used the default. This erased later message models, system flags, and authoritative usage-cost provenance, changing usage totals and analytics results. Enable per-row DEFAULT placeholders, normalize colliding tool-result coordinates, and retain call/event ordering in shared content search.
SQLite pricing writes still bypassed the shared Bun contract, so revision behavior could diverge across adapters. Route the public writer through the canonical transaction and pin SQLite to the same pricing contract. Ownership locks must identify a complete session key, and source usage events must preserve chronological order even when valid RFC3339 offsets have different textual forms. These fixes keep replication behavior stable without widening dialect-specific storage seams.
The write cutover no longer maintains a separate usage-event fingerprint loader. Keep exact microdollar coverage on the production skip-check boundary by comparing canonical dependent rows and proving a one-microdollar difference is detected.
66ea17d to
a560de7
Compare
roborev: Combined Review (
|
Independent archive, PostgreSQL replication, and DuckDB mirror writers manually projected overlapping schemas, which made column ownership and transaction semantics vulnerable to drift. This layer routes canonical session, pricing, cursor, identity, and dependent writes through Bun transactions and the shared model registry.
PostgreSQL ownership serialization uses collision-resistant full-digest keys, SQLite pricing uses the canonical Bun writer and pricing contract, and usage-event snapshots order mixed-offset timestamps chronologically. Operational synchronization metadata and backend lifecycle remain adapter-owned. Stack 4 of 5.
generated by a clanker