refactor(storage): complete unified Bun storage cutover - #1347
refactor(storage): complete unified Bun storage cutover#1347mariusvniekerk wants to merge 20 commits into
Conversation
roborev: Combined Review (
|
Canonical pricing rows must carry real timestamps, workflow staging must remain valid before publication, and stamped PostgreSQL schemas must be validated under the migration lock. Keeping these invariants in the foundation prevents later stack layers from inheriting dialect-specific repair paths. Move SQLite pricing refresh state into dedicated metadata, normalize PostgreSQL pricing columns in the existing convergence transaction, and require explicit DuckDB mapping timestamps. Trigger removal remains owned by the final cutover in PR #1347 and kata vzty.
df16698 to
3ee3fdb
Compare
roborev: Combined Review (
|
Canonical pricing rows must carry real timestamps, workflow staging must remain valid before publication, and stamped PostgreSQL schemas must be validated under the migration lock. Keeping these invariants in the foundation prevents later stack layers from inheriting dialect-specific repair paths. Move SQLite pricing refresh state into dedicated metadata, normalize PostgreSQL pricing columns in the existing convergence transaction, and require explicit DuckDB mapping timestamps. Trigger removal remains owned by the final cutover in PR #1347 and kata vzty.
3ee3fdb to
ebb6997
Compare
roborev: Combined Review (
|
ebb6997 to
73c0e3d
Compare
roborev: Combined Review (
|
9917e39 to
0c7127c
Compare
roborev: Combined Review (
|
0c7127c to
73fef1d
Compare
Canonical pricing rows must carry real timestamps, workflow staging must remain valid before publication, and stamped PostgreSQL schemas must be validated under the migration lock. Keeping these invariants in the foundation prevents later stack layers from inheriting dialect-specific repair paths. Move SQLite pricing refresh state into dedicated metadata, normalize PostgreSQL pricing columns in the existing convergence transaction, and require explicit DuckDB mapping timestamps. Trigger removal remains owned by the final cutover in PR #1347 and kata vzty.
roborev: Combined Review (
|
Project identity and worktree metadata have one portable shape, but their mirror publishers still rebuilt identical rows with adapter SQL. PostgreSQL vector documents likewise need Bun-owned row state while keeping dynamic halfvec chunk tables at the dialect boundary.\n\nPublish archive scope, observations, snapshots, and worktree mappings through shared canonical rows. Keep PostgreSQL filtered ownership and vector generation/chunk policy local while moving vector documents and delta state into the same Bun transactions.
Identity and worktree publication must remain complete as canonical models evolve, without reintroducing adapter-specific projections or false PostgreSQL ownership contention. The target write contract also needs executable ownership and retry semantics so partial upgrades cannot appear complete.\n\nDerive replacement projections from Bun models, preserve SQLite worktree CRUD behavior through shared transactions, and bind PostgreSQL repair state to exhaustive column ownership. Cross-engine row contracts and real lock/vector schema invariants now make drift observable while keeping search-specific DDL at its dialect seam.
Parser ingestion and background identity repair still wrote observations and snapshots through a parallel SQLite SQL projection even after mirrors adopted canonical Bun rows. That left the archive able to drift from the schema and replacement semantics shared by PostgreSQL and DuckDB.\n\nRoute live session, batch, and backfill identity publication through the canonical converters and Bun transactions while preserving evidence precedence and atomic progress. Keep only the pre-Bun startup credential scrub as an explicitly migration-owned raw transaction seam.
PostgreSQL and DuckDB inherited identical archive-ingest rejection methods solely to satisfy the Store interface, leaving concrete adapters as redundant common-method owners. Read-only identity was likewise repeated on every wrapper.\n\nMove those defaults onto the embedded BunStore so remote adapters expose only their actual lifecycle and capability responsibilities. SQLite keeps its writable archive methods while sharing the same backend-derived ReadOnly result.
Conversation-unit boundary and extent resolution had separate PostgreSQL and DuckDB implementations even though the shared Bun backend already owns the portable query contract. Keeping those copies made search hydration sensitive to adapter drift and preserved direct driver execution in otherwise common behavior. Promote the guarded Bun implementation to the embedded store and let remote adapters inherit it. SQLite retains its narrowly scoped local search seam while PostgreSQL and DuckDB now share one canonical implementation.
Subagent rollups still depended on three independent ordering, deduplication, and pricing pipelines after activity reports moved to Bun. That left a hidden application query path in each adapter and allowed authoritative session costs to diverge. Make the guarded Bun store the sole owner of the rollup stream and remove the obsolete PostgreSQL and DuckDB usage builders, scanners, and implementation-shaped tests. Explicit canonical usage-event columns also prevent DuckDB batch contents from depending on the first row’s default-valued fields, preserving Copilot reported-cost authority.
Project reclassification could bypass canonical sanitization and model evolution by rebuilding identity aggregates with a manual SQL projection. PostgreSQL session ownership also treated every unknown canonical column as source-owned, allowing schema drift to silently overwrite target curation. Republish latest-winner identity evidence through canonical Bun rows and make PostgreSQL source/target ownership explicitly exhaustive at the push boundary. This keeps existing archive transactions atomic while forcing new session columns to receive an intentional owner.
Moving metadata reconciliation onto a Bun transaction bypassed the raw writer facade that previously rejected read-only and closed stores. A nil Bun writer could panic while holding the connection lock and deadlock cleanup. Restore the lifecycle guard before acquiring the Bun connection so metadata copy preserves the existing read-only and writer-closed contracts.
Common session filtering, sorting, analytics, search scope, and pagination still depended on a custom placeholder and SQL dialect renderer after reads moved to Bun. That preserved a fourth query abstraction and let SQLite/PostgreSQL/DuckDB behavior drift outside the approved adapter seams. Use Bun-native question-mark binding throughout the shared store and retain only the SQLite instant-order transform for shipped text timestamps. PostgreSQL FTS keeps a narrow exported base-scope predicate, while the legacy dialect types, builders, and renderer tests are removed.
The shared Bun backend accepted arbitrary timestamp SQL operands, but its contract described only columns and recent-edits inferred nullable storage semantics from whether an adapter rewrote the expression. That made a valid future adapter capable of silently changing empty-timestamp handling.\n\nNormalize nullable timestamp values before adapter ordering and preserve the writer-barrier and session-filter behavior with focused lifecycle, DST, recursive-scope, and cutoff regressions.
Common storage behavior was still reachable through guarded database/sql facades, concrete SQLite upload shadows, and PostgreSQL replication helpers using driver-native placeholders. Those paths weakened the single-store ownership boundary and could bypass Bun formatting or fail when a Bun handle executed raw PostgreSQL bindings. Make BunStore the sole public db.Store owner, route backend schema and replication operations through Bun transactions, and reserve direct driver access for documented lifecycle, transport, compatibility-probe, and vector seams. Legacy archive fixture setup uses the explicit raw test seam so shipped corrupt-data recovery remains covered without weakening application execution.
The shared Bun cutover exposed parity and performance gaps in analytics, usage, search hydration, timestamp ordering, and conflict-heavy ingestion. Leaving those gaps open would make the unified adapters preserve behavior only at the cost of backend drift or common-workload regressions. Keep canonical scans and reductions in Bun, preserve the shipped SQLite lexical and timestamp semantics at the capability seam, and reuse stable write clauses so SQLite, PostgreSQL, and DuckDB stay within the established performance gates.
Cross-engine review and an identical-input runtime comparison exposed places where the shared Bun cutover could still round aggregates differently, lose the created-at usage fallback, or let a padded Cursor row suppress an in-range event. Literal suffix matching also needed to remain case-sensitive without relying on SQLite LIKE semantics. Make those behaviors explicit in the common store and contracts while documenting that dialect-specific SQL remains valid behind adapter boundaries whenever engine semantics or query plans require it.
The rebased common-schema convergence helper accepts a Bun connection, but the stamped-schema fast path still passed the raw PostgreSQL pool. That broke the top branch build and would bypass the canonical query formatter during startup validation. Reuse the sync store's Bun handle for the stamp probe and locked convergence transaction.
The landing review exposed that the storage guide's query-plan allowance was broad enough to recreate backend-specific common query paths. Define the finite adapter seams and reconcile the design with the search capability shapes the stack actually implements. Record the branch-by-branch cleanup plan so the stack can be restacked with fixes at their introducing layers while the larger incremental archive-writer migration remains separately owned.
The final layer removes the legacy QueryDialect type, so the search capability cleanup must use the backend's surviving timestamp-order seam directly. Keep the replaying read fixture on that same interface and remove the last orphaned PostgreSQL placeholder builder after the dead renderer cleanup.
The cutover changes PostgreSQL common convergence from a raw sql.DB entry point to a Bun handle. Keep the stamped drift and bounded-row-scan regressions on that final interface.
The review plan still showed completed Quack, analytics, restack, and verification work as pending, which made the landing state ambiguous after the branch fixes moved to their owning layers. Record the evidence-backed completion state while leaving publication and RoboRev re-enablement open until the remote stack is updated.
The verified five-branch stack and summary-only PR descriptions are now published with recovery refs retained. Close the remaining delivery checklist so the plan reflects the review-ready remote state and the handoff can return RoboRev reminders to normal.
The DuckDB analytics contract still parses exact dollar fixtures after the stack rebase. Restore its domain import on the layer that introduced those assertions so this PR and every dependent layer compile independently.
73fef1d to
773e891
Compare
roborev: Combined Review (
|
Shared reads and writes still left a legacy query renderer, concrete adapter shadows, and several identity, vector, unit-range, and usage paths outside the Bun ownership boundary. This final layer removes those alternate paths and completes the shared Bun storage cutover.
Bun owns the common schema and public store behavior. Dialect-specific SQL remains allowed behind the documented closed seams for lifecycle, schema convergence, synchronization metadata, timestamp ordering, capability probes, FTS, and vectors. Observable usage dollar values remain unchanged under the cross-backend pricing and usage contracts, providing contract-verified dollar parity rather than relying on a one-off output comparison. Stack 5 of 5.
generated by a clanker