Skip to content

refactor(storage): complete unified Bun storage cutover - #1347

Open
mariusvniekerk wants to merge 20 commits into
t3code/bun-write-unificationfrom
t3code/unify-databases-with-bun-duckdb
Open

refactor(storage): complete unified Bun storage cutover#1347
mariusvniekerk wants to merge 20 commits into
t3code/bun-write-unificationfrom
t3code/unify-databases-with-bun-duckdb

Conversation

@mariusvniekerk

@mariusvniekerk mariusvniekerk commented Aug 5, 2026

Copy link
Copy Markdown
Collaborator

Shared reads and writes still left a legacy query renderer, concrete adapter shadows, and several identity, vector, unit-range, and usage paths outside the Bun ownership boundary. This final layer removes those alternate paths and completes the shared Bun storage cutover.

Bun owns the common schema and public store behavior. Dialect-specific SQL remains allowed behind the documented closed seams for lifecycle, schema convergence, synchronization metadata, timestamp ordering, capability probes, FTS, and vectors. Observable usage dollar values remain unchanged under the cross-backend pricing and usage contracts, providing contract-verified dollar parity rather than relying on a one-off output comparison. Stack 5 of 5.

generated by a clanker

@roborev-ci

roborev-ci Bot commented Aug 5, 2026

Copy link
Copy Markdown

roborev: Combined Review (df16698)

Code review passed: no Medium, High, or Critical findings were reported.


Reviewers: 2 done | Synthesis: codex, 6s | Total: 33m13s

mariusvniekerk added a commit that referenced this pull request Aug 5, 2026
Canonical pricing rows must carry real timestamps, workflow staging must remain valid before publication, and stamped PostgreSQL schemas must be validated under the migration lock. Keeping these invariants in the foundation prevents later stack layers from inheriting dialect-specific repair paths.

Move SQLite pricing refresh state into dedicated metadata, normalize PostgreSQL pricing columns in the existing convergence transaction, and require explicit DuckDB mapping timestamps. Trigger removal remains owned by the final cutover in PR #1347 and kata vzty.
@mariusvniekerk
mariusvniekerk force-pushed the t3code/unify-databases-with-bun-duckdb branch from df16698 to 3ee3fdb Compare August 7, 2026 03:46
@roborev-ci

roborev-ci Bot commented Aug 7, 2026

Copy link
Copy Markdown

roborev: Combined Review (3ee3fdb)

The refactor is generally sound, but one medium-severity SQLite date-bucketing regression needs correction.

Medium

  • internal/db/bun_analytics.go:490 — The aggregate summary derives UTC dates with SUBSTR(..., 1, 10). Stored RFC3339 timestamps may include offsets, so 2026-08-03T23:30:00-05:00 is bucketed as August 3 instead of August 4 UTC. This can produce incorrect date-filtered totals and ActiveDays. Normalize timestamps before extracting the date (for example, with strftime('%Y-%m-%d', ...)) or disable this aggregate path for SQLite, and add a behavioral test covering an offset timestamp that crosses UTC midnight.

Reviewers: 2 done | Synthesis: codex, 9s | Total: 26m8s

mariusvniekerk added a commit that referenced this pull request Aug 7, 2026
Canonical pricing rows must carry real timestamps, workflow staging must remain valid before publication, and stamped PostgreSQL schemas must be validated under the migration lock. Keeping these invariants in the foundation prevents later stack layers from inheriting dialect-specific repair paths.

Move SQLite pricing refresh state into dedicated metadata, normalize PostgreSQL pricing columns in the existing convergence transaction, and require explicit DuckDB mapping timestamps. Trigger removal remains owned by the final cutover in PR #1347 and kata vzty.
@mariusvniekerk
mariusvniekerk force-pushed the t3code/unify-databases-with-bun-duckdb branch from 3ee3fdb to ebb6997 Compare August 7, 2026 14:26
@roborev-ci

roborev-ci Bot commented Aug 7, 2026

Copy link
Copy Markdown

roborev: Combined Review (ebb6997)

No Medium, High, or Critical findings; the reviewers found no actionable issues at the requested severity threshold.


Reviewers: 2 done | Synthesis: codex, 5s | Total: 42m24s

@mariusvniekerk
mariusvniekerk force-pushed the t3code/unify-databases-with-bun-duckdb branch from ebb6997 to 73c0e3d Compare August 7, 2026 16:34
@roborev-ci

roborev-ci Bot commented Aug 7, 2026

Copy link
Copy Markdown

roborev: Combined Review (9917e39)

Medium

  • internal/db/bun_activity_report.go:108 — Activity reports load all matching sessions and complete message histories before applying the requested date range. Even a one-day report may scale with the entire archive, causing severe latency or memory exhaustion. Apply activity-window predicates in the session query, then restrict message and usage queries to the candidate session IDs.

Reviewers: 2 done | Synthesis: codex, 8s | Total: 30m14s

@mariusvniekerk
mariusvniekerk force-pushed the t3code/unify-databases-with-bun-duckdb branch from 9917e39 to 0c7127c Compare August 9, 2026 03:46
@roborev-ci

roborev-ci Bot commented Aug 9, 2026

Copy link
Copy Markdown

roborev: Combined Review (0c7127c)

The storage refactor is generally sound, but two medium-severity correctness and performance issues should be addressed.

Medium

  • UTC date handling can misclassify sessionsinternal/db/bun_analytics.go:493

    The analytics fast path derives dates from the first ten timestamp characters. RFC3339 timestamps with offsets can cross a UTC date boundary, causing ActiveDays and From/To filtering to disagree with the previous parsed-time behavior.

    Fix: Normalize timestamps to UTC before extracting the date with backend-specific SQL, or retain the parsed Go fallback for SQLite. Add coverage for an offset timestamp that crosses midnight UTC.

  • Identity reconciliation introduces an unnecessary cross-joininternal/db/project_identity.go:1262

    Model(&winner) already supplies the model table, while TableExpr(...) adds a second snapshots-table instance. The resulting unused self-cross-join can substantially multiply reconciliation work.

    Fix: Replace TableExpr(...) with ModelTableExpr("source_session_project_identity_snapshots AS snap") so the existing model table is aliased instead of duplicated.


Reviewers: 2 done | Synthesis: codex, 12s | Total: 34m30s

@mariusvniekerk
mariusvniekerk force-pushed the t3code/unify-databases-with-bun-duckdb branch from 0c7127c to 73fef1d Compare August 10, 2026 15:39
mariusvniekerk added a commit that referenced this pull request Aug 10, 2026
Canonical pricing rows must carry real timestamps, workflow staging must remain valid before publication, and stamped PostgreSQL schemas must be validated under the migration lock. Keeping these invariants in the foundation prevents later stack layers from inheriting dialect-specific repair paths.

Move SQLite pricing refresh state into dedicated metadata, normalize PostgreSQL pricing columns in the existing convergence transaction, and require explicit DuckDB mapping timestamps. Trigger removal remains owned by the final cutover in PR #1347 and kata vzty.
@roborev-ci

roborev-ci Bot commented Aug 10, 2026

Copy link
Copy Markdown

roborev: Combined Review (73fef1d)

Changes need revision: three medium-severity issues were identified.

Medium

  • internal/postgres/schema.go:1576 — Unstamped PostgreSQL convergence validates canonical foreign-key parents before provenance repair. Sessions with blank source_archive_id fail EnsureSchema, preventing the subsequent supported provenance backfill.

    • Fix: Repair provenance before validation or exclude incomplete session provenance from this invariant. Add migration coverage for an unstamped schema containing a blank-provenance session.
  • internal/db/bun_analytics.go:492 — The aggregate summary derives a UTC date by slicing the first ten characters from stored timestamps. Valid RFC3339 timestamps with non-UTC offsets that cross UTC midnight can be assigned to the wrong day and incorrectly included or excluded by From/To.

    • Fix: Convert timestamps to UTC before extracting the date, or use the parsed-row fallback for SQLite text timestamps. Add a regression test covering an offset that crosses UTC midnight.
  • internal/db/bun_activity_report.go:174 — Activity-report range filtering occurs only after all matching sessions and messages are materialized; the usage query at line 298 also loads every usage row in the date window. Narrow reports can therefore scale with the entire archive and consume excessive memory.

    • Fix: Apply session-overlap filtering in the initial query, hydrate messages only for candidate session IDs, and load usage for those candidates plus only the cross-session snapshot peers required for deduplication.

Reviewers: 2 done | Synthesis: codex, 14s | Total: 31m16s

Project identity and worktree metadata have one portable shape, but their mirror publishers still rebuilt identical rows with adapter SQL. PostgreSQL vector documents likewise need Bun-owned row state while keeping dynamic halfvec chunk tables at the dialect boundary.\n\nPublish archive scope, observations, snapshots, and worktree mappings through shared canonical rows. Keep PostgreSQL filtered ownership and vector generation/chunk policy local while moving vector documents and delta state into the same Bun transactions.
Identity and worktree publication must remain complete as canonical models evolve, without reintroducing adapter-specific projections or false PostgreSQL ownership contention. The target write contract also needs executable ownership and retry semantics so partial upgrades cannot appear complete.\n\nDerive replacement projections from Bun models, preserve SQLite worktree CRUD behavior through shared transactions, and bind PostgreSQL repair state to exhaustive column ownership. Cross-engine row contracts and real lock/vector schema invariants now make drift observable while keeping search-specific DDL at its dialect seam.
Parser ingestion and background identity repair still wrote observations and snapshots through a parallel SQLite SQL projection even after mirrors adopted canonical Bun rows. That left the archive able to drift from the schema and replacement semantics shared by PostgreSQL and DuckDB.\n\nRoute live session, batch, and backfill identity publication through the canonical converters and Bun transactions while preserving evidence precedence and atomic progress. Keep only the pre-Bun startup credential scrub as an explicitly migration-owned raw transaction seam.
PostgreSQL and DuckDB inherited identical archive-ingest rejection methods solely to satisfy the Store interface, leaving concrete adapters as redundant common-method owners. Read-only identity was likewise repeated on every wrapper.\n\nMove those defaults onto the embedded BunStore so remote adapters expose only their actual lifecycle and capability responsibilities. SQLite keeps its writable archive methods while sharing the same backend-derived ReadOnly result.
Conversation-unit boundary and extent resolution had separate PostgreSQL and DuckDB implementations even though the shared Bun backend already owns the portable query contract. Keeping those copies made search hydration sensitive to adapter drift and preserved direct driver execution in otherwise common behavior.

Promote the guarded Bun implementation to the embedded store and let remote adapters inherit it. SQLite retains its narrowly scoped local search seam while PostgreSQL and DuckDB now share one canonical implementation.
Subagent rollups still depended on three independent ordering, deduplication, and pricing pipelines after activity reports moved to Bun. That left a hidden application query path in each adapter and allowed authoritative session costs to diverge.

Make the guarded Bun store the sole owner of the rollup stream and remove the obsolete PostgreSQL and DuckDB usage builders, scanners, and implementation-shaped tests. Explicit canonical usage-event columns also prevent DuckDB batch contents from depending on the first row’s default-valued fields, preserving Copilot reported-cost authority.
Project reclassification could bypass canonical sanitization and model evolution by rebuilding identity aggregates with a manual SQL projection. PostgreSQL session ownership also treated every unknown canonical column as source-owned, allowing schema drift to silently overwrite target curation.

Republish latest-winner identity evidence through canonical Bun rows and make PostgreSQL source/target ownership explicitly exhaustive at the push boundary. This keeps existing archive transactions atomic while forcing new session columns to receive an intentional owner.
Moving metadata reconciliation onto a Bun transaction bypassed the raw writer facade that previously rejected read-only and closed stores. A nil Bun writer could panic while holding the connection lock and deadlock cleanup.

Restore the lifecycle guard before acquiring the Bun connection so metadata copy preserves the existing read-only and writer-closed contracts.
Common session filtering, sorting, analytics, search scope, and pagination still depended on a custom placeholder and SQL dialect renderer after reads moved to Bun. That preserved a fourth query abstraction and let SQLite/PostgreSQL/DuckDB behavior drift outside the approved adapter seams.

Use Bun-native question-mark binding throughout the shared store and retain only the SQLite instant-order transform for shipped text timestamps. PostgreSQL FTS keeps a narrow exported base-scope predicate, while the legacy dialect types, builders, and renderer tests are removed.
The shared Bun backend accepted arbitrary timestamp SQL operands, but its contract described only columns and recent-edits inferred nullable storage semantics from whether an adapter rewrote the expression. That made a valid future adapter capable of silently changing empty-timestamp handling.\n\nNormalize nullable timestamp values before adapter ordering and preserve the writer-barrier and session-filter behavior with focused lifecycle, DST, recursive-scope, and cutoff regressions.
Common storage behavior was still reachable through guarded database/sql facades, concrete SQLite upload shadows, and PostgreSQL replication helpers using driver-native placeholders. Those paths weakened the single-store ownership boundary and could bypass Bun formatting or fail when a Bun handle executed raw PostgreSQL bindings.

Make BunStore the sole public db.Store owner, route backend schema and replication operations through Bun transactions, and reserve direct driver access for documented lifecycle, transport, compatibility-probe, and vector seams. Legacy archive fixture setup uses the explicit raw test seam so shipped corrupt-data recovery remains covered without weakening application execution.
The shared Bun cutover exposed parity and performance gaps in analytics, usage, search hydration, timestamp ordering, and conflict-heavy ingestion. Leaving those gaps open would make the unified adapters preserve behavior only at the cost of backend drift or common-workload regressions.

Keep canonical scans and reductions in Bun, preserve the shipped SQLite lexical and timestamp semantics at the capability seam, and reuse stable write clauses so SQLite, PostgreSQL, and DuckDB stay within the established performance gates.
Cross-engine review and an identical-input runtime comparison exposed places where the shared Bun cutover could still round aggregates differently, lose the created-at usage fallback, or let a padded Cursor row suppress an in-range event. Literal suffix matching also needed to remain case-sensitive without relying on SQLite LIKE semantics.

Make those behaviors explicit in the common store and contracts while documenting that dialect-specific SQL remains valid behind adapter boundaries whenever engine semantics or query plans require it.
The rebased common-schema convergence helper accepts a Bun connection, but the stamped-schema fast path still passed the raw PostgreSQL pool. That broke the top branch build and would bypass the canonical query formatter during startup validation.

Reuse the sync store's Bun handle for the stamp probe and locked convergence transaction.
The landing review exposed that the storage guide's query-plan allowance was broad enough to recreate backend-specific common query paths. Define the finite adapter seams and reconcile the design with the search capability shapes the stack actually implements.

Record the branch-by-branch cleanup plan so the stack can be restacked with fixes at their introducing layers while the larger incremental archive-writer migration remains separately owned.
The final layer removes the legacy QueryDialect type, so the search capability cleanup must use the backend's surviving timestamp-order seam directly. Keep the replaying read fixture on that same interface and remove the last orphaned PostgreSQL placeholder builder after the dead renderer cleanup.
The cutover changes PostgreSQL common convergence from a raw sql.DB entry point to a Bun handle. Keep the stamped drift and bounded-row-scan regressions on that final interface.
The review plan still showed completed Quack, analytics, restack, and verification work as pending, which made the landing state ambiguous after the branch fixes moved to their owning layers. Record the evidence-backed completion state while leaving publication and RoboRev re-enablement open until the remote stack is updated.
The verified five-branch stack and summary-only PR descriptions are now published with recovery refs retained. Close the remaining delivery checklist so the plan reflects the review-ready remote state and the handoff can return RoboRev reminders to normal.
The DuckDB analytics contract still parses exact dollar fixtures after the stack rebase. Restore its domain import on the layer that introduced those assertions so this PR and every dependent layer compile independently.
@mariusvniekerk
mariusvniekerk force-pushed the t3code/unify-databases-with-bun-duckdb branch from 73fef1d to 773e891 Compare August 13, 2026 13:26
@roborev-ci

roborev-ci Bot commented Aug 13, 2026

Copy link
Copy Markdown

roborev: Combined Review (773e891)

The PR has one medium-severity correctness issue in UTC analytics bucketing.

Medium

  • internal/db/bun_analytics.go:492 — UTC day bucketing takes the first ten characters of stored timestamps instead of converting each instant to UTC. Existing SQLite RFC3339 timestamps with offsets can therefore be assigned to the wrong day; for example, 2024-06-01T23:30:00-05:00 is treated as June 1 rather than June 2.
    • Fix: Use a backend-specific UTC date expression such as SQLite strftime('%Y-%m-%d', ...), or retain the normalized Go path for SQLite. Add coverage for an offset timestamp crossing a UTC day boundary.

Reviewers: 2 done | Synthesis: codex, 10s | Total: 24m51s

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Development

Successfully merging this pull request may close these issues.

1 participant