Consolidate archive sync writers on Bun transactions - #1361
Consolidate archive sync writers on Bun transactions#1361mariusvniekerk wants to merge 40 commits into
Conversation
roborev: Combined Review (
|
005ad77 to
a37d0df
Compare
roborev: Combined Review (
|
roborev: Combined Review (
|
935b4e0 to
178a8ff
Compare
roborev: Combined Review (
|
roborev: Combined Review (
|
roborev: Combined Review (
|
roborev: Combined Review (
|
roborev: Combined Review (
|
roborev: Combined Review (
|
roborev: Combined Review (
|
roborev: Combined Review (
|
roborev: Combined Review (
|
roborev: Combined Review (
|
roborev: Combined Review (
|
roborev: Combined Review (
|
roborev: Combined Review (
|
roborev: Combined Review (
|
roborev: Combined Review (
|
roborev: Combined Review (
|
roborev: Combined Review (
|
roborev: Combined Review (
|
roborev: Combined Review (
|
The five-layer cutover still leaves ordinary sync, incremental repair, and orphan-copy projections on independent archive writer paths. Define the sixth stack layer that converges those paths on the existing session-batch and canonical Bun transaction primitives while preserving SQLite-only FTS and recovery seams.
Parse-diff repair must keep stable message identities and untouched tool graphs while the archive writer moves to the shared Bun path. Add a scoped canonical repair primitive so later writer cutovers can replace only affected ordinals without treating a partial tool graph as a full-session replacement.
Standalone secret scans must use the same guarded canonical write path as archive replication. This keeps finding sanitization and nullable coordinates aligned while preserving the session summary and sync marker contract.
The rebased PostgreSQL tests still depended on adapter-private query helpers and non-timestamp pricing markers. Remove obsolete implementation-detail cases and keep the remaining parity fixtures valid against the canonical Bun types. Shared SQLite usage behavior remains authoritative when malformed nested server-tool data cannot be parsed.
PostgreSQL rounds TIMESTAMPTZ values to microseconds while SQLite can retain nanoseconds. Normalize the shared pricing metadata timestamp before it reaches reports so identical rows produce identical output across Bun adapters.
The Bun cutover changed equal-snapshot ranking and materialized the full daily usage projection, producing different usage totals and excessive allocations. Restore explicit timestamp precedence and reduce rows while they stream. SQLite archives must retain malformed provider timestamp text that origin/main accepted, including through atomic sync and parse-diff repair. Keep that compatibility seam inside guarded Bun reads and writes while PostgreSQL and DuckDB canonical conversion stays strict and DuckDB counts unsupported rows as session-local failures. Keep archive batches below SQLite’s historical variable ceiling and wait for populated usage-filter options before exercising bulk selection.
The final Bun cutover exposed insert-only recall reconciliation and legacy malformed-timestamp readers that still depended on stricter typed scans. It also normalized report timestamps with Go midpoint semantics instead of PostgreSQL's persisted half-even behavior. Retain the canonical 100-row archive batching used by current SQLite builds; do not introduce a historical 999-variable compatibility target.
A multi-session source can commit early results before a later member write or final scoped link fails. Queue each committed session itself so deferred repair can restore authoritative incoming-edge parentage instead of only repairing its outgoing children.
Append-only message paths could introduce duplicate evidence endpoints while leaving trusted recall provenance unchanged. Reconcile each affected session inside the message transaction and emit revocation diagnostics only after the write commits.
Bun bulk inserts render an entire model slice as one dialect-aware VALUES statement, so the manual 100-row loops added statement overhead without protecting a driver bind limit. Use Bun's documented slice API throughout canonical writes and keep malformed SQLite timestamps in a one-field archive model extension while strict adapters retain the shared timestamp type.
Secret findings must use the same result-event coordinate that persistence stores or reveal cannot relocate their source. Share the canonical distinct-index and collision-fallback rule between Bun writes and scanning, and cover a non-positional write/scan/reveal round trip.
Late tool results and subagent links change fields included in trusted recall digests even when no message row is appended. Reconcile whenever an incremental write changes transcript content so stale evidence cannot remain trusted.
Bun formats slice inserts into SQL, so unbounded dynamic payloads can create avoidable query and hook working sets even without a driver bind limit. Bound canonical writes by the archive's intentional maximum payload while keeping oversized logical rows intact. SQLite's permissive archive timestamp seam also needs the same canonical availability policy at every shared read boundary. Normalize timing, recent-edit, and content-search hydration, invalidate secret coordinates written before canonical result indexes, and pin PostgreSQL-compatible pricing rounding edges.
The payload estimator runs once per canonical row. Go's reflect.Value.Fields iterator allocates on each struct traversal, doubling bulk-insert allocation counts and pushing archive sync beyond the benchmark gate even though the byte-budget behavior itself was sound. Keep indexed reflection on this hot path and document the targeted modernize suppression so lint cannot rewrite the regression back in.
PostgreSQL rounds TIMESTAMPTZ fractions to microseconds while DuckDB truncates them, so preserving nanoseconds until adapter persistence can move pricing metadata by one microsecond across otherwise identical reports. Apply the existing PostgreSQL half-even rule to canonical pricing rows and revision timestamps before any Bun adapter writes them. Pin the three-backend parity fixture to a fractional timestamp that deterministically exercises the boundary.
SQLite accepts date-only, time-only, and numeric values in julianday even though the canonical Bun timestamp scanner rejects them. Ranking those legacy values before hydration can let an unavailable edit displace a valid file from a bounded page. Use the canonical parser through the SQLite adapter availability seam before recent-edit ranking, while native typed backends keep their existing expression. The regression combines an unsupported future value with a canonical edit so pagination order is observable.
Schema-v11 mirrors can retain timestamps truncated before canonical pricing normalization because timestamp-only differences are intentionally excluded from incremental pricing updates. Advance the disposable mirror boundary so the next push rebuilds and atomically swaps affected files. The upgrade regression simulates the old persisted value and verifies the rebuilt mirror carries the shared half-even microsecond timestamp.
The batch estimator measures static Go row size plus dynamic values before Bun formats SQL. Calling 16 MiB a formatted-query working-set bound overstated what the calculation guarantees because syntax, escaping, and Bun copies add overhead. Describe it consistently as a pre-format payload target while retaining the same byte budget, oversized-row behavior, and transaction boundaries.
Daily usage filtering and Claude snapshot deduplication can discard most raw candidates, so counting every message and usage event first adds two full scans and can reserve memory that the result never uses. Let the accepted result slice grow from the rows that survive streaming. Keep the existing candidate order, filters, and deduplication unchanged.
Removing separate candidate-count scans exposed the normalized usage path to repeated growth and backing-array copies on large archives. Preserve Claude cross-session deduplication with its widened stream, while filtering ordinary messages and usage events in SQL and deriving their allocation capacity from the streaming queries themselves. This keeps the query path unbounded by arbitrary row or bind limits and avoids reintroducing standalone full-table count queries.
Bun formats tuple lists into the SQL statement, so removing the old fixed-row batching left large worktree-mapping publication windows able to build one unbounded query. Apply the canonical pre-format payload budget to tombstone keys so statement growth follows actual string size without reintroducing a guessed row or bind-variable ceiling.
SQLite julianday accepts date-only, time-only, numeric, and some invalid calendar text that the canonical Bun timestamp scanner rejects. Usage projections and bounded queries must classify those message timestamps the same way so they fall back to the session timestamp instead of disappearing or failing during scan. Keep canonical UTC rows on a validated SQLite-native fast path and reserve the exact Go parser UDF for legacy or suspicious text, preserving the usage benchmark allocation profile.
The Windows DuckDB driver converts a scanned time.Time one microsecond below the persisted value, which made the mirror-rebuild contract depend on driver decoding. Assert DuckDB's stored epoch microseconds directly so the test protects canonical pricing precision on every platform.
Timing responses must use the same persisted message IDs as message reads so frontend turn and tool-call joins remain valid. Malformed provider timestamps are repaired at ingestion and during the v84 archive rebuild instead of being tolerated through backend-specific read or fingerprint fallbacks. This keeps SQLite, PostgreSQL, and DuckDB on one strict canonical timestamp contract.
The Bun writer cutover made ID-less parser messages escape and allocated fresh sanitation and canonical-row backing for every session. That pushed cold sync bytes and contributor rebuild allocations over the benchmark gate despite unchanged behavior. Keep per-database schema creation and validation intact. Format canonical timestamps directly into Bun queries, separate target-assigned message conversion from source-ID preservation, and recycle only cleared fixed-row backing bounded by the existing payload budget.
PostgreSQL shortened UTC offsets when shared queries cast canonical timestamps to text, which made strict scans reject valid rows. Keep real timestamp columns typed and type only PostgreSQL's null-only union branches. The allocation-free timestamp appender also cannot receive a typed nil pointer through a direct Bun query argument. Convert that nullable policy value to SQL NULL at the call boundary while preserving value-based timestamp formatting.
DuckDB on Windows can round an RFC3339 string cast one microsecond below the canonical value. That made rebuilt mirrors lose the PostgreSQL-compatible pricing revision even though the shared model had already normalized it. Let DuckDB format canonical Bun timestamps with its typed microsecond literal. Keep SQLite and PostgreSQL on the existing RFC3339 path so archive text remains unchanged.
The data-mode workflow can finish its mapping mutation before the follow-up project inventory reaches the page under CI load. Await both successful responses so the DOM assertion observes the completed user workflow instead of racing the refresh.
Analytics and search planners need one shared relational shape while SQLite, PostgreSQL, and DuckDB retain only the SQL expressions and native sources that differ. Define compositional fragments so Bun keeps query construction and execution ownership, including paired model turns, direct tool facts, timezone projection, lexical ranking, and vector distance/setup seams.
Shared analytics filters must retain the indexed UTC prefilter before exact local-date matching, and usage SQL must use the same activity cutoff as its final reducer. Make the analytics date grain explicit, pass one captured reference time through usage queries, and keep FTS-dependent contract assertions conditional when the optional SQLite module is not compiled.
e02f880 to
0ee0579
Compare
roborev: Combined Review (
|
Ordinary sync, full-session replacement, incremental repair, dependent accounting writes, and orphan recovery now share the canonical Bun transaction core. This removes staged commits that could expose mixed session state after a late failure and eliminates hand-maintained common child projections during archive recovery.
SQLite still owns the operations that genuinely depend on its adapter: FTS5 maintenance, pins and physical-ID remapping, ATTACH and temporary-table lifecycle, provenance repair, sanitization, and legacy capability probes. Transcript comparison remains proportional to session history, while changed-row persistence and FTS mutation stay scoped to the affected diff.
Review the atomic session-batch routing, canonical finding helper, and registry-derived orphan-copy boundary.
make test-shortremains affected only by the existing macOS FSEvents delivery timeouts; focused sync, full database, DuckDB, and PostgreSQL/activity suites pass.generated by a clanker