Skip to content

feat(desktop): B1 transactional agent store - #4567

Open
wpfleger96 wants to merge 5 commits into
mainfrom
duncan/b1-transactional-agent-store
Open

feat(desktop): B1 transactional agent store#4567
wpfleger96 wants to merge 5 commits into
mainfrom
duncan/b1-transactional-agent-store

Conversation

@wpfleger96

@wpfleger96 wpfleger96 commented Aug 3, 2026

Copy link
Copy Markdown
Member

Replaces the autocommit+sequential-rename agent store with a substrate that makes transactional claims true end-to-end.

What changed

Commit protocol (store_journal/txn.rs)

mutate_store is the only write entry point. It owns the full lock sequence:

  1. In-process mutex → OS advisory lock (flock / named mutex) → fresh fail-closed decode
  2. BEGIN SQLite transaction — all journal mutations plus file serialization run inside
  3. Stage both JSON files with fsync (serialization failure rolls back)
  4. Insert file_commit_phases intent row inside the transaction (commit-unique stage paths)
  5. COMMIT
  6. Rename managed-agents.<commit_id>.stage → record first_renamed → rename teams.<commit_id>.stage → record committed
  7. Release advisory lock

A closure returning Err rolls back the SQLite transaction — no journal row is written, no file is staged.

Stage paths are commit-unique (managed-agents.<uuid>.stage) so two concurrent mutate_store calls or a recovery-vs-mutation race cannot overwrite each other's temp files.

Boot recovery

run_file_commit_recovery runs synchronously under the anchored advisory lock during app setup, before any commands are admitted. This ensures the JSON store is in a consistent state before the first write arrives.

run_boot_recovery_at (called from spawn_boot_recovery) performs two phases: file-commit recovery (under advisory lock) then publication recovery (outbox re-drive into the active scoped retention DB).

Recovery uses the active scoped retention DB (anchor/retention/<sha256(relay,owner)>.db) resolved via active_retention_scope — the same database the flush loop drains. The prior flat anchor/retention.db path did not exist in production and silently nullified publication re-drive.

File recovery is fail-closed on ambiguous missing stages (intent phase, agents-absent teams-present) rather than treating absence as proof of success.

Publication authority (prepare_publication)

Single function that establishes durable evidence in a recoverable ordered protocol across two SQLite connections (journal then retention). IdentityCollisionErr. Wired for all production publication paths: agent retain/tombstone/archive, team retain/tombstone, persona pending/tombstone.

Tombstone and archive helpers (tombstone_managed_agent_pending, archive_managed_agent_pending, tombstone_team_pending, tombstone_persona_pending) return Result<(), String> — callers log errors rather than swallowing them.

Keyring chokepoint restored

persist_agent_keys_pub is called on a save-local record clone before mutate_agent_store for all creates and imports (agents.rs, personas/snapshot/import.rs). The nsec is stripped after keyring write+verify; the closure only receives a cleared record. Inline fallback preserved when keyring unavailable. Team snapshot already had this pattern.

Fail-closed event-sync / reconcile

migrate_personas_to_events and reconcile_agents_to_events resolve the advisory lock and file paths from store_anchor_dir(). Lock-acquire failure returns early.

Retention DB migration under advisory lock

active_retention_scope migrates the old per-process DB to the anchored path using the SQLite backup API under the advisory lock. Fail-closed.

Windows abandoned-mutex handling

WaitForSingleObject accepts WAIT_OBJECT_0 and WAIT_ABANDONED; rejects WAIT_FAILED with GetLastError context.

Generation intents at identity mutations

cas_generation wired for agent create/update/delete, team create/update/delete, agent settings, and team snapshot import. CasOutcome::Tombstoned and Conflict matched — create rejects tombstoned keys.

Boundary

All changes are within desktop/src-tauri/**. No mobile, frontend, or relay files.

Gates

  • just desktop-tauri-clippy — zero errors, zero warnings (-D warnings)
  • just desktop-tauri-test — 2207 passed, 0 failed
  • just desktop-check — no ratchet violations
  • cargo test --workspace (test target) — 0 warnings

Deferred to B1.1

Three items explicitly out of scope per Will's ruling (2026-08-04), tracked as B1.1 follow-up — not separate issues:

Keyring compensation protocol (CRIT 4 residual). The restore-on-crash protocol for overwritten agent keys (persist_agent_keys_journaled / pin_compensation) is not wired in production. Delete is generation-fenced via tombstone_key; what's missing is the two-phase lock-release/reacquire mechanism that would restore an old key value if the app crashed mid-overwrite. Deferred because: (a) it's substantial new surface, (b) if agent nsecs move to relay aggregates under Wes's relay-only migration plan, the local keyring-compensation protocol may shrink or disappear.

Lifecycle-only CAS (IMP residual). Start/stop/restore/runtime/shutdown mutations do not call cas_generation. These paths do not write new identity records and are already serialized by the advisory lock. Accepted as a documented design position: CAS guards identity-key ownership; the lock prevents lost updates on lifecycle writes.

Bulk boot migrations + outbox-join flusher (CRIT 2 residual). Boot event-sync/reconcile bulk writers (event_sync.rs, reconcile.rs, team_snapshot.rs snapshot-import path) still write retention directly without outbox evidence. The background flush loop still drains the retention table rather than validating a 1:1 join against outbox rows. Both require design work and belong in a dedicated B1.1 pass.

@wpfleger96
wpfleger96 requested a review from a team as a code owner August 3, 2026 17:45
@wpfleger96 wpfleger96 changed the title feat(desktop): B1 transactional agent store substrate feat(desktop): B1 transactional agent store Aug 3, 2026
@wpfleger96
wpfleger96 force-pushed the duncan/b1-transactional-agent-store branch 3 times, most recently from 8ae5eae to 2badedb Compare August 4, 2026 18:41
npub1mn7jgtj4w2pd0g0zeuhxsa6jy6p0rewxz4kujt98my82ahfmp72sxjexk7 and others added 3 commits August 4, 2026 15:37
Co-authored-by: Will Pfleger <pfleger.will@gmail.com>
Signed-off-by: Will Pfleger <pfleger.will@gmail.com>
…opagation, tests

True git rebase onto bc9e652 (current origin/main). Conflict resolution
preserves #3637 (spawn_snapshot, restart-diff) and #4522 (media_raw, deferred
uploads) alongside all B1 content.

Fix 3b (hash-verified file recovery):
- schema.rs: add agents_content_hash / teams_content_hash columns to
  file_commit_phases (schema v3 with ALTER TABLE migration for existing DBs).
- txn.rs: compute SHA-256 of staged payloads and record them in the intent row.
- Recovery now verifies canonical file hash against recorded hash before
  treating absent stage files as 'already completed'. All three missing-stage
  branches (intent/both-absent, intent/teams-absent, first_renamed/teams-absent)
  fail closed unless canonical matches. Only hash-verified canonicals advance
  to committed.

Fix 4 (tombstone/archive propagation):
- agents.rs: tombstone_managed_agent_pending and archive_managed_agent_pending
  use .map_err(|e| ...)?  — failures surface in the delete command's Result.
- personas/mod.rs: same pattern for cascaded agent tombstones and persona tombstone.
- teams.rs: tombstone_team_pending and cascaded persona tombstones propagate.
- agents_retain.rs: retain_managed_agent_pending keeps its outer log-and-swallow
  with an explicit comment documenting the boot-reconcile-recoverable contract
  (Thufir accepted this distinction for updates vs deletes).

Tests:
- store_journal_fix_tests.rs (5 new tests):
  - test_file_recovery_rename_done_phase_not_updated_succeeds: both stages absent,
    canonicals match hashes → advance to committed.
  - test_file_recovery_hash_mismatch_fails_closed: hash mismatch → uncommitted.
  - test_file_recovery_first_renamed_teams_done_hash_verified: teams hash verified.
  - test_file_recovery_first_renamed_teams_absent_no_hash_fails_closed: fail closed.
  - test_boot_recovery_inserts_into_supplied_retention_path_not_flat: scoped path.
- storage_tests.rs (2 new tests): keyring round-trip (Fix 2 coverage).

Co-authored-by: Will Pfleger <pfleger.will@gmail.com>
Signed-off-by: Will Pfleger <pfleger.will@gmail.com>
…inate/race/failure tests

BLOCKING 1: ensure_retention_row_for_payload was re-deriving the retention
d_tag from the event's d-tag field, which is absent for kind-5 tombstones
(they carry only an a-tag) and kind-9035 archives (p-tag only). Two distinct
tombstones for different agents both derived d_tag="" and the second
retention upsert silently clobbered the first.

Fix: add retention_d_tag column to outbox_events (schema v4, ALTER TABLE
migration). insert_outbox_event accepts the d_tag at enqueue time;
prepare_publication passes it through. ensure_retention_row_for_payload
uses the persisted column value directly — no per-kind tag parsing.

BLOCKING 2 (test gaps):
- test_boot_recovery_tombstone_redrives_at_correct_retention_coordinate:
  two distinct agent tombstones, verifies both survive re-drive without
  collision and that each row's d_tag equals the synthetic key used at
  enqueue time (not empty).
- test_boot_recovery_archive_redrives_at_correct_retention_coordinate:
  kind-9035 archive, verifies d_tag = agent_pubkey after re-drive.
- test_file_commit_recovery_is_serialized_by_advisory_lock: spawns two
  threads competing for JournalLockGuard, proves the second blocks until
  the first releases (advisory-lock serialization of recovery vs mutations).
- test_prepare_publication_propagates_retention_write_failure: schemaless
  retention connection, verifies prepare_publication returns Err and that
  the outbox row was still written (journal-first ordering). Stated limit:
  command-path propagation requires AppHandle, covered by ?-propagation
  already in agents.rs:1315,1320 / personas/mod.rs:283-288 / teams.rs:324.
- test_prepare_publication_propagates_journal_write_failure: schemaless
  journal connection, verifies Err propagation and that retention DB is
  untouched (journal-first ordering enforced).

Gates at this commit: just desktop-tauri-clippy clean, just desktop-tauri-test
2254/0, just desktop-check ratchet clean, boundary diff empty.

Co-authored-by: Will Pfleger <pfleger.will@gmail.com>
Signed-off-by: Will Pfleger <pfleger.will@gmail.com>
@wpfleger96
wpfleger96 force-pushed the duncan/b1-transactional-agent-store branch from 2badedb to 78c977e Compare August 4, 2026 19:41
npub1mn7jgtj4w2pd0g0zeuhxsa6jy6p0rewxz4kujt98my82ahfmp72sxjexk7 and others added 2 commits August 4, 2026 16:39
…tion

Fix A: single-owner file recovery ordering and seam test.
- lib.rs: run_file_commit_recovery() now runs BEFORE spawn_boot_recovery()
  is spawned, establishing synchronous pre-admission file repair.
- txn.rs: run_boot_recovery_at() (the background publication task) no longer
  calls recover_interrupted_file_commits — it is publication-only. File repair
  has exactly one owner: the synchronous run_file_commit_recovery call.
- txn.rs: add file_commit_recovery_at_pub() (cfg(test)) for testing the sync
  path without AppHandle.
- Two seam tests in store_journal_fix_tests.rs:
  test_background_recovery_does_not_repair_files: proves run_boot_recovery_at
  leaves stage files and phase rows untouched.
  test_sync_file_recovery_repairs_dangling_intent_row: proves
  file_commit_recovery_at_pub renames stages and advances phase to committed.

Fix B: verifiable schema migration.
- schema.rs: replace let _ = conn.execute_batch() with PRAGMA table_info-based
  column inspection, ADD COLUMN only for missing columns, errors propagated,
  user_version stamped inside the same transaction as the migration. Post-commit
  verification ensures required columns are present.
- New table_column_names() helper for column introspection.
- Three migration tests in store_journal_fix_tests.rs:
  test_schema_migration_from_v2_adds_all_columns: v2 DB gets all three columns.
  test_schema_migration_from_partial_v3_adds_missing_d_tag: partial v3 gets d-tag.
  test_schema_migration_idempotent_on_v4: re-running on v4 is a no-op.

Also update existing file-commit tests to use file_commit_recovery_at_pub
(store_journal_tests.rs crash test, store_journal_fix_tests.rs hash-verify tests)
since run_boot_recovery_at no longer performs file repair.

Co-authored-by: Will Pfleger <pfleger.will@gmail.com>
Signed-off-by: Will Pfleger <pfleger.will@gmail.com>
Apply PRAGMA query_only=ON on a v2 in-memory connection before calling
apply_journal_schema_pub: ALTER TABLE fails, the transaction rolls back,
and user_version stays at 2. Disabling query_only and retrying succeeds
with all three columns present and version 4 -- proving the retry path.

Also corrects the false doc comment on test_schema_migration_idempotent_on_v4
(was incorrectly claiming in-memory DBs cannot be made read-only via
query_only; restated as a straightforward idempotency check).

Co-authored-by: Will Pfleger <pfleger.will@gmail.com>
Signed-off-by: Will Pfleger <pfleger.will@gmail.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant