feat(desktop): B1 transactional agent store - #4567
Open
wpfleger96 wants to merge 5 commits into
Open
Conversation
wpfleger96
force-pushed
the
duncan/b1-transactional-agent-store
branch
3 times, most recently
from
August 4, 2026 18:41
8ae5eae to
2badedb
Compare
Co-authored-by: Will Pfleger <pfleger.will@gmail.com> Signed-off-by: Will Pfleger <pfleger.will@gmail.com>
…opagation, tests True git rebase onto bc9e652 (current origin/main). Conflict resolution preserves #3637 (spawn_snapshot, restart-diff) and #4522 (media_raw, deferred uploads) alongside all B1 content. Fix 3b (hash-verified file recovery): - schema.rs: add agents_content_hash / teams_content_hash columns to file_commit_phases (schema v3 with ALTER TABLE migration for existing DBs). - txn.rs: compute SHA-256 of staged payloads and record them in the intent row. - Recovery now verifies canonical file hash against recorded hash before treating absent stage files as 'already completed'. All three missing-stage branches (intent/both-absent, intent/teams-absent, first_renamed/teams-absent) fail closed unless canonical matches. Only hash-verified canonicals advance to committed. Fix 4 (tombstone/archive propagation): - agents.rs: tombstone_managed_agent_pending and archive_managed_agent_pending use .map_err(|e| ...)? — failures surface in the delete command's Result. - personas/mod.rs: same pattern for cascaded agent tombstones and persona tombstone. - teams.rs: tombstone_team_pending and cascaded persona tombstones propagate. - agents_retain.rs: retain_managed_agent_pending keeps its outer log-and-swallow with an explicit comment documenting the boot-reconcile-recoverable contract (Thufir accepted this distinction for updates vs deletes). Tests: - store_journal_fix_tests.rs (5 new tests): - test_file_recovery_rename_done_phase_not_updated_succeeds: both stages absent, canonicals match hashes → advance to committed. - test_file_recovery_hash_mismatch_fails_closed: hash mismatch → uncommitted. - test_file_recovery_first_renamed_teams_done_hash_verified: teams hash verified. - test_file_recovery_first_renamed_teams_absent_no_hash_fails_closed: fail closed. - test_boot_recovery_inserts_into_supplied_retention_path_not_flat: scoped path. - storage_tests.rs (2 new tests): keyring round-trip (Fix 2 coverage). Co-authored-by: Will Pfleger <pfleger.will@gmail.com> Signed-off-by: Will Pfleger <pfleger.will@gmail.com>
…inate/race/failure tests BLOCKING 1: ensure_retention_row_for_payload was re-deriving the retention d_tag from the event's d-tag field, which is absent for kind-5 tombstones (they carry only an a-tag) and kind-9035 archives (p-tag only). Two distinct tombstones for different agents both derived d_tag="" and the second retention upsert silently clobbered the first. Fix: add retention_d_tag column to outbox_events (schema v4, ALTER TABLE migration). insert_outbox_event accepts the d_tag at enqueue time; prepare_publication passes it through. ensure_retention_row_for_payload uses the persisted column value directly — no per-kind tag parsing. BLOCKING 2 (test gaps): - test_boot_recovery_tombstone_redrives_at_correct_retention_coordinate: two distinct agent tombstones, verifies both survive re-drive without collision and that each row's d_tag equals the synthetic key used at enqueue time (not empty). - test_boot_recovery_archive_redrives_at_correct_retention_coordinate: kind-9035 archive, verifies d_tag = agent_pubkey after re-drive. - test_file_commit_recovery_is_serialized_by_advisory_lock: spawns two threads competing for JournalLockGuard, proves the second blocks until the first releases (advisory-lock serialization of recovery vs mutations). - test_prepare_publication_propagates_retention_write_failure: schemaless retention connection, verifies prepare_publication returns Err and that the outbox row was still written (journal-first ordering). Stated limit: command-path propagation requires AppHandle, covered by ?-propagation already in agents.rs:1315,1320 / personas/mod.rs:283-288 / teams.rs:324. - test_prepare_publication_propagates_journal_write_failure: schemaless journal connection, verifies Err propagation and that retention DB is untouched (journal-first ordering enforced). Gates at this commit: just desktop-tauri-clippy clean, just desktop-tauri-test 2254/0, just desktop-check ratchet clean, boundary diff empty. Co-authored-by: Will Pfleger <pfleger.will@gmail.com> Signed-off-by: Will Pfleger <pfleger.will@gmail.com>
wpfleger96
force-pushed
the
duncan/b1-transactional-agent-store
branch
from
August 4, 2026 19:41
2badedb to
78c977e
Compare
…tion Fix A: single-owner file recovery ordering and seam test. - lib.rs: run_file_commit_recovery() now runs BEFORE spawn_boot_recovery() is spawned, establishing synchronous pre-admission file repair. - txn.rs: run_boot_recovery_at() (the background publication task) no longer calls recover_interrupted_file_commits — it is publication-only. File repair has exactly one owner: the synchronous run_file_commit_recovery call. - txn.rs: add file_commit_recovery_at_pub() (cfg(test)) for testing the sync path without AppHandle. - Two seam tests in store_journal_fix_tests.rs: test_background_recovery_does_not_repair_files: proves run_boot_recovery_at leaves stage files and phase rows untouched. test_sync_file_recovery_repairs_dangling_intent_row: proves file_commit_recovery_at_pub renames stages and advances phase to committed. Fix B: verifiable schema migration. - schema.rs: replace let _ = conn.execute_batch() with PRAGMA table_info-based column inspection, ADD COLUMN only for missing columns, errors propagated, user_version stamped inside the same transaction as the migration. Post-commit verification ensures required columns are present. - New table_column_names() helper for column introspection. - Three migration tests in store_journal_fix_tests.rs: test_schema_migration_from_v2_adds_all_columns: v2 DB gets all three columns. test_schema_migration_from_partial_v3_adds_missing_d_tag: partial v3 gets d-tag. test_schema_migration_idempotent_on_v4: re-running on v4 is a no-op. Also update existing file-commit tests to use file_commit_recovery_at_pub (store_journal_tests.rs crash test, store_journal_fix_tests.rs hash-verify tests) since run_boot_recovery_at no longer performs file repair. Co-authored-by: Will Pfleger <pfleger.will@gmail.com> Signed-off-by: Will Pfleger <pfleger.will@gmail.com>
Apply PRAGMA query_only=ON on a v2 in-memory connection before calling apply_journal_schema_pub: ALTER TABLE fails, the transaction rolls back, and user_version stays at 2. Disabling query_only and retrying succeeds with all three columns present and version 4 -- proving the retry path. Also corrects the false doc comment on test_schema_migration_idempotent_on_v4 (was incorrectly claiming in-memory DBs cannot be made read-only via query_only; restated as a straightforward idempotency check). Co-authored-by: Will Pfleger <pfleger.will@gmail.com> Signed-off-by: Will Pfleger <pfleger.will@gmail.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Replaces the autocommit+sequential-rename agent store with a substrate that makes transactional claims true end-to-end.
What changed
Commit protocol (
store_journal/txn.rs)mutate_storeis the only write entry point. It owns the full lock sequence:flock/ named mutex) → fresh fail-closed decodeBEGINSQLite transaction — all journal mutations plus file serialization run insidefsync(serialization failure rolls back)file_commit_phasesintent row inside the transaction (commit-unique stage paths)COMMITmanaged-agents.<commit_id>.stage→ recordfirst_renamed→ renameteams.<commit_id>.stage→ recordcommittedA closure returning
Errrolls back the SQLite transaction — no journal row is written, no file is staged.Stage paths are commit-unique (
managed-agents.<uuid>.stage) so two concurrentmutate_storecalls or a recovery-vs-mutation race cannot overwrite each other's temp files.Boot recovery
run_file_commit_recoveryruns synchronously under the anchored advisory lock during app setup, before any commands are admitted. This ensures the JSON store is in a consistent state before the first write arrives.run_boot_recovery_at(called fromspawn_boot_recovery) performs two phases: file-commit recovery (under advisory lock) then publication recovery (outbox re-drive into the active scoped retention DB).Recovery uses the active scoped retention DB (
anchor/retention/<sha256(relay,owner)>.db) resolved viaactive_retention_scope— the same database the flush loop drains. The prior flatanchor/retention.dbpath did not exist in production and silently nullified publication re-drive.File recovery is fail-closed on ambiguous missing stages (intent phase, agents-absent teams-present) rather than treating absence as proof of success.
Publication authority (
prepare_publication)Single function that establishes durable evidence in a recoverable ordered protocol across two SQLite connections (journal then retention).
IdentityCollision→Err. Wired for all production publication paths: agent retain/tombstone/archive, team retain/tombstone, persona pending/tombstone.Tombstone and archive helpers (
tombstone_managed_agent_pending,archive_managed_agent_pending,tombstone_team_pending,tombstone_persona_pending) returnResult<(), String>— callers log errors rather than swallowing them.Keyring chokepoint restored
persist_agent_keys_pubis called on a save-local record clone beforemutate_agent_storefor all creates and imports (agents.rs,personas/snapshot/import.rs). The nsec is stripped after keyring write+verify; the closure only receives a cleared record. Inline fallback preserved when keyring unavailable. Team snapshot already had this pattern.Fail-closed event-sync / reconcile
migrate_personas_to_eventsandreconcile_agents_to_eventsresolve the advisory lock and file paths fromstore_anchor_dir(). Lock-acquire failure returns early.Retention DB migration under advisory lock
active_retention_scopemigrates the old per-process DB to the anchored path using the SQLite backup API under the advisory lock. Fail-closed.Windows abandoned-mutex handling
WaitForSingleObjectacceptsWAIT_OBJECT_0andWAIT_ABANDONED; rejectsWAIT_FAILEDwithGetLastErrorcontext.Generation intents at identity mutations
cas_generationwired for agent create/update/delete, team create/update/delete, agent settings, and team snapshot import.CasOutcome::TombstonedandConflictmatched — create rejects tombstoned keys.Boundary
All changes are within
desktop/src-tauri/**. No mobile, frontend, or relay files.Gates
just desktop-tauri-clippy— zero errors, zero warnings (-D warnings)just desktop-tauri-test— 2207 passed, 0 failedjust desktop-check— no ratchet violationscargo test --workspace(test target) — 0 warningsDeferred to B1.1
Three items explicitly out of scope per Will's ruling (2026-08-04), tracked as B1.1 follow-up — not separate issues:
Keyring compensation protocol (CRIT 4 residual). The restore-on-crash protocol for overwritten agent keys (
persist_agent_keys_journaled/pin_compensation) is not wired in production. Delete is generation-fenced viatombstone_key; what's missing is the two-phase lock-release/reacquire mechanism that would restore an old key value if the app crashed mid-overwrite. Deferred because: (a) it's substantial new surface, (b) if agent nsecs move to relay aggregates under Wes's relay-only migration plan, the local keyring-compensation protocol may shrink or disappear.Lifecycle-only CAS (IMP residual). Start/stop/restore/runtime/shutdown mutations do not call
cas_generation. These paths do not write new identity records and are already serialized by the advisory lock. Accepted as a documented design position: CAS guards identity-key ownership; the lock prevents lost updates on lifecycle writes.Bulk boot migrations + outbox-join flusher (CRIT 2 residual). Boot event-sync/reconcile bulk writers (
event_sync.rs,reconcile.rs,team_snapshot.rssnapshot-import path) still write retention directly without outbox evidence. The background flush loop still drains the retention table rather than validating a 1:1 join against outbox rows. Both require design work and belong in a dedicated B1.1 pass.