Repository navigation
Recover from an empty or corrupt base snapshot - #198
Conversation
Before: a cut write left empty base files and every sync failed with EOF while parsing. Now: writes are synced to disk, and an unreadable base falls back to base.old, else base.broken and an empty base.
Before: the base.old fixture was saved under a wrong name. Now: it is moved into place, and the no-base sync test asserts the kept edit and held delete.
|
Plan review Verdict: the diagnosis and the split (app recovery plus a probe guard) are right. Fix the points below before building. File:line facts
Part A gaps
Part B gaps
Simplicity
Done-when
|
Before: save_base_dir synced only files; a corrupt base deleted the good base.old on the next save; any read error moved base aside, overwriting base.broken; read_base_file ignored a bad head.json; a failed folder sync failed the write. Now: files, folders and the parent folder are synced; a good base.old is restored over a corrupt base; only parse errors count as corruption and each bad copy gets a free base.broken name; read_base_file falls back to base.old; a failed folder sync after the rename is logged.
|
Fixes:
|
|
Diff review: Plan: issue 197 has only the late-step note (no Plan comment before the code), so I checked against the PR body and the earlier plan-review points. The 7 earlier findings are fixed correctly in the main paths: the sync order, restore of
|
…lder sync errors (#197) Before: a failed folder sync failed the save; a failed move-aside left base.old to be deleted by the next save; base.broken.N copies were never pruned. Now: folder sync errors are logged; the bad base is deleted so base.old is restored, and a save keeps a good base.old over a bad base; only the newest 3 base.broken copies stay.
Before: sort_by tripped clippy. Now: sort_by_key with Reverse.
Before: the test opened a folder to set its time, which fails on Windows. Now: the test is Unix only.
|
Fixes:
|
|
Diff review: no problems found |
|
Check: edge cases from #197 against PR 198 (head c25df27)
|
Before: no test covered base and base.old both bad, or a folder sync failure in save_base_dir. Now: save_base_dir takes the sync function through a private seam; two tests cover both cases.
|
Check: acceptance points against PR 198 head 34a99a0
|
|
Check: edge cases from #197 against PR 198 (head 34a99a0)
|
Before: the one warning for a bad base and the folder sync failure log were not asserted. Now: both tests capture WARN output with the tracing-subscriber already in dev-dependencies and check it.
|
Check: edge cases from #197 against PR 198 (head 1e93635)
|
|
Check: (original Plan wording, head 1e93635)
|
|
Check: (head 1e93635, Plan wording)
|
|
Diff review:
|
…rialize base reads and saves (#197) Before: base.broken copies were pruned by folder mtime, so the copy just moved aside could be deleted at once; a valid head.json with 0-byte files loaded as a good base; a status read could move a folder while a save swapped it. Now: copies are named base.broken.<n> with n above every existing copy and pruned by n; a 0-byte file not recorded in empty.json makes the snapshot bad (base.old, else quarantine); load and save share one lock.
|
Check: (head a936a37, CI green on macOS, Ubuntu, Windows; edge cases quoted from the Plan comment in #197)
|
|
Diff review:
|
…heck (#197) Before: status reads called load_base_dir, which can rename, restore or delete base folders, racing a save in another process; read_base_file skipped the empty-file check and the lock. Now: group_auto_accept, group_proposal_mode, the tick head checks and read_base_file use peek_base_dir (base, else base.old, else empty; nothing moved); only the sync pass repairs. Tests for no-move reads and the upgrade case.
Before: read_base_file via peek_base_dir ignored a folder with files but no head.json, so job tests failed. Now: such a folder is read as before, under the lock.
|
Check: (head 8c219a5, CI green on macOS, Ubuntu, Windows; edge cases word for word from the Edge cases comment on #197)
|
|
Diff review:
Minor, no action needed: |
Closes #197.
A restart in the middle of a write left the base snapshot's files empty, and every later sync failed with
EOF while parsing.state::write_atomic: sync the temp file before the rename and the parent folder after it (Unix).save_base_dir: sync each file it writes.load_base_dir: whenhead.jsonor the files tree cannot be read, trybase.old; if that fails too, rename the folder tobase.broken(a failed rename is ignored), log one warning, and return an empty base so the next sync fetches the group again.head.jsonwith 0-byte files underfiles/is now a bad base too (see Diff review fixes).Checks (macOS):
cargo fmt --all -- --checkok;cargo clippy --workspace --all-targets -- -D warningsok;cargo test -p claude-sync-core: 554 passed, 0 failed (lib), other targets ok.Review fixes:
save_base_dirnow syncs every file it writes, each folder and the parent folder after the final rename, and a goodbase.oldis restored over a corruptbase. Only parse errors count as corruption; each bad copy gets a freebase.broken/base.broken.Nname.read_base_filefalls back tobase.old. A failed folder sync after a rename inwrite_atomicis logged, not an error.Cost: one fsync per file written, per base save (only files actually written, plus one per folder and the parent folder). A save writes the whole snapshot, so a group with many files pays that many syncs each time its base changes.
Second review: folder sync errors in
save_base_dirare logged (likewrite_atomic) and cleanup still runs; a badbasethat cannot be moved aside is deleted sobase.oldis restored, and a save never deletes a goodbase.oldwhilebaseis bad; only the newest 3base.broken*copies are kept.Cost, second part:
write_atomicnow does a file sync and a folder sync on every state, status and review write, including the per-round status writes. On macOSsync_allis a full flush (milliseconds to tens of ms each), so a base save with hundreds of files can take seconds. Accepted for this fix; syncing only changed files is a possible follow-up.Tests:
cargo test -p claude-sync-core: 554 passed, 0 failed (lib).Scope: this PR is the app part (Part A). The origin guard (Part B) ships in claude-sync-data PR 1949; #197 is closed together with it. The 134 load average from the issue is not explained by this fix.
Diff review fixes (third round):
base.brokencopies are now namedbase.broken.<n>,nabove every existing copy (time in ms, or highest + 1 if the clock went back), and pruned byn, not by folder mtime: the copy just moved aside always survives. Tests set no times.read_base_dirtreats a 0-byte file underfiles/as corrupt (sobase.oldis tried, else the folder is moved aside), unless the save recorded it as empty inempty.json(a file that is empty in the group itself, so it never loops). An older base withoutempty.jsonthat holds an empty file is moved aside once and fetched again.load_baseis also called from status reads (group_auto_accept,group_proposal_mode, the window) while a sync can save, and only the sync pass holdssync.lock; so loads and saves were not serialized. Added one in-process lock aroundload_base_dir_withandsave_base_dir_with. Across processes (app and CLI) a sync is serialized bysync.lock, a status read is not; not added.cargo test -p claude-sync-coreon macOS: 559 passed, 0 failed.Diff review fixes (fourth round):
group_auto_accept,group_proposal_mode, the tick's head checks andread_base_fileusepeek_base_dir:base, else a goodbase.old, else an empty base; it never renames, restores or deletes (so it cannot race a save in another process). Only the sync pass (holdingsync.lock) usesload_base_dir, which repairs. Both take the in-processBASE_IOlock.read_base_filenow applies the same empty-file check (it reads throughpeek_base_dir); a folder with files but nohead.jsonis read as before.empty.json, so it is moved aside once (onebase.broken.<n>copy), the group is fetched again and the next save writes the list. Tested (an_older_base_with_an_empty_file_is_moved_aside_once).cargo test -p claude-sync-coreon macOS: 562 passed, 0 failed.