Skip to content

Recover from an empty or corrupt base snapshot - #198

Merged
MehdiMamas merged 11 commits into
mainfrom
fix1879-app
Nov 14, 2026
Merged

MehdiMamas merged 11 commits into
mainfrom
fix1879-app

Conversation

@MehdiMamas

@MehdiMamas MehdiMamas commented Nov 14, 2026 •

Copy link
Copy Markdown
Owner

Closes #197.

A restart in the middle of a write left the base snapshot's files empty, and every later sync failed with EOF while parsing.

  • state::write_atomic: sync the temp file before the rename and the parent folder after it (Unix). save_base_dir: sync each file it writes.
  • load_base_dir: when head.json or the files tree cannot be read, try base.old; if that fails too, rename the folder to base.broken (a failed rename is ignored), log one warning, and return an empty base so the next sync fetches the group again.
  • A valid head.json with 0-byte files under files/ is now a bad base too (see Diff review fixes).
  • With an empty base, a local edit is kept and a local delete stays a held local change; nothing is pushed or restored (tested).

Checks (macOS): cargo fmt --all -- --check ok; cargo clippy --workspace --all-targets -- -D warnings ok; cargo test -p claude-sync-core: 554 passed, 0 failed (lib), other targets ok.

Review fixes: save_base_dir now syncs every file it writes, each folder and the parent folder after the final rename, and a good base.old is restored over a corrupt base. Only parse errors count as corruption; each bad copy gets a free base.broken / base.broken.N name. read_base_file falls back to base.old. A failed folder sync after a rename in write_atomic is logged, not an error.

Cost: one fsync per file written, per base save (only files actually written, plus one per folder and the parent folder). A save writes the whole snapshot, so a group with many files pays that many syncs each time its base changes.

Second review: folder sync errors in save_base_dir are logged (like write_atomic) and cleanup still runs; a bad base that cannot be moved aside is deleted so base.old is restored, and a save never deletes a good base.old while base is bad; only the newest 3 base.broken* copies are kept.

Cost, second part: write_atomic now does a file sync and a folder sync on every state, status and review write, including the per-round status writes. On macOS sync_all is a full flush (milliseconds to tens of ms each), so a base save with hundreds of files can take seconds. Accepted for this fix; syncing only changed files is a possible follow-up.

Tests: cargo test -p claude-sync-core: 554 passed, 0 failed (lib).

Scope: this PR is the app part (Part A). The origin guard (Part B) ships in claude-sync-data PR 1949; #197 is closed together with it. The 134 load average from the issue is not explained by this fix.

Diff review fixes (third round):

  • base.broken copies are now named base.broken.<n>, n above every existing copy (time in ms, or highest + 1 if the clock went back), and pruned by n, not by folder mtime: the copy just moved aside always survives. Tests set no times.
  • read_base_dir treats a 0-byte file under files/ as corrupt (so base.old is tried, else the folder is moved aside), unless the save recorded it as empty in empty.json (a file that is empty in the group itself, so it never loops). An older base without empty.json that holds an empty file is moved aside once and fetched again.
  • Callers: load_base is also called from status reads (group_auto_accept, group_proposal_mode, the window) while a sync can save, and only the sync pass holds sync.lock; so loads and saves were not serialized. Added one in-process lock around load_base_dir_with and save_base_dir_with. Across processes (app and CLI) a sync is serialized by sync.lock, a status read is not; not added.
  • Fsync cost: accepted (small files), no code change.
  • Tests: cargo test -p claude-sync-core on macOS: 559 passed, 0 failed.

Diff review fixes (fourth round):

  • Status reads no longer repair. group_auto_accept, group_proposal_mode, the tick's head checks and read_base_file use peek_base_dir: base, else a good base.old, else an empty base; it never renames, restores or deletes (so it cannot race a save in another process). Only the sync pass (holding sync.lock) uses load_base_dir, which repairs. Both take the in-process BASE_IO lock.
  • read_base_file now applies the same empty-file check (it reads through peek_base_dir); a folder with files but no head.json is read as before.
  • Upgrade: a base saved before this PR that holds a file empty in the group has no empty.json, so it is moved aside once (one base.broken.<n> copy), the group is fetched again and the next save writes the list. Tested (an_older_base_with_an_empty_file_is_moved_aside_once).
  • Fsync cost stays accepted.
  • Tests: cargo test -p claude-sync-core on macOS: 562 passed, 0 failed.

Before: a cut write left empty base files and every sync failed with EOF while parsing.
Now: writes are synced to disk, and an unreadable base falls back to base.old, else base.broken and an empty base.
Before: the base.old fixture was saved under a wrong name.
Now: it is moved into place, and the no-base sync test asserts the kept edit and held delete.
@MehdiMamas

Copy link
Copy Markdown
Owner Author

Plan review

Verdict: the diagnosis and the split (app recovery plus a probe guard) are right. Fix the points below before building.

File:line facts

  • nodes.mjs:414 freshRef is correct. classifyProbe is at node-probe-core.mjs:20, and the behind block is at lines 50-60.
  • engine.rs line numbers are off by 1-2. load_base_dir is 2256-2272 and save_base_dir is 2274-2292, not 2257-2273 and 2276-2294.
  • The claim that save_base_dir renames without flushing is true, but it is incomplete. head.json is written through write_json, which is state::write_atomic (state.rs:40-52). That function does std::fs::write then rename, with no sync_all. So "sync the head" is not a change inside save_base_dir. The same function writes state, status and review files, so those can also come back 0 bytes after a cut. The simplest fix is one sync_all in state::write_atomic, as apply.rs:1355 already does, plus one in the files loop. Also fsync the parent dir after the final rename. Without that the swap itself can be lost, and the rename is what the old-base guarantee rests on.

Part A gaps

  1. load_base_dir picks the first dir where head.json is_file(). A 0-byte head.json is a file, so it never falls through to base.old. Before returning "no base", try base.old when the main dir fails to parse. A cut between the two renames in save_base_dir leaves a good base.old. Throwing that away forces a full refetch for no reason.
  2. A valid head.json with 0-byte files under files/ is not detected. read_tree reads them fine and returns empty content. Head holds commit, tree, date and etag, with no content hash to check against. The plan's "files/ unreadable" case does not cover this. State it as a known residual, which the fsync fix prevents, or do not claim it is covered.
  3. If renaming to base.broken fails (Windows sharing, permissions), load_base_dir must still return the default and not error. Otherwise sync stays stuck. Add a test for it. The warn line also needs a place to go that is visible. Name where, since the daemon runs unattended.
  4. "No base" is not a first join. With an empty base, the three-way merge sees every local-only edit and every local deletion since the last good sync as a difference. Deleted files can come back, and edits can become review items or conflicts. The plan's risk line says "may raise a review item". Say what happens to a local delete, and add a test with a locally edited file and a locally deleted file against a no-base sync. This is the data-loss path, so it deserves more than a one-line risk.
  5. Check read_base_file (engine.rs:2297) with a 0-byte head.json. It uses the same head.json is_file() || files is_dir() selection. It should stay tolerant, and a test should say so.

Part B gaps

  1. freshRef takes one injectable run, and the existing tests stub it with one canned reply. Calling gh through the same run makes those stubs answer for both calls. Add a separate ghRun parameter, and memoise it with the same key rules.
  2. The memo and return value are the parsed fresh object, and null when the hub's own script fails. Hanging origin on that object means a failed local script also drops origin, so the guard goes quiet in exactly the broken-hub case. Return { ...fresh, origin }, or a separate field, and build origin even when fresh is null. Make classifyProbe accept ref.origin without ref.mods.
  3. If a node trips both the hub-based reason and the origin reason, it prints two near-identical reasons. When both fire, give only the origin one, because main is the truth and the hub is a proxy.
  4. The origin check compares only the sha. A node that is ahead of or different from main (a force-push or rewrite of main, a gh read that lags) is flagged as behind for as long as main stays older than 30 min. Use the node's own COMMIT date (line 2 of the file) and require it to be older than origin.at, like the existing older(). The node's modsAt is the file mtime, so do not use that.
  5. A silent gh failure turns the guard off with no sign. This repo is private, so a hub without gh auth is the normal failure. Put "origin unavailable" in the probe log or --once --dry-run output, so a dead guard is visible. Keep the probe state ok, as planned.
  6. The edge-case list is missing: gh returns a sha but a bad date (no origin); at is in the future (clock skew); a node COMMIT with trailing text or an empty first line.

Simplicity

  • Part B is fine as one gh call per round. A simpler alternative is to have the hub check its own sync result (claude-sync-cli status error or last-sync age). That would not catch a node that syncs but is stuck on a stale head, so keep the gh compare.
  • Part A can shrink. One sync_all in state::write_atomic removes the cause for every state file. The recovery in load_base_dir is the second layer, so test both.

Done-when

  • Add that the desktop's base/ is not hand-edited. The plan already relies on the app doing the recovery, so the evidence should be the warn line and a base.broken that exists, from the real run.
  • Also name the 134 load average from the issue as not explained. A cut write at restart explains the 0-byte head. It does not say whether the load contributed. Do not close the issue with that left open.

Before: save_base_dir synced only files; a corrupt base deleted the good base.old on the next save; any read error moved base aside, overwriting base.broken; read_base_file ignored a bad head.json; a failed folder sync failed the write.
Now: files, folders and the parent folder are synced; a good base.old is restored over a corrupt base; only parse errors count as corruption and each bad copy gets a free base.broken name; read_base_file falls back to base.old; a failed folder sync after the rename is logged.
@MehdiMamas

Copy link
Copy Markdown
Owner Author

Fixes:

  • (2) save_base_dir syncs each written file, every folder under base.new, and the parent folder after the final rename.
  • (3) load_base_dir restores a good base.old to base when base is corrupt (bad folder moved aside); it also restores base.old when base is missing. The next save can no longer delete the only good copy.
  • (4) Only parse/invalid-content errors count as corruption; other I/O errors are returned and nothing is moved. Bad copies go to base.broken, base.broken.1, ... and an earlier one is never overwritten.
  • (5) read_base_file skips a folder whose head.json does not parse and reads base.old (read-only, nothing moved).
  • (6) write_atomic logs a failed parent-folder sync after the rename and still succeeds.
  • (7) PR body notes the per-save fsync cost; only written files are synced.
    Tests added for each (restore, missing base, unique broken name, stuck rename, I/O error, read_base_file fallback). fmt, clippy and cargo test -p claude-sync-core (549 passed) ran on mac; CI green on all 3 OSes.

@MehdiMamas

Copy link
Copy Markdown
Owner Author

Diff review:

Plan: issue 197 has only the late-step note (no Plan comment before the code), so I checked against the PR body and the earlier plan-review points. The 7 earlier findings are fixed correctly in the main paths: the sync order, restore of base.old, the unique base.broken.N names, the I/O-vs-corruption split, the read_base_file fallback, the logged folder sync in write_atomic, and the cost note. Remaining problems:

  1. crates/core/src/engine.rs:2372 and :2380: save_base_dir uses sync_dir(...)?, so a folder fsync that fails (some network or FUSE filesystems return EINVAL or EPERM) now fails every base save. write_atomic logs the same failure and carries on (fix 6), so the two are inconsistent. Before this PR such a machine synced fine. Now each sync errors and refetches. The failure at :2380 comes after the swap, so it also skips remove_dir_if_exists(&old) and returns an error for a save that worked. Log these like write_atomic does.

  2. crates/core/src/engine.rs:2361 and crates/core/src/state.rs:49: cost. sync_all on macOS is F_FULLFSYNC, which is slow (milliseconds to tens of ms each). save_base_dir now does one per file plus one per folder on every base change, and write_atomic does one on every state, status and review write, including the per-round status writes. A group with hundreds of skill files can make each sync take seconds. The PR body notes the base cost but not the write_atomic hot path. Consider syncing only the files whose content changed, or syncing only head.json and the folders and relying on the file sync only for new files.

  3. crates/core/src/engine.rs:2316: quarantine keeps up to 100 whole snapshot copies and never deletes any. A machine that keeps corrupting (bad disk) grows without limit up to that cap, and after 100 the bad folder stays in place and load_base_dir warns on every load. Cap or prune old base.broken.*.

  4. crates/core/src/engine.rs load_base_dir: if quarantine(dir) fails while base.old is good, the bad base stays and base.old is returned. The next save_base_dir then deletes base.old (the only good copy) before the swap, so a cut during that save leaves nothing. This is narrow, but it is the case fix 3 said it closes. No test covers it (the stuck-rename test only covers the no-old case).

  5. Scope: the plan-review comment describes a Part B (the classifyProbe / freshRef origin guard). This PR contains only Part A and closes A cut write leaves an empty base snapshot and every sync fails with EOF while parsing #197. The 134 load average noted in that review is also not addressed anywhere. If Part B is wanted, A cut write leaves an empty base snapshot and every sync fails with EOF while parsing #197 should not close on this PR alone.

  6. Minor: the PR body says 544 tests passed, the Fixes comment says 549.

…lder sync errors (#197)

Before: a failed folder sync failed the save; a failed move-aside left base.old to be deleted by the next save; base.broken.N copies were never pruned.
Now: folder sync errors are logged; the bad base is deleted so base.old is restored, and a save keeps a good base.old over a bad base; only the newest 3 base.broken copies stay.
Before: sort_by tripped clippy.
Now: sort_by_key with Reverse.
Before: the test opened a folder to set its time, which fails on Windows.
Now: the test is Unix only.
@MehdiMamas

Copy link
Copy Markdown
Owner Author

Fixes:

  • (1) save_base_dir logs a failed folder sync (like write_atomic) and cleanup of base.old still runs.
  • (3) Only the newest 3 base.broken* copies are kept; older ones are pruned after each move-aside.
  • (4) If the bad base cannot be moved aside while base.old is good, it is deleted so base.old is restored. A save also keeps a good base.old when base is bad, instead of deleting it first. Tests: stuck rename with a good old base, save over bad base with good old.
  • (2)/(6) PR body now covers the write_atomic fsync cost and the real test count (552).
  • (5) PR body says A cut write leaves an empty base snapshot and every sync fails with EOF while parsing #197 closes together with claude-sync-data PR 1949.
    fmt, clippy and cargo test -p claude-sync-core (552 passed) ran on mac; CI green on all 3 OSes.

@MehdiMamas

Copy link
Copy Markdown
Owner Author

Diff review: no problems found

@MehdiMamas

Copy link
Copy Markdown
Owner Author

Check: edge cases from #197 against PR 198 (head c25df27)

  • pass: given head.json is 0 bytes and base.old is good -> expect load returns base.old content and sync continues (crates/core/src/engine.rs: an_empty_head_loads_the_old_base)
  • fail: given base and base.old are both bad -> expect the bad folder is moved to base.broken, one warning is logged, an empty base is returned - no test covers both-bad; the code also moves base.old to base.broken.1 (a second bad folder) and no test counts the warnings
  • pass: given the move to base.broken fails and base.old is good -> expect the bad base is deleted and base.old is restored (crates/core/src/engine.rs: a_stuck_rename_still_restores_the_good_old_base)
  • fail: given a folder fsync fails during save -> expect it is logged, the save still succeeds and base.old cleanup runs - no test injects a sync failure, so the sync_dir_logged path is untested
  • pass: given more than 3 base.broken copies exist -> expect only the newest 3 are kept (crates/core/src/engine.rs: only_the_newest_broken_copies_are_kept)
  • pass: given a crash between removing base and swapping in the new folder -> expect base.old is still there to recover from (crates/core/src/engine.rs: a_missing_base_with_an_old_one_restores_it)

Before: no test covered base and base.old both bad, or a folder sync failure in save_base_dir.
Now: save_base_dir takes the sync function through a private seam; two tests cover both cases.
@MehdiMamas

Copy link
Copy Markdown
Owner Author

Check: acceptance points against PR 198 head 34a99a0

  • pass: head.json and every base file are fsynced (sync_all) and each folder is synced before the swap rename; the parent folder is synced after it (engine.rs save_base_dir, state.rs write_atomic).
  • pass: load_base_dir falls back to base.old when base is missing, 0-byte or unparsable (an_empty_head_loads_the_old_base, a_garbage_head_counts_as_no_base, a_good_old_base_is_restored_over_a_bad_one).
  • pass: both-bad case moves base to base.broken and base.old to base.broken.1, and returns an empty base (both_bases_bad_are_moved_aside_and_the_base_is_empty). The test does not count the warning lines.
  • pass: prune_broken keeps only the newest KEEP_BROKEN (3) base.broken* folders (only_the_newest_broken_copies_are_kept, Unix-only test; the prune code itself is platform-neutral).
  • pass: a folder fsync error is logged, not returned, in save_base_dir and write_atomic, and the save succeeds (a_failed_folder_sync_does_not_fail_the_save).
  • pass: CI green at the head on macos-latest, ubuntu-latest and windows-latest (run 43572347387, head_sha 34a99a0).

@MehdiMamas

Copy link
Copy Markdown
Owner Author

Check: edge cases from #197 against PR 198 (head 34a99a0)

  • pass: given head.json is 0 bytes and base.old is good -> expect load returns base.old content and sync continues — crates/core/src/engine.rs:5009
  • fail: given base and base.old are both bad -> expect the bad folder is moved to base.broken, one warning is logged, an empty base is returned — crates/core/src/engine.rs:5187 (the move and the empty base are asserted; no test counts the warning, which is emitted at crates/core/src/engine.rs:2298)
  • pass: given the move to base.broken fails and base.old is good -> expect the bad base is deleted and base.old is restored — crates/core/src/engine.rs:5111
  • fail: given a folder fsync fails during save -> expect it is logged, the save still succeeds and base.old cleanup runs — crates/core/src/engine.rs:5175 (success and cleanup are asserted; no test asserts the log, which is emitted at crates/core/src/engine.rs:2438)
  • pass: given more than 3 base.broken copies exist -> expect only the newest 3 are kept — crates/core/src/engine.rs:5140 (Unix-only test)
  • pass: given a crash between removing base and swapping in the new folder -> expect base.old is still there to recover from — crates/core/src/engine.rs:5216

Before: the one warning for a bad base and the folder sync failure log were not asserted.
Now: both tests capture WARN output with the tracing-subscriber already in dev-dependencies and check it.
@MehdiMamas

Copy link
Copy Markdown
Owner Author

Check: edge cases from #197 against PR 198 (head 1e93635)

  • pass: given head.json is 0 bytes and base.old is good -> expect load returns base.old content and sync continues — crates/core/src/engine.rs:5009
  • pass: Given base and base.old are both bad, the code moves base to base.broken and also base.old to base.broken.1 (not only base), logs one warning (for base), and returns an empty base so the next sync fetches the group again — crates/core/src/engine.rs:5219
  • pass: given the move to base.broken fails and base.old is good -> expect the bad base is deleted and base.old is restored — crates/core/src/engine.rs:5111 (asserts the restore; the deletion of the bad folder is in the code path, not asserted)
  • pass: given a folder fsync fails during save -> expect it is logged, the save still succeeds and base.old cleanup runs — crates/core/src/engine.rs:5200
  • pass: given more than 3 base.broken copies exist -> expect only the newest 3 are kept — crates/core/src/engine.rs:5140 (Unix-only test)
  • pass: given a crash between removing base and swapping in the new folder -> expect base.old is still there to recover from — crates/core/src/engine.rs:5252

@MehdiMamas

Copy link
Copy Markdown
Owner Author

Check: (original Plan wording, head 1e93635)

  • pass: given base and base.old are both bad -> expect the bad folder is moved to base.broken, one warning is logged, an empty base is returned — crates/core/src/engine.rs:5219

@MehdiMamas

Copy link
Copy Markdown
Owner Author

Check: (head 1e93635, Plan wording)

  • pass: given head.json is 0 bytes and base.old is good -> expect load returns base.old content and sync continues — crates/core/src/engine.rs:5009
  • pass: given base and base.old are both bad -> expect the bad folder is moved to base.broken, one warning is logged, an empty base is returned — crates/core/src/engine.rs:5219
  • pass: given the move to base.broken fails and base.old is good -> expect the bad base is deleted and base.old is restored — crates/core/src/engine.rs:5111
  • pass: given a folder fsync fails during save -> expect it is logged, the save still succeeds and base.old cleanup runs — crates/core/src/engine.rs:5200
  • pass: given more than 3 base.broken copies exist -> expect only the newest 3 are kept — crates/core/src/engine.rs:5140
  • pass: given a crash between removing base and swapping in the new folder -> expect base.old is still there to recover from — crates/core/src/engine.rs:5252

@MehdiMamas

Copy link
Copy Markdown
Owner Author

Diff review:

  1. crates/core/src/engine.rs prune_broken (about line 2330): it sorts base.broken* by the folder mtime. A rename does not change a folder mtime, so the copy just moved aside keeps its old mtime. If 3 or more older-named copies have newer mtimes, the copy just quarantined is pruned at once, and the evidence is gone. The test only_the_newest_broken_copies_are_kept sets mtimes by hand and misses this. Fix: exclude the just-moved target from pruning, or order by name or by a timestamp in the name.
  2. The known limit in the PR body is the exact failure in the issue: a valid head.json with 0-byte files under files/ is not detected, so it still loads as a base. Only files that fail to parse (Json/Invalid errors) trigger recovery. If the incident left head.json valid and the files empty, the fix does not cover it. The new sync_all calls only prevent new cases. The Plan has no edge case for it, and sync_recovers_from_an_empty_base zeroes head.json as well, so it does not test it.
  3. load_base_dir_with (about line 2270) renames and deletes folders inside a function named load. If a save runs at the same time (for example a UI read through load_base during a sync), the load can quarantine the base.new to base swap. No lock is visible in the diff. Please confirm the callers are serialized.
  4. Cost: write_atomic now does two fsyncs on every state, status and review write, and save_base_dir does one per file. The PR body accepts this, but there is no measurement for a large group.
    Nothing found on security.

…rialize base reads and saves (#197)

Before: base.broken copies were pruned by folder mtime, so the copy just moved aside could be deleted at once; a valid head.json with 0-byte files loaded as a good base; a status read could move a folder while a save swapped it.
Now: copies are named base.broken.<n> with n above every existing copy and pruned by n; a 0-byte file not recorded in empty.json makes the snapshot bad (base.old, else quarantine); load and save share one lock.
@MehdiMamas

Copy link
Copy Markdown
Owner Author

Check: (head a936a37, CI green on macOS, Ubuntu, Windows; edge cases quoted from the Plan comment in #197)

  • pass: given head.json is 0 bytes and base.old is good -> expect load returns base.old content and sync continues — crates/core/src/engine.rs:2268 (test :5055)
  • pass: given base and base.old are both bad -> expect the bad folder is moved to base.broken, one warning is logged, an empty base is returned — crates/core/src/engine.rs:2268 (test :5325)
  • pass: given the move to base.broken fails and base.old is good -> expect the bad base is deleted and base.old is restored — crates/core/src/engine.rs:2268 (test :5159)
  • pass: given a folder fsync fails during save -> expect it is logged, the save still succeeds and base.old cleanup runs — crates/core/src/engine.rs:2460 (test :5306)
  • pass: given more than 3 base.broken copies exist -> expect only the newest 3 are kept — crates/core/src/engine.rs:2371 (tests :5206, :5218, no hand-set mtimes)
  • pass: given a crash between removing base and swapping in the new folder -> expect base.old is still there to recover from — crates/core/src/engine.rs:2464 (test :5113)
  • pass: (new) given a valid head.json with 0-byte files under files/ -> expect base.old, else quarantine; an empty file recorded in empty.json stays good — crates/core/src/engine.rs:2379 (tests :5240, :5254, :5270, :5437)

@MehdiMamas

Copy link
Copy Markdown
Owner Author

Diff review:

  • crates/core/src/engine.rs, load_base_dir_with (the else if !dir.exists() { rename(&old, dir) } branch): the read path moves folders. Another process (CLI or app) that is in the middle of save_base_dir_with, between rename(dir, &old) and rename(&new, dir), sees base missing and base.old good. The reader then renames base.old to base. The saver's rename(&new, dir) then fails on a non-empty target, so the save errors. BASE_IO is in-process only, and the PR body says status reads are not serialized across processes. Status reads (group_auto_accept, group_proposal_mode, the window) should not rename or quarantine anything. Either do the repair only under sync.lock, or have read-only callers use a non-mutating load like read_base_file.
  • Same function, quarantine on a read-only status path: a reader that races a save in another process can see a half-written base as corrupt (parse error on a file being replaced). It then moves base aside or deletes it (remove_dir_if_exists(dir)), which can destroy the folder the saver just swapped in.
  • read_base_file takes no BASE_IO lock, while load_base_dir and save_base_dir do. It can read between the two renames in one process. Its usable check also does not apply the 0-byte-file test, so it can return an empty file from a snapshot that load_base_dir calls bad.
  • read_base_dir: an existing base saved before this PR that holds a legitimately empty file has no empty.json. It is treated as corrupt and quarantined, then refetched. This happens once for every such group, and each one leaves a base.broken.<n> copy behind. The PR body admits it. No test covers the upgrade case.
  • Process: the issue has no Plan comment from before the code. All the Plan comments are marked "Late step" and were written after the commits, and the last one changed the design (empty.json, BASE_IO) after the earlier Plan comments. So there is no pre-code Plan to check the diff against, and the edge-case list does not cover the empty.json case or the cross-process race.
  • Cost: write_atomic now does two fsyncs on every state, status and review write, and save_base_dir does one fsync per file on every base save. Hot paths pay this, including the per-round status writes. The PR accepts it, but there is no measurement and no cap.

…heck (#197)

Before: status reads called load_base_dir, which can rename, restore or delete base folders, racing a save in another process; read_base_file skipped the empty-file check and the lock.
Now: group_auto_accept, group_proposal_mode, the tick head checks and read_base_file use peek_base_dir (base, else base.old, else empty; nothing moved); only the sync pass repairs. Tests for no-move reads and the upgrade case.
Before: read_base_file via peek_base_dir ignored a folder with files but no head.json, so job tests failed.
Now: such a folder is read as before, under the lock.
@MehdiMamas

Copy link
Copy Markdown
Owner Author

Check: (head 8c219a5, CI green on macOS, Ubuntu, Windows; edge cases word for word from the Edge cases comment on #197)

  • pass: given head.json is 0 bytes and base.old is good -> expect load returns base.old content and sync continues — crates/core/src/engine.rs:2289 (test :5078)
  • pass: given base and base.old are both bad -> expect the bad folder is moved to base.broken, one warning is logged, an empty base is returned — crates/core/src/engine.rs:2289 (test :5411; base.old also goes to a second base.broken., per the correction comment)
  • pass: given the move to base.broken fails and base.old is good -> expect the bad base is deleted and base.old is restored — crates/core/src/engine.rs:2289 (test :5182)
  • pass: given a folder fsync fails during save -> expect it is logged, the save still succeeds and base.old cleanup runs — crates/core/src/engine.rs:2484 (test :5392)
  • pass: given more than 3 base.broken copies exist -> expect only the newest 3 are kept — crates/core/src/engine.rs:2392 (test :5229, no hand-set mtimes)
  • pass: given a crash between removing base and swapping in the new folder -> expect base.old is still there to recover from — crates/core/src/engine.rs:2488 (test :5136)
  • pass: (new) given a valid head.json with 0-byte files under files/ -> expect base.old, else quarantine; a file recorded in empty.json stays good — crates/core/src/engine.rs:2403 (tests :5262, :5277)
  • pass: (new) given a status read (group_auto_accept, group_proposal_mode, tick, read_base_file) during or after a bad save -> expect nothing is renamed, restored or deleted — crates/core/src/engine.rs:2275 (test :5286)
  • pass: (new) given read_base_file on a snapshot with an empty file -> expect it is skipped like load_base_dir does — crates/core/src/engine.rs:2514 (test :5310)
  • pass: (new) given a base saved before empty.json with a legitimately empty file -> expect it is moved aside once, then good after the next save — crates/core/src/engine.rs:2403 (test :5324)

@MehdiMamas

Copy link
Copy Markdown
Owner Author

Diff review:
Plan steps (write_atomic and save_base_dir syncs, load fallback to base.old then quarantine with an empty base, tests) are all covered. I found no correctness bug. Two small problems:

  1. crates/core/src/engine.rs, read_base_file (~line 2400 of the new file): it now goes through peek_base_dir, which parses head.json and reads the whole files/ tree to return one file. It used to read a single file. If the app calls it on a poll or render path (schedules.json), that is a full-snapshot read per call, and it is a cost the issue and PR cost notes do not mention. Reading just the one file after the same head and empty-file checks would avoid it.
  2. crates/core/src/state.rs write_atomic (~line 47-50): if write_all or sync_all fails, the ? returns before the cleanup, so the .name.pid.tmp file is left behind. Only the rename failure removes it. The old std::fs::write leaked on write failure too, but sync_all is a new failure point.

Minor, no action needed: write_atomic now does a folder sync on every state write, not just base saves. Only the base-save cost is stated in the PR body.

@MehdiMamas
MehdiMamas merged commit 87a6e19 into main Nov 14, 2026
3 checks passed
@MehdiMamas
MehdiMamas deleted the fix1879-app branch November 14, 2026 08:22
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

A cut write leaves an empty base snapshot and every sync fails with EOF while parsing

1 participant