Skip to content

fix(persistence): write state.json and the search index atomically (#854) - #855

Open
ydflow wants to merge 3 commits into
Tencent:mainfrom
ydflow:fix/atomic-state-writes
Open

ydflow wants to merge 3 commits into
Tencent:mainfrom
ydflow:fix/atomic-state-writes

Conversation

@ydflow

@ydflow ydflow commented Sep 27, 2026

Copy link
Copy Markdown
Contributor

Closes #854.

Problem

saveState, saveStateForScope and the buildIndex save write in place (writeJson = open + truncate + write). A crash, kill, ENOSPC or power loss mid-write leaves truncated JSON, and the readers answer null for what they cannot parse — the loss is silent, which is what makes the state.json case dangerous:

The repo already treats this exact failure mode as a bug: #831 made config.yaml saves atomic (Issue #823 item 14 — "a command reading it mid-save saw an empty file"), and saveUserVotes documents it for votes ("A torn plain overwrite would leave truncated YAML … silently wiping every doc's counts, all pending deltas, and the whole upvote ledger"). writeJsonAtomic (src/utils/fs.ts:161) already exists — temp file + rename, preserving the target's permission bits — these three writers just predate it.

Fix

Move the three writers to writeJsonAtomic:

  • saveState / saveStateForScope (src/config.ts)
  • the index save in buildIndex (src/utils/search-index.ts)

No reader changes; a successful write behaves as before, including file modes (existing files keep theirs; new state files get 0600, the same default writeJsonAtomic applies to the local-agent config today).

Tests

New src/__tests__/state-atomic-save.test.ts, mirroring the #831 method: intercept fse.writeFile to truncate the target, run a concurrent reader, then complete or fail the write with ENOSPC.

  • saveState / saveStateForScope: the concurrent loadStateForScope sees the old complete state (lastPullRev intact, never parse({})), and a failed write leaves the previous state on disk.
  • buildIndex: a concurrent loadIndex mid-rebuild sees the previous index, never null.

All 5 tests fail on main (the concurrent reader observes exactly the torn state: expected null to be 'abc1234', expected undefined to deeply equal ['a.md']) and pass on this branch. npx tsc --noEmit clean; npm run lint 0 warnings under --deny-warnings.

Real-CLI verification (607de7c)

npm run build, then node dist/index.js against a sandbox HOME and a local team-repo fixture with a learnings/ note — the writers behave end-to-end and leave no temp residue:

$ teamai pull
- [user] Pulling team repo...

state.json parses OK; lastPullRev = "73687bd"; workspaces = 6686d2852646
$ teamai recall "retry budget"
ℹ No matching learnings found for "retry budget".   ← relevance floor on a 1-entry corpus, as designed

search-index.json parses OK; entries = atomic-check.md
no tmp residue

The atomicity itself is what the unit tests above prove (a torn write cannot be reproduced in a passing CLI run by construction); this record shows the two writers work unchanged on the happy path.

ydflow and others added 3 commits September 27, 2026 13:13
…encent#854)

saveState, saveStateForScope and buildIndex wrote in place: open + truncate +
write. A crash, kill, ENOSPC or power loss mid-write leaves truncated JSON,
and the readers answer null for what they cannot parse:

- state.json: loadStateForScope falls back to StateSchema.parse({}) —
  lastPullRev and every per-checkout pushBaseRevs entry are gone without a
  word, so the next push compares against a stale base. That is the
  stale-base overwrite class the worktree fixes (Tencent#827) closed.
- search-index.json: loadIndex returns null, so recall silently loses the
  whole corpus until the next rebuild, and the shrink guard loses its
  baseline.

The repo already writes config.yaml (Tencent#831) and the votes file atomically for
exactly this reason, and writeJsonAtomic exists: temp file + rename,
preserving the target's permission bits. Move the three writers to it. No
reader changes; a successful write behaves as before.
CI failed recall-rebuild-roots: the 'cannot be written' test chmod'd the
index file 0o444, which fails the in-place writer but not the atomic one —
writeJsonAtomic stages a temp sibling (the directory is writable) and renames
over the target, and rename needs only the directory's permission. Same for
the EISDIR test: renaming a file over a directory is EISDIR on POSIX but
EPERM on Windows.

Fail the fse.writeFile calls at the index path itself — the staged temp file
included — so the injection works for both writers on every platform. The
root-skip guard goes with the chmod it existed for.
…, not the file mode

CI failed the 'cannot be recorded' case: it chmod'd state.json 0o444, which
stops the in-place writer but not the atomic one — writeJsonAtomic stages a
temp sibling and renames, and rename needs only the directory's permission,
so the push completed normally and exited 0. (The e2e config retries once;
the retry then failed teammatePublishes' git commit with 'nothing to
commit', the run's second error.)

The CLI runs as a subprocess, so the write call cannot be spied on the way
state-atomic-save does it. Inject the failure from outside instead: a
preload hook (NODE_OPTIONS --require) fails every fs.writeFile targeting
the staged temp of this state file inside the CLI process — the same seam
as the unit tests, on every platform.
@jeff-r2026 jeff-r2026 self-assigned this Sep 27, 2026
@jeff-r2026
jeff-r2026 self-requested a review September 27, 2026 10:17
@github-actions

Copy link
Copy Markdown

No findings.

The PR description includes sufficient testing for the runtime change, including a representative real-CLI verification. The record names commit 607de7c, which is older than the test-only head commits, but per the review rules this is non-blocking.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

[bug] state.json and the search index are written in place: a torn write silently resets push bases / recall

2 participants