Skip to content

[sentry:7849148048] The load itself succeeded; the failure was the model-claim record write that follows it. src/core/sessions.ts:112 is await claim.update('ready'), reached only after ` - #115

Merged
Ooooze merged 4 commits into
mainfrom
codex/sentry-fix/7849148048
Nov 18, 2026
Merged

Ooooze merged 4 commits into
mainfrom
codex/sentry-fix/7849148048

Conversation

@Ooooze

@Ooooze Ooooze commented Nov 16, 2026

Copy link
Copy Markdown
Contributor

The load itself succeeded; the failure was the model-claim record write that follows it. src/core/sessions.ts:112 is await claim.update('ready'), reached only after runtime.load(modelId, …) resolved, and the frames place both the ENOENT cause and the IO_ERROR wrapper inside writeClaim of src/lock/model-claim.ts (update → writeClaim → temp writeFile/rename, wrapped by claimIoError). The claims folder and the per-model claim directory are created before the engine start, and nothing in the core removes a live claim directory (ModelClaimHandle.release() is the only remover and it is serialized against the load or awaited by releaseAll() after the runtimes stop), so the directory was removed by something outside the claim protocol on that Windows host. The defect the event proves is the handling: writeClaim answered any failure of the post-load record write as a failed load, and acquireNow's catch keeps the handle because runtime.findSession(modelId) is now true, so every later load or attach of the same resident model repeated update('ready') on the same dead path and failed identically until unload. The repair in src/lock/model-claim.ts treats ENOENT from the record write as a vanished directory: it re-creates path with mkdir(path, { recursive: true }) and writes the record once more; a second failure and every other errno keep the existing claimIoError('write', …) wrapping, so a full volume, a file where the folder belongs or an unwritable folder still surface as coded IO_ERROR and still refuse. Probe src/lock/model-claim-recovery.test.ts was mined on base (named assertion writes the record again instead of failing the update) and now passes on the repaired code, together with its guard case that still expects a coded IO_ERROR. The controller's full-suite run reported one unrelated failure in src/core/create.test.ts (wires the diffusion service to the control API, the journal, the events and the shutdown order — the abandoned job was not cancelled after its own 10 s wait). That path is unreachable from this patch: the only added behaviour is the ENOENT branch of writeClaim, which runs solely when a claim-directory write fails with ENOENT, and the diffusion job cancellation neither creates nor deletes model claims. The same file documents in its header that whole-core tests failed at random under full-suite parallel load (T-282) and raises the per-test timeout to 20 s, while that inner 10 s poll loop is outside it — consistent with a load-induced flake rather than a regression.

Evidence:
src/core/sessions.ts:96 acquires the claim before the engine start; line 112 is await claim.update('ready'), the only update call site in acquireNow, reached only after runtime.load resolved.
src/lock/model-claim.ts update → writeClaim → temp writeFile + rename, with claimIoError producing new AtomicCoreError('IO_ERROR', …) — the message and code in the event.
Event facts: error_code IO_ERROR, cause_code/diagnostic_cause_code ENOENT, operation load, component runtime, failure_reason unknown, release atomic-chat-core@0.23.4, os.name win32.
src/lock/model-claim.ts creates the claims folder recursively and the per-model directory with mkdir(path) before writing the first record, so a later ENOENT means the directory disappeared between those calls and the post-load write.
Only ModelClaimHandle.release() removes a claim directory; its call sites are serialized by withModelTransition or awaited by releaseAll() after runtimes stop, so an in-process deletion during update('ready') is unreachable.
src/core/sessions.ts keeps the claim handle when the failure happens after runtime.findSession(modelId) is true, so the dead path poisoned every later attach of the resident model.
Repaired writeClaim re-creates the claim directory once on ENOENT and retries the record write; a second failure or any other errno stays claimIoError('write', …).
Probe src/lock/model-claim-recovery.test.ts now passes (2/2) in isolation: the vanished-directory case writes the record again with the same claim_id and state 'ready', and the claims-folder-replaced-by-a-file case still rejects with AtomicCoreError code IO_ERROR.
Controller full-suite output: 1 failed / 671 passed files, 11816 passed tests — the single failure is src/core/create.test.ts:1050, message 'the abandoned job was not cancelled'.
src/core/create.test.ts declares vi.setConfig({ testTimeout: 20_000 }) with a comment that these whole-core tests failed at random under the full-suite's parallel load (T-282), and the failing assertion is an inner 10 s poll loop, independent of the model-claim code.
The added patch branch executes only when a claim record write fails with ENOENT; no test in the suite deletes a live claim directory except the new probe, so diffusion job cancellation sees byte-identical behaviour.
New ADR docs/decisions/2026-11-16-re-establish-a-model-claim-whose-directory-vanished.md documents the decision and is linked by the single bullet at docs/decisions/INDEX.md:261.

Regression tests: src/lock/model-claim-recovery.test.ts

Full CI and human review are required.

Report: http://127.0.0.1:8080/tenants/707d0855-80ab-4e1f-a156-f1c4546cbf52/runs/0a81fd6c-094f-4449-8e48-cb02f1b6c96e

Sentry: https://atomic-chat.sentry.io/issues/7849148048/

@Ooooze
Ooooze merged commit 212d2eb into main Nov 18, 2026
15 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant