Skip to content

fix(sessions): stop orphaning sessions at their writers and repair existing orphans (#6144) - #6640

Merged
5 commits merged into
mainfrom
fix/6144-session-orphans
Sep 28, 2026
Merged

5 commits merged into
mainfrom
fix/6144-session-orphans

Conversation

@Hmbown

@Hmbown Hmbown commented Sep 26, 2026 •

Copy link
Copy Markdown
Owner

Closes #6144
Refs #6639 (follow-ups split out of this PR)

What orphans sessions, and where each one is fixed

The session document is the one authority for a conversation. Each of the two links has one home:

  • document → Runtime store: metadata.runtime_store
  • Runtime thread → document: ThreadRecord.session_id plus its checkpoint

A directory name does not say who owns it. On one real store, 0 of 82 documents were bound to the store under their own id. 79 were bound to a store under another id's directory, and 78 of the "document-less" store directories were live bindings. The host's boot anchor store holds every conversation that process saved. Any cleanup keyed on "sessions/<id>/runtime with no <id>.json" would have set aside live stores. Every rule here is keyed on the reference set instead: all document and checkpoint bindings, plus every thread's session_id and derived id.

Path Cause Fix
P1a /load, /resume and switches rebind a document onto this host's store and leave the old store on disk with nothing pointing at it The abandoned store is set aside right there, off the UI runtime. If another document binds it, it is kept
P1b A host that exits without binding its own store On exit, the store is retired if unbound. If anything in the process still holds it, the next launch's repair handles it by the same rule
P2 remove_session looked only at sessions/<id>/runtime, never at the store the document is actually bound to. It also deleted runtime-recovered-* stores under that directory (via remove_dir_all), even when another document was bound to them It follows the binding and sets the store aside unless another document binds it or it holds work. runtime-recovered-* directories are never deleted
P3 POST /v1/sessions created a new document on every call, then bound the thread in a second write Idempotent: it writes only the document whose id is derived from the thread, creating it (201) and updating it after (200), so a retry after a crash between the two writes binds the document it already wrote. A session the thread was resumed from is never rewritten. PUT with a session_id that another thread is bound to returns 409
P4 The checkpoint fingerprinted the whole document, so any legitimate append (a TUI autosave of the same id, a later PUT) stranded the thread messages_len plus a hash of that prefix. The thread keeps its prefix plus its own turns. Legacy checkpoints migrate on read in one O(n) pass that serializes each message once, and the length is written back. The thread unbinds, with a receipt, only when the document is missing or has really diverged; it then loads from its own turns. resume-thread reuses an open thread only when its checkpoint covers the whole document; a document that grew since gets a fresh thread holding all of it
P4 The live-writer guard was in-process only An open session holds an OS lock on .late-usage/<id>.live, separate from the per-write lock. PUT and DELETE return 409 if the session is open in any process
P5 Deleting a document left threads saying "Cannot read saved session" DELETE unbinds the threads that referenced it. A missing document now also unbinds at read time, and repair unbinds threads in unheld stores
P7 One corrupt thread record failed the whole thread list The record is skipped and reported. A newer schema is still refused
P8 The read-modify-write of session_boot_owners.json was not locked Now locked across processes
P9 Runtime-thread engines ran with session_id: None, so each engine load used a new random id. Compaction transfers (context-transfer-*.json, 411 of the stray artifact files measured), background-shell evidence and spills went to directories that no document names, and the thread lost them at its next load The engine runs as thread.session_id, or the id derived from the thread, which is the same id export uses

Repair on load (session_reconcile.rs)

It runs once the TUI host holds its own store, so that store and any store it resumed or recovered into count as in use and are never candidates. It also runs at codewhale serve start and from codewhale doctor --repair-sessions [--dry-run].

  • It is single-flight (.reconcile.lock) and bounded (500 actions per run; the next run continues).
  • It never deletes. Items move to sessions/.set-aside/<run>/, and MANIFEST.jsonl records each one's original path and reason, plus a sha256 for documents. Actions are logged in sessions/.reconcile/receipts.jsonl, and the summary is written to .reconcile/last.json.

What each rule does:

  • R4 also runs in an unbound store, before R3: a thread naming a missing document (for example one deleted from the session picker) is unbound with a receipt, then recovered.
  • R1 A document that cannot be read is set aside. A newer-schema document is left in place and reported.
  • R2/R5 A store no document binds, holding no work, is set aside while its process-owner lock is held, so no host can open it mid-move. A store held by another process is skipped and counted as in use.
  • R3 A store no document binds that holds threads with turns gets one Recovered: … document per thread, bound to that store, and each thread is bound back to its document.
  • R4 A thread whose document is gone is unbound.
  • R6 A directory with no document and no store, holding only artifacts or approval receipts, is set aside when no document, thread or derived thread id names it.

Surfaces: a one-line toast when something changed; codewhale doctor (human output, and a sessions.last_repair key in JSON); GET /v1/sessions/repair.

Before/after on a copy of one real ~/.codewhale/sessions

The copy was 447 MB, with bindings rebased onto the copy. codewhale doctor --repair-sessions:

dry run run 1 run 2
stores set aside (empty, unbound) 139 139 0
sessions recovered (threads in an unbound store) 4 4 0
artifact dirs set aside 128 128 0
artifact dirs kept (named or recent) 51 51 51
unreadable / newer-schema documents 0 / 0 0 / 0 0 / 0

After run 1 there were 85 documents, 83 of them bound. Every bound store is still in place except one binding that was already missing in the original store (it is not in the manifest). All 267 moves are recorded in the manifest.

Where this differs from the approved design, and why

  • P9 needed no trace commit. Reading the code together with the measurement pinned the writer (above). The "rename the pre-sync directory at first save" step is not needed.
  • The derived id is sha256("codewhale-thread-session-v1:" + thread_id) as a UUIDv8, not uuid_v5, which avoids a new uuid feature and dependency.
  • Receipts for a thread's dropped binding live in that thread's store (session-unbind-receipts.jsonl), not in sessions/.reconcile. The thread manager does not know the sessions directory, and writing to the default one leaked into a developer home during tests. The file is outside the store's work directories, so it never makes a store look busy.
  • R6 keeps a 24h idle window. The mtime heuristic is gone for stores, which the owner lock makes exact. An artifact directory has no lock that proves nobody is still writing to it (an older build, for example).
  • Legacy checkpoint migration happens on read, not in R4. A legacy session_id-only link gets a real checkpoint when the projection matches. When it does not, the thread loads from turns, and it unbinds only if the stored record still carries that same link.
  • list_sessions still leaves unreadable documents out of the list. R1 sets them aside at launch and reports them, instead of changing the return type for every caller.
  • Boot repair runs inside run_tui after TaskManager::start, not in the pre-TUI janitor. This closes the race between the janitor and a launch opening a runtime-recovered-* store.
  • Dropped as agreed and filed in Sessions follow-up: StateStore as a view, pre-fix duplicate exports, offline-queue lock sweep #6639: the state.db view or retirement, archiving pre-fix duplicate exports. Also filed there: the checkpoints/*.offline_queue.lock leak found during measurement (about 3.8k files).

Note for the founder: the #6094 release-plan text and CURRENT_DECISIONS §18 still describe a report-only doctor. §18 is updated locally in codewhale-ops (see below). #6094 needs your edit, since agents do not comment.

Verification (run locally)

  • CARGO_BUILD_JOBS=3 scripts/dev-cargo.sh nextest run -p codewhale-tui --locked -E 'test(runtime_threads::) | test(runtime_api::tests::session) | test(runtime_api::tests::thread) | test(snapshot) | test(session_control) | test(task_manager::) | test(session_picker) | test(doctor) | test(session_reconcile::) | test(runtime_store_binding) | test(picker_adopt) | test(session_manager::) | test(live_session) | test(delete)': 989 run, 989 passed
  • cargo fmt --all -- --check: clean
  • scripts/dev-cargo.sh clippy -p codewhale-tui --all-targets --all-features --locked -- -D warnings -A clippy::uninlined_format_args -A clippy::too_many_arguments -A clippy::unnecessary_map_or: clean
  • check-blocking-calls-budget.py: within budget after --update, which adds 3 canonicalize sites in session_reconcile.rs. That module is synchronous by design: its callers run it on spawn_blocking, from the doctor CLI, or on the already-synchronous delete path.
  • check-dead-code-budget.py, check-command-crate-boundaries.py, split/module_graph.py --check: pass
  • CHANGELOG gates: sync-changelog.sh, derive-changelog, derive-facts, derive-install, check-versions.sh --range-audit-advisory, check-contributor-credit.py v0.10.0, and vitest lib/public-copy.test.ts (6 passed)

New tests cover:

  • the P1a abandoned store being retired, and kept when another document binds it
  • P2 binding-following delete: shared store kept, busy store kept, a recovered store bound elsewhere never deleted
  • P3 idempotent export and crash-retry, plus the 409 on another thread's session
  • P4 prefix checkpoint, legacy migration, and prefix search matching serde
  • P4 lease-held DELETE returning 409
  • P5 unbind on delete and on a missing document
  • P7 a corrupt thread record
  • P8 concurrent boot-owner stamps
  • P9 the engine's session id
  • R1, R2 (the measured "bound to another id's directory" shape, and a held store), R3 with the second run a no-op, R4, R6, the per-run limit, single-flight, and the notice and doctor text

Moving a held store (set-aside and retire)

HeldRuntimeStore::move_to holds the store's owner lock for the whole move and unlinks nothing. The store directory is not renamed, because Windows refuses to rename a directory with an open handle inside it. Each entry is renamed into the destination instead, with the lock file last, while the lock is still held. std opens files with FILE_SHARE_DELETE, and CreateFileW documents that delete access "allows both delete and rename operations", so this rename works on Windows too. The emptied source directory is then removed, still under the lock. If the directory is no longer empty, an opener created a fresh store there after the lock file moved; that store belongs to the opener and stays. The lock is dropped last.

An earlier revision unlinked the lock file after dropping it. On unix, remove_file succeeds even while another process holds an flock, so a gap opener could end up holding a lock on an unlinked inode while a second opener took a fresh lock. That gave two owners. One narrower race remains: an opener can open the old path before the move and lock after it. RuntimeProcessOwnerLock::acquire/try_acquire_file now check after locking that the locked file is still the one at the path, and reopen it if not. Two tests cover this: opener_in_the_set_aside_gap_is_the_only_owner and opener_that_locks_a_moved_lock_file_reopens_its_path. The second fails when the identity check is stubbed out.

Focused run after this change: 337 run, 337 passed (adoption_refusal, session_reconcile, runtime_store_binding, task_manager, runtime_threads and related filters). fmt and CI-flag clippy are clean. Not verified locally: Windows. The Windows CI job is the proof.

🤖 Generated with Claude Code

@Hmbown Hmbown added this to the v0.10.1 milestone Sep 26, 2026
Copilot AI lite review requested due to automatic review settings September 26, 2026 13:05
@chatgpt-codex-connector

Copy link
Copy Markdown

You have reached your Codex usage limits for code reviews. You can see your limits in the Codex usage dashboard.
To continue using code reviews, add credits to your account and enable them for code reviews in your settings.

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Copilot was unable to review this pull request because the user who requested the review has reached their quota limit.

Comment thread crates/tui/src/runtime_api/tests.rs Fixed
Comment thread crates/tui/src/runtime_api/tests.rs Fixed
Hmbown pushed a commit that referenced this pull request Sep 26, 2026
…verage threads, recover orphans in unbound stores (#6144)

Review follow-ups on #6640:

- POST /v1/sessions writes only the document derived from the thread id.
  A thread opened with resume-thread is bound to the original saved
  session; updating that in place from the thread's projection would drop
  content the projection does not carry. Re-export and crash retry stay
  idempotent on the derived id.
- resume-thread reuses an open thread only when its checkpoint covers the
  whole document. A document that grew after the bind (TUI autosave) gets
  a fresh thread holding all of it, instead of a thread missing the
  appended messages.
- Reconcile applies R4 to unbound stores too: threads naming a missing
  document are unbound with a receipt, then recovered, instead of keeping
  the store forever.

Tests: focused nextest (session_reconcile, resume/export/fork filters)
126 passed, 0 failed, including 3 new tests. cargo fmt --check clean;
clippy -p codewhale-tui --all-targets --all-features with CI flags clean;
blocking-calls, dead-code, command-crate-boundaries and module_graph
gates pass.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Comment thread crates/tui/src/runtime_api/tests.rs Dismissed
Comment thread crates/tui/src/runtime_api/tests.rs Dismissed
@Hmbown
Hmbown marked this pull request as draft September 26, 2026 17:54
@Hmbown
Hmbown marked this pull request as ready for review September 26, 2026 18:16
@chatgpt-codex-connector

chatgpt-codex-connector Bot commented Sep 26, 2026 •

Copy link
Copy Markdown

Codex Review Summary

This comment shows the latest Codex review activity on this pull request.

Review Status Commit Review trigger
📝 Code Review ✅ Completed 2026-09-26T18:27:14.050848Z a507c4e Draft marked ready
ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review" or "@codex security review".

Codex reacts with 👀 while any review is running, comments if it has suggestions, and reacts with 👍 once all reviews finish with no findings.

@chatgpt-codex-connector chatgpt-codex-connector Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: a507c4e1c8

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

Comment thread docs/RUNTIME_API.md
Comment on lines +682 to +686
autosave. `PATCH`, `PUT` and `DELETE` therefore fail closed on it with
`409 Conflict` rather than writing something that would be silently reverted.
Change it in the terminal instead. The open process holds a lock on the
session (`sessions/.late-usage/<id>.live`), so this holds whether the request
reaches the API inside that process or a separate `codewhale serve`.

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

P1 Badge Enforce the live lease for PATCH requests

When the API runs in a separate codewhale serve process, patch_session still calls rename_session and set_session_archived, which consult only the process-local is_live_session registry rather than the new cross-process lease. Consequently, a PATCH /v1/sessions/{id} can race an interactive TUI autosave and be silently overwritten even though this documented contract promises a 409 Conflict; the PATCH path must check is_session_live_anywhere as PUT and DELETE do.

Useful? React with 👍 / 👎.

Comment on lines +737 to +742
if references.binds(&store.path) {
// R4 for a bound store nobody holds.
if self.options.dry_run {
return;
}
match held.unbind_threads_without_documents(exists) {

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

P1 Badge Retry binding documents left by interrupted recovery

If repair stops after recovered_document saves the derived session but before bind_recovered_thread updates the thread, the next run sees that document's runtime_store binding and enters this branch. It then only unbinds references to missing documents, so the still-unbound thread never reaches recover_store and the retry logic at recovered_document is unreachable; the thread remains permanently detached and later turns are not reflected in the recovered document.

Useful? React with 👍 / 👎.

Comment on lines +628 to +631
if self.options.skip_session.as_deref() == Some(id.as_str())
|| SessionManager::load_session_metadata(&path).is_ok()
{
continue;

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

P2 Badge Validate the full document before marking it readable

For a session file whose metadata object is intact but whose later messages or tool_log JSON is truncated or malformed, load_session_metadata succeeds because it intentionally parses only the metadata prefix. This early continue therefore bypasses classify_unlisted_document's full SavedSession deserialization, leaving the corrupt document listed and counted as a valid store reference while every actual resume fails; R1 never sets it aside.

Useful? React with 👍 / 👎.

Comment on lines +739 to +740
if self.options.dry_run {
return;

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

P2 Badge Simulate unbinding when producing a dry-run plan

With --dry-run, a bound store containing a thread whose referenced document was deleted returns here without even counting the pending unbind, while a real run increments threads_unbound. The analogous skip in recover_store also leaves such threads marked bound, causing recoverable_threads to omit the sessions that the real run would recover, so the command advertised as a repair preview substantially under-reports its planned actions.

Useful? React with 👍 / 👎.

Comment on lines +874 to +877
let target_id = req
.session_id
.clone()
.unwrap_or_else(|| snapshot.session_id.clone());

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

P1 Badge Update the derived target instead of recreating it

When PUT /v1/sessions omits session_id for a resumed thread, the new engine configuration makes snapshot.session_id equal the existing bound document ID, but target_id is used only for the lease check. The construction below still enters the req.session_id == None branch and creates a blank SavedSession over that ID instead of loading and updating it, which discards the existing title, archive/fork metadata, artifacts, work state, and runtime_store binding; the latter recreates the orphan this change is intended to prevent.

Useful? React with 👍 / 👎.

Comment on lines +2664 to +2673
let thread: ThreadRecord = match read_store_file(&path)
.with_context(|| format!("Failed to read {}", path.display()))
.and_then(|raw| {
serde_json::from_str(&raw)
.with_context(|| format!("Failed to parse {}", path.display()))
}) {
Ok(thread) => thread,
Err(error) => {
skipped.push(format!("{error:#}"));
continue;

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

P1 Badge Read the schema version before skipping parse failures

A valid record written by a newer Codewhale can fail ThreadRecord deserialization when that schema changes an enum or field shape. Because this branch classifies every deserialization failure as a skippable corrupt record before inspecting schema_version, such a future record disappears from thread listings instead of triggering the promised newer-schema refusal, allowing an older binary to operate while silently hiding work it cannot understand.

Useful? React with 👍 / 👎.

fs::remove_dir(session_dir)?;
}
}
self.retire_released_stores(id, bound_store);

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

P1 Badge Keep store retirement off Tokio handlers

This newly added call synchronously scans every saved document, canonicalizes paths, inspects automations, and may move a whole Runtime store. SessionManager::delete_session is called directly by the async Runtime API delete handler and by the TUI event path, so a large session store now blocks the Tokio/UI thread during deletion; schedule the retirement work through spawn_blocking or another dedicated worker.

AGENTS.md reference: AGENTS.md:L164-L170

Useful? React with 👍 / 👎.

Comment on lines +135 to +145
parts.push(format!("{} recovered", self.sessions_recovered));
}
if self.threads_unbound > 0 {
parts.push(format!("{} threads re-linked", self.threads_unbound));
}
let set_aside =
self.stores_set_aside + self.artifact_dirs_set_aside + self.documents_set_aside;
if set_aside > 0 {
parts.push(format!("{set_aside} unused items set aside"));
}
let mut line = format!("Sessions repaired: {}", parts.join(", "));

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

P1 Badge Route the repair toast through localization

The result of ReconcileSummary::notice is displayed directly as a TUI status toast, but all of its newly added user-visible fragments are hard-coded English. Non-English users therefore receive an untranslated launch notification; define message IDs with parameters for the counts/path and construct the toast through tr(locale, MessageId::...).

AGENTS.md reference: crates/tui/AGENTS.md:L25-L26

Useful? React with 👍 / 👎.

Comment on lines +864 to +869
let mut session = crate::session_manager::create_saved_session_with_id_and_mode(
id,
&recovered.messages,
&thread.model,
&thread.workspace,
0,

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

P1 Badge Preserve usage when rebuilding recovered sessions

Every R3 recovery passes a hard-coded zero token count and leaves the new session's cost snapshot at its default even though the recovered thread's durable turn records contain usage and route settlement data. A normal orphan with completed paid turns therefore reappears with zero tokens and zero historical cost, and resuming/autosaving that document makes the incorrect accounting durable; aggregate the recovered turns using the same usage projection used by Runtime API exports.

Useful? React with 👍 / 👎.

Comment on lines 998 to +1001
let sessions_dir = default_sessions_dir().unwrap_or_else(|_| fallback_sessions_dir());
// Repair the saved-session store once per server start (#6144); this
// server's own store is open by now, so it reads as in use.
crate::session_reconcile::spawn_background_reconcile(None);

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

P2 Badge Reconcile the same fallback store the server exposes

If default_sessions_dir() fails, the server deliberately continues with fallback_sessions_dir() for all session endpoints, but spawn_background_reconcile independently calls SessionManager::default_location() and silently returns on that same failure. In exactly the environment that selects the fallback, codewhale serve therefore never repairs the store it is serving; pass the selected sessions_dir into the worker and surface initialization failures rather than skipping repair.

AGENTS.md reference: AGENTS.md:L79-L81

Useful? React with 👍 / 👎.

CodeWhale Bot and others added 4 commits September 26, 2026 19:44
…isting orphans (#6144)

The session document is the one authority for a conversation. The
document-to-store link lives only in metadata.runtime_store, and the
thread-to-document link lives only in ThreadRecord.session_id plus its
checkpoint. Directory names are not ownership.

Fixed at the writers:
- P1a: a switch that rebinds a conversation to this host's store sets aside
  the empty store it left, unless another document binds it.
- P1b: on exit, the host retires its own store if nothing binds it.
- P2: remove_session follows the binding instead of sessions/<id>/runtime.
  It also no longer deletes runtime-recovered-* stores, which another
  document may bind.
- P3: POST /v1/sessions is idempotent. It reuses thread.session_id, or an id
  derived from the thread, so a retry after a crash binds the document that
  was already written. PUT with another thread's session_id returns 409.
- P4: checkpoints cover a prefix (messages_len), so appends to the document
  no longer strand the thread. Legacy checkpoints migrate on read in one
  O(n) pass. A thread unbinds, with a receipt, only when the document is
  missing or has diverged.
- P4: a cross-process live lease (.late-usage/<id>.live). PUT and DELETE
  refuse a session that is open in any process.
- P5: DELETE unbinds the threads that referenced the document.
- P7: one corrupt thread record no longer fails list_threads.
- P8: session_boot_owners.json read-modify-write is locked.
- P9: Runtime-thread engines write under the thread's session id. They
  previously used a random id on each load.

Repair on load (new session_reconcile.rs): runs at TUI launch after the
host holds its store, at codewhale serve start, and from
`codewhale doctor --repair-sessions [--dry-run]`. It is bounded,
single-flight, and moves items aside with a manifest; it never deletes.

Tests: 989 run, 989 passed (nextest, focused on runtime_threads,
runtime_api session/thread, session_manager, session_reconcile,
runtime_store_binding, snapshot, task_manager, doctor and delete filters).
Clippy (CI flags) is clean on codewhale-tui. fmt is clean. The blocking-call
budget was raised by 3 canonicalize sites in the synchronous session_reconcile
module.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
…verage threads, recover orphans in unbound stores (#6144)

Review follow-ups on #6640:

- POST /v1/sessions writes only the document derived from the thread id.
  A thread opened with resume-thread is bound to the original saved
  session; updating that in place from the thread's projection would drop
  content the projection does not carry. Re-export and crash retry stay
  idempotent on the derived id.
- resume-thread reuses an open thread only when its checkpoint covers the
  whole document. A document that grew after the bind (TUI autosave) gets
  a fresh thread holding all of it, instead of a thread missing the
  appended messages.
- Reconcile applies R4 to unbound stores too: threads naming a missing
  document are unbound with a receipt, then recovered, instead of keeping
  the store forever.

Tests: focused nextest (session_reconcile, resume/export/fork filters)
126 passed, 0 failed, including 3 new tests. cargo fmt --check clean;
clippy -p codewhale-tui --all-targets --all-features with CI flags clean;
blocking-calls, dead-code, command-crate-boundaries and module_graph
gates pass.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
… set them aside (#6144)

HeldRuntimeStore::move_to renamed the store directory while holding the
open owner-lock file inside it. Windows refuses to rename a directory with
an open handle beneath it, so every set-aside and retire failed there
(five session_reconcile / runtime_store_binding tests red on
Test (windows-latest)). Now each entry except the lock file is renamed
into the destination while the lock is still held (renaming siblings of an
open file is allowed on every platform), a partial failure moves entries
back, and only after the lock drops are the lease file and the emptied
directory removed.

The session export retry test now addresses DELETE by the session id the
server returned instead of the SHA-derived thread_session_id, which
removes the taint source behind the two CodeQL cleartext-transmission
alerts in runtime_api/tests.rs.

Tests (macOS): cargo test -p codewhale-tui --lib -- session_reconcile::
runtime_store_binding:: session_export_retry_after_a_crash_binds_the_document_it_wrote
-> 25 passed, 0 failed. cargo fmt --check clean; clippy -p codewhale-tui
--all-targets --all-features with CI flags clean. Windows proof comes from
the Windows CI job.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
…e move (#6144)

a4abe8a moved every entry except the owner-lock file, dropped the lock,
then unlinked the lock file and removed the directory. On unix remove_file
succeeds while another process holds an flock on the file, so an opener
that locked in that gap held a lock on an unlinked inode; the directory was
recreated and a second opener took a fresh lock: two owners of one store.

HeldRuntimeStore::move_to now renames the lock file into the destination
too, last, while it is still held; removes the emptied source directory
while still holding it (a non-empty directory means an opener created a
fresh store there, which is its own and stays); and drops the lock last.
Nothing is unlinked. The Windows fix is kept: the directory is still not
renamed, and renaming the open, locked file is allowed there because std
opens files with FILE_SHARE_DELETE (library/std/src/sys/fs/windows.rs,
OpenOptions::new share_mode) and CreateFileW documents that delete access
"allows both delete and rename operations"; LockFile byte-range locks do
not block a rename.

Renaming the lock file leaves one narrower race: an opener that opened
the old path before the move and locks after it holds the moved file.
RuntimeProcessOwnerLock::acquire and try_acquire_file now check, after
locking, that the locked file is still the one at the path (dev/inode, or
volume serial/file index on Windows) and reopen the path when it is not.

Tests: opener_in_the_set_aside_gap_is_the_only_owner takes the lock at
the old path after the lock file moved and before the lock drops, and
proves a second opener is refused and the moved store's lock is free.
opener_that_locks_a_moved_lock_file_reopens_its_path runs a whole
set-aside between an opener's open and lock; with the identity check
stubbed out it fails (second opener also acquires), with it it passes.

Verification (macOS):
- scripts/dev-cargo.sh nextest run -p codewhale-tui --lib --locked -E
  'test(adoption_refusal::) | test(session_reconcile::) |
  test(runtime_store_binding) | test(task_manager::) | test(owner_lock) |
  test(live_session) | test(picker_adopt) | test(process_owner) |
  test(session_export_retry) | test(runtime_threads::)': 337 run, 337 passed
- cargo fmt --all -- --check: clean
- clippy -p codewhale-tui --all-targets --all-features --locked with CI
  flags: clean
- check-blocking-calls-budget.py, check-dead-code-budget.py,
  check-command-crate-boundaries.py: pass
Windows: not run locally; the Windows CI job is the proof.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
@Hmbown
Hmbown force-pushed the fix/6144-session-orphans branch from a507c4e to cf42682 Compare September 27, 2026 02:55
…egacy_root

main (#6641) removed Config's top-level api_key/base_url fields, so the
session_reconcile fixture no longer compiled on CI (E0560).

Checks: cargo test -p codewhale-tui --lib -- session_reconcile:
11 passed; 0 failed.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Hmbown pushed a commit that referenced this pull request Sep 27, 2026
Reconciles #6640 with #6645 (already merged here):
- Engine identity: #6645's rule stands (every Runtime thread's engine runs
  under thread.id). #6640 had also set the field to the bound document or a
  derived UUID; git kept both assignments, so the duplicate is dropped.
- thread_session_id(thread_id) now returns the thread id itself, so the
  export document (POST /v1/sessions), the engine's session directory and
  the thread name one conversation, and the export stays idempotent. The
  derived-UUID scheme never shipped.
- save_current_session keeps #6640's guards (409 when naming another
  thread's document; live-session refusal) and #6645's target choice (the
  bound document, else the thread's own id).

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Hmbown pushed a commit that referenced this pull request Sep 27, 2026
Conflict in RuntimeProcessOwnerLock::acquire (runtime_threads.rs): #6640
wrapped the open+flock in a retry loop that re-opens when a maintenance
move renamed the lock file (OWNER_LOCK_MOVED_RETRIES, owner_lock_is_at_path),
while #6601 changed the contention-deadline exit from an untyped bail! to an
io::Error of kind WouldBlock so session_secret_scrub can skip a busy
Runtime store and report it instead of failing the whole scrub. Kept
#6640's retry structure and applied #6601's typed error to both "held"
exits (deadline and moved-retries exhausted) through a small held_error()
helper, so the scrub's WouldBlock downcast sees every held case.

scripts/check-blocking-calls-budget.json: both sides added a new file
entry at the same spot (session_reconcile.rs from #6640,
session_secret_scrub.rs from #6601); kept both, sorted.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Hmbown pushed a commit that referenced this pull request Sep 27, 2026
Each fix below reconciles two PRs that merged textually but not
semantically.

- snapshot/repo.rs, receipts.rs: #6645/#6682 added
  SnapshotRepo::changed_paths_between(from, to) -> Vec<PathBuf> (undo,
  turn artifacts) and #6591 added a different
  changed_paths_between(from, to, limit) -> (Vec<SnapshotPathChange>, bool)
  (receipts). Duplicate definition; #6591's is renamed
  path_changes_between and its one caller (receipts) updated.
- core/engine/turn_loop.rs: #6673's repl-fence approval match did not
  cover #6601's ApprovalResult::TimedOut. A timed-out card now refunds the
  tool-call budget slot and reports a timeout, like direct and code-mode
  calls; the audit line records "timeout" instead of "denied".
- runtime_api/sessions.rs: #6640's session-owner 409 predates #6645's
  ApiError.code field; code: None.
- tools/verifier.rs: #6671's env-scrub test called run_gate(gate) without
  the session_id argument run_gate takes on main (#6508).
- skills/install.rs + integration harness: #6679 made install.rs read
  downloads through crate::utils::read_response_body_capped, but the
  integration harness #[path]-includes install.rs and has no utils
  module, so the integration test target did not compile (also on the
  #6679 branch). The capped reader moves to utils/response_body.rs
  (re-exported from utils, unchanged API) and the harness includes just
  that file as crate::utils.
- Test files where an add-only conflict was auto-resolved by
  concatenating both sides lost the shared closing lines of the first
  test: commands/groups/debug/tests.rs (#6682 + #6591), tui/ui/tests.rs
  (#6635 + main), tools/shell/tests.rs (#6674 + #6679), and
  runtime_api/tests.rs (#6645 merge in round 1). Restored the missing
  `}` / `);` so each test is whole again; no assertions were dropped.
- tools/shell/tests.rs: #6637's executor test expected `sort * | cat`
  to be admitted and then fail at run time; with #6675's rule ported into
  the #6637 lexer (see the #6675 merge), a word-leading unquoted `*` is
  refused before anything runs. The test now asserts that refusal and
  still checks the sentinel and option-named files are untouched.
- core/engine/tests.rs -> tui/history/tests.rs: #6601's engine test
  asserted crate::tui::history on the trust warning, raising the
  runtime->UI test reference ratchet 40 -> 41 (check-command-crate-
  boundaries FAIL). That assertion moved to a tui::history test on
  workspace_trust_runtime_message, so the ratchet is back at 40.
- scripts/check-blocking-calls-budget.json: runtime_api/git.rs 5 -> 6.
  #6648 justified this budget in its PR body (working-tree fingerprint
  reads in sync fns reached only from spawn_blocking); its own branch
  already has six such sites (the File::open used for hashing), so the
  recorded 5 was stale. subagent/worktree.rs tightened 7 -> 6.

Checks: cargo check --workspace --tests clean (no warnings);
cargo test -p codewhale-execpolicy 221+1+5+7+1 passed;
cargo test -p codewhale-tui --test integration 187 passed.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
@Hmbown Hmbown closed this pull request by merging all changes into main in 0bfe04e Sep 28, 2026
@Hmbown
Hmbown deleted the fix/6144-session-orphans branch September 28, 2026 08:40
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Session persistence: decide whether session_manager or codewhale-state owns session truth

3 participants