[myles332]

Closes OR-564. This is phase 4 of OR-569. It is stacked on #757, which is stacked on #754; retarget it to main as those merge.

Summary

• CLI: orx exp delete <expId> permanently deletes an experiment node. It removes:

  • its run rows
  • their local run artifacts (logs, run dirs, submission handles, and source snapshots no other run uses)
  • its orx/<slug> git branch

  Its SSH/Slurm runs' remote dirs are queued for full removal through Clean up older remote run dirs and unused source tars (OR-575) #757's cleanup queue. The command prints the deleted branch's commit with a git branch <b> <sha> restore hint.

• API and dashboard: DELETE /api/experiments/{id} backs a new "Delete permanently…" item in the experiment menu. It uses danger styling and a confirm dialog, and only appears for nodes without children, matching the existing archive menu.

• Refusals (conservative): the delete is refused, with nothing changed, when:

  • the node has child experiments (no re-parent or cascade flag; use orx exp move first)
  • any of its runs is non-terminal, or a terminal run's job or supervisor is still winding down
  • git refuses to delete the branch because a worktree (a chat session's or the main checkout) has it checked out

• Branches that are never deleted: the project's base branch (legacy roots) and a branch another experiment uses.

• Order and retry:

  1. All checks run under the exclusive run-artifacts.lock.
  2. The branch is deleted.
  3. Local artifacts are removed.
  4. Rows and the remote-cleanup queue entry go in one transaction that rechecks for children.

  A failure after the branch is gone still reports its restore commit, and retrying completes the delete.

• Store refactor: project delete and experiment delete now share the artifacts-lock wait (while_deleting) and run-id-scoped artifact cleanup. Project-delete behaviour is unchanged.

• Docs: orx-experiment-tree and the root skill table say to use delete only when the user asks, and archive otherwise.

Judgment calls

• orx create-experiment and orx exp move now hold run-artifacts.lock shared, so they wait (up to 60s) while an experiment or project delete or an orphan scan holds it. They then refuse if the project or parent was deleted meanwhile ("… no longer exists"), or fail with "A deletion or artifact cleanup is in progress; retry shortly."
• Children are refused rather than re-parented or cascaded.
• The GitHub copy of a pushed branch is left alone, since deleting it is outward-facing.
• Open dashboard tabs for the deleted node aren't closed; they show the existing "unavailable" state.
• Another window learns of a CLI delete on its next refetch, since there is no experiment.deleted SSE event yet.

Test plan

• cargo fmt --all --check, cargo clippy --all-targets -- -D warnings, cargo build --locked, cargo test --locked
• UI: i18n check, paraglide, tsc --noEmit, vite build (ui/dist committed), node --test (405 pass)
• Tests with a real git repo:
  • a child refusal leaves the parent and its branch intact
  • a branch checked out in a worktree is refused, leaving rows, run and queue untouched
  • a leaf delete removes rows, runs and the branch, returns the commit and queues remote cleanup
  • a second delete errors
  • a legacy root keeps the base branch
• Store tests:
  • an in-flight run, or a held supervisor lock, is refused before the callback runs
  • the run's artifacts and unshared snapshot are removed
  • a sibling experiment's run and its shared snapshot survive
• Dashboard: delete a leaf from the tree and from the history table; the confirm shows, and the node and its runs disappear
• Delete a node whose branch a chat session has checked out; the error names the worktree
• Delete a node with a finished SSH/Slurm run; the remote dir is removed by the cleanup task

🤖 Generated with Claude Code

[greptile-apps]

Confidence Score: 4/5

[High impact] The change reaches local experiment records, files, branches, and remote cleanup.

Fix the race with experiment creation and reparenting before merging.

Findings

1.  Refused deletion still removes files ▶

Fix with agent prompt

### Issue 1
src/store.rs:2229-2232
Concurrent experiment creation or `orx exp move` can add a child after the first child check. Neither takes the artifacts lock. By the time the final check refuses deletion, `before_rows()` has deleted the branch and `cleanup_deleted_run_artifacts()` has removed the run files. Rolling back the rows cannot restore those files.

A child creation that forks before deletion and inserts after commit can also retain a deleted parent ID. Make creation, reparenting, and deletion share one lock, and recheck the parent while holding it, before deleting anything.

---

For each issue above, determine whether it is valid and should be fixed. If so, fix it directly.

Summary

Adds permanent experiment deletion through the CLI, API, and dashboard.

• A safe leaf experiment can be permanently removed with its local run data.
• The dashboard can permanently delete a leaf experiment after confirmation.

myles332 explicitly deferred deletion events and closing open dashboard tabs. Pushed GitHub branches are intentionally retained.

Diagram

%%{init: {'theme': 'neutral'}}%%
flowchart TD
  A[Delete experiment] --> B[Acquire artifacts lock]
  B --> C[Check children and runs]
  C --> D[Delete owned branch]
  D --> E[Remove local run files]
  E --> F[Begin transaction]
  F --> G[Queue remote cleanup and delete run rows]
  G --> H{Child now exists?}
  H -->|No| I[Delete experiment and commit]
  H -->|Yes| J[Roll back rows and return error]
  J --> K[Branch and files remain deleted]

Loading

_{Reviews (1) · Last reviewed commit: "Check run ids and artifact roots before ..." · Reviewed by Greptile}

[greptile-apps]

Refused deletion still removes files

Concurrent experiment creation or orx exp move can add a child after the first child check. Neither takes the artifacts lock. By the time the final check refuses deletion, before_rows() has deleted the branch and cleanup_deleted_run_artifacts() has removed the run files. Rolling back the rows cannot restore those files.

A child creation that forks before deletion and inserts after commit can also retain a deleted parent ID. Make creation, reparenting, and deletion share one lock, and recheck the parent while holding it, before deleting anything.

Prompt To Fix With AI

This is a comment left during a code review.
Path: src/store.rs
Line: 2229-2232

Comment:
**Refused deletion still removes files**

Concurrent experiment creation or `orx exp move` can add a child after the first child check. Neither takes the artifacts lock. By the time the final check refuses deletion, `before_rows()` has deleted the branch and `cleanup_deleted_run_artifacts()` has removed the run files. Rolling back the rows cannot restore those files.

A child creation that forks before deletion and inserts after commit can also retain a deleted parent ID. Make creation, reparenting, and deletion share one lock, and recheck the parent while holding it, before deleting anything.

---

For each issue above, determine whether it is valid and should be fixed. If so, fix it directly.

[myles332]

Fixed in 1344342 / 837d6e3. Experiment create and orx exp move now hold run-artifacts.lock shared for their whole operation (Store::with_run_artifacts_shared). Experiment and project deletes hold it exclusively from the child check through branch, artifact and row deletion, so a child can no longer appear mid-delete. Under the lock, create re-reads the project and the parent and forks from the fresh rows, so a create that waited behind a delete refuses with "no longer exists" instead of leaving a dangling parent. Delete also refuses if its row is already gone, so an overlapping second delete can't remove a reused branch. With every writer now excluded, the in-transaction child recheck became dead code and was removed. Covered by tree_edits_wait_out_deletes_and_recheck_what_they_build_on and the repeat-delete assertion in the store test.