diff --git a/changelog.d/fixed-18767-session-delete-background.md b/changelog.d/fixed-18767-session-delete-background.md new file mode 100644 index 0000000000..93edc2860c --- /dev/null +++ b/changelog.d/fixed-18767-session-delete-background.md @@ -0,0 +1 @@ +- **Deleting a session now runs in the background, shows its progress, and can no longer leave a session listed whose files are gone** ([#18767](https://github.com/hivecommons/hive/issues/18767)). Deleting an ended session looked like it did nothing: the request stayed open for about a minute with no indicator, then the dashboard said "could not remove the session's files" although the files had been removed, and the session stayed in the list. Three things caused it. The sandbox was deleted under the 10-second timeout meant for tmux calls, which a HOME with a CLI's caches outlasts. The cancel that the timeout triggered ran `su-exec root kill`, and the image has no `kill` binary, so the cancel itself failed. The hive then reported that cancel error over a removal that had finished, and kept the record. Now `DELETE /api/sessions/{id}` on an ended or failed session answers 202 at once and the session is `deleting`: the tab shows "Deleting…" with a spinner and no controls, and disappears by itself when the hive is done. One job per session removes the files under its own 15-minute ceiling and removes the record last; what is on disk decides the outcome, so an error from a command that left nothing behind is a success. A real failure puts the session back as it was with the reason on the tab and a **Retry delete** button. The login safety from #18728 is unchanged in what it protects: a session holding a strictly newer, usable Claude login that cannot be adopted is not deleted, and now says so ("this session holds the hive's current sign-in; adopting it failed: ..."); having nothing to adopt never blocks or slows a delete. The cancel of a privileged child now uses the shell's builtin `kill`, so a real timeout ends the child instead of waiting for it. Sessions already left half-deleted are repaired without a manual step: at the next sweep (boot, and every list) an `ended` record whose sandbox and worktree are both gone is dropped, and a delete click on one completes at once. Every delete outcome writes one log line with its duration and a `session_closed` audit entry, failures included. `DELETE ...?remove=true` on a running session closes and deletes it in one call. diff --git a/dashboard/openapi.json b/dashboard/openapi.json index 14373a6618..f74def1bad 100644 --- a/dashboard/openapi.json +++ b/dashboard/openapi.json @@ -45108,17 +45108,23 @@ "tags": [ "Sessions" ], - "summary": "Close a session (creator or owner)", - "description": "Kills tmux and keeps the metadata marked ended; on an ended session removes its metadata.", + "summary": "Close or delete a session (creator or owner)", + "description": "On a running session: kills tmux and keeps the metadata marked ended (with remove=true it is then deleted). On an ended or failed session: starts the delete and answers 202 at once with the session as deleting; the files are removed by a background job, the record goes when they are gone, and a failure puts the previous status back with delete_error.", "responses": { "200": { - "description": "OK" + "description": "Closed (running session), or removed=true: nothing was left on disk and the record is gone" + }, + "202": { + "description": "The delete is running in the background; body carries the session with status deleting" }, "403": { "description": "Not the creator or an owner" }, "404": { "description": "Unknown session" + }, + "409": { + "description": "The session is still being prepared, or changed state meanwhile" } }, "parameters": [ @@ -45129,6 +45135,15 @@ "schema": { "type": "string" } + }, + { + "name": "remove", + "in": "query", + "required": false, + "description": "true: delete a running session after closing it", + "schema": { + "type": "string" + } } ] } diff --git a/src/crux/agent/usersession_login.go b/src/crux/agent/usersession_login.go index 3563c834fd..48a2e01446 100644 --- a/src/crux/agent/usersession_login.go +++ b/src/crux/agent/usersession_login.go @@ -303,7 +303,7 @@ func (o sessionLoginOps) reconcileClaudeLogin(id string, mode sessionLoginMode) if sessionClaudeLoginNewer(sess, info.ModTime(), shared, fileModTime(target), now) { staleShared, perr := promoteClaudeCredential(data, target, now) if perr != nil { - return none, fmt.Errorf("adopting the session's newer login into the shared login: %w", perr) + return none, fmt.Errorf("%w; adopting it failed: %w", crux.ErrUserSessionHoldsLogin, perr) } res := sessionLoginResult{action: sessionLoginAdopted, staleShared: staleShared} if mode == sessionLoginModeRemove { diff --git a/src/crux/agent/usersession_priv.go b/src/crux/agent/usersession_priv.go index 8bf148cb86..fcc23e81e0 100644 --- a/src/crux/agent/usersession_priv.go +++ b/src/crux/agent/usersession_priv.go @@ -245,11 +245,20 @@ func newUserSessionPrivCmd(ctx context.Context, user string, stdin io.Reader, ar return cmd, nil } +// userSessionRootKillScript signals the process group $1 with KILL. `kill` is +// the shell's builtin: the image ships no kill binary, so `su-exec root kill` +// could never run ("su-exec: kill: No such file or directory"), the cancel +// failed, and Wait kept blocking until the child finished by itself and then +// reported the cancel's error over a command that had succeeded. The -s form +// is the one dash and bash both take with a negative pid; the second form is +// for a shell that only knows the signal-name flag. +const userSessionRootKillScript = `kill -s KILL -- "-$1" 2>/dev/null || kill -KILL "-$1"` + // killUserSessionRootGroup ends a root child (and everything it spawned, via // its process group) through the setuid helper, the only way the hive's uid // can signal a root process. func killUserSessionRootGroup(pid int, env []string) error { - kill := exec.Command(userSuExecBinary, userSessionRootSpec, "kill", "-KILL", "--", "-"+strconv.Itoa(pid)) + kill := exec.Command(userSuExecBinary, userSessionRootKillArgv(pid)...) kill.Env = env if out, err := kill.CombinedOutput(); err != nil { return outputErr("killing privileged child", err, out) @@ -257,6 +266,11 @@ func killUserSessionRootGroup(pid int, env []string) error { return nil } +// userSessionRootKillArgv is the su-exec argv killUserSessionRootGroup runs. +func userSessionRootKillArgv(pid int) []string { + return []string{userSessionRootSpec, "sh", "-c", userSessionRootKillScript, "sh", strconv.Itoa(pid)} +} + // privRun is r.priv bound to the standard command timeout. func (r *UserSessionRuntime) privRun(user string, stdin io.Reader, argv ...string) ([]byte, error) { return r.privRunFor(userSessionCmdTimeout, user, stdin, argv...) @@ -353,57 +367,3 @@ func (r *UserSessionRuntime) openSocketDirToAttacher(uid int) error { } return nil } - -// Remove ends the session and deletes everything it owned: its worktree of the -// shared workspace clone (uncommitted work is auto-saved onto its branch first, -// see userSessionWorktreeRemoveScript), the sandbox (home, work dir, uploads, -// tmp) and the uid's tmux socket dir, so a later session that is allocated the -// same uid starts clean. Directories earlier releases created for the id are -// removed too. A failure is returned and the caller must keep the session -// record (and its uid slot) so the removal is retried. -func (r *UserSessionRuntime) Remove(id string) error { - if err := checkUserSessionID(id); err != nil { - return err - } - uid, uidErr := r.sessionUID(id) - if uidErr == nil { - if err := r.killAs(uid, id); err != nil { - r.logger.Warn("user session kill before removal failed", "id", id, "error", err) - } - } - // The session's CLI is gone (killed above), so its login can no longer - // change. If that login is the newest one on this hive, it goes into the - // shared file BEFORE the sandbox is deleted: removing a session must never - // remove the only valid login. A login that cannot be judged or adopted - // keeps the sandbox, and the caller retries. - // (The CLI creates files and directories with private modes; the hive - // must be able to look at the login path to judge it.) - r.makeHomeReadable(id) - if err := r.syncClaudeLogin(id, sessionLoginTriggerRemove, sessionLoginModeRemove); err != nil { - return fmt.Errorf("keeping the session's login before removal: %w", err) - } - r.workspaceMu.Lock() - err := r.removeWorktrees(id) - r.workspaceMu.Unlock() - if err != nil { - return fmt.Errorf("removing session worktree: %w", err) - } - dir := userSessionSandboxDir(id) - if _, err := os.Lstat(dir); err == nil { - if out, err := r.privRun(userSessionRootSpec, nil, "rm", "-rf", "--one-file-system", "--", dir); err != nil { - return outputErr("removing session sandbox", err, out) - } - } - if uidErr == nil { - sock := userSessionTmuxSocketDir(uid) - if out, err := r.privRun(userSessionRootSpec, nil, "rm", "-rf", "--one-file-system", "--", sock); err != nil { - return outputErr("removing session tmux socket dir", err, out) - } - } - for _, p := range legacyUserSessionPaths(id) { - if err := os.RemoveAll(p); err != nil { - return fmt.Errorf("removing legacy session dir %s: %w", p, err) - } - } - return nil -} diff --git a/src/crux/agent/usersession_remove.go b/src/crux/agent/usersession_remove.go new file mode 100644 index 0000000000..9655b8611f --- /dev/null +++ b/src/crux/agent/usersession_remove.go @@ -0,0 +1,135 @@ +package agent + +import ( + "errors" + "fmt" + "io/fs" + "os" + "time" + + "github.com/hivecommons/hive/crux" + "github.com/hivecommons/hive/crux/sharedfile" +) + +// Removing a session's files. +// +// The failure this closes (#18767). The sandbox was deleted with the +// 10-second userSessionCmdTimeout that bounds a tmux call. A HOME holding a +// CLI's caches takes longer than that to delete (a minute was seen), so the +// timeout fired, the cancel of the root child could not run (see +// userSessionRootKillScript), `rm` finished by itself, and the caller was +// handed the cancel's error for a removal that had succeeded: the files were +// gone and the session record was kept. +// +// Two rules follow. A removal has its own, generous deadline. And what decides +// whether a tree was removed is the tree: a command that reports an error +// after the directory is gone has removed it. + +// userSessionRemoveTimeout bounds deleting one session tree (the sandbox, the +// tmux socket dir). It is a ceiling for a wedged filesystem, not an estimate. +const userSessionRemoveTimeout = 15 * time.Minute + +// removeTree deletes dir as root. An error from a command that left nothing +// behind is logged and dropped: the directory is gone, which is what was +// asked for. +func (r *UserSessionRuntime) removeTree(what, dir string) error { + out, err := r.privRunFor(userSessionRemoveTimeout, userSessionRootSpec, nil, "rm", "-rf", "--one-file-system", "--", dir) + if err == nil { + return nil + } + if _, serr := os.Lstat(dir); errors.Is(serr, fs.ErrNotExist) { + r.logger.Info("user session removal: the command reported an error, but the directory is gone; treated as removed", "what", what, "path", dir, "error", err) + return nil + } + return outputErr(what, err, out) +} + +// Remove ends the session and deletes everything it owned: its worktree of the +// shared workspace clone (uncommitted work is auto-saved onto its branch first, +// see userSessionWorktreeRemoveScript), the sandbox (home, work dir, uploads, +// tmp) and the uid's tmux socket dir, so a later session that is allocated the +// same uid starts clean. Directories earlier releases created for the id are +// removed too. It is safe to call again after any failure, and on a session +// that is already partly or wholly gone. A failure is returned and the caller +// must keep the session record (and its uid slot) so the removal is retried. +// +// The order puts what can refuse first and what cannot be undone last: the +// login check removes nothing, the worktree step keeps the work on a branch +// (a resume gets the worktree back from it), and only then is the sandbox +// deleted. +func (r *UserSessionRuntime) Remove(id string) error { + if err := checkUserSessionID(id); err != nil { + return err + } + uid, uidErr := r.sessionUID(id) + if uidErr == nil { + if err := r.killAs(uid, id); err != nil { + r.logger.Warn("user session kill before removal failed", "id", id, "error", err) + } + } + // The session's CLI is gone (killed above), so its login can no longer + // change. If that login is the newest one on this hive, it goes into the + // shared file BEFORE the sandbox is deleted: removing a session must never + // remove the only valid login. "Nothing to adopt" (no file, an expired + // copy, a copy without a refresh token, a file that is not a credential) + // is not a failure and costs two stats and one small read. + // (The CLI creates files and directories with private modes; the hive + // must be able to look at the login path to judge it.) + r.makeHomeReadable(id) + if err := r.syncClaudeLogin(id, sessionLoginTriggerRemove, sessionLoginModeRemove); err != nil { + switch { + case errors.Is(err, sharedfile.ErrTooLarge): + // Over the size of any credentials file: not a login, so there + // is nothing to keep. syncClaudeLogin logged it. + case errors.Is(err, crux.ErrUserSessionHoldsLogin): + return err + default: + return fmt.Errorf("this session's sign-in could not be checked, so its files are kept: %w", err) + } + } + r.workspaceMu.Lock() + err := r.removeWorktrees(id) + r.workspaceMu.Unlock() + if err != nil { + return fmt.Errorf("removing session worktree: %w", err) + } + dir := userSessionSandboxDir(id) + if _, err := os.Lstat(dir); err == nil { + if err := r.removeTree("removing session sandbox", dir); err != nil { + return err + } + } + if uidErr == nil { + if err := r.removeTree("removing session tmux socket dir", userSessionTmuxSocketDir(uid)); err != nil { + return err + } + } + for _, p := range legacyUserSessionPaths(id) { + if err := os.RemoveAll(p); err != nil { + return fmt.Errorf("removing legacy session dir %s: %w", p, err) + } + } + return nil +} + +// Removed reports whether nothing of session id is left on disk: no sandbox, +// no worktrees and none of the directories earlier releases made. It answers +// false when that cannot be told (the sandbox root itself is missing or +// unreadable, as on a volume that is not mounted), so a caller that drops +// records of removed sessions never drops one whose files it merely cannot +// see. +func (r *UserSessionRuntime) Removed(id string) bool { + if checkUserSessionID(id) != nil { + return false + } + if fi, err := os.Stat(userSessionSandboxRoot()); err != nil || !fi.IsDir() { + return false + } + paths := append([]string{userSessionSandboxDir(id), userSessionWorktreesDir(id)}, legacyUserSessionPaths(id)...) + for _, p := range paths { + if _, err := os.Lstat(p); !errors.Is(err, fs.ErrNotExist) { + return false + } + } + return true +} diff --git a/src/crux/agent/usersession_remove_test.go b/src/crux/agent/usersession_remove_test.go new file mode 100644 index 0000000000..ed90b33b67 --- /dev/null +++ b/src/crux/agent/usersession_remove_test.go @@ -0,0 +1,244 @@ +package agent + +import ( + "bytes" + "context" + "errors" + "io" + "os" + "os/exec" + "path/filepath" + "reflect" + "strconv" + "strings" + "syscall" + "testing" + "time" + + "github.com/hivecommons/hive/crux" +) + +// "Nothing to adopt" is success: an ended session whose HOME holds an expired +// login, no login file at all, or a copy without a refresh token is deleted, +// and the shared login is not touched. Synthetic credentials only. +func TestSessionRemove_NothingToAdoptIsSuccess(t *testing.T) { + now := time.Now() + cases := map[string]func(t *testing.T, path string){ + "expired login copy": func(t *testing.T, path string) { + writeLoginAt(t, path, bridgeTestCreds(t, "old", now.Add(-time.Hour).UnixMilli()), now) + }, + "no login file": func(t *testing.T, path string) { + if err := os.Remove(path); err != nil { + t.Fatal(err) + } + }, + "copy without a refresh token": func(t *testing.T, path string) { + // What Start installed: asserted, then left as it is. + assertRefreshless(t, path, "acc-shared-"+bridgeTestTokenSentinel) + }, + "not a credentials file": func(t *testing.T, path string) { + writeLoginAt(t, path, []byte("not json"), now) + }, + } + for name, prepare := range cases { + t.Run(name, func(t *testing.T) { + f := newSessionLoginFixture(t) + shared := bridgeTestCreds(t, "shared", now.Add(time.Hour).UnixMilli()) + writeLoginAt(t, f.target, shared, now.Add(-time.Hour)) + f.start(t) + prepare(t, sessionLoginPath(testSessionID)) + + if err := f.r.Remove(testSessionID); err != nil { + t.Fatalf("Remove = %v, want success: there is nothing to adopt", err) + } + if _, err := os.Lstat(userSessionSandboxDir(testSessionID)); !errors.Is(err, os.ErrNotExist) { + t.Errorf("the sandbox survived Remove (%v)", err) + } + if !f.r.Removed(testSessionID) { + t.Error("Removed = false after a successful Remove") + } + if !bytes.Equal(mustRead(t, f.target), shared) || f.adoptions() != 0 { + t.Error("deleting a session with nothing to adopt changed the shared login") + } + f.assertNoTokenMaterial(t) + }) + } +} + +// A real adoption failure of a strictly newer, usable login refuses the +// delete with the reason, before anything is removed. +func TestSessionRemove_AdoptionFailureRemovesNothing(t *testing.T) { + f := newSessionLoginFixture(t) + now := time.Now() + f.start(t) + path := sessionLoginPath(testSessionID) + rotated := bridgeTestCreds(t, "rotated", now.Add(8*time.Hour).UnixMilli()) + writeLoginAt(t, path, rotated, now) + if err := os.Symlink(filepath.Join(t.TempDir(), "elsewhere"), f.target); err != nil { + t.Fatal(err) + } + f.p.calls = nil + + err := f.r.Remove(testSessionID) + if !errors.Is(err, crux.ErrUserSessionHoldsLogin) { + t.Fatalf("Remove = %v, want ErrUserSessionHoldsLogin", err) + } + if !strings.Contains(err.Error(), "this session holds the hive's current sign-in; adopting it failed: ") { + t.Errorf("reason = %q", err) + } + if strings.Contains(err.Error(), bridgeTestTokenSentinel) { + t.Error("token material in the error") + } + if !bytes.Equal(mustRead(t, path), rotated) { + t.Error("the session's login was lost") + } + for _, c := range f.p.calls { + if len(c.argv) > 0 && c.argv[0] == "rm" { + t.Errorf("a refused delete removed something: %v", c.argv) + } + if len(c.argv) > 2 && c.argv[2] == userSessionWorktreeRemoveScript { + t.Error("a refused delete removed the worktree") + } + } + if f.r.Removed(testSessionID) { + t.Error("Removed = true for a session whose delete was refused") + } + f.assertNoTokenMaterial(t) +} + +// The incident (#18767): the removal command reports an error (the cancel of +// its timeout failed) although it deleted the sandbox. What decides is the +// directory. +func TestSessionRemove_CommandErrorWithDirectoryGoneIsSuccess(t *testing.T) { + r, _, p := startedRuntime(t) + p.handler = func(c fakePrivCall) (bool, []byte, error) { + if len(c.argv) > 0 && c.argv[0] == "rm" { + _ = os.RemoveAll(c.argv[len(c.argv)-1]) + return true, []byte("su-exec: kill: No such file or directory"), errors.New("exec: canceling Cmd: killing privileged child: exit status 1") + } + return false, nil, nil + } + if err := r.Remove(testSessionID); err != nil { + t.Fatalf("Remove = %v, want success: the directory is gone", err) + } + if !r.Removed(testSessionID) { + t.Error("Removed = false") + } +} + +func TestSessionRemove_CommandErrorWithDirectoryLeftIsFailure(t *testing.T) { + r, _, p := startedRuntime(t) + p.handler = func(c fakePrivCall) (bool, []byte, error) { + if len(c.argv) > 0 && c.argv[0] == "rm" { + return true, []byte("Device or resource busy"), errors.New("exit status 1") + } + return false, nil, nil + } + err := r.Remove(testSessionID) + if err == nil || !strings.Contains(err.Error(), "removing session sandbox") { + t.Fatalf("Remove = %v, want the sandbox failure", err) + } + if r.Removed(testSessionID) { + t.Error("Removed = true although the sandbox is still there") + } + // A second call, with the cause gone, completes. + p.handler = nil + if err := r.Remove(testSessionID); err != nil { + t.Fatalf("second Remove = %v", err) + } + if !r.Removed(testSessionID) { + t.Error("Removed = false after the retry") + } +} + +// Deleting a sandbox is not a tmux call: it gets its own deadline, far past +// the 10 seconds that used to cancel it half-way. +func TestSessionRemove_SandboxRemovalHasItsOwnDeadline(t *testing.T) { + r, _, p := startedRuntime(t) + var left time.Duration + r.priv = func(ctx context.Context, user string, stdin io.Reader, argv ...string) ([]byte, error) { + if len(argv) > 0 && argv[0] == "rm" && argv[len(argv)-1] == userSessionSandboxDir(testSessionID) { + if dl, ok := ctx.Deadline(); ok { + left = time.Until(dl) + } + } + return p.run(ctx, user, stdin, argv...) + } + if err := r.Remove(testSessionID); err != nil { + t.Fatal(err) + } + if left <= userSessionCmdTimeout*6 { + t.Errorf("the sandbox removal had %v, want far more than the tmux command timeout %v", left, userSessionCmdTimeout) + } +} + +func TestUserSessionRemoved(t *testing.T) { + r, _, _ := startedRuntime(t) + if r.Removed(testSessionID) { + t.Error("Removed = true for a session whose sandbox exists") + } + if r.Removed("../x") { + t.Error("Removed = true for a bad id") + } + const other = "ba9876543210" + if !r.Removed(other) { + t.Error("Removed = false for an id with nothing on disk") + } + if err := os.MkdirAll(userSessionWorktreesDir(other), 0o755); err != nil { + t.Fatal(err) + } + if r.Removed(other) { + t.Error("Removed = true while a worktrees dir is left") + } + // A sandbox root that cannot be seen proves nothing. + t.Setenv(userSessionSandboxDirEnv, filepath.Join(t.TempDir(), "not-mounted")) + if r.Removed("aaaaaaaaaaaa") { + t.Error("Removed = true although the sandbox root is missing") + } +} + +// The cancel of a root child goes through the shell's builtin kill: the image +// has no kill binary, so `su-exec root kill` could never run. +func TestUserSessionRootKillUsesTheShellBuiltin(t *testing.T) { + const pid = 4242 + want := []string{userSessionRootSpec, "sh", "-c", userSessionRootKillScript, "sh", strconv.Itoa(pid)} + if got := userSessionRootKillArgv(pid); !reflect.DeepEqual(got, want) { + t.Errorf("argv = %v, want %v", got, want) + } + if got := userSessionRootKillArgv(pid); got[1] == "kill" { + t.Error("the cancel execs a kill binary") + } + + // The script really ends a process group, with PATH emptied so that only + // a builtin can do it. + sh, err := exec.LookPath("sh") + if err != nil { + t.Skip("no sh") + } + child := exec.Command(sh, "-c", "sleep 300 & wait") + child.SysProcAttr = &syscall.SysProcAttr{Setpgid: true} + if err := child.Start(); err != nil { + t.Fatal(err) + } + done := make(chan error, 1) + go func() { done <- child.Wait() }() + kill := exec.Command(sh, "-c", userSessionRootKillScript, "sh", strconv.Itoa(child.Process.Pid)) + kill.Env = []string{"PATH=/nonexistent"} + if out, err := kill.CombinedOutput(); err != nil { + _ = syscall.Kill(-child.Process.Pid, syscall.SIGKILL) + t.Fatalf("kill script: %v: %s", err, out) + } + select { + case werr := <-done: + var ee *exec.ExitError + if !errors.As(werr, &ee) { + t.Fatalf("child ended with %v, want a signal", werr) + } + if ws, ok := ee.Sys().(syscall.WaitStatus); !ok || !ws.Signaled() || ws.Signal() != syscall.SIGKILL { + t.Errorf("child ended with %v, want SIGKILL", werr) + } + case <-time.After(10 * time.Second): + _ = syscall.Kill(-child.Process.Pid, syscall.SIGKILL) + t.Fatal("the process group survived the kill script") + } +} diff --git a/src/crux/agent/usersession_worktree.go b/src/crux/agent/usersession_worktree.go index 5d15181050..e6c4e3305e 100644 --- a/src/crux/agent/usersession_worktree.go +++ b/src/crux/agent/usersession_worktree.go @@ -361,8 +361,8 @@ func (r *UserSessionRuntime) removeWorktrees(id string) error { return err } } - if out, err := r.privRunFor(userSessionWorktreeTimeout, userSessionRootSpec, nil, "rm", "-rf", "--one-file-system", "--", dir); err != nil { - return outputErr("removing session worktrees dir", err, out) + if err := r.removeTree("removing session worktrees dir", dir); err != nil { + return err } return os.RemoveAll(userSessionSpekInitMarkerDir(id)) } diff --git a/src/crux/usersession_types.go b/src/crux/usersession_types.go index 1add94c4fa..6aa472c691 100644 --- a/src/crux/usersession_types.go +++ b/src/crux/usersession_types.go @@ -139,4 +139,9 @@ var ( // .hive/inbox in the session's working directory is a symlink; nothing is // written. ErrUserSessionInboxSymlink = errors.New("refusing to write through a symlinked .hive/inbox directory") + // ErrUserSessionHoldsLogin is wrapped by Remove when the session's HOME + // holds a Claude login that is usable and strictly newer than the shared + // one and adopting it into the shared login failed. Nothing of the session + // is removed: deleting it would delete the hive's only valid sign-in. + ErrUserSessionHoldsLogin = errors.New("this session holds the hive's current sign-in") ) diff --git a/src/docs/api-reference/sessions.md b/src/docs/api-reference/sessions.md index 9884d4f063..8981b7f825 100644 --- a/src/docs/api-reference/sessions.md +++ b/src/docs/api-reference/sessions.md @@ -6,21 +6,21 @@ Sessions are interactive, named CLI-agent terminals. Each runs as its own unpriv | Method | Path | Auth | Purpose | Source | |---|---|---|---|---| -| `GET` | `/api/sessions` | Dashboard auth/session | Sessions, vendors and limits | `pkg/dashboard/api_sessions.go:182` | -| `POST` | `/api/sessions` | Owner or read-write | Create a session `{name, group, vendor, shared?, seed_items?}`; answers 202 at once with the session `preparing`, the start runs in the background (`seed_items`: see [Seeding from the Inbox](#seeding-from-the-inbox)) | `pkg/dashboard/api_sessions.go:183` | -| `PATCH` | `/api/sessions/{id}` | Session creator or owner | Rename, regroup or (un)share `{name?, group?, shared?}` | `pkg/dashboard/api_sessions.go:184` | -| `DELETE` | `/api/sessions/{id}` | Session creator or owner | Close a running session (metadata kept, marked `ended`); on an ended session, remove its metadata and delete its sandbox files | `pkg/dashboard/api_sessions.go:185` | -| `GET` | `/api/sessions/{id}/log` | Dashboard auth/session | Redacted scrollback `?lines=N` (capped at 5000) | `pkg/dashboard/api_sessions.go:186` | -| `POST` | `/api/sessions/{id}/input` | Owner or read-write | Queue `{text, submit}` for the pane; answers 202 with `{queued: {id, position}, queue}` and the text is typed once the CLI prompt is idle (size-capped, audited at delivery without the text) | `pkg/dashboard/api_sessions.go:187` | -| `DELETE` | `/api/sessions/{id}/input/{queue_id}` | Owner or read-write | Cancel your own queued send (the creator or an owner may cancel anyone's); 409 while it is being delivered | `pkg/dashboard/api_sessions.go:191` | -| `POST` | `/api/sessions/{id}/upload` | Owner or read-write | Multipart `file`; returns the absolute path inside the session directory | `pkg/dashboard/api_sessions.go:188` | -| `POST` | `/api/sessions/{id}/resume` | Owner or read-write | Reopen an ended session's last conversation where the vendor supports it (409 otherwise); answers 202 with the session `preparing` | `pkg/dashboard/api_sessions.go:189` | -| `POST` | `/api/sessions/{id}/keyboard` | Owner or read-write | `{"action":"take"\|"release"}` the keyboard lease: the one writable terminal | `pkg/dashboard/api_sessions.go:190` | -| `POST` | `/api/sessions/{id}/seed` | Owner or read-write | Send Inbox items `{items: [{source, id}]}` into a running session: one enveloped file each under `.hive/inbox/`, then a queued Send naming them (see [Seeding from the Inbox](#seeding-from-the-inbox)) | `pkg/dashboard/api_sessions.go:192` | +| `GET` | `/api/sessions` | Dashboard auth/session | Sessions, vendors and limits | `pkg/dashboard/api_sessions.go:188` | +| `POST` | `/api/sessions` | Owner or read-write | Create a session `{name, group, vendor, shared?, seed_items?}`; answers 202 at once with the session `preparing`, the start runs in the background (`seed_items`: see [Seeding from the Inbox](#seeding-from-the-inbox)) | `pkg/dashboard/api_sessions.go:189` | +| `PATCH` | `/api/sessions/{id}` | Session creator or owner | Rename, regroup or (un)share `{name?, group?, shared?}` | `pkg/dashboard/api_sessions.go:190` | +| `DELETE` | `/api/sessions/{id}` | Session creator or owner | Close a running session (metadata kept, marked `ended`; `?remove=true` also deletes it). On an ended or failed session, start the delete: answers 202 at once with the session `deleting`, the files are removed in the background (see [Deleting a session](#deleting-a-session)) | `pkg/dashboard/api_sessions.go:191` | +| `GET` | `/api/sessions/{id}/log` | Dashboard auth/session | Redacted scrollback `?lines=N` (capped at 5000) | `pkg/dashboard/api_sessions.go:192` | +| `POST` | `/api/sessions/{id}/input` | Owner or read-write | Queue `{text, submit}` for the pane; answers 202 with `{queued: {id, position}, queue}` and the text is typed once the CLI prompt is idle (size-capped, audited at delivery without the text) | `pkg/dashboard/api_sessions.go:193` | +| `DELETE` | `/api/sessions/{id}/input/{queue_id}` | Owner or read-write | Cancel your own queued send (the creator or an owner may cancel anyone's); 409 while it is being delivered | `pkg/dashboard/api_sessions.go:197` | +| `POST` | `/api/sessions/{id}/upload` | Owner or read-write | Multipart `file`; returns the absolute path inside the session directory | `pkg/dashboard/api_sessions.go:194` | +| `POST` | `/api/sessions/{id}/resume` | Owner or read-write | Reopen an ended session's last conversation where the vendor supports it (409 otherwise); answers 202 with the session `preparing` | `pkg/dashboard/api_sessions.go:195` | +| `POST` | `/api/sessions/{id}/keyboard` | Owner or read-write | `{"action":"take"\|"release"}` the keyboard lease: the one writable terminal | `pkg/dashboard/api_sessions.go:196` | +| `POST` | `/api/sessions/{id}/seed` | Owner or read-write | Send Inbox items `{items: [{source, id}]}` into a running session: one enveloped file each under `.hive/inbox/`, then a queued Send naming them (see [Seeding from the Inbox](#seeding-from-the-inbox)) | `pkg/dashboard/api_sessions.go:198` | ## Session object -`{id, name, group, vendor, creator, created_at, last_activity, status, attached, tmux_name, resumable, shared, workspace_repo, workspace, message, error, keyboard_holder, keyboard_since, keyboard_mine, viewers, queue, startup}` where `status` is `preparing`, `running`, `failed` or `ended`, `attached` is the number of live attached tmux clients (each list probe detaches clients whose process is gone or no longer descends from the terminal attacher, so a stale client never counts) and `tmux_name` is `hive-sess-`. `resumable`, `shared`, `workspace_repo`, `workspace`, `message`, `error` and the keyboard, presence and queue fields are additive. `keyboard_holder` is the user whose terminal is writable (`""` when released), `keyboard_since` when they took it and `keyboard_mine` whether that is the caller. `viewers` is `[{user, attached_since, keyboard}]`, one entry per person with a live terminal on the session (several tabs collapse into one). `queue` is `{length, head, position, items: [{id, user, bytes, submit, queued_at, delivering, mine}]}`: the pending sends, never their text, with `position` the caller's own first item (0 when none). `startup` (additive, running sessions only, absent when the hive has none) is `{state, screen, message, answered, since}`, see [Startup auto-answer](#startup-auto-answer). `GET /api/sessions` lists only sessions the caller created, owners (all) and `shared` sessions; the log, input, upload, resume, keyboard and terminal routes answer 404 for a session the caller cannot see and 403 when it is visible but not usable. +`{id, name, group, vendor, creator, created_at, last_activity, status, attached, tmux_name, resumable, shared, workspace_repo, workspace, message, error, keyboard_holder, keyboard_since, keyboard_mine, viewers, queue, startup, delete_error}` where `status` is `preparing`, `running`, `failed`, `ended` or `deleting` (`delete_error` is why the last delete failed, empty otherwise), `attached` is the number of live attached tmux clients (each list probe detaches clients whose process is gone or no longer descends from the terminal attacher, so a stale client never counts) and `tmux_name` is `hive-sess-`. `resumable`, `shared`, `workspace_repo`, `workspace`, `message`, `error` and the keyboard, presence and queue fields are additive. `keyboard_holder` is the user whose terminal is writable (`""` when released), `keyboard_since` when they took it and `keyboard_mine` whether that is the caller. `viewers` is `[{user, attached_since, keyboard}]`, one entry per person with a live terminal on the session (several tabs collapse into one). `queue` is `{length, head, position, items: [{id, user, bytes, submit, queued_at, delivering, mine}]}`: the pending sends, never their text, with `position` the caller's own first item (0 when none). `startup` (additive, running sessions only, absent when the hive has none) is `{state, screen, message, answered, since}`, see [Startup auto-answer](#startup-auto-answer). `GET /api/sessions` lists only sessions the caller created, owners (all) and `shared` sessions; the log, input, upload, resume, keyboard and terminal routes answer 404 for a session the caller cannot see and 403 when it is visible but not usable. ## Roles @@ -36,6 +36,25 @@ A private session is visible only to its creator and owners, whatever the role. A create (or resume) never blocks the request on the start. The record is saved and answered as `preparing` with a human `message` (`preparing workspace: cloning org/repo (first session on this hive)` when the workspace does not exist yet, otherwise the vendor being started), and the start runs in the background: the one-time clone of the workspace, its hand-over to the sessions group, `spektacular init` and the tmux server. Poll `GET /api/sessions` until the session is `running` (the terminal can then be attached) or `failed`, in which case `error` carries the reason (tokens redacted) and `DELETE` removes the record; a resume whose start fails goes back to `ended` with `error` set and stays resumable. Preparing sessions count against the limits, answer 409 to input, upload, terminal handoff and `DELETE`, and a second create that needs the same workspace waits for the clone in flight instead of starting another. A `preparing` record found after a hive restart, with no start behind it, is marked `failed` on the next list. The create is audited as `session_created` and its outcome as `session_started` (`outcome=running` or `outcome=failed` with the error). +## Deleting a session + +`DELETE /api/sessions/{id}` on an ended or failed session never waits for the removal. Deleting a session's files (a HOME with a CLI's caches, removed through the privileged helper) can take a minute or more, so the hive answers at once and one background job per session does the work: + +| Answer | Meaning | +|---|---| +| `202 {ok, deleting: true, session}` | The delete started (or one is already running for this session; a second request never starts a second removal). The session's `status` is `deleting`. | +| `200 {ok, removed: true}` | Nothing of the session was left on disk, so the record is gone already. | +| `200 {ok}` | A running session was closed and kept, `ended`. With `?remove=true` it is closed and then deleted (202). | +| `409` | A start is in flight (`preparing`), or the session changed state meanwhile. A record a hive restart left `preparing` is not in this case: it is failed and deleted. | + +The job removes things in an order that can stop without losing anything: the login check first (it removes nothing, see [Vendor logins in a session](#vendor-logins-in-a-session)), then the session's worktree (uncommitted work is saved onto its `hive-session/` branch of the shared clone, so a resume gets it back), then the sandbox, and the record **last**, only once the files are gone. What decides the outcome is what is on disk: a removal command that reports an error after everything is gone has removed the session. + +- **Success:** the record disappears from `GET /api/sessions`. The dashboard shows the tab as "Deleting…" with a spinner and no controls until then, polls faster while any session is deleting, and announces the result. +- **Failure:** the record goes back to the status it had, with the reason in `delete_error`; nothing else changed and a new `DELETE` retries. The dashboard brings the tab back, shows the reason and offers **Retry delete**. +- **Repair:** a `deleting` record that has no job (the hive restarted in the middle) gets its job restarted, and an `ended` record whose sandbox and worktree are both gone is dropped, by the session sweep at boot and on every list. No manual step. + +Every outcome writes one log line with the duration (`sessions: session deleted`, or WARN `sessions: session delete failed; ...`) and one `session_closed` audit entry with `removed`, `outcome` (`success` or `failed`), `trigger` (`request`, `retention`, `interrupted-delete` or `files-already-gone`), `duration` and, for a failure, `error`. Sessions pruned after the retention window go through the same job. + ## Workspace Every session's CLI starts inside the hive's **session workspace**. The hive keeps one persistent clone per repository, `//` (default `/data/session-sandbox/workspace/...`), and gives **each session its own git worktree** of it, `/.sessions///`, checked out on its own branch `hive-session/` (created at the clone's `HEAD`; `git fetch` is never run for you). Sessions share the clone's objects and branches, so a commit in one session is visible to `git log`/`git switch` in another, but not a working tree: two sessions no longer see each other's uncommitted edits, a branch switched in one does not move another, and each worktree has its own `.spektacular/` state, so one person's unfinished spec never shows up in another session's "Another spec workflow" prompt ([#18588](https://github.com/hivecommons/hive/issues/18588)). People who share one session (several viewers, one keyboard) still share everything. The repository is `dashboard.sessions.workspace_repo` (`HIVE_SESSIONS_WORKSPACE_REPO`), defaulting to the hive's own `project.org/primary_repo`; with neither set, a session opens in its own empty directory. The hive clones once, with a short-lived App token in a one-shot credential helper that is deleted afterwards, so the clone's config and `origin` carry no credential: pushing from a session uses the person's own login. A session's first start adds its worktree with `git worktree add` against that one clone, as the session's own uid with git hooks and `core.fsmonitor` disabled (no second clone); later starts and resumes reuse it. The clone's shared git config and hooks are locked (root-owned and read-only to sessions, rewritten at every boot), so one session cannot plant a config entry, hook, filter or alias that runs as another session; keep personal git settings with `git config --worktree` (see [Security model](../security-model.md)). Writes to the shared config fail (`git push -u` warns and sets no upstream), and deleting a ref the clone still holds in `packed-refs` fails. The hive then runs `spektacular init ` in the session's worktree once per vendor spek supports (`claude`, `bob`, `codex`, `copilot`, `omp`) when `/usr/local/bin/spektacular` (`runs.spektacular.binary`) is installed; a missing binary or a failed init is logged and the session still starts. Deleting a session (or pruning an ended one) removes its worktree (`git worktree remove`, then `git worktree prune`) and never silently drops work: anything uncommitted, untracked files included (ignored files are not), is first auto-saved as a commit on the worktree's branch, `hive: auto-save of work left uncommitted when the session was deleted`, a detached `HEAD` is kept as `hive-session/-detached`, and a session branch is deleted only when the clone's `HEAD` already contains it, otherwise it stays in the clone and the hive logs its name. If the worktree cannot be saved that way the removal fails and the session is kept for a retry. The periodic session sweep also removes worktrees whose session record no longer exists; it is skipped while the sessions store has failed to load (or an unreadable copy of it is still set aside as `sessions.json.corrupt-