Skip to content

πŸ› Sandbox toggle writes a complete block; reap orphaned sandbox Jobs - #18566

Merged
hivecommons-hive[bot] merged 4 commits into
v5from
fix/18525-sandbox-toggle-and-job-reap
Nov 17, 2026
Merged

hivecommons-hive[bot] merged 4 commits into
v5from
fix/18525-sandbox-toggle-and-job-reap

Conversation

@clubanderson

Copy link
Copy Markdown
Member

Fixes #18525

What changed

Run in sandbox toggle (PUT /api/config/agent/{name}/general)

  • Turning it on resolves the agent's effective sandbox block: the agent's own fields first, then the agent_sandbox defaults. It writes the block with the image and runtime pinned.
  • When no image, no usable runtime, or (for the Job runtime) no job.workspace_claim resolves, it refuses with 400. The message names what is missing and lists the agents a block can be copied from.
  • copy_from (the dashboard sends sandboxCopyFrom) copies another agent's complete block, minus its publish block. The receiving agent keeps its own publish.verify.
  • Turning it off keeps the block and sets enabled: false.
  • The General tab shows a "Sandbox settings from" picker when the toggle is turned on. It offers the defaults when they are complete, plus every agent with a complete block. It shows an inline error when there are none. GET /api/config/agent/{name} now carries general.sandboxDefaultsProblem and general.sandboxCopySources.

Entering sandbox mode stops the tmux session

  • When an agent's config moves it into sandbox mode (through UpdateConfig, ReconcileAgents, or the global gate in SetSandboxConfig), the manager does the following:
    • archives the scrollback
    • reaps the CLI
    • kills the session
    • parks the agent in idle
    • writes a sandbox_mode_entered audit line
  • A kick refused because the agent is in running now names the real state: a sandbox kick is already running or agent has a live interactive session; sandbox kicks are blocked until it stops.

Job deadline and boot sweep

  • Every sandbox Job carries activeDeadlineSeconds: the kick timeout plus kubejob.DeadlineMargin (15 minutes). When the run's context deadline gives a later value, the Job uses that one.
  • At boot, when any agent selects the Job runtime, hive lists the app.kubernetes.io/managed-by=hive-kubejob Jobs in its namespace. It deletes unfinished agent-sandbox Jobs (hive-<agent>-sandbox and -verify) in two cases:
    • no live kick owns the Job and it is older than the longest kick timeout
    • it never got a ready pod and is past the deadline
  • Each deleted Job is logged and audited as sandbox_job_reaped.
  • The sweep leaves these Jobs alone:
    • finished Jobs, which are left to their TTL
    • Jobs without the label
    • contained runs (crux-run-*)

Tests

  • toggle with no defaults (refused), with defaults (prefilled), copy_from, the copy_from refusals, toggling off
  • tmux teardown via UpdateConfig, ReconcileAgents and the global gate; agents mid-kick and paused agents are left alone
  • the improved refusal message for both states
  • activeDeadlineSeconds in the Job spec, both configured and default
  • the boot sweep: deletes orphans, keeps live, young and finished Jobs, ignores unlabeled Jobs and contained runs; error paths

Docs: src/docs/sandbox-isolation.md, src/docs/dashboard-sections/workflow-and-agents.md. Changelog: changelog.d/fixed-18525-sandbox-toggle-and-job-reap.md.

cc @joerunde

πŸ€– Generated with Claude Code

The per-agent Run in sandbox toggle now resolves the agent's effective
sandbox block (own fields, then agent_sandbox defaults) and refuses with
400 when no image or usable runtime resolves. copy_from copies another
agent's complete block minus publish.verify. The General tab shows a
source picker with an inline error when there is none. Toggling off keeps
the block with enabled: false.

Switching an agent into sandbox mode now stops its tmux session, and the
running-state kick refusal names the real state.

Every sandbox Job gets activeDeadlineSeconds (kick timeout plus a named
margin), and hive sweeps orphaned hive-kubejob sandbox Jobs at boot,
logging and auditing each.

Fixes #18525

Signed-off-by: Andy Anderson <andy@clubanderson.com>
@kubestellar-prow kubestellar-prow Bot added dco-signoff: yes Indicates the PR's author has signed the DCO. size/XXL Denotes a PR that changes 1000+ lines, ignoring generated files. labels Nov 17, 2026
@github-actions

github-actions Bot commented Nov 17, 2026 •

Copy link
Copy Markdown
Contributor

Open PR overlap check

Base branch: v5

Conflicts are checked with every PR merged onto the current tip of v5, so a PR that only needs a rebase is not reported as conflicting with its siblings.

PR Title Relation Files
#18507 ✨ governor: release-feed watcher kicks agents when a tracked upstream dependency releases same files src/cmd/hive/main.go
#18587 ✨ Sandboxed fix kicks push to the agent's own PR; sandbox PRs are credited to their agent same files src/crux/agent/manager_sandbox.go
src/crux/agent/sandbox_executor.go
src/docs/sandbox-isolation.md
#18586 ✨ Sandboxed agents open issues and keep topic claims through the request spool same files src/cmd/hive/main.go
src/crux/agent/sandbox_executor.go
src/docs/sandbox-isolation.md
#18583 ✨ Governor: opt-in advisory report in every repo (default off) same files src/pkg/dashboard/static/index.html
#18536 ✨ feature: choose remote repository write identity same files src/cmd/hive/main.go
src/pkg/dashboard/static/index.html
#18576 ✨ Owner-only Send to on model-owned stage-tracker rows same files src/cmd/hive/main.go
src/pkg/dashboard/static/index.html
#18584 ✨ Show why a calendar-email proof failed on its Settings row same files src/pkg/dashboard/static/index.html
#18562 πŸ› Dashboard: keep the Collaborate sidebar group and place it after Dashboard same files src/pkg/dashboard/static/index.html

@hivecommons-hive

hivecommons-hive Bot commented Nov 17, 2026 •

Copy link
Copy Markdown
Contributor

Hive status β€” updated in place; edits do not notify. Full history is in the Activity Monitor.

@clubanderson this PR is held at stage/071-needs-author and the hive will not merge it yet.

Failing checks:

  • crux-file-size-ratchet

Next: the hive does not push to your branch. Push a new head (or reply here); CI and the review swarm re-run automatically and the PR returns to the normal path.

Evidence bundle `#185669f08e3d` sha256:89e42977324eed2efb17a7a7c2b4621baa9696ecd195bea8f8da0a628fa4d1aa β€” download from the dashboard or `hivectl review evidence #18566`.

@github-actions

Copy link
Copy Markdown
Contributor

Mutation testing

❌ gremlins did not produce a report for 2 package(s), so no verdict can be derived from this run (see the job log):

  • crux/agent: gremlins exited 1
  • pkg/dashboard: gremlins exited 137

gremlins found 4 surviving mutant(s) in changed files: the code ran under tests, but changing it did not fail any test.

File Line Mutation
src/crux/kubejob/kubejob.go 176 CONDITIONALS_BOUNDARY
src/crux/kubejob/sweep.go 79 CONDITIONALS_BOUNDARY
src/crux/kubejob/sweep.go 79 CONDITIONALS_NEGATION
src/crux/kubejob/sweep.go 113 CONDITIONALS_NEGATION

9 mutant(s) in changed files are not reached by any test (NOT COVERED). A maintainer may apply the mutation/acknowledged label to accept this.

File Line Mutation
src/cmd/hive/sandbox_job_sweep.go 18 ARITHMETIC_BASE
src/crux/kubejob/kubejob.go 67 ARITHMETIC_BASE
src/crux/kubejob/kubejob.go 75 ARITHMETIC_BASE
src/crux/kubejob/sweep.go 126 CONDITIONALS_BOUNDARY
src/crux/kubejob/sweep.go 126 CONDITIONALS_NEGATION
src/crux/kubejob/sweep.go 126 CONDITIONALS_NEGATION
src/crux/kubejob/sweep.go 130 CONDITIONALS_BOUNDARY
src/crux/kubejob/sweep.go 130 CONDITIONALS_NEGATION
src/crux/kubejob/sweep.go 130 CONDITIONALS_NEGATION

Verdict: error (#17349).

…under its size ratchet

Signed-off-by: Andy Anderson <andy@clubanderson.com>
@kubestellar-prow

Copy link
Copy Markdown
Contributor

[APPROVALNOTIFIER] This PR is APPROVED

Approval requirements bypassed by manually added approval.

This pull-request has been approved by:

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Details Needs approval from an approver in each of these files:

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

@kubestellar-prow kubestellar-prow Bot removed the lgtm Indicates that a PR is ready to be merged. label Nov 17, 2026
@kubestellar-prow

Copy link
Copy Markdown
Contributor

New changes are detected. LGTM label has been removed.

@hivecommons-hive
hivecommons-hive Bot merged commit e3b2bcf into v5 Nov 17, 2026
90 checks passed
@kubestellar-prow
kubestellar-prow Bot deleted the fix/18525-sandbox-toggle-and-job-reap branch November 17, 2026 02:11
@hivecommons-hive

Copy link
Copy Markdown
Contributor

merged by Hive trusted-author auto-merge: author @clubanderson holds role owner and GitHub permission admin on repo; CI green at 9f08e3d.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

approved Indicates a PR has been approved by an approver from all required OWNERS files. dco-signoff: yes Indicates the PR's author has signed the DCO. size/XXL Denotes a PR that changes 1000+ lines, ignoring generated files.

Projects

None yet

Development

Successfully merging this pull request may close these issues.

πŸ› Per-agent Run in sandbox toggle writes enabled: true only and wedges the agent; orphaned sandbox Jobs are never reaped

1 participant