Skip to content

✨ feature: sandbox node preflight and fail-fast on an unschedulable Job - #18891

Merged
kubestellar-prow[bot] merged 2 commits into
v5from
feat/sandbox-preflight-18811
Nov 18, 2026
Merged

kubestellar-prow[bot] merged 2 commits into
v5from
feat/sandbox-preflight-18811

Conversation

@clubanderson

Copy link
Copy Markdown
Member

What

Two checks so a bad or missing node does not cost a whole kick (#18811). Both are for sandbox.runtime: job.

sandbox.preflight (off unless commands is set; agent level and agent_sandbox default, same merge rules as publish): commands run as sh -lc in the Job's main container after the relay check and before the agent, with a per-command and a total timeout. A failure exits 75 after printing a marker line, and hive reports node_unhealthy:

  • output tail in the transcript and kick log, node (spec.nodeName) in the run record and audit log;
  • not an agent failure: the agent returns to idle, not failed, so restart/escalation/fix-loop logic never sees it;
  • one retry after retry_delay_s with a kubernetes.io/hostname NotIn for the failed node, ANDed into every required term of the operator's own affinity (nodes are never pinned); a second failure fails the kick as node_unhealthy.
  • commands come from config only, never from a kick message; only agent kicks run it (publish.verify Jobs do not).

job.schedule_timeout_s (default 600 s, per-agent job block falls back to global): the pod is read while it has no node; PodScheduled=False/Unschedulable for longer than the timeout fails the kick as waiting_for_node with the scheduler's message, deletes the Job and releases the agent instead of waiting out the 45m+15m deadline.

Dashboard: waiting_for_node (live, amber) and node_unhealthy (last run, red) are their own health states with the reason and node.

Notes for review

  • Refactor: Run is split into runWithRetry/runAttempt; types and name helpers moved out of kubejob.go and the Job types out of agent_config.go to stay under the crux file-size ratchet (no behavior change).
  • schedule_timeout_s is on by default (the issue asks for 5 to 10 minutes); an autoscaler that needs longer should raise it. Preflight is off unless configured.
  • New settings leaves are in the parity allowlist (policy-file knobs, like publish.verify).

Fixes #18811

🤖 Generated with Claude Code

Add sandbox.preflight: configurable commands run in the Job's main container
before the agent. A failure ends the kick in seconds as node_unhealthy with
the node recorded, retries once with a hostname NotIn affinity, and is not an
agent failure.

Add job.schedule_timeout_s (default 10m): a pod Unschedulable that long fails
the kick as waiting_for_node with the scheduler's message, deletes the Job and
releases the agent. The dashboard shows both states.

Fixes #18811

Signed-off-by: Andrew Anderson <andy@clubanderson.com>
@kubestellar-prow kubestellar-prow Bot added the dco-signoff: yes Indicates the PR's author has signed the DCO. label Nov 18, 2026
@kubestellar-prow kubestellar-prow Bot added the size/XXL Denotes a PR that changes 1000+ lines, ignoring generated files. label Nov 18, 2026
@github-actions

github-actions Bot commented Nov 18, 2026 •

Copy link
Copy Markdown
Contributor

Open PR overlap check

Base branch: v5

Conflicts are checked with every PR merged onto the current tip of v5, so a PR that only needs a rebase is not reported as conflicting with its siblings.

PR Title Relation Files
#18868 ✨ Sandbox Jobs: authenticated GitHub reads via a read-only hive relay same files src/crux/agent/manager_sandbox.go
src/crux/config/agent_config.go
src/crux/config/sandbox_job_types.go
src/docs/sandbox-isolation.md
src/pkg/config/crux_aliases.go
src/pkg/config/validate.go
src/pkg/dashboard/settings_parity_allowlist.txt
#18893 🐛 Hold decisions on a PR whose CI or review is still running on its current head same files src/pkg/dashboard/static/index.html
#18873 🐛 needs-sha: exempt owner/detailed issues, overrideable waiver, visible hold on the row same files src/pkg/dashboard/static/index.html

@github-actions

Copy link
Copy Markdown
Contributor

Open PR overlap check

Base branch: v5

Conflicts are checked with every PR merged onto the current tip of v5, so a PR that only needs a rebase is not reported as conflicting with its siblings.

PR Title Relation Files
#18888 ✨ Customers list: GitHub avatars, tag pills and one-line rows same files src/pkg/dashboard/static/index.html
#18868 ✨ Sandbox Jobs: authenticated GitHub reads via a read-only hive relay CONFLICT src/crux/agent/manager_sandbox.go
src/crux/config/agent_config.go (conflict)
src/crux/config/sandbox_job_types.go (conflict)
src/docs/sandbox-isolation.md
src/pkg/config/crux_aliases.go
src/pkg/dashboard/settings_parity_allowlist.txt
#18873 🐛 needs-sha: exempt owner/detailed issues, overrideable waiver, visible hold on the row same files src/pkg/dashboard/static/index.html

@hivecommons-hive

Copy link
Copy Markdown
Contributor

Hive status — updated in place; edits do not notify. Full history is in the Activity Monitor.

@clubanderson this PR is held at stage/071-needs-author and the hive will not merge it yet.

Failing checks:

  • test (rest 2/4)

Next: the hive does not push to your branch. Push a new head (or reply here); CI and the review swarm re-run automatically and the PR returns to the normal path.

The sleep ratchet counts time.Sleep calls in tests.

Signed-off-by: Andrew Anderson <andy@clubanderson.com>
@github-actions github-actions Bot removed the conflicts-with-open-pr Open PR has a merge conflict with another open PR label Nov 18, 2026
@clubanderson clubanderson added lgtm Indicates that a PR is ready to be merged. approved Indicates a PR has been approved by an approver from all required OWNERS files. labels Nov 18, 2026
@kubestellar-prow

Copy link
Copy Markdown
Contributor

[APPROVALNOTIFIER] This PR is APPROVED

Approval requirements bypassed by manually added approval.

This pull-request has been approved by:

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Details Needs approval from an approver in each of these files:

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

@kubestellar-prow
kubestellar-prow Bot merged commit d5dfc48 into v5 Nov 18, 2026
98 checks passed
@kubestellar-prow
kubestellar-prow Bot deleted the feat/sandbox-preflight-18811 branch November 18, 2026 04:52
clubanderson added a commit that referenced this pull request Nov 18, 2026
's sandbox job types)

Signed-off-by: Andy Anderson <andy@clubanderson.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

approved Indicates a PR has been approved by an approver from all required OWNERS files. dco-signoff: yes Indicates the PR's author has signed the DCO. lgtm Indicates that a PR is ready to be merged. size/XXL Denotes a PR that changes 1000+ lines, ignoring generated files.

Projects

None yet

Development

Successfully merging this pull request may close these issues.

✨ Sandbox Jobs: pre-run node health check (sandbox.preflight) and fail-fast on an unschedulable Job

1 participant