Repository navigation
✨ feature: sandbox node preflight and fail-fast on an unschedulable Job - #18891
Conversation
Add sandbox.preflight: configurable commands run in the Job's main container before the agent. A failure ends the kick in seconds as node_unhealthy with the node recorded, retries once with a hostname NotIn affinity, and is not an agent failure. Add job.schedule_timeout_s (default 10m): a pod Unschedulable that long fails the kick as waiting_for_node with the scheduler's message, deletes the Job and releases the agent. The dashboard shows both states. Fixes #18811 Signed-off-by: Andrew Anderson <andy@clubanderson.com>
Open PR overlap checkBase branch: Conflicts are checked with every PR merged onto the current tip of
|
Open PR overlap checkBase branch: Conflicts are checked with every PR merged onto the current tip of
|
|
Hive status — updated in place; edits do not notify. Full history is in the Activity Monitor. @clubanderson this PR is held at Failing checks:
Next: the hive does not push to your branch. Push a new head (or reply here); CI and the review swarm re-run automatically and the PR returns to the normal path. |
The sleep ratchet counts time.Sleep calls in tests. Signed-off-by: Andrew Anderson <andy@clubanderson.com>
|
[APPROVALNOTIFIER] This PR is APPROVED Approval requirements bypassed by manually added approval. This pull-request has been approved by: The full list of commands accepted by this bot can be found here. The pull request process is described here DetailsNeeds approval from an approver in each of these files:Approvers can indicate their approval by writing |
What
Two checks so a bad or missing node does not cost a whole kick (#18811). Both are for
sandbox.runtime: job.sandbox.preflight(off unlesscommandsis set; agent level andagent_sandboxdefault, same merge rules aspublish): commands run assh -lcin the Job's main container after the relay check and before the agent, with a per-command and a total timeout. A failure exits 75 after printing a marker line, and hive reportsnode_unhealthy:spec.nodeName) in the run record and audit log;idle, notfailed, so restart/escalation/fix-loop logic never sees it;retry_delay_swith akubernetes.io/hostnameNotInfor the failed node, ANDed into every required term of the operator's own affinity (nodes are never pinned); a second failure fails the kick asnode_unhealthy.job.schedule_timeout_s(default 600 s, per-agent job block falls back to global): the pod is read while it has no node;PodScheduled=False/Unschedulablefor longer than the timeout fails the kick aswaiting_for_nodewith the scheduler's message, deletes the Job and releases the agent instead of waiting out the 45m+15m deadline.Dashboard:
waiting_for_node(live, amber) andnode_unhealthy(last run, red) are their own health states with the reason and node.Notes for review
Runis split intorunWithRetry/runAttempt; types and name helpers moved out ofkubejob.goand the Job types out ofagent_config.goto stay under the crux file-size ratchet (no behavior change).schedule_timeout_sis on by default (the issue asks for 5 to 10 minutes); an autoscaler that needs longer should raise it. Preflight is off unless configured.publish.verify).Fixes #18811
🤖 Generated with Claude Code