Repository navigation
Description
Problem
A sandbox Job's placement can only be constrained with sandbox.job.node_selector (and tolerations). src/crux/kubejob/kubejob.go passes NodeSelector straight into the pod spec; there is no way to send node affinity. A node selector can only say "must carry these labels", never "must not be this node", so an operator who needs to avoid a known-bad node has to pin the Job to one specific node instead.
Seen 2026-11-17 on a customer's standalone hive (OpenShift, Spyre accelerator nodes): the quality agent's Job was pinned with kubernetes.io/hostname: p1-worker-45 (the ConfigMap comment says "p1-worker-93 card unusable (VFIO mmap)"). When the pinned node had no free card the Job sat Pending until its deadline (backoffLimit 0, deadline 3600s), and nothing could reschedule it. The customer's infra team cordons nodes with hardware errors, but until that happens a pin-free Job can still land on a bad node and fail, since hive has no way to say "any Spyre node except X".
Expected
- A
sandbox.job.affinityblock (and the same underagent_sandbox.job) that accepts the standard KubernetesnodeAffinityshape:requiredDuringSchedulingIgnoredDuringExecution.nodeSelectorTerms[].matchExpressions[](In,NotIn,Exists,DoesNotExist,Gt,Lt) andpreferredDuringSchedulingIgnoredDuringExecutionwith weights. Optionally pod anti-affinity is out of scope. - Passed through to the Job's pod spec next to
nodeSelector, with the same merge rules as the otherjobfields (agent block overrides theagent_sandboxdefault). - Validation: reject malformed operators/values with the field named; reject a
valueslist onExists/DoesNotExist; cap term and value counts. - Example in
src/docs/sandbox-isolation.md: "any Spyre node except one":spyre.node-classification In [all]andkubernetes.io/hostname NotIn [<bad-node>]. - A warning at config load when the node selector pins
kubernetes.io/hostname(a single node) so the operator is told that a cordoned or down node will leave the Job Pending until its deadline. - Dashboard (after ✨ Dashboard/API: set review.perspectives_source and agent sandbox image/block without editing config files #18800): an optional "Node affinity" section in the agent's Sandbox settings, behind the same validation (can be a follow-up PR).
Acceptance
- Unit tests in
src/crux/kubejobthat the pod spec carries the affinity unchanged, with and without a node selector; config tests for merge precedence and validation errors. - Config round trip: set, save, reload, still present (see 🐛 Kubernetes reload may drop dashboard-saved review and agent_sandbox settings (loadWithDashboardOverlay does not adopt them) #18804 on overlay adoption of
agent_sandbox). - changelog.d fragment and docs.
Verify before relying on it: the cluster's custom scheduler (spyre-scheduler here) must honour nodeAffinity; most scheduler forks do, but state in the docs that placement is the scheduler's decision.
Related: #18797 / #18800 (sandbox settings UI/API), #18804 (overlay adoption), #18525 (orphaned Job reaping).
Activity
clubanderson commentedon Nov 17, 2026
Claimed: operator-filed; to be dispatched after #18800 merges. Hive agents, please do not pick it up.
hivecommons-hive commentedon Nov 17, 2026
🔓 Claim by @Karibhou released (covering PR merged or closed).
Claim history
kellyaa(contributor on hive hosted-available-oke-11-placeholder-r05x)kellyaa(contributor; lease revoked)kellyaa(contributor on hive hosted-available-oke-11-placeholder-r05x)kellyaa(contributor; lease revoked)Karibhou(contributor on hive hosted-available-oke-11-placeholder-r05x)Karibhou(contributor; covering PR merged or closed)3 remaining items
hivecommons-hive commentedon Nov 18, 2026
@Karibhou you now hold this issue (contributor on hive hosted-available-oke-11-placeholder-r05x); the claim ledger comment above tracks it. Reply
/unclaimto release.✨ feature: support node affinity for sandbox jobs (#18861)