Skip to content

✨ Sandbox Jobs: support node affinity (e.g. any Spyre node except a known-bad one), not only nodeSelector #18809

Closed

Description

@clubanderson

Problem

A sandbox Job's placement can only be constrained with sandbox.job.node_selector (and tolerations). src/crux/kubejob/kubejob.go passes NodeSelector straight into the pod spec; there is no way to send node affinity. A node selector can only say "must carry these labels", never "must not be this node", so an operator who needs to avoid a known-bad node has to pin the Job to one specific node instead.

Seen 2026-11-17 on a customer's standalone hive (OpenShift, Spyre accelerator nodes): the quality agent's Job was pinned with kubernetes.io/hostname: p1-worker-45 (the ConfigMap comment says "p1-worker-93 card unusable (VFIO mmap)"). When the pinned node had no free card the Job sat Pending until its deadline (backoffLimit 0, deadline 3600s), and nothing could reschedule it. The customer's infra team cordons nodes with hardware errors, but until that happens a pin-free Job can still land on a bad node and fail, since hive has no way to say "any Spyre node except X".

Expected

  • A sandbox.job.affinity block (and the same under agent_sandbox.job) that accepts the standard Kubernetes nodeAffinity shape: requiredDuringSchedulingIgnoredDuringExecution.nodeSelectorTerms[].matchExpressions[] (In, NotIn, Exists, DoesNotExist, Gt, Lt) and preferredDuringSchedulingIgnoredDuringExecution with weights. Optionally pod anti-affinity is out of scope.
  • Passed through to the Job's pod spec next to nodeSelector, with the same merge rules as the other job fields (agent block overrides the agent_sandbox default).
  • Validation: reject malformed operators/values with the field named; reject a values list on Exists/DoesNotExist; cap term and value counts.
  • Example in src/docs/sandbox-isolation.md: "any Spyre node except one": spyre.node-classification In [all] and kubernetes.io/hostname NotIn [<bad-node>].
  • A warning at config load when the node selector pins kubernetes.io/hostname (a single node) so the operator is told that a cordoned or down node will leave the Job Pending until its deadline.
  • Dashboard (after ✨ Dashboard/API: set review.perspectives_source and agent sandbox image/block without editing config files #18800): an optional "Node affinity" section in the agent's Sandbox settings, behind the same validation (can be a follow-up PR).

Acceptance

Verify before relying on it: the cluster's custom scheduler (spyre-scheduler here) must honour nodeAffinity; most scheduler forks do, but state in the docs that placement is the scheduler's decision.

Related: #18797 / #18800 (sandbox settings UI/API), #18804 (overlay adoption), #18525 (orphaned Job reaping).

Activity

  1. added
    kind/featureCategorizes issue or PR as related to a new feature.
    triage/acceptedIndicates an issue or PR is ready to be actively worked on.
    on Nov 17, 2026
  2. self-assigned this
    on Nov 17, 2026
  3. clubanderson commented on Nov 17, 2026

    @clubanderson
    MemberAuthor

    Claimed: operator-filed; to be dispatched after #18800 merges. Hive agents, please do not pick it up.

  4. added
    help wantedDenotes an issue that needs help from a contributor. Must meet "help wanted" guidelines.
    on Nov 17, 2026
  5. hivecommons-hive commented on Nov 17, 2026

    @hivecommons-hive
    Contributor

    🔓 Claim by @Karibhou released (covering PR merged or closed).

    Claim history
    • 2026-11-17 18:54 UTC — claimed by kellyaa (contributor on hive hosted-available-oke-11-placeholder-r05x)
    • 2026-11-17 18:55 UTC — released by kellyaa (contributor; lease revoked)
    • 2026-11-17 18:58 UTC — claimed by kellyaa (contributor on hive hosted-available-oke-11-placeholder-r05x)
    • 2026-11-17 18:59 UTC — released by kellyaa (contributor; lease revoked)
    • 2026-11-18 01:12 UTC — claimed by Karibhou (contributor on hive hosted-available-oke-11-placeholder-r05x)
    • 2026-11-18 01:25 UTC — released by Karibhou (contributor; covering PR merged or closed)
  6. 3 remaining items

  7. added
    stage/020-claimedClaimed by the scanner or a relay contributor
    stage/stalledIn its current stage longer than expected
    stage/010-actionableTriaged; eligible for the hive to pick up
    and removed
    stage/020-claimedClaimed by the scanner or a relay contributor
    stage/stalledIn its current stage longer than expected
    on Nov 17, 2026
  8. hivecommons-hive commented on Nov 18, 2026

    @hivecommons-hive
    Contributor

    @Karibhou you now hold this issue (contributor on hive hosted-available-oke-11-placeholder-r05x); the claim ledger comment above tracks it. Reply /unclaim to release.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

Labels

help wantedDenotes an issue that needs help from a contributor. Must meet "help wanted" guidelines.kind/featureCategorizes issue or PR as related to a new feature.triage/acceptedIndicates an issue or PR is ready to be actively worked on.

Type

No type

Projects

No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions