Repository navigation
Description
Problem
A sandbox Job whose project image lives in a private registry can fail with ImagePullBackOff even though the Job's service account carries the right registry secrets, because the launcher overwrites the pod's imagePullSecrets with the hive pod's own list.
src/crux/kubejob/kubejob.go:662 sets podSpec["imagePullSecrets"] = relays.pullSecrets, and relays.go:128 fills that from the running hive pod's spec.imagePullSecrets (added by #18512 so the hive-relays init container can pull the hive image). Kubernetes' ServiceAccount admission only adds a service account's imagePullSecrets to a pod that has none of its own. So once the launcher sets a list, the Job's service account secrets are not applied.
Seen 2026-11-17 on a customer's standalone hive (OpenShift, 5.293.x candidate): the reviewer sandbox Job was created with serviceAccountName: hive-spyre-job and imagePullSecrets: [hive-dockercfg-tglnf] (the hive pod's own OpenShift dockercfg). The service account hive-spyre-job carries wxpe-cicd-aiu-internal and wxpe-cicd-vllm-iccr-internal (the secrets 24 other pods in the namespace use for this registry), but they were not on the pod. The hive-relays init container pulled the hive image fine; the kick container then failed:
Failed to pull image "us.icr.io/wxpe-cicd-internal/amd64/spyre-inference-dev:latest": ... unable to retrieve auth token: invalid username/password: unauthorized: Authorization required
The pod is stuck until the Job deadline. Any Job that lands on a node that does not already have the image cached fails the same way, so deps, quality and the reviewer all fail on a fresh node.
Expected
The Job pod's pull secrets are the union of: the hive pod's secrets (for the relays init image), the Job service account's imagePullSecrets (read the service account through the API if the launcher's grant allows get serviceaccounts; otherwise skip with a warning), and an explicit list sandbox.job.image_pull_secrets (names only; new config field, same merge precedence as the other job fields, settable from the dashboard sandbox section added in #18800). Names are deduplicated, order stable.
- At config load, warn when
sandbox.job.service_accountis set and the launcher cannot resolve its pull secrets (so the operator knows to setimage_pull_secrets). - A clearer failure: when a Job pod is in
ErrImagePull/ImagePullBackOfffor more than a configurable time (see ✨ Sandbox Jobs: pre-run node health check (sandbox.preflight) and fail-fast on an unschedulable Job #18811 for the stalled-Job detector), fail the kick with reasonimage_pull_failedand the registry message, instead of waiting for the deadline.
Acceptance
- Unit tests in
src/crux/kubejob: the pod spec carries the union (hive pod secrets + SA secrets + configured names, deduplicated); with no hive pod secrets and a configured list it still carries the configured names; an SA read denied by RBAC falls back with a warning, not an error. - Config tests for
image_pull_secretsmerge precedence and name validation (names only, no values); round trip set/save/reload (see 🐛 Kubernetes reload may drop dashboard-saved review and agent_sandbox settings (loadWithDashboardOverlay does not adopt them) #18804). - Docs (
src/docs/sandbox-isolation.md), changelog.d fragment; the sandbox UI field if ✨ Dashboard/API: set review.perspectives_source and agent sandbox image/block without editing config files #18800's surface makes it a small addition. - No secret values are read or logged; only names.
Workaround used on the affected hive until this ships: add the same secret names to the hive Deployment's pod template imagePullSecrets, so the launcher forwards them.
Related: #18512 (relays init container), #18811, #18797 / #18800, #18804.
Activity
clubanderson commentedon Nov 17, 2026
Claimed: an operator-dispatched agent is working on this now. Hive agents, please do not pick it up.
hivecommons-hive commentedon Nov 17, 2026
Thanks for filing this issue! To help us reproduce and investigate, could you please include the commit SHA of the build you're running?
You can find it by running:
Or check the bottom of the console UI for the version string.
This issue will be on hold until a SHA is provided. Simply add a comment with the SHA and the label will be automatically removed.
🐛 Sandbox Jobs keep their service account's imagePullSecrets (#18820)