Skip to content

🐛 Sandbox Jobs lose their service account's imagePullSecrets (launcher overwrites them with the hive pod's list), so private-registry project images fail ImagePullBackOff #18819

Closed

Description

@clubanderson

Problem

A sandbox Job whose project image lives in a private registry can fail with ImagePullBackOff even though the Job's service account carries the right registry secrets, because the launcher overwrites the pod's imagePullSecrets with the hive pod's own list.

src/crux/kubejob/kubejob.go:662 sets podSpec["imagePullSecrets"] = relays.pullSecrets, and relays.go:128 fills that from the running hive pod's spec.imagePullSecrets (added by #18512 so the hive-relays init container can pull the hive image). Kubernetes' ServiceAccount admission only adds a service account's imagePullSecrets to a pod that has none of its own. So once the launcher sets a list, the Job's service account secrets are not applied.

Seen 2026-11-17 on a customer's standalone hive (OpenShift, 5.293.x candidate): the reviewer sandbox Job was created with serviceAccountName: hive-spyre-job and imagePullSecrets: [hive-dockercfg-tglnf] (the hive pod's own OpenShift dockercfg). The service account hive-spyre-job carries wxpe-cicd-aiu-internal and wxpe-cicd-vllm-iccr-internal (the secrets 24 other pods in the namespace use for this registry), but they were not on the pod. The hive-relays init container pulled the hive image fine; the kick container then failed:

Failed to pull image "us.icr.io/wxpe-cicd-internal/amd64/spyre-inference-dev:latest": ... unable to retrieve auth token: invalid username/password: unauthorized: Authorization required

The pod is stuck until the Job deadline. Any Job that lands on a node that does not already have the image cached fails the same way, so deps, quality and the reviewer all fail on a fresh node.

Expected

The Job pod's pull secrets are the union of: the hive pod's secrets (for the relays init image), the Job service account's imagePullSecrets (read the service account through the API if the launcher's grant allows get serviceaccounts; otherwise skip with a warning), and an explicit list sandbox.job.image_pull_secrets (names only; new config field, same merge precedence as the other job fields, settable from the dashboard sandbox section added in #18800). Names are deduplicated, order stable.

Acceptance

Workaround used on the affected hive until this ships: add the same secret names to the hive Deployment's pod template imagePullSecrets, so the launcher forwards them.

Related: #18512 (relays init container), #18811, #18797 / #18800, #18804.

Activity

  1. added
    kind/bugCategorizes issue or PR as related to a bug.
    triage/acceptedIndicates an issue or PR is ready to be actively worked on.
    on Nov 17, 2026
  2. self-assigned this
    on Nov 17, 2026
  3. clubanderson commented on Nov 17, 2026

    @clubanderson
    MemberAuthor

    Claimed: an operator-dispatched agent is working on this now. Hive agents, please do not pick it up.

  4. added
    help wantedDenotes an issue that needs help from a contributor. Must meet "help wanted" guidelines.
    on Nov 17, 2026
  5. hivecommons-hive commented on Nov 17, 2026

    @hivecommons-hive
    Contributor

    Thanks for filing this issue! To help us reproduce and investigate, could you please include the commit SHA of the build you're running?

    You can find it by running:

    git rev-parse --short HEAD
    

    Or check the bottom of the console UI for the version string.

    This issue will be on hold until a SHA is provided. Simply add a comment with the SHA and the label will be automatically removed.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

Labels

help wantedDenotes an issue that needs help from a contributor. Must meet "help wanted" guidelines.kind/bugCategorizes issue or PR as related to a bug.needs-shatriage/acceptedIndicates an issue or PR is ready to be actively worked on.

Type

No type

Projects

No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions