Skip to content

✨ Sandbox Jobs: authenticated GitHub reads via a read-only hive relay - #18868

Merged
kubestellar-prow[bot] merged 10 commits into
v5from
feat/18865-sandbox-github-read-relay
Nov 18, 2026
Merged

kubestellar-prow[bot] merged 10 commits into
v5from
feat/18865-sandbox-github-read-relay

Conversation

@clubanderson

Copy link
Copy Markdown
Member

Fixes #18865

Owner decision on the issue: option 1. Sandbox Job GitHub reads go through hive, which attaches a read identity the Job never sees.

Why this is a relay and not just the inject flag

Sandbox Jobs run in their own pods. Today they bypass the hive proxy entirely: the proxy listens on loopback, identifies callers by socket UID, and kubejob sets no proxy or CA env. So HIVE_PROXY_INJECT_GH_AUTH cannot reach a Job. This PR wires the Job through a second, narrow listener that reuses the proxy's CA, cert forging, destination-guarded dialers and upstream chain.

What it does (default off)

Config: agent_sandbox.job.github_read_proxy with enabled, endpoint (required, how Job pods reach hive), listen_addr (default :18445), no_proxy. Hive-wide only; a per-agent job block cannot enable or repoint it. Documented in src/docs/sandbox-isolation.md.

When enabled, each sandbox.runtime: job agent kick gets HTTPS_PROXY (with a per-kick relay key), a CA bundle written to <workspace>/.hive/github-read-ca.pem (system roots plus hive CA), and the usual SSL_CERT_FILE/GIT_SSL_CAINFO/etc. env.

Host Request Result
api.github.com, codeload.github.com, raw.githubusercontent.com GET, HEAD agent Authorization stripped, read token attached
github.com GET info/refs?service=git-upload-pack, POST git-upload-pack same, Basic x-access-token
github.com other GET/HEAD forwarded with no credential
those four hosts any other method, GraphQL, git-receive-pack (incl. push discovery) 403, never sent upstream
any other host CONNECT 443 opaque tunnel, no injection, destination guard applies

Credential

The existing shared-uid-read tier (metadata, contents, actions, checks, pull requests, issues: all read) via ScopedTokenForRepos, narrowed to project.repos. The codebase does distinguish read and write tiers, so no new credential type. It is never the write token, never in the Job env, files or logs. The Job holds only a per-kick relay key (revoked when the kick ends) that authenticates it to hive's listener. Token cached 45 min in hive memory. If no read token can be minted, reads go out anonymously as today and the agent's own Authorization is still stripped.

Tests (not run locally per repo rules; CI is the gate)

  • Policy table: every write method, GraphQL, receive-pack, push discovery, non-GitHub, lookalike and GHE hosts are not injected.
  • End-to-end through the real handler with a fake upstream: GET gets the read token and the agent's header is replaced, POST is 403 and opens no upstream connection, wrong/absent/released key is 407, non-443 CONNECT refused, no-token path is anonymous.
  • Enroll output and Job env carry no GitHub token; kubejob manifest drops GITHUB_TOKEN/GH_TOKEN even if allowlisted.
  • Config: default off, endpoint validation, per-agent block cannot override.

Notes for review

🤖 Generated with Claude Code

@kubestellar-prow kubestellar-prow Bot added dco-signoff: yes Indicates the PR's author has signed the DCO. size/XXL Denotes a PR that changes 1000+ lines, ignoring generated files. labels Nov 18, 2026
@hivecommons-hive

hivecommons-hive Bot commented Nov 18, 2026 •

Copy link
Copy Markdown
Contributor

Hive status — updated in place; edits do not notify. Full history is in the Activity Monitor.

clubanderson this PR is held at stage/071-needs-author and the hive will not merge it yet.

Why:

  • the branch conflicts with the base and needs a rebase

Next: the hive does not push to your branch. Push a new head (or reply here); CI and the review swarm re-run automatically and the PR returns to the normal path.

Review findings from reviewer (folded here because this head already carries its hive review):

Read the relay (proxy/sandbox_read_relay.go), its boot wiring and the executor enrol/release path. The read-only policy, key-hash lookup, header stripping and dial-on-first-allowed-request look correct to me, and I found no path that attaches a credential off the four read hosts or on a write method. One robustness finding, on the line below; it needs no maintainer decision.

Open question (not verified, no code cited): whether ScopedTokenForRepos returns a cached token whose remaining life is shorter than the relay's 45-minute sandboxReadTokenTTL; if it does, injected reads can 401 near expiry.

Not approving; that is a human's call.

Findings not on a changed line

  • src/crux/proxy/sandbox_read_relay.go:229: Medium: readToken holds tokMu across r.opts.Token(), which can block up to sandboxReadTokenTimeout (20s) in sandboxReadToken. On error it sets r.tok = "" and leaves tokFetched unchanged, so there is no negative cache. During a GitHub App/token-endpoint outage every injected request on every Job connection re-attempts the mint, serialized behind the mutex: each waits up to 20s per attempt, the clients' reads stall, and the token endpoint is hit once per request. Suggested fix: remember the failure time and skip re-minting for ~30-60s (return "" immediately in that window), and/or mint outside the lock.

Review findings from reviewer-2 (folded here because this head already carries its hive review):

Needs a maintainer call because this adds a new credential-bearing listener. I read the relay, policy, enroll path and config in the diff; I found no read-only bypass.

Verified OK: SandboxReadPolicy (src/crux/proxy/sandbox_read_relay.go:~95-125) refuses every non-GET/HEAD to the four read hosts, push discovery and git-receive-pack. applySandboxReadAuth strips any Job-supplied Authorization before attaching the token. The per-agent GitHubReadProxy override is reset in SandboxJob(). Release is deferred in sandbox_executor.go. Per #18865 option 1 the token is a read tier that never enters the Job. The policy table covers each of these.

Open decision (the question above). DefaultSandboxGitHubReadProxyListen = ":18445" binds every interface, and Validate() accepts an http:// endpoint, so the per-kick key travels as Basic auth in cleartext on the pod network. The key is worthless to GitHub but grants use of the repo-scoped read token for the kick's lifetime. The PR body says Service/NetworkPolicy is the operator's job; that is a reasonable position, but it is not stated in the issue, so it is yours to accept or tighten (for example a loopback/Service-IP default, or a note in sandbox-isolation.md).

Low, noted: the relay key is checked only at CONNECT. An already-established tunnel keeps working after release() for as long as the client keeps sending requests (each read gets a fresh httpReadTimeout). The Job pod is torn down at kick end, so impact is small.

One line finding is inline (token mint failure path).

Findings not on a changed line

  • src/crux/proxy/sandbox_read_relay.go:229: Mint failures are never cached: on error r.tok is cleared and tokFetched is left stale, so the next injectable request calls r.opts.Token() again, and tokMu is held across the call (up to sandboxReadTokenTimeout, 20s). If the App token endpoint is down or rate-limited, every concurrent Job read serializes behind a 20s mint attempt and each one re-hits GitHub's App API. Suggest recording the failure time and skipping retries for a short backoff (e.g. 30s), returning "" (anonymous) meanwhile, and not holding the lock during the network call. Severity: medium-low; only bites during a token outage.

Review findings from adjudicator-2 (folded here because this head already carries its hive review):

HUMAN DECISION NEEDED — confirm a sandbox Job may read every project repo (incl. private) through the relay's single shared read token. clubanderson

Delta re-review at fdf97f0. The relay code (src/crux/proxy/sandbox_read_relay.go) holds up: reads only (GET/HEAD plus git-upload-pack), Job-supplied Authorization stripped, no injection off the four GitHub hosts, Host/CONNECT mismatch refused, TLS-only listener, per-kick key stored hashed and revoked on release. No new exploitable issue found in the delta.

Previous finding src/cmd/hive/sandbox_read_relay_boot.go:92 (relay token spans all project repos): still present. ScopedTokenForRepos(ctx, SharedUIDReadTokenTier, b.cfg.Project.Repos) mints one token for all project repos, and the relay attaches it for any repo a Job names. A sandbox agent for one repo can therefore read the others, private ones included. Issue #18865 says "read-only" but not which repos, so the repo scope is a human call. If per-agent narrowing is wanted, mint per kick from the agent's repo set; otherwise record the broader scope in docs/security-model.md.

Limits: I read only the security-relevant files in the 84-file delta (relay, boot wiring, executor, CA forging); the dashboard, automerge and config-UI changes were not reviewed.

Review findings from adjudicator-4 (folded here because this head already carries its hive review):

Style re-review at fdf97f0: no blockers. Advisory: the PR body still says listen_addr defaults to :18445. The code (src/crux/config/sandbox_job_types.go:41-44), sandbox-isolation.md:295 and the changelog all say there is no default and https:// is required, so please update the PR description. The prior security finding (sandbox_read_relay_boot.go:92, token spans all project repos) is still present and not re-filed here.

@hivecommons-hive hivecommons-hive Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Correctness review at head 4327d29.

  1. High — does not compile. src/cmd/hive/sandbox_read_relay_boot.go:82 calls b.agentMgr.SetSandboxGitHubReadRelay, but agentMgr is hostedRuntime (crux.Runtime + crux.Host) and the method is only added to *agent.Manager. crux.Host (src/crux/host_wiring.go) and the other implementers are untouched; CI build-go / build-and-test / golangci-lint are failing. Add it to crux.Host and implement it everywhere; decide what sidecar mode does, since its Job kicks run out of process.
  2. Low — read-token mint failures are not backed off (see inline).

The policy table, key auth, header stripping and read-only enforcement read correctly.

— hive: agent=adjudicator backend=claude model=claude-sonnet-5-5 claude=2.1.326

// issues, all read), narrowed to the project's repositories. It is never the
// write-capable credential.
func (b *boot) sandboxReadToken() (string, error) {
auth := b.appAuth

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Build break. b.agentMgr is hostedRuntime (crux.Runtime + crux.Host). SetSandboxGitHubReadRelay exists only on *agent.Manager; it is not in crux.Host (src/crux/host_wiring.go) and the PR doesn't touch that file, sidecarHost, or the cruxtest fakes. CI build-go/build-and-test/golangci-lint are red. Add the method to crux.Host and implement it on every Host (sidecarHost: forward or no-op with a warning, since sidecar-mode Job kicks run out of process).

}, nil
}

// authenticate returns the agent a Proxy-Authorization header belongs to.

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Mint failures aren't negative-cached: tokMu is held across opts.Token() (up to 20s), and a failure leaves tokFetched stale, so during an App outage every injectable request retries serially. Suggest a short failure backoff (e.g. 30s) before retrying.

@hivecommons-hive

Copy link
Copy Markdown
Contributor

Scanner (suggestion only; branch is busy): build-go fails to compile: cmd/hive/sandbox_read_relay_boot.go:73:13: b.agentMgr.SetSandboxGitHubReadRelay undefined (type hostedRuntime has no field or method SetSandboxGitHubReadRelay). Add SetSandboxGitHubReadRelay to the hostedRuntime interface (or call it on the concrete manager type) and any test fakes implementing it. The test/coverage, golangci-lint, govulncheck, crux-file-size-ratchet and docker failures likely follow from the same compile break or need their own logs once it is fixed.
hive/awaiting-ci: held for the author's push; not touched by scanner.


🐝 Hive Agent: scanner | Instance: hosted-available-oke-11-placeholder-r05x | SHA: unknown

— hive: agent=scanner backend=claude model=claude-sonnet-5-5 claude=2.1.326

@github-actions

github-actions Bot commented Nov 18, 2026 •

Copy link
Copy Markdown
Contributor

Open PR overlap check

Base branch: v5

Conflicts are checked with every PR merged onto the current tip of v5, so a PR that only needs a rebase is not reported as conflicting with its siblings.

PR Title Relation Files
#18873 🐛 needs-sha: exempt owner/detailed issues, overrideable waiver, visible hold on the row same files src/hive.yaml.example
#18904 ✨ Expose the knowledge advisor rank-versus-holdout summary through a read-only route and Advisor card same files src/cmd/hive/main.go
#18913 🐛 Contributor lane: grace window before claiming owner/hive-filed issues same files src/cmd/hive/main.go
src/pkg/dashboard/settings_parity_allowlist.txt

@clubanderson

Copy link
Copy Markdown
Member Author

Owner decision on the reviewer-2 needs-decision item (listener bind and cleartext key), implemented in head a0579d9:

  1. No default bind. agent_sandbox.job.github_read_proxy.listen_addr has no default; enabled without it fails config validation with an error naming the key. The feature stays default off.
  2. TLS, not cleartext. The listener serves TLS only, with a certificate issued by the existing hive proxy CA for the endpoint host (DNS or IP SAN; no new CA, no new key on disk). endpoint must be https://; http:// is rejected at validation. Jobs verify the proxy against the hive CA bundle, and git gets http.proxySSLCAInfo for libcurl. Tests: plain HTTP is refused and its key is never logged, no key still gets 407, SAN verification.
  3. Docs: example NetworkPolicy in sandbox-isolation.md, stating the NetworkPolicy is the first layer and the per-kick key the second, and that the feature is default off. Clients that cannot speak TLS to a proxy fall back to anonymous reads; hive does not downgrade to cleartext.

Left out by the same decision: the tunnel that outlives its released key, tracked in #18895.

@hivecommons-hive hivecommons-hive Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Decision needed: @clubanderson should every sandbox Job kick get a read token covering all of project.repos, or should the relay scope it to the repos that kick is working on?

This adds a credential-attaching network listener and mints a GitHub App read token for Jobs, so it is a trust-surface change a person should sign off on. I read crux/proxy/sandbox_read_relay.go and cmd/hive/sandbox_read_relay_boot.go in full and did not find a defect in the mechanics.

What I checked and found sound:

  • SandboxReadPolicy only attaches a credential to GET/HEAD on the four fixed hosts, plus git-upload-pack POST. Push discovery and every other method get a 403.
  • applySandboxReadAuth strips the Job's own Authorization and Proxy-Authorization before attaching anything.
  • sandboxReadRequestTargetsHost stops a request inside the CONNECT tunnel from naming another host and riding the credential decision.
  • The listener is TLS-only, and the per-kick key is stored as a SHA-256 digest and revoked on release.

Open question (the only reason this is routed): sandboxReadToken (sandbox_read_relay_boot.go:92) mints one shared-uid-read token for all of b.cfg.Project.Repos. Any Job that holds a relay key can therefore read every repo in the project, including private ones, through api.github.com and raw.githubusercontent.com. The linked issue's option-1 decision does not say whether that breadth is intended. I could not find a per-agent or per-kick narrowing in the diff; please correct me if it exists.

Not verified: the 1,891-line diff is mostly tests and docs, and I did not run anything. Concurrency of the keys map and readToken looked correct on a read.

— hive: agent=reviewer-2 backend=claude model=claude-sonnet-5-5 claude=2.1.326

}
ctx, cancel := context.WithTimeout(context.Background(), sandboxReadTokenTimeout)
defer cancel()
return auth.ScopedTokenForRepos(ctx, crux.SharedUIDReadTokenTier, b.cfg.Project.Repos)

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This token is scoped to all b.cfg.Project.Repos and cached for 45 minutes, then attached to every Job's reads. A Job working on one repo can read every other project repo, private ones included. If that is intended, say so in sandbox-isolation.md. If not, mint per-kick tokens for the repos the kick needs.

@github-actions github-actions Bot removed the conflicts-with-open-pr Open PR has a merge conflict with another open PR label Nov 18, 2026
…elay (#18865)

Adds an opt-in relay (agent_sandbox.job.github_read_proxy, default off) that sandbox Jobs reach through HTTPS_PROXY. It attaches a read-only, repo-scoped App token (shared-uid-read tier) to GET/HEAD for api.github.com, codeload.github.com, raw.githubusercontent.com and git fetch on github.com; refuses writes, GraphQL and push discovery; tunnels other hosts untouched. The Job holds only a per-kick relay key, never a GitHub credential.

Signed-off-by: Andrew Anderson <andy@clubanderson.com>
…host check (#18865)

- Add SetSandboxGitHubReadRelay to crux.Host and implement it on every
  Host: Manager, the sidecar host (in-process-only), the cruxtest recorder,
  and the api HostSetters table. boot calls it through the interface.
- Move SandboxGitHubReadRelay into the crux package (agent aliases it) and
  keep the Manager's relay wiring in one atomic value so the relay and its
  NO_PROXY hosts can never be read from different installs.
- Keep agent/manager.go and config/agent_config.go under their ratchets by
  moving code out (ACMM level accessors, sandbox job types and the relay
  config) instead of raising any limit.
- Refuse a request inside a tunnel whose Host or absolute URL names a host
  other than the CONNECT target, so the credential decision cannot be
  borrowed for another host.
- Fix the boot test's Config literal (AgentSandbox is promoted).
- Tests: relay logs never contain the token, relay key or agent header;
  keys are per kick; Host mismatch is refused without an upstream dial;
  manager install/clear copies noProxy.

Signed-off-by: Andy Anderson <andy@clubanderson.com>
…ratchet (#18865)

Signed-off-by: Andy Anderson <andy@clubanderson.com>
… SA4023) (#18865)

Signed-off-by: Andy Anderson <andy@clubanderson.com>
… retrying per request (#18865)

Signed-off-by: Andy Anderson <andy@clubanderson.com>
… v5 merge (#18865)

Signed-off-by: Andy Anderson <andy@clubanderson.com>
…int, NetworkPolicy docs (#18865)

Owner decision on the needs-decision review item:
- listen_addr has no default; enabling without it fails config validation
  with an error naming the key (default off is unchanged).
- The listener serves TLS only, with a certificate issued by the existing
  hive proxy CA for the endpoint host (DNS or IP SAN); endpoint must be
  https:// and http:// is rejected at validation. No new CA or key on disk.
- Jobs verify the https proxy against the hive CA bundle, including
  libcurl's separate proxy CA setting for git (http.proxySSLCAInfo).
- Tests: plain HTTP is refused and its key never logged, no key still gets
  407 over TLS, certificate SANs, config validation, env.
- Docs: example NetworkPolicy, two-layer statement, client TLS-to-proxy
  limits, example config, changelog fragment.

Follow-up for the tunnel that outlives its key: #18895.

Signed-off-by: Andy Anderson <andy@clubanderson.com>
@clubanderson
clubanderson force-pushed the feat/18865-sandbox-github-read-relay branch from a0579d9 to f05bcfd Compare November 18, 2026 04:51
Signed-off-by: Andrew Anderson <andy@clubanderson.com>
@github-actions github-actions Bot removed the conflicts-with-open-pr Open PR has a merge conflict with another open PR label Nov 18, 2026
@clubanderson

Copy link
Copy Markdown
Member Author

Owner confirms the reviewer-2 needs-decision answers above (no default bind address, TLS only with a certificate from the hive proxy CA, feature default off). Proceeding to rebase onto v5 and merge.

's sandbox job types)

Signed-off-by: Andy Anderson <andy@clubanderson.com>
@github-actions github-actions Bot added the conflicts-with-open-pr Open PR has a merge conflict with another open PR label Nov 18, 2026

@hivecommons-hive hivecommons-hive Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

HUMAN DECISION NEEDED — @clubanderson: accept or narrow the read token's scope (credential policy): the relay mints one shared-uid-read token spanning every project.repos entry for all Jobs.

Intent-alignment at fdf97f0 (delta from a0579d9, which is mostly merges from base; the PR's own 31 files stay scoped to #18865, no scope creep):

  • Issue #18865 acceptance (authenticated reads, no write capability, token out of the Job): met by src/crux/proxy/sandbox_read_relay.go (GET/HEAD plus git-upload-pack only; other methods get 403) and src/crux/agent/sandbox_github_read_relay.go (the env carries a relay key, no GitHub token).
  • Previous security finding (src/cmd/hive/sandbox_read_relay_boot.go:92, token spans all project repos): still present. ScopedTokenForRepos(ctx, SharedUIDReadTokenTier, b.cfg.Project.Repos) is unchanged, and no per-agent or per-kick narrowing was added. It needs a human decision, not an agent fix.
  • Description drift (low): the PR body says listen_addr defaults to :18445. src/crux/config/sandbox_job_types.go says there is deliberately no default and requires it when enabled. Update the body or the code so they agree.
  • Not verified: I did not run the tests, and the uv lock --upgrade no-clone acceptance is not tested in the diff.

— hive: agent=adjudicator-3 backend=claude model=claude-sonnet-5-5 claude=2.1.326

@github-actions github-actions Bot removed the conflicts-with-open-pr Open PR has a merge conflict with another open PR label Nov 18, 2026
…ub-read-relay

Signed-off-by: Andrew Anderson <andy@clubanderson.com>

# Conflicts:
#	src/hive.yaml.example
@clubanderson clubanderson added lgtm Indicates that a PR is ready to be merged. approved Indicates a PR has been approved by an approver from all required OWNERS files. labels Nov 18, 2026
@kubestellar-prow

Copy link
Copy Markdown
Contributor

[APPROVALNOTIFIER] This PR is APPROVED

Approval requirements bypassed by manually added approval.

This pull-request has been approved by:

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Details Needs approval from an approver in each of these files:

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

@kubestellar-prow
kubestellar-prow Bot merged commit db67bf9 into v5 Nov 18, 2026
98 checks passed
@kubestellar-prow
kubestellar-prow Bot deleted the feat/18865-sandbox-github-read-relay branch November 18, 2026 05:25
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

approved Indicates a PR has been approved by an approver from all required OWNERS files. dco-signoff: yes Indicates the PR's author has signed the DCO. lgtm Indicates that a PR is ready to be merged. size/XXL Denotes a PR that changes 1000+ lines, ignoring generated files.

Projects

None yet

Development

Successfully merging this pull request may close these issues.

🐛 Sandbox agents hit GitHub's anonymous 60/hr rate limit; uv falls back to full git clones

1 participant