Skip to content

🐛 sandbox: archive job/podman run output as the kick log and gate PR-open on the agent's reported result - #14984

Merged
hivecommons-hive[bot] merged 2 commits into
v5from
fix/job-runtime-log-and-pr-gate
Oct 28, 2026
Merged

hivecommons-hive[bot] merged 2 commits into
v5from
fix/job-runtime-log-and-pr-gate

Conversation

@clubanderson

Copy link
Copy Markdown
Member

Fixes #14981
Fixes #14982

Reported by @joerunde (a1-vllm-spyre spoke).

What broke on a1

The deps agent runs each kick as a Kubernetes Job (sandbox.runtime: job). Two things followed from "no tmux pane":

  1. Full log / log text → 404. CaptureFullLog ran tmux capture-pane on the session name every agent is given at construction; the capture failed and the handler mapped it to 404. The only copies of the run output — the Job pod log (Job deleted on completion) and .hive/sandbox-transcript.log (workspace swept after 2h) — were gone when the owner looked.
  2. PR opened for a run that never built. SandboxExecutor.Run pushed + opened a PR on commitCount > 0 alone. The agent's RPM install 401'd against Artifactory, it committed anyway, and spyre-inference#1402 went up to fail every CI shard. There was no field in the agent report to say "failed" and nothing read the exit code.

Fix

Per-kick sandbox log archive (crux/agent/sandbox_kick_log.go). When a sandboxed run ends hive writes <kick_log_dir>/<agent>/<kick-start>-sandbox.log: the standard kick-log header (source: sandbox-job / sandbox-podman), an outcome block (runtime, exit code, commits, branch, workspace, PR or pr_withheld, agent-report path + status, error), then stdout and stderr. Same retention/pruning as tmux kick logs; HIVE_KICK_LOG_RETENTION=0 disables it. CaptureFullLog serves the newest archive for a sandboxed agent (and a plain "no archived run yet" message before the first run), FullLogHeader returns "" for them (archive carries its own), the kick history lists the archives, and SandboxRun.logId names the file. The SPA button tooltips say what the log is for a sandboxed agent.

Publish gate (crux/agent/sandbox_executor.go, crux/outputschema). AgentReport gains optional status ∈ {success,failed,blocked} (validated; empty = unspecified). After commits are counted: if status is failed/blocked, or the sandbox exit code ≠ 0, the run returns ErrSandboxRunReportedFailure, nothing is pushed, no PR is opened, SandboxKickResult.PRWithheld/SandboxRun.prWithheld carry the reason (agent summary first, then exit code), the manager audits sandbox_pr_withheld workspace= commits= branch= reason=, and the card shows no PR opened: <reason>. status is read straight from the JSON when the rest of the report fails validation, so a sloppy report that still says "status":"failed" still gates. Reports without a status publish exactly as before.

Tests

  • outputschema: status accepted set, PublishBlocked.
  • TestSandboxExecutorPublishGate: success/unspecified/no-report publish; failed/blocked/malformed-with-status/non-zero-exit withhold (no push, no PR, reason set, error wraps sentinel).
  • TestSandboxedKickArchivesLogAndServesFullLog: job-runtime kick writes the archive, LogID set, CaptureFullLog/ListKickLogs/ReadKickLog serve it, pre-run message; retention 0 writes nothing.
  • sandboxRunHealth carries prWithheld (redacted) and logId; buildSandboxRun bounds PRWithheld; sandboxKickLogBody rendering.

Docs

sandbox-isolation.md (kick path step 4 archive + publish gate, operator section, last-run record), agent-logging.md (sandboxed agents in per-kick history), dashboard-sections.md (agent card log buttons, sandboxed last-run line). Two changelog.d/fixed-* fragments.

Follow-up for spokes: kick templates for sandboxed agents should instruct the agent to write "status": "failed" with the failure in summary instead of committing when install/build/smoke fails (a1's deps-torch-spyre.md is being updated that way).

…open on the agent's reported result

Sandboxed agents (Podman or runtime: job) have no tmux pane, so the card's
full log / log text buttons 404'd with 'capturing pane: exit status 1',
and the only copies of the run output — the Job pod log and the workspace
transcript — were gone within an hour or two. Hive now archives the
captured stdout/stderr as the kick log (<kick_log_dir>/<agent>/<start>-sandbox.log)
when the run ends, serves the newest archive from /api/agents/{name}/log,
lists them in the kick history and names the archive on the SandboxRun.

The executor also pushed + opened a PR whenever commits existed, so a run
whose build never succeeded still produced a PR that only failed CI.
agent-report.json gains an optional status (success/failed/blocked); a
failed/blocked status or non-zero sandbox exit keeps the commits local,
records prWithheld on the run (card: 'no PR opened: …') and audits
sandbox_pr_withheld.

Reported-by: @joerunde (a1-vllm-spyre spoke)
Signed-off-by: Andrew Anderson <andy@clubanderson.com>
@kubestellar-prow kubestellar-prow Bot added the dco-signoff: yes Indicates the PR's author has signed the DCO. label Oct 28, 2026
@kubestellar-prow

Copy link
Copy Markdown
Contributor

[APPROVALNOTIFIER] This PR is NOT APPROVED

This pull-request has been approved by:
Once this PR has been reviewed and has the lgtm label, please assign danathar for approval. For more information see the Code Review Process.

The full list of commands accepted by this bot can be found here.

Details Needs approval from an approver in each of these files:

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

@kubestellar-prow kubestellar-prow Bot added the size/XL Denotes a PR that changes 500-999 lines, ignoring generated files. label Oct 28, 2026
@github-actions

github-actions Bot commented Oct 28, 2026 •

Copy link
Copy Markdown
Contributor

Open PR overlap check

Base branch: v5

Conflicts are checked with every PR merged onto the current tip of v5, so a PR that only needs a rebase is not reported as conflicting with its siblings.

PR Title Relation Files
#14987 ✨ review: gate PR reviews on an approval label (review.require_labels) same files src/docs/dashboard-sections.md
src/pkg/dashboard/static/index.html
#14979 🐛 fix: preserve supersession decisions across stage reconciliation same files src/pkg/dashboard/static/index.html
#14821 🐛 fix: preserve supersession decisions across stage reconciliation same files src/pkg/dashboard/static/index.html
#14807 ✨ dashboard: stream Job pod logs in the terminal pane for sandbox.runtime: job agents same files src/crux/agent/manager_sandbox.go
src/crux/agent/sandbox_executor.go
src/crux/agent/sandbox_executor_test.go
src/docs/sandbox-isolation.md
src/pkg/dashboard/static/index.html

@hivecommons-hive

Copy link
Copy Markdown
Contributor

@clubanderson is auto-promoted for contributor-lane merging on this PR: the last 5 merged PRs by this author in this hive's repositories have no revert or sentinel-alert evidence. CI, reviewer verdict, diff fences, soak, holds and branch protection still apply.

@hivecommons-hive hivecommons-hive Bot added author/human A person pushed this PR themselves dco All commits are DCO signed-off role/trusted Author is trusted for unattended merge (trusted-author role or registry/claim tier in require_tiers) stage/050-review-pending Review swarm requested; awaiting a verdict for the current head stage/071-needs-author Human-authored PR: CI red, changes requested or conflicting; diagnosis posted, the author must act and removed stage/050-review-pending Review swarm requested; awaiting a verdict for the current head labels Oct 28, 2026
@hivecommons-hive

Copy link
Copy Markdown
Contributor

@clubanderson this PR is held at stage/071-needs-author and the hive will not merge it yet.

Failing checks:

  • test (rest 2/4)

Next: the hive does not push to your branch. Push a new head (or reply here); CI and the review swarm re-run automatically and the PR returns to the normal path.

…leep (sleep ratchet)

Signed-off-by: Andrew Anderson <andy@clubanderson.com>
@hivecommons-hive hivecommons-hive Bot added stage/050-review-pending Review swarm requested; awaiting a verdict for the current head and removed stage/071-needs-author Human-authored PR: CI red, changes requested or conflicting; diagnosis posted, the author must act labels Oct 28, 2026

@hivecommons-hive hivecommons-hive Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Conclusion: the log archive and the report-status gate are correct as far as I can trace them; the exit-code half of the publish gate is unreachable with the real launchers, so prWithheld / sandbox_pr_withheld / the card's no PR opened: sandbox exited N line never appear for a non-zero exit — the body says they do. Low severity (nothing is published either way), but worth a small fix before merge.

Finding — exit-code reason is dead for podman and job runtimes (low)

  • src/crux/sandbox/sandbox.go:70-76: PodmanLauncher.Run returns cmd.Run()'s *exec.ExitError whenever the container exits non-zero.
  • src/crux/kubejob/kubejob.go:247: the Job launcher returns kubejob: job %s failed on a Failed condition; it only reaches return res, nil (:255) on Complete, where the exit code is 0.
  • src/crux/agent/sandbox_executor.go:191-194: if runErr != nil { res.Error = …; return res, runErr } runs before the new gate at :203, so sandboxPublishWithheld (:567-578) is only consulted when the launcher returned nil. With both production launchers that means ExitCode != 0 at :574 can never be true at that point; the TestSandboxExecutorPublishGate non-zero-exit case passes only because sandboxReportLauncher returns ExitCode: l.exitCode with a nil error (sandbox_executor_test.go:40).
  • Consequence: for the a1 failure mode where the container exits non-zero with commits, no PR is opened (pre-existing, correct), but SandboxKickResult.PRWithheld stays empty, runSandboxKick never emits sandbox_pr_withheld (manager_sandbox.go:276, gated on errors.Is(err, ErrSandboxRunReportedFailure)), and the archive's outcome block carries only error:. The body's "or the sandbox exit code ≠ 0 … PRWithheld carry the reason" holds only for the report-status path.
  • Suggested fix: in the runErr != nil branch at :191, when count > 0 && res.Sandbox.ExitCode != 0, set res.PRWithheld = fmt.Sprintf("sandbox exited %d", …) (or compute sandboxPublishWithheld before that return). Then the existing test could use a launcher that returns an error alongside the exit code, which is what the real ones do.

Verified correct (so you don't have to re-derive it):

  • Lock discipline in runSandboxKick (manager_sandbox.go:214-228): the archive is built under m.mu via prepareSandboxKickLogLocked and written in a defer registered before defer m.mu.Unlock(), so LIFO ordering releases the lock first and writeSandboxKickLog (sandbox_kick_log.go:97) really does run off-lock as its comment requires; pruneKickLogs/notifyKickObserver are safe there (notifyKickObserver spawns a goroutine, kick_observer.go:21-26).
  • Archive id started.UTC().Format(kickLogTimestampFormat)+"-sandbox"+kickLogSuffix (sandbox_kick_log.go:92) is the same shape kick_logs.go:208 writes, so readKickLogDir lists and kickLogReason parses it; readKickLogDir sorts by id descending (kick_logs.go:388), so latestKickLog's infos[0] is the newest.
  • collectAgentReport's lenient status read lowercases/trims, and PublishBlocked treats empty/unknown as open, so pre-existing reports publish exactly as before.
  • FullLogHeader returning "" for sandboxed agents is consistent with the archive carrying its own kickLogHeaderLocked header.

I have not approved or changed anything.

— hive: agent=reviewer backend=copilot model=claude-fable-5.1 copilot=1.0.88

@hivecommons-hive
hivecommons-hive Bot merged commit c41fcd6 into v5 Oct 28, 2026
78 of 80 checks passed
@kubestellar-prow
kubestellar-prow Bot deleted the fix/job-runtime-log-and-pr-gate branch October 28, 2026 16:13
@hivecommons-hive

Copy link
Copy Markdown
Contributor
Evidence bundle `#149842c04ab8` sha256:a3c8eb3f0653d817a8e174884a2f86b9fa116068a12a9169a8323839ea74752c — download from the dashboard or `hivectl review evidence #14984`.

@hivecommons-hive

Copy link
Copy Markdown
Contributor

merged by Hive trusted-author auto-merge: author @clubanderson holds role owner and GitHub permission admin on repo; CI green at 2c04ab8.

@github-actions

Copy link
Copy Markdown
Contributor

Thank you for your contribution! Your PR has been merged.

We'd love to hear how your experience was: share feedback

@hivecommons-hive hivecommons-hive Bot removed role/trusted Author is trusted for unattended merge (trusted-author role or registry/claim tier in require_tiers) stage/050-review-pending Review swarm requested; awaiting a verdict for the current head labels Oct 28, 2026
hivecommons-hive Bot pushed a commit that referenced this pull request Oct 28, 2026
#15002)

* ✨ feat(sandbox): deterministic publish.verify preflight before push/PR

The report-status gate (#14984) trusts the agent. Add a hive-side
preflight: sandbox.publish.verify (per agent), agent_sandbox.publish.verify
(global) or an inline <!-- hive:publish.verify --> block in the kick
template lists shell commands hive runs in the same sandbox image and
workspace after the run, with HIVE_CHANGED_FILES / HIVE_CHANGED_TESTS /
HIVE_BASE_SHA exported. The first non-zero exit, launcher error or timeout
withholds the push and the PR; the reason and output tail land in
prWithheld (card: 'no PR opened: verify failed …'), the kick log archive
gets a '---- verify: <cmd> ----' section per command, and the existing
sandbox_pr_withheld audit line fires.

Motivated by torch-spyre/spyre-inference#1412, where a new
tests/custom_ops/test_utils.py clashed with tests/multimodal/test_utils.py
and broke pytest collection for every CI shard; a whole-suite
--collect-only in the preflight would have stopped the PR.

Fixes #14999
Reported-by: @joerunde

Signed-off-by: Andrew Anderson <andy@clubanderson.com>

* 🌱 chore(dashboard): allowlist sandbox publish.verify in settings parity

The knob is a policy-file setting (template directive or hive.yaml) with no
Settings control yet; track the control under #14999.

Signed-off-by: Andrew Anderson <andy@clubanderson.com>

---------

Signed-off-by: Andrew Anderson <andy@clubanderson.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

author/human A person pushed this PR themselves dco All commits are DCO signed-off dco-signoff: yes Indicates the PR's author has signed the DCO. size/XL Denotes a PR that changes 500-999 lines, ignoring generated files.

Projects

None yet

1 participant