Repository navigation
🐛 sandbox: archive job/podman run output as the kick log and gate PR-open on the agent's reported result - #14984
Conversation
…open on the agent's reported result
Sandboxed agents (Podman or runtime: job) have no tmux pane, so the card's
full log / log text buttons 404'd with 'capturing pane: exit status 1',
and the only copies of the run output — the Job pod log and the workspace
transcript — were gone within an hour or two. Hive now archives the
captured stdout/stderr as the kick log (<kick_log_dir>/<agent>/<start>-sandbox.log)
when the run ends, serves the newest archive from /api/agents/{name}/log,
lists them in the kick history and names the archive on the SandboxRun.
The executor also pushed + opened a PR whenever commits existed, so a run
whose build never succeeded still produced a PR that only failed CI.
agent-report.json gains an optional status (success/failed/blocked); a
failed/blocked status or non-zero sandbox exit keeps the commits local,
records prWithheld on the run (card: 'no PR opened: …') and audits
sandbox_pr_withheld.
Reported-by: @joerunde (a1-vllm-spyre spoke)
Signed-off-by: Andrew Anderson <andy@clubanderson.com>
|
[APPROVALNOTIFIER] This PR is NOT APPROVED This pull-request has been approved by: The full list of commands accepted by this bot can be found here. DetailsNeeds approval from an approver in each of these files:Approvers can indicate their approval by writing |
Open PR overlap checkBase branch: Conflicts are checked with every PR merged onto the current tip of
|
|
@clubanderson is auto-promoted for contributor-lane merging on this PR: the last 5 merged PRs by this author in this hive's repositories have no revert or sentinel-alert evidence. CI, reviewer verdict, diff fences, soak, holds and branch protection still apply. |
|
@clubanderson this PR is held at Failing checks:
Next: the hive does not push to your branch. Push a new head (or reply here); CI and the review swarm re-run automatically and the PR returns to the normal path. |
…leep (sleep ratchet) Signed-off-by: Andrew Anderson <andy@clubanderson.com>
There was a problem hiding this comment.
Conclusion: the log archive and the report-status gate are correct as far as I can trace them; the exit-code half of the publish gate is unreachable with the real launchers, so prWithheld / sandbox_pr_withheld / the card's no PR opened: sandbox exited N line never appear for a non-zero exit — the body says they do. Low severity (nothing is published either way), but worth a small fix before merge.
Finding — exit-code reason is dead for podman and job runtimes (low)
src/crux/sandbox/sandbox.go:70-76:PodmanLauncher.Runreturnscmd.Run()'s*exec.ExitErrorwhenever the container exits non-zero.src/crux/kubejob/kubejob.go:247: the Job launcher returnskubejob: job %s failedon aFailedcondition; it only reachesreturn res, nil(:255) onComplete, where the exit code is 0.src/crux/agent/sandbox_executor.go:191-194:if runErr != nil { res.Error = …; return res, runErr }runs before the new gate at:203, sosandboxPublishWithheld(:567-578) is only consulted when the launcher returnednil. With both production launchers that meansExitCode != 0at:574can never be true at that point; theTestSandboxExecutorPublishGatenon-zero-exit case passes only becausesandboxReportLauncherreturnsExitCode: l.exitCodewith a nil error (sandbox_executor_test.go:40).- Consequence: for the a1 failure mode where the container exits non-zero with commits, no PR is opened (pre-existing, correct), but
SandboxKickResult.PRWithheldstays empty,runSandboxKicknever emitssandbox_pr_withheld(manager_sandbox.go:276, gated onerrors.Is(err, ErrSandboxRunReportedFailure)), and the archive's outcome block carries onlyerror:. The body's "or the sandbox exit code ≠ 0 …PRWithheldcarry the reason" holds only for the report-status path. - Suggested fix: in the
runErr != nilbranch at:191, whencount > 0 && res.Sandbox.ExitCode != 0, setres.PRWithheld = fmt.Sprintf("sandbox exited %d", …)(or computesandboxPublishWithheldbefore that return). Then the existing test could use a launcher that returns an error alongside the exit code, which is what the real ones do.
Verified correct (so you don't have to re-derive it):
- Lock discipline in
runSandboxKick(manager_sandbox.go:214-228): the archive is built underm.muviaprepareSandboxKickLogLockedand written in a defer registered beforedefer m.mu.Unlock(), so LIFO ordering releases the lock first andwriteSandboxKickLog(sandbox_kick_log.go:97) really does run off-lock as its comment requires;pruneKickLogs/notifyKickObserverare safe there (notifyKickObserverspawns a goroutine,kick_observer.go:21-26). - Archive id
started.UTC().Format(kickLogTimestampFormat)+"-sandbox"+kickLogSuffix(sandbox_kick_log.go:92) is the same shapekick_logs.go:208writes, soreadKickLogDirlists andkickLogReasonparses it;readKickLogDirsorts by id descending (kick_logs.go:388), solatestKickLog'sinfos[0]is the newest. collectAgentReport's lenient status read lowercases/trims, andPublishBlockedtreats empty/unknown as open, so pre-existing reports publish exactly as before.FullLogHeaderreturning "" for sandboxed agents is consistent with the archive carrying its ownkickLogHeaderLockedheader.
I have not approved or changed anything.
— hive: agent=reviewer backend=copilot model=claude-fable-5.1 copilot=1.0.88
|
merged by Hive trusted-author auto-merge: author @clubanderson holds role owner and GitHub permission admin on repo; CI green at 2c04ab8. |
|
Thank you for your contribution! Your PR has been merged. We'd love to hear how your experience was: share feedback |
#15002) * ✨ feat(sandbox): deterministic publish.verify preflight before push/PR The report-status gate (#14984) trusts the agent. Add a hive-side preflight: sandbox.publish.verify (per agent), agent_sandbox.publish.verify (global) or an inline <!-- hive:publish.verify --> block in the kick template lists shell commands hive runs in the same sandbox image and workspace after the run, with HIVE_CHANGED_FILES / HIVE_CHANGED_TESTS / HIVE_BASE_SHA exported. The first non-zero exit, launcher error or timeout withholds the push and the PR; the reason and output tail land in prWithheld (card: 'no PR opened: verify failed …'), the kick log archive gets a '---- verify: <cmd> ----' section per command, and the existing sandbox_pr_withheld audit line fires. Motivated by torch-spyre/spyre-inference#1412, where a new tests/custom_ops/test_utils.py clashed with tests/multimodal/test_utils.py and broke pytest collection for every CI shard; a whole-suite --collect-only in the preflight would have stopped the PR. Fixes #14999 Reported-by: @joerunde Signed-off-by: Andrew Anderson <andy@clubanderson.com> * 🌱 chore(dashboard): allowlist sandbox publish.verify in settings parity The knob is a policy-file setting (template directive or hive.yaml) with no Settings control yet; track the control under #14999. Signed-off-by: Andrew Anderson <andy@clubanderson.com> --------- Signed-off-by: Andrew Anderson <andy@clubanderson.com>
Fixes #14981
Fixes #14982
Reported by @joerunde (a1-vllm-spyre spoke).
What broke on a1
The
depsagent runs each kick as a Kubernetes Job (sandbox.runtime: job). Two things followed from "no tmux pane":CaptureFullLograntmux capture-paneon the session name every agent is given at construction; the capture failed and the handler mapped it to 404. The only copies of the run output — the Job pod log (Job deleted on completion) and.hive/sandbox-transcript.log(workspace swept after 2h) — were gone when the owner looked.SandboxExecutor.Runpushed + opened a PR oncommitCount > 0alone. The agent's RPM install 401'd against Artifactory, it committed anyway, and spyre-inference#1402 went up to fail every CI shard. There was no field in the agent report to say "failed" and nothing read the exit code.Fix
Per-kick sandbox log archive (
crux/agent/sandbox_kick_log.go). When a sandboxed run ends hive writes<kick_log_dir>/<agent>/<kick-start>-sandbox.log: the standard kick-log header (source: sandbox-job/sandbox-podman), an outcome block (runtime, exit code, commits, branch, workspace, PR orpr_withheld, agent-report path + status, error), then stdout and stderr. Same retention/pruning as tmux kick logs;HIVE_KICK_LOG_RETENTION=0disables it.CaptureFullLogserves the newest archive for a sandboxed agent (and a plain "no archived run yet" message before the first run),FullLogHeaderreturns "" for them (archive carries its own), the kick history lists the archives, andSandboxRun.logIdnames the file. The SPA button tooltips say what the log is for a sandboxed agent.Publish gate (
crux/agent/sandbox_executor.go,crux/outputschema).AgentReportgains optionalstatus∈ {success,failed,blocked} (validated; empty = unspecified). After commits are counted: if status is failed/blocked, or the sandbox exit code ≠ 0, the run returnsErrSandboxRunReportedFailure, nothing is pushed, no PR is opened,SandboxKickResult.PRWithheld/SandboxRun.prWithheldcarry the reason (agent summary first, then exit code), the manager auditssandbox_pr_withheld workspace= commits= branch= reason=, and the card showsno PR opened: <reason>.statusis read straight from the JSON when the rest of the report fails validation, so a sloppy report that still says"status":"failed"still gates. Reports without a status publish exactly as before.Tests
outputschema: status accepted set,PublishBlocked.TestSandboxExecutorPublishGate: success/unspecified/no-report publish; failed/blocked/malformed-with-status/non-zero-exit withhold (no push, no PR, reason set, error wraps sentinel).TestSandboxedKickArchivesLogAndServesFullLog: job-runtime kick writes the archive,LogIDset,CaptureFullLog/ListKickLogs/ReadKickLogserve it, pre-run message; retention 0 writes nothing.sandboxRunHealthcarriesprWithheld(redacted) andlogId;buildSandboxRunboundsPRWithheld;sandboxKickLogBodyrendering.Docs
sandbox-isolation.md(kick path step 4 archive + publish gate, operator section, last-run record),agent-logging.md(sandboxed agents in per-kick history),dashboard-sections.md(agent card log buttons, sandboxed last-run line). Twochangelog.d/fixed-*fragments.Follow-up for spokes: kick templates for sandboxed agents should instruct the agent to write
"status": "failed"with the failure insummaryinstead of committing when install/build/smoke fails (a1'sdeps-torch-spyre.mdis being updated that way).