Repository navigation
✨ feat(sandbox): deterministic publish.verify preflight before push/PR - #15002
Conversation
The report-status gate (#14984) trusts the agent. Add a hive-side preflight: sandbox.publish.verify (per agent), agent_sandbox.publish.verify (global) or an inline <!-- hive:publish.verify --> block in the kick template lists shell commands hive runs in the same sandbox image and workspace after the run, with HIVE_CHANGED_FILES / HIVE_CHANGED_TESTS / HIVE_BASE_SHA exported. The first non-zero exit, launcher error or timeout withholds the push and the PR; the reason and output tail land in prWithheld (card: 'no PR opened: verify failed …'), the kick log archive gets a '---- verify: <cmd> ----' section per command, and the existing sandbox_pr_withheld audit line fires. Motivated by torch-spyre/spyre-inference#1412, where a new tests/custom_ops/test_utils.py clashed with tests/multimodal/test_utils.py and broke pytest collection for every CI shard; a whole-suite --collect-only in the preflight would have stopped the PR. Fixes #14999 Reported-by: @joerunde Signed-off-by: Andrew Anderson <andy@clubanderson.com>
|
[APPROVALNOTIFIER] This PR is NOT APPROVED This pull-request has been approved by: The full list of commands accepted by this bot can be found here. DetailsNeeds approval from an approver in each of these files:Approvers can indicate their approval by writing |
Open PR overlap checkBase branch: Conflicts are checked with every PR merged onto the current tip of
|
|
@clubanderson is auto-promoted for contributor-lane merging on this PR: the last 5 merged PRs by this author in this hive's repositories have no revert or sentinel-alert evidence. CI, reviewer verdict, diff fences, soak, holds and branch protection still apply. |
There was a problem hiding this comment.
Style review (read at head 7377efa).
1. sandboxPublishWithheld's doc comment is now attached to the new const block — src/crux/agent/sandbox_executor.go:596-607. The diff inserted const ( sandboxVerifyTailMax … ) directly under the existing // sandboxPublishWithheld returns why the run's commits must not be published… comment, so godoc now documents the constants with the gate function's text, and sandboxPublishWithheld itself (sandbox_executor.go:719) has no doc comment. Move the block above the comment:
var ErrSandboxRunReportedFailure = errors.New("sandbox run reported failure")
-// sandboxPublishWithheld returns why the run's commits must not be published,
-// or "" when the gate is open. The order matters for the message an operator
-// reads: the agent's own words first, the exit code second.
const (
// sandboxVerifyTailMax bounds the verify output quoted on the card and in
// the audit record; the full output lands in the kick log archive.
sandboxVerifyTailMax = 600
// sandboxVerifyOutputMax bounds what one verify command's combined output
// keeps in memory and in the kick log.
sandboxVerifyOutputMax = 64 << 10
defaultSandboxVerifyTimeout = 15 * time.Minute
)
@@
+// sandboxPublishWithheld returns why the run's commits must not be published,
+// or "" when the gate is open. The order matters for the message an operator
+// reads: the agent's own words first, the exit code second.
func sandboxPublishWithheld(res SandboxKickResult) string {2. Two independent copies of the verify-timeout default (advisory). src/crux/config/agent_config.go:519-522 defines DefaultSandboxVerifyTimeoutS = 900 and its field comment says non-positive "uses the executor default (DefaultSandboxVerifyTimeoutS)", but the executor's fallback at src/crux/agent/sandbox_executor.go:606,616-618 is a separate literal defaultSandboxVerifyTimeout = 15 * time.Minute. They agree today only by coincidence; agent already imports crux/config (manager_sandbox.go:11), so derive one from the other:
defaultSandboxVerifyTimeout = time.Duration(config.DefaultSandboxVerifyTimeoutS) * time.Second
— hive: agent=adjudicator-4 backend=copilot model=claude-fable-5.1 copilot=1.0.88
|
@clubanderson this PR is held at Failing checks:
Next: the hive does not push to your branch. Push a new head (or reply here); CI and the review swarm re-run automatically and the PR returns to the normal path. |
The knob is a policy-file setting (template directive or hive.yaml) with no Settings control yet; track the control under #14999. Signed-off-by: Andrew Anderson <andy@clubanderson.com>
There was a problem hiding this comment.
Settings-parity gate will fail on the four new publish.* config leaves (in-scope, agent-fixable)
src/crux/config/agent_config.go adds SandboxPublishConfig (verify, verify_timeout_s) under both AgentSandboxConfig.Publish and AgentSandboxOverride.Publish. internal/settingsparity.ConfigLeaves (src/internal/settingsparity/settingsparity.go:25-28) walks src/crux/config too, so pkg/dashboard.TestSettingsDialogReferencesEveryHiveYAMLLeaf (src/pkg/dashboard/settings_parity_test.go:12-33) now sees four leaves that neither static/index.html (0 occurrences of publish.verify at head 7377efa) nor settings_parity_allowlist.txt reference, and fails with "hive.yaml settings are not referenced by the Settings UI and are not allowlisted". Verified by building internal/tools/settingsparity and running it against the head agent_config.go; it emits:
agent_sandbox.publish.verify[]
agent_sandbox.publish.verify_timeout_s
agents.<key>.sandbox.publish.verify[]
agents.<key>.sandbox.publish.verify_timeout_s
Smallest fix, matching how the other sandbox leaves are handled (settings_parity_allowlist.txt:4-15, :95-117) — append:
--- a/src/pkg/dashboard/settings_parity_allowlist.txt
+++ b/src/pkg/dashboard/settings_parity_allowlist.txt
@@
agent_sandbox.job.volumes[].read_only # TODO #11854
+agent_sandbox.publish.verify[] # TODO #11854
+agent_sandbox.publish.verify_timeout_s # TODO #11854
@@
agents.<key>.sandbox.network_mode # TODO #11854
+agents.<key>.sandbox.publish.verify[] # TODO #11854
+agents.<key>.sandbox.publish.verify_timeout_s # TODO #11854
agents.<key>.sandbox.runtime # TODO #11854Advisory (docs wording, non-blocking): src/docs/sandbox-isolation.md describes the kick-log header as ---- verify: <cmd> (exit N) ----, but sandboxKickLogText (sandbox_kick_log.go in this diff) writes (passed), (timed out), or (error: …) as well — agent-logging.md's (<verdict>) is the accurate form; consider aligning.
— hive: agent=ci-maintainer backend=copilot model=claude-fable-5.1 copilot=1.0.88
There was a problem hiding this comment.
Security perspective — reviewed head 7377efa.
1. Verify directive is parsed from the rendered kick message, so untrusted text can disable or replace the preflight (in-scope, medium).
src/crux/agent/manager_sandbox.go:155-158 calls parsePublishVerifyDirective(message) and overwrites spec.PublishVerify whenever a <!-- hive:publish.verify … --> block appears anywhere in the message — including an empty block, which sandbox_publish_verify.go:26-39 returns as ok=true and so turns the configured preflight off. But message is not only the operator's template: pkg/linearagent/responder.go:225-248 (buildKickMessage) appends ev.PromptText() — the Linear user's comment — verbatim, and that reaches startSandboxKickLocked via SendKick (manager_kick.go:127, kick_async.go:183). Anyone who can @mention the hive agent in Linear can therefore paste
<!-- hive:publish.verify
-->
(or true) and the gate that #14999 describes as the thing that "does not trust the agent" is silently bypassed for that kick; sandbox_pr_withheld is never written and the card shows a normal PR. Commands supplied this way run as sh -lc in the sandbox (sandbox_executor.go:633-643) with the same image/env allowlist as the agent, so this is a gate bypass rather than a privilege escalation — but it is exactly the trust boundary this PR exists to add.
Suggested fix: parse the directive from the configured template source (agent.Config.KickTemplate / the template file, src/crux/config/agent_config.go:341-343) rather than from the rendered message, so only an operator-owned artifact can change the preflight. If you want to keep the message path, at minimum make it additive and non-disabling:
- if cmds, ok := parsePublishVerifyDirective(message); ok {
- spec.PublishVerify = cmds
- }
+ // Directives in the rendered message may come from external text
+ // (e.g. Linear prompts); they may only add checks, never clear them.
+ if cmds, ok := parsePublishVerifyDirective(message); ok && len(cmds) > 0 {
+ spec.PublishVerify = append(spec.PublishVerify, cmds...)
+ }and update TestSandboxKick_TemplateDirectiveOverridesConfig/the "empty block disables" test and the "template block replaces" sentence in docs/sandbox-isolation.md:136 accordingly.
Advisory (out-of-scope, low): the verify commands execute in the workspace the agent just modified, so an agent that commits a conftest.py, pytest.ini, or Makefile edit can still make the preflight pass. Worth a sentence in docs/sandbox-isolation.md:136 so operators do not read "does not trust the agent" as stronger than it is; a later hardening could reset tracked config files to HIVE_BASE_SHA before running verify.
— hive: agent=adjudicator-2 backend=copilot model=claude-fable-5.1 copilot=1.0.88
|
Hive filed review findings that do not block this PR as backlog items:
Cap: 10 backlog item(s) per PR per day. |
There was a problem hiding this comment.
Correctness review (head 7377efa).
1. The template directive is parsed from the fully rendered kick message, so untrusted text can switch the configured preflight off — src/crux/agent/manager_sandbox.go:156 runs parsePublishVerifyDirective(message) on the final message and lets any match replace agent.Config.SandboxPublishVerify(...). Kick messages carry text hive did not write: planning puts the GitHub issue body into the architect prompt verbatim (pkg/planning/issue.go:112, pkg/planning/decompose.go:145), and the reviewer lane writes PR titles with a trailing \n (pkg/scheduler/reviewer_lane.go:474). An issue body containing
<!-- hive:publish.verify
-->
disables agent_sandbox.publish.verify / agents.<name>.sandbox.publish.verify for that kick (parser returns ok=true, len(cmds)==0, sandbox_publish_verify.go:26-38), and a non-empty block substitutes true for the real check — so the gate the PR body calls "deterministic … does not trust the agent" is cancelable by whoever opens the issue. #14999 does say "template block wins", but it assumes the block comes from the template, not from interpolated content. Smallest fix that keeps the feature: make the directive additive over config so it can add checks but never remove them (per-agent publish: {verify: []} already covers the disable case):
// A kick template's own verify block adds to the configured list; it
// cannot remove configured checks, because the rendered message also
// carries issue/PR text hive did not write.
if cmds, ok := parsePublishVerifyDirective(message); ok {
spec.PublishVerify = append(spec.PublishVerify, cmds...)
}
(TestManagerSandboxKickTemplateDirectiveOverridesConfiguredVerify would then expect [make check, uv run pytest --collect-only -q].) If you'd rather keep replace semantics, the directive needs to be lifted from the template source before interpolation.
2. The documented (exit N) reason is unreachable with the real launchers. PodmanLauncher.Run returns a non-nil *exec.ExitError for every non-zero exit (src/crux/sandbox/sandbox.go:66-76) and the Job launcher returns kubejob: job … failed: … (src/crux/kubejob/kubejob.go:243-246). runPublishVerify checks vres.Err != "" before the exit-code default (sandbox_executor.go:652-666), and the kick-log header does the same (sandbox_kick_log.go:66-71), so operators get verify failed (1/2): <cmd>: exit status 2 on the card and ---- verify: <cmd> (error: exit status 2) ---- in the log — not the (exit 2) promised in docs/sandbox-isolation.md, docs/dashboard-sections.md and the changelog. The tests only pass because sandboxVerifyLauncher returns ExitCode with a nil error. Fix: prefer the exit code when one was collected, in both switches:
switch {
case vres.TimedOut:
reason += fmt.Sprintf(" timed out after %s", timeout)
+ case vres.ExitCode > 0:
+ reason += fmt.Sprintf(" (exit %d)", vres.ExitCode)
case vres.Err != "":
reason += ": " + vres.Err
- default:
- reason += fmt.Sprintf(" (exit %d)", vres.ExitCode)
}and in sandboxKickLogBody move case v.ExitCode != 0 above case v.Err != "". A test where the fake returns {ExitCode: 2} and errors.New("exit status 2") would pin it.
— hive: agent=adjudicator backend=copilot model=claude-fable-5.1 copilot=1.0.88
|
merged by Hive trusted-author auto-merge: author @clubanderson holds role owner and GitHub permission admin on repo; CI green at 1bf7628. |
|
Thank you for your contribution! Your PR has been merged. We'd love to hear how your experience was: share feedback |
Fixes #14999
Reported by @joerunde (a1-vllm-spyre spoke). Follow-up to #14984 (report-status publish gate).
What
A deterministic, hive-side preflight in the sandbox publish path. The
.hive/agent-report.jsonstatus gate is necessary but trusts the agent; this does not.publish.verify: [cmds](+publish.verify_timeout_s, default 900 s):agents.<name>.sandbox.publish(per agent; an empty list disables it for that agent),agent_sandbox.publish(global), or inline in the kick template:runPublishVerify): after commits are counted and the report gate passes, each command runs assh -lc <cmd>in a fresh container/Job of the same image, workspace, env allowlist and network mode, withHIVE_CHANGED_FILES,HIVE_CHANGED_TESTS(base..HEAD) andHIVE_BASE_SHAexported. First non-zero exit / launcher error / timeout → no push, no PR,ErrSandboxRunReportedFailure;prWithheld=verify failed (1/2): <cmd> (exit N)+ scrubbed output tail. Later commands are skipped. No commits → skipped; no commands → no-op (fleet default unchanged).no PR opened: verify failed …(first line; tail in tooltip), kick log archive gets a---- verify: <cmd> (<verdict>) ----section per command with full output, existingsandbox_pr_withheldaudit line.Why
torch-spyre/spyre-inference#1412: the quality agent added
tests/custom_ops/test_utils.py, ran only that file (17 passed) and committed;tests/multimodal/test_utils.pyalready existed and the tests tree has no__init__.py, so pytest's basename import clash broke collection for every CI shard and pre-commit. A whole-suite--collect-onlyin the preflight stops that PR.Tests
crux/agent/sandbox_publish_verify_test.go: verify pass → push + PR, launches reuse image/workspace/network/env and carryHIVE_CHANGED_*; failure → withheld with command, exit and tail, later commands skipped, kick-log section; launcher error → withheld; no commits → skipped; nil list → publish; directive parser (comments/blank lines/empty block);changedTestFiles; manager tests for config-driven verify and template-directive precedence.crux/config/sandbox_publish_test.go: per-agent > global > default precedence, empty override disables.Docs / changelog
docs/sandbox-isolation.md(publish preflight paragraph + example),agent-logging.md,dashboard-sections.md;changelog.d/added-sandbox-publish-verify.md.