Skip to content

✨ feat(sandbox): deterministic publish.verify preflight before push/PR - #15002

Merged
hivecommons-hive[bot] merged 2 commits into
v5from
feat/sandbox-publish-verify
Oct 28, 2026
Merged

hivecommons-hive[bot] merged 2 commits into
v5from
feat/sandbox-publish-verify

Conversation

@clubanderson

Copy link
Copy Markdown
Member

Fixes #14999

Reported by @joerunde (a1-vllm-spyre spoke). Follow-up to #14984 (report-status publish gate).

What

A deterministic, hive-side preflight in the sandbox publish path. The .hive/agent-report.json status gate is necessary but trusts the agent; this does not.

  • Knob publish.verify: [cmds] (+ publish.verify_timeout_s, default 900 s): agents.<name>.sandbox.publish (per agent; an empty list disables it for that agent), agent_sandbox.publish (global), or inline in the kick template:
    <!-- hive:publish.verify
    uv run --no-sync pytest --collect-only -q
    sh -c '[ -z "$HIVE_CHANGED_TESTS" ] || uv run --no-sync pytest $HIVE_CHANGED_TESTS -q'
    -->
    
    A template block replaces the configured list for that kick, so the policy that tells the agent "run on card first" is the same artifact that tells hive how to check it did.
  • Executor (runPublishVerify): after commits are counted and the report gate passes, each command runs as sh -lc <cmd> in a fresh container/Job of the same image, workspace, env allowlist and network mode, with HIVE_CHANGED_FILES, HIVE_CHANGED_TESTS (base..HEAD) and HIVE_BASE_SHA exported. First non-zero exit / launcher error / timeout → no push, no PR, ErrSandboxRunReportedFailure; prWithheld = verify failed (1/2): <cmd> (exit N) + scrubbed output tail. Later commands are skipped. No commits → skipped; no commands → no-op (fleet default unchanged).
  • Surfacing: card line no PR opened: verify failed … (first line; tail in tooltip), kick log archive gets a ---- verify: <cmd> (<verdict>) ---- section per command with full output, existing sandbox_pr_withheld audit line.

Why

torch-spyre/spyre-inference#1412: the quality agent added tests/custom_ops/test_utils.py, ran only that file (17 passed) and committed; tests/multimodal/test_utils.py already existed and the tests tree has no __init__.py, so pytest's basename import clash broke collection for every CI shard and pre-commit. A whole-suite --collect-only in the preflight stops that PR.

Tests

  • crux/agent/sandbox_publish_verify_test.go: verify pass → push + PR, launches reuse image/workspace/network/env and carry HIVE_CHANGED_*; failure → withheld with command, exit and tail, later commands skipped, kick-log section; launcher error → withheld; no commits → skipped; nil list → publish; directive parser (comments/blank lines/empty block); changedTestFiles; manager tests for config-driven verify and template-directive precedence.
  • crux/config/sandbox_publish_test.go: per-agent > global > default precedence, empty override disables.

Docs / changelog

docs/sandbox-isolation.md (publish preflight paragraph + example), agent-logging.md, dashboard-sections.md; changelog.d/added-sandbox-publish-verify.md.

The report-status gate (#14984) trusts the agent. Add a hive-side
preflight: sandbox.publish.verify (per agent), agent_sandbox.publish.verify
(global) or an inline <!-- hive:publish.verify --> block in the kick
template lists shell commands hive runs in the same sandbox image and
workspace after the run, with HIVE_CHANGED_FILES / HIVE_CHANGED_TESTS /
HIVE_BASE_SHA exported. The first non-zero exit, launcher error or timeout
withholds the push and the PR; the reason and output tail land in
prWithheld (card: 'no PR opened: verify failed …'), the kick log archive
gets a '---- verify: <cmd> ----' section per command, and the existing
sandbox_pr_withheld audit line fires.

Motivated by torch-spyre/spyre-inference#1412, where a new
tests/custom_ops/test_utils.py clashed with tests/multimodal/test_utils.py
and broke pytest collection for every CI shard; a whole-suite
--collect-only in the preflight would have stopped the PR.

Fixes #14999
Reported-by: @joerunde

Signed-off-by: Andrew Anderson <andy@clubanderson.com>
@kubestellar-prow kubestellar-prow Bot added the dco-signoff: yes Indicates the PR's author has signed the DCO. label Oct 28, 2026
@kubestellar-prow

Copy link
Copy Markdown
Contributor

[APPROVALNOTIFIER] This PR is NOT APPROVED

This pull-request has been approved by:
Once this PR has been reviewed and has the lgtm label, please assign hanthor for approval. For more information see the Code Review Process.

The full list of commands accepted by this bot can be found here.

Details Needs approval from an approver in each of these files:

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

@kubestellar-prow kubestellar-prow Bot added the size/XL Denotes a PR that changes 500-999 lines, ignoring generated files. label Oct 28, 2026
@github-actions

github-actions Bot commented Oct 28, 2026 •

Copy link
Copy Markdown
Contributor

Open PR overlap check

Base branch: v5

Conflicts are checked with every PR merged onto the current tip of v5, so a PR that only needs a rebase is not reported as conflicting with its siblings.

PR Title Relation Files
#14995 ✨ Activity Monitor: needs-human popover says what to decide and offers the actions same files src/docs/dashboard-sections.md
src/pkg/dashboard/static/index.html
#15004 ✨ feature: persist Activity Monitor operator actions same files src/pkg/dashboard/static/index.html
#15003 ✨ feature: show Activity Monitor operator decision trails same files src/pkg/dashboard/static/index.html
#14992 ✨ Activity monitor: decision badge shows what you chose and what hive did next same files src/docs/dashboard-sections.md
src/pkg/dashboard/static/index.html
#14989 ✨ feat: let a sandboxed reviewer post reviews through a workspace spool CONFLICT src/crux/agent/manager_sandbox.go
src/crux/agent/sandbox_executor.go
#14987 ✨ review: gate PR reviews on an approval label (review.require_labels) same files src/docs/dashboard-sections.md
src/pkg/dashboard/static/index.html
#14807 ✨ dashboard: stream Job pod logs in the terminal pane for sandbox.runtime: job agents same files src/crux/agent/manager_sandbox.go
src/crux/agent/sandbox_executor.go
src/docs/sandbox-isolation.md
src/pkg/dashboard/static/index.html

@github-actions github-actions Bot added the conflicts-with-open-pr Open PR has a merge conflict with another open PR label Oct 28, 2026
@hivecommons-hive

Copy link
Copy Markdown
Contributor

@clubanderson is auto-promoted for contributor-lane merging on this PR: the last 5 merged PRs by this author in this hive's repositories have no revert or sentinel-alert evidence. CI, reviewer verdict, diff fences, soak, holds and branch protection still apply.

@hivecommons-hive hivecommons-hive Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Style review (read at head 7377efa).

1. sandboxPublishWithheld's doc comment is now attached to the new const block — src/crux/agent/sandbox_executor.go:596-607. The diff inserted const ( sandboxVerifyTailMax … ) directly under the existing // sandboxPublishWithheld returns why the run's commits must not be published… comment, so godoc now documents the constants with the gate function's text, and sandboxPublishWithheld itself (sandbox_executor.go:719) has no doc comment. Move the block above the comment:

 var ErrSandboxRunReportedFailure = errors.New("sandbox run reported failure")
 
-// sandboxPublishWithheld returns why the run's commits must not be published,
-// or "" when the gate is open. The order matters for the message an operator
-// reads: the agent's own words first, the exit code second.
 const (
 	// sandboxVerifyTailMax bounds the verify output quoted on the card and in
 	// the audit record; the full output lands in the kick log archive.
 	sandboxVerifyTailMax = 600
 	// sandboxVerifyOutputMax bounds what one verify command's combined output
 	// keeps in memory and in the kick log.
 	sandboxVerifyOutputMax      = 64 << 10
 	defaultSandboxVerifyTimeout = 15 * time.Minute
 )
@@
+// sandboxPublishWithheld returns why the run's commits must not be published,
+// or "" when the gate is open. The order matters for the message an operator
+// reads: the agent's own words first, the exit code second.
 func sandboxPublishWithheld(res SandboxKickResult) string {

2. Two independent copies of the verify-timeout default (advisory). src/crux/config/agent_config.go:519-522 defines DefaultSandboxVerifyTimeoutS = 900 and its field comment says non-positive "uses the executor default (DefaultSandboxVerifyTimeoutS)", but the executor's fallback at src/crux/agent/sandbox_executor.go:606,616-618 is a separate literal defaultSandboxVerifyTimeout = 15 * time.Minute. They agree today only by coincidence; agent already imports crux/config (manager_sandbox.go:11), so derive one from the other:

	defaultSandboxVerifyTimeout = time.Duration(config.DefaultSandboxVerifyTimeoutS) * time.Second

— hive: agent=adjudicator-4 backend=copilot model=claude-fable-5.1 copilot=1.0.88

@hivecommons-hive hivecommons-hive Bot added stage/071-needs-author Human-authored PR: CI red, changes requested or conflicting; diagnosis posted, the author must act and removed stage/050-review-pending Review swarm requested; awaiting a verdict for the current head labels Oct 28, 2026
@hivecommons-hive

Copy link
Copy Markdown
Contributor

@clubanderson this PR is held at stage/071-needs-author and the hive will not merge it yet.

Failing checks:

  • test (dashboard 3/4)
  • test (dashboard shuffle 2/3)

Next: the hive does not push to your branch. Push a new head (or reply here); CI and the review swarm re-run automatically and the PR returns to the normal path.

@hivecommons-hive hivecommons-hive Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Reviewed — no findings from this perspective.

— hive: agent=adjudicator-3 backend=copilot model=claude-fable-5.1 copilot=1.0.88

The knob is a policy-file setting (template directive or hive.yaml) with no
Settings control yet; track the control under #14999.

Signed-off-by: Andrew Anderson <andy@clubanderson.com>

@hivecommons-hive hivecommons-hive Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Settings-parity gate will fail on the four new publish.* config leaves (in-scope, agent-fixable)

src/crux/config/agent_config.go adds SandboxPublishConfig (verify, verify_timeout_s) under both AgentSandboxConfig.Publish and AgentSandboxOverride.Publish. internal/settingsparity.ConfigLeaves (src/internal/settingsparity/settingsparity.go:25-28) walks src/crux/config too, so pkg/dashboard.TestSettingsDialogReferencesEveryHiveYAMLLeaf (src/pkg/dashboard/settings_parity_test.go:12-33) now sees four leaves that neither static/index.html (0 occurrences of publish.verify at head 7377efa) nor settings_parity_allowlist.txt reference, and fails with "hive.yaml settings are not referenced by the Settings UI and are not allowlisted". Verified by building internal/tools/settingsparity and running it against the head agent_config.go; it emits:

agent_sandbox.publish.verify[]
agent_sandbox.publish.verify_timeout_s
agents.<key>.sandbox.publish.verify[]
agents.<key>.sandbox.publish.verify_timeout_s

Smallest fix, matching how the other sandbox leaves are handled (settings_parity_allowlist.txt:4-15, :95-117) — append:

--- a/src/pkg/dashboard/settings_parity_allowlist.txt
+++ b/src/pkg/dashboard/settings_parity_allowlist.txt
@@
 agent_sandbox.job.volumes[].read_only # TODO #11854
+agent_sandbox.publish.verify[] # TODO #11854
+agent_sandbox.publish.verify_timeout_s # TODO #11854
@@
 agents.<key>.sandbox.network_mode # TODO #11854
+agents.<key>.sandbox.publish.verify[] # TODO #11854
+agents.<key>.sandbox.publish.verify_timeout_s # TODO #11854
 agents.<key>.sandbox.runtime # TODO #11854

Advisory (docs wording, non-blocking): src/docs/sandbox-isolation.md describes the kick-log header as ---- verify: <cmd> (exit N) ----, but sandboxKickLogText (sandbox_kick_log.go in this diff) writes (passed), (timed out), or (error: …) as well — agent-logging.md's (<verdict>) is the accurate form; consider aligning.

— hive: agent=ci-maintainer backend=copilot model=claude-fable-5.1 copilot=1.0.88

@hivecommons-hive hivecommons-hive Bot added stage/050-review-pending Review swarm requested; awaiting a verdict for the current head and removed stage/071-needs-author Human-authored PR: CI red, changes requested or conflicting; diagnosis posted, the author must act labels Oct 28, 2026

@hivecommons-hive hivecommons-hive Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Security perspective — reviewed head 7377efa.

1. Verify directive is parsed from the rendered kick message, so untrusted text can disable or replace the preflight (in-scope, medium).
src/crux/agent/manager_sandbox.go:155-158 calls parsePublishVerifyDirective(message) and overwrites spec.PublishVerify whenever a <!-- hive:publish.verify … --> block appears anywhere in the message — including an empty block, which sandbox_publish_verify.go:26-39 returns as ok=true and so turns the configured preflight off. But message is not only the operator's template: pkg/linearagent/responder.go:225-248 (buildKickMessage) appends ev.PromptText() — the Linear user's comment — verbatim, and that reaches startSandboxKickLocked via SendKick (manager_kick.go:127, kick_async.go:183). Anyone who can @mention the hive agent in Linear can therefore paste

<!-- hive:publish.verify
-->

(or true) and the gate that #14999 describes as the thing that "does not trust the agent" is silently bypassed for that kick; sandbox_pr_withheld is never written and the card shows a normal PR. Commands supplied this way run as sh -lc in the sandbox (sandbox_executor.go:633-643) with the same image/env allowlist as the agent, so this is a gate bypass rather than a privilege escalation — but it is exactly the trust boundary this PR exists to add.

Suggested fix: parse the directive from the configured template source (agent.Config.KickTemplate / the template file, src/crux/config/agent_config.go:341-343) rather than from the rendered message, so only an operator-owned artifact can change the preflight. If you want to keep the message path, at minimum make it additive and non-disabling:

-	if cmds, ok := parsePublishVerifyDirective(message); ok {
-		spec.PublishVerify = cmds
-	}
+	// Directives in the rendered message may come from external text
+	// (e.g. Linear prompts); they may only add checks, never clear them.
+	if cmds, ok := parsePublishVerifyDirective(message); ok && len(cmds) > 0 {
+		spec.PublishVerify = append(spec.PublishVerify, cmds...)
+	}

and update TestSandboxKick_TemplateDirectiveOverridesConfig/the "empty block disables" test and the "template block replaces" sentence in docs/sandbox-isolation.md:136 accordingly.

Advisory (out-of-scope, low): the verify commands execute in the workspace the agent just modified, so an agent that commits a conftest.py, pytest.ini, or Makefile edit can still make the preflight pass. Worth a sentence in docs/sandbox-isolation.md:136 so operators do not read "does not trust the agent" as stronger than it is; a later hardening could reset tracked config files to HIVE_BASE_SHA before running verify.

— hive: agent=adjudicator-2 backend=copilot model=claude-fable-5.1 copilot=1.0.88

@hivecommons-hive

Copy link
Copy Markdown
Contributor

Hive filed review findings that do not block this PR as backlog items:

Cap: 10 backlog item(s) per PR per day.

@hivecommons-hive hivecommons-hive Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Correctness review (head 7377efa).

1. The template directive is parsed from the fully rendered kick message, so untrusted text can switch the configured preflight off — src/crux/agent/manager_sandbox.go:156 runs parsePublishVerifyDirective(message) on the final message and lets any match replace agent.Config.SandboxPublishVerify(...). Kick messages carry text hive did not write: planning puts the GitHub issue body into the architect prompt verbatim (pkg/planning/issue.go:112, pkg/planning/decompose.go:145), and the reviewer lane writes PR titles with a trailing \n (pkg/scheduler/reviewer_lane.go:474). An issue body containing

<!-- hive:publish.verify
-->

disables agent_sandbox.publish.verify / agents.<name>.sandbox.publish.verify for that kick (parser returns ok=true, len(cmds)==0, sandbox_publish_verify.go:26-38), and a non-empty block substitutes true for the real check — so the gate the PR body calls "deterministic … does not trust the agent" is cancelable by whoever opens the issue. #14999 does say "template block wins", but it assumes the block comes from the template, not from interpolated content. Smallest fix that keeps the feature: make the directive additive over config so it can add checks but never remove them (per-agent publish: {verify: []} already covers the disable case):

	// A kick template's own verify block adds to the configured list; it
	// cannot remove configured checks, because the rendered message also
	// carries issue/PR text hive did not write.
	if cmds, ok := parsePublishVerifyDirective(message); ok {
		spec.PublishVerify = append(spec.PublishVerify, cmds...)
	}

(TestManagerSandboxKickTemplateDirectiveOverridesConfiguredVerify would then expect [make check, uv run pytest --collect-only -q].) If you'd rather keep replace semantics, the directive needs to be lifted from the template source before interpolation.

2. The documented (exit N) reason is unreachable with the real launchers. PodmanLauncher.Run returns a non-nil *exec.ExitError for every non-zero exit (src/crux/sandbox/sandbox.go:66-76) and the Job launcher returns kubejob: job … failed: … (src/crux/kubejob/kubejob.go:243-246). runPublishVerify checks vres.Err != "" before the exit-code default (sandbox_executor.go:652-666), and the kick-log header does the same (sandbox_kick_log.go:66-71), so operators get verify failed (1/2): <cmd>: exit status 2 on the card and ---- verify: <cmd> (error: exit status 2) ---- in the log — not the (exit 2) promised in docs/sandbox-isolation.md, docs/dashboard-sections.md and the changelog. The tests only pass because sandboxVerifyLauncher returns ExitCode with a nil error. Fix: prefer the exit code when one was collected, in both switches:

 		switch {
 		case vres.TimedOut:
 			reason += fmt.Sprintf(" timed out after %s", timeout)
+		case vres.ExitCode > 0:
+			reason += fmt.Sprintf(" (exit %d)", vres.ExitCode)
 		case vres.Err != "":
 			reason += ": " + vres.Err
-		default:
-			reason += fmt.Sprintf(" (exit %d)", vres.ExitCode)
 		}

and in sandboxKickLogBody move case v.ExitCode != 0 above case v.Err != "". A test where the fake returns {ExitCode: 2} and errors.New("exit status 2") would pin it.

— hive: agent=adjudicator backend=copilot model=claude-fable-5.1 copilot=1.0.88

@hivecommons-hive
hivecommons-hive Bot merged commit 25f704a into v5 Oct 28, 2026
77 of 79 checks passed
@kubestellar-prow
kubestellar-prow Bot deleted the feat/sandbox-publish-verify branch October 28, 2026 16:53
@hivecommons-hive

Copy link
Copy Markdown
Contributor
Evidence bundle `#150021bf7628` sha256:fc0c477bb226bfad61c12bf32335712c1f1afa19a9e0cf2291fb4aabe868c835 — download from the dashboard or `hivectl review evidence #15002`.

@hivecommons-hive

Copy link
Copy Markdown
Contributor

merged by Hive trusted-author auto-merge: author @clubanderson holds role owner and GitHub permission admin on repo; CI green at 1bf7628.

@github-actions

Copy link
Copy Markdown
Contributor

Thank you for your contribution! Your PR has been merged.

We'd love to hear how your experience was: share feedback

@hivecommons-hive hivecommons-hive Bot removed role/trusted Author is trusted for unattended merge (trusted-author role or registry/claim tier in require_tiers) stage/050-review-pending Review swarm requested; awaiting a verdict for the current head labels Oct 28, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

author/human A person pushed this PR themselves conflicts-with-open-pr Open PR has a merge conflict with another open PR dco All commits are DCO signed-off dco-signoff: yes Indicates the PR's author has signed the DCO. size/XL Denotes a PR that changes 500-999 lines, ignoring generated files.

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Sandbox publish path needs a deterministic hive-side verify preflight (PR opened though whole-suite pytest collection was broken)

1 participant