Skip to content

fix(sandbox): preserve capability labels on workload resume#993

Open
tarrencev wants to merge 1 commit into
paradigmxyz:mainfrom
cartridge-gg:fix/resume-capability-labels
Open

fix(sandbox): preserve capability labels on workload resume#993
tarrencev wants to merge 1 commit into
paradigmxyz:mainfrom
cartridge-gg:fix/resume-capability-labels

Conversation

@tarrencev

Copy link
Copy Markdown
Contributor

Summary

When a session sandbox's agent pod is deleted out from under a live Sandbox CR (janitor sweep, node pressure eviction, manual reap), the pod that comes back through the resume path is missing the centaur.ai/api-server-enabled and centaur.ai/observability-enabled capability labels that build_agent_sandbox applies at create time.

Reproduction: delete a session's sandbox pod directly while its Sandbox CR survives, so AgentSandboxBackend::resume() (ExistingSandboxAction::ResumeOrReplace in centaur-session-runtime) drives recreation. The pod comes back, but the chart's centaur-centaur-sandbox-api-server NetworkPolicy selects on centaur.ai/api-server-enabled=true for sandbox → api-rs:8080 egress (the model-proxy path). Without the label, that egress is cut and the harness inside the pod loops "Reconnecting... N/5: request timed out" — the session looks dead even though the pod is technically running. The interior iron-proxy pod is unaffected since it's rebuilt fresh on every resume with its own labels; this only affects the agent sandbox pod itself.

Root cause: resume() only ever patched spec.replicas and cleared the paused-at annotation via a minimal JSON merge patch. It never had a path to reassert the two capability labels, because — per the existing code comment — resume only has the sandbox id, not the original SandboxSpec/SandboxCapabilities that produced them at create time. This logic is untouched from upstream; build_agent_sandbox, resume(), and the create/replace paths in centaur-session-runtime carry zero fork diff against origin/main, so this is a plain upstream gap, not something introduced by any fork-specific change.

Fix: resolve_iron_proxy_for_resume already solved the identical problem for the iron-proxy's NetworkPolicy scoping, by reading back the durable CENTAUR_SANDBOX_OBSERVABILITY_ENABLED / CENTAUR_SANDBOX_API_SERVER_ENABLED env vars that apply_sandbox_capabilities stamps onto the agent container at create time. This PR applies the same trick to the pod labels: a new sandbox_capability_labels helper re-derives both booleans from that same durable env, and resume()'s merge patch now reasserts them on both metadata.labels and spec.podTemplate.metadata.labels. A disabled capability is patched with null (removing the key) rather than "false", matching how build_agent_sandbox only ever inserts these labels and never falsifies them.

Verified in production: hit this exact failure mode in a live deployment — deleting a session's sandbox pod and letting resume recreate it dropped the two labels and cut model-proxy egress. Re-checking after applying this fix confirmed the recreated pod carries the correct labels again and connectivity is restored.

Changes

  • centaur-sandbox-agent-k8s/src/iron_proxy.rs: expose sandbox_observability_enabled / sandbox_api_server_enabled as pub(crate) so lib.rs can reuse them for the resume path.
  • centaur-sandbox-agent-k8s/src/lib.rs:
    • sandbox_capability_labels() — re-derives the two capability labels from the sandbox's recorded env, mirroring what build_agent_sandbox would have applied.
    • sandbox_resume_patch() now takes the derived labels and includes them (or null for a disabled capability) in both metadata.labels and spec.podTemplate.metadata.labels.
    • resume() computes the labels from the fetched sandbox before building the patch.

Tests

  • resume_reasserts_capability_labels_from_recorded_env — simulates the observed bug (pod template/metadata labels stripped while the container's capability env is untouched) and asserts the resume patch restores both labels for an enabled-capabilities sandbox.
  • resume_patch_clears_labels_for_restricted_capabilities — asserts a sandbox built with both capabilities disabled produces a resume patch that nulls both label keys rather than writing "false".

Test plan

  • cargo fmt --all --check
  • cargo clippy -p centaur-sandbox-agent-k8s --all-targets -- -D warnings
  • cargo test -p centaur-sandbox-agent-k8s (43 passed, including the 2 new regression tests)
  • Verified in a production deployment: delete-pod-and-recheck restored the capability labels and model-proxy connectivity

🤖 Generated with Claude Code

Production symptom tonight: when a session sandbox's agent pod was deleted
(janitor, node pressure, manual reap) but the Sandbox CR survived, the pod
that came back was missing `centaur.ai/api-server-enabled` and
`centaur.ai/observability-enabled`. The chart's
`centaur-centaur-sandbox-api-server` NetworkPolicy selects on
`centaur.ai/api-server-enabled=true` to allow sandbox -> api-rs:8080 egress
(the model-proxy path), so the recreated sandbox lost all model access —
codex looped "Reconnecting... N/5: request timed out" and Slack threads
looked dead. The interior iron-proxy pod was unaffected since it's rebuilt
fresh on every resume with its own labels.

Root cause: `AgentSandboxBackend::resume()` (services/api-rs/crates/
centaur-sandbox-agent-k8s/src/lib.rs) is the recreation path api-rs drives
for an existing sandbox id whose pod went missing (`ExistingSandboxAction::
ResumeOrReplace` in centaur-session-runtime). It only patches
`spec.replicas` and the paused-at annotation; it has never had a way to
reassert the two capability labels, since — per its own comment — resume
only has the sandbox id, not the original `SandboxSpec`/capabilities.
`resolve_iron_proxy_for_resume` already solved the identical problem for
the iron-proxy's NetworkPolicy scoping by reading the durable
`CENTAUR_SANDBOX_OBSERVABILITY_ENABLED` / `CENTAUR_SANDBOX_API_SERVER_ENABLED`
env vars `apply_sandbox_capabilities` stamped on the agent container at
create time; `resume()` never applied the same trick to the pod labels
themselves. This logic is untouched from upstream (`build_agent_sandbox`,
`resume()`, and the create/replace paths in centaur-session-runtime all
have zero fork diff against origin/main) — this is an upstream gap that
our warm-pool/model-proxy fork commits (31f77c8, ae880a9) didn't
introduce or touch.

Fix: derive the capability labels the same way `resolve_iron_proxy_for_
resume` derives its booleans, and include them (or `null` to clear a
disabled capability, matching how `build_agent_sandbox` omits rather than
falsifies the label) in the resume patch's `metadata.labels` and
`spec.podTemplate.metadata.labels`, so a workload recreated via resume
gets exactly the labels the create path would have applied for that
session's capabilities.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant