fix(sandbox): preserve capability labels on workload resume#993
Open
tarrencev wants to merge 1 commit into
Open
fix(sandbox): preserve capability labels on workload resume#993tarrencev wants to merge 1 commit into
tarrencev wants to merge 1 commit into
Conversation
Production symptom tonight: when a session sandbox's agent pod was deleted (janitor, node pressure, manual reap) but the Sandbox CR survived, the pod that came back was missing `centaur.ai/api-server-enabled` and `centaur.ai/observability-enabled`. The chart's `centaur-centaur-sandbox-api-server` NetworkPolicy selects on `centaur.ai/api-server-enabled=true` to allow sandbox -> api-rs:8080 egress (the model-proxy path), so the recreated sandbox lost all model access — codex looped "Reconnecting... N/5: request timed out" and Slack threads looked dead. The interior iron-proxy pod was unaffected since it's rebuilt fresh on every resume with its own labels. Root cause: `AgentSandboxBackend::resume()` (services/api-rs/crates/ centaur-sandbox-agent-k8s/src/lib.rs) is the recreation path api-rs drives for an existing sandbox id whose pod went missing (`ExistingSandboxAction:: ResumeOrReplace` in centaur-session-runtime). It only patches `spec.replicas` and the paused-at annotation; it has never had a way to reassert the two capability labels, since — per its own comment — resume only has the sandbox id, not the original `SandboxSpec`/capabilities. `resolve_iron_proxy_for_resume` already solved the identical problem for the iron-proxy's NetworkPolicy scoping by reading the durable `CENTAUR_SANDBOX_OBSERVABILITY_ENABLED` / `CENTAUR_SANDBOX_API_SERVER_ENABLED` env vars `apply_sandbox_capabilities` stamped on the agent container at create time; `resume()` never applied the same trick to the pod labels themselves. This logic is untouched from upstream (`build_agent_sandbox`, `resume()`, and the create/replace paths in centaur-session-runtime all have zero fork diff against origin/main) — this is an upstream gap that our warm-pool/model-proxy fork commits (31f77c8, ae880a9) didn't introduce or touch. Fix: derive the capability labels the same way `resolve_iron_proxy_for_ resume` derives its booleans, and include them (or `null` to clear a disabled capability, matching how `build_agent_sandbox` omits rather than falsifies the label) in the resume patch's `metadata.labels` and `spec.podTemplate.metadata.labels`, so a workload recreated via resume gets exactly the labels the create path would have applied for that session's capabilities. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
When a session sandbox's agent pod is deleted out from under a live
SandboxCR (janitor sweep, node pressure eviction, manual reap), the pod that comes back through the resume path is missing thecentaur.ai/api-server-enabledandcentaur.ai/observability-enabledcapability labels thatbuild_agent_sandboxapplies at create time.Reproduction: delete a session's sandbox pod directly while its
SandboxCR survives, soAgentSandboxBackend::resume()(ExistingSandboxAction::ResumeOrReplacein centaur-session-runtime) drives recreation. The pod comes back, but the chart'scentaur-centaur-sandbox-api-serverNetworkPolicy selects oncentaur.ai/api-server-enabled=truefor sandbox → api-rs:8080 egress (the model-proxy path). Without the label, that egress is cut and the harness inside the pod loops"Reconnecting... N/5: request timed out"— the session looks dead even though the pod is technically running. The interior iron-proxy pod is unaffected since it's rebuilt fresh on every resume with its own labels; this only affects the agent sandbox pod itself.Root cause:
resume()only ever patchedspec.replicasand cleared the paused-at annotation via a minimal JSON merge patch. It never had a path to reassert the two capability labels, because — per the existing code comment — resume only has the sandbox id, not the originalSandboxSpec/SandboxCapabilitiesthat produced them at create time. This logic is untouched from upstream;build_agent_sandbox,resume(), and the create/replace paths in centaur-session-runtime carry zero fork diff againstorigin/main, so this is a plain upstream gap, not something introduced by any fork-specific change.Fix:
resolve_iron_proxy_for_resumealready solved the identical problem for the iron-proxy's NetworkPolicy scoping, by reading back the durableCENTAUR_SANDBOX_OBSERVABILITY_ENABLED/CENTAUR_SANDBOX_API_SERVER_ENABLEDenv vars thatapply_sandbox_capabilitiesstamps onto the agent container at create time. This PR applies the same trick to the pod labels: a newsandbox_capability_labelshelper re-derives both booleans from that same durable env, andresume()'s merge patch now reasserts them on bothmetadata.labelsandspec.podTemplate.metadata.labels. A disabled capability is patched withnull(removing the key) rather than"false", matching howbuild_agent_sandboxonly ever inserts these labels and never falsifies them.Verified in production: hit this exact failure mode in a live deployment — deleting a session's sandbox pod and letting resume recreate it dropped the two labels and cut model-proxy egress. Re-checking after applying this fix confirmed the recreated pod carries the correct labels again and connectivity is restored.
Changes
centaur-sandbox-agent-k8s/src/iron_proxy.rs: exposesandbox_observability_enabled/sandbox_api_server_enabledaspub(crate)solib.rscan reuse them for the resume path.centaur-sandbox-agent-k8s/src/lib.rs:sandbox_capability_labels()— re-derives the two capability labels from the sandbox's recorded env, mirroring whatbuild_agent_sandboxwould have applied.sandbox_resume_patch()now takes the derived labels and includes them (ornullfor a disabled capability) in bothmetadata.labelsandspec.podTemplate.metadata.labels.resume()computes the labels from the fetched sandbox before building the patch.Tests
resume_reasserts_capability_labels_from_recorded_env— simulates the observed bug (pod template/metadata labels stripped while the container's capability env is untouched) and asserts the resume patch restores both labels for an enabled-capabilities sandbox.resume_patch_clears_labels_for_restricted_capabilities— asserts a sandbox built with both capabilities disabled produces a resume patch thatnulls both label keys rather than writing"false".Test plan
cargo fmt --all --checkcargo clippy -p centaur-sandbox-agent-k8s --all-targets -- -D warningscargo test -p centaur-sandbox-agent-k8s(43 passed, including the 2 new regression tests)🤖 Generated with Claude Code