fix(sandbox): support Clearwing workspace transfer on Kubernetes

[jorge-garcia-le]

Why this is needed

Clearwing’s workspace setup was built around Docker sandbox behavior. With Docker, a read-only source workspace can be exposed through a host-path bind mount, and the client can signal EOF after streaming an archive. Neither assumption holds for the Kubernetes backend.

A Kubernetes pod cannot mount an arbitrary path from the process that launched it. The Dagster worker’s workspace is inside a different pod, so it must be copied into the sandbox pod. The Kubernetes Python WSClient also supports writing stdin but has no half-close operation, so archive transfer must terminate by consuming a known byte count.

The Kubernetes backend additionally imposed activeDeadlineSeconds = max(900, command_timeout * 3). With the default command timeout, Kubernetes killed the entire sandbox after 15 minutes even when the enclosing SourceHunt had a multi-hour timeout. Error paths during workspace transfer could also leave pods behind, and a hard worker crash could bypass Python cleanup completely.

Summary

• force writable workspaces and disable host scratch mounts for Kubernetes sandboxes
• stream workspace archives without relying on unsupported websocket stdin half-close
• remove the pod-wide activeDeadlineSeconds; retain per-command and outer SourceHunt timeouts
• make sandbox startup transactional so transfer or initialization failures stop the pod
• register Kubernetes sandboxes for process-exit cleanup and retain failed deletions for retry
• attach sandbox pods to their caller pod using an owner reference, allowing Kubernetes garbage collection after hard crashes
• document the Kubernetes lifecycle and cleanup contract
• add regression coverage for workspace transfer, deadlines, owner references, failed deletion retries, and partial-startup cleanup

Verification

• .venv/bin/pytest -q tests/test_kubernetes_sandbox.py tests/test_hunter_sandbox.py — 34 passed
• .venv/bin/ruff check clearwing/sandbox/kubernetes.py clearwing/sandbox/hunter_sandbox.py tests/test_kubernetes_sandbox.py — passed
• git diff --check — passed
• validated through a local k3s Dagster cvehunt_experiment run

Latest lifecycle hardening commit: afd79ea.