ST5: integration & live gate — cross-transport conformance + end-to-end leaf (Kind → OCP)

[pdettori]

ST5 — Integration & live gate: cross-transport conformance + end-to-end leaf

Part of the SandboxTransport epic (see epic #89). Both tracks. Depends on ST3 (backend) and ST4 (worker).

Spec: docs/specs/2026-07-08-sandbox-transport-grpc-design.md §11, §13.3.

Problem

KubectlTransport and GrpcRelayTransport are only safely swappable (driver #2) if they demonstrably satisfy the same SandboxTransport contract, and the remote path only counts as real once a genuine Go worker pod serves a leaf end-to-end over :443.

Goal

Prove equivalence and land the live gate (§11 testing table):

• Conformance battery — one shared spec run against both KubectlTransport and GrpcRelayTransport: round-trip every op (read/write/bash/grep), abort mid-stream, timeout, reconnect→dedup, output-cap. This is what makes the two transports safely interchangeable. (The battery is authored here and is the same suite ST2 uses to prove its refactor inert.)
• Manifest-shape test — a vitest that parses the relay + worker Deployment YAML directly (no kustomize in CI), matching the existing manifest-shape pattern.
• Live gate — a real Go worker pod dialing the in-cluster relay over :443; one leaf end-to-end on Kind, then OCP, following the existing leaf-smoke pattern.

Non-goals (this task)

• New transport behavior (owned by ST2/ST3) or worker behavior (ST4).
• The deferred items (mTLS, multi-replica HA, private mesh, extra-language workers, HTTP/1.1 traversal — §13.4).

Acceptance

• The same conformance suite is green against both transports (identical observable contract).
• Manifest-shape vitest parses relay + worker Deployment YAML and passes in CI.
• Live leaf smoke passes with the Go worker → relay :443 path on Kind, then on OCP (gated by the existing live-smoke env vars).

Touches

Shared conformance suite (k8s-sandbox tests, exercised by ST2 + ST3); Go contract tests (ST4); deploy/knative/ relay + worker manifests + smoke drivers (leaf-smoke.sh extension); CI wiring.

Next step

Green here retires the epic's core scope; deferred follow-ups (§13.4) stay tracked in the epic's Deferred section.

[pdettori]

ST5 scope — integration & live gate

ST5 is the slice that proves the remote path end-to-end with a real Go worker over the network: worker dials the relay, the harness leases the mirrored grpc record and runs a leaf through GrpcRelayTransport, first on Kind then on OCP. It depends on #87 (ST4, the Go worker) and on #86 (ST3, backend — merged).

Already landed in ST3 (#86 / PR #93) — do NOT redo

• Unit-level cross-transport conformance is done. The shared runConformance battery already runs against both transports — runConformance("KubectlTransport", …) and runConformance("GrpcRelayTransport", …) — proving they satisfy the identical SandboxTransport contract. ST5's job is the live analog, not re-proving this at the unit level.
• Relay Deployment + its manifest-shape vitest exist (deploy/knative/relay-deployment.yaml, single replica, Service on :8443; packages/knative-server/test/relay-deployment.test.ts), wired into the base + OCP kustomizations (OCP overlay already rewrites the relay image to GHCR).
• Harness dispatch + flag gating (SH_REMOTE_SANDBOX, select-sandbox → GrpcRelayTransport, transport-blind run-leaf) is in and inert when off.

ST5 deliverables (the live gate)

1. Worker deploy surface — a worker Deployment manifest for the in-cluster smoke (there is none yet) + a manifest-shape vitest mirroring relay-deployment.test.ts; wire it into the base + OCP kustomizations. (The worker binary/image is ST4: worker — Go reference worker (dial Attach, run/stream/abort, dedup, dual timeout) #87's; ST5 owns its deployment for the gate.)
2. Reachability + auth wiring — for the Kind/OCP smoke an in-cluster worker can dial the relay ClusterIP sandbox-relay.default.svc:8443 directly (plaintext) — start there, it's the simplest path. The relay needs SH_RELAY_TOKEN/SH_RELAY_TOKEN_<id> + REDIS_URL; the worker Deployment carries the matching Bearer token + its sandbox_id; the harness needs SH_REMOTE_SANDBOX=1 + SH_RELAY_ADDR. The production external-worker path (relay Attach on :443 behind an ingress/Route with TLS, and SandboxExec locked to in-cluster via a NetworkPolicy per spec §9) is a follow-on — decide whether ST5 includes the NetworkPolicy/Route or defers them to a hardening slice.
3. leaf-smoke.sh relay path — deploy/knative/leaf-smoke.sh has no relay/SH_REMOTE_SANDBOX hooks yet. Extend it (or add leaf-relay-smoke.sh) to: deploy relay + worker, wait for the presence record to appear in the Redis pool (redis-cli HGETALL sh:sandbox:records), run one leaf with SH_REMOTE_SANDBOX=1 so it leases the grpc record, assert the verdict/output, and assert the record is removed when the worker stops.
4. Live cross-transport gate — run the same leaf workload through the pod path (KubectlTransport, flag off) and the worker path (GrpcRelayTransport, flag on) and assert identical verdicts — the live equivalent of the unit conformance battery.
5. Kind → OCP progression — green on Kind first, then OCP 4.20 (the repo's live-gate pattern).

Acceptance gates (spec §10 / §13.2)

• Flag off ⇒ existing e2e green (path fully inert).
• Worker connect ⇒ presence record appears in the pool; removed on stream close.
• A leaf completes end-to-end via the relay (streamed stdout, correct exit code).
• Relay restart (or worker disconnect) ⇒ in-flight exec fails ⇒ leaf retry re-leases a healthy sandbox (no mid-exec durability).
• Output-cap truncation and dual-ended timeout exercised against the live worker.

Starting point

The Level-2 (full-path) recipe in #87 is the entry point — SH_REMOTE_SANDBOX=1 + SH_RELAY_ADDR + a leased grpc record. Reference material: proto/sandbox/v1/sandbox.proto (§8 semantics), packages/sandbox-relay/src/relay.ts (park/presence/routing), packages/k8s-sandbox/src/grpc-relay-transport.ts (harness-side reqId correlation / Chunk.stream split / dedup / deadline / output cap), and the existing deploy/knative/leaf-smoke.sh as the template to extend.

Assisted-By: Claude Code