ST4: worker — Go reference worker (dial Attach, run/stream/abort, dedup, dual timeout)

[pdettori]

ST4 — Worker: Go reference worker

Part of the SandboxTransport epic (see epic #89). Worker track (Go / sandbox side) — a separate contributor from the backend. Depends only on ST1 (the published sandbox/v1 proto); builds in parallel with ST2/ST3.

Spec: docs/specs/2026-07-08-sandbox-transport-grpc-design.md §7, §8.

Problem

The contract is language-neutral (ST1), but nothing yet proves it: we need one worker that a) implements the SandboxWorker.Attach client, and b) is deliberately not TypeScript — the honest proof the seam isn't secretly TS-shaped (driver #2, §2). A Go static binary drops into any sandbox image with no runtime dependency.

Goal

Ship a thin, single-purpose Go static binary that holds no LLM key and no orchestration — it only executes commands and returns bytes (what makes "central brain" trust-correct, §7). Behavior per §7:

1. On start: read SANDBOX_ID, RELAY_ADDR, SANDBOX_TOKEN from env; dial SandboxWorker.Attach over TLS :443 (HTTP/2) with Authorization: Bearer <token>.
2. Send Hello (id, labels, capabilities, capacity_max, trust).
3. Loop on ServerFrame:
   • Exec → run bash -c <command> locally, feeding stdin. If streaming: emit Chunk* then End{exit_code}. Else: a single End carrying full stdout. Enforce timeout_s locally (SIGKILL on expiry).
   • Abort → SIGKILL the in-flight child for that req_id.
4. Send Heartbeat periodically (liveness + NAT/proxy keepalive).
5. Maintain a bounded req_id → End dedup cache; on a redelivered req_id, re-emit the cached result instead of re-running (at-least-once + dedup, §8).

Persistent-shell semantics (one long-lived bash preserving cwd/env across execs) are an internal choice; the wire only requires ordered, one-in-flight execs (§7).

Non-goals (this task)

• The relay and GrpcRelayTransport (ST3 — backend).
• mTLS / SPIFFE (bearer token only day-one, §9); additional-language workers (§13.4).
• Any orchestration or credential handling in the worker.

Acceptance

• Go contract test (Go test, real Attach against a relay, worker pointed at local bash) round-trips read/write/bash/grep, plus abort mid-stream, timeout (worker SIGKILLs child), and reconnect → dedup (redelivered req_id re-emits cached End, no re-run).
• Static binary, no runtime deps; drops into a sandbox image.
• Chunk size capped so a single frame stays small (backpressure, §8).

Touches

New Go module for the worker (e.g. worker/ or a dedicated repo dir), consuming ST1's generated Go stubs; a minimal container build for the static binary; Go unit + contract tests.

Next step

Join ST3 for the ST5 live gate (real worker pod → in-cluster relay on :443). Worker plan lives locally in docs/plans/, authored by the worker contributor.

[pdettori]

Testing the Go worker against the ST3 backend

ST3 (#86, PR #93) is merged, so the relay + GrpcRelayTransport are available to test against now — you don't need to wait for ST5. There are two levels; Level 1 is the fast inner loop and exercises everything the worker must get right.

Level 1 — worker ↔ relay contract loop (no harness needed)

1. Redis + relay:

docker run --rm -p 6379:6379 redis:7            # or reuse an existing one
# from repo root:
SH_RELAY_TOKEN=dev-token SH_RELAY_PORT=8443 REDIS_URL=redis://127.0.0.1:6379 \
  pnpm --filter @sh/sandbox-relay start          # binds 0.0.0.0:8443, plaintext (TLS terminates at ingress in prod)

2. Worker connects: dial localhost:8443 (h2c / insecure), open SandboxWorker.Attach, attach metadata authorization: Bearer dev-token, and send WorkerFrame{hello:{sandbox_id:"sbx-dev-1", ...}} as the first frame. The token must match SH_RELAY_TOKEN (or set SH_RELAY_TOKEN_sbx-dev-1 for a per-sandbox token) — auth is fail-closed: if no token is configured for the sandbox, or it doesn't match, the Attach is rejected before the stream is parked.

3. Verify presence — the live Attach stream is the registration:

redis-cli HGETALL sh:sandbox:records     # → field "sbx-dev-1" = {"transport":"grpc",...}
# close the worker's Attach stream → the field disappears (removed on stream close/error)

4. Drive an exec from the harness-facing side — two options:

(a) The real harness client (recommended — it's the exact code the harness runs):

import { SandboxExecClient, GrpcRelayTransport } from "@sh/k8s-sandbox";
import { credentials } from "@grpc/grpc-js";
const client = new SandboxExecClient("localhost:8443", credentials.createInsecure());
const t = GrpcRelayTransport("sbx-dev-1", client as any);
const r = await t.exec("echo hello; echo oops >&2; exit 7", { onData: c => process.stderr.write(c) });
console.log("stdout:", r.stdout.toString(), "exit:", r.exitCode);  // "hello\n", 7
await t.close();

Run it with tsx. This proves the whole relay→worker→relay round-trip and the stdout/stderr split in one shot.

(b) grpcurl (the relay does not register gRPC reflection, so pass the proto):

grpcurl -plaintext -proto proto/sandbox/v1/sandbox.proto \
  -d '{"sandbox_id":"sbx-dev-1","exec":{"req_id":1,"command":"echo hi","timeout_s":10,"streaming":true}}' \
  localhost:8443 sandbox.v1.SandboxExec/Exec

The wire contract the worker must satisfy

From proto/sandbox/v1/sandbox.proto (§8) and what GrpcRelayTransport expects:

• stdout/stderr split: stdout → Chunk{stream: STREAM_STDOUT} (collected into the returned stdout); stderr → Chunk{stream: STREAM_STDERR} (streamed to onData, excluded from the returned stdout). This is the reason ST3 added Chunk.stream.
• terminate each exec with End{req_id, exit_code}; exit_code < 0 means signal/none (the harness maps it to null). Failures → ExecError{req_id, message}.
• abort: on ServerFrame{abort}, SIGKILL the child and emit a terminal frame for that req_id.
• worker-side timeout: kill the child at exec.timeout_s (the harness enforces its own deadline too — dual-ended).
• dedup / at-least-once: cache req_id → End; on a redelivered req_id after a reconnect, re-emit the cached terminal result rather than re-running.
• stdin: feed exec.stdin bytes to the child's stdin. Heartbeat frames are liveness-only (the harness owns lease counts).

Level 2 — full path (worker ↔ relay ↔ harness)

Run the harness with SH_REMOTE_SANDBOX=1 and SH_RELAY_ADDR pointed at the relay; with a grpc pool record present, select-sandbox leases it and run-leaf drives a real leaf through GrpcRelayTransport. The end-to-end leaf-smoke for this (extending deploy/knative/leaf-smoke.sh with the relay path) is ST5 (#88)'s deliverable — so Level 1 is what unblocks worker development now; Level 2 lands with ST5.

Reference pointers

• Source of truth: proto/sandbox/v1/sandbox.proto (§4 messages/services, §8 wire semantics).
• Go stubs: gen/go/sandbox/v1/ (module github.com/kagenti/serverless-harness/gen/go) — there's already a contract_test.go there.
• Relay behavior your worker must interoperate with: packages/sandbox-relay/src/relay.ts (park / presence / routing).
• Harness-side expectations: packages/k8s-sandbox/src/grpc-relay-transport.ts (reqId correlation, Chunk.stream split, dedup, deadline, per-exec output cap).

Assisted-By: Claude Code