Claude eval sessions run in a container seeded from managed settings

[jbachorik]

What was wrong

Claude Code's managed settings (managed-settings.json) are the highest-precedence scope and cannot be overridden by env vars or --settings. They pin ANTHROPIC_BASE_URL at the host omlx_proxy and carry ANTHROPIC_CUSTOM_HEADERS, so drydock could not add an x-target-account: eval header to route an eval session's traffic to the eval account. The previous approach marked the session id out-of-band with the host's omlx_proxy, which injected the header per request — but it depended on omlx_proxy running, on session-id correlation, and left the header source outside drydock's control.

What happens now

An eval session runs inside a container (docker on Linux, colima on macOS) where there is no managed-settings file, so a settings file drydock seeds in CLAUDE_CONFIG_DIR is the only source of truth. The seed is copied from the host's managed settings with three edits: ANTHROPIC_BASE_URL is rewritten to host.docker.internal so the container still reaches the host's omlx_proxy, x-target-account: eval is appended to ANTHROPIC_CUSTOM_HEADERS, and apiKeyHelper is dropped (the token is resolved on the host and passed in as ANTHROPIC_API_KEY). Non-eval sessions are unchanged: they keep launching claude directly on the host.

The auth token is written to an owner-only file in the mounted config dir and the entrypoint exports it from there — it never appears in ps or the command preview the UI shows. The token is a JWT; its exp claim is decoded (no signature verification — display only) so the UI shows a countdown in the session tooltip. The token's real TTL is 6h (not the 2h CLAUDE_CODE_API_KEY_HELPER_TTL_MS, which is claude's own re-invocation cache for the helper script).

Mounts

A git worktree's .git is a file pointing back into the main repo's object store via an absolute path, so for git to work inside the container both the worktree and the main repo root are bind-mounted at their original host paths. The activity-hook script, the activity state directory, and the per-session MCP config file are mounted at their host paths too, so the existing --settings/--mcp-config flags resolve unchanged and the host-side activity watcher keeps reading the same files. When the main repo root cannot be resolved (an unusual .git layout), only the worktree is mounted and a WARNING is logged so a user whose git ops break inside eval has a breadcrumb.

Token resolution SPI

The ddtool command that resolves the token is behind an EvalTokenResolver SPI; the only implementation is DtoolEvalTokenResolver (DataDog-specific). Once drydock grows plugin support it is intended to move behind a DD plugin so the eval integration is not bound to a single credential source.

Image

The image is built on demand: ./gradlew claudeEvalImage. evalAvailable() probes both that the daemon answers and that the image is present, so the UI checkbox is honestly gated. The image's ENTRYPOINT is sh, so the generated docker run command passes the entrypoint script path directly — no extra sh, or it would be sh sh <script> and fail.

Verification

Built the image (./gradlew claudeEvalImage), generated the docker run command from ClaudeEvalContainer.wrap, and ran it under a real PTY (simulating ghostty's launch path): the entrypoint executed inside the container, the token was read from the file (not the argv), and interactive TTY detection succeeded. Unit tests cover the pure logic: JWT exp decode, worktree .git resolution, managed-settings seed edits (base URL rewrite, header append idempotency, apiKeyHelper/TTL drop), and the wrap output (no token in argv, no double sh). The pre-existing SessionForkServiceTest timeouts are unrelated to this change (reproduced on clean main).

Housekeeping

The ClaudeEvalProxy (omlx marking) is deleted; nothing references it. Run ./gradlew claudeEvalImage once on each host that wants eval mode — the image is not built by the regular build.