Embedded Tunnel goes terminal Failed (tunnel_health_unavailable) after a single readiness.json write error, no retry #1410

Summary

On v0.5.0 (commit 354541eaa729a1c4bb6f35d4add46a95db4fbca4, Windows 10 19045.6466), an embedded (server-owned) OpenAI tunnel profile can go to terminal Failed because of a single transient error while writing readiness.json. The profile never recovers on its own; the only fix is restarting the owning Server.

Observed twice in one day on the same machine, ~5h40m apart:

{"schema_version":1,"service_pid":21872,"profile_revision":1,
 "tunnel_ready":false,"local_mcp_ready":false,
 "state":"failed","diagnostic":"tunnel_health_unavailable"}

At both moments the Server process itself was healthy: /healthz = 200, the same Server PID before and after, Runner connected=true / status=online, 39 projects visible, and the local MCP endpoint answering tools/list. Only the tunnel was dead.

Root cause

src/server_tunnels.rs treats a readiness persist error as fatal:

if persist_status(&profile, &status, index).is_err() {
    break Some("tunnel_health_unavailable".into());   // task ends, no retry
}

and per docs/architecture/server-owned-tunnels.md the supervisor deliberately does not restart a failed/uncertain tunnel task:

"It does not automatically restart a failed or uncertain task."

The persist path is a blocking wait-free exclusive lock:

• crates/webcodex-environment/src/tunnel.rs write_tunnel_health_inner → fs2::FileExt::try_lock_exclusive(&file), and any error becomes SetupDiagnostic::io()
• read_tunnel_health → fs2::FileExt::try_lock_shared(&file)

So any concurrent reader in another process (Desktop status polling, webcodex environment status / doctor, tunnel-diagnose, an AV scanner scanning the file) that holds the shared lock at that instant makes the exclusive attempt fail. Since the heartbeat runs roughly every 6 s (observed observed_at_ms deltas ≈ 6 s) and the freshness window is 7 s (now - time < 7000), the file is written thousands of times per day — a collision over a multi-hour session is practically inevitable.

#1029 fixed leaked locks inside the process (merged 2026-10-11, before this build); this report is about the legitimate cross-process shared-lock contention, which still takes the tunnel down permanently.

Environment / evidence

• Install: Desktop app, persistent environment, single embedded profile, host_mode=embedded, autostart=true
• Tunnel lifecycle is server-owned, so recovery commands refuse:
  • environment start tunnel --profile <id> → tunnel_host_transfer_required
  • environment restart tunnel --profile <id> → tunnel_server_owned ("This Tunnel lifecycle belongs to the Server. Control the owning Server")
• Not an ACL / disk problem: readiness.json ACL is SYSTEM / Administrators / Administrator FullControl, owner is the signed-in user, file is a regular non-symlink file, ~56 GB free on the volume, archive attribute only
• Not a network or credential problem: after a Server restart the same profile reaches state:"running", tunnel_ready:true, local_mcp_ready:true within seconds and keeps beating; control-plane reachability (chatgpt.com:443, api.openai.com:443) was fine before and after

Suggested fix

1. Make the readiness persist resilient: retry try_lock_exclusive + write with a short bounded backoff (e.g. 3 attempts / 200 ms), and only escalate to Failed after N consecutive failures — do not tear the tunnel task down on the first error.
2. Alternatively keep the last known-good readiness instead of forcing state: failed when the heartbeat write fails; a missed beat is not evidence that the tunnel is broken (the 7 s freshness window already expires on its own).
3. Consider temp-file + atomic rename for readiness.json so readers never contend with the writer at all.

Impact

A ready tunnel silently becomes unusable (ChatGPT reports the tunnel client as absent) while every other component still reports healthy; nothing in the UI offers a retry, and the CLI refuses to operate the tunnel because it is server-owned. Users have to restart the whole Server task to recover, interrupting any in-flight session.