Skip to content

fix(tunnel): keep embedded tunnels alive through heartbeat write failures - #1411

Merged
swjnxyf merged 1 commit into
mainfrom
fix/1410-tunnel-health-contention
Nov 9, 2026
Merged

swjnxyf merged 1 commit into
mainfrom
fix/1410-tunnel-health-contention

Conversation

@swjnxyf

@swjnxyf swjnxyf commented Nov 9, 2026

Copy link
Copy Markdown
Collaborator

A periodic readiness.json write failure currently tears down the embedded OpenAI Tunnel task and leaves the profile terminally failed until the owning Server restarts. Holding a separately opened shared file lock deterministically reproduces the writer failure; the existing supervisor then maps it to tunnel_health_unavailable.

Keep heartbeat publication recoverable after launch: retain the same owned task and live readiness observations, expose tunnel_health_unavailable in runtime status, and try publication again at the normal health cadence. A successful write clears the diagnostic and publishes the latest observation. Log failure/recovery transitions only. Saved observations still expire after seven seconds, and the writer retains its existing private file, ACL, validation and nonblocking lock.

The initial status write remains a launch prerequisite. Actual task errors, startup deadlines, shutdown and durable restart-fence behavior retain their existing handling; this change does not restart a task or replay work.

Add deterministic tests for shared-reader contention, recovery with updated readiness, missing files, stale saved readiness, terminal diagnostics and the initial launch gate. Run the new tests in Windows core CI as well as the ordinary Linux suite, and document the runtime contract.

Fixes #1410.

Validation on macOS:

  • cargo test --locked -p webcodex-environment --lib tunnel::health_tests — 2 passed.
  • cargo test --locked -p webcodex --lib server_tunnels — 3 passed.
  • cargo test --locked -p webcodex --lib project_entry::openai_tunnel_service::tests — 16 passed.
  • cargo test --locked -p webcodex --lib server_shutdown — 9 passed.
  • python3 -m unittest scripts.tests.test_ci_contract_ownership scripts.tests.test_ci_path_risk — 45 passed.
  • cargo fmt --all -- --check, python3 scripts/check_markdown_links.py, and git diff --check passed.

The Rust test build emitted three warnings in unchanged code (RunnerPollRequest, wait_for_admitted_action, and ManagedWorktreeSource). The lock conflict was reproduced locally; the reporter's Windows machine and a live OpenAI connection were not exercised. Native Windows validation is delegated to the added CI step.

@swjnxyf swjnxyf left a comment

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Code review of ef2c504a6535953174b381a2514dc966ebc09f64 for #1410: No blocking issues found (LGTM).

  • The change prevents a transient readiness.json publication failure from terminating the already-owned Tunnel, while preserving the existing startup gate, shutdown behavior, and durable restart-fence handling.
  • Runtime status exposes tunnel_health_unavailable during publication failure and clears it after recovery. The persisted heartbeat continues to expire after seven seconds, so it cannot be mistaken for a fresh readiness signal.
  • The regression tests cover shared-reader lock contention, subsequent recovery, missing health files, stale observations, and initial-launch failure. Windows core CI, including the new Tunnel tests, passed, along with the relevant Linux/macOS core jobs.

Non-blocking observation: persistent ACL or I/O errors still require operator intervention to restore publication; this is consistent with the focused scope of the fix.

Review outcome: Ready from the code-review perspective. GitHub does not allow the PR author to approve their own PR, so this is submitted as a review comment rather than an APPROVE event.

swjnxyf commented Nov 9, 2026

Copy link
Copy Markdown
Collaborator Author

Thanks for the focused fix for #1410. The code review found no blocking issues, and the full CI run, including the Windows-native Tunnel heartbeat contention tests, has passed. This preserves the running Tunnel on transient health-file write failures without relaxing the initial launch gate or restart-fence protections. Merging this fix into main.

@swjnxyf
swjnxyf merged commit 6d50934 into main Nov 9, 2026
19 checks passed
@swjnxyf
swjnxyf deleted the fix/1410-tunnel-health-contention branch November 9, 2026 15:10
9holy added a commit to 9holy/webcodex that referenced this pull request Nov 9, 2026
Port the bounded heartbeat reliability fix from yyjeqhc#1411 (6d50934). Preserve initial publication, stale observations, fatal lifecycle and ownership boundaries. Add deterministic contention/recovery tests and native Windows coverage.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Embedded Tunnel goes terminal Failed (tunnel_health_unavailable) after a single readiness.json write error, no retry

2 participants