Repository navigation
Description
Bug: v0.4.6 detached supervisor has terminal receipt, but Runner/Server reports the same Job as running indefinitely
Environment / observed symptom
Windows 11, WebCodex Desktop/Server/Runner v0.4.6; one healthy local Runner reporting 37/37 Projects online and two run_detached_process Jobs as running. Both remained in the live list_jobs / observe_jobs inventory for ~8 days with terminal=false, absent Exit Code and no new stdout/stderr, though their original OS supervisors and children are no longer present.
This is an observed live inventory vs exact durable Supervisor receipt mismatch, not merely a quiet process.
Read-only native evidence (no secrets, paths to user data, or original command bodies disclosed)
- Identified original Runner-owned per-Job
runner-detached-jobs-v1/<namespace>/<job-hash>/state.json, not Server's ordinary SQLite execution table. - Exact Job identity from the two state files matches the still-reported opaque runtime Job IDs. Each includes independently recorded Supervisor PID, process creation fence, tree-leader PID/creation fence, timestamps, and a terminal object.
- Job A:
phase=terminal,terminal.status=stopped,terminal.exit_code=-1,stop_requested=true,terminal.completed_at_unix_ms=1793601755182(2026-11-02T06:42:35.182Z). State SHA256:9d70ef2aca6f92832084a8afb47f3b7fe4976e82fb459b922e0d0d436015821a. - Job B:
phase=terminal,terminal.status=completed,terminal.exit_code=0,stop_requested=false,terminal.completed_at_unix_ms=1793596272098(2026-11-02T05:11:12.098Z). State SHA256:a43e85cfd58d912c3a9282ec8862c9449e5509e8cb71f99fe573c5f4eca8c5b7. - A contemporaneous read-only
Win32_Processlookup confirms all four original Supervisor/tree-leader PID numbers are absent; PID reuse is not assumed based on number alone, and the durable state also carries native process creation fences. - Native
observe_jobsstill says bothrunning / terminal=false.list_jobsshows each withended_at=nullandexit_code=null. - The two state file last-write UTC times match their
terminal.completed_at_unix_ms; they are not recent manually rewritten JSON. read_tool_traceaudit confirms two originalrun_detached_processadmission calls, but detailed tool payloads have expired (trace_not_retained). Ordinary Server SQLitewc_job_receipts,wc_executions,wc_job_terminal_waitsall have zero rows; this does not negate Runner-owned state.- Existing Desktop SQLite has been backed up online with SQLite backup API and validated
integrity_check=okafter an isolated DB-level roundtrip. No native service restart, stop_job, Runner replacement, JSON edit, DB mutation, WAL deletion or binary upgrade was performed.
Expected
The same Runner's existing-job inventory should reconcile these atomic terminal receipts and display stopped/-1 and completed/0 respectively, not keep capacity reserved as running. Per upstream #88 acceptance criteria, Supervisor terminal evidence should be authoritative with PID+start-time fencing when the Supervisor is no longer alive.
Request
- What is the supported non-destructive reconciliation mechanism in v0.4.6 for already-terminal detached supervisor records, without restarting Desktop/Runner/Server or misclassifying a completed Job as cancelled?
- Does current main or a post-v0.5.0 release fix missing terminal aggregation / stale in-memory
runningstate when the same Runner process has remained online? - If a controlled Runner restart is the only supported recovery, what are the exact safety gates for preserving the original 37 Projects, other Runner identities, credentials, and prior terminal receipts, and avoiding duplicate execution?
- Would an authenticated, bounded read-only Job→Supervisor state diagnostic endpoint and a terminal-only reconcile operation be appropriate?
This report does not request disabling safeguards, force-stop, editing Runner state or premature Production migration. The normal existing independent GitHub AUTO GO workflow continues.
Related design: #88 detached ownership handoff.
Activity
hanklee5677 commentedon Nov 10, 2026
Additional read-only evidence (2026-11-10)
We recovered the exact Runner-owned per-Job state under the local detached-supervisor state directory, with the original atomically committed terminal record and OS-process start identity. We then independently checked the four original PID numbers through Windows
Win32_Process: none exist in the current process table. This is not a guess from inactivity or absence of cargo.terminal.status=stopped, Exit -1,stop_requested=true, terminal timestamp Nov 2 06:42:35.182 UTC. The retained bounded test log contains onetest result: ok. 8 passed; 0 failedsummary, but whole Job is stopped, not completed. Further original native WebCodex ActionAudit lists four historicalstop_jobtool calls for this Project at approximately the same time (first call1793601755075ms, 107 ms before the Job A terminal record). Each call's detailed trace payload is no longer retained, so we cannot attribute exact args to all four events; do not assume current stop request or manually rewrite outcomes.terminal.status=completed, Exit 0,stop_requested=false, terminal Nov 2 05:11:12.098 UTC. Its retained bounded output contains onetest result: ok. 4 passed; 0 failedsummary (not a claim of a complete test-suite count).observe_jobsafter discovery still reports BOTHrunning,terminal=false, with old log tails and no Exit Code, including Job A where terminal explicitly says stopped by request.5c5cbaaca9c27b51cda75d15ddd338e52eb91c6173b39671cbace2252790aba4. No raw command, credential, Runner token, private data, or full local path was published.We request a terminal-only non-destructive reconciliation contract that returns these original recorded terminal statuses and releases incorrect running capacity without re-executing or overwriting original stopped/completed results. We have not invoked
stop_job, service restart, DB mutation, or a new Runner on the impacted HOME since discovering the mismatch.hanklee5677 commentedon Nov 10, 2026
Critical source-level finding: 24-hour terminal retention makes blind Runner restart unsafe
After filing this issue, I inspected current upstream source (read-only):
crates/webcodex-core/src/runner_protocol/job.rssetsJOB_TERMINAL_RETENTION_SECS = 24 * 60 * 60.crates/webcodex-runner/src/webcodex_runner/job_manager/detached.rs::recover_detached_jobsskips a terminal detached record whennow - snapshot.ended_at >= JOB_TERMINAL_RETENTION_SECS(the relevant source check is around lines 110–118).crates/webcodex-runner/src/webcodex_runner/detached_job/store.rs::reclaim_expired_terminal_records_lockedcan reclaim an old terminal Job directory once terminal age passes the same threshold, with identity checks (around lines 246–315).stopped/-1andcompleted/0states; the implementation may skip them at recovery and reclamation could discard the remaining original evidence. Please confirm whether any supported migration/reconciliation path can preserve the exact outcomes while freeing stale inventory capacity without a restart or retention loss.Additional evidence preservation actually performed on HOME (no live runtime mutation): created an access-restricted backup directory outside the active Runner state containing two copied raw terminal
state.jsonfiles, with source vs copy exact SHA-256 equal and size match:9d70ef2aca6f92832084a8afb47f3b7fe4976e82fb459b922e0d0d436015821a(8770 bytes).a43e85cfd58d912c3a9282ec8862c9449e5509e8cb71f99fe573c5f4eca8c5b7(7283 bytes).Copy ACL: SYSTEM, Administrators, current HOME User only. The raw states remain local and are not attached here because their payload/context may be private. Original Runner-owned state files untouched.
Suggested safe fix design for upstream consideration: terminal-only authenticated reconciliation in the same running Runner, read the exact original state with owner namespace/Job+execution identity/creation-fence validation, preserve original terminal status/exit code, monotonically update inventory and queue an authoritative terminal update to Server, do not respawn/rewrite old outcomes, and clarify how to handle records beyond 24h retention without silently losing previous closure evidence. A read-only preview/dry-run before any state change would make this easier to accept.
Still no stop_job, Runner restart, DB edit, WAL deletion, or source patch of previously denied Platform A Host Grant/Release.
hanklee5677 commentedon Nov 10, 2026
Source tag verification: this is also true in the installed v0.4.6 code, not only current main
Independently fetched authentic GitHub
v0.4.6tag contents:crates/webcodex-core/src/runner_protocol/job.rsblobf38f44b6ae4fd05eb3071f9b48d37c84d9ed18a1, line 53:JOB_TERMINAL_RETENTION_SECS = 24 * 60 * 60.crates/webcodex-runner/src/webcodex_runner/job_manager/detached.rsblob71bab97c68f60c32bdf214c8497a6a7be8e38aa7, around line 142:recover_detached_jobsskips a terminal record whennow - ended_at >= JOB_TERMINAL_RETENTION_SECS.detached_job.rs(monolithic pre-refactor), blobcfb54cead5384ebf3b844828b4aad1ad10ce4997:reclaim_expired_terminal_records_lockedcomputesterminal_expiredand permits identity-fenced reclamation at the same 24h threshold.These exact older-tag source checks establish that blind restart is insufficient and possibly destructive for Nov 2 terminal records now over eight days old. Prior comment used current main; this follow-up removes version mismatch uncertainty.
Two original unmodified
state.jsonfiles are now locally preserved under the existing HOME ACL-protected recovery directory, verified exact SHA256 source/copy and file ACL:A
9d70ef2aca6f92832084a8afb47f3b7fe4976e82fb459b922e0d0d436015821a,B
a43e85cfd58d912c3a9282ec8862c9449e5509e8cb71f99fe573c5f4eca8c5b7.Copies are not used as a replacement live Runner state and have not been uploaded.
Please prioritize a supported same-live-Runner terminal-only reconcile or a sanctioned migration/archival path that preserves Job A's
stopped/-1, Job B'scompleted/0, does not rerun either Job, and does not allow 24h expiry to erase final audit evidence before Server inventory is repaired. We need accurate ownership/exit-code propagation, not simply hide two stale rows or blindly call stop_job.hanklee5677 commentedon Nov 10, 2026
Root cause now source-confirmed: older v0.4.6 detached Job heartbeat/durable sequence regression; prevention fixed by merged PR #970
New precise source-to-symptom linkage found without modifying HOME:
v0.4.6crates/webcodex-runner/src/webcodex_runner/job_manager/delivery.rs::queue_job_heartbeat_batchfilters all active Jobs, then incrementsjob.snapshot.update_seqevery heartbeat (lines ~388-390), including supervisor-ownedrun_detached_process.v0.4.6job_manager/detached.rs::sync_detached_record, an authentic record from Runner-owned durable Supervisor is rejected ifsnapshot.update_seq < job.snapshot.update_seq, with errordetached Job durable update sequence regressed(lines ~287-288), and the observer logsdetached Job inventory sync failed closedthen returns (around lines ~253-257). This logically explains indefinite in-memory Running after the original detached state advanced to terminal.0d65db122d8d841cc9a98e0a5b5d2fc11a6d70f3; named fix2e2340235297133dc8b8c51d503739bdf12f59a6fix(runner): keep detached update sequence durable. The PR description explicitly calls out v0.4-era real dogfood symptomlocal durable state terminal; Server continued running; heartbeat advances ahead of durable sequence; original tests listed under PR. Current maindelivery.rsexcludesexecution_source == run_detached_processfrom generic heartbeat batch. GitHub commit ancestry confirms v0.5.0 includes PR fix(detached jobs): surface timeout clamping and preserve durable sequence #970 (compare/v0.5.0...0d65db...: status behind, meaning fix commit is ancestor of the v0.5.0 tag).v0.4.6code does not have that exclusion. This is a source-confirmed matching mechanism but without original Runner logs, the precise contemporaneous error instance remainsORIGINAL_RUNTIME_ERROR_LOG=NOT_CAPTURED.stopped/-1, Job Bcompleted/0; native Server inventory still reports bothrunning. Original files and protected copies SHA verified.v0.4.6terminal retention is 24h, and its recovery logic skips expired terminal records; those Nov 2 receipts are now >8 days old.Important distinction for fix acceptance
Prevention for newly created detached jobs is fixed upstream #970; retroactive historical reconciliation is NOT fixed by merely installing v0.5.0. Need sanctioned same-live-Runner terminal-only reconciliation/migration that preserves original stop/completion statuses beyond 24h, or official data-preserving approach. Do not trigger a blind upgrade/restart and lose terminal status evidence. HOME Production 37/37 Projects and 2 stale Running entries kept unchanged, no DB/JSON edits or stop_job.
Can upstream clarify if existing present-day native command/API can safely re-admit these exact terminal snapshots to Server registry on v0.4.6 or newer without expiry, and whether a small separate repair PR for historical terminal state replay is welcome? We can verify a read-only dry-run and native regression in an authorized isolated environment; original Host Grant/Release denied operations remain out of scope.
hanklee5677 commentedon Nov 10, 2026
Exact root cause found: already fixed for new detached Jobs in merged PR #970 (v0.5.0), historical v0.4.6 inventory still requires supported recovery
We traced the installed
v0.4.6source to a concrete race:job_manager/delivery.rs::queue_job_heartbeat_batch(v0.4.6 near lines 387–390) chooses all active Jobs, includingrun_detached_process, and increments the memory-onlyjob.snapshot.update_seqon each heartbeat.job_manager/detached.rs::sync_detached_record(v0.4.6 near lines 275–288) rejects an authentic durable Supervisor snapshot whensnapshot.update_seq < job.snapshot.update_seqwithdetached Job durable update sequence regressed. The detached Observer exits after loggingdetached Job inventory sync failed closed(near lines 253–257). Once the generic heartbeat has advanced the in-memory sequence ahead of supervisor's durable sequence, the actual terminal record may never reach Server. This matches the two HOME records: original supervisor terminalstopped/-1andcompleted/0with Server stillrunning.0d65db122d8d841cc9a98e0a5b5d2fc11a6d70f3, constituent commit2e2340235297133dc8b8c51d503739bdf12f59a6titled fix(runner): keep detached update sequence durable. The PR explicitly documents the same real-world failure: durable Job terminal but Server still running due to heartbeats racing ahead; it included native regression tests. Current maindelivery.rsexcludesstructured_execution.execution_source == run_detached_processfrom generic heartbeats. GitHub compare shows the fix(detached jobs): surface timeout clamping and preserve durable sequence #970 merge commit is an ancestor of official v0.5.0. The installed v0.4.6 code still uses the buggy filter.JOB_TERMINAL_RETENTION_SECS=86400; v0.4.6 recovery skips such expired terminal entries and reclamation may erase originals. Two raw original state files have already been copied with exact SHA256 and ACL into protected HOME recovery storage. DO NOT blindly restart or upgrade just to free slots, and do not issuestop_jobagainst already-completed B.What remains for #1417 is a supported, non-destructive terminal-only reconciliation/migration for already-stale v0.4.6 Job inventory preserving exact A
stopped/-1and Bcompleted/0before deploying any officially accepted future Release. Please consider adding acceptance tests for importing historical >24h terminal receipts, not just preventing new races. Existing 37 Projects, Qwen/AI Hank work, and independent free GitHub AUTO GO must remain unchanged.hanklee5677 commentedon Nov 10, 2026
Exact installed build provenance removes the v0.4.6 tag ambiguity
The live HOME WebCodex MCP
runtime_statusreports build Git SHA5072608fe74b2bfa6e8f208515c04242f475582f(version label v0.4.6, git_dirty=false). We independently fetched this exact source commit from the official upstream GitHub repository; its committer date is 2026-10-03T12:34:23Z, before PR #970 merged.The exact commit's
crates/webcodex-runner/src/webcodex_runner/job_manager/delivery.rshas Git blob35ff648a9972a9a1d565fbc4cbd775fd86b382d0; it does containjob.snapshot.update_seq = job.snapshot.update_seq.saturating_add(1)and does NOT contain themetadata.execution_source == "run_detached_process"heartbeat exclusion. This is not merely an approximation from the v0.4.6 tag; the advertised running build source has the PR #970 bug.PR #970 fix merge
0d65db122d8d841cc9a98e0a5b5d2fc11a6d70f3came after that build and is an ancestor of official v0.5.0. Historical original Runner stderr for Nov 2 remains unavailable, so the source/symptom match is strong evidence, not a reconstructed contemporaneous runtime error.Needed safe action remains retroactive same-live-Runner terminal-only reconciliation of the two durable, already-stopped/completed Jobs (now >24h old) preserving exact historical exit code, followed by officially allowed version validation/cutover—not an unsafe immediate restart.
hanklee5677 commentedon Nov 10, 2026
Decisive HOME live-vs-durable sequence evidence (2026-11-10)
New independent same-job numeric evidence strengthens the upstream #970 race diagnosis beyond source-code similarity.
From authorized native HOME
list_jobsandobserve_jobs, on exact Projectautonomous-development-loop-v0-1-0fbcbe5e-70e4de77, both Jobs still showrunningwith no exit/ended_at:detached_y1UOCOartZMaJLDxk5s7xpBSRGqhmlwTsdnWS2g01k4: live publiclast_update_seq=47695.detached_zCpXL2fY20miHBHBazephnM3VXA5r4lGEsFVFPNgVMY: live publiclast_update_seq=48110.At the same inspection, the exact corresponding unmodified Runner-owned
runner-detached-jobs-v1originalstate.jsonrecords show:durable.update_seq=12,phase=terminal,terminal.status=stopped,exit_code=-1.durable.update_seq=13,phase=terminal,terminal.status=completed,exit_code=0.Thus both live inventory sequences are thousands ahead of the authoritative durable terminal sequences, exactly satisfying the
v0.4.6failure conditionsnapshot.update_seq < job.snapshot.update_seqthat rejectssync_detached_record. This directly supports PR #970's diagnosis (generic heartbeat increments detached memory-only seq while Supervisor independently owns durable sequence). Both original states have been backed up and verified SHA256 in ACL-protected local recovery directory.For a genuine safe fix, please distinguish:
A supported repair must authenticate exact Job ID, execution ID, Runner/Project owner, process creation fence and original terminal record, not directly reset update_seq in state.json, not call stop_job, not rerun tasks. It should conservatively reconcile the public registry with a monotonically admissible terminal update while preserving original durable terminal result, or provide an approved migration contract. Blind Runner restart risks 24h expired-terminal omission/deletion; no live mutation attempted.
Observation limitation: prior original Runner stderr error text from Nov 2 was not retained, so this is direct numeric mismatch plus code-matched rejection, not a captured old exception.
hanklee5677 commentedon Nov 10, 2026
Reproduced LIVE sequence drift on the still-running v0.4.6 HOME Runner — two native observations, not a simulated test
Using the existing authenticated HOME
list_jobstwice in this investigation (no Runner restart, process mutation, DB edit, Job stop or re-execution), publiclast_update_seqadvanced while both Jobs stayedrunning:47695 → 47701, +6; its exact Runner-owned original durablestate.jsonremainsupdate_seq=12,phase=terminal,stopped, Exit -1.48110 → 48116, +6; exact durablestate.jsonremainsupdate_seq=13,phase=terminal,completed, Exit 0.This is live, repeated public-sequence inflation with unchanged authentic terminal receipts, not merely a historical example. Installed build
5072608fe74b2bfa6e8f208515c04242f475582fcontains the pre-PR-970 generic-heartbeat path, confirmed by exact GitHub source blob. It directly matches thesnapshot.update_seq < job.snapshot.update_seqrejection path in the detached observer.Created a protected local summary-only diagnostic JSON
HOME_DETACHED_SEQUENCE_DRIFT_20261110.jsonin the existing restricted HOME recovery directory; 12/12 assertions PASS across source/copy exact bytes, original terminal/exit outcomes and two public-sequence observations. Independent PowerShell SHA256 readback3e29f4fa42ceb95c0d7917283567b1762b06ae24bb3317058965a63ed6e44e2e; ACL SYSTEM/Admin/current User only. No original state, command or secrets uploaded.Request: upstream please confirm an approved terminal-only same-live-Runner historical reconciliation/migration for >24h records. Existing #970 prevents NEW sequence drift in v0.5.0 but does not retroactively release two v0.4.6 stale entries without potentially losing original old terminal receipts at 24h retention. Avoid declaring these tasks newly completed by forcing Server state; preserve original A stopped/-1 and B completed/0.
hanklee5677 commentedon Nov 10, 2026
2026-11-11 post-Windows-reboot reconciliation evidence — IMPORTANT correction to historical symptom
As of 2026-11-11 06:39–06:41 Asia/Taipei, HOME v0.4.6 still has 37/37 online Projects, healthy Server and Runner, but native
runtime_statusnow reportsactive_count=0, not the previous 2 stale Running Jobs. Exact originallist_jobsreturns zero entries; authenticatedobserve_jobsfor both original Job IDs returnsunknown_job, notcompleted/stoppedwith original exit codes.Read-only Windows process census confirms current WebCodex Desktop v0.4.6 and its Server/Runner were created at ~2026-11-11 06:36:36–06:36:40 Taipei, following an OS boot at 06:36:00. Windows EventLog:
MoUsoCoreWorker.exerequested a restart at ~03:05:59 Taipei.TrustedInstaller.exerequested a planned OS restart at ~03:08:47 Taipei.REBOOT_CAUSE=NOT_VERIFIED.The exact original Runner-owned private per-Job
state.jsondirectories are still present, unchanged last write Nov 2; the true terminal records Astopped/-1and Bcompleted/0are preserved on local disk, plus separately protected copy files. Independently rehashed protected sources (2/2 original terminal copies, live-consistent SQLite backup, recovery evidence ledger, 37 Project file restore receipt), all PASS. Original 37 Project TOML files match pre-reboot SHA256s (37/37), and both Runner/Server TOML SHA256s are unchanged. No active HOME state was edited, no service restarted by this investigation.Interpretation: the two false Running inventory entries disappeared during unexpected host reboot/Runner replacement. v0.4.6
recover_detached_jobsdeliberately skips already-expired (>24h) terminal records, explainingunknown_jobeven while durable terminal files remain. This is not a successful historical terminal-only replay, and the original stopped/completed outcomes are still absent from native public inventory. The stale slot symptom is no longer active, but the historic audit/replay discrepancy persists. If upstream proposes #1417 as resolved due to '0 running', do not classify the original terminal correctness requirement as PASS without native accepted exact status+exit readback.New locally protected summary-only incident receipt
HOME_REBOOT_JOB_RETENTION_AUDIT_20261111.json, SHA256eba692ff7bb1b5db93e210157e1c6fc425e29af8b9ab09a293c7a87dcd7f2666; ACL SYSTEM/Admin/current User only, no raw states/credentials sent to GitHub.Upstream #970 already prevents new Detached heartbeat sequence regression in v0.5.0, but HOME is on v0.4.6 and latest official v0.5.0 still lacks later PR #1393/#1413. This is not permission to run installer or retry previously denied Host Grant/Release writes. Please clarify a documented historical native terminal evidence / archive-reconciliation path after 24h expiry, while preserving original outcomes and 37 Project registrations.