Skip to content

status reports a background job as running forever when its worker dies before writing a terminal status #704

Description

@djd1m

Version: plugin codex 1.0.5 (marketplace openai-codex 1.0.5), Claude Code on Linux, Node 22.

What happens

codex-companion.mjs task --background spawns a detached worker (spawn(..., { detached: true, stdio: "ignore" }), then child.unref()spawnDetachedTaskWorker, ~line 671) and writes a job record with status: "queued" and the worker's pid (enqueueBackgroundTask, ~line 690). Writing the terminal status is the worker's own job.

Nothing reconciles that record with reality afterwards. If the worker dies before it writes its terminal status — SIGKILL, OOM, an unhandled rejection, a lost connection — the record keeps status: "running" indefinitely, and status keeps reporting it as running with a climbing elapsed time, even though the recorded PID is no longer in the process table.

Deterministic reproducer (no race needed)

The point is the missing check, so this reproduces it without having to win a race against a real worker:

CC=~/.claude/plugins/cache/openai-codex/codex/1.0.5/scripts/codex-companion.mjs
mkdir -p /tmp/codex-repro && cd /tmp/codex-repro && git init -q

# 1. create any background job so the workspace state exists
node $CC task --background "Reply with exactly: OK"

# 2. take the state dir for this workspace
D=$(find ~/.claude/plugins/data/codex-openai-codex/state -maxdepth 1 -name 'codex-repro-*' | head -1)

# 3. write a record in the state a dying worker leaves behind:
#    status=running, and a PID that provably does not exist
node -e '
const fs=require("fs"), D=process.argv[1];
const any=fs.readdirSync(D+"/jobs").find(f=>f.endsWith(".json"));
const j=JSON.parse(fs.readFileSync(D+"/jobs/"+any,"utf8"));
j.id="task-repro00-deadpid"; j.status="running"; j.phase="editing"; j.pid=999999;
delete j.completedAt; delete j.result;
fs.writeFileSync(D+"/jobs/task-repro00-deadpid.json", JSON.stringify(j,null,2));
const s=JSON.parse(fs.readFileSync(D+"/state.json","utf8"));
(s.jobs=s.jobs||[]).push({...j});
fs.writeFileSync(D+"/state.json", JSON.stringify(s,null,2));
' "$D"

# 4. confirm the PID really is absent, then ask the plugin
ps -p 999999 || echo "pid 999999 does not exist"
node $CC status task-repro00-deadpid

Observed:

# Codex Job Status
- task-repro00-deadpid | running | rescue | Codex Task
  Phase: editing
  Elapsed: 1m 20s

Elapsed keeps growing on every subsequent call. Nothing ever reports the worker as gone.

Expected: a job whose recorded PID is absent from the process table should be reported as terminated-with-unknown-outcome (or similar) rather than running — the record cannot be trusted more than the operating system.

Field impact

Observed six times in one night of automated use on this machine (2026-08-30/31). Jobs stayed running with an absent recorded PID for as long as 8h51m. Every consumer of the status — supervisors, an orchestration barrier that waits for a coder to finish, an idle-detection rule — reads "work in progress" and waits. In our case that produced ~4.5h of a lead agent monitoring jobs that had been dead for hours.

Note on evidence: the raw records for those six were overwritten when we cancelled the jobs (cancel writes a terminal status and clears pid), so the reproducer above is the reliable artifact; the field numbers come from our own supervisor logs.

Interesting detail, in case it helps localise: three of the six were the same stage of our pipeline and each had already written its output file before the record went stale — the work completed, only the bookkeeping did not.

Suggested fix

Reconcile on read rather than trusting the record. In status (and wherever jobs are listed), for any job whose stored status is non-terminal and whose pid is a positive integer, check liveness (process.kill(pid, 0)ESRCH means gone, EPERM means alive but not ours) and report a job with an absent PID as terminated-unknown. That is a few lines, requires no change to the worker, and fixes every stale record already on disk.

A stronger variant, if you prefer to keep status read-only: add a reaper pass on startup that rewrites such records once, with a reason.

Workaround we are using

We stopped trusting the status field: liveness = terminal status or the recorded PID being absent from the process table. That works, but every consumer of your API has to reimplement it.

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions