Repository navigation
Description
Status: fix written, tested, and pushed to the fork. Not yet emailed upstream.
Branch: fix/timeout-process-group — one commit, cb2561584, based on master 17eb78e6d. Purely additive: 27 insertions, 0 deletions, 2 files.
Companion to #4, but independent of it — the two touch disjoint files and either can land alone.
The bug
exec_timeout fires, GracefulTerminate() signals the process we started, and anything that process spawned survives it. The survivor still holds the write end of the pipe, so the parent stays blocked in its read loop on a command it has already given up on. exec_timeout therefore does not bound the promise's wall clock at all.
This is the ordinary case rather than a corner case: any shell script that runs another program has this shape.
body common control { bundlesequence => { "t" }; }
body contain c { useshell => "noshell"; exec_timeout => "2"; }
bundle agent t
{
commands:
"/bin/sh" arglist => { "-c", "sleep 30; exit 0" }, contain => c;
}
| case | before | after |
|---|---|---|
sh -c 'sleep 30', exec_timeout 2 |
30.3 s, sleep orphaned and left running |
5.0 s, no orphan |
sh -c "trap '' INT TERM; sleep 30", exec_timeout 2 |
30.3 s | 4.4 s |
(The "after" numbers are with #4 also applied, since that one governs how long the termination ladder itself takes. With this change alone the first case is 11.9 s instead of 30.3 s, and still leaves no orphan.)
The fix
cf_popen()'s child calls setpgid(0, 0) so a timed-out command and its descendants can be signalled as a unit. setpgid() is async-signal-safe, so it is legal in the post-fork child alongside the existing sigprocmask() call.
TimeOut() reads the process group before calling GracefulTerminate() and sweeps it after. Reading it first is not cosmetic — once the process has been killed, getpgid() fails with ESRCH and there is no safe way left to tell whether it led a group of its own. I found that the hard way: the first version of this patch queried the pgid after termination, the guard silently never fired, and the bug appeared unfixed.
The sweep is guarded on pgid == pid specifically so that a child which somehow is not a group leader cannot turn into a negative kill() against the agent's own group.
Two things I am not certain about
Flagging these rather than presenting the patch as obviously right:
setpgid()applies to everycf_popen()child, not only those with anexec_timeout. That detaches children from the agent's process group, so a terminal SIGINT (Ctrl-C) sent to the foreground group no longer reaches a running child. For non-interactive agent runs this is invisible, but it is a real behaviour change for interactive use. The alternative — setting the group only when anexec_timeoutis present — would make the timeout path structurally different from the normal one, which seemed worse. I do not have strong grounds for preferring one.- The sweep is an unconditional
SIGKILLto the group, after the leader has already been through the graceful ladder. Escalating over the whole group would be gentler, but that means changingGracefulTerminate(), which is shared with the stale-lock path inlocks.cwhere group semantics would be wrong.
Testing
tests/unit/process_terminate_unix_test— 6/6.- Full
make checkintests/unit— 64 PASS, exit 0, and the FAIL/ERROR/XFAIL set is identical to before the change (compared line by line). The pre-existingprocess_testXFAILs are the Darwin process stub and are unrelated. - Checked separately that argv fidelity through
commands:arglistis unaffected (spaces, both quote characters, shell metacharacters, empty arguments and tabs all still arrive verbatim, ARGC unchanged), and that a command with noexec_timeoutstill runs normally and still leads to a correct promise outcome.
Activity
upstream: B-2 fixed and filed; branch layout and email duty recorded
djbclark commentedon Aug 17, 2026
Second-opinion review found a regression in this patch (2026-08-16)
Three independent non-Claude reviewers (Cursor, Gemini, Grok) were given this
patch adversarially. All three said the unconditional
setpgid(0, 0)mustnot ship as written, and the reason is worse than the Ctrl-C tradeoff flagged
in the original report.
The regression, measured
setpgid(0, 0)put everycf_popen()child in its own process group. A childin its own group is no longer in the terminal's foreground group, so it is
stopped by SIGTTIN the first time it reads the terminal — and type-
'r'pipes do not redirect stdin, so the child inherits the agent's.
Measured under a pty,
sh -c 'read x; echo GOT-$x', noexec_timeoutatall:
GOT-helloProcess state during the hang:
PGID 2710= its own group (thesetpgidtook effect),TPGID 2669= theterminal's foreground group is still the agent's,
STAT T= stopped. The agentthen blocks forever reading a pipe held by a stopped child.
That is the same unbounded hang this patch exists to remove, reintroduced on
a path that has no timeout to end it. Strictly worse than the Ctrl-C concern.
Two further consequences the reviewers raised, both from the same cause:
exits immediately — so Ctrl-C used to take its children with it and now
orphans them.
agent_expireafterkills by process group(
cf-execd-runner.cdoeskill(-agent)whengetpgid(agent) == agent).Children detached from that group survive the watchdog — which exists for
precisely the hung command this patch is about. Grok called this the
production hit, and it is the strongest argument of the three.
The blast radius is the whole popen family, since
GenericCreatePipeAndFork()is the single fork behindcf_popen,cf_popen_select,cf_popensetuid,cf_popen_sh,cf_popen_sh_select,cf_popen_shsetuidandcf_popen_full_duplex—execresult(), package listcommands,
ps,getent, monitord pipes, custom modules, cf-execd's own shellaround cf-agent.
My stated reason for doing it unconditionally — not wanting the timeout path
to be structurally different from the normal path — was backwards. The
timeout path is different: it is the only one that has to kill a tree. The
normal path has to stay reachable by the things that already kill trees.
Fix — commit
847373cf6, same branchThe process group is now created only when a timeout is armed, which is
exactly when
TimeOut()may need to sweep it.SetTimeOut()records that,ClearTimeOut()andTimeOut()clear it. The four call sites that open-codedalarm(0); signal(SIGALRM, SIG_DFL);now callClearTimeOut(), so a commandthat completes in time cannot leave the flag set for the next, unrelated child.
Those are the only
SetTimeOut()users in the tree (verify_exec.c,nfs.c,cf-monitord/history.c).Verified after the fix:
sh -c 'read x', no timeout, under a ptysh -c 'sleep 30; exit 0',exec_timeout => "2"sleeporphaned(10.8s rather than ~4.4s because this branch alone does not include #4's
deadline fix; merged with it, 4.7s.)
Reviewer answers to the other two open questions
timeout path, and the shared-function argument is true: putting group
semantics into
GracefulTerminate()would letlocks.c's stale-lock reaperkill(-agent)and take down an entire execd generation. The sweep belongs inTimeOut()where it is.pgid == pidbefore termination"closes" a recycle race is wrong, and two reviewers said so independently.
It closes a different, real bug:
getpgid()returns ESRCH once the leader isdead, so reading first is mandatory or the sweep never runs. It does not bind
the pid against reuse in the window between
GracefulTerminate()returningand
kill(-pid, SIGKILL). The window is a handful of syscalls; neitherreviewer would block on it; neither would describe it as closed. Corrected
here rather than left in the commit message.
Still outstanding on this branch
setpgid()/getpgid()return values are still unchecked. On failure thesweep is silently skipped and the patch becomes a no-op for that child. All
three reviewers asked for a log line; not yet done.
setpgid(pid, pid)(thePOSIX both-sides pattern), since a timeout firing before the child reaches
its own
setpgidwould skip the sweep. Unlikely atexec_timeout >= 1s;not yet done.
CONTRIBUTING.mdasks for one and thesh -c 'sleep 30'shape is exactly that. Not yet done.
Tests:
process_terminate_unix_test6/6; fulltests/unitsuite all 68 asexpected with 4 expected failures, identical to baseline.
djbclark commentedon Aug 17, 2026
Now tracked upstream in Jira as CFE-4729.
This item was reported to security@northern.tech on 2026-08-17. The Jira ticket carries the corrected claims from the review panel, not the original wording. The unconditional-setpgid regression and the PID-recycling correction are both recorded there.
This fork issue stays as the working record; CFE-4729 is the upstream reference. Going forward we file upstream items in the CFE Jira project rather than by email or GitHub Discussion.
djbclark commentedon Aug 22, 2026
Upstream disposition: closed
Filed as CFE-4729, now Rejected / Done. Upstream PR cfengine/core#6305 closed 2026-08-19T10:45:01Z on review volume.
Ticket-only content worth keeping
This item got the only substantive maintainer code review of the whole 26. nickanderson reviewed cfengine#6305 on 2026-08-18 with one inline note on
cf-agent/verify_exec.c: the added comments are too heavy, terse ones would do.Trimmed across the whole diff as
0e06ad3d7— 37 lines removed, comment volume roughly halved, keeping only the non-obvious constraints (the sampling order aftercf_pclose, the pgid guard, and the SIGTTIN reason for theTimeOutIsArmedcondition). Verified comment-only: with comments stripped, the token stream is byte-identical to the previous head in all four files. Clean rebuild,timeout_test6/6.That feedback generalises past this one diff and is worth applying to everything on this fork.
Branch naming: the shipped fix is on
fix/timeout-process-group-merged(merged with the CFE-4726 reporting fix), not thefix/timeout-process-groupthe ticket named.djbclark commentedon Nov 1, 2026
Reproduces on upstream/master 5ac9805. Fresh, smaller branch fix/timeout-process-group-r2 at 3dc774b (3 files + acceptance test; the old -merged branch bundled 11 commits): acceptance
timeout_kills_descendants.cffails without the fix, passes with it; unit check 78/78. Not yet a PR.djbclark commentedon Nov 1, 2026
Rework after review: fix/timeout-process-group-r3 (7b1d260): agent SIGTERM/SIGINT now passed on to the child's group; follow-up limits in #76. Not a PR yet.
djbclark commentedon Nov 1, 2026
Reworked as
fix/timeout-process-group-r4(1fdb294), a linear series on upstream/master carrying #83, #84 and #69 as its first commits; land after those. Changes from -r3: the agent's SIGTERM/SIGINT handler kills the timed child's group with SIGKILL instead of forwarding the signal (a command ignoring SIGINT survived -r3), and the own-group record replaces getpgid(). Details in #76.djbclark commentedon Nov 1, 2026
Reworked as
fix/timeout-process-group-r5(b0ee8fd), a linear series on upstream/master carrying #83, #84 and #69 as its first commits; land after those. Change from -r4: the agent's SIGINT/SIGTERM handler gives the timed child's group master's graceful stop (the signal it got, SIGTERM after 1 s, SIGKILL after 3 s) instead of SIGKILL at once. Details in #76.