Skip to content

PR 5: Signal the whole process group when a command times out #5

Open

Description

@djbclark

Status: fix written, tested, and pushed to the fork. Not yet emailed upstream.

Branch: fix/timeout-process-group — one commit, cb2561584, based on master 17eb78e6d. Purely additive: 27 insertions, 0 deletions, 2 files.

Companion to #4, but independent of it — the two touch disjoint files and either can land alone.

The bug

exec_timeout fires, GracefulTerminate() signals the process we started, and anything that process spawned survives it. The survivor still holds the write end of the pipe, so the parent stays blocked in its read loop on a command it has already given up on. exec_timeout therefore does not bound the promise's wall clock at all.

This is the ordinary case rather than a corner case: any shell script that runs another program has this shape.

body common control { bundlesequence => { "t" }; }
body contain c { useshell => "noshell"; exec_timeout => "2"; }
bundle agent t
{
  commands:
      "/bin/sh" arglist => { "-c", "sleep 30; exit 0" }, contain => c;
}
case before after
sh -c 'sleep 30', exec_timeout 2 30.3 s, sleep orphaned and left running 5.0 s, no orphan
sh -c "trap '' INT TERM; sleep 30", exec_timeout 2 30.3 s 4.4 s

(The "after" numbers are with #4 also applied, since that one governs how long the termination ladder itself takes. With this change alone the first case is 11.9 s instead of 30.3 s, and still leaves no orphan.)

The fix

cf_popen()'s child calls setpgid(0, 0) so a timed-out command and its descendants can be signalled as a unit. setpgid() is async-signal-safe, so it is legal in the post-fork child alongside the existing sigprocmask() call.

TimeOut() reads the process group before calling GracefulTerminate() and sweeps it after. Reading it first is not cosmetic — once the process has been killed, getpgid() fails with ESRCH and there is no safe way left to tell whether it led a group of its own. I found that the hard way: the first version of this patch queried the pgid after termination, the guard silently never fired, and the bug appeared unfixed.

The sweep is guarded on pgid == pid specifically so that a child which somehow is not a group leader cannot turn into a negative kill() against the agent's own group.

Two things I am not certain about

Flagging these rather than presenting the patch as obviously right:

  1. setpgid() applies to every cf_popen() child, not only those with an exec_timeout. That detaches children from the agent's process group, so a terminal SIGINT (Ctrl-C) sent to the foreground group no longer reaches a running child. For non-interactive agent runs this is invisible, but it is a real behaviour change for interactive use. The alternative — setting the group only when an exec_timeout is present — would make the timeout path structurally different from the normal one, which seemed worse. I do not have strong grounds for preferring one.
  2. The sweep is an unconditional SIGKILL to the group, after the leader has already been through the graceful ladder. Escalating over the whole group would be gentler, but that means changing GracefulTerminate(), which is shared with the stale-lock path in locks.c where group semantics would be wrong.

Testing

  • tests/unit/process_terminate_unix_test — 6/6.
  • Full make check in tests/unit — 64 PASS, exit 0, and the FAIL/ERROR/XFAIL set is identical to before the change (compared line by line). The pre-existing process_test XFAILs are the Darwin process stub and are unrelated.
  • Checked separately that argv fidelity through commands: arglist is unaffected (spaces, both quote characters, shell metacharacters, empty arguments and tabs all still arrive verbatim, ARGC unchanged), and that a command with no exec_timeout still runs normally and still leads to a correct promise outcome.

Activity

  1. djbclark commented on Aug 17, 2026

    @djbclark
    OwnerAuthor

    Second-opinion review found a regression in this patch (2026-08-16)

    Three independent non-Claude reviewers (Cursor, Gemini, Grok) were given this
    patch adversarially. All three said the unconditional setpgid(0, 0) must
    not ship as written
    , and the reason is worse than the Ctrl-C tradeoff flagged
    in the original report.

    The regression, measured

    setpgid(0, 0) put every cf_popen() child in its own process group. A child
    in its own group is no longer in the terminal's foreground group, so it is
    stopped by SIGTTIN the first time it reads the terminal — and type-'r'
    pipes do not redirect stdin, so the child inherits the agent's.

    Measured under a pty, sh -c 'read x; echo GOT-$x', no exec_timeout at
    all
    :

    build result
    stock 3.27.1 finishes in 0.1s, prints GOT-hello
    this branch, before the follow-up hangs indefinitely

    Process state during the hang:

      PID  PPID  PGID TPGID STAT COMMAND
     2669  2667  2669  2669 Ss+  cf-agent -KI -f ...
     2710  2669  2710  2669 T    /bin/sh -c read x; echo GOT-$x
    

    PGID 2710 = its own group (the setpgid took effect), TPGID 2669 = the
    terminal's foreground group is still the agent's, STAT T = stopped. The agent
    then blocks forever reading a pipe held by a stopped child.

    That is the same unbounded hang this patch exists to remove, reintroduced on
    a path that has no timeout to end it
    . Strictly worse than the Ctrl-C concern.

    Two further consequences the reviewers raised, both from the same cause:

    • A terminal SIGINT reaches only the foreground group, and cf-agent's handler
      exits immediately — so Ctrl-C used to take its children with it and now
      orphans them.
    • cf-execd's agent_expireafter kills by process group
      (cf-execd-runner.c does kill(-agent) when getpgid(agent) == agent).
      Children detached from that group survive the watchdog — which exists for
      precisely the hung command this patch is about. Grok called this the
      production hit, and it is the strongest argument of the three.

    The blast radius is the whole popen family, since
    GenericCreatePipeAndFork() is the single fork behind cf_popen,
    cf_popen_select, cf_popensetuid, cf_popen_sh, cf_popen_sh_select,
    cf_popen_shsetuid and cf_popen_full_duplex — execresult(), package list
    commands, ps, getent, monitord pipes, custom modules, cf-execd's own shell
    around cf-agent.

    My stated reason for doing it unconditionally — not wanting the timeout path
    to be structurally different from the normal path — was backwards.
    The
    timeout path is different: it is the only one that has to kill a tree. The
    normal path has to stay reachable by the things that already kill trees.

    Fix — commit 847373cf6, same branch

    The process group is now created only when a timeout is armed, which is
    exactly when TimeOut() may need to sweep it. SetTimeOut() records that,
    ClearTimeOut() and TimeOut() clear it. The four call sites that open-coded
    alarm(0); signal(SIGALRM, SIG_DFL); now call ClearTimeOut(), so a command
    that completes in time cannot leave the flag set for the next, unrelated child.
    Those are the only SetTimeOut() users in the tree (verify_exec.c, nfs.c,
    cf-monitord/history.c).

    Verified after the fix:

    case before series this branch now
    sh -c 'read x', no timeout, under a pty 0.1s (stock) 0.1s — hang gone
    sh -c 'sleep 30; exit 0', exec_timeout => "2" 30.3s, sleep orphaned 10.8s, no orphan

    (10.8s rather than ~4.4s because this branch alone does not include #4's
    deadline fix; merged with it, 4.7s.)

    Reviewer answers to the other two open questions

    • The unconditional group SIGKILL — all three said acceptable on the
      timeout path, and the shared-function argument is true: putting group
      semantics into GracefulTerminate() would let locks.c's stale-lock reaper
      kill(-agent) and take down an entire execd generation. The sweep belongs in
      TimeOut() where it is.
    • PID recycling — my claim that reading pgid == pid before termination
      "closes" a recycle race is wrong, and two reviewers said so independently.
      It closes a different, real bug: getpgid() returns ESRCH once the leader is
      dead, so reading first is mandatory or the sweep never runs. It does not bind
      the pid against reuse in the window between GracefulTerminate() returning
      and kill(-pid, SIGKILL). The window is a handful of syscalls; neither
      reviewer would block on it; neither would describe it as closed. Corrected
      here rather than left in the commit message.

    Still outstanding on this branch

    • setpgid() / getpgid() return values are still unchecked. On failure the
      sweep is silently skipped and the patch becomes a no-op for that child. All
      three reviewers asked for a log line; not yet done.
    • Two reviewers suggested the parent should also call setpgid(pid, pid) (the
      POSIX both-sides pattern), since a timeout firing before the child reaches
      its own setpgid would skip the sweep. Unlikely at exec_timeout >= 1s;
      not yet done.
    • No acceptance test. CONTRIBUTING.md asks for one and the sh -c 'sleep 30'
      shape is exactly that. Not yet done.

    Tests: process_terminate_unix_test 6/6; full tests/unit suite all 68 as
    expected with 4 expected failures, identical to baseline.

  2. djbclark commented on Aug 17, 2026

    @djbclark
    OwnerAuthor

    Now tracked upstream in Jira as CFE-4729.

    This item was reported to security@northern.tech on 2026-08-17. The Jira ticket carries the corrected claims from the review panel, not the original wording. The unconditional-setpgid regression and the PID-recycling correction are both recorded there.

    This fork issue stays as the working record; CFE-4729 is the upstream reference. Going forward we file upstream items in the CFE Jira project rather than by email or GitHub Discussion.

  3. djbclark commented on Aug 22, 2026

    @djbclark
    OwnerAuthor

    Upstream disposition: closed

    Filed as CFE-4729, now Rejected / Done. Upstream PR cfengine/core#6305 closed 2026-08-19T10:45:01Z on review volume.

    Ticket-only content worth keeping

    This item got the only substantive maintainer code review of the whole 26. nickanderson reviewed cfengine#6305 on 2026-08-18 with one inline note on cf-agent/verify_exec.c: the added comments are too heavy, terse ones would do.

    Trimmed across the whole diff as 0e06ad3d7 — 37 lines removed, comment volume roughly halved, keeping only the non-obvious constraints (the sampling order after cf_pclose, the pgid guard, and the SIGTTIN reason for the TimeOutIsArmed condition). Verified comment-only: with comments stripped, the token stream is byte-identical to the previous head in all four files. Clean rebuild, timeout_test 6/6.

    That feedback generalises past this one diff and is worth applying to everything on this fork.

    Branch naming: the shipped fix is on fix/timeout-process-group-merged (merged with the CFE-4726 reporting fix), not the fix/timeout-process-group the ticket named.

  4. djbclark commented on Nov 1, 2026

    @djbclark
    OwnerAuthor

    Reproduces on upstream/master 5ac9805. Fresh, smaller branch fix/timeout-process-group-r2 at 3dc774b (3 files + acceptance test; the old -merged branch bundled 11 commits): acceptance timeout_kills_descendants.cf fails without the fix, passes with it; unit check 78/78. Not yet a PR.

  5. djbclark commented on Nov 1, 2026

    @djbclark
    OwnerAuthor

    Rework after review: fix/timeout-process-group-r3 (7b1d260): agent SIGTERM/SIGINT now passed on to the child's group; follow-up limits in #76. Not a PR yet.

  6. djbclark commented on Nov 1, 2026

    @djbclark
    OwnerAuthor

    Reworked as fix/timeout-process-group-r4 (1fdb294), a linear series on upstream/master carrying #83, #84 and #69 as its first commits; land after those. Changes from -r3: the agent's SIGTERM/SIGINT handler kills the timed child's group with SIGKILL instead of forwarding the signal (a command ignoring SIGINT survived -r3), and the own-group record replaces getpgid(). Details in #76.

  7. djbclark commented on Nov 1, 2026

    @djbclark
    OwnerAuthor

    Reworked as fix/timeout-process-group-r5 (b0ee8fd), a linear series on upstream/master carrying #83, #84 and #69 as its first commits; land after those. Change from -r4: the agent's SIGINT/SIGTERM handler gives the timed child's group master's graceful stop (the signal it got, SIGTERM after 1 s, SIGKILL after 3 s) instead of SIGKILL at once. Details in #76.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions