Skip to content

Terminated the whole process group of a command that exceeded exec_timeout - #87

Open
djbclark wants to merge 49 commits into
masterfrom
fix/timeout-process-group-r5
Open

djbclark wants to merge 49 commits into
masterfrom
fix/timeout-process-group-r5

Conversation

@djbclark

@djbclark djbclark commented Nov 1, 2026 •

Copy link
Copy Markdown
Owner

When exec_timeout fired only the process cf_popen() had started was terminated, and what it had spawned kept the output pipe open, so the agent stayed blocked reading from a command it had given up on. A child started under a timeout now leads a process group of its own, recorded when it is forked (getpgid() cannot be asked later: on macOS it fails for a group leader that has exited and not been reaped), and the group is killed after the child is terminated; a command started without a timeout stays in the agent's group, and its group is never killed, even if it made one for itself with setsid(1). Such a child no longer receives what is sent to the agent's group, and nothing would end it when the agent exits (the alarm dies with the agent). On master the command got a graceful stop in two ways: from cf-execd's ladder (ProcessSignalTerminate): SIGINT, 1 s, SIGTERM, 5 s, SIGKILL, sent to the agent's group; and, for an interactive Ctrl-C from a root shell, directly from the terminal, since the agent stays in the terminal's foreground group when setsid() fails. So the agent's SIGINT and SIGTERM handler now gives the command's group a stop of its own: it passes on the signal it received, adds SIGTERM after 1 s if that was SIGINT, and sends SIGKILL after 3 s, sooner as soon as the group is empty (it reaps the leader, which counts as a member while it is a zombie), so a command that ignores both signals still dies with the agent (as under cf-execd on master; an interactive Ctrl-C now ends it too, where master left it running). The wait ends well before cf-execd would kill the agent (6 s after its SIGINT), a second signal during it (cf-execd's SIGTERM) neither extends it nor repeats the forward, and only kill(), waitpid() and nanosleep() are called from the handler. A command that needs more than 2 s after cf-execd's SIGTERM (3 s if the agent itself is sent SIGTERM) loses master's 5 s, Ctrl-C holds the agent for up to 3 s (a second Ctrl-C is blocked meanwhile), a lock-expiry kill of a stale agent (SIGINT, 1 s, SIGTERM, 1 s, SIGKILL) can SIGKILL the agent before the group is finished and skip its cleanup (#76, #88), and kill <agent pid>, which on master left the command alone, now ends it. SIGINT and SIGTERM are blocked from the fork until the child is published, so one arriving in between is not lost (untested: the window is microseconds), and cf_pwait() no longer leaves ALARM_PID naming a child it could not wait for. Land after #83 (fix/alarm-pid-reaped-r3), #84 (fix/exec-timeout-prefork-race-r3) and #69 (fix/exec-timeout-output-closed-r2): this is a linear series on upstream/master whose first three commits are signed-off cherry-picks of those branches, to drop on rebase, and without #69 cf_pclose() clears ALARM_PID before waiting, so the handler finds no child. A signal can still miss a child that a shell is in the middle of forking, as with any group signal (#76). On head b0ee8fd make -C tests/unit check passes 80 of 80 programs on Linux (Ubuntu 24.04 aarch64, --enable-debug, root), timeout_test 12 of 12 and 08_commands/04_exec_timeout/ 2 of 2; on macOS arm64 timeout_test passes 12 of 12. Reverting signals.c fails all four agent tests; sending SIGKILL at once, as the previous revision did, fails the three that need the command to handle a signal; dropping the SIGTERM step, the leader reaping or the re-entry guard each fails the test for it.

Fixes #5

Adversary review (omp Claude Opus 5.5, read-only; r3 and r4 heads were BLOCKed, this rework checked finding by finding): APPROVE, no P0-P2. P3 notes, all disclosed above or in #76/#79/#88: the SIGTERM budget cut from 5 s to 2 s; the lock-expiry kill skipping cleanup; the handler reaps the leader without clearing ALARM_PID (negligible race); two test timing margins (min 800 ms); ALARM_PID is not sig_atomic_t.

🤖 Generated with Claude Code

victormlg and others added 30 commits October 28, 2026 12:55
Added the reactor tests to jenkins, to be able to run them in jenkins. This
means we can test cf-reactor on all the different platforms.

Signed-off-by: Victor Moene <victor.moene@northern.tech>
Made the logs prettier and added a log file where the xtrace is
printed

Signed-off-by: Victor Moene <victor.moene@northern.tech>
The guest environment cannot contain "-". On hosts containing "-", the
promise fails. The host name is now canonofied.

Signed-off-by: Victor Moene <victor.moene@northern.tech>
X509_get_subject_name() now returns a const name. Build a new
X509_NAME and set it as both subject and issuer instead.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Signed-off-by: Lars Erik Wik <lars.erik.wik@northern.tech>
Tests that print Pass message but have logs errors will fail with `FAIL (Error printed)` message.
Tests which are expected to log errors should use `.error.cf` or `.error.pass.cf` suffix. see: `tests/acceptance/README`

Ticket: CFE-3041
Signed-off-by: Ihor Aleksandrychiev <ihor.aleksandrychiev@northern.tech>
These tests read files created by their own init bundle, but pre-evaluation calls the read functions before the files exist, logging "Could not examine file" errors which now fail regular tests.

Ticket: CFE-3041
Signed-off-by: Ihor Aleksandrychiev <ihor.aleksandrychiev@northern.tech>
Errors printed by the command were reported on lines starting with "error:", so testall mistook them for errors of the test itself.

Ticket: CFE-3041

Signed-off-by: Ihor Aleksandrychiev <ihor.aleksandrychiev@northern.tech>
These tests read files and directories created by their own policy, but pre-evaluation calls the functions before they exist, logging errors which now fail regular tests.

Ticket: CFE-3041
Signed-off-by: Ihor Aleksandrychiev <ihor.aleksandrychiev@northern.tech>
The agent loads def.json from the directory of the input policy file, so tests in 00_basics/ tried to read this directory as their augments file and logged "Could not open JSON file" errors.

Ticket: CFE-3041
Signed-off-by: Ihor Aleksandrychiev <ihor.aleksandrychiev@northern.tech>
The test used the 'immediate' body from plucked.sub.cf before it was
loaded during pre-evaluation, so the promise is now skipped until the
test framework is loaded.

Ticket: CFE-3041
Signed-off-by: Ihor Aleksandrychiev <ihor.aleksandrychiev@northern.tech>
The test referred to a promise module which never existed and did not run its init and cleanup bundles.

Ticket: CFE-3041
Signed-off-by: Ihor Aleksandrychiev <ihor.aleksandrychiev@northern.tech>
The check bundle counted lines in files which do not exist until the test bundle has run, but the agent evaluates the functions of all promises before any bundle runs.

Ticket: CFE-3041
Signed-off-by: Ihor Aleksandrychiev <ihor.aleksandrychiev@northern.tech>
wrong-bundle-type.cf printed the sub-agent output in its DEBUG report, so the sub-agent errors were mistaken for errors of the test, and ifelse.cf used invalid class expressions as false conditions.

Ticket: CFE-3041
Signed-off-by: Ihor Aleksandrychiev <ihor.aleksandrychiev@northern.tech>
These tests intentionally exercise error paths, the reason is documented in the header of each test.

Ticket: CFE-3041
Signed-off-by: Ihor Aleksandrychiev <ihor.aleksandrychiev@northern.tech>
Signed-off-by: Ihor Aleksandrychiev <ihor.aleksandrychiev@northern.tech>
Ticket: CFE-3041
Signed-off-by: Nick Anderson <nick@cmdln.org>
Ticket: CFE-3041
Signed-off-by: Nick Anderson <nick@cmdln.org>
The errorpass branch only checked for logged errors and the Pass
message, so a test marked test_soft_fail or test_skip_unsupported was
reported as FAIL (e.g. process_select_fncalls.error.pass.cf on Windows,
which soft fails due to ENT-10217 and logs no errors there). Errorpass
tests now go through the regular outcome handling and only differ in
requiring at least one logged error on Pass.
Ticket: CFE-3041
Signed-off-by: Nick Anderson <nick@cmdln.org>
Signed-off-by: Dhanraj E <dhanraj@labs.digiscrypt.com>
Signed-off-by: Lars Erik Wik <lars.erik.wik@northern.tech>
Signed-off-by: Lars Erik Wik <lars.erik.wik@northern.tech>
…ptions() public

Signed-off-by: Lars Erik Wik <lars.erik.wik@northern.tech>
Signed-off-by: Lars Erik Wik <lars.erik.wik@northern.tech>
Signed-off-by: Lars Erik Wik <lars.erik.wik@northern.tech>
Signed-off-by: Lars Erik Wik <lars.erik.wik@northern.tech>
The test promised the current user with policy "present" and expected
the promise to be kept. For an existing user, "present" also means
"not locked". When run as root, cf-reactor can read /etc/shadow, and if
the account's password hash starts with '!' (as root's does on some
systems), it is considered locked and gets unlocked. The promise was
then repaired instead of kept, and the test modified the root account.
As a normal user, /etc/shadow is unreadable, so the account was
considered unlocked and the test passed.

Instead, we modify the policy to always return "kept"

Signed-off-by: Victor Moene <victor.moene@northern.tech>
Signed-off-by: Lars Erik Wik <lars.erik.wik@northern.tech>
Signed-off-by: Lars Erik Wik <lars.erik.wik@northern.tech>
larsewi and others added 19 commits October 29, 2026 13:05
Signed-off-by: Lars Erik Wik <lars.erik.wik@northern.tech>
…be created

Signed-off-by: Lars Erik Wik <lars.erik.wik@northern.tech>
…per.c

Signed-off-by: Lars Erik Wik <lars.erik.wik@northern.tech>
Signed-off-by: Lars Erik Wik <lars.erik.wik@northern.tech>
Signed-off-by: Lars Erik Wik <lars.erik.wik@northern.tech>
Signed-off-by: Lars Erik Wik <lars.erik.wik@northern.tech>
Signed-off-by: Lars Erik Wik <lars.erik.wik@northern.tech>
ENT-14766: Prepared cf-reactor event types for package events
ENT-14766: Moved package module wrapper from cf-agent to libpromises
Ticket: ENT-14765
Changelog: `cf-reactor` can now react to a user being added or removed, with the `user_added` and `user_removed` attributes of `when` bodies
Signed-off-by: Lars Erik Wik <lars.erik.wik@northern.tech>
CFE-3041: Made testall fail regular acceptance tests which log errors
…Jenkins

Added reactor tests to make check
ENT-14765: cf-reactor: Added user_added and user_removed events
…-path-overflow

fix stack buffer overflow in SaveAsFile backup path construction
cf_pwait() and ShellCommandReturnsZero() reaped their child but left
ALARM_PID naming it, so a later alarm could signal an unrelated process
that had been given the recycled pid. RepairExec() also left its
exec_timeout alarm armed when cf_popen*() failed, so it could fire during
the next command. Both are now cleared. ShellCommandReturnsZero() also
clears ALARM_PID when waitpid() itself fails.

The new unit test builds pipes_unix.c into its own binary, so that its
fdopen() override reaches cf_popen*() on macOS as well.

Changelog: Title
Ticket: none
Signed-off-by: Daniel JB Clark <djbclark@gmail.com>
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
SetTimeOut() armed the alarm at once, but cf_popen() publishes the pid of
the child in ALARM_PID only after fork(). If the alarm expired in between,
for example because the agent was stalled by a heavily loaded host,
TimeOut() found nothing to terminate and the command then ran without
any limit.

The new SetChildTimeOut() only records the timeout and cf_popen*() starts
the clock right after publishing the pid; RepairExec() and the mount
commands in nfs.c use it. SetTimeOut() still arms at once, for callers
that do not fork. A request is good for one cf_popen*() call, and the new
DisarmTimeOut() drops it, so an abandoned request cannot start the clock
for an unrelated child.

Changelog: Title
Ticket: none
Signed-off-by: Daniel JB Clark <djbclark@gmail.com>
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
cf_pclose() cleared ALARM_PID before waiting for the child. A command
that closes its stdout and stderr but keeps running ends the read loop at
EOF and is already in that wait when the alarm fires, so TimeOut() found
no pid and signalled nothing. ALARM_PID is now cleared after the reap.

Changelog: Title
Ticket: none
Signed-off-by: Daniel JB Clark <djbclark@gmail.com>
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
…meout

When exec_timeout fired only the process cf_popen() had started was
terminated. Anything it had spawned survived and kept the output pipe
open, so the agent stayed blocked reading from a command it had given up
on until the survivor exited. A child started under a timeout now leads its
own process group, noted when it is forked, and the group is killed after
the child has been terminated. Children started without a timeout stay in
the agent's group, and a group is never killed for them.

A child in a group of its own no longer dies with the agent when cf-execd
signals the agent's group, so the agent's SIGTERM and SIGINT handler ends
that group before it exits: it passes on the signal it got, adds SIGTERM
after 1 s if that was SIGINT, and sends SIGKILL after 3 s, sooner once the
group is empty. That is the command's graceful stop on master, and ends
well before cf-execd would kill the agent (6 s after its SIGINT). A second
signal during the wait does not extend it. SIGINT and SIGTERM are held
from the fork until the child is published, so that one arriving in
between is not missed. cf_pwait() no longer leaves ALARM_PID naming a
child it could not wait for.

Changelog: Title
Ticket: none
Signed-off-by: Daniel JB Clark <djbclark@gmail.com>
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
@chatgpt-codex-connector

chatgpt-codex-connector Bot commented Nov 1, 2026 •

Copy link
Copy Markdown

Codex Review Summary

This comment shows the latest Codex review activity on this pull request.

Review Status Commit Review trigger
📝 Code Review ✅ Completed 2026-11-01T21:27:24.453410Z b0ee8fd PR opened
🔒 Security Review ✅ Completed 2026-11-01T21:27:45.476378Z b0ee8fd PR opened
ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review" or "@codex security review".

Codex reacts with 👀 while any review is running, comments if it has suggestions, and reacts with 👍 once all reviews finish with no findings.

@chatgpt-codex-connector chatgpt-codex-connector Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: b0ee8fd166

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

Comment thread libpromises/pipes_unix.c
Comment on lines +268 to +270
else if (own_group)
{
setpgid(pid, pid); /* the child may not have run yet */

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

P2 Badge Forward shutdown signals to timed commands in cf-reactor

When foreground cf-reactor is executing a command promise with exec_timeout, moving that command into its own process group prevents it from receiving terminal SIGINT or a shutdown signal sent to the reactor's process group. The new forwarding only runs in HandleSignalsForAgent, whereas cf-reactor/cf-reactor.c:318-319 installs HandleSignalsForDaemon, which merely sets PENDING_TERMINATION. Since reactor bundles execute commands through ScheduleAgentOperations and RepairExec does not check that flag while reading command output, shutdown now waits until the command finishes or its timeout expires instead of interrupting it. Extend the timed-group shutdown handling to the reactor's termination path as well.

Useful? React with 👍 / 👎.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

PR 5: Signal the whole process group when a command times out

7 participants