Skip to content

PR 2: Machine-readable (JSON) rendering of the --simulate change set #3

Open

Description

@djbclark

Status: working document on a personal fork. Not an upstream submission. This
is PR 2 of two; see #1 for the investigation this came out of, and #2 for the
companion chroot-retention change. Both are intended to go upstream under a single
Jira ticket describing the pair.

Explain it like I'm 5

Imagine you're about to let someone make changes to your computer, and you ask a
careful friend to go check what those changes would actually do. Your friend comes
back and hands you a note:

I'd change three files in your settings folder, and install one new program
called caddy.

Perfect. For a human being, that note is exactly right.

Now imagine you want a robot to check the note instead of you — because the note
might be four hundred lines long, because you're tired, because you're going to
read a note like this every week for years and eventually you'll skim one. The
robot's job is to confirm that nothing got quietly added that you didn't agree to.

But the robot has to read your friend's handwriting. And your friend writes it a
little differently every time — sometimes "I'd change", sometimes "changes to",
sometimes a helpful extra sentence. One day your friend rephrases something, the
robot misreads the note, and it waves through a change nobody approved. The robot
was never given anything solid to hold onto.

This change asks CFEngine to also hand over the same information as a filled-in
form with labeled boxes — path here, kind-of-change there, package name in this
box — instead of only as a paragraph. Same facts, same friend, same trip. One is
prose written for a person; the other is a form written for a program.

Why anyone would want that: the moment a program can read the answer reliably,
you can build things on top of it. Show a person a clean summary of exactly what
would change. Refuse to proceed if anything appears that wasn't in what they
approved. Keep a signed record of what was agreed to. Have CI fail when a change
touches something it shouldn't. None of that is safe to build on text that was
written to be read by a human and can be reworded at any time.

What is wanted

A machine-readable rendering of the change set --simulate already computes:
changed files, renames, and package operations, as JSON.

Proposed surface, to be confirmed with maintainers first:
--simulate-output=json, orthogonal to the existing
--simulate=<manifest|manifest-full|diff> mode selector.

Why today's output can't be parsed instead

cf-agent/simulate_mode.c renders with printf() throughout — '%s' is a %s,
'%s' no longer exists, Package '%s-%s [%s]' would be %s — and for
--simulate=diff it shells out to diff -u (RunDiff(), simulate_mode.c:360)
and passes the result through.

Parsing that back into structure is possible and unwise: it is a stat(1)-shaped
text format with no stability contract, and any consumer doing it is one wording
change away from silently misreading a security-relevant artifact. Note that
tests/acceptance/29_simulate_mode/ already carries a
normalize_agent_output.cf.sub — the project's own tests need output
normalisation to be comparable at all, which is itself an argument for structured
output.

Why this is cheaper than it looks, as of 17eb78e

  • The data already exists and is already structured. Four record files are
    written during a simulate run (libpromises/changes_chroot.h:28-31). Package
    operations are already a four-field CSV row with an operation code from a
    defined enum (ChrootPkgOperationCode, cf-agent/simulate_mode.h:38-43).
  • The renderer already runs before cleanup. cf-agent.c:392-428 dispatches on
    EVAL_MODE and calls the manifest/diff renderers; only afterwards, at
    cf-agent.c:435, does CallCleanupFunctions() tear down the chroot. A JSON
    renderer slots in beside the existing prose ones with all the same data live.
  • The serializer is already linked in. libntech/libutils/json.h provides
    JsonObjectCreate(), JsonArrayCreate(), JsonObjectAppendString(),
    JsonWrite(), and libntech is already a submodule — so this is a single-repo
    change
    , avoiding the coordinated multi-repo PR that CONTRIBUTING.md
    otherwise requires.

The prose renderers stay untouched and remain the default.

What this does not get, and must not claim

  • Files and packages only. Service activation is not simulated. commands
    promises are not executed — see the still-open
    CFE-3548.
  • No runtime confinement. Simulate mode computes a would-be change set; it is
    not a capability sandbox and nothing here makes it one.
  • --simulate=manifest is stat(1) prose. ManifestStatInfo() prints
    size/mode/uid/gid/atime/mtime/ctime. Structuring that is straightforward but is
    distinct work from structuring the change list.

Open questions — deliberately not answered on paper

These get decided against a running prototype, not by reasoning. An earlier note
in this project reasoned about simulate mode from source alone and got the chroot
lifetime wrong; the correction only came from running cf-agent.

  1. Destination. stdout, interleaved with Log() output? A separate
    --simulate-output-file=PATH? Anything on stdout must survive log interleaving
    to be parseable.
  2. Schema. What does a changed-file entry contain — path, change kind, old and
    new stat, content hashes, the unified diff as a string, or a structured hunk
    list? The security-relevant question downstream is whether content is committed
    to by digest.
  3. Diff representation. Keep the diff -u shell-out and embed its text, or
    restructure? Embedding is far cheaper and probably the right first cut.
  4. Stability contract. Supported interface with a version field, or explicitly
    unstable? Maintainers will care about this more than about the shape.

Definition of done

  • Surface agreed with maintainers before implementation
  • JSON renderer beside the existing prose renderers; prose unchanged and still
    the default
  • Unit tests for the new C functions
  • Acceptance tests in the style of tests/acceptance/29_simulate_mode/
  • Built and run for real against a policy with file and package promises
  • clang-format clean; matches the local idiom of simulate_mode.c
  • Changelog: and Ticket: trailers in the commit message

Activity

  1. djbclark commented on Aug 15, 2026

    @djbclark
    OwnerAuthor

    Finding: --simulate=diff depends on a diff binary in CFEngine's own bindir

    Found by running the built agent, not by reading source. On a real run, diff mode failed:

    info: Showing diff for changed files (in the changes chroot)
    ===========================================================================
    error: Couldn't run '.../workdir/bin/diff'. (execv: No such file or directory)
    ===========================================================================
    

    The cause is RunDiff() at cf-agent/simulate_mode.c:363-364:

    strncpy(diff_path, GetBinDir(), sizeof(diff_path) - 1);
    JoinPaths(diff_path, sizeof(diff_path), "diff");

    It resolves diff from GetBinDir(), not $PATH — CFEngine's own bin directory. Neither a make install tree nor the Homebrew cfengine package puts a diff there, so --simulate=diff fails out of the box on macOS while --simulate=manifest works fine. (Whether the Linux packages ship one is unverified and worth checking before saying anything upstream — the finding is about the lookup path, not about any particular package being broken.)

    Why this matters for this ticket, in both directions:

    • It strengthens the case for structured output. The existing diff mode has an external-binary dependency resolved from a non-standard location, and its output is then whatever that particular diff implementation emits — flags, labels, and all. That is a second stability problem layered on the first.
    • It complicates open question 3 (embed the diff -u text vs. restructure). Embedding is still the cheap first cut, but embedding means the JSON inherits the external dependency and the failure mode above. A structured representation computed in-process would not. Worth putting to maintainers explicitly rather than deciding quietly.

    Also relevant: a JSON mode that reports changed files without needing diff at all would make simulate genuinely useful on platforms where this currently silently degrades.

  2. djbclark commented on Aug 17, 2026

    @djbclark
    OwnerAuthor

    Now tracked upstream in Jira as CFE-4716.

    Jira became usable to us on 2026-08-17 — the Atlassian account alone wasn't enough, project-level permission was the missing piece. This fork issue stays as the working record; CFE-4716 is the upstream reference, and the two should be kept in step.

    Going forward we file upstream items in the CFE Jira project rather than by email or GitHub Discussion.

  3. djbclark commented on Aug 22, 2026

    @djbclark
    OwnerAuthor

    Upstream disposition: closed

    Filed as CFE-4716, now Rejected / Won't Do. Upstream PR cfengine/core#6294 closed 2026-08-19T10:45:41Z. Review volume, not the merits. One of the two features in the set of 26; the other 24 were resolved Done.

    Ticket-only content worth keeping — the UTF-8 interaction, tested for real

    Built a throwaway core-p2 worktree with its libntech submodule repointed at a merge of fix/json-double-decode + fix/mustache-minor-defects on top of 0c0620d (the codec fixes now tracked as djbclark/libntech#6 and #7), then ran tests/unit/simulate_mode_test plus a standalone probe isolating RestoreUtf8InJson()/GetJsonEscapedByte()/ValidUtf8SequenceLength() against the fixed JsonEncodeStringWriter(). Two different results:

    • Valid multi-byte UTF-8 (e.g. U+00E9): confirmed safe. The fixed encoder emits one escape per code point instead of one \u00XX per UTF-8 byte, so RestoreUtf8InJson() never collects the 2+ consecutive escapes it looks for and passes the text through unchanged — dead code for this case, not corruption. test_special_characters_in_path's failure under the fixed library is cosmetic: it asserts the old raw-byte output shape, and the parsed value itself is correct.

    • Invalid non-UTF-8 bytes (e.g. a lone 0xE9 filename byte): a real gap, but not the one the workaround guards against. A genuine U+00E9 code point and a lone invalid byte 0xE9 encode to the identical escape sequence; RestoreUtf8InJson() cannot tell them apart. test_invalid_utf8_in_path only ever passed because the old non-conformant parser and writer were exact inverses — the same "round-trips its own corruption" shape the codec issue itself describes — and cancelled out. Once the parser side is also conformant, the escape correctly decodes to the valid code point, which is not the original byte. A non-UTF-8-valid path byte cannot be represented exactly in a JSON string at all, conformant or not. Pre-existing representational gap the old test accidentally papered over, not a new defect.

    This needs a real design decision — replacement character, error out, or a separate raw/base64 field for the rare non-UTF-8 path — before test_invalid_utf8_in_path can be fixed for real, whenever the codec fix lands here and this branch is rebased onto it. Carried into the consolidated questions issue.

  4. djbclark commented on Aug 22, 2026

    @djbclark
    OwnerAuthor

    Open decision from this item is tracked in the consolidated list: #23

  5. djbclark commented on Sep 30, 2026

    @djbclark
    OwnerAuthor

    2026-09-30: upstream said yes; the split is ready on the fork.

    On cfengine#6332 (2026-09-23), larsewi wrote: "Feel free to open a PR. It would help if you split the work in logical parts." The rebased and trimmed version is fork PR #27, branch simulate-json-v2, with two commits:

    1. 09d8d9f1f Shared the reduction of recorded package operations (a refactor with no output change)
    2. 4c4919ede Added --simulate-json option to write the simulated change set as JSON

    Changes from the 08-15 version:

    1. The "install cancels removal" fix is gone. It is already upstream as CFE-4742: Fixed simulate mode reporting both removal and install of the same package cfengine/core#6332.
    2. Symlink entries no longer carry permissions. Symlink modes are meaningless: always 0777 on Linux, 0700 here on macOS. The Linux gate caught this.
    3. The acceptance test no longer depends on libxml2 (no XML promises).
    4. Comments, unit tests (+542 → +305 lines) and the acceptance expected file (158 → 50 lines) are trimmed.
    5. The UTF-8 path handling moved to the fork-only branch simulate-json-utf8 (b9a02866e), because it works around libntech's JsonWrite() escaping and belongs there.

    Results:

    1. Unit tests pass at each commit: 2/2, 11/11, then 13/13 with the UTF-8 commit.
    2. simulate_json.cf passes on macOS and in an Ubuntu 24.04 arm64 container.
    3. Fork CI is running on Added --simulate-json option to write the simulated change set as JSON #27.

    Open questions:

    1. A new Jira ticket, or no Ticket: trailer? CFE-4716 is Rejected/Won't Do.
    2. The second-opinion review must report before anything is sent.
  6. djbclark commented on Sep 30, 2026

    @djbclark
    OwnerAuthor

    Independent review of #27: no blockers. Fixes are folded into commit 2, which is now 4c7d20e1a. The UTF-8 commit on top of it is now 313e9b7ff.

    Verified by the reviewer:

    1. Commit 1 produced output byte-identical to upstream's over 2,000 random package-operation sequences.
    2. macOS leaks found 0 leaks across the error paths: corrupt, truncated or empty records, an output path that is a directory, an unreadable file.
    3. The O_EXCL + rename() security design holds up.

    Fixed:

    1. The temp file was <FILE>.new, which was deleted before writing, so an existing file of that name was silently removed. It is now <FILE>.<6 digits>.cfnew with O_EXCL, the same idiom as override_fsattrs.c, and nothing is deleted first.
    2. A failure was logged twice: main() repeated what WriteChangesJson() had already logged on every failure path.
    3. Permissions are formatted with %04o, matching the rest of the file.
    4. Removed the only Windows-only unlink() block in the tree.
    5. Fixed a leak when the option is given twice.
    6. Ran cfengine format on the acceptance test. Fork CI's format --check had failed on it, which skipped every workflow that depends on the lint job.

    Left as they are, with reasons:

    1. Non-ASCII paths are wrong in Added --simulate-json option to write the simulated change set as JSON #27 on its own. libntech JsonWrite() escapes each byte separately. simulate-json-utf8 fixes it; whether it goes upstream is an operator decision.
    2. A digest that can't be computed is logged at ERROR, but the exit code stays 0.
    3. No tests yet for corrupt records, rejected option combinations, or several packages at once.

    Tests: unit 2/2, 11/11 and 13/13 at each commit, and the acceptance test passes, on macOS and in the Ubuntu 24.04 container.

  7. djbclark commented on Sep 30, 2026

    @djbclark
    OwnerAuthor

    Decision (operator, 2026-09-30): the upstream PR carries commits 1 and 2 only, with no Jira ticket.

    Non-ASCII path handling will be offered upstream as a libntech encoder fix, not as a core workaround. The fix writes valid UTF-8 as-is and keeps escaping invalid bytes. It changes only the encoder, so CFEngine still reads its own output identically and no migration is needed. It is djbclark/libntech#9 (1a3b12fef, +77/−1 C plus a test). Results:

    1. libntech make check: 39/39 programs pass.
    2. Core unit tests built against it: 70 pass, plus the macOS XFAILs that are expected to fail.
    3. No core test depends on the old escaping.

    The fork-only core workaround stays on simulate-json-utf8.

    The ELI5 for the logic is in #27's description. A pre-existing relative-symlink target bug found while writing it is filed as #28.

  8. djbclark commented on Sep 30, 2026

    @djbclark
    OwnerAuthor

    Upstream plan: three PRs (decided 2026-09-30)

    On cfengine#6332, larsewi invited the PR "split in logical parts". After two independent reviews (Claude, plus a Fable workflow with adversarial verification) and a size analysis, the operator chose to send it as a sequence of small PRs. Tests may run a bit longer, since they carry the correctness case. This comment is the tracker and is edited as work lands.

    PR 0: two simulate mode package reporting fixes (pre-existing bugs), fork PR #39

    • dba2a1032: version and architecture were written swapped; regression test fails without the fix
    • 98006a443 (Pre-existing: --simulate=diff never reports a versioned removal with no prior install record #38): --simulate=diff omitted a removal with a version unless it cancelled a recorded install; regression test fails without the fix
    • Test placed so PR 0 merges cleanly with PR 1 (checked with git merge-tree)
    • macOS + Linux gate (3/3, 4/4); review with no blockers
    • Fork CI on the reworked commits

    PR 1: --simulate-json for files and renames, fork PR #40 (d5cc2c4e0, +716)

    • Option handling: IsAbsoluteFileName(); the dead optarg == NULL branch removed
    • Files: path, change, type, permissions (none for symlinks; MinGW-safe mask), uid, gid, size, sha256 (compact all-zero check), target
    • A file created and then deleted in the same run is not reported
    • Renames; one shared ReadChrootRecords() reader
    • failsafe_fallback (hard class only); simulate_mode field dropped
    • Atomic write (RAND_bytes() temp name, O_EXCL, rename()); write, close and rename failures each reported accurately
    • Unit tests 11/11 and acceptance test on macOS, including the option checks
    • MinGW: the mask checked with a scratch cross-compile. A full cross-compile needs the buildscripts toolchain.
    • Linux gate 11/11 + acceptance; final review with no blockers; failsafe lookup only with --simulate-json
    • Fork CI on the reworked commit

    PR 2: packages in the JSON, fork PR #41 (stacked on PR 1; full CI via #42)

    • Refactor 16a6b6567: PkgOperationRecord stores fields and both modes format at print time, with one shared print helper. Output identical over 12,314 generated logs.
    • Rebuilt on the new PR 1 with both PR 0 commits; the Pre-existing: --simulate=diff never reports a versioned removal with no prior install record #38 fix carried through the refactor (c1bf747ea; again identical output over 12,314 logs)
    • packages array 29fce4663, sorted by name then architecture; unit tests 15/15
    • Acceptance comment explaining the pre-existing "Error installing package" message
    • Linux gate at every commit; final and delta reviews with no blockers
    • Fork CI on the reworked commits

    Then

    Fork-only: simulate-json-utf8 rebased onto PR 1 (1b0fb89db, 13/13). Deferred items are in #28–#38 and #43. #27 is closed as superseded.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions