Repository navigation
Conversation
DiffPkgOperations() now renders its messages from the reduced records at print time, so that the reduction can be reused by other renderers. Changelog: None Signed-off-by: Daniel JB Clark <djbclark@gmail.com> Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
7 tasks
djbclark
force-pushed
the
simulate-json-v2
branch
2 times, most recently
from
September 30, 2026 13:28
313e9b7 to
4c7d20e
Compare
This was referenced Sep 30, 2026
With --simulate-json=FILE, cf-agent also writes the files, renames and net package operations of a --simulate run to FILE as one JSON document. Changelog: Title Signed-off-by: Daniel JB Clark <djbclark@gmail.com> Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
djbclark
force-pushed
the
simulate-json-v2
branch
from
September 30, 2026 15:03
4c7d20e to
480acaf
Compare
This was referenced Sep 30, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.


Fork-internal PR, used to run upstream's CI before anything goes upstream. Tracks #3.
Two commits, split as larsewi asked on cfengine#6332:
09d8d9f1fShared the reduction of recorded package operations (a refactor with no output change)4c7d20e1aAdded--simulate-jsonThe UTF-8 path workaround is on the fork-only branch
simulate-json-utf8. The proper fix belongs in libntech's JSON encoder and is being prepared separately.Checked on macOS and in an Ubuntu 24.04 arm64 container:
29_simulate_mode/simulate_json.cfpasses.ELI5 — auditable without C
You know shell and a little Python. Five facts carry the whole argument, and a sixth note covers non-ASCII file names. Each fact can be checked on its own. If any one is wrong, the argument collapses, so that's where to attack it.
What the code is for.
cf-agent --simulateruns the policy against a scratch copy of the files it touches (the "changes chroot"), not the real system. It keeps a log of what it did and, at the end, prints a report for humans. Commit 2 adds--simulate-json=FILE, which writes the same change set as one JSON document for programs to read. Commit 1 only prepares for that: it lets the JSON writer reuse the package part of the report.Fact 1 — commit 1 changes what the package summarizer remembers, not what it decides. The summarizer reads the package log and keeps two dictionaries,
installedandremoved. Before, each entry held a finished sentence. After, it holds the three facts the sentence is made from, and the same unchanged function builds the sentence at print time:The decision logic (which log line cancels which) isn't edited. Inside the loop, the only changed lines are the record's type name and the lines that build a record. Outside it, the "no package operations" log line moved to the caller. The only field the decisions ever read is the version, and both versions store a copy of the same value. So the output could only change if
sentence()got different inputs, and it gets copies of the same three values. This was also checked by running it: old and new code were given the same 2000 random package logs, covering all four operation kinds. The printed output was byte-identical.Fact 2 — the JSON reads only what the text report reads. Simulate mode already leaves three logs in its scratch area: the paths it changed, the renames it did (old/new pairs) and the package operations. The JSON is built from those logs plus the scratch copies:
pathchangedeletedif the scratch copy is gone. Otherwisecreatedormodified, depending on whether the real path exists. The text report makes the same two checks.type,permissions,uid,gid,size,sha256,targetstat/readlinkwould, without following linksrenamespackages--simulate=diffprintsSo the JSON can't know anything the text report doesn't. It also inherits the report's blind spots (see the end). The document is the same in all three
--simulatemodes except for itssimulate_modefield.Fact 3 — FILE only ever changes through
rename(). In Python terms, the writer does this:Three properties follow. Each comes from the standard meaning of these system calls, not from anything CFEngine-specific.
O_EXCLmakes the create fail if anything already has that name, including a symlink. So the temp file is always a new file that cf-agent made, never something planted.rename(). A failed or killed write leaves FILE exactly as it was, at worst with a stray*.cfnewfile next to it.rename()replaces the name FILE in one step. Readers see either the old file or the complete new one, never half of one. If FILE was a symlink, the link itself is replaced and its target is never opened.The six digits aren't a security measure. They come from C's
rand(), which is predictable. The protection isO_EXCL: a name collision makes the run fail, it never overwrites.Fact 4 — any failure to write the whole document makes cf-agent exit non-zero, with one deliberate exception. The document is built in memory first. If any of the three logs can't be read, nothing is written. Failing to create, write, close or rename the temp file also counts as a failure. In every one of those cases cf-agent exits 1, unless it was already going to exit with some other non-zero code, which it keeps.
The deliberate exception is a file whose SHA-256 can't be computed. It is written without a
sha256key, an error is logged, and the exit code doesn't change. Leaving the key out is safer than writing zeros. The hashing function has no way to report failure; it just leaves its output all zeros. Written out,"sha256": "0000…0000"looks exactly like a real digest, so a program comparing digests would get a wrong answer with full confidence. A missing key can't be mistaken for anything else. (The code treats an all-zero digest as failure. The chance of a real file hashing to all zeros is 2⁻²⁵⁶.)The document is written just before cf-agent's shutdown routine, because that routine shuts down the OpenSSL library that computes SHA-256.
Fact 5 — a symlink's permission bits carry no information. What you can do through a symlink depends on the target's permissions, not the link's. Linux reports every symlink as
0777. macOS reports whatever the umask gave it. Including them would make the same change look different on two platforms while meaning nothing on either. So symlinks gettargetinstead ofpermissions.Fact 6 (note) — non-ASCII names come out escaped byte by byte because of libntech, not this change. A file name is bytes, but a JSON string is characters:
libntech's
JsonWrite()escapes each byte outside printable ASCII on its own, as\u00XX. In JSON, though,Ãmeans "the character U+00C3" (Ã), not "the byte 0xC3". Socaféis written ascafé, and a standard parser such as Python's reads backcafé. The existingstorejson()policy function produces the same escape, which shows the cause isn't this change. CFEngine's ownreadjson()reverses the escape and gets the right bytes back. The document is pure ASCII, so it's always valid JSON; only non-CFEngine readers misread non-ASCII names. A separate fix to libntech's encoder is being prepared.What is not guaranteed.
fsync, so the all-or-nothing replacement in Fact 3 protects against other programs and a killed cf-agent, not against a power loss.failsafe.cf, and the JSON then describes that run.rel -> modify-meis reported, in both the text and the JSON, as target/modify-me, while a real run createsmodify-me.uid/gidshow the user who ran it, because cf-agent can'tchownits scratch copies.manifest-full. That mode's listing of unchanged files is not in the JSON.How you'd catch me lying, without reading C.
a. Commit 1: build the old and new code, give both the same random package logs, and
cmpthe output. If Fact 1 were wrong, some log would print differently.b. The 11 unit tests (
tests/unit/simulate_mode_test) include these two cases:The acceptance test
29_simulate_mode/simulate_json.cfruns a real cf-agent and compares the whole document with an expected file.c. Point FILE into a read-only directory that already holds an old FILE. You get exit 1, the old contents are intact, and no
.cfnewfile is left.d. Make FILE a symlink to a victim file. You get exit 0, the victim is unchanged, and FILE is now a regular 0600 file.
e. Without root, have a policy create a file and then set it to mode
000, so cf-agent can't read its own scratch copy. The entry has nosha256, the log says "Failed to compute the SHA-256 digest", and the exit code is 0.f. Run
printf created | shasum -a 256. It matches thesha256ofcreate-mein the acceptance test's expected output.g. Create a file named
caféand read the JSON with Python'sjson. You getcafé.storejson()on the same string gives the sameé.Checks c through g were run against an installed build of
4c7d20e1a. If Fact 3 were wrong, c or d would show a changed victim or a clobbered FILE. If Fact 4 were wrong, e would show zeros or a wrong exit code.🤖 Generated with Claude Code