Repository navigation
Description
Migrated from CFE-4730 (Rejected 2026-08-19 on review volume). Upstream PR NorthernTechHQ/libntech#293 closed the same day. Branch fix/json-string-codec (commit 90cf8cc, cut from upstream master 0c0620d) is live on this fork.
libntech's JSON string codec is not conformant in both directions. Measured on libntech 5b5d04e1 (1.1.0-139-g5b5d04e), built from source on macOS 15, and reproduced through readjson() in a real policy.
Reading standard JSON silently corrupts the data
JsonDecodeStringWriter() (libutils/json.c:1108-1120) handles a \uXXXX escape by calling HexStringToChar(), which rejects any code point above U+00FF (:1063: (c > 255) returns false). On that failure the case 'u': arm simply breaks — without advancing the cursor past the escape and without emitting anything. The enclosing for (const char *c = ...; *c != '\0'; c++) then advances by one, landing on the u, which is copied literally along with the four hex digits. The backslash is dropped and the escape sequence becomes the data.
$ cat data.json
{"city": "中国", "cafe": "café"}
$ python3 -c "import json; print(json.load(open('data.json')))"
{'city': '中国', 'cafe': 'café'}
$ cf-promises -f p.cf --show-vars # "d" data => readjson("data.json", 4096)
default:main.city u4e2du56fd
default:main.cafe <non-printable>
default:main.d {"cafe":"café","city":"u4e2du56fd"}
中国 became the seven-character literal u4e2du56fd. Silent data corruption on input, reachable from anything that reads externally-produced JSON: readjson(), parsejson(), data containers, augments files, and CMDB host_specific.json — which is routinely generated by tooling that emits \uXXXX for non-ASCII.
The second line is the same defect's other half: an escape in U+0080–U+00FF is accepted, but written as a single raw byte, i.e. decoded as Latin-1. café should decode to 0xC3 0xA9; it produces the single byte 0xE9, which is not valid UTF-8 at all.
Writing mangles UTF-8
JsonEncodeStringWriter() (:1015-1022) escapes every byte that is not printable ASCII individually as \u%04x of the byte value. Well-formed JSON, but \u00XX denotes code point U+00XX, so a conformant parser decodes each byte of a multi-byte UTF-8 character as its own wrong character: café is emitted as café and read back by any standard parser as café.
Why the existing tests cannot catch either half
The two halves are exact inverses, so libntech round-trips its own output perfectly. Any test that writes with JsonWrite() and reads back with JsonParseFile()/JsonDecodeString() passes while both halves are wrong. Only a conformant third-party decoder, or an assertion on the raw bytes on disk, can see it. Testing the round trip through python3 -c 'import json; json.load(...)' rather than through CFEngine's own decoder is what surfaced this.
Fix
Both halves together, since each currently compensates for the other — fixing only the writer would make output libntech itself then misreads. The encoder escapes the code point rather than the byte, surrogate-pairing outside the BMP, keeping output pure ASCII and byte-identical for ASCII input; the decoder handles the full BMP, emits UTF-8, combines surrogate pairs, and substitutes U+FFFD for an unpaired surrogate or malformed escape instead of corrupting it into literal text.
Regression tests deliberately assert against values derived from a conformant implementation (python3's json module) rather than round-tripping through libntech's own codec.
Compatibility consequence — decision needed
Documents written by the old encoder no longer round-trip through the fixed library, because their per-byte escapes are now read per code point, as every conformant reader always did. See the consolidated questions issue.
Activity
djbclark commentedon Aug 22, 2026
Open decision from this item is tracked in the consolidated list: djbclark/core#23
djbclark commentedon Oct 9, 2026
Re-cut onto master 3c21479 as branch
claudehelm/libntech-6(0b0b91b, same change as 90cf8cc / fix/json-string-codec, with trailers). Chose the both-halves codec fix over PR #9's writer-only raw-UTF-8 variant because the reading half (\u4e2dcorrupting to literal text) is the data-loss case and PR #9 leaves it. Verified both ways: with libutils at master 3 of 72 tests fail ("u4e2du56fd" != "中国"); with the fix 72/72 pass, build 0 warnings. The old-encoder compatibility question (djbclark/core#23 item 3) is still the operator's call. Upstream PR body drafted, not posted. Left open for operator review (ClaudeHelm).