Skip to content

JSON string codec is non-conformant both ways: reading standard \uXXXX escapes silently corrupts data, writing mangles UTF-8 #6

Open

Description

@djbclark

Migrated from CFE-4730 (Rejected 2026-08-19 on review volume). Upstream PR NorthernTechHQ/libntech#293 closed the same day. Branch fix/json-string-codec (commit 90cf8cc, cut from upstream master 0c0620d) is live on this fork.

libntech's JSON string codec is not conformant in both directions. Measured on libntech 5b5d04e1 (1.1.0-139-g5b5d04e), built from source on macOS 15, and reproduced through readjson() in a real policy.

Reading standard JSON silently corrupts the data

JsonDecodeStringWriter() (libutils/json.c:1108-1120) handles a \uXXXX escape by calling HexStringToChar(), which rejects any code point above U+00FF (:1063: (c > 255) returns false). On that failure the case 'u': arm simply breaks — without advancing the cursor past the escape and without emitting anything. The enclosing for (const char *c = ...; *c != '\0'; c++) then advances by one, landing on the u, which is copied literally along with the four hex digits. The backslash is dropped and the escape sequence becomes the data.

$ cat data.json
{"city": "中国", "cafe": "café"}

$ python3 -c "import json; print(json.load(open('data.json')))"
{'city': '中国', 'cafe': 'café'}

$ cf-promises -f p.cf --show-vars        # "d" data => readjson("data.json", 4096)
default:main.city   u4e2du56fd
default:main.cafe   <non-printable>
default:main.d      {"cafe":"café","city":"u4e2du56fd"}

中国 became the seven-character literal u4e2du56fd. Silent data corruption on input, reachable from anything that reads externally-produced JSON: readjson(), parsejson(), data containers, augments files, and CMDB host_specific.json — which is routinely generated by tooling that emits \uXXXX for non-ASCII.

The second line is the same defect's other half: an escape in U+0080–U+00FF is accepted, but written as a single raw byte, i.e. decoded as Latin-1. café should decode to 0xC3 0xA9; it produces the single byte 0xE9, which is not valid UTF-8 at all.

Writing mangles UTF-8

JsonEncodeStringWriter() (:1015-1022) escapes every byte that is not printable ASCII individually as \u%04x of the byte value. Well-formed JSON, but \u00XX denotes code point U+00XX, so a conformant parser decodes each byte of a multi-byte UTF-8 character as its own wrong character: café is emitted as café and read back by any standard parser as café.

Why the existing tests cannot catch either half

The two halves are exact inverses, so libntech round-trips its own output perfectly. Any test that writes with JsonWrite() and reads back with JsonParseFile()/JsonDecodeString() passes while both halves are wrong. Only a conformant third-party decoder, or an assertion on the raw bytes on disk, can see it. Testing the round trip through python3 -c 'import json; json.load(...)' rather than through CFEngine's own decoder is what surfaced this.

Fix

Both halves together, since each currently compensates for the other — fixing only the writer would make output libntech itself then misreads. The encoder escapes the code point rather than the byte, surrogate-pairing outside the BMP, keeping output pure ASCII and byte-identical for ASCII input; the decoder handles the full BMP, emits UTF-8, combines surrogate pairs, and substitutes U+FFFD for an unpaired surrogate or malformed escape instead of corrupting it into literal text.

Regression tests deliberately assert against values derived from a conformant implementation (python3's json module) rather than round-tripping through libntech's own codec.

Compatibility consequence — decision needed

Documents written by the old encoder no longer round-trip through the fixed library, because their per-byte escapes are now read per code point, as every conformant reader always did. See the consolidated questions issue.

Activity

  1. added
    bugSomething isn't working
    jira-migratedCopied from a rejected CFE Jira ticket
    on Aug 22, 2026
  2. djbclark commented on Aug 22, 2026

    @djbclark
    OwnerAuthor

    Open decision from this item is tracked in the consolidated list: djbclark/core#23

  3. djbclark commented on Oct 9, 2026

    @djbclark
    OwnerAuthor

    Re-cut onto master 3c21479 as branch claudehelm/libntech-6 (0b0b91b, same change as 90cf8cc / fix/json-string-codec, with trailers). Chose the both-halves codec fix over PR #9's writer-only raw-UTF-8 variant because the reading half (\u4e2d corrupting to literal text) is the data-loss case and PR #9 leaves it. Verified both ways: with libutils at master 3 of 72 tests fail ("u4e2du56fd" != "中国"); with the fix 72/72 pass, build 0 warnings. The old-encoder compatibility question (djbclark/core#23 item 3) is still the operator's call. Upstream PR body drafted, not posted. Left open for operator review (ClaudeHelm).

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    bugSomething isn't workingjira-migratedCopied from a rejected CFE Jira ticketneeds-decisionBlocked on a judgment call

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions