Skip to content

chore(usage): escape the raw NUL bytes in jsonl-parser.ts as \0 - #84

Merged
pitimon merged 1 commit into
mainfrom
chore/escape-nul-jsonl-parser
Aug 3, 2026
Merged

chore(usage): escape the raw NUL bytes in jsonl-parser.ts as \0#84
pitimon merged 1 commit into
mainfrom
chore/escape-nul-jsonl-parser

Conversation

@pitimon

@pitimon pitimon commented Aug 3, 2026

Copy link
Copy Markdown
Owner

Mechanical prerequisite for #83, landed separately on purpose.

src/usage/jsonl-parser.ts holds 4 raw NUL bytes: 2 are the Map-key
separators in template literals (offsets 7260, 7365) and 2 sit in the comment
describing them (5708, 5745). Git classifies a blob binary when it finds a NUL
in the first 8000 bytes, so it was the comment's NUL that tripped the check
and suppressed this file's diff entirely — including from the repo's own audit
greps in coding-style.md, which then return 0 matches as a false negative.

\0 in a TypeScript template literal produces the same U+0000 character, so
this is a source-encoding change with no runtime effect. Verified by undoing
the escape and comparing: it reproduces the previous blob byte-for-byte
(8052 bytes, 4 NUL → 8056 bytes, 0 NUL, delta exactly 4).

Why this is its own PR

GitHub decides binary-vs-text by sniffing the pre-image and ignores the head
tree's .gitattributes — measured on #83, where the commits API reported no
patch for this file even though .gitattributes sat in the same tree. Splitting
the escape into its own commit on the same branch fixed the per-commit view but
not the PR-level "Files changed" tab, which still diffs against main.

Landing the escape on main first is what makes #83's diff reviewable in the
normal view.

Note on the diff

This PR's own diff renders as "Binary files differ" — unavoidable, since its
pre-image is the NUL-bearing blob. It is 4 bytes. To verify:

git fetch origin chore/escape-nul-jsonl-parser
git show FETCH_HEAD:src/usage/jsonl-parser.ts | tr -d '\0' | cmp - <(git show origin/main:src/usage/jsonl-parser.ts | tr -d '\0')

(identical with NULs stripped from both sides = nothing but the escapes changed)

Test plan

  • bun test at this commit — 180 pass / 0 fail
  • Round-trip: re-inserting the NULs reproduces origin/main's blob exactly

The file held 4 raw NUL bytes: 2 are the Map-key separators in template
literals (offsets 7260, 7365) and 2 sit in the comment describing them
(5708, 5745). Git classifies a blob binary when it finds a NUL in the first
8000 bytes, so it was the COMMENT's NUL that tripped the check and suppressed
the file's diff entirely -- including from the repo's own audit greps, which
then returned 0 matches as a false negative.

`\0` in a TypeScript template literal produces the same U+0000 character, so
this is a source-encoding change with no runtime effect. Verified by undoing
the escape and comparing: it reproduces the previous blob byte-for-byte.

Split out as its own commit deliberately. GitHub sniffs the PRE-image when
deciding binary-vs-text and does not apply the head tree's .gitattributes --
measured on an earlier revision of this branch, where the commits API reported
no patch for this file even though .gitattributes was present in the same tree.
Landing the escape separately gives the following commit a NUL-free pre-image,
which is what GitHub actually looks at.

This commit's own diff still renders as binary. It is 4 bytes; `git show <sha>
-- src/usage/jsonl-parser.ts` or the round-trip check above settles it.
@pitimon
pitimon merged commit 5f31f59 into main Aug 3, 2026
@pitimon
pitimon deleted the chore/escape-nul-jsonl-parser branch August 3, 2026 08:28
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant