You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
{{ message }}
Repository navigation
[PR] fix: align watermark scanner with JSON whitespace and number grammar - #6599
The watermark structural scanner returns scan-broken for valid JSON with whitespace after commas inside nested arrays, including pretty-printed multipart content and tool-schema enums. Positive exponents such as 1e+3 fail for the same reason: the token scanner stops before +, even though the final number grammar supports it. Eligible seeded requests then miss the existing prefix reuse and repeat full-body stripping/conversion.
The top-level object loop also accepts a trailing comma, contrary to the scanner's fail-soft JSON contract. The HTTP handler already parses the full body before this evaluator; this is not a claim of an HTTP validation bypass.
Changes
Skip JSON whitespace before each nested array value.
Include + in number token collection, leaving the existing strict number expression responsible for validity.
Require a key after every top-level comma, so a trailing comma returns null.
Add 23 focused cases, including seeded evaluator sequences and comparison with the legacy full-parse message reconstruction.
These changes retain the depth cap, code-unit spans, duplicate-key rules and watermark identity/eligibility gates. They add no configuration and do not remove the HTTP handler's initial full parse. No end-to-end latency or memory improvement is claimed.
Validation
On unchanged master e1f31ad9d5b47500362699a9867819b861c7de96, the new suite has 14 failing regressions and 9 passing controls. After the fix, all 58 tests across the new grammar suite, existing scanner suite, and watermark evaluator suite pass. The existing dual-proxy HTTP A/B test also passes, checking outbound byte equality, watermark-header containment, and the fastpath residency band: 59 total tests passed.
The new cases cover all four JSON whitespace characters plus a mixed sequence; positive/negative/unsigned exponents; malformed commas and numbers; Unicode span preservation; pretty-printed multipart/tool-schema arrays; and seeded hit/fallback decisions. Valid reconstructed messages match JSON.parse/openaiToCore output.
Passed on Node 24.22.1: npm run typecheck, npm run build, npm run check:unused-exports, npm run check:test-import-graph, and git diff --check. Full repository and real-provider matrices remain for hosted CI.
Prior work and scope
All-state issue/PR searches and my prior contribution history found no matching correction. Open #5662 changes scanner traversal/resumption; its current patch preserves these three grammar behaviors. This patch only corrects the existing parser contract, without changing that proposal's identity or resumption design.
Suggested triage: bug, p2, scope/wiring; Topic N — server-side miscellaneous.
ranxianglei
changed the title
fix: align watermark scanner with JSON whitespace and number grammar
[PR] fix: align watermark scanner with JSON whitespace and number grammar
Nov 13, 2026
Superseded by #6677 (merged as 3858b6956, 2026-11-14).
Landed into master via the real merge commit a74980e4d95 (Merge Afloat16/2026-11-13_fastpath-json-grammar (PR #6599)), so your original commits are preserved verbatim in master history — only hashes of the batch wrapper changed, your authorship and commit messages are intact.
The fix for #6598 is now in master. Closing this PR as superseded — thanks for the careful, well-tested fix.
Verified the supersession against live origin/master before acknowledging:
3858b6956a is the current master tip: Merge pull request #6677 from ranxianglei/batch5-wave5 — the batch wrapper is merged.
a74980e4d95 is in master history: Merge Afloat16/2026-11-13_fastpath-json-grammar (PR #6599) into batch5-wave5.
The original commit deff69e125 (fix: align watermark scanner with JSON whitespace and number grammar, author Afloat <ShuffleMonkey@163.com>) is an ancestor of origin/master — authorship and message preserved verbatim, only the wrapper hashes changed, exactly as stated.
The landed diff matches the PR description point-for-point: skipWs() before nested-array walkValue() (src/server/fastpath-scan.ts:91), + (charcode 43) added to the number-token set (src/server/fastpath-scan.ts:116), top-level trailing-comma acceptance removed so it now fails soft (src/server/fastpath-scan.ts:136), plus the 134-line tests/fastpath-json-grammar.test.ts suite.
So the #6598 fix is confirmed in master with the original work intact. No further changes needed from this side.
One note: per the repo's closing policy agents don't close PRs — the close itself is left to you (the mirror still shows it OPEN).
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Fixes #6598
Problem
The watermark structural scanner returns
scan-brokenfor valid JSON with whitespace after commas inside nested arrays, including pretty-printed multipart content and tool-schema enums. Positive exponents such as1e+3fail for the same reason: the token scanner stops before+, even though the final number grammar supports it. Eligible seeded requests then miss the existing prefix reuse and repeat full-body stripping/conversion.The top-level object loop also accepts a trailing comma, contrary to the scanner's fail-soft JSON contract. The HTTP handler already parses the full body before this evaluator; this is not a claim of an HTTP validation bypass.
Changes
+in number token collection, leaving the existing strict number expression responsible for validity.null.These changes retain the depth cap, code-unit spans, duplicate-key rules and watermark identity/eligibility gates. They add no configuration and do not remove the HTTP handler's initial full parse. No end-to-end latency or memory improvement is claimed.
Validation
On unchanged master
e1f31ad9d5b47500362699a9867819b861c7de96, the new suite has 14 failing regressions and 9 passing controls. After the fix, all 58 tests across the new grammar suite, existing scanner suite, and watermark evaluator suite pass. The existing dual-proxy HTTP A/B test also passes, checking outbound byte equality, watermark-header containment, and the fastpath residency band: 59 total tests passed.The new cases cover all four JSON whitespace characters plus a mixed sequence; positive/negative/unsigned exponents; malformed commas and numbers; Unicode span preservation; pretty-printed multipart/tool-schema arrays; and seeded hit/fallback decisions. Valid reconstructed messages match
JSON.parse/openaiToCoreoutput.Passed on Node 24.22.1:
npm run typecheck,npm run build,npm run check:unused-exports,npm run check:test-import-graph, andgit diff --check. Full repository and real-provider matrices remain for hosted CI.Prior work and scope
All-state issue/PR searches and my prior contribution history found no matching correction. Open #5662 changes scanner traversal/resumption; its current patch preserves these three grammar behaviors. This patch only corrects the existing parser contract, without changing that proposal's identity or resumption design.
Suggested triage:
bug,p2,scope/wiring; Topic N — server-side miscellaneous.