Skip to content

fix(engine): decode JSON escapes in stop_sequences - #819

Open
breken-ai wants to merge 1 commit into
cactus-compute:mainfrom
breken-ai:fix/stop-sequence-escapes
Open

breken-ai wants to merge 1 commit into
cactus-compute:mainfrom
breken-ai:fix/stop-sequence-escapes

Conversation

@breken-ai

Copy link
Copy Markdown

What

parse_inference_options_json read stop_sequences by copying each string from one " to the next. It never decoded JSON escapes. This PR reads the array with parse_json_string_array_field, the helper the custom_vocabulary option already uses, which does decode them. The fix removes 17 lines and adds 1.

Why

Any JSON encoder escapes these characters, so the problem reaches every binding:

stop_sequences sent parsed on main parsed with this PR
"\n\n" \n\n (4 characters: backslash, n, backslash, n) two newlines
"C:\\" C:\\ C:\
"caf\u00e9" caf\u00e9 café
"\"\"" \, then an empty string ""

Such a stop never matches, so generation runs to max_tokens. stop: ["\n"] is a common OpenAI-style request, and python/cactus/server.py passes req.stop straight into stop_sequences.

It can also hang. When a stop holds an odd number of escaped quotes and stop_sequences is the last option, the scan runs off the array. json.find('"', npos + 1) then restarts at offset 0, and the loop appends strings forever. server.py's _chat_options puts stop_sequences last. So {"max_tokens": 64, "stop_sequences": ["\""]} never returns: the test binary on main reached 9.3 GB resident within 10 s before alarm killed it.

Tests

I added cactus-engine/tests/test_inference_options.cpp. It needs no model, and it builds through the existing test_*.cpp glob.

  • On main (fa094ce8): stop_sequences_decode_json_escapes fails, and stop_sequence_with_quote_as_last_option hangs (killed at 10 s, 9.3 GB max RSS).
  • With this PR, both pass (All 2 tests passed!).
  • All engine test targets build on macOS arm64. The model-free suites test_bucket (10/10) and test_kv_compress (29/29) still pass. The model-backed suites were not run: they need a bundle.

AI disclosure

I found and fixed this with help from an AI coding assistant. I checked the code path by hand and reproduced the failure on main before writing the fix. The commit is signed off for the DCO.

🤖 Generated with Claude Code

parse_inference_options_json copied each stop sequence verbatim from the
first quote to the next one, so an escaped character was never decoded:
"\n" became a literal backslash and n, "é" stayed six ASCII bytes,
and "\"" split the string. The stop never matched and generation ran to
max_tokens. When the escaped quotes were odd in number and stop_sequences
was the last option (the order python/cactus/server.py builds), the scan
wrapped to the start of the JSON and appended forever: the call hangs and
grows memory without bound (9.3 GB in 10 s).

Read the array with parse_json_string_array_field, which the custom
vocabulary option already uses and which decodes escapes.

Signed-off-by: breken <hello@breken.ai>

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant