Repository navigation
Conversation
parse_inference_options_json copied each stop sequence verbatim from the first quote to the next one, so an escaped character was never decoded: "\n" became a literal backslash and n, "é" stayed six ASCII bytes, and "\"" split the string. The stop never matched and generation ran to max_tokens. When the escaped quotes were odd in number and stop_sequences was the last option (the order python/cactus/server.py builds), the scan wrapped to the start of the JSON and appended forever: the call hangs and grows memory without bound (9.3 GB in 10 s). Read the array with parse_json_string_array_field, which the custom vocabulary option already uses and which decodes escapes. Signed-off-by: breken <hello@breken.ai>
This branch has not been deployed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What
parse_inference_options_jsonreadstop_sequencesby copying each string from one"to the next. It never decoded JSON escapes. This PR reads the array withparse_json_string_array_field, the helper thecustom_vocabularyoption already uses, which does decode them. The fix removes 17 lines and adds 1.Why
Any JSON encoder escapes these characters, so the problem reaches every binding:
stop_sequencessentmain"\n\n"\n\n(4 characters: backslash, n, backslash, n)"C:\\"C:\\C:\"caf\u00e9"caf\u00e9café"\"\""\, then an empty string""Such a stop never matches, so generation runs to
max_tokens.stop: ["\n"]is a common OpenAI-style request, andpython/cactus/server.pypassesreq.stopstraight intostop_sequences.It can also hang. When a stop holds an odd number of escaped quotes and
stop_sequencesis the last option, the scan runs off the array.json.find('"', npos + 1)then restarts at offset 0, and the loop appends strings forever.server.py's_chat_optionsputsstop_sequenceslast. So{"max_tokens": 64, "stop_sequences": ["\""]}never returns: the test binary onmainreached 9.3 GB resident within 10 s beforealarmkilled it.Tests
I added
cactus-engine/tests/test_inference_options.cpp. It needs no model, and it builds through the existingtest_*.cppglob.main(fa094ce8):stop_sequences_decode_json_escapesfails, andstop_sequence_with_quote_as_last_optionhangs (killed at 10 s, 9.3 GB max RSS).All 2 tests passed!).test_bucket(10/10) andtest_kv_compress(29/29) still pass. The model-backed suites were not run: they need a bundle.AI disclosure
I found and fixed this with help from an AI coding assistant. I checked the code path by hand and reproduced the failure on
mainbefore writing the fix. The commit is signed off for the DCO.🤖 Generated with Claude Code