Partial tool-call openers leak into message.content on seven other detectors (same class as #203)

[vianbas]

Splitting this out of #203 so that PR stays one change, as CONTRIBUTING asks.

Summary

#203 is a gemma-4 report, but the shape that causes it is not gemma-4 specific. Seven other
detectors in server/function_call_parser.py carry the identical line:

normal_text = text[:idx].strip() if idx != -1 else text

The opener lookup only matches a complete token. Five of the seven use
text.find(self.bot_token); DeepSeekV32Detector and GptOssDetector use
_first_existing_pos() over two openers each. Either way, when the model emits a partial
or malformed opener nothing matches, idx == -1, the whole text becomes normal_text,
and the marker reaches message.content verbatim.

Reproduction

Feeding each detector its own bot_token minus the final character, prefixed with ordinary
prose. Run against main at bd372b6 with PR #346 applied — so this is what remains after
the gemma-4 fix, not something that PR introduces:

--tool-call-parser	bot_token	sent	message.content
qwen25	<tool_call>	<tool_call	Working on it.<tool_call
mistral	[TOOL_CALLS] [	[TOOL_CALLS]	Working on it.[TOOL_CALLS]
glm47	<tool_call>	<tool_call	Working on it.<tool_call
deepseekv32	<｜DSML｜function_calls>	<｜DSML｜function_calls	Working on it.<｜DSML｜function_calls
qwen3_coder	<tool_call>	<tool_call	Working on it.<tool_call
minimax	<minimax:tool_call>	<minimax:tool_call	Working on it.<minimax:tool_call
gpt-oss	`<	channel	>`

from freetoken.server.function_call_parser import FunctionCallParser

TOOLS = [{"type": "function", "function": {"name": "get_weather", "parameters": {"type": "object"}}}]
for name in ["qwen25", "mistral", "glm47", "deepseekv32", "qwen3_coder", "minimax", "gpt-oss"]:
    p = FunctionCallParser(TOOLS, tool_call_parser=name)
    partial = p.detector.bot_token[:-1]
    print(name, repr(p.parse_non_stream(f"Working on it.{partial}").normal_text))

What I am NOT claiming

I have only shown that the parser surfaces a partial opener when it is given one. I have
not observed any of these seven models actually emitting one in production — #203 is the
only field report I know of, and it is gemma-4. So please read this as a latent path rather
than seven live bugs. Whether it fires in practice depends on whether a given format's
opener can survive detokenisation in pieces, which I am not in a position to measure.

Same caveat as on #346: no GPU here, no model served. This is parser-level only, on an
Apple M4 with torch.cuda.is_available() == False.

Possible fix

#346 adds BaseFormatDetector.scrub_markup(), called on the one-shot parse paths and the
end-of-stream drain (not per streaming chunk — that path still releases a partial opener,
noted as a known gap on #346) and defaulting to returning the text unchanged. Each of these detectors could
override it with its own marker prefix, which would close the path without touching the
formats that deliberately surface raw text.

Two things worth deciding before anyone writes that:

1. What the prefix is per format. gemma-4 was easy — its opener stabilises on
   <|tool_call (its closer <tool_call|> does not share that prefix and is handled
   separately). [TOOL_CALLS] [ and <|channel|> need a judgement call about how much of
   the marker is safe to treat as "unambiguously ours" without eating legitimate text.
2. Whether truncating is right at all. scrub_markup() as implemented for gemma-4 cuts
   everything from the marker onward. For formats where prose can legitimately follow a
   malformed block, deleting just the marker substring may be the better trade.

Happy to send a PR for whichever direction you prefer, or to leave it if the latent-path
framing means it is not worth the churn.

[vianbas]

Following up with the Roadmap/FAQ/duplicate check CONTRIBUTING asks for, so a maintainer does not have to redo it.

• Roadmap (FreeToken Roadmap (2026) #79): no item covers this. It lists New Model Support, Hardware Support (macOS Metal, AMD ROCm, DGX Spark), Multimodal, Multi-GPU, Quantization and Speculative Decoding — nothing on the tool-call parsers.
• FAQ (FAQ #84): covers supported hardware/OS, RAM requirements and supported models only.
• Closest existing issue: Qwen semantic responses can leak transport special-token markers #265 ("Qwen semantic responses can leak transport special-token markers"), with PR fix(server): hide qwen transport markers from semantic responses #266 open against it.

On that last one — I do not think #265/#266 subsumes this, and the reason is worth stating explicitly since the two read alike at a glance:

1. Different layer. fix(server): hide qwen transport markers from semantic responses #266 changes server/generation.py and adds tests/server/test_qwen_semantic_special_tokens.py; it does not touch server/function_call_parser.py, where the line reported here lives.
2. Different markers, and Qwen semantic responses can leak transport special-token markers #265 rules these out on purpose. Its expected behaviour says: "Reasoning tags, tool-call tags, and grounding markers remain available to their parsers or clients as appropriate." So the transport-marker filter is designed not to strip tool-call markup — which is exactly what leaks here.

So the two are complementary rather than overlapping: #265 keeps transport framing out of semantic content at the generation boundary, while this is the detectors surfacing their own tool-call openers when the opener arrives incomplete.

Happy to be told otherwise if the intent is for the generation-layer filter to eventually catch all of it — in that case this is better closed as a duplicate than fixed separately.