TITO: non_assistant_text session-mismatch ≈ 0.99 in production agentic-RL

[nightlessbaron]

Summary

In production agentic-RL rollouts, tito_session_mismatch_rate/non_assistant_text is ≈ 0.99 — i.e. almost every session shows a non-assistant-text token divergence.

# miles/utils/chat_template_utils/token_seq_comparator.py
NON_ASSISTANT_TEXT = "non_assistant_text"
# Non-assistant content (user, system, tool, etc.) differs.  This indicates
# a bug in the TITO algorithm — these regions should match exactly.

Evidence (two independent runs, identical tokenizer/config)

metric	run A	run B
tito_session_mismatch_rate	1.0	1.0
…/non_assistant_text	~0.99	~0.99
…/assistant_text	~0.99	~0.99
…/special_token_count	~0.006–0.012	~0.006
…/special_token_type	~0.002–0.004	~0.004

So the special-token structure is essentially fine (~0), and assistant_text divergence is the documented/expected "non-severe" case (assistant tokens are inherited from the pretokenized prefix). The concern is non_assistant_text (which "should match exactly") sitting at ~0.99.

Relevant config:

• tito_model = k2v3
• chat_template = autofix
• 8B Llama-family tokenizer

Analysis — where the divergence comes from

SessionRegistry.compute_session_mismatch (miles/rollout/session/linear_trajectory.py) compares:

• expected_ids = apply_chat_template(session.messages, tokenizer=<HF>, add_generation_prompt=False, tokenize=True) — a canonical full HF render, vs
• session.token_ids — the accumulated checkpoint.

session.token_ids is not a pure HF render. In chat_completions / LinearTrajectory.update_pretokenized_state (miles/rollout/session/sessions.py) it is assembled from:

• turn 0: SGLang-returned choice["prompt_token_ids"], plus
• subsequent turns: incremental merge_tokens deltas for appended non-assistant (user/tool/system) messages + SGLang completion tokens.

So non_assistant_text divergence has two candidate sources:

1. SGLang/gateway tokenization of non-assistant text differs from HF apply_chat_template (chat-template / whitespace / boundary differences — plausibly the autofix template path or a gateway-side template that isn't byte-identical to the HF one), and/or
2. incremental tokenization seams in merge_tokens when appending user/tool/system messages, vs a single full-sequence render.

Questions

1. Is non_assistant_text ≈ 0.99 expected for the tito_model=k2v3 + chat_template=autofix configuration, or a genuine TITO/chat-template bug as the comparator comment states?
2. If this is non-severe in practice (training is self-consistent — generation and stored tokens both come from SGLang, so the loss is computed on actually-generated tokens), should non_assistant_text be downgraded from the strict set for these configs (or the seam behavior documented) so the metric isn't silently pinned at 1.0?

Impact

No crash. Training appears self-consistent, so this reads as a correctness-signal / metric-integrity concern rather than training-data corruption. Filing because (a) the comparator explicitly classifies non_assistant_text divergence as a bug, and (b) the rate is ~1.0 across runs, which makes tito_session_mismatch_rate uninformative as a guardrail.