Skip to content

fix: preserve rendered native chat prompts - #16

Closed
0z5a wants to merge 1 commit into
data/math-quality-inputsfrom
fix/native-chat-prompt
Closed

0z5a wants to merge 1 commit into
data/math-quality-inputsfrom
fix/native-chat-prompt

Conversation

@0z5a

@0z5a 0z5a commented Oct 5, 2026

Copy link
Copy Markdown
Owner

When --apply-chat-template is enabled, the standard dataset has already rendered each prompt to a string. Native rollout applied the template again, causing the official Ouro-Thinking tokenizer to fail before the engine call. Apply the template only to structured message lists; encode rendered strings directly.

Incremental draft on #15. The production diff is one condition.

Comparison Parent #15 Candidate Speedup
Rendered chat prompt Fails before engine call Exact prepared token IDs preserved Undefined; parent cannot execute
Full online RL Pending Pending Not measured

Validation: 631 CPU checks pass, 6 CUDA skips, 3 known factory deselections. Six new prompt-contract cases cover rendered strings, structured messages and plain text, with thinking on/off; the unchanged parent reproduces both rendered-string failures. An official-tokenizer audit checks all 500 MATH-500 prompts in both thinking modes and two completions per prompt: 2,000 exact input-token matches and known-answer rewards.

The official audit uses an explicit Ray transport seam and canonical answer replay, not model generation or quality measurement. Original production and initial fixture failures are retained. See docs/native_chat_prompt_results.md and benchmarks/results/native-chat-prompt/ for complete scope, source hashes and raw results.

Signed-off-by: 0z5a <dezhen.lu@student.uni-tuebingen.de>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant