Skip to content

Add held-out evaluation for the native recurrent backend - #13

Closed
0z5a wants to merge 1 commit into
perf/packed-prefix-suffixfrom
feature/native-heldout-evaluation
Closed

0z5a wants to merge 1 commit into
perf/packed-prefix-suffixfrom
feature/native-heldout-evaluation

Conversation

@0z5a

@0z5a 0z5a commented Oct 5, 2026

Copy link
Copy Markdown
Owner

Periodic held-out evaluation is currently rejected by the native recurrent backend, so the training recipe cannot record a reward curve. This adds complete text-dataset evaluation with fixed completion seeds, separate evaluation sampling, per-dataset completion counts and one committed policy cohort across all batches.

Native result indices count completed updates: the initial policy remains eval_0, and the first updated policy is eval_1. Evaluation-only resume uses the restored cursor. Publication versions remain separately recorded.

Stacked on #12; this diff contains the evaluation implementation, tests and raw CPU evidence. Author and committer: 0z5a.

Validation

508 distinct CPU checks pass across the final release and argument suites; six existing CUDA checks skip and three known missing-Triton factory checks are deselected. Ruff and source whitespace checks pass. Unmodified raw logs retain upstream warning whitespace.

Tiny Ouro, Nanbeige and Huginn each perform three actual native/MCore/AdamW updates. Inserting held-out evaluation preserves every training record, final weight and optimizer state exactly. Restoring into fresh objects reproduces the next update and evaluation. Ray transport and tokenization are test doubles, and recovery occurs in the same test process.

Scope Parent #12 This change Parent/current speed ratio
Native periodic held-out evaluation Rejected CPU component tests pass Undefined: parent cannot execute this path
Official-model full RL NOT_RUN NOT_RUN —
Matched peak CUDA memory NOT_RUN NOT_RUN —
Reward time-to-quality NOT_RUN NOT_RUN —

No acceleration or convergence claim is made from these tests. Formal evaluation must include official checkpoints/tokenizers, actual Ray/CUDA workers, complete evaluation cost and independent training seeds. The initial implementation is text-only and completes the full dataset; unsupported execution and partial-dataset contracts are rejected explicitly.

Report: docs/native_evaluation_results.md. Source hashes, raw test receipts and per-update/evaluation records are in benchmarks/results/native-evaluation-cpu/.

Preserve fixed evaluation seeds and committed policy cohorts across updates and resume. Keep initial and post-update results distinct, honor per-dataset completion counts, and retain CPU learning-isolation evidence.

Signed-off-by: 0z5a <dezhen.lu@student.uni-tuebingen.de>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant