Repository navigation
Conversation
Preserve fixed evaluation seeds and committed policy cohorts across updates and resume. Keep initial and post-update results distinct, honor per-dataset completion counts, and retain CPU learning-isolation evidence. Signed-off-by: 0z5a <dezhen.lu@student.uni-tuebingen.de>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Periodic held-out evaluation is currently rejected by the native recurrent backend, so the training recipe cannot record a reward curve. This adds complete text-dataset evaluation with fixed completion seeds, separate evaluation sampling, per-dataset completion counts and one committed policy cohort across all batches.
Native result indices count completed updates: the initial policy remains
eval_0, and the first updated policy iseval_1. Evaluation-only resume uses the restored cursor. Publication versions remain separately recorded.Stacked on #12; this diff contains the evaluation implementation, tests and raw CPU evidence. Author and committer: 0z5a.
Validation
508 distinct CPU checks pass across the final release and argument suites; six existing CUDA checks skip and three known missing-Triton factory checks are deselected. Ruff and source whitespace checks pass. Unmodified raw logs retain upstream warning whitespace.
Tiny Ouro, Nanbeige and Huginn each perform three actual native/MCore/AdamW updates. Inserting held-out evaluation preserves every training record, final weight and optimizer state exactly. Restoring into fresh objects reproduces the next update and evaluation. Ray transport and tokenization are test doubles, and recovery occurs in the same test process.
No acceleration or convergence claim is made from these tests. Formal evaluation must include official checkpoints/tokenizers, actual Ray/CUDA workers, complete evaluation cost and independent training seeds. The initial implementation is text-only and completes the full dataset; unsupported execution and partial-dataset contracts are rejected explicitly.
Report:
docs/native_evaluation_results.md. Source hashes, raw test receipts and per-update/evaluation records are inbenchmarks/results/native-evaluation-cpu/.