Skip to content

Verify native rollout and held-out outputs after recovery - #21

Closed
0z5a wants to merge 1 commit into
feature/native-runtime-identityfrom
test/native-output-recovery
Closed

0z5a wants to merge 1 commit into
feature/native-runtime-identityfrom
test/native-output-recovery

Conversation

@0z5a

@0z5a 0z5a commented Oct 5, 2026

Copy link
Copy Markdown
Owner

Checkpoint equality does not establish that a resumed worker generated the same samples or evaluated every held-out question. Add an output audit that reopens both runs, binds evaluation to the recorded worker and publication, re-decodes saved tokens, regrades rewards, and compares training tensors plus all four held-out rounds.

The frozen qualification covers 96 training and 32 held-out samples per run. Tokens and semantic traces must match; numeric comparisons use the predeclared FP32 tolerance and report maximum differences. Identical outputs with no useful learning signal do not qualify. This change leaves training and serving behavior unchanged.

Stacked on #20 with feature/native-runtime-identity as the base. One incremental commit, authored and signed off by 0z5a.

Validation Result
CPU regression 245 passed; 20 new audit cases
Actual tiny Ouro / Nanbeige / Huginn sampling and Adam recovery 8 paired held-out outputs per family; exact tokens/rewards/traces, zero score difference
Official retained Ouro tokenizer 40 answer-replay controls; six config/tokenizer files match the pinned manifest
Full official Ray/CUDA E2E, speedup, memory reduction, reward convergence Not run / not established

The tiny model tests restore fresh objects in one process and use tokenizer/transport doubles. CLI tests use synthetic worker/process records; official tokenizer controls use known answers and invented scores. These are not fresh Ray-worker or reasoning-convergence results. Weight/checkpoint/optimizer state and frozen-source membership remain separate gates.

Results and speed-comparison table, with raw logs/XML, tokenizer controls and hashes in benchmarks/results/native-output-recovery/.

Signed-off-by: 0z5a <dezhen.lu@student.uni-tuebingen.de>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant