Skip to content

Enable fixed-depth RLTT training with response-only readout - #8

Closed
0z5a wants to merge 1 commit into
feature/looped-streamed-readoutfrom
feature/looped-rltt-training
Closed

0z5a wants to merge 1 commit into
feature/looped-streamed-readoutfrom
feature/looped-rltt-training

Conversation

@0z5a

@0z5a 0z5a commented Oct 4, 2026 •

Copy link
Copy Markdown
Owner

Fixed-depth recurrent training can now opt into rltt_loss through the standard MCore model call. The actor consumes response-only scores from every loop, while reference evaluation consumes terminal scores. Logical-step weights preserve the declared loss across microbatch partitions. The initial reference remains frozen and is loaded even at zero KL; the recipe preserves its path across actor resume.

This is an incremental draft on #7. --algorithm rltt selects grouped GRPO advantages, static uniform/progressive credit and terminal sampled k3 for Ouro, Nanbeige and Huginn. It is an objective reimplementation; learned exit credit, online full KL, distributed scaling and the actual Ray/CUDA lifecycle remain unfinished.

Validation: 257 CPU checks passed; three factory/HF-loader checks failed at the existing runtime's missing Triton import. Three tiny native sampling → output-derived reward → AdamW update → physical publication cycles each complete three nonzero updates and reproduce the next step exactly after fresh actor/reference/optimizer recovery. CLI/placement checks capture calls and do not launch Ray workers. MCore's pinned patched source matches all 2232 audited files.

The validation report and comparison table includes raw update records and source hashes. Existing readout-only CPU ratios remain 0.414–0.722× (slower); saved intermediates are not peak GPU memory. No official-model acceleration or reward-convergence result is claimed.

Signed-off-by: 0z5a <dezhen.lu@student.uni-tuebingen.de>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant