Conversation
Phase C of #104: training integration for augmented noul questions. - Add _select_teacher_probs() to data/pipeline.py: selects best teacher from nested multi-teacher cache (jev_augmented > gpt_5_6_luna_augmented for augmented noul, jev > luna > qwen for standard items). Fixes existing dead code where synthetic teacher_probs were never used. - Augmented noul items with 5-option teacher_probs are routed through the choice head (KL-divergence against teacher distribution) instead of the binary noul head, in both training and eval paths. - Uncertainty weighting correctly uses "choice" log_var for augmented noul items. - Eval maps augmented noul predictions back to binary accuracy (pred_key == "true"/"false"). - Standard noul fallback when teacher_probs has 2-key {"true","false"} format (uses true probability as BCE target). - 9 new tests for teacher selection logic, 557 total pass.
There was a problem hiding this comment.
PR #113 (krino) — APPROVED ✅
Clean routing of augmented noul through the choice head. Teacher selection logic is well-structured with clear priority chains.
| Path | Teacher Priority | Head | Loss |
|---|---|---|---|
| Augmented noul | jev_augmented > luna_augmented > standard | Choice (5-way softmax) | KL divergence |
| Standard noul | jev > luna > qwen | Binary noul | BCE |
Key changes verified:
_select_teacher_probs()— Correct priority fallback; empty dicts properly skipped- Routing gate —
augmented_options and len(teacher_probs) > 2cleanly separates paths - Uncertainty weighting — Augmented noul correctly uses
"choice"log_var - Eval mapping —
pred_key == target_keycorrectly maps 5-way predictions back to binary accuracy
Minor observation: _compute_augmented_noul_batch processes items sequentially (not true batch), matching the unified head path for standard noul. Fine for now; batch optimization can come later if needed.
Tests: 9 new tests cover teacher selection edge cases comprehensively.
CI green. Ready to merge.
This branch has not been deployed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Phase C of #104: training integration for augmented noul questions.
data/pipeline.py):_select_teacher_probs()selects the best teacher from the nested multi-teacher cache (jev_augmented > luna_augmented for augmented noul, jev > luna > qwen for standard). Fixes existing dead code where synthetic teacher_probs were never consumed by the training loop.model/training/supervised.py): Augmented noul items with 5-option teacher_probs are routed through the choice head (KL-divergence against teacher distribution) instead of the binary noul head. Standard noul items unchanged.Test plan