Repository navigation
Add LLM confounder soft labels for augmented noul questions - #111
Merged
Merged
Conversation
87,719/87,822 items labeled via GPT-5.6 Luna (low reasoning effort). Remaining 103 items blocked by Azure content filter.
Jev API labels for LLM-generated confounder options (augmented_options_llm). 17,904/17,904 items labeled, 0 failures across 23 domains. Includes label_confounder_noul_jev.py script.
GPT-5.6 Luna labels for LLM-generated confounder options. 19 items blocked by Azure content filter. Includes label_confounder_noul_llm.py script and label_llm_confounders.py utility.
24 of 27 tasks
![milo-oaklight[bot]](https://avatars.githubusercontent.com/in/3956677?s=60&v=4){kind=small}
There was a problem hiding this comment.
LGTM ✅ Phase B soft labels complete.
Verified:
- CI green (all checks pass)
- Label structure correct:
{id, teacher_probs: {label_key: {option: prob}}} - Probability distributions sum to ~1.0 (spot-checked academic_reasoning samples)
- Scripts well-structured: resumable via existing ID check, shardable via
--shard/--num-shards - 72 files: 69 data files + 3 scripts
Counts (from PR description):
| Label Type | Count | Coverage |
|---|---|---|
| Luna augmented (template) | 87,719 | 99.9% |
| Jev confounder (LLM) | 17,904 | 100% |
| Luna confounder (LLM) | 17,885 | 99.9% |
Azure content filter blocked 103 + 19 = 122 items total — acceptable loss for sensitive domains.
Ready to merge.
![clementine-oaklight[bot]](https://avatars.githubusercontent.com/in/3956685?s=60&v=4){kind=small}
There was a problem hiding this comment.
PR #111 (krino) — APPROVED ✅
| Metric | Jev Teacher | Luna Teacher |
|---|---|---|
| Template labels | 87,822 (100%) | 87,719 (99.9%) |
| LLM confounder labels | 17,904 (100%) | 17,885 (99.9%) |
| Azure content filter blocks | — | 103 template, 19 LLM |
Scripts added:
| Script | Purpose |
|---|---|
label_confounder_noul_jev.py |
Jev API labeling for LLM confounders |
label_confounder_noul_llm.py |
Luna labeling for LLM confounders |
label_llm_confounders.py |
Combined orchestration script |
All three scripts follow the established pattern: resumable, shardable, dry-run by default.
Data format verified:
- ID pattern:
synthetic-{domain}-{fidx:04d}-noul-{cognitive_type}-{idx}-llm - teacher_probs distributions sum to ~1.0 ✓
- Soft labels show genuine uncertainty (not degenerate 0.99/0.01)
Relationship with prior PRs:
| PR | Phase | Content |
|---|---|---|
| #107 | — | Template-based confounders |
| #109 | — | Calibration-aware labeling prompts |
| #110 | A | LLM-generated confounders (Qwen3.8-27B) |
| #111 | B | Soft labels for LLM confounders |
Closes Phase B checklist items in #104. CI green.
This was referenced Sep 29, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Complete Phase B of issue #104: multi-teacher soft labels for augmented noul questions using both template-based and LLM-generated confounders.
label_confounder_noul_jev.py,label_confounder_noul_llm.pyfor resumable, shardable labelingData inventory after this PR
Builds on PR #110 (LLM confounder generation) and earlier commits for routing augmented noul through choice labeling path, Qwen3.8-27B gap filling, and Luna low-reasoning-effort labeling.
Closes Phase B checklist items in #104.
Test plan