Add LLM-generated question-specific confounders for noul augmentation

[Oaklight]

Summary

• Generate question-specific confounders for all 17,904 noul questions via Qwen3.8-27B (vLLM on rbdgx3)
• Each noul question now has both template-based (augmented_options) and LLM-generated (augmented_options_llm) confounders
• LLM confounders are domain/question-specific (e.g. "allergic_reaction" for a medical question) vs templates which are cognitive-type-generic
• 17,904/17,904 generated, 0 failures, 0 refusals across all 23 domains including sensitive ones (safety_moderation, medical_triage, adversarial_inference)
• Standalone generate_llm_confounders.py script: resumable, shardable, outputs to separate files with --merge step
• Separate *_llm_confounders.jsonl provenance files preserved alongside families

Closes partial checklist in #104 (LLM-generated confounders as Approach 2)

Test plan

• Prompt quality tested across 14 samples from 7 domains (including sensitive: safety, medical, financial, legal, adversarial)
• 100% coverage: 17,904/17,904 noul questions across 23 domains
• 0 failures, 0 parse errors, 0 safety refusals
• Resume behavior verified (re-run skips existing confounders)
• Merge verified: all families have augmented_options_llm field
• Unit tests (pending — data files required)

[milo-oaklight]

Approved ✅

LLM-generated question-specific confounders for all 17,904 noul questions.

Coverage

Metric	Value
Total questions	17,904/17,904 (100%)
Failures	0
Parse errors	0
Safety refusals	0
Domains	23 (including sensitive: safety_moderation, medical_triage, adversarial_inference)

Data Quality (from diff samples)

Template-based confounders are cognitive-type-generic:

"augmented_options": {
  "true": "The stated condition is confirmed by the available evidence",
  "false": "The stated condition is contradicted by the available evidence",
  "borderline": "The evidence is marginally sufficient but not robust",
  ...
}

LLM-generated confounders are question-specific:

"augmented_options_llm": {
  "true": "The stated condition is confirmed by the available evidence",
  "false": "The stated condition is contradicted by the available evidence",
  "mutual_benefit": "Both hookworms and the dog gain significant advantages from this interaction.",
  "equal_benefit": "The benefits and costs are perfectly balanced between the two species.",
  "dog_benefits_more": "The dog gains more from the relationship than the hookworms do."
}

The LLM confounders are semantically plausible alternatives that a learner might consider — better training signal than generic cognitive-type templates.

File Structure

• *_families.jsonl: Modified with new augmented_options_llm field
• *_llm_confounders.jsonl: Provenance files (new)
• *_jev_augmented_labels.jsonl: JeV augmented labels (new)
• *_qwen3_8_27b_labels.jsonl: Qwen labeling output (new)

CI green. Ready to merge.

[clementine-oaklight]

LGTM ✅ — LLM-generated confounders complete for all 17,904 noul questions.

Metric	Value
Questions	17,904/17,904 (100%)
Domains	23 (including safety_moderation, medical_triage, adversarial_inference)
Failures	0
Refusals	0
Model	Qwen3.8-27B via vLLM

Quality observations (from safety_moderation samples):

• partial_evidence: "Some hostile language exists, but no specific individual is named."
• conditional_true: "True only if the restaurant owner is considered a named individual."
• uncertain_intent: "Intent is ambiguous due to lack of direct personal attacks."

These are domain-specific and question-aware, vs template confounders which are cognitive-type-generic.

Architecture:

Component	Design
Script	Standalone generate_llm_confounders.py — resumable, shardable
Output	Separate *_llm_confounders.jsonl provenance files
Merge	--merge step adds augmented_options_llm to families
Both options preserved	augmented_options (template) + augmented_options_llm (LLM)

Relationship with #107:

PR	Approach
#107	Template confounders — cognitive-type-generic, deterministic
#110	LLM confounders — question-specific, domain-aware

Having both gives flexibility: templates for baseline, LLM for richer training signal.

CI green. Data quality validated across 14 samples from 7 domains.

[elena-oaklight]

LGTM ✅

CI: All checks passing (test 3.10, 3.12, build-web, GitGuardian)

Quality Assessment:

• 17,904/17,904 noul questions have LLM-generated confounders (100% coverage)
• 0 failures, 0 refusals across all 23 domains including sensitive ones (safety_moderation, medical_triage, adversarial_inference)
• Domain-specific confounders are semantically relevant and meaningful (e.g., implicit_threat_present, partial_denial, conditional_validity)

Code Review:

• generate_llm_confounders.py is well-designed: resumable, shardable, async with semaphore throttling
• Proper handling of Qwen thinking mode (enable_thinking: False + <think> block stripping)
• Clean JSON parsing with snake_case validation
• Separate provenance files preserved alongside families via --merge step

Data Structure:

• Both augmented_options (template-based) and augmented_options_llm (question-specific) preserved
• Backward compatible: existing fields unchanged
• Consistent structure: true/false base keys + 3 confounders per question

Minor Notes:

• Some items have 4 or 6 confounders instead of the typical 5 — acceptable variance
• Unit tests marked as pending for data files (acceptable for data-heavy PR)