Lab efficiency meta-analysis on checkpoint cycles #8
Description
Problem
The framework tracks individual experiments but doesn't meta-analyze its own efficiency. Questions like "what fraction of budget is going to dead branches?", "what's our cost per champion?", and "are we exploring too much or too little?" are left to human intuition. Without this, labs can waste 50%+ of experiments without anyone noticing until manual review.
In a real deployment, 51% of experiments were on branches that produced zero champions. This wasn't surfaced until a manual audit at the end of the session.
Proposal
Add automatic lab efficiency meta-analysis to human checkpoint reports (every 15 cycles) and optionally to every handoff.
Metrics to compute
## Lab Efficiency Report (Cycle {N})
### Exploration vs Exploitation Balance
- Experiments on branches with no champion yet: {X} (exploration)
- Experiments iterating on existing champions: {Y} (exploitation)
- Ratio: {X/Y} (guideline: 2:1 early, 1:2 late in lab lifecycle)
- Current phase: {early / mid / late} (based on % search space explored)
### Cost per Champion
- Total experiments: {N}
- Total champions: {M}
- Experiments per champion: {N/M}
- Most efficient branch: {branch} ({K} experiments to first champion)
- Least efficient branch: {branch} ({K} experiments, 0 champions)
### Budget Utilization
- Total budget spent: {X}
- Budget on dead branches: {Y}
- Waste rate: {Y/X * 100}%
- Budget remaining: {Z}
- Estimated cycles to exhaustion: {Z / avg_spend_per_cycle}
### Signal-to-Noise Ratio
- Promotion rate: {M/N * 100}%
- Marginal rate: {J/N * 100}%
- Rejection rate: {(N-M-J)/N * 100}%
- Trend (last 10 cycles): {improving / stable / degrading}
### Phase Detection
- If promotion_rate > 30% and exploration_ratio > 1.5: "Productive exploration phase"
- If promotion_rate > 30% and exploitation_ratio > 1.5: "Productive exploitation phase"
- If promotion_rate < 10% and exploration_ratio > 1.5: "Unproductive exploration — consider pivoting"
- If promotion_rate < 10% and exploitation_ratio > 1.5: "Diminishing returns — consider new branches"Automatic recommendations
Based on the efficiency metrics, append to the handoff:
| Condition | Recommendation |
|---|---|
| waste_rate > 50% | "Over half of budget is going to dead branches. Consider pruning or scouting." |
| experiments_per_champion > 15 | "High cost per champion. Consider tightening search spaces or adding diagnostics." |
| promotion_rate trending down | "Diminishing returns. Consider expansion cycle or frame challenge." |
| exploration_ratio < 0.5 late in lifecycle | "Under-exploring. Consider reopening exhausted branches with scouts." |
Implementation
Compute at the end of Step 7 (Update State). Write to the handoff and to a dedicated state/lab_efficiency.json for trend tracking. Full report on checkpoint cycles; summary line on every cycle.
Why this matters
A lab that doesn't measure its own productivity can't improve its process. Meta-analysis turns the lab's experiment history into a feedback signal on the RESEARCH PROCESS itself, not just the research outcomes. This is the difference between doing science and doing science well.
Relationship to existing features
- Natural extension of human checkpoint reports (Step 1d)
- Feeds into the diminishing returns detector (Step 8c) with richer signal
- Informs the expansion scout: "we're wasting budget on calibration branches, scout for content-based approaches"
- Phase detection helps the orchestrator adjust exploration/exploitation balance dynamically