Great proposal — cost as the competitive axis is the right long-term design. "$0.04/task, -75% vs baseline" sells to enterprises in a way "score: 0.91" never will.
A few concerns on the qualification gate:
1. p^59 math is harsh. With 59 required checks across 5 scenarios and LLM non-determinism flipping 2-5 checks per run, even a perfect policy fails qualification regularly. At p=0.98 per check: 0.98^59 = 0.30 — 30% pass rate per run. Majority-vote (2/3) helps but doesn't eliminate this. Risk: epochs where no miner qualifies and emission goes to uniform weights.
2. Consider 4/5 scenario pass instead of 5/5. One flipped check in one scenario = full disqualification is fragile. Allowing 4/5 accounts for LLM variance without sacrificing quality.
3. 5 consensus runs instead of 3. A check needs to fail 3/5 instead of 2/3 to count as failed — dramatically reduces false-disqualification. More eval compute, but worth it for fairness.
4. 10% cost delta may be too thin. Token counts vary ~5-10% between runs from pure noise. At 10%, a miner could dethrone the incumbent through luck, not optimization. 15-20% would be safer.
5. Consider keeping efficiency checks as gates. Without them, a pack could use 30 tool calls and still qualify as long as safety+correctness pass. The cost metric partially handles this, but a gate prevents degenerate strategies.
Overall: strong direction, just needs softer edges on the qualification design.
Ref: https://github.com/trajectoryRL/clawbench/blob/main/proposal/0303-cost-based-scoring.md
Great proposal — cost as the competitive axis is the right long-term design. "$0.04/task, -75% vs baseline" sells to enterprises in a way "score: 0.91" never will.
A few concerns on the qualification gate:
1.
p^59math is harsh. With 59 required checks across 5 scenarios and LLM non-determinism flipping 2-5 checks per run, even a perfect policy fails qualification regularly. At p=0.98 per check:0.98^59 = 0.30— 30% pass rate per run. Majority-vote (2/3) helps but doesn't eliminate this. Risk: epochs where no miner qualifies and emission goes to uniform weights.2. Consider 4/5 scenario pass instead of 5/5. One flipped check in one scenario = full disqualification is fragile. Allowing 4/5 accounts for LLM variance without sacrificing quality.
3. 5 consensus runs instead of 3. A check needs to fail 3/5 instead of 2/3 to count as failed — dramatically reduces false-disqualification. More eval compute, but worth it for fairness.
4. 10% cost delta may be too thin. Token counts vary ~5-10% between runs from pure noise. At 10%, a miner could dethrone the incumbent through luck, not optimization. 15-20% would be safer.
5. Consider keeping efficiency checks as gates. Without them, a pack could use 30 tool calls and still qualify as long as safety+correctness pass. The cost metric partially handles this, but a gate prevents degenerate strategies.
Overall: strong direction, just needs softer edges on the qualification design.
Ref: https://github.com/trajectoryRL/clawbench/blob/main/proposal/0303-cost-based-scoring.md