docs(eval): eval scoring design doc

[henrycgbaker]

Summary

Proposed design doc for the eval scoring layer: turning per-row labels into corpus metrics with a confidence interval on every reported metric, exposed via a public API + new CLI command, reusing/generalising the existing IAA stats machinery.

Adds docs/design/eval-scoring-uncertainty.md and an index entry in docs/design/README.md. Docs-only - no code changes.

Proposed shape

• core/stats.py (new) — shared wilson_interval + percentile_bootstrap; refactor iaa.py:bootstrap_alpha to reuse.
• core/eval/metrics.py (new) — pure per-query metric functions matching the taxonomy.
• core/eval/scoring.py (new) — run_scoring(frame, ...), pure compute mirroring run_iaa.
• api/eval_score.py (new) — score_eval(...), the I/O wrapper mirroring compute_iaa (resolve settings/paths, read frame, write *_scores.json).
• cli/commands/eval.py (new) — pragmata eval score.
• Output contract — nested MetricScore (point + CI + method + n) per metric, replacing bare Rate floats.

Uncertainty: Wilson for the 12 proportions, query-level percentile bootstrap for the 5 continuous metrics; query is the resampling (cluster) unit.

Status - draft for discussion

NB content is still rough in places, just an initial proposal!

[saschagobel]

Good design doc overall; it attaches well to the existing eval work and the proposed scoring layer is directionally right. My comments are mostly about tightening API/CLI boundaries, input contracts, uncertainty/reporting semantics, and resolving the remaining TODOs/open decisions before implementation.

[saschagobel]

"per-row labels" is a bit misleading as the metric grain is query/example-level. I'd rephrase to "turn row-level label columns into per-query values and corpus-level metrics", with retrieval called out as chunk rows grouped by query.

[saschagobel]

Confidence intervals need a precise scope statement. These CIs estimate sampling uncertainty over query examples; they do not capture annotator disagreement, evaluator/model label error, or benchmark construction bias. That distinction should be explicit before making "CI on every metric" part of the contract.

[saschagobel]

This background is outdated. The current flow should start at annotation export and then branch: human-labeled exports can go directly into pragmata eval score, while evaluator-based scoring goes through pragmata eval train followed by pragmata eval predict and then pragmata eval score. Scoring is the missing piece; train/predict are already in development and should be represented as part of the eval workflow rather than as an external tlmtc-only block.

[saschagobel]

This is accurate, but worth phrasing more precisely: paths, settings, and report schemas are already in place; what is missing is the scoring computation itself and the API/CLI wiring around it.

[saschagobel]

Correct, but the eval CLI should be framed as an eval command group. pragmata eval train and pragmata eval predict are expected to land first, and pragmata eval score should extend that surface rather than being designed independently.

[saschagobel]

I think this is mixing input and output identifiers. Annotation export_id / prediction run_id select the input; score_id identifies where score artifacts are written under eval/scores/<score_id>/. I'd keep score_id as an optional output identifier, defaulting to the generated value already provided by EvalScoreSettings, and avoid using export_id for output naming.

[saschagobel]

I'd make this a scoring-specific input contract rather than extending the train schemas. validate_eval_score_frame currently reuses _TRAIN_SCHEMAS_BY_TASK, but scoring has stricter structural requirements than training: retrieval needs record_uuid, rank, and chunk_id, while grounding/generation need record_uuid. I suggest adding dedicated *_SCORE_SCHEMA definitions in eval_input.py and have validate_eval_score_frame use those, so training inputs are not unnecessarily constrained by scoring-only metadata.

[saschagobel]

After the API/CLI discussion above, this should no longer be an open TODO. The design should settle on one eval API/CLI shape and document the input selectors and precedence rules explicitly.

[saschagobel]

K should be inferred from the ranked rows in the scoring input, not supplied as --top-k. Exposing top_k suggests a tunable choice where the scored artifact already has a fixed cutoff, and a wrong or post-hoc value would either bias the metric or force avoidable error handling.

[saschagobel]

I'd add two items to this list:

1. input selection semantics. Scoring appears to have three input selectors: direct labeled_input_path, annotation export_id, and prediction_run_id. The design should define which are mutually exclusive, which are required, and what (if any) fallback applies if none is supplied.

2. incomplete and degenerate scoring data. Retrieval metrics assume complete ranked chunk labels per query; if some chunks are unlabeled, we need a deliberate policy: fail with an informative message, skip affected queries, or compute an alternative fallback metric with clear caveats. Similarly, all-0/all-1 or otherwise degenerate labels may make some estimates uninformative and should be handled explicitly. Ideally the validation/guard logic is reusable between eval train and eval score where the constraints overlap.