Build execution-derived eval cohorts and review packets

[lluisinthedesert]

Problem

The current understudy evals flow can freeze a deterministic recent cohort, but its catalog metadata only supports request-level filters such as model, status, tool presence, and structured output. A random recent sample can therefore miss rare or high-consequence behaviors whose complexity is only visible after the execution finishes.

That leaves agents manually joining completed executions back to captures, discovering hard paths late, and rebuilding spot-check packets by hand. Fixed small cohorts also become accidental evidence caps: agents stop when the initial sample is convenient instead of collecting enough data for the decision.

Desired state

Extend the existing cohort workflow so a developer can discover, expand, balance, freeze, and inspect cohorts by completed-execution behavior while keeping payloads local.

• Derive configurable strata from local materialized traces: terminal outcome, tool/action classes, write vs no-op, error/recovery, turn count, and user-supplied tags.
• Join those derived strata back to capture and eval row ids.
• Select deterministically toward a decision-sized per-stratum plan, and report every uncovered or underfilled stratum.
• Support incremental cohort expansion and emit stability or uncertainty evidence after each meaningful batch.
• Treat pilot sizes as minimums, never caps; include every available rare or high-consequence case when tractable.
• Preserve an accumulator of rare/high-consequence cases so later random windows do not lose them.
• Generate a local redacted review packet with coverage counts, baseline/candidate result refs, scorer rationale, and exact trace/log drilldown paths.
• Let developers add human-guided value/consumer tags during drilldown; do not infer a universal low-value taxonomy automatically.

Proposed surface

Build on understudy evals catalog|create|cohort rather than creating a parallel workflow. Keep server catalogs metadata-only; any payload-derived classification should run after explicit local export approval.

Acceptance criteria

• A synthetic tool workflow demonstrates simple, multi-turn read→write, recovery, and rare/high-consequence strata.
• The command emits deterministic selection plus source/captured/split counts and selection rules per stratum.
• Cohorts can expand incrementally without mutating prior frozen splits, and each increment reports whether material metrics are stable within a declared decision tolerance.
• An important underfilled stratum or unstable material result is machine-readable and prevents a whole-workload-ready status.
• The review packet links selected rows to local trace and eval artifacts without embedding raw prompts or completions, while retaining counts and links for the full evaluated cohort.
• Frozen split and holdout boundaries remain unchanged; expanding coverage creates a new split contract.
• Tests prove payload-derived fields never enter stdout, the redacted catalog, telemetry, or committed fixtures.

Why this is roadmap work

The public skills can immediately enforce adaptive data-sufficiency and row-review gates, but durable execution-derived sampling needs CLI schemas, incremental cohort mechanics, local trace parsing, artifact contracts, and likely compatible server metadata/API changes. That is broader than a safe skills-only patch.

[Anirudh2627]

Posting the design I implemented in #480 as a first phase of this, so the approach is reviewable in one place.

Scope (phase 1): local stratified cohort planning from materialized traces.

A new understudy evals stratify command + src/evals/cohort-strata.ts that reads the foundry's local execution-index.jsonl (+ optional tasks.jsonl) and emits a versioned plan artifact (understudy.eval-cohort-strata-plan.v1). Server catalogs stay metadata-only; everything here is payload-local, per the issue.

Strata derived from completed-execution behavior

• outcome — terminal lineage status
• mode — write vs read-only vs no-tools, reusing the foundry's exported isMutatingTool classifier so selection and replay score writes with the same rule
• turns — single / short / long capture-count buckets
• confidence — foundry machine confidence
• tag — human-supplied only (--tags), never auto-inferred, per "do not infer a universal low-value taxonomy"

Selection semantics

• Deterministic seeded ordering (byte-identical plans for same seed/inputs)
• Pilot sizes are minimums, never caps: rare strata (≤ --rare-threshold) and --high-consequence-tags strata are saturated — every available rare/high-consequence case is included
• Greedy fill toward per-stratum targets in axis priority order, counting members already selected by earlier strata

Incremental expansion

• --expand-from <plan> preserves every prior frozen selection (split boundaries unchanged) and rejects seed/axis drift
• Reports whether the new batch moved stratum prevalence beyond --tolerance (stable/unstable + per-stratum moved list)

Readiness evidence

• Underfilled rare/high-consequence strata populate machine-readable blocking[] and set ready:false, so a later gate can block whole-workload-ready status on this evidence

Privacy contract

• The module only reads metadata fields; payload-derived fields (e.g. task titles) are never read, and tests assert no payload text can appear in the plan, stdout, or --json output.

Deliberately deferred to later phases: joining strata back to hosted capture/eval row ids, the redacted review packet with drilldown links, and the rare-case accumulator across random windows. Happy to reshape any of the above — in particular whether prevalence is the right stability metric vs. a scorer-driven material metric — before going deeper.