feat(evals): build and publish trace-derived evals locally

[aamir]

Summary

A developer can now ask their coding agent to turn a hosted Understudy workload into a checked, publishable eval without using the web app. The CLI downloads and authenticates the complete frozen week, keeps workload reasoning local, proves the verifier against independent good and wrong fixtures, and uploads only the exact owner-approved release.

This is the agent-tools half of the CLI-first eval workflow. It uses the companion platform PR for complete trace export and immutable release storage. Hosted model runs, GEPA, and cleanup of the old web authoring path remain separate follow-up work.

Related: understudylabs/understudy-platform#646

Workflow

1. capture-evidence runs understudy evals build --last 7d and materializes every receipt- and digest-verified export segment into one private local project.
2. The coding agent searches the compact execution index, reconstructs provable lineage, confirms the workload profile and metric with the owner, and authors representative tasks, coverage, environment, and verifier.
3. understudy evals check replays immutable module snapshots in fresh deterministic workers. It requires a known-good pass, an intentionally wrong rejection, repeatability, complete source accounting, and bound owner approvals.
4. understudy evals publish --preview prints the exact manifest, release ID, bundle hash, size, and ordered inventory without a network request.
5. Upload requires a matching --expect-release-id. Any local change invalidates the preview before the CLI sends data.

Data and execution boundaries

• The whole week is stored locally, but the agent can inspect the compact index and individual traces instead of placing the corpus in one prompt.
• Repository builds must use a child directory under .understudy; staging and downloaded payloads remain inside that ignored private tree, and an empty authenticated export fails before a project is created.
• The backend freezes ingestion at export start, at or after the exact seven-day trace window. The CLI omits a cutoff on page one, preserves the returned canonical cutoff across committed resumed segments, and safely restarts an interrupted first page so already-arrived, late-ingested rows from the week are included without caller-selected or later corpus drift.
• Each local trace must match the content SHA-256 in the authenticated export manifest. Fresh downloads and resumed same-size files fail closed on a digest mismatch.
• The backend and CLI independently derive the same ordered rolling commitment from every source row's identity, byte size, and content digest. The verified terminal export binds that exact corpus into a compact durable source attestation.
• The check report and publication hash the exact attestation token, and the backend verifies it against the frozen workload scope, totals, and corpus commitment before sealing a release.
• Publication sends exactly two reviewed objects: the displayed manifest, which contains the attestation and its hash, and its gzip bundle; the ordered inventory rejects file/descendant path overlaps so the archive can always be materialized exactly.
• The bundle excludes source/, raw traces, the expiring receipt and local export proof, the execution index, analysis, and mutable eval-project.json state.
• The historical incumbent answer is never treated as gold. Both positive and negative fixtures require independent correctness evidence.
• Build, check, and publish do not call a model, start a hosted eval, change a prompt, or mutate serving.
• The local JavaScript worker is a bounded deterministic verifier runtime, not an operating-system sandbox for arbitrary untrusted code.

Session-settled decisions carried from planning: the coding agent owns eval semantics; the source is a complete frozen week; the backend only transports evidence and stores the final release; publication is a separate explicit action.

Validation

• npm run check: 1,275 tests passed across 165 suites.
• Public skill validation: 43 skills passed.
• Package dry-run smoke passed.
• Packaged JSON Schemas name the exported runtime validators required for cross-field release invariants; platform golden publication/release bytes match.
• Review covered correctness, upload consent, local data boundaries, deterministic execution, API parity, reliability, tests, public-repo standards, and agent usability. Every actionable finding to date was fixed and its thread resolved.

Deliberately deferred

• Running models or prompts against the release, background benchmarks, GEPA, and result-ledger integration.
• Treating the local module worker as a general-purpose hostile-code sandbox.
• Removing the existing web eval authoring code before Cedar validates this flow.

Post-Deploy Monitoring & Validation

• Owner/window: an Understudy operator stays with Cedar through the first end-to-end run and reviews the first published release before any later benchmark work.
• Start with a synthetic or Understudy-owned workload. Verify evals build reaches a terminal seven-day receipt and that every local source count, byte total, and content digest matches the authenticated export.
• Verify evals check reports complete lineage accounting, explicit coverage gaps, one good pass, one wrong rejection, stable repeated results, and the final artifact hashes.
• Review the complete evals publish --preview inventory. Upload only after explicit permission and only with its exact --expect-release-id.
• Healthy publication returns the expected release ID; platform readback must return the same manifest and bundle hashes. A changed preview, source-accounting mismatch, unexpected upload file, 4xx scope/approval error, or 5xx is a stop signal.
• Rollback: revert the package change or keep Cedar on the previous CLI version. No eval command in this PR can modify live routing, prompts, models, or serving traffic.

---

---

[chatgpt-codex-connector]

Codex Review Summary

This comment shows the latest Codex review activity on this pull request.

Review	Status	Commit	Review trigger
📝 Code Review	✅ Completed 2026-08-31T09:26:18.886040Z	8b19cd2	Manual request

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

• Open a pull request for review
• Mark a draft as ready
• Comment "@codex review" or "@codex security review".

Codex reacts with 👀 while any review is running, comments if it has suggestions, and reacts with 👍 once all reviews finish with no findings.

[devin-ai-integration]

Devin Review found 4 potential issues.

[chatgpt-codex-connector]

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: 86a344f8e2

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

• Open a pull request for review
• Mark a draft as ready
• Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

[aamir]

@codex review

[chatgpt-codex-connector]

Codex Review: Didn't find any major issues. Bravo.

Reviewed commit: 5ac430e969

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

• Open a pull request for review
• Mark a draft as ready
• Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

[aamir]

@codex review

[chatgpt-codex-connector]

Codex Review: Didn't find any major issues. Bravo.

Reviewed commit: 4705607c61

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

• Open a pull request for review
• Mark a draft as ready
• Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

[aamir]

@codex review

[chatgpt-codex-connector]

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: 4c49fa85f5

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

• Open a pull request for review
• Mark a draft as ready
• Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

[aamir]

@codex review

[chatgpt-codex-connector]

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: 3ec1c43715

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

• Open a pull request for review
• Mark a draft as ready
• Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

[aamir]

@codex review

[aamir]

@codex review

[chatgpt-codex-connector]

Codex Review: Didn't find any major issues. Nice work!

Reviewed commit: 8b19cd2541

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

• Open a pull request for review
• Mark a draft as ready
• Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".