feat(evals): build drafts from one day of raw traces - #477
Conversation
Codex Review SummaryThis comment shows the latest Codex review activity on this pull request.
ℹ️ About Codex in GitHubYour team has set up Codex to review pull requests in this repo. Reviews are triggered when you
Codex reacts with 👀 while any review is running, comments if it has suggestions, and reacts with 👍 once all reviews finish with no findings. |
![devin-ai-integration[bot]](https://avatars.githubusercontent.com/in/811515?s=60&v=4){kind=small}
There was a problem hiding this comment.
Devin Review found 3 potential issues.
2 flags not posted on this PR by your GitHub settings — view them in Devin Review. (Configure)
![chatgpt-codex-connector[bot]](https://avatars.githubusercontent.com/in/1144995?s=60&v=4){kind=small}
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: 724895a572
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
- preserve rolling versus dated export resume semantics - validate timeout, cutoff, skipped-index scope, order, and uniqueness
1 participant
What changes
understudy evals buildis now the recommended coding-agent workflow for a captured workload. It resolves the selected project and workload, downloads one raw 24-hour trace window, writes a private and resumable local source package, and creates aneval-project.jsonhandoff for the coding agent.The user can select a complete UTC day with
--date YYYY-MM-DD; otherwise the command uses the latest rolling 24 hours. Downloads run concurrently, retry transient failures, verify capture identity and content hashes, and record unavailable captures explicitly instead of hiding them.The coding-agent handoff asks the agent to infer the workload goal, output contract, success criteria, execution modes, and failure taxonomy from the raw traces and repository. It then points to a deterministic local draft check. No model provider is called during source materialization.
Why
We want the backend to remain a generic authorized trace data plane while the customer’s coding agent performs workload-specific eval and verifier authoring. One day is the smallest useful default for tomorrow’s Cedar test and should be practical to download within the five-minute target through direct R2 downloads.
Contract changes
source/skipped.jsonlto the v2 project source contract.evals createcommand as legacy;evals buildis the front door.Validation
npm run checkgit diff --checkReview follow-up hardened timeout retries, rejected future ingestion cutoffs, preserved explicit-date versus rolling-window resume semantics, and fully validated the skipped-capture ledger.
Dependency
Requires https://github.com/understudylabs/understudy-platform/pull/647 to deploy the scoped raw trace endpoint.