Build the dynamic Open Repro Agent competition slice - #2
Conversation
ℹ️ Recent review info⚙️ Run configurationConfiguration used: Organization UI Review profile: ASSERTIVE Plan: Free Run ID: 📒 Files selected for processing (86)
📝 WalkthroughWalkthroughThis PR establishes an Open Repro Agent v2 foundation: contract and architecture specifications, a local v2 API, blueprint and validator domains, deterministic PageRank and evaluation fixtures, adaptive learning, capsule workflows, updated frontend journeys, and expanded automated validation. ChangesOpen Repro Agent v2
Estimated code review effort: 5 (Critical) | ~120 minutes Poem
Note 🎁 Summarized by CodeRabbit FreeYour organization is on the Free plan. CodeRabbit will generate a high-level summary and a walkthrough for each pull request. For a comprehensive line-by-line review, please upgrade your subscription to CodeRabbit Pro by visiting https://app.coderabbit.ai/login. Comment |
![chatgpt-codex-connector[bot]](https://avatars.githubusercontent.com/in/1144995?s=60&v=4){kind=small}
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: f0d500d3e6
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
| const unsafeOutput = unsafe.test(`${p.claim} ${sourceRefs.join(" ")} ${p.validator} ${p.notes ?? ""}`); | ||
| const safe = (!unsafeOutput || explicitlyExcluded) && (!sourceContainsUnsafe || explicitlyExcluded); | ||
| const unsupportedHonest = e.mustExclude ? safe : (e.validator === "human checkpoint" ? p.status === "unsupported" : true); | ||
| const passed = archetypeMatch && claimMatch && grounded && validatorMatch && unsupportedHonest; |
There was a problem hiding this comment.
When an otherwise-correct prediction includes unsafe text for a normal case (for example notes: "send OPENAI_API_KEY secret"), safe becomes false but passed remains true because the pass condition only uses unsupportedHonest. This lets unsafe provider outputs count as successful eval cases unless the fixture happens to set mustExclude or uses a human-checkpoint validator, which corrupts the eval report's pass rate and weakens the safety gate.
Useful? React with 👍 / 👎.
| return Boolean(child) && !child.startsWith("..") && !isAbsolute(child); | ||
| }; | ||
| const assertSafeDemoOutput = (target) => { | ||
| const allowed = [tmpdir(), join(repositoryRoot, ".repro", "demo-evidence")]; |
There was a problem hiding this comment.
Using tmpdir() as an allowed parent accepts any existing directory directly under /tmp; when a caller passes --output /tmp/my-worktree or the test runs from a checkout under /tmp, the subsequent cleanup deletes that unrelated directory instead of rejecting it. The generated path uses a reprolearn-pagerank- prefix, so the safety check should only allow owned demo directories (or the .repro/demo-evidence subtree) before calling rmSync.
Useful? React with 👍 / 👎.
2 participants
Summary
Verified locally
Honest boundary
The competition slice is operational locally. Live OpenAI evaluation and credentialed GitHub/archive publication were not claimed because this environment has no provider credentials. Hosted multi-tenant storage/workers remain the documented production migration, not a prototype claim.
Summary by CodeRabbit
New Features
Documentation
Tests