Skip to content

[BENCH-1131] Add check-s2s to check the S2S fetch settings against Coval - #744

Merged
seribaymadina merged 3 commits into
mainfrom
madina/bench-1131-check-s2s
Oct 19, 2026
Merged

seribaymadina merged 3 commits into
mainfrom
madina/bench-1131-check-s2s

Conversation

@seribaymadina

Copy link
Copy Markdown
Contributor
  • Problem: a wrong Coval ID in the fetch settings only shows up at the 05:00 fetch, so the board can miss a day.
  • Fix: coval-bench check-s2s checks the fetch settings against Coval before or after an apply.
  • Input: the env, or --settings-file with the planned env from an infra PR.
  • Checks: the runner accepts the settings, every scenario × model has recent runs per condition, and every persona is mapped.
  • Output: a table of runs found, and a non-zero exit on any flag.
  • Safety: read-only. It never writes to the database or to Coval.

@seribaymadina
seribaymadina marked this pull request as ready for review October 16, 2026 22:39
@seribaymadina
seribaymadina requested a review from a team as a code owner October 16, 2026 22:39
@chatgpt-codex-connector

chatgpt-codex-connector Bot commented Oct 16, 2026 •

Copy link
Copy Markdown

Codex Review Summary

This comment shows the latest Codex review activity on this pull request.

Review Status Commit Review trigger
📝 Code Review ✅ Completed 2026-10-16T22:41:13.461432Z f68643f Draft marked ready
ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review" or "@codex security review".

Codex reacts with 👀 while any review is running, comments if it has suggestions, and reacts with 👍 once all reviews finish with no findings.

coval-cale
coval-cale previously approved these changes Oct 16, 2026

@coval-cale coval-cale left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Oh this is SLICK!!!!

@chatgpt-codex-connector chatgpt-codex-connector Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: f68643fb6b

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

Comment thread runner/src/coval_bench/s2s/check.py
@seribaymadina
seribaymadina added this pull request to the merge queue Oct 19, 2026
Merged via the queue into main with commit 40c971f Oct 19, 2026
5 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants