Repository navigation
Golden set: 29 additional issues on the cal.com PRs, each with public evidence - #66
Open
pranav100000 wants to merge 1 commit into
Open
pranav100000 wants to merge 1 commit into
pranav100000 wants to merge 1 commit into
Conversation
Each added comment is an issue the current gold set lacks, with a public review comment stating the trigger, and in most cases the same finding among other published tools' unmatched candidates or a recorded reproduction. See the pull request description for the per-entry evidence table. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Golden set: 29 additional issues on the calcom/cal.com PRs
Each entry below is an issue that a review of the benchmark PR surfaced, that the current gold set does not contain, and that is corroborated by at least one of: a recorded reproduction in the review, or the same finding appearing among other published tools' unmatched candidates in
results/anthropic_claude-opus-4-5-20251101/evaluations.json(the methodology's "multiple strong tools agree against the gold set" signal).Disclosure: these were found by Aether Review, a vendor tool, on public forks in the
aether-benchGitHub org. Every linked review comment is public and states the trigger; the entries stand or fall on that evidence.Adding them changes the gold set for every tool, so this is a new golden-set version in the sense of #52; re-running step 3 against stored candidates re-scores existing tools without re-running them.