Repository navigation
Add kadamba-fit, a harder synthetic fixture with an answer list - #3
Merged
Merged
Conversation
examples/acme-demo overstates accuracy: its comments state its bugs and its key has one should-not-flag item. This fixture states no gap in any comment, doc line or identifier, and its key lists clean controls so false positives can be measured. The run sheet covers a blind audit from an exported commit and scoring by a separate session. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The first scored cold run found real gaps the key missed, and three should-not-flag notes were too broad or wrong. Narrow those notes and add known_gaps, so a matching finding counts as correct. Each gap was checked against the app, which is unchanged. The run sheet records the change; the first score stands against the original key. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
examples/acme-demooverstates Vigil's accuracy: its comments state its bugs, and its answer list has one should-not-flag item. This addsexamples/kadamba-fit, a harder synthetic app whose files state no gap, with an answer list that also names clean controls, so false positives can be measured.What's here
examples/kadamba-fit/: a made-up fitness and cycle-tracking app for members in India and the EU, in 55 files of code, migrations, deployment config, vendor contracts and policies. Every person, company, credential and domain is invented, and domains are reserved for documentation.examples/kadamba-fit.expected.json: 16 planted issues, 13 should-not-flag items, and known real gaps a finding may correctly report.examples/kadamba-fit.run-sheet.md: how to run a blind audit from an exported commit, and how a separate session scores it.CHANGELOG.mdline.First run
One cold audit ran on
masterat259d7e0, and a separate session scored it against the original answer list:The scorer judged two of the three flagged items to be defects in the answer list.
a2083c0narrows those notes and adds the real gaps the run found asknown_gaps. The app is unchanged, and the first score stands against the original list.These are still one run on synthetic code, scored by a model. The run's own output is gitignored and not in this PR.
Keep it blind
The answer list is public once this merges. A fair run copies the app out of the repo and never opens
examples/*.expected.jsonor the run sheet, as the run sheet describes.Checks
npm run validate -- --strict: 0 errors.npm test: 23 pass, 0 fail, 1 skipped. The branch already contains currentmasterand merges cleanly.Compliance tooling, not legal advice.
🤖 Generated with Claude Code