Repository navigation
test: AI評価の検出率、誤検出、揺れを手元のセッションで測る仕組みを作る（#77） - #87
Merged
Merged
Conversation
- tests/ai-eval/に、対の文、評価する文書の準備、結果のまとめ、集計のスクリプトと手順を置いた - 評価はCIやAPIでなく、手元のClaude Codeのセッションのサブエージェントが行う - 禁止語のAIルール5つの対の文25対を、claude-opus-5-5で2回評価した結果を残した - CIでは、対の文と結果のファイルの形と、集計が動くことだけを確かめる Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
|
Thank you for your contribution! 🎉 |
This was referenced Nov 3, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
🙋 誰が出すか（Author）
✍️ 人が書く要約（Human-authored summary）
👏 解決する issue / Resolved Issues
📝 関連する issue / Related Issues
⛏ 変更内容 / Details of Changes
AI評価（
prompts/evaluate.md）の検出率、誤検出、実行ごとの揺れを測る仕組みを、tests/ai-eval/に作った。評価はCIやAPIではなく、手元のClaude Codeのセッションのサブエージェントが行う。APIの従量課金をかけないためである。tests/ai-eval/pairs/<ルールのファイル名>.jsonltests/corpus/pairs/と同じtests/ai-eval/prepare.mjstests/ai-eval/collect.mjstests/ai-eval/report.mjstests/ai-eval/results/tests/ai-eval/README.mdtests/ai-eval.test.mjs試しに、禁止語のAIルール5つ（25対）を、
claude-opus-5-5で2回評価した。残りのAIで評価するルール（約52件）は、ルールのファイルごとに別のPRで足す。
🔎 既存のPR / Existing PRs
🔍 確認したこと / Verification
skills/docs-rules/rules/のルールに照らして点検した（node --test tests/*.test.mjsの採点のテストを含め、67件が通った）tests/ai-eval/を足した）.claude-plugin/やスキルの置き場所を変えた場合、CONTRIBUTING.mdの「プラグイン配布の検証」を通した（claude plugin validate .が通った）python3 tests/test_skill_size.pyを通した🤖 Generated with Claude Code