Repository navigation
Add HumaneBench transcript-scoring Claude Code skill - #87
Closed
ErikaOnFire wants to merge 1 commit into
Closed
ErikaOnFire wants to merge 1 commit into
ErikaOnFire wants to merge 1 commit into
Conversation
Self-contained skill (embedded rubric v3) that scores existing AI conversation transcripts against the 8 HumaneBench principles. Claude-only default judge (sonnet-4-5) with opt-in cross-family --ensemble (Sonnet 4.5 + GPT-5.1 + Gemini 2.5 Pro). 22/22 unit tests pass. Staged for relocation to the humanebench repo (see MOVE.md). Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Deploying humanebench-prod with
|
| Latest commit: |
a1c5d64
|
| Status: | ✅ Deploy successful! |
| Preview URL: | https://eca659dd.humanebench-prod.pages.dev |
| Branch Preview URL: | https://sparkle-agent-ed662a71-0b6d.humanebench-prod.pages.dev |
Contributor
Author
|
Closing — this skill's intended home is the separate humanebench repo (alongside the rubric and harness), not the website repo. It's self-contained/portable; relocation guide is in the skill's MOVE.md. Not merging here. |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What
Adds a HumaneBench transcript-scoring Claude Code skill at
skills/humanebench-transcript-score/. It scores an existing AI conversation transcript against the eight HumaneBench principles (rubric v3), producing a per-principle breakdown (-1 / -0.5 / +0.5 / +1) and an overall HumaneScore.It's the transcript-scoring companion to the published harness in the
humanebenchrepo: that repo runs standardized scenarios through a model; this scores conversations you already have from a product.Why the ensemble matters
A single LLM judge inherits its own temperament, and LLM judges favor their own model family's outputs. On one real transcript, two single judges disagreed by 0.37 (0.13 vs 0.50) with three principles flipping sign. So the skill has two modes:
--ensemble(recommended)Every ensemble report surfaces per-judge scores, judge spread, and sign-flips, so divergence is reported rather than averaged away. The default judge is Sonnet 4.5 on purpose — it matches the published leaderboard so single-judge scores stay comparable.
Contents
SKILL.md,README.md,MOVE.md(relocation guide for thehumanebenchrepo)references/— embedded rubric v3, output template, transcript-format specscripts/—humanebench_score.py(Claude-only default;--ensemble),test_scoring.py(22 tests, no network),requirements.txtexamples/sample_transcript.txtTesting
22/22 pass, no keys/network needed. Judge network calls are thin wrappers over the official SDKs and are exercised separately with live keys (not done here — costs money).
Notes
humanebenchrepo (seeMOVE.md); it's self-contained and portable.🤖 Generated with Claude Code
https://claude.ai/code/session_01FAzJ1KraFKVMFVWsYE4EWd