Skip to content

About

Recurring personal agent-session benchmark. Score whether you accepted the output. HTML sheets + X cards. @BlackWolfNews

Resources

Stars

0 stars

Watchers

0 watching

Forks

Latest commit

 

History

2 Commits

Folders and files

Repository files navigation

session-bench

Recurring personal agent-session benchmark. Score whether you accepted the model's next output — not a lab leaderboard.

Inspired by @nkgoutham. Built by @BlackWolfNews (ProtoEngineer). First public skill/repo.

Public output is branded session-bench · @BlackWolfNews. You keep the HTML and PNG cards; post them on your own blog or X.

What you get

File Share?
sheets.html Yes — blog, gist, GitHub Pages
cards/card-01.png … Yes — X / social
private-report.md No — five next-session suggestions
submit.json Optional, stripped (counts + token burn, no quotes)
~/.session-bench/history.jsonl Local status checks (python3 -m session_bench status)

No InfoBlocks required. Logs on disk are enough.

Install

git clone https://github.com/BlackWolfNews/session-bench.git
cd session-bench
python3 -m session_bench extract -o ./run

Then tag run/batches/batch-*.json with any agent (see prompts/TAGGER.md and /session-bench), and:

python3 -m session_bench finish -o ./run --handle @you
python3 -m session_bench status

finish writes sheets, PNG cards (Chrome headless on macOS), the private report, tokens.json, submit.json, and a history snapshot.

Token burn

  • Session tokens — estimate from extracted user + assistant text (chars/4).
  • Analysis tokens — reported via --analysis-tokens / SESSION_BENCH_ANALYSIS_TOKENS, else estimate from digest+tag JSON.
  • Optional USD: SESSION_BENCH_USD_PER_MTOK.

Same numbers appear on sheet 09, in tokens.json, and in submit.json.

Compare / send back

submit.json has no transcripts. If you want a community pile later, that file is the unit — not your episodes.jsonl. v1 is opt-in: you publish it.

Local compare is automatic: the next finish shows avg/win/pain vs the last snapshot.

Skill

In Grok: copy skill/SKILL.md to ~/.grok/skills/session-bench/SKILL.md (or use this repo as cwd). Slash command: /session-bench.

License

MIT. Scoring rubric is harsh on purpose. Cells with fewer than 8 scored episodes get a ★.

About

Recurring personal agent-session benchmark. Score whether you accepted the output. HTML sheets + X cards. @BlackWolfNews

Resources

Stars

0 stars

Watchers

0 watching

Forks

Releases

Packages

Contributors

Languages