Skip to content

feat(benchmark): observable tool-call event logs with pass/fail grading for agent tasks and live benchmark runs - #232

Open
vu1n wants to merge 1 commit into
elder-plinius:mainfrom
vu1n:feat/agent-task-events
Open

vu1n wants to merge 1 commit into
elder-plinius:mainfrom
vu1n:feat/agent-task-events

Conversation

@vu1n

@vu1n vu1n commented Nov 12, 2026

Copy link
Copy Markdown

Summary

Bottom line: every tool call an agent makes can now be written to an ordered event log and graded pass or fail, for scripted tasks, Arsenal calls and live benchmark runs. It is opt-in: default benchmark runs and reports are unchanged. To try it: npm run build && npm run test:agent-task && npm run test:cybench-events (no network, no corpus, no key).

What it adds:

  • Event log (src/benchmark/task-events.ts, schema t3mp3st_agent_task_event/v1): append-only JSONL. The log assigns seq and uses a logical clock (ts === seq), so a scripted run is byte-identical on any machine. Parsing is strict, with exactly one task_started and one task_finished.
  • Task contract (bench/tasks/<id>/task.json): the prompt, the allowed tools, and an event-policy grader (required calls with argument constraints, a denied-call budget, sha256-pinned artifacts). A task can carry a deterministic stub script, and negative fixtures name the check each one must fail.
  • Grader (src/benchmark/task-grader.ts): named checks and four outcomes: pass, fail, abstain, infrastructure_error. A stream with dropped events grades infrastructure_error, so a scope denial that could not be logged can never vanish from a passing run.
  • Live benchmark runs: node scripts/cybench-bench.mjs --hunter live-tools --events-out DIR writes one stream per run from the live-tools loop. Each dispatched command is a tool_call (arguments, duration, capped result preview), an integrity-gate refusal is a tool_denied, and the run outcome closes the stream. The report row gains events: { path, count, dropped }. It is off by default, and without the flag nothing is loaded and the report is unchanged. Flags are still graded by the benchmark's existing scorer; this PR adds no second flag grader.
  • Arsenal: tool:executed and tool:error now carry secret-redacted call arguments, computed only when something subscribes. tapArsenal mirrors them into the log and is fault-isolated, so logging never changes what execute() returns.

Why: the repo can already say whether a run got the flag. This makes how it got there reviewable and gradable (which tools, with what arguments, what was refused), in one format shared by scripted tasks, Arsenal calls and live benchmark runs.

Not covered: a bare local-agent CLI run (Claude Code / Codex) is still not observable per tool call, because its tool use sits outside both paths.

Linked issue

None.

Prior work and attribution

  • Prior PR or issue used: none
  • Reused/adapted areas: not applicable
  • Fresh verification performed for reused material: not applicable

Not derived from #163.

Contribution receipt

  • Scope class: static_fixture
  • Target authority: not_applicable (scripted stub runs and a synthetic one-file challenge staged in a temp dir)
  • Network use: none (the live-loop test replaces fetch with a scripted mock and uses a dummy key)
  • Run mode labels: static_test, mocked
  • Model/harness labels: model not_applicable; harness agent-task and cybench-bench; tool access mocked/local_only; target class static_fixture
  • Claims or evidence changed: none. No README or headline number changes. npm run verify-claims passes.
  • Redaction/provenance notes: Arsenal event arguments go through the existing configured-secret scrubber, which only masks configured target-header secrets. A credential the model puts in an ordinary argument is recorded as-is. Benchmark streams record each command verbatim, as the report transcript already does. Treat events.jsonl as operator-private. bench/tasks/README.md says so.
  • Abstentions/refusals: graded as their own outcome (abstain), separate from fail.

Verification

  • npm run typecheck
  • npm test
  • npm run doctor
  • Risk-specific checks are listed below

Exact commands and results (all on this head, Node 22, Linux):

Command Result
npm run test:pr pass (90 files, 1013 tests; doctor warnings only for optional tools not installed)
npm run test:pr-coverage with COVERAGE_BASE=upstream/main pass, changed executable lines 496/496 (100%)
npm run test:agent-task pass (committed task plus 3 negative fixtures graded as pinned)
npm run test:cybench-events pass (23 checks: helpers, mocked live-tools loop, CLI on/off/repeat)
npm run test:cybench-ci, npm run test:flag-grading pass (existing benchmark contracts unchanged)
npm run verify-claims, npm run test:no-phantom-tools, npm run docs:check pass
npm run test:release pass

Skipped: npm run arsenal:smoke and npm run field:drill (they need the API server running; this change does not touch the server).

Risk and rollback

  • Residual risk: the live-tools insertion is one recordToolEvent call next to the existing bashTool dispatch. It is a no-op without --events-out, and a test asserts the tool output is identical with and without a log attached. --events-out needs npm run build and exits 2 with a clear message if dist/ is missing. Dropped events on a benchmark run are reported (events.dropped) but do not change the flag verdict.
  • Rollback: revert the squash commit. Nothing reads the new files or report field unless the flag is set.

Delivery checks

  • Diff is scoped against current upstream/main (15 files: 12 added, 3 modified, no deletions)
  • No unrelated protected evidence, safety tests, or provider/config files were removed
  • Published review history was not force-pushed
  • Reused or adapted prior work is identified, re-verified, and credited (none used)
  • Exact-head PR CI is green (including the 50% changed-line coverage floor)
  • I understand PR acceptance is not release certification

…ng for agent tasks and live benchmark runs

One ordered event format (t3mp3st_agent_task_event/v1) for what an agent
actually did, three producers, one grader:

- Event log: append-only JSONL, log-assigned seq, logical clock (ts === seq)
  so a scripted run is byte-reproducible. Strict validation on parse; exactly
  one task_started (first) and one task_finished.
- Task contract (bench/tasks/<id>/task.json, t3mp3st_agent_task/v1): prompt,
  allowed tools, an event-policy grader (required calls with argument
  constraints, denied-call budget, sha256-pinned artifacts) and an optional
  deterministic stub script. Negative fixtures pin the check each must fail.
- Grader: named checks, outcomes pass | fail | abstain | infrastructure_error.
  A stream with dropped events grades infrastructure_error, so an unloggable
  scope denial can never disappear from a passing run.
- Live benchmark runs: `cybench-bench.mjs --events-out DIR` writes one stream
  per run from the live-tools loop (each dispatched command, integrity-gate
  refusals as tool_denied, the run outcome) and points the report row at it.
  Off by default; the default path and report are unchanged. Flags are still
  graded by the benchmark's own scorer.
- Arsenal: tool:executed / tool:error carry secret-redacted call arguments
  (computed only when something subscribes); tapArsenal mirrors them into the
  log and is fault-isolated so logging never changes what execute() returns.

Runner guarantees: fresh workspace or refusal, reserved bundle names, a
backend crash still persists the partial log and a verdict.

Tests: src/__tests__/agent-task.test.ts (vitest), npm run test:agent-task
(committed task + negatives), npm run test:cybench-events (mocked model,
synthetic challenge, no network). Both scripts join test:release:contracts.
@vu1n
vu1n marked this pull request as ready for review November 12, 2026 13:02
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant