Repository navigation
Conversation
…ng for agent tasks and live benchmark runs One ordered event format (t3mp3st_agent_task_event/v1) for what an agent actually did, three producers, one grader: - Event log: append-only JSONL, log-assigned seq, logical clock (ts === seq) so a scripted run is byte-reproducible. Strict validation on parse; exactly one task_started (first) and one task_finished. - Task contract (bench/tasks/<id>/task.json, t3mp3st_agent_task/v1): prompt, allowed tools, an event-policy grader (required calls with argument constraints, denied-call budget, sha256-pinned artifacts) and an optional deterministic stub script. Negative fixtures pin the check each must fail. - Grader: named checks, outcomes pass | fail | abstain | infrastructure_error. A stream with dropped events grades infrastructure_error, so an unloggable scope denial can never disappear from a passing run. - Live benchmark runs: `cybench-bench.mjs --events-out DIR` writes one stream per run from the live-tools loop (each dispatched command, integrity-gate refusals as tool_denied, the run outcome) and points the report row at it. Off by default; the default path and report are unchanged. Flags are still graded by the benchmark's own scorer. - Arsenal: tool:executed / tool:error carry secret-redacted call arguments (computed only when something subscribes); tapArsenal mirrors them into the log and is fault-isolated so logging never changes what execute() returns. Runner guarantees: fresh workspace or refusal, reserved bundle names, a backend crash still persists the partial log and a verdict. Tests: src/__tests__/agent-task.test.ts (vitest), npm run test:agent-task (committed task + negatives), npm run test:cybench-events (mocked model, synthetic challenge, no network). Both scripts join test:release:contracts.
vu1n
marked this pull request as ready for review
November 12, 2026 13:02
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Bottom line: every tool call an agent makes can now be written to an ordered event log and graded pass or fail, for scripted tasks, Arsenal calls and live benchmark runs. It is opt-in: default benchmark runs and reports are unchanged. To try it:
npm run build && npm run test:agent-task && npm run test:cybench-events(no network, no corpus, no key).What it adds:
src/benchmark/task-events.ts, schemat3mp3st_agent_task_event/v1): append-only JSONL. The log assignsseqand uses a logical clock (ts === seq), so a scripted run is byte-identical on any machine. Parsing is strict, with exactly onetask_startedand onetask_finished.bench/tasks/<id>/task.json): the prompt, the allowed tools, and anevent-policygrader (required calls with argument constraints, a denied-call budget, sha256-pinned artifacts). A task can carry a deterministic stub script, and negative fixtures name the check each one must fail.src/benchmark/task-grader.ts): named checks and four outcomes:pass,fail,abstain,infrastructure_error. A stream with dropped events gradesinfrastructure_error, so a scope denial that could not be logged can never vanish from a passing run.node scripts/cybench-bench.mjs --hunter live-tools --events-out DIRwrites one stream per run from the live-tools loop. Each dispatched command is atool_call(arguments, duration, capped result preview), an integrity-gate refusal is atool_denied, and the run outcome closes the stream. The report row gainsevents: { path, count, dropped }. It is off by default, and without the flag nothing is loaded and the report is unchanged. Flags are still graded by the benchmark's existing scorer; this PR adds no second flag grader.tool:executedandtool:errornow carry secret-redacted call arguments, computed only when something subscribes.tapArsenalmirrors them into the log and is fault-isolated, so logging never changes whatexecute()returns.Why: the repo can already say whether a run got the flag. This makes how it got there reviewable and gradable (which tools, with what arguments, what was refused), in one format shared by scripted tasks, Arsenal calls and live benchmark runs.
Not covered: a bare local-agent CLI run (Claude Code / Codex) is still not observable per tool call, because its tool use sits outside both paths.
Linked issue
None.
Prior work and attribution
Not derived from #163.
Contribution receipt
fetchwith a scripted mock and uses a dummy key)agent-taskandcybench-bench; tool access mocked/local_only; target class static_fixturenpm run verify-claimspasses.events.jsonlas operator-private.bench/tasks/README.mdsays so.abstain), separate fromfail.Verification
npm run typechecknpm testnpm run doctorExact commands and results (all on this head, Node 22, Linux):
npm run test:prnpm run test:pr-coveragewithCOVERAGE_BASE=upstream/mainnpm run test:agent-tasknpm run test:cybench-eventsnpm run test:cybench-ci,npm run test:flag-gradingnpm run verify-claims,npm run test:no-phantom-tools,npm run docs:checknpm run test:releaseSkipped:
npm run arsenal:smokeandnpm run field:drill(they need the API server running; this change does not touch the server).Risk and rollback
recordToolEventcall next to the existingbashTooldispatch. It is a no-op without--events-out, and a test asserts the tool output is identical with and without a log attached.--events-outneedsnpm run buildand exits 2 with a clear message ifdist/is missing. Dropped events on a benchmark run are reported (events.dropped) but do not change the flag verdict.Delivery checks
upstream/main(15 files: 12 added, 3 modified, no deletions)