Skip to content

feat(benchmark): agent task contract — reproducible tasks with observable tool calls and explicit pass/fail grading - #231

Closed
vu1n wants to merge 5 commits into
elder-plinius:mainfrom
vu1n:feat/agent-task-contract
Closed

vu1n wants to merge 5 commits into
elder-plinius:mainfrom
vu1n:feat/agent-task-contract

Conversation

@vu1n

@vu1n vu1n commented Nov 12, 2026 •

Copy link
Copy Markdown

What

A committed agent task contract (t3mp3st_agent_task/v1) plus the runner/grader machinery that makes an agent run reproducible, observable, and explicitly graded — the same discipline verify-claims applies to headline numbers, applied to individual agent tasks.

  • Task format (bench/tasks/<id>/task.json): prompt, allowed tools, timeout, grader, and an optional deterministic stub script — strictly validated, fail-closed on malformed files.
  • Ordered event stream (events.jsonl): one JSON line per observable step — every tool call with its secret-redacted arguments, every scope/approval denial, notes, and the final answer. Logical clock (ts === seq) so a stub run is byte-reproducible on any machine.
  • Graders with named success AND failure criteria:
    • flag-exact — CTF-shaped exact-match oracle;
    • event-policy — process-shaped: required calls (tool + min count + argument constraints), denied-call budget (default: any tool_denied fails the run), sha256-pinned artifacts.
    • Outcomes follow the model-matrix taxonomy: pass / fail / abstain / infrastructure_error (infrastructure errors are never capability misses).
  • Negative fixtures pin the failure criteria: each committed bad run must fail on its named check (scope-denial, missing-call, wrong-artifact, wrong-flag).
  • Arsenal observability: tool:executed / tool:error events now carry redacted parameters, and tapArsenal() mirrors live Arsenal calls (and gate refusals) into the same event format. The tap is fault-isolated: a logging failure drops the event, never the tool result.
  • scripts/agent-task.mjs: --list / --task <id> / self-test via npm run test:agent-task; two seed tasks (egress-receipt-check, flag-oracle-demo) naming only registered tools (http_request, header_analysis, jwt_decode), guarded by an advertised-=-wired test over bench/tasks/; docs in bench/tasks/README.md.

Scope (read this first)

This PR ships the contract + event-stream + grading layer and a deterministic stub backend, plus the Arsenal tap hook. It does not wire a live LLM or server backend — those are follow-ups, as is wiring tapArsenal into a live runner outside tests.

Why a new module rather than extending scripts/cybench-bench.mjs: cybench is a single-suite harness whose per-run artifact is a prose transcript (stripped for privacy, so verdicts re-derive only at the graded-verdict level) with flag-only oracles. The task contract gives a schema any suite can adopt — structured per-call events, pluggable graders, argument-aware required-call checks — without entangling cybench's committed artifacts or its privacy-stripping guarantees. Migrating cybench tasks onto this format (and a t3mp3st-server backend that drives /api/general/auto with the tap attached) is the natural next PR.

Runner guarantees (review-hardened)

  • A provided workspace must be empty or nonexistent — a leftover artifact from an earlier run can never re-pass an artifact check.
  • events.jsonl / verdict.json are runner-reserved; stub/grader paths cannot shadow them.
  • A backend exception persists the partial event log and grades as infrastructure_error — the bundle is always diagnosable.
  • Exactly one task_started (first) and one task_finished per stream, enforced on append and on JSONL parse. A backend that finishes and then throws keeps its graded finish; the throw is preserved as a note.
  • Dropped events fail closed. The grader accepts the TaskEventLog itself (which carries its dropped count), so drop-aware grading cannot forget it; Tap subscribers are fault-isolated (a logging failure never changes a tool result), and drops are counted: any dropped event grades infrastructure_error — an unloggable scope denial can never quietly vanish from a passing run.
  • Empty tool output / empty answers are valid events (a success with no output stays a success).

Contribution Receipt

  • Change: new agent-task contract + event stream + graders + CLI + two seed tasks (all stub-backed, deterministic); additive parameters on Arsenal tool events (computed only when a subscriber is listening).
  • Scope class: static_fixture
  • Target authority: not_applicable (stub backend executes no tools, touches no network)
  • Network use: none
  • Run mode labels: static_test, mocked, tool_backed (Arsenal tap path only)
  • Model/harness labels: model not_applicable; harness agent-task (new); agent_runtime none; tool_access mocked; target_class static_fixture; attempts: 2 tasks + 4 negative fixtures per self-test run; successes: 2 tasks; failures: 4 negative fixtures (by design, each on its pinned check); abstentions: 0.
  • Commands run (at head 98f37ba, clean clone of main @ 29824d5 + patch):
    • npm run lint → pass (0 errors; pre-existing warnings unchanged)
    • npm run typecheck → pass
    • npm run test:pr → pass end-to-end (lint 0 errors, typecheck, npm test 90 files / 1014 tests incl. ctf-rsa-static, doctor PASS, 0 blockers) — run with homebrew python 3.15 first on PATH; the system python3 (3.9) cannot parse the CTF challenge's int | str unions, a pre-existing environment quirk, not a repo defect
    • npm run doctor → pass (optional-tool warnings only)
    • npm run verify-claims → pass, 27/27 (no claims touched)
    • npm run build && npm run test:agent-task → pass (all tasks + negatives graded as pinned)
    • changed-line coverage (vitest --coverage --config vitest.pr.config.ts + check-changed-line-coverage origin/main) → 506/506 (100%; required 50%)
  • Artifacts: bench/tasks/README.md, two seed task dirs with negative fixtures; run bundles are written to fresh temp dirs on demand, never committed.
  • Redaction: Arsenal-emitted parameters pass through the configured-secret scrubber (same path as results) — that scrubber masks only configured target-header secrets; a credential the model itself places in an ordinary parameter is recorded as-is, documented in bench/tasks/README.md. Event previews are capped; stub fixtures contain no real hosts, flags, or secrets.
  • Claims changed: none — verify-claims output is byte-identical before/after.
  • Abstentions/refusals: n/a (no model behavior measured).
  • Residual risk: stub backend replays scripts (that is what makes grading CI-safe); live LLM / local-agent / server backends are follow-ups, and a bare local-agent CLI run is not yet per-tool-call observable (outside the Arsenal receipt boundary). timeoutMs is declared in the contract but enforcement is the backend's responsibility in v1. Instrumented events include model-supplied parameter values (see Redaction).

Prior work and attribution

No prior T3MP3ST PR is used as a code source. The format follows existing repo patterns: cybench challenge dirs, the model-matrix outcome taxonomy, recovery-snapshot-style strict validation, and the no-phantom-tools advertised-=-wired discipline (extended over bench/tasks/ by a new guard test). External inspiration (own prior systems: sealed-session eval harness with per-session event logs) noted; all code here is fresh and verified at this head.

…able tool calls and explicit pass/fail grading

- t3mp3st_agent_task/v1: committed task dirs (prompt, allowed tools, grader,
  deterministic stub script) with strict fail-closed validation
- ordered append-only event stream (events.jsonl): every tool call with
  secret-redacted args, every scope/approval denial, logical clock so stub
  runs are byte-reproducible
- graders: flag-exact (CTF oracle) and event-policy (required calls, denied-call
  budget, sha256-pinned artifacts); outcomes pass/fail/abstain/infrastructure_error
- negative fixtures pin failure criteria: each committed bad run must fail on
  its named check (scope denial, missing call, wrong artifact, wrong flag)
- Arsenal tool events now carry redacted parameters so live runs are observable
  per-call; tapArsenal mirrors them (and gate refusals) into the same stream
- scripts/agent-task.mjs: --list / --task / --self-test; two seed tasks
@vu1n
vu1n marked this pull request as draft November 12, 2026 09:08
vu1n added 4 commits November 12, 2026 18:12
…ents, fresh workspaces, crash persistence, no phantom tools

Review findings fixed (all with regression tests):
- empty tool output/answer no longer throws from the event logger; the tap
  wraps its subscribers so instrumentation can never flip an Arsenal success
  into an ExecutionError or replace a SCOPE DENIED result with an exception
  (tested with an empty-output tool and an over-long tool name)
- runner refuses a non-empty workspace: a stale artifact from an earlier run
  can no longer satisfy an artifact check (false pass); stub/grader artifact
  paths may not shadow events.jsonl/verdict.json (rejected at validation)
- a backend exception now persists the partial event log and grades as
  infrastructure_error instead of losing the bundle
- task fixtures renamed to registered tools (http_request, header_analysis,
  hash_crack) and a guard test enforces advertised = wired over bench/tasks
- exactly-one task_started/task_finished enforced in the log and on JSONL parse
- CLI: fileURLToPath (paths with spaces), harness loaded once for --list;
  npm run test:agent-task wired into test:release:contracts
- Arsenal: redacted parameters computed once per execute and only when a
  subscriber is listening; README documents the redaction scope honestly
…ts verdict, jwt_decode seed task

- tap drops are now counted: any dropped event grades infrastructure_error
  (no_dropped_events check) so an unloggable scope denial can never vanish
  from a passing run; the count is recorded on the verdict and as a note
- a backend that finishes and then throws keeps its graded finish; the
  post-finish throw is preserved as a note instead of an illegal second
  task_finished (which previously aborted the run before writing the bundle)
- flag-oracle-demo seed task uses jwt_decode (harmless, plausible decode of a
  real base64url JWT payload) instead of hash_crack
- tests renamed to describe behavior, not review provenance
…count parameter

gradeAgentTask now accepts TaskEventLog | readonly TaskEvent[]: a log carries
its own dropped-event count, so drop-aware grading cannot accidentally grade
an incomplete stream as complete. Raw arrays remain for re-deriving complete
streams (documented as drop-free).
@vu1n
vu1n marked this pull request as ready for review November 12, 2026 11:55
@vu1n
vu1n marked this pull request as draft November 12, 2026 12:54
@vu1n vu1n closed this Nov 12, 2026
@vu1n
vu1n deleted the feat/agent-task-contract branch November 12, 2026 13:02
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant