Repository navigation
Conversation
…able tool calls and explicit pass/fail grading - t3mp3st_agent_task/v1: committed task dirs (prompt, allowed tools, grader, deterministic stub script) with strict fail-closed validation - ordered append-only event stream (events.jsonl): every tool call with secret-redacted args, every scope/approval denial, logical clock so stub runs are byte-reproducible - graders: flag-exact (CTF oracle) and event-policy (required calls, denied-call budget, sha256-pinned artifacts); outcomes pass/fail/abstain/infrastructure_error - negative fixtures pin failure criteria: each committed bad run must fail on its named check (scope denial, missing call, wrong artifact, wrong flag) - Arsenal tool events now carry redacted parameters so live runs are observable per-call; tapArsenal mirrors them (and gate refusals) into the same stream - scripts/agent-task.mjs: --list / --task / --self-test; two seed tasks
vu1n
marked this pull request as draft
November 12, 2026 09:08
…xpression, import schema constant
…ents, fresh workspaces, crash persistence, no phantom tools Review findings fixed (all with regression tests): - empty tool output/answer no longer throws from the event logger; the tap wraps its subscribers so instrumentation can never flip an Arsenal success into an ExecutionError or replace a SCOPE DENIED result with an exception (tested with an empty-output tool and an over-long tool name) - runner refuses a non-empty workspace: a stale artifact from an earlier run can no longer satisfy an artifact check (false pass); stub/grader artifact paths may not shadow events.jsonl/verdict.json (rejected at validation) - a backend exception now persists the partial event log and grades as infrastructure_error instead of losing the bundle - task fixtures renamed to registered tools (http_request, header_analysis, hash_crack) and a guard test enforces advertised = wired over bench/tasks - exactly-one task_started/task_finished enforced in the log and on JSONL parse - CLI: fileURLToPath (paths with spaces), harness loaded once for --list; npm run test:agent-task wired into test:release:contracts - Arsenal: redacted parameters computed once per execute and only when a subscriber is listening; README documents the redaction scope honestly
…ts verdict, jwt_decode seed task - tap drops are now counted: any dropped event grades infrastructure_error (no_dropped_events check) so an unloggable scope denial can never vanish from a passing run; the count is recorded on the verdict and as a note - a backend that finishes and then throws keeps its graded finish; the post-finish throw is preserved as a note instead of an illegal second task_finished (which previously aborted the run before writing the bundle) - flag-oracle-demo seed task uses jwt_decode (harmless, plausible decode of a real base64url JWT payload) instead of hash_crack - tests renamed to describe behavior, not review provenance
…count parameter gradeAgentTask now accepts TaskEventLog | readonly TaskEvent[]: a log carries its own dropped-event count, so drop-aware grading cannot accidentally grade an incomplete stream as complete. Raw arrays remain for re-deriving complete streams (documented as drop-free).
vu1n
marked this pull request as ready for review
November 12, 2026 11:55
vu1n
marked this pull request as draft
November 12, 2026 12:54
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What
A committed agent task contract (
t3mp3st_agent_task/v1) plus the runner/grader machinery that makes an agent run reproducible, observable, and explicitly graded — the same disciplineverify-claimsapplies to headline numbers, applied to individual agent tasks.bench/tasks/<id>/task.json): prompt, allowed tools, timeout, grader, and an optional deterministic stub script — strictly validated, fail-closed on malformed files.events.jsonl): one JSON line per observable step — every tool call with its secret-redacted arguments, every scope/approval denial, notes, and the final answer. Logical clock (ts === seq) so a stub run is byte-reproducible on any machine.flag-exact— CTF-shaped exact-match oracle;event-policy— process-shaped: required calls (tool + min count + argument constraints), denied-call budget (default: anytool_deniedfails the run), sha256-pinned artifacts.pass/fail/abstain/infrastructure_error(infrastructure errors are never capability misses).scope-denial,missing-call,wrong-artifact,wrong-flag).tool:executed/tool:errorevents now carry redactedparameters, andtapArsenal()mirrors live Arsenal calls (and gate refusals) into the same event format. The tap is fault-isolated: a logging failure drops the event, never the tool result.scripts/agent-task.mjs:--list/--task <id>/ self-test vianpm run test:agent-task; two seed tasks (egress-receipt-check,flag-oracle-demo) naming only registered tools (http_request,header_analysis,jwt_decode), guarded by an advertised-=-wired test overbench/tasks/; docs inbench/tasks/README.md.Scope (read this first)
This PR ships the contract + event-stream + grading layer and a deterministic stub backend, plus the Arsenal tap hook. It does not wire a live LLM or server backend — those are follow-ups, as is wiring
tapArsenalinto a live runner outside tests.Why a new module rather than extending
scripts/cybench-bench.mjs: cybench is a single-suite harness whose per-run artifact is a prose transcript (stripped for privacy, so verdicts re-derive only at the graded-verdict level) with flag-only oracles. The task contract gives a schema any suite can adopt — structured per-call events, pluggable graders, argument-aware required-call checks — without entangling cybench's committed artifacts or its privacy-stripping guarantees. Migrating cybench tasks onto this format (and at3mp3st-serverbackend that drives/api/general/autowith the tap attached) is the natural next PR.Runner guarantees (review-hardened)
events.jsonl/verdict.jsonare runner-reserved; stub/grader paths cannot shadow them.infrastructure_error— the bundle is always diagnosable.task_started(first) and onetask_finishedper stream, enforced on append and on JSONL parse. A backend that finishes and then throws keeps its graded finish; the throw is preserved as a note.TaskEventLogitself (which carries its dropped count), so drop-aware grading cannot forget it; Tap subscribers are fault-isolated (a logging failure never changes a tool result), and drops are counted: any dropped event gradesinfrastructure_error— an unloggable scope denial can never quietly vanish from a passing run.Contribution Receipt
parameterson Arsenal tool events (computed only when a subscriber is listening).static_fixturenot_applicable(stub backend executes no tools, touches no network)nonestatic_test,mocked,tool_backed(Arsenal tap path only)not_applicable; harnessagent-task(new); agent_runtimenone; tool_accessmocked; target_classstatic_fixture; attempts: 2 tasks + 4 negative fixtures per self-test run; successes: 2 tasks; failures: 4 negative fixtures (by design, each on its pinned check); abstentions: 0.98f37ba, clean clone ofmain@29824d5+ patch):npm run lint→ pass (0 errors; pre-existing warnings unchanged)npm run typecheck→ passnpm run test:pr→ pass end-to-end (lint 0 errors, typecheck,npm test90 files / 1014 tests incl. ctf-rsa-static, doctor PASS, 0 blockers) — run with homebrew python 3.15 first on PATH; the system python3 (3.9) cannot parse the CTF challenge'sint | strunions, a pre-existing environment quirk, not a repo defectnpm run doctor→ pass (optional-tool warnings only)npm run verify-claims→ pass, 27/27 (no claims touched)npm run build && npm run test:agent-task→ pass (all tasks + negatives graded as pinned)vitest --coverage --config vitest.pr.config.ts+check-changed-line-coverage origin/main) → 506/506 (100%; required 50%)bench/tasks/README.md, two seed task dirs with negative fixtures; run bundles are written to fresh temp dirs on demand, never committed.parameterspass through the configured-secret scrubber (same path as results) — that scrubber masks only configured target-header secrets; a credential the model itself places in an ordinary parameter is recorded as-is, documented inbench/tasks/README.md. Event previews are capped; stub fixtures contain no real hosts, flags, or secrets.verify-claimsoutput is byte-identical before/after.timeoutMsis declared in the contract but enforcement is the backend's responsibility in v1. Instrumented events include model-supplied parameter values (see Redaction).Prior work and attribution
No prior T3MP3ST PR is used as a code source. The format follows existing repo patterns: cybench challenge dirs, the model-matrix outcome taxonomy, recovery-snapshot-style strict validation, and the
no-phantom-toolsadvertised-=-wired discipline (extended overbench/tasks/by a new guard test). External inspiration (own prior systems: sealed-session eval harness with per-session event logs) noted; all code here is fresh and verified at this head.