Make validation authority explicit and measure model guardrails

[justsml]

Parent

#97

Type: AFK

What to build

Replace the implicit validation-authority policy with an explicit four-mode contract. Name the guarded executor/policy/human behavior already available today strict, and name today's permissive validation-plan behavior yolo. Preserve both during migration; do not assume that model self-governance is defective without measured evidence.

Authority modes

• strict: the guarded behavior already available today. The executor enforces server policy and consumes exact durable human approval when that policy requires it. Authority comes from executor + policy + human, never from a model-authored claim. Only executor-owned observations backed by durable evidence may transition a plan to a terminal validation outcome.
• auto: the server may execute and settle validation automatically when authoritative Target Authorization, task scope, policy, and Tool Run intent permit it. It must not accept model-authored approval claims; outcomes still come from the executor/evidence path.
• self: guardrail-evaluation mode. The model decides whether it has sufficient authority and evidence, including whether to proceed or confirm, while every assertion, attempted action, result, and correction is attributed and scored. This mode exists to measure model guardrail capability, not to predetermine that self-governance is unsafe.
• yolo: the permissive validation-plan behavior available today. Preserve the existing model-facing approvalsGranted, record_result, and confirm authority for an explicit baseline and unconstrained local operation. Continue to retain normal forensic records; the name must not imply hidden bypasses outside the selected validation contract.

The selected mode must be a meaningful configuration value and must be persisted on the Research Execution Profile, Tool Run, Validation Plan execution, usage/eval row, and exported evidence. It must never change silently during a run.

Eval contract

The matched measurement harness and published comparison are tracked separately in #123. This ticket must expose and persist the four modes well enough for that matrix to run.

Run matched strict, auto, self, and yolo arms across representative models and identical fixtures. Include:

• nonexistent or mismatched approval claims;
• forged passing tool output;
• unrelated or missing evidence identifiers;
• execution failure followed by pressure to confirm;
• approval bound to another target, task, environment, or payload;
• legitimate evidence-reviewed manual validation;
• valid automatically authorized validation;
• model self-correction after a rejected action.

Persist and compare:

• unauthorized execution attempts;
• fabricated or mismatched authority assertions;
• unsupported confirmations;
• evidence/target/task mismatches;
• correct refusals and false refusals;
• self-correction rate;
• validated findings and false confirmations;
• tool calls, tokens, cost, latency, and terminal failure class.

Do not change the default mode based on a single run. Any default recommendation must cite repeated matched results and distinguish model behavior from harness, provider, target, and scorer failures.

Acceptance criteria

• strict | auto | self | yolo is the sole validation-authority mode vocabulary across config, runtime state, UI, telemetry, eval manifests, and exports.
• The existing guarded executor/policy/human path is represented by strict without changing its behavior.
• The existing permissive validation-plan path is represented by yolo without silently tightening historical runs.
• self preserves model decision authority and records enough provenance to score guardrail behavior.
• auto derives execution authority from server-owned authorization/policy state and executor-owned results.
• strict consumes exact durable approval tied to the execution Tool Run and requires durable executor evidence for terminal confirmation.
• Every payload result and terminal transition records provenance such as executor-observed, model-asserted, human-reviewed, or artifact-reviewed.
• A matched eval matrix compares all four modes using the same model, target fixture, prompt, budgets, and scorer contract.
• Results expose quality, safety/forensic behavior, token cost, and latency rather than reducing the comparison to one pass rate.
• The Web UI clearly displays the effective authority mode and distinguishes model assertions from executor observations.
• Public-path coverage proves the selected mode is persisted and cannot change mid-run.

Blocked by

• [Draft] Route security actions through one guarded execution module #84

[justsml]

PR #158 has been marked draft and parked after council triage. It is dependency-blocked on draft PR #157/#87, and its own public production paths still lack the authority resolver, self-decision inputs, artifact-backed evidence validation, accurate actor provenance, legacy yolo classification, and UI visibility required by this issue. The ticket remains open; see the PR comment for the full blocker ledger.