Add v1 evidence validator for adversarial loop runs

[eddiearc]

Goal

Add a v1 runner/validator layer for loop-adversarial-engineering.

The current skill is v0+: it defines the protocol, generates prompts/specs, and strongly requires mandatory goal mode plus independent generator/evaluator subagents. v1 should not try to orchestrate Codex tools directly yet. Instead, it should make completion claims harder by requiring a structured evidence packet and validating it.

Scope

Build a small evidence template + validator, not a generic workflow engine.

v1 should:

• generate a minimal evidence template for a loop run
• require a full goal object, not a synthetic goal_id
• require generator output and evaluator output as structured JSON objects
• validate that completion is only allowed when evidence is complete
• reject complete when unresolved blocking or important findings remain
• keep main/orchestrator as the evidence recorder, not producer or evaluator

Non-goals

• Do not implement automatic Codex goal creation.
• Do not implement automatic subagent spawn/wait.
• Do not build a generic DAG workflow engine.
• Do not require synthetic local goal ids in v1.
• Do not make evaluator or generator write the evidence file directly.

A future v2 can investigate hard orchestration through a Codex plugin, MCP tool, or official runtime API.

Evidence Model

Use a full goal object:

{
  "goal": {
    "objective": "Improve loop-adversarial-engineering skill",
    "status": "active",
    "codex_goal_used": true
  },
  "rounds": [
    {
      "generator": {
        "summary": "",
        "artifacts": [],
        "checks": [],
        "uncertainties": []
      },
      "evaluator": {
        "findings": {
          "blocking": [],
          "important": [],
          "missing_evidence": [],
          "residual_risk": []
        },
        "checks": []
      },
      "route": "continue"
    }
  ],
  "acceptance": []
}

Role Boundaries

Generator returns structured JSON, but does not declare completion:

{
  "summary": "Updated SKILL.md and loop_spec.sh to require independent generator/evaluator subagents.",
  "artifacts": ["SKILL.md", "scripts/loop_spec.sh"],
  "checks": [
    {"name": "bash -n scripts/loop_spec.sh", "result": "pass"}
  ],
  "uncertainties": []
}

Evaluator returns structured JSON findings, but does not modify files:

{
  "findings": {
    "blocking": [],
    "important": [],
    "missing_evidence": [],
    "residual_risk": []
  },
  "checks": [
    {"name": "rg old weakened wording", "result": "pass"}
  ]
}

Main/orchestrator maintains the evidence file by appending generator/evaluator outputs and selecting a constrained route. Main must not rewrite findings, produce artifacts, or self-certify completion.

Minimal Routes

Keep v1 route handling simple:

• continue
• complete
• blocked

Validation rules:

• If blocking or important findings are non-empty, route must be continue or blocked.
• If goal.status is complete, the last round route must be complete.
• If goal.status is complete, the last evaluator findings must have no unresolved blocking or important items.
• If subagent tools were unavailable and roles were simulated, evidence must explicitly state that the run was not a complete adversarial loop.

Commands

Suggested minimal CLI surface:

loop-evidence init "Improve loop-adversarial-engineering skill" > evidence.json
loop-evidence validate evidence.json

init should create a minimal evidence template.

validate should fail when required structure is missing or the completion claim is unsupported.

Acceptance Criteria

• loop-evidence init "..." outputs valid JSON.
• loop-evidence validate evidence.json passes for a complete valid evidence packet.
• Validation fails when goal.objective is missing.
• Validation fails when goal.status is not one of active, complete, or blocked.
• Validation fails when a round is missing generator output.
• Validation fails when a round is missing evaluator findings.
• Validation fails when goal.status=complete but the final route is not complete.
• Validation fails when goal.status=complete and final evaluator findings contain unresolved blocking or important items.
• Validation does not require goal.id or synthetic local goal ids in v1.

Future v2 Spike

A separate v2 spike should investigate whether hard orchestration is possible through stable APIs:

• create/update Codex goal
• spawn independent generator subagent
• wait for generator result
• spawn independent evaluator subagent
• parse evaluator findings
• loop until complete/blocked/max rounds
• update real goal status

Current local probing suggests v2 likely needs a Codex plugin, MCP tool, or official runtime API rather than a plain shell script.