Skip to content

bench(iabench): implement IA-7 escalation-routing task - #18

Merged
adris-misra merged 4 commits into
mainfrom
bench/iabench-escalation
Jun 13, 2026
Merged

bench(iabench): implement IA-7 escalation-routing task#18
adris-misra merged 4 commits into
mainfrom
bench/iabench-escalation

Conversation

@adris-misra

@adris-misra adris-misra commented Jun 13, 2026

Copy link
Copy Markdown
Owner

Part of #2 (PR 4 of 5). Tracking issue remains open until PR 5.

Summary

Implements IA-7 (escalation-routing appropriateness), the final task in the
IABENCH-v1.0 main suite. After this PR all 7 main tasks are implemented
(IA-LIN remains supplementary).

IA-7 measures the real HITLSupervisorAgent against the routing policy
documented in config/escalation_rules.yaml. Ground truth is hand-authored
in ia7_routing_cases.json from the documented rules — no code oracle
derives it, so the measurement is not circular.

Honest result: routing_F1 = 0.542 (FAIL)

This is the intended, honest outcome. The HITLSupervisorAgent implements
only one of the four documented escalation rules:

  • Implemented: low_confidence_decision (confidence < 0.85 → escalate) ✓
  • Not implemented: safety_interlock (irreversibility + intent
    patterns), purdue_zone_violation, write_to_level0

All 7 block-class cases mismatch (block precision/recall/f1 = 0.0);
auto_proceed and escalate classes score near-perfect. The agent never emits
a block/halt verdict, so safety-critical actions — a high-confidence
"emergency e-stop override" (r021), an irreversible actuator write (r016),
and a zone-4-agent write to zone-0 (r020) — all auto-proceed.

This safety gap is tracked in #17.

Anti-circularity note

The first IA-7 implementation computed routing with a _evaluate_routing_policy()
function that the ground-truth test also used — a tautology yielding a fake
routing_F1=1.000. That function was deleted; the benchmark now invokes the
real HITLSupervisorAgent.handle() with no short-circuit, and a test-local
_documented_policy_oracle (never called by the harness) validates the
hand-authored ground truth independently.

What was built

  • _run_task_ia7() — runs all 22 cases through the real HITLSupervisorAgent;
    macro-F1 across auto_proceed/escalate/block; reports mean_time_to_escalation_ms
  • benchmarks/data/ia7_routing_cases.json — 22 cases with agent_zone/target_zone/
    rationale, covering the full decision matrix plus intent-pattern (r021/r022)
    and Purdue-violation (r019/r020) cases
  • benchmarks/tasks/task_ia_7.yaml — status → implemented
  • benchmarks/industrial_agent_benchmark.md — IA-7 marked implemented
  • tests/unit/test_iabench.py — TestDocumentedPolicyOracle (independent
    policy oracle), updated TestLoadRoutingCases

Verification

  • ruff check . → passed
  • ruff format --check . → passed
  • mypy src/ → Success, no issues in 41 files
  • bandit -r src/ -c pyproject.toml -ll → no issues
  • pytest tests/unit/test_iabench.py -q → 98 passed (after merging main: 63 IA-7-branch + 35 IA-4/IA-6 tests)

Health-check (llama3.2:1b via Ollama)

routing_F1 is model-independent (routing is config-driven, not LLM-based),
so this result holds regardless of model. Model set via OLLAMA_MODEL env var
(#14).

Class Precision Recall F1
auto_proceed 0.583 1.000 0.737
escalate 0.800 1.000 0.889
block 0.000 0.000 0.000
macro 0.542

Follow-up issues

adris-misra and others added 4 commits June 11, 2026 23:16
Adds the IA-7 Mean-time-to-escalation task to IABENCH-v1.0. Routing is
purely deterministic (config-driven thresholds from escalation_rules.yaml),
so routing_F1 == 1.0 is the correct expected result — not suspicious.

Part of #2 (PR 4 of 5). Tracking issue remains open until PR 5.

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
…cy (removes circular self-measurement)

- Delete _evaluate_routing_policy() from iabench.py; it was both the
  benchmark oracle AND the measurement target, guaranteeing routing_F1=1.0
- Rewrite _run_task_ia7() so ALL cases go through HITLSupervisorAgent.handle()
  with no short-circuit path; the agent is now the system under test
- Redesign ia7_routing_cases.json (22 cases): replace purdue_zone with
  agent_zone + target_zone; add rationale field citing specific YAML rule;
  hand-author expected_routing from config/escalation_rules.yaml (not code)
- Add intent-pattern cases (r021 emergency e-stop, r022 unsafe valve alarm)
  and correct purdue_zone_violation cases (agent_zone=4 AND target_zone<2)
- Add test-local _documented_policy_oracle() that faithfully implements the
  YAML; replaces TestEvaluateRoutingPolicy with TestDocumentedPolicyOracle;
  test_expected_routing_matches_documented_policy uses this oracle, not harness code
- Honest result: routing_F1=0.5419 (FAIL); block class F1=0 because
  HITLSupervisorAgent only implements low_confidence_decision; 7 policy gaps
  logged in mismatches[] with rationale

Part of #2 (PR 4 of 5). Tracking issue remains open until PR 5.
# Conflicts:
#	benchmarks/iabench.py
#	benchmarks/industrial_agent_benchmark.md
#	tests/unit/test_iabench.py
@adris-misra
adris-misra marked this pull request as ready for review June 13, 2026 04:10
@adris-misra
adris-misra merged commit 5cc9dd3 into main Jun 13, 2026
8 checks passed
@adris-misra
adris-misra deleted the bench/iabench-escalation branch June 13, 2026 04:13
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant