Skip to content

🐛 ioscan: unavailable classifier consumes per-eval budget, so fail-closed leaks text as budget_exhausted; fail-open warns per string #14627

Closed

Description

@clubanderson

Observed (kubestellar/console hosted hive, ACMM L6, copilot-only, build e568d94 / 9170788 v6)

Since the 2026-10-24 06:52Z redeploy picked up #13858 (classifier on + fail-closed at L5/L6) the hive has no classifier backend (ioscan classifier required at this ACMM level but not running ... no model endpoint resolved). Expected under fail-closed: every kick carrying untrusted text withheld. Actual, per /data/audit.jsonl, every governor eval shows exactly 64 error_fail_closed + 26–44 budget_exhausted classifier skips. The first 64 strings are withheld, the rest pass on unclassified. Which agents' kicks survive therefore depends on render order; reviewer (rendered last) got one kick per pod lifetime from Oct 24 06:38Z to Oct 27 (no priced session for 3 days), while architect/ci-maintainer/quality/dedupe were withheld in 161/169 evals.

Cause: scheduler.enforceTextFrom calls consumeClassifierBudget() before classifier.Score, even when the classifier is ioscan.UnavailableClassifier (which never makes a model call). Once the budget is gone the budget_exhausted branch returns the text in either fail mode.

Second problem: after switching the hive to ioscan.classifier.fail_mode: open (via PUT /api/config/governor/security), the fail-open branch logs Warn("ioscan classifier failed open") once per string (not warnOnce), i.e. 64 WARNs per 2-minute eval, which auto-filed selfreport #14620 within 2 minutes.

Repro

  1. ACMM L6 hive, all agents on copilot, no governor.litellm/governor.trajectory, build without ✨ ioscan: copilot tool-less classifier backend and blocked-kick alert #14501.
  2. Watch audit.jsonl ioscan_classifier_skip reasons per eval: 64 × error_fail_closed then budget_exhausted for the rest.
  3. Set fail_mode: open → 64 × ioscan classifier failed open WARN per eval.

Fix

  • Do not consume budget for UnavailableClassifier; route it straight to the fail-mode handler so fail-closed withholds every string.
  • warnOnce keyed by error in the fail-open branch.

Related: #14499 / #14501 (adds a copilot backend so the classifier actually runs; v5 only, not yet on v6 edge), #14073 (daily quota bypass), #14620 (selfreport caused by the per-string warn).

Activity

  1. github-actions commented on Oct 27, 2026

    @github-actions
    Contributor

    @clubanderson this issue was closed by @hivecommons-hive[bot] as completed.

    If this was closed by a merged PR, the fix only takes effect once it ships in a tagged release and your deployment updates to it — check your running version against the fix before concluding the fix failed.

    If you are still seeing the problem, comment /reopen and it will be reopened for you — you do not need write access. Please add what you are still observing (running version, logs, steps) so it is not closed the same way again.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions