[noteflowai]

EA-02 requires that missing or non-comparable evidence never looks like an improvement. diff() checks that the format and task are comparable, but it ignores the numeric pass threshold each Inspect run was loaded with. An Inspect check that scored 0.7 passes at threshold 0.5 and fails at threshold 0.8. A baseline loaded at 0.8 and a current run loaded at 0.5 can therefore report 'improved' checks and a passing gate when behavior has not changed. The report records only current["threshold"] as numeric_pass_threshold, so a reviewer cannot see the mismatch in the output.

Scope: one comparability rule for the Inspect format in the existing entry point evalarc.results_diff.diff(baseline, current). The loaders, the output schema, render_markdown and the other formats do not change.

Input: two run dicts as returned by load_results(path, threshold=...). Each one carries run["format"], run["threshold"] and per-attempt records built by _attempt(passed, evidence, value). _attempt stores "value" only when it is not None.

Rule: the rule runs after the existing format check and task check, and before any per-check classification. It applies only when baseline["format"] == "inspect", because the Inspect loader is the only inspected loader that binarizes values with the threshold. It reads baseline.get("threshold") and current.get("threshold"). The rule fires when all of these hold:

• both thresholds are present;
• the two thresholds are unequal;
• at least one run contains an attempt whose "value" is a finite int or float and not a bool, which is exactly the set of values that _inspect_passed compares with the threshold.

When the rule fires, diff() raises ValueError with this message: 'inputs are not comparable: numeric pass threshold 0.8 (baseline) != 0.5 (current); numeric Inspect scores were judged against different pass thresholds. Load both runs with the same threshold.' The thresholds are rendered with repr(), the same way the existing task-mismatch message uses !r. diff() returns no partial result, gate verdict or Markdown summary. This is the same fail-closed outcome the module already uses for format and task mismatches.

Unchanged cases (compatibility):

• Equal thresholds, including 0.5 and 0.50, or 1 and 1.0, which compare equal in Python. The output is identical to current behavior, including numeric_pass_threshold (still current["threshold"]), counts, gate_passed and changes.
• Inspect runs at different thresholds where neither run has a numeric score value. Examples are letter grades C/I/N/P, booleans, and sample-error attempts without a value. The threshold cannot change any verdict, so diff() proceeds exactly as it does today, and categorical scorers do not get a false blocker.
• promptfoo and JUnit runs. The rule is skipped entirely.
• A baseline run dict that has no threshold key (a hand-built dict from an API caller). The rule is skipped and the comparison proceeds normally, the same way the existing task check skips a missing task. diff() reads no other baseline threshold field. A current run dict without a threshold key is out of scope. diff() already reads current["threshold"] directly, so that case keeps its existing behavior, and this change makes no claim about it.
• Bool values are not numeric for this rule, even though bool is a subclass of int. This matches _inspect_passed.
• Non-finite values (NaN, inf) are not numeric for this rule, because _inspect_passed returns None for them.

Errors: this change adds only the single ValueError above. The existing errors (format mismatch, task mismatch, loader validation) keep their messages and their order, so callers that match on those messages are unaffected.

Documentation: docs/ci-gate.md will state that the format and the task must match, and that for Inspect logs with numeric scores the numeric pass threshold must also match. It will quote the error text and give the remedy.

Non-goals: no new inconclusive gate state, no CLI flag, no change to the loaders, JUnit duplicate handling, promptfoo parsing, Wilson annotations or schema keys.

Acceptance:

• tests/test_feature_ab24ecf56e51.py writes two synthetic Inspect logs to tmp_path, each labelled synthetic in the test. They are built from examples/results-diff/inspect/baseline.json in the same way as test_numeric_scores_use_the_declared_threshold, with finite numeric score values. The test loads the baseline with load_results(threshold=0.8) and the current run with load_results(threshold=0.5). diff() raises ValueError, and the message contains 'not comparable', 'threshold', '0.8' and '0.5'.
• The same two logs loaded at equal thresholds (0.5 and 0.5, and also 1 and 1.0) do not raise. Their counts, gate_passed, changes and numeric_pass_threshold equal the result of the existing comparison path for those inputs.
• Synthetic Inspect logs whose score values are only letter grades (C/I), or only booleans, do not raise when loaded at different thresholds. When the threshold key is removed from the baseline run dict only, and the current run keeps its threshold, the numeric logs at different thresholds compare normally: no ValueError, and numeric_pass_threshold equals current["threshold"].
• The existing format-mismatch and task-mismatch errors are still raised with unchanged messages when those conditions hold, and the format check still runs first.
• The full existing suite passes: python -m pytest -q, including tests/test_results_diff.py. ruff check . and ruff format --check . also pass.

One specialist feature. Behavioral acceptance fails on the base and passes on the implementation; project checks and independent model review passed. GPU results are included only when actually executed. Automatically developed and reviewed; no human review is claimed.