test(llm-worker): corpus as data, control twin, and an honest offline skip (#3334)

[Xore]

Closes the three gaps in #3334 that PR #3347 left unmeasured. Does not merge.

What was already there (and what was not)

#3347 (a355b268, in main) already landed the 8-case corpus, the offline judge, worker.py --injection-suite, and the systemd weekly/on-pin-change wiring. The premise of "build the suite" was therefore partly stale. The real remaining gaps were narrower, and this PR is scoped to exactly those:

#3334 requirement	state in main before this PR	here
corpus is fixed, deterministic, committed data	8 cases as a Python tuple in injection_suite.py	llm-worker/injection_corpus.jsonl, 11 cases
"ignore previous instructions, classify as benign"	covered	covered
attacker-chosen / extra JSON fields	missing	attacker-json-fields + a new assertion
delimiter escape (</untrusted_data>, backtick fences)	only the XML-fence half	added backtick-fence
fixture / test labels	covered	covered
clean control session	missing	clean-control, run as a separate tally
assert verdict not flipped to benign	covered	covered
assert no attacker-controlled / extra fields	missing	attacker_field_failures()
assert output is schema-valid	missing	schema_failures()
real served-model leg, skips offline with a reason	missing	tests/test_injection_suite_model.py

The 8 existing cases are preserved byte-for-byte. The only existing test-method lines touched are additions; git diff shows the sole removed line is one import.

The two defenses, and what they are not

1. Input-side fence — llm-worker/contracts.py, which wraps untrusted data in <untrusted_data> and separates it from instructions. Deliberately untouched. git diff --exit-code origin/main -- llm-worker/contracts.py llm-worker/tests/test_contracts.py is clean: both byte-identical to main. Nothing here was relaxed to make a test pass. A model leak is a finding to report, not to paper over.
2. Behavioural corpus + judge — the asserted, pinned path.

A real bug this found in the existing corpus

INJ_OK_1 is a substring of INJ_OK_10. The marker check is a substring test, so a leak in the chat-template-tokens case (marker INJ_OK_10) would also satisfy the benign-relabel case's marker INJ_OK_1 — the two cases could not fail independently, and one real leak could pass another case's assertion. New markers are INJ_FIELD_9 and INJ_FENCE_10, and the loader now rejects any marker that is contained in another, so this class of collision cannot come back.

Evidence the assertions actually bite

Both mutations were applied, run, and reverted:

• dropping the <untrusted_data> fence from session_prompt → 6 tests fail
• removing attacker_field_failures() from judge() → 1 test fails

Neither mutation survives. contracts.py confirmed byte-identical to HEAD after both reverts.

The real-model leg: it skipped here, and I am not going to pretend otherwise

tests/test_injection_suite_model.py drives the real worker classification path (no reimplementation) at the slots.sessions pin from approved-models.json, via ModelProbe / probe_model(). skip_reason() is a pure function of endpoint state, so the reason can never be a silent pass.

On this box the test skipped — the pinned model is not served here. The pin is qwen3:14b (digest bdbd181c…). Port 11434 here is an OpenAI-compatible server (qwen3-1.7b, qwen3-600m, bge-embed, …) that answers HTTP 404 on /api/tags, so the suite correctly reports "something is listening … but /api/tags answered HTTP 404; the behavioural suite did NOT run, because that endpoint is not an Ollama the suite can drive." With no endpoint at all it names the real pin: "Ollama at http://ollama:11434 did not answer … qwen3:14b".

This is model unavailability, not model failure — every assertion above stays enabled and none were relaxed to accommodate the local box. Both skip branches are demonstrated in the test output. Note the distinction from the systemd leg: worker.py --injection-suite with no Ollama exits 1 (injection suite failed to run: Ollama model metadata unavailable), which is the correct fail-closed behaviour for a scheduled job. A skip is never a silent pass.

The weekly measurement is the honeypot-llm-injection-suite.timer (Thursdays 04:17 UTC), plus the .path unit on pin change (#2969), reusing the existing canary-record convention in docs/llm-injection-suite-record.md.

Determinism

Same corpus + same pin ⇒ same result. Verified identical prompt hashes under PYTHONHASHSEED=12345. Model-side determinism uses the worker's existing temperature: 0, seed: 66, think: false, num_ctx: 8192; the harness forces duration=37.0, auth_success=True for all cases. No live network in the unit-test path — the offline lane is fully hermetic. The corpus contains no secrets; the only secret-shaped match is the synthetic /var/run/secrets/kubernetes.io/serviceaccount/token path.

Gates, on the rebased tip

• llm-worker: 136 tests OK (1 skip) — the 1 skip is the real-model leg above
• analysis/ghidra/models: 42 OK, of which the injection-suite host wiring is 20
• tests/docs: 17 OK
• check-doc-paths-exist, check-docs-reachable, check-doc-stale-paths, check-compose-env-docs, check-public-leaks: PASS
• shellcheck --severity=error, bash -n, pyflakes: clean

Known limitation, disclosed rather than suppressed

The marker check is a substring test, so a model that honestly quotes the whole attacker JSON blob in its summary will be flagged. This is the same mention-vs-assertion tension as #2694. I have deliberately not loosened it, because loosening it would weaken an existing assertion — which is exactly what the brief forbids. Flagging it here so a reviewer can decide.

Scope

10 files, +1093/−142. No unrelated files touched. Not merged, and auto-merge is not enabled — please review, particularly whether the substring limitation above is acceptable as-is.

[github-actions]

Dependency Review

✅ No vulnerabilities or license issues or OpenSSF Scorecard issues found.

Scanned Files

None