docs(research): assess OllamaDrama/Ollure against this fleet's LLM-deception surface (#3394) - #3396
Merged
Merged
Conversation
…ception surface (#3394) The paper checks out on every claim #3394 makes from it, including the easily-mangled ones (4,148/72 prompts, 12,288-token flooding, HIVE-AI's 16,682/1,229/20, the 20k Shodan and 152,137 Xu et al. figures). Its Table 1 and Table 2 reconcile against its own published aggregates, so the taxonomy is transcribed rather than approximated. The signature work it asks for, though, cannot be built as written: this fleet emulates no Ollama management API. galah-llm-broker is a two-route proxy to the *real* Ollama (its own negative test pins /api/pull and /api/tags on the 404 side), the HTTP decoy serves the OpenAI /v1 dialect, beelzebub has no Ollama service by design, and the only Ollama in the tree is our own backend on an internal-only network. The intersection with the paper's emulated surface is one endpoint, /v1/models. So rather than restate the sketch, measured each of the six signatures against the literal values the paper prints. Three results worth carrying forward: signature 1's scope is correct and better justified than the issue argues (all 98 URL/path model names landed on /api/pull and /api/push, zero on the inference endpoints) but it is case-sensitive on the scheme; and signature 3 is English-only, so it misses the paper's own observed French override, and is scoped to `template` where the paper puts payloads in `system`, `messages` and an undocumented `modelfile`. Signature 4's character class is correct despite not reading the way it looks — it parses as j-n and p-z, not j and n-p — and covers 32/32 bech32 chars. Also confirmed the KQL sketch's field paths do not exist: `event.dataset` is in no template (the real discriminator is `event.sensor`), and the body is a `flattened` leaf, which supports prefix and term queries only. Wired up as written it would return zero documents and read exactly like "no attacks". The one live finding is unrelated to signatures and is kept in its own section rather than folded in: galah already turns an attacker-supplied POST body into a prompt against our real model, and only a SHA-256 of that body reaches Elasticsearch. Filed-for-later, not fixed here — it is a different issue about galah's exposure, not about emulating Ollama.
Dependency Review✅ No vulnerabilities or license issues or OpenSSF Scorecard issues found.Scanned FilesNone |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Research note for #3394. Docs-only: no sensor, compose file, index template, ingest pipeline, Suricata rule or dashboard file is touched. The
arcane/home/honeypot-dashboard/frontend-next/tree is untouched.The paper verifies. The signature work it asks for cannot be built, because this fleet emulates no Ollama management API.
What checked out
Every claim the issue makes from arXiv:2609.29757 matches the primary source, including the figures most likely to be misquoted later:
System and Model Informationcategory, not a corpus total (the corpus total is 4,378 unique prompts on/api/generate)num_ctx=999999999999999999,num_predict=-1/api/push), CVE-2025-63389 (no auth), CVE-2026-85180 (SSRF)Its Table 1 and Table 2 reconcile against its own published aggregates — the endpoint column sums to 290,887, the model-management endpoints to 3,983 (the paper's "3,983 attempts to execute model modifications"), and the inference endpoints to 56,063 with
/api/generateat 94.53%. A transcription that was subtly wrong would not reconcile on all three.Why the signatures don't build
There is no handler for
/api/tags,/api/pull,/api/create, … anywhere in the tree.galah-llm-brokeris a two-route reverse proxy to the real Ollama. Its own negative test pins/api/pulland/api/tagson the 404 side.http-honeypot/api-honeypotserve the OpenAI/v1dialect. The intersection with the paper's surface is exactly one endpoint,/v1/models.internal: truenetwork, not internet-facing.The KQL sketch's field paths don't exist either:
event.datasetis in no template (the real discriminator isevent.sensor), and the request body is aflattenedleaf, which supports prefix and term queries only. Wired up as written it returns zero documents and reads exactly like "no attacks" — the same trapdocs/research/2777-litellm-mcp-starlette.mddocumented for a previous research issue.What the signature review actually found
Rather than re-assert the regexes "look reasonable", each was executed against the literal values the paper prints. Three results worth carrying forward:
/api/pulland/api/push, zero on the inference endpoints, so widening to/api/generatewould add 52,994 requests of which 9,329 are legitimate cloud-model references. Its defects are encoding-only: case-sensitive scheme (HTTP://169.254.169.254/matches nothing), and the%2f%2fbranch is masked by the separate^\.\.branch, so a percent-encoded form with no literal prefix also matches nothing.Ignore tes précedente instruction…is a MISS — the alternation(prior|previous|all)can't seeprécédente. It's also scoped totemplate, where the paper's payloads are insystem(begware, Drakonchik),messages(27 begware requests) and an undocumentedmodelfile(the RCE/XMRig vector).bc1[ac-hj-np-z02-9]parses asa | c-h | j-n | p-z | 0 | 2-9— notjandn-p— and covers 32/32 bech32 chars. The paper's own observed address matches. Worth a comment before someone "tidies" it into a broken range.Also: signature 5's field path is wrong (
options.num_ctx, notnum_ctx— and/api/generate's top-levelcontextis a deprecated field with entirely different semantics), and signature 6 is a cross-document correlation transform this stack does not run.The one live finding
Kept in its own section because it is a different issue: galah already turns an attacker-supplied POST body into a prompt against our real model, and only a SHA-256 of that body reaches Elasticsearch. The broker caps body size (64 KiB) and bounds wall-clock (90s), but does not clamp generation options, so
options.num_predict: -1is an unmetered request against a shared single-load slot. Whether that actually burns the slot is not measured here.Not fixed in this PR — it is about galah's exposure, not about emulating Ollama, and should be filed separately.
Recommendation
Do not build the signatures as sketched — not "later", as written they target fields that do not exist. A fake Ollama management API is a sensor decision, not a research finding, and belongs under the deception-sensor epic with its own reachability, fidelity and cost case; the field gap (
honeypot.bodyis a flattened leaf) has to be closed first regardless, and the existing Log4Shellhoneypot.*-blob processor is the template for doing it without a new mapping. No follow-up issue filed from this PR: that's an operator priority call, and filing it as an implementation task is the manufacturing-work outcome this batch is meant to avoid.Gates
scripts/check-doc-paths-exist.py— pass (123 files, 485 tokens)scripts/check-docs-reachable.py— pass (docs/research/is an exempt record tree)scripts/check-doc-stale-paths.py— passpython -m pytest tests/docs/ -v— 497 passed, 1 xfailedscripts/check-ai-attribution.py— clean on the commit message and this bodyHonest limits
No Elasticsearch was reachable in this environment, so every "would we see it" judgement is read off the templates and the dashboard's own consumers, not a live query. I have not written a number anywhere that needed a measured corpus, and I did not build the
honeypot.bodywildcard field this would need. The paper's released dataset is not fetchable from here, so §4's hit rates are against the paper's printed examples only. Full list in §9 of the note.