From e447521a79b19e56a2bacc6d36ed9608f67faa8c Mon Sep 17 00:00:00 2001 From: xore Date: Sun, 27 Sep 2026 14:43:40 +0200 Subject: [PATCH 01/17] docs(readme): correct sensor directory names, retired-worker row, profile set, stack and diagram counts - row 'more sensors' listed honeypot-citrix / honeypot-cisco-asa / honeypot-rdp, but the directories are honeypot-citrix-honeypot, honeypot-cisco-asa-honeypot and honeypot-rdp-honeypot, so the row's own 'arcane/home/honeypot-/compose.yml' pattern did not resolve - the three worker stacks in the next row were all retired by #1649 and run as Rust WORKER_LOOPS now; the row still described them as live writers - 'the only profile is geoip-update' omitted the threat-intel maintenance job, the four ["legacy"] rollback definitions and ghosts' ["test"] client - '20 split home stacks' -> 26, the roster deploy-profiles/full.txt names - ARCHITECTURE.md '6 diagrams' -> 4, its actual mermaid block count --- README.md | 15 +++++++++------ 1 file changed, 9 insertions(+), 6 deletions(-) diff --git a/README.md b/README.md index 8e401a15..7e82aae5 100644 --- a/README.md +++ b/README.md @@ -25,8 +25,11 @@ flowchart LR wg --> home["home APIARY stacks
@ 10.8.0.2"] ``` -**All core sensors run without compose profiles.** The only profile is the -optional on-demand `geoip-update` maintenance job. 39 deployment pieces — +**All core sensors run without compose profiles.** The only profiles are the +optional on-demand maintenance jobs `geoip-update` and `threat-intel` (both in +`honeypot-init`); the four `["legacy"]` worker stacks are defined for rollback +but do not run, and `sandbox/ghosts`'s `["test"]` client is not a sensor. +39 deployment pieces — 33 independent Arcane-managed stacks under `arcane/home/` plus 6 more at their own repository-root paths, all at home, plus the VPS (see [docs/ARCANE-GIT-SYNC.md](docs/ARCANE-GIT-SYNC.md) for how a repo commit @@ -38,12 +41,12 @@ why the home side split into this many Compose stacks): | `honeypot-keycloak` ([arcane/home/honeypot-keycloak/compose.yml](arcane/home/honeypot-keycloak/compose.yml)) | **home** | Arcane-managed Keycloak/PostgreSQL identity stack; only Keycloak is reachable from VPS Traefik over WireGuard | | `honeypot-init` ([arcane/home/honeypot-init/compose.yml](arcane/home/honeypot-init/compose.yml)) | **home** | one-shot bootstrap jobs: log paths, Elasticsearch templates, Arkime schema, persona validation | | `honeypot-cowrie`, `honeypot-dionaea`, `honeypot-conpot`, `honeypot-dnp3`, `honeypot-http`, `honeypot-multipot` (`arcane/home/honeypot-/compose.yml`, one directory each) | **home** | the sensors: Cowrie, Dionaea (+ TFTP relay), Conpot personas, DNP3, HTTP/API honeypots, multipot | -| `honeypot-dicompot`, `honeypot-dns-honeypot`, `honeypot-citrix`, `honeypot-cisco-asa`, `honeypot-sonicwall-sma`, `honeypot-rdp`, `honeypot-endlessh`, `honeypot-beelzebub`, `honeypot-hellpot`, `honeypot-elasticpot`, `honeypot-galah`, `honeypot-sentrypeer`, `honeypot-mailoney` (`arcane/home/honeypot-/compose.yml`, one directory each) | **home** | more sensors: DICOM medical-imaging decoy, DNS UDP reflection bait (response-capped, never a real amplification vector), Citrix ADC/NetScaler Gateway decoy (CVE-2019-19781), Cisco ASA WebVPN+IKE decoy (CVE-2018-0101), SonicWall SMA1000 Work Place/AMC decoy (CVE-2026-83548 Work Place SSRF), RDP decoy, SSH pre-auth tarpit, vendored multi-protocol deception runtime (SSH/LDAP/MCP/HTTP, #1418), vendored HTTP bot tarpit (#1419), vendored Elasticsearch decoy distinct from multipot's own (#1423), vendored LLM-powered HTTP honeypot behind its own broker-guarded bridge onto the shared Ollama instance (#1420), vendored SIP/VoIP fraud-detection honeypot (#1424), vendored SMTP honeypot taking over port 25 from multipot's own retired handler (#1422) — the row's `honeypot-wordpot` / WordPress/CMS decoy slot was removed when wordpot retired (#2381) | +| `honeypot-dicompot`, `honeypot-dns-honeypot`, `honeypot-citrix-honeypot`, `honeypot-cisco-asa-honeypot`, `honeypot-sonicwall-sma`, `honeypot-rdp-honeypot`, `honeypot-endlessh`, `honeypot-beelzebub`, `honeypot-hellpot`, `honeypot-elasticpot`, `honeypot-galah`, `honeypot-sentrypeer`, `honeypot-mailoney` (`arcane/home/honeypot-/compose.yml`, one directory each) | **home** | more sensors: DICOM medical-imaging decoy, DNS UDP reflection bait (response-capped, never a real amplification vector), Citrix ADC/NetScaler Gateway decoy (CVE-2019-19781), Cisco ASA WebVPN+IKE decoy (CVE-2018-0101), SonicWall SMA1000 Work Place/AMC decoy (CVE-2026-83548 Work Place SSRF), RDP decoy, SSH pre-auth tarpit, vendored multi-protocol deception runtime (SSH/LDAP/MCP/HTTP, #1418), vendored HTTP bot tarpit (#1419), vendored Elasticsearch decoy distinct from multipot's own (#1423), vendored LLM-powered HTTP honeypot behind its own broker-guarded bridge onto the shared Ollama instance (#1420), vendored SIP/VoIP fraud-detection honeypot (#1424), vendored SMTP honeypot taking over port 25 from multipot's own retired handler (#1422) — the row's `honeypot-wordpot` / WordPress/CMS decoy slot was removed when wordpot retired (#2381) | | `honeypot-canarytokens` ([arcane/home/honeypot-canarytokens/compose.yml](arcane/home/honeypot-canarytokens/compose.yml)) | **home** | self-hosted honeytoken platform (#1426) -- planted-artifact deception, not a listening protocol decoy; `canarytokens-adapter` translates its webhook alerts into this repo's shared JSON event shape. The dashboard's Settings > Canarytokens pane (#1487) creates PDF/Word/Excel/custom-image/Windows-Folder/QR tokens on demand for use *outside* this honeypot (#1662 dropped the stale design doc that described the pre-cutover plan; the shipped pane is authoritative) | | `honeypot-tanner` ([arcane/home/honeypot-tanner/compose.yml](arcane/home/honeypot-tanner/compose.yml)) | **home** | SNARE + TANNER application-emulation boundary | | `honeypot-elk` ([arcane/home/honeypot-elk/compose.yml](arcane/home/honeypot-elk/compose.yml)) | **home** | Filebeat, Elasticsearch, Kibana, EveBox, Arkime | | `honeypot-agent-intrusion-worker` ([arcane/home/honeypot-agent-intrusion-worker/compose.yml](arcane/home/honeypot-agent-intrusion-worker/compose.yml)) | **home** | the labelled corpus plus the Tier 1 contract benchmark. The worker itself was ported to Rust in #1610 and now runs as `WORKER_LOOPS=agent-intrusion` inside `honeypot-dashboard`'s `backend-worker`; the Python stack is retained under the `legacy` profile for rollback only, and the live `agent-intrusion-campaigns` index is written by the Rust loop | -| `honeypot-attacker-identity-worker`, `honeypot-correlator-worker`, `honeypot-payload-inventory-worker` (`arcane/home/honeypot-/compose.yml`, one directory each) | **home** | three more workers that had their own top-level compose file but had drifted out of the deploy/installer inventory before #1502's audit caught it (same class of gap #560 and #891 each fixed once before) -- attacker-identity correlation, cross-sensor campaign correlation, and payload inventory tracking | +| `honeypot-attacker-identity-worker`, `honeypot-correlator-worker`, `honeypot-payload-inventory-worker` (`arcane/home/honeypot-/compose.yml`, one directory each) | **home** | three more workers that had their own top-level compose file but had drifted out of the deploy/installer inventory before #1502's audit caught it (same class of gap #560 and #891 each fixed once before) -- attacker-identity correlation, cross-sensor campaign correlation, and payload inventory tracking. All three were retired by the same #1649 pass as the agent-intrusion worker: #1610 ported them to Rust, where they now run as `WORKER_LOOPS=attacker-identity` and `WORKER_LOOPS=correlator` on `honeypot-dashboard`'s `backend-worker` and `WORKER_LOOPS=payload-inventory` on `backend-worker-payload-inventory`. Like row above, the Python stacks are kept under the `legacy` profile for rollback only and are not the live writers | | `honeypot-dashboard` ([arcane/home/honeypot-dashboard/compose.yml](arcane/home/honeypot-dashboard/compose.yml)) | **home** | the live investigation dashboard: TanStack Start frontend (`dashboard-next`) in front of the Rust axum `backend-service` API tier, plus its worker containers (importer, networkless enrichment, the `WORKER_LOOPS` aggregation loops) and the services-adapter Docker control surface. Live since #1628's cutover completed 2026-08-22 -- the Go dashboard is deleted; see [docs/DASHBOARD-CUTOVER.md](docs/DASHBOARD-CUTOVER.md) and [docs/ARCHITECTURE.md](docs/ARCHITECTURE.md) | | `honeypot-dashboard-backend` ([arcane/home/honeypot-dashboard-backend/compose.yml](arcane/home/honeypot-dashboard-backend/compose.yml)) | **home** | the unprivileged read-only `backend-service` API tier (:8081), split out from `honeypot-dashboard` by #1622 so Arcane can redeploy the API tier without touching `dashboard-next`; the write-capable, host-spool-mounted instance is `backend-service-mounted` (:8082), which stayed in `honeypot-dashboard` | | `honeypot-payload-analysis` ([arcane/home/honeypot-payload-analysis/compose.yml](arcane/home/honeypot-payload-analysis/compose.yml)) | **home** | payload dedup + YARA scanning | @@ -91,7 +94,7 @@ for where those fit. | [docs/BACKUP-ESSENTIALS.md](docs/BACKUP-ESSENTIALS.md) | What is backed up so the stack can be rebuilt, where the three copies go, and the full restore procedure | | [scripts/install.sh](scripts/install.sh) | Single entry point for host provisioning — `sudo ./scripts/install.sh --profile home\|vps`, which dispatches to the installer below (or [scripts/install-vps.sh](scripts/install-vps.sh)) with that profile's answers file. Both share their retry/step/logging framework via [scripts/lib/install-common.sh](scripts/lib/install-common.sh) ([#1609](https://github.com/Xore/APIARY/issues/1609)) | | [scripts/install-homeserver.sh](scripts/install-homeserver.sh) | Unattended provisioning script (Docker, GPU/NVIDIA, WireGuard, Arcane, the stacks themselves) for a manually-installed base Ubuntu system — fill in [scripts/install-homeserver.conf.example](scripts/install-homeserver.conf.example) first, same idea as a Windows `autounattend.xml` answer file. First cut, see [#518](https://github.com/Xore/APIARY/issues/518) | -| [docs/ARCHITECTURE.md](docs/ARCHITECTURE.md) | System architecture and data flow — trust boundaries, container map, event ingestion, correlation/enrichment (p0f, HASSH/JA3/JA4, GeoIP), payload lifecycle, sandbox detonation, evidence types (6 diagrams) | +| [docs/ARCHITECTURE.md](docs/ARCHITECTURE.md) | System architecture and data flow — trust boundaries, container map, event ingestion, correlation/enrichment (p0f, HASSH/JA3/JA4, GeoIP), payload lifecycle, sandbox detonation, evidence types (4 diagrams) | | [docs/SENSORS.md](docs/SENSORS.md) | The sensor table, resource budgets, investigation UIs, SNARE+TANNER, Suricata, Arkime, and how real attacker IPs survive the tunnel | | [docs/OPERATIONS.md](docs/OPERATIONS.md) | Persona inventory, the seeded cowrie filesystem, GeoIP, and how to actually read the data (dashboard, Kibana, Arkime, backups) | | [docs/ip-reporting-plan.md](docs/ip-reporting-plan.md) | Defensive IP-blocklist reporting (AbuseIPDB/Blocklist.de), dry-run by default | @@ -104,7 +107,7 @@ for where those fit. | [docs/CONTAINER-UPDATES.md](docs/CONTAINER-UPDATES.md) | How to check pinned images for updates, assess compatibility, verify empirically, and pin by digest | | [docs/TESTING.md](docs/TESTING.md) | The three testing tiers -- CI, live feature smoke tests, and the full clean-reinstall release gate -- and how to repeat each one | | [docs/STACK-REBUILD.md](docs/STACK-REBUILD.md) | Runbook for a full deliberate reset — stop order, what's preserved vs wiped, and the ordering/permission pitfalls to avoid | -| [deploy-profiles/](deploy-profiles/) | Named deployment shapes (full / ICS-only / web-only) — which of the 20 split home stacks run for a given deployment, plus a validator catching cross-stack drift before deploy | +| [deploy-profiles/](deploy-profiles/) | Named deployment shapes (full / ICS-only / web-only) — which of the 26 split home stacks run for a given deployment, plus a validator catching cross-stack drift before deploy | | [docs/RECOVERY.md](docs/RECOVERY.md) | `factory-reset.sh` — one entry point for "back up, optionally wipe/reset, restart" on the same host | | [docs/ROADMAP.md](docs/ROADMAP.md) / [docs/WORK-LEDGER.md](docs/WORK-LEDGER.md) | What order work happens in, and how issues are claimed/reviewed | | [docs/ml-worker-plan.md](docs/ml-worker-plan.md), [docs/gpu-llm-analysis-worker.md](docs/gpu-llm-analysis-worker.md), [docs/gpu-ml-worker-acceleration.md](docs/gpu-ml-worker-acceleration.md) | The homeserver's NVIDIA GPU running local LLM log/payload analysis and CUDA-accelerated anomaly detection — no data leaves the machine | From 711ad7a7e492d85c6d6c96292de0c92986f55d5c Mon Sep 17 00:00:00 2001 From: xore Date: Sun, 27 Sep 2026 14:44:50 +0200 Subject: [PATCH 02/17] docs(map): drop the nonexistent archive/ tree, add design-lab/ to the exempt list docs/archive/ has no directory and no tracked files, but the map listed it twice (as a subdirectory, and as a dated record tree). The reachability script does exempt docs/design-lab/ -- kept as a near-duplicate of branding/design-lab/ -- and the map's description of that gate omitted it. Reachability result is unchanged: 85 reachable, 35 exempt. --- docs/README.md | 6 +++--- 1 file changed, 3 insertions(+), 3 deletions(-) diff --git a/docs/README.md b/docs/README.md index 77b79d3c..a41a8f9a 100644 --- a/docs/README.md +++ b/docs/README.md @@ -97,8 +97,8 @@ Analysis and sandbox components: era references inside are historical)](dashboard-manual-ip-block-design.md) Subdirectories (`analysis/`, `research/`, `sandbox/`, `vps/`, `autoinstall/`, -`deploy-profiles/`, `archive/`) hold the same kinds of documents scoped to +`deploy-profiles/`, `design-lab/`) hold the same kinds of documents scoped to their component. Every doc must be reachable from this page through links; dated record trees (`research/`, `benchmarks/`, the VM-detection results, -`archive/`) are exempt. `scripts/check-docs-reachable.py` enforces this in CI -(#3332). +`design-lab/`, kept as a near-duplicate of `branding/design-lab/`) are +exempt. `scripts/check-docs-reachable.py` enforces this in CI (#3332). From 29065ac3870ec38bd650c61e9a5cf3ca01c89625 Mon Sep 17 00:00:00 2001 From: xore Date: Sun, 27 Sep 2026 14:54:10 +0200 Subject: [PATCH 03/17] docs(sensors): fix retired dashboard container, ES budget, TANNER roster, endlessh PROXY - the investigation-UI table still named the deleted Go 'dashboard' container on :8090. The live view is dashboard-next, container :8080, published on home 19090/19092 and bridged from VPS 8090/8092 - Elasticsearch is limits.memory 12G with ES_JAVA_OPTS -Xms6g -Xmx6g, not 8 GiB / 4 GiB; dashboard-next gets 2 CPUs, not 1 - the TANNER container list omitted tanner_docker and listed snare_clone as one of this stack's services; it is a honeypot-init one-shot (hp-snare-clone) writing the snare-pages volume. The seven tanner_local services are now listed - endlessh parses PROXY_PROTOCOL=1 behind its :pp 2022 rule but was missing from the list of sensors that recover the real attacker IP that way --- docs/SENSORS.md | 21 +++++++++++++-------- 1 file changed, 13 insertions(+), 8 deletions(-) diff --git a/docs/SENSORS.md b/docs/SENSORS.md index 36faf392..1c2326c2 100644 --- a/docs/SENSORS.md +++ b/docs/SENSORS.md @@ -75,9 +75,9 @@ and dead-letter records for 60 days so high-volume scans cannot fill the disk. Every service has a CPU, memory, and Docker `json-file` log budget. The limits are intentionally generous relative to the host (16 logical CPUs and -91 GiB RAM): Elasticsearch gets 8 GiB with a 4 GiB heap; Arkime capture 6 GiB; +91 GiB RAM): Elasticsearch gets 12 GiB with a 6 GiB heap; Arkime capture 6 GiB; Kibana, Filebeat, and the TANNER analyzer receive 2 GiB; EveBox, Dionaea, Arkime -viewer, and the live dashboard receive 1 GiB (the dashboard also has one CPU). The +viewer, and the live dashboard receive 1 GiB (the dashboard also has two CPUs). The remaining lightweight sensors receive 128-512 MiB. Docker console logs rotate at 25 MiB with three files, independently from sensor event files under `./logs`. @@ -86,7 +86,7 @@ at 25 MiB with three files, independently from sensor event files under | Dashboard | Subdomain | Container | |---|---|---| -| Live sensor view (ours) | `honeypot.` | `dashboard` :8090 | +| Live sensor view (ours) | `honeypot.` | `dashboard-next` :8080 | | Kibana (ELK + Suricata) | `kibana.` | `kibana` :5601 | | TANNER web-attack analysis | `tanner.` | `tanner_web` :8091 | | EveBox (Suricata events) | `evebox.` | `evebox` :5636 | @@ -116,10 +116,13 @@ XSS, command execution, PHP code/object injection, XXE, CRLF and template injection) and stores sessions in Redis. TANNER's emulation containers are isolated from the homeserver Docker socket and are not a malware detonation environment. Suspicious payload detonation belongs in the separate KVM/libvirt -sandbox described in [`sandbox/README.md`](sandbox/README.md). Containers: `tanner_redis`, -`tanner_phpox`, `tanner_api`, `tanner` (analyzer, `:8090`), `tanner_web` -(dashboard, `:8091`), `snare_clone` (one-shot deterministic persona installer), -`snare` (`:8080`). The page source lives under [snare/persona](../arcane/home/honeypot-tanner/snare/persona) +sandbox described in [`sandbox/README.md`](sandbox/README.md). Containers: `tanner_docker` +(the nested disposable emulator daemon), `tanner_redis`, `tanner_phpox`, `tanner_api`, +`tanner` (analyzer, `:8090`), `tanner_web` (dashboard, `:8091`), `snare` (`:8080`) -- +seven services, all on `tanner_local`. The persona clone itself is not one of them: +`snare-clone` is a one-shot job in `honeypot-init` (`hp-snare-clone`) that writes the +`snare-pages` volume `snare` reads. The page source lives under +[snare/persona](../arcane/home/honeypot-tanner/snare/persona) and is transformed into SNARE's content-addressed store during the image build; no third-party site is cloned. All `mushorg/*` images are third-party — verify tags/args upstream (needs a live build/pull). @@ -180,7 +183,9 @@ Web UI: `http://:19080` (`arkime.` via Traefik). > **http/api-honeypots** (`PROXY_PROTOCOL=1`), **dnp3** (`PROXY_PROTOCOL=1`), > **dicompot** (`PROXY_PROTOCOL=1`), **citrix-honeypot**, > **sonicwall-sma-honeypot** (`PROXY_PROTOCOL=1`), -> **cisco-asa-honeypot**'s WebVPN side and **rdp-honeypot** (`PROXY_PROTOCOL=1`) and **all conpot sensors** (`CONPOT_PROXY_PROTOCOL=1`, gevent shim baked in +> **cisco-asa-honeypot**'s WebVPN side and **rdp-honeypot** (`PROXY_PROTOCOL=1`), +> **endlessh** (`PROXY_PROTOCOL=1`, public 2022) and **all conpot sensors** +> (`CONPOT_PROXY_PROTOCOL=1`, gevent shim baked in > by `conpot/proxy_patch.py`) parse it, so those events log the true IP and > port. The http listener sniffs the header, so Traefik-routed requests (no > header) keep working too. From e9117f797526095963db5b3bfcf10c967eea2dd1 Mon Sep 17 00:00:00 2001 From: xore Date: Sun, 27 Sep 2026 14:57:30 +0200 Subject: [PATCH 04/17] docs(deception): add the missing SonicWall SMA decoy row The 'Service-specific decoys' section states its purpose as backfilling rows so every running sensor traces back to a decision, but honeypot-sonicwall-sma (#3033, CVE-2026-83548 / CVE-2026-83549) had no row anywhere in the file. Every other one of the 20 sensor stacks does. All other claims verified against compose and the source: wordpot's directory is gone, the 2026-08-27 retirement date matches, conpot's six personas, and #233/#242 as cited in community-threat-intel-sharing.md. --- docs/DECEPTION-EXTENSIONS.md | 1 + 1 file changed, 1 insertion(+) diff --git a/docs/DECEPTION-EXTENSIONS.md b/docs/DECEPTION-EXTENSIONS.md index 14255209..57fcb72d 100644 --- a/docs/DECEPTION-EXTENSIONS.md +++ b/docs/DECEPTION-EXTENSIONS.md @@ -109,6 +109,7 @@ traces back to at least one decision. | RDP decoy (rdphoneypot lineage) | Integrated | `arcane/home/honeypot-rdp-honeypot/` | #238 batch, per-decoy plan #412 | | Cisco ASA VPN gateway decoy | Integrated | `arcane/home/honeypot-cisco-asa-honeypot/` | #238 batch, CVE context #414 | | Citrix ADC gateway decoy | Integrated | `arcane/home/honeypot-citrix-honeypot/` | #238 batch, CVE context #414 | +| SonicWall SMA1000 Work Place/AMC decoy | Integrated | `arcane/home/honeypot-sonicwall-sma/` | #3033; CVE-2026-83548 SSRF / CVE-2026-83549 AMC command-injection chain | | DNS amplification bait | Integrated | `arcane/home/honeypot-dns-honeypot/` | #238 batch, safety-sensitive design #415 | | Mailoney (SMTP) | Integrated | `arcane/home/honeypot-mailoney/` | #1422 | | SentryPeer (VoIP/SIP) | Integrated | `arcane/home/honeypot-sentrypeer/` | #1424 | From 8eec9c7fa34db977a7ecd18a93ad03f5e43d6549 Mon Sep 17 00:00:00 2001 From: Xore Date: Sun, 27 Sep 2026 14:58:42 +0200 Subject: [PATCH 05/17] docs(sensors): correct the Suricata capture interface, ILM retention and payload-dedupe logging claims --- docs/SENSORS.md | 16 ++++++++++++---- 1 file changed, 12 insertions(+), 4 deletions(-) diff --git a/docs/SENSORS.md b/docs/SENSORS.md index 1c2326c2..e6a0b020 100644 --- a/docs/SENSORS.md +++ b/docs/SENSORS.md @@ -52,8 +52,9 @@ The `payload-dedupe` service scans these stores hourly and atomically replaces same-filesystem duplicates with hard links. Existing event/download URLs remain valid while duplicate disk blocks are reclaimed; its last-run report is stored at `state/dedupe/payload-dedupe.json`. -Its diagnostic logger is limited to `info,warning,error` so debug chatter cannot -consume the data disk. The `log-maintenance` sidecar copy-truncates and gzips +Neither service produces per-event chatter: `payload-dedupe` prints a single +JSON result line per pass, and `log-maintenance` writes one stderr line per +rotation. The `log-maintenance` sidecar copy-truncates and gzips human-readable Dionaea, Conpot, and Cowrie logs at 256 MiB (four archives). Structured JSON event streams are deliberately never rotated by that sidecar, which preserves Filebeat offsets and dashboard ingestion. @@ -70,6 +71,11 @@ GeoIP enrichment is best-effort: empty or malformed addresses are skipped, but the original event is always retained. ILM keeps raw Suricata indices for 7 days, honeypot data streams for 30 days, and dead-letter records for 60 days so high-volume scans cannot fill the disk. +Those three are the values at the default `HONEYPOT_RETENTION_DAYS=30`: every +window derives from that one variable (Suricata `retention*7/30`, dead-letter +`retention*2`), so lowering it reclaims disk across all of them at once. Only +the ILM *policy names* (`suricata-7d`, `honeypot-30d`, `dead-letter-60d`) stay +fixed. ## Runtime resource budgets @@ -130,8 +136,10 @@ no third-party site is cloned. All `mushorg/*` images are third-party ## Suricata — analysing all the traffic `suricata` runs **on the VPS** (host networking, sniffing the public interface -`SURICATA_IFACE`, default `ens6`) so it sees real attacker source IPs before the -tunnel. It writes to `/opt/stacks/apiary/logs/suricata/` on the VPS: +`CAPTURE_INTERFACE`, written at boot by `detect-capture-interface.service`, +falling back to the legacy `SURICATA_IFACE` and then to `eth0` — *not* the old +`ens6`, which a reboot has already renamed) so it sees real attacker source IPs +before the tunnel. It writes to `/opt/stacks/apiary/logs/suricata/` on the VPS: - `eve.json` (alerts, http, dns, tls, flow) — Filebeat on the home server ships it to the `suricata-*` Elasticsearch index (stats events are dropped, see From 3186eb47910b22cfb114e1662eda9dde74a92951 Mon Sep 17 00:00:00 2001 From: Xore Date: Sun, 27 Sep 2026 15:00:41 +0200 Subject: [PATCH 06/17] docs(roadmap,design-lab): correct CAPEv2 build status, doc-sweep scope and the design-lab harness pointer --- docs/ROADMAP.md | 27 +++++++++++++++++++++------ docs/design-lab/README.md | 16 ++++++++++++++-- 2 files changed, 35 insertions(+), 8 deletions(-) diff --git a/docs/ROADMAP.md b/docs/ROADMAP.md index 50dc0c04..c3256004 100644 --- a/docs/ROADMAP.md +++ b/docs/ROADMAP.md @@ -13,6 +13,11 @@ Last audited: 2026-08-05 > Deleted by the 2026-08-30 bulk purge and restored verbatim by #2896/#2947 > on 2026-09-04. Nothing below was updated during that window — treat > "last audited" above as still true, not as of the restore date. +> +> The *repository* did keep moving across that same window, though, so the +> audit date is not a safe lower bound for "still true": reconcile any status +> claim here against the code before relying on it. The CAPEv2 line above is +> one already re-verified (2026-09-27) and found stale. ## Current baseline @@ -35,12 +40,22 @@ Last audited: 2026-08-05 booting; the end-to-end submit-to-report path is verified for the Linux/Wine sandbox and GitHub-analysis publishing. The Windows-11 golden image epic ([#47](https://github.com/Xore/APIARY/issues/47)) is - still open — see the Windows sandbox section below. CAPEv2 (#314-322) - remains unbuilt and is post-0.1.0 backlog. -- Documentation has been consolidated: every doc that used to be scattered - next to its source now lives under `docs/`, mirroring the source tree - ([#670](https://github.com/Xore/APIARY/issues/670), closed - 2026-08-05). + still open — see the Windows sandbox section below. CAPEv2 (#314-322) is + **authored, not deployed**: [#843](https://github.com/Xore/APIARY/issues/843) + (2026-09-01) landed `sandbox/cape/` — compose stack, Packer + `win11-cape.pkr.hcl`, CAPEv2 override units and a spool worker — but unlike + GHOSTS it has no entry in `arcane/manifests/home-production.json`, so nothing + Dockge-managed deploys it. Treat it as post-0.1.0 backlog whose build work + has already started, not as unbuilt. +- Documentation has been consolidated: the subsystem docs that used to be + scattered next to their source now live under `docs/`, mirroring the source + tree ([#670](https://github.com/Xore/APIARY/issues/670), closed + 2026-08-05). The exceptions are deliberate and catalogued in + [`docs/README.md`](README.md) — per-stack and vendored `README.md` files stay + next to their code, and a few dated record trees are exempt from the + reachability gate (#3332). Root-level dated task records (`DIFF.md`, + `EVIDENCE.md`, `HANDOFF-3097.md`) are the residue of that sweep and are not + covered by it. - The dashboard rewrite is done, not pending: the TanStack Start frontend/BFF + Rust service tier ([#1608](https://github.com/Xore/APIARY/issues/1608) and its diff --git a/docs/design-lab/README.md b/docs/design-lab/README.md index 2b4b5071..d658e325 100644 --- a/docs/design-lab/README.md +++ b/docs/design-lab/README.md @@ -67,5 +67,17 @@ The original lab served variant builds against real Elasticsearch data on ports 19201–19205, driven by an env-guarded Go test that booted the dashboard with a stubbed OIDC session, a `STATIC_DIR` override and nil write-services so the real index stayed read-only. That harness depended on the Go dashboard -and went away with it. A `frontend-next` equivalent needs the same read-only -guarantees; scoped separately. +and went away with it. + +It has since been rebuilt for `frontend-next` as +[`branding/design-lab/lab.mjs`](../../branding/design-lab/lab.mjs) (#1828, +#1935), which serves variants on the same 19201–19205 range and the elements +playground on 19300. `frontend-next` has no nil-write-services handle — it +reaches data over HTTP through two bases — so the read-only guarantee is made +at that seam instead: `BACKEND_URL` goes through a gate that forwards +GET/HEAD and answers 405 to everything else, and `BACKEND_MOUNTED_URL` is left +unset so every request 503s. That is stronger than the original, which relied +on remembering to pass nil. + +This directory is the redacted public copy and does not carry the harness +itself; run it from `branding/design-lab/`. From 788beee59c9ce27b70d675a7db6bb5916d00c8b4 Mon Sep 17 00:00:00 2001 From: Xore Date: Sun, 27 Sep 2026 15:01:42 +0200 Subject: [PATCH 07/17] docs(security-fixes): date the CodeQL record and point its Go-era examples at the Rust tier --- docs/security-fixes.md | 11 +++++++++++ 1 file changed, 11 insertions(+) diff --git a/docs/security-fixes.md b/docs/security-fixes.md index 3d2b9f49..cf9dbc23 100644 --- a/docs/security-fixes.md +++ b/docs/security-fixes.md @@ -1,5 +1,16 @@ # Code scanning +> **Dated record — 2026-07-30.** The CodeQL history below is a worked example, +> not a live status page; it is here for the lesson, per "do not mirror a live +> system's state into a markdown file" (which is also why the current alerts +> live in [#80](https://github.com/Xore/APIARY/issues/80) and the +> [Security tab](https://github.com/Xore/APIARY/security/code-scanning)). Every +> code path it names — `dashboard/sandbox.go`, `ghidra.go`, +> `dashboard/static/hp-adminlte.js` — is Go-era and no longer exists: the Go +> dashboard was deleted at #1628 (2026-08-22). The equivalents are now Rust in +> `arcane/home/honeypot-dashboard/backend-service/src/` (`artifacts.rs`, +> `report_pdf.rs`). The method below is unchanged and still current. + Open CodeQL alerts are tracked in [#80](https://github.com/Xore/APIARY/issues/80) and in the [Security tab](https://github.com/Xore/APIARY/security/code-scanning), From 4f9fa6443fd5a4f47b7727cc19bc74e2bd3530c4 Mon Sep 17 00:00:00 2001 From: Xore Date: Sun, 27 Sep 2026 15:02:26 +0200 Subject: [PATCH 08/17] docs(sensors): state the ILM windows at the shipped HONEYPOT_RETENTION_DAYS=21, not the code fallback --- docs/SENSORS.md | 13 ++++++++----- 1 file changed, 8 insertions(+), 5 deletions(-) diff --git a/docs/SENSORS.md b/docs/SENSORS.md index e6a0b020..b7316ca7 100644 --- a/docs/SENSORS.md +++ b/docs/SENSORS.md @@ -71,11 +71,14 @@ GeoIP enrichment is best-effort: empty or malformed addresses are skipped, but the original event is always retained. ILM keeps raw Suricata indices for 7 days, honeypot data streams for 30 days, and dead-letter records for 60 days so high-volume scans cannot fill the disk. -Those three are the values at the default `HONEYPOT_RETENTION_DAYS=30`: every -window derives from that one variable (Suricata `retention*7/30`, dead-letter -`retention*2`), so lowering it reclaims disk across all of them at once. Only -the ILM *policy names* (`suricata-7d`, `honeypot-30d`, `dead-letter-60d`) stay -fixed. +Those are the values at the *code* fallback `HONEYPOT_RETENTION_DAYS=30`. Every +window derives from that one knob, and the shipped configuration does **not** +use the fallback: all four tracked `.env.example` files set +`HONEYPOT_RETENTION_DAYS=21` (#2820), at which the three windows above become +**4d / 21d / 42d** — Suricata is `retention*7/30` (integer-truncated, floored +at 1) and dead-letter is `retention*2`. Only the ILM *policy names* +(`suricata-7d`, `honeypot-30d`, `dead-letter-60d`) stay fixed; they are +identifiers referenced by the index templates, not claims about duration. ## Runtime resource budgets From 4017eefc5d35be88e8ab03e1044ea59d25a49184 Mon Sep 17 00:00:00 2001 From: Xore Date: Sun, 27 Sep 2026 15:03:36 +0200 Subject: [PATCH 09/17] docs(reporting,bistreams): redraw the reporter diagrams against the Go service, date the retention decision --- docs/dionaea-bistreams-retention.md | 10 +++++ docs/ip-reporting-plan.md | 63 ++++++++++++++++++++--------- 2 files changed, 55 insertions(+), 18 deletions(-) diff --git a/docs/dionaea-bistreams-retention.md b/docs/dionaea-bistreams-retention.md index 48892313..df6fdea5 100644 --- a/docs/dionaea-bistreams-retention.md +++ b/docs/dionaea-bistreams-retention.md @@ -1,5 +1,15 @@ # Dionaea bistreams retention — consumer inventory and decision (#2862) +> **Dated decision record — 2026-09-03.** `BISTREAMS_RETENTION_DAYS=30` is +> still the pinned value in `arcane/home/honeypot-payload-analysis/.env.example` +> and the decision has not been reopened. Read the forward-looking sections +> ("nothing is 30 days old yet", "when this window first destroys something — +> 2026-09-09") as written on 2026-09-03: that date has passed, so the pruning +> path is now live rather than pending, and the size projections below are the +> ones that were made then, not current measurements. Re-measure before acting +> on the capacity numbers. The consumer inventory and the forensic argument are +> unaffected by the passage of time and are the substance of this document. + `dionaea-lib`'s `bistreams/` tree holds Dionaea's raw per-connection capture stream: every accepted connection gets a date-named subdirectory (`YYYY-MM-DD/`) full of raw capture files, payload or not — a superset of diff --git a/docs/ip-reporting-plan.md b/docs/ip-reporting-plan.md index 1b396d40..443841c6 100644 --- a/docs/ip-reporting-plan.md +++ b/docs/ip-reporting-plan.md @@ -61,12 +61,15 @@ Primary target: **AbuseIPDB** — widely used, has a public confidence score, an ```mermaid flowchart TD Sensors["Cowrie / Dionaea / Conpot / HTTP-honeypot / DNP3"] - Reporter["reporter (Python)
new Docker Compose service"] + Reporter["reporter (Go)
hp-reporter, dry-run by default"] AbuseIPDB["AbuseIPDB"] Blocklist["Blocklist.de"] + GreyNoise["GreyNoise RIOT
(read-only pre-check)"] Sensors -->|"JSON event logs on the shared Docker volumes, tailed —
not Redis pub-sub; see 'Resolved design decisions'"| Reporter Reporter -->|"POST /api/v2/reports"| AbuseIPDB + Reporter -->|"POST /api"| Blocklist + Reporter -->|"GET /v3/riot/{ip}"| GreyNoise AbuseIPDB -->|optional| Blocklist ``` @@ -165,27 +168,32 @@ Before reporting, cross-check against: Add to `docker-compose.yml`: +> The block below is the original sketch. It is superseded — see the status +> banner. Three things in it are simply wrong against what shipped, and are +> called out because they are the kind of detail that gets copy-pasted: +> there is **no Prometheus port** (Phase 4 emits `metrics.json` into the data +> volume instead), the Blocklist.de credentials are **`BLOCKLISTDE_SENDER` + +> `BLOCKLISTDE_API_KEY`**, not `BLOCKLIST_DE_EMAIL`/`BLOCKLIST_DE_PASSWORD`, +> and the live switch is `REPORTER_LIVE`. The `whitelist.txt` mount path and +> `REPORTER_COOLDOWN_HOURS` did land as drawn. + ```yaml reporter: build: ./reporter restart: unless-stopped environment: ABUSEIPDB_API_KEY: ${ABUSEIPDB_API_KEY} - BLOCKLIST_DE_EMAIL: ${BLOCKLIST_DE_EMAIL} - BLOCKLIST_DE_PASSWORD: ${BLOCKLIST_DE_PASSWORD} - GREYNOISE_API_KEY: ${GREYNOISE_API_KEY:-} # optional + BLOCKLISTDE_SENDER: ${BLOCKLISTDE_SENDER} + BLOCKLISTDE_API_KEY: ${BLOCKLISTDE_API_KEY} + GREYNOISE_API_KEY: ${GREYNOISE_API_KEY:-} # inert unless GREYNOISE_ENABLED=1 + REPORTER_LIVE: ${REPORTER_LIVE:-} # unset = dry-run REPORTER_COOLDOWN_HOURS: ${REPORTER_COOLDOWN_HOURS:-24} - REPORTER_WHITELIST: /config/whitelist.txt volumes: - cowrie_logs:/logs/cowrie:ro - dionaea_logs:/logs/dionaea:ro - conpot_logs:/logs/conpot:ro - reporter_data:/data - ./reporter/whitelist.txt:/config/whitelist.txt:ro - ports: - - "127.0.0.1:9101:9101" # Prometheus metrics - networks: - - honeypot_internal ``` Add to `.env.example`: @@ -193,9 +201,11 @@ Add to `.env.example`: ```dotenv # IP Blocklist Reporting ABUSEIPDB_API_KEY= -BLOCKLIST_DE_EMAIL= -BLOCKLIST_DE_PASSWORD= +BLOCKLISTDE_SENDER= +BLOCKLISTDE_API_KEY= GREYNOISE_API_KEY= +GREYNOISE_ENABLED=0 +REPORTER_LIVE= # leave empty: the reporter is dry-run until set REPORTER_COOLDOWN_HOURS=24 ``` @@ -225,20 +235,37 @@ The reporter will track a daily counter and pause with exponential back-off on ` ## Files To Create +The Python layout originally sketched here was never built — the shipped +service is Go. This is what +`arcane/home/honeypot-utilities/reporter/` actually contains: + ```mermaid flowchart TD - ReporterDir["reporter/"] --> Dockerfile["Dockerfile"] - ReporterDir --> Requirements["requirements.txt"] - ReporterDir --> ReporterPy["reporter.py
main loop"] - ReporterDir --> SourcesPy["sources.py
per-sensor log parsers"] - ReporterDir --> ApisPy["apis.py
AbuseIPDB + Blocklist.de clients"] - ReporterDir --> DedupPy["dedup.py
SQLite-backed deduplication"] + ReporterDir["reporter/ (Go)"] --> MainGo["main.go
entrypoint, run loop wiring"] + MainGo --> RunloopGo["runloop.go
poll tick"] + RunloopGo --> TailGo["tail.go
per-sensor log tailing"] + TailGo --> EventGo["event.go
normalised event"] + EventGo --> CategorizeGo["categorize.go
sensor/kind to upstream category"] + CategorizeGo --> DedupGo["dedup.go
SQLite-backed deduplication"] + DedupGo --> GreynoiseGo["greynoise.go
RIOT pre-check (Phase 3)"] + GreynoiseGo --> WhitelistGo["whitelist.go
CIDR/IP allowlist"] + WhitelistGo --> BlocklistdeGo["blocklistde.go
Blocklist.de client"] + WhitelistGo --> ReportGo["report.go
AbuseIPDB client"] + GreynoiseGo --> ProcessGo["process.go
report decision + dispatch"] + ProcessGo --> ReportGo + ProcessGo --> BlocklistdeGo + ProcessGo --> MetricsGo["metrics.go
counters to metrics.json"] + ProcessGo --> AuditGo["audit.go
bounded rotating audit log"] ReporterDir --> WhitelistTxt["whitelist.txt
safe IPs/CIDRs to never report"] - ReporterDir --> MetricsPy["metrics.py
Prometheus exporter"] + ReporterDir --> Dockerfile["Dockerfile
FROM scratch, runs as 0:0"] DocsDir["docs/"] --> PlanMd["ip-reporting-plan.md
this file"] ``` +Every `.go` file has a matching `_test.go`. There is no `requirements.txt`, +no `sources.py`/`apis.py`/`dedup.py`/`metrics.py`, and no Prometheus +exporter — see the Phase 4 note in the status banner. + --- ## Resolved design decisions From 6a2eee8c1ec6862d03f62df7a0dacb0489f28c17 Mon Sep 17 00:00:00 2001 From: Xore Date: Sun, 27 Sep 2026 15:05:14 +0200 Subject: [PATCH 10/17] docs(threat-intel): correct #153 to closed-and-implemented in the reporter comparison --- docs/community-threat-intel-sharing.md | 7 +++++-- 1 file changed, 5 insertions(+), 2 deletions(-) diff --git a/docs/community-threat-intel-sharing.md b/docs/community-threat-intel-sharing.md index 64a558d2..89a89f05 100644 --- a/docs/community-threat-intel-sharing.md +++ b/docs/community-threat-intel-sharing.md @@ -54,8 +54,11 @@ commitment than this repo's existing IP-blocklist reporting: destinations (AbuseIPDB, Blocklist.de), stays dry-run by default, and needed its own multi-phase build (#68 for the dry-run foundation and safeguards, #69 for validation and metrics, #153 for reputation - filtering and observability, still open) to get the privacy posture - right. + filtering and observability) to get the privacy posture + right. All three are closed and implemented: #153 shipped GreyNoise + RIOT pre-checks (`reporter/greynoise.go`, off unless `GREYNOISE_ENABLED=1`) + and a `metrics.json` counter snapshot that the dashboard mirrors into + `reporter-metrics-v1`. - A generic `hpfeeds` publisher would share *richer* structured data (commands, credentials, payload hashes, session metadata -- whatever TANNER or another sensor chose to publish) with *whichever broker an From b5572f674beea62fc5d2c2c40bfa8f6557cb2e7e Mon Sep 17 00:00:00 2001 From: xore Date: Sun, 27 Sep 2026 15:05:28 +0200 Subject: [PATCH 11/17] docs(threat-model): resolve two follow-ups the body still reported as open Both were already marked Done in the doc's own Follow-up scope section; the section bodies and the applicability matrix were never updated to match. - item 9: llm-analysis severity is wired into the Rust alert sink as llm_flagged_alerts (worker.rs:1277), not browse-only via /llm-analysis - item 5: hp-autoheal no longer bind-mounts /var/run/docker.sock. #592 moved it onto hp-docker-socket-proxy, which holds the socket :ro scoped to CONTAINERS/IMAGES/POST on a private network. The residual is that CONTAINERS=1 stays daemon-wide, which is now what the text says - the docker.sock grep aside claimed 'absent' from the dashboard compose, but that file does contain one mount (services-adapter's) -- made explicit so someone re-running the grep is not surprised ROADMAP.md: verified, no drift. No 'next' profile survives anywhere, CAPE is authored but absent from the manifest while GHOSTS is present, sandbox/cape has the Packer file and spool worker it claims, reporter/ sits under honeypot-utilities, and llm-worker's selftest exists. --- docs/agent-intrusion-threat-model.md | 51 +++++++++++++++------------- 1 file changed, 28 insertions(+), 23 deletions(-) diff --git a/docs/agent-intrusion-threat-model.md b/docs/agent-intrusion-threat-model.md index fbaad1c9..99789f98 100644 --- a/docs/agent-intrusion-threat-model.md +++ b/docs/agent-intrusion-threat-model.md @@ -233,20 +233,24 @@ gap is outbound-to-internet egress policy, tracked separately in #538. **Substantially addressed for the dashboard; inconsistent elsewhere.** - The dashboard's own Docker-socket boundary is the strongest example in - this tree: the dashboard container itself never mounts `/var/run/docker.sock` - (`arcane/home/honeypot-dashboard/compose.yml`, grepped directly — absent). All + this tree: the dashboard containers themselves never mount + `/var/run/docker.sock` (`arcane/home/honeypot-dashboard/compose.yml`, grepped + directly — its one socket mount belongs to `services-adapter`, below). All Docker-lifecycle actions (start/stop/restart) go through `hp-services-adapter`, a separate container that is `cap_drop: [ALL]`, `read_only: true`, `network_mode: none`, and reachable only via an AF_UNIX socket the dashboard also holds — no TCP path exists to abuse it remotely even if the dashboard container itself were compromised. -- `hp-autoheal` is the one other service that does bind-mount the real - `/var/run/docker.sock` (`arcane/home/honeypot-utilities/compose.yml`) — it watches - containers by label daemon-wide and restarts unhealthy ones. This is a - broad, standing grant (full Docker API access, not scoped to specific - containers) held by a long-running service; workload-identity-scoped - alternatives (e.g. a narrower label-filtered API surface) were not found - in this tree. +- `hp-autoheal` was the one other service that bind-mounted the real + `/var/run/docker.sock` (`arcane/home/honeypot-utilities/compose.yml`) — it + watches containers by label daemon-wide and restarts unhealthy ones. **That + grant has since been narrowed by #592** (the strikethrough item in + [Follow-up scope](#follow-up-scope)): it now talks to `hp-docker-socket-proxy` + at `tcp://docker-socket-proxy:2375` with no socket bind mount of its own, and + the proxy holds the socket `:ro` scoped to `CONTAINERS=1`, `IMAGES=1`, + `POST=1` on a private network. What remains is that `CONTAINERS=1` is still + daemon-wide rather than label-filtered — the label scoping is + `AUTOHEAL_CONTAINER_LABEL` inside autoheal, not an API-side restriction. - `tanner_docker` (`arcane/home/honeypot-tanner/compose.yml`) is `privileged: true` with its own `tmpfs /var/lib/docker` — explicitly isolated Docker-in-Docker on the private `tanner_local` network, not a bind mount of the host socket. @@ -259,11 +263,10 @@ gap is outbound-to-internet egress policy, tracked separately in #538. repo has to short-lived workload identity today. **Verdict:** the dashboard/services-adapter split is a good existing -least-privilege pattern worth citing as the template for any future -privileged-access surface. `hp-autoheal`'s standing daemon-wide Docker -socket grant is the one credential-lifetime/least-privilege gap worth a -scoped look (whether its watch scope can be narrowed), separate from #88's -network-isolation focus. +least-privilege pattern worth citing as the template for any new +privileged-access surface. It is also now the template autoheal was moved onto: +its raw socket grant is gone, replaced by the same narrow-proxy shape, leaving +only the daemon-wide `CONTAINERS=1` scope. --- @@ -400,8 +403,10 @@ finding.** state; the same fragmentation carried into the Rust cutover's per-source worker functions): Suricata/sensor event alerts, `ghidraAlerts`, `githubAnalysisAlerts`, sandbox queue/verdict alerts, ML anomaly severity, and (as of #150) the - new `llm-analysis` index's own severity field is not yet wired into - `alerts.go` at all — it is currently browse-only via `/llm-analysis`. + new `llm-analysis` index's own severity field — which was browse-only via + `/llm-analysis` until it was wired into the sink as `llm_flagged_alerts` in + the Rust cutover, the "Done" item in [Follow-up scope](#follow-up-scope) + below. - No single "this source_ip/session/sample crossed N independent trust boundaries in a Y-minute window" correlation exists — each alert source answers its own narrow question. This is exactly the shape the campaign's @@ -413,11 +418,11 @@ finding.** identity for investigation, not by trust-boundary-crossing count for alerting. -**Verdict:** this is the clearest concrete gap this research surfaced. Wiring -`llm-analysis`'s severity into the existing alert sink is a small, immediate -follow-up; a genuine cross-source trust-boundary-crossing correlation engine -is squarely phase 3's scope, not something to build inside this research -pass. +**Verdict:** this is the clearest concrete gap this research surfaced. Its +smallest piece — wiring `llm-analysis`'s severity into the alert sink — has +since landed (`llm_flagged_alerts` in the Rust `alert-notifier`); what remains +is a genuine cross-source trust-boundary-crossing correlation engine, which is +squarely phase 3's scope, not something to build inside this research pass. --- @@ -429,11 +434,11 @@ pass. | 2 | Untrusted structured-data processing | Yes, broadly | `html/template` auto-escaping; CI YARA corpus gate | No archive/container-format parsing exists yet — must inherit this discipline when added | | 3 | Env/`/proc/*/environ` secret exposure | Yes | `secretFromEnvironment`'s `_FILE` pattern (one use) | Pattern not applied to `ARKIME_*`, `GH_PAT`, VPS SSH key | | 4 | Metadata-service / RFC 1918 reachability | No cloud metadata surface exists | Per-sensor private Docker networks | Outbound-to-internet egress policy (tracked in #538) | -| 5 | Credential lifetime / workload identity | Yes | dashboard/services-adapter split (strong pattern, since carried into the backend-service/worker split) | `hp-autoheal`'s standing daemon-wide docker.sock grant | +| 5 | Credential lifetime / workload identity | Yes | dashboard/services-adapter split (strong pattern, since carried into the backend-service/worker split); autoheal moved onto the same narrow-proxy shape by #592 | Proxy's `CONTAINERS=1` is still daemon-wide, not label-filtered | | 6 | Encoded/chunked C2 | Yes, as honeypot capture surface | Raw payload capture (tanner/Suricata); narrow fixed-destination outbound HTTP clients | No network-layer egress enforcement (folds into #538) | | 7 | Repeated recon / low-signal escalation | Yes — core motivating gap | ml-worker anomaly scoring; dashboard campaign clustering | No behavioral-phase correlation or combination-based severity escalation | | 8 | Source-control/CI write paths | Yes | `analysis/github/` publish gate (CI-tested); vendored-dep hash pinning | No image digest pinning for this repo's own built images | -| 9 | Cross-source alert correlation | Yes — core motivating gap | Multiple independent alert sources feed one sink | No trust-boundary-crossing correlation; `llm-analysis` severity not yet wired into alerts | +| 9 | Cross-source alert correlation | Yes — core motivating gap | Multiple independent alert sources feed one sink, `llm-analysis` severity included | No trust-boundary-crossing correlation engine | --- From 5d6f4d80d971f29dbcb7af205bd42c00bcc8de1f Mon Sep 17 00:00:00 2001 From: Xore Date: Sun, 27 Sep 2026 15:06:30 +0200 Subject: [PATCH 12/17] docs(ip-block,agent-intrusion): record the ip-block export path move and the unported sidecar default --- docs/agent-intrusion-threat-model.md | 7 +++++-- docs/dashboard-manual-ip-block-design.md | 21 +++++++++++++++++---- 2 files changed, 22 insertions(+), 6 deletions(-) diff --git a/docs/agent-intrusion-threat-model.md b/docs/agent-intrusion-threat-model.md index 99789f98..2bb52e11 100644 --- a/docs/agent-intrusion-threat-model.md +++ b/docs/agent-intrusion-threat-model.md @@ -36,9 +36,12 @@ For each of the nine areas #154 asked to cover, this maps to APIARY's **actual** current architecture — verified against the real compose files, -Go/Python source, and docs in this tree, not assumed from what a "typical" +the source, and docs in this tree, not assumed from what a "typical" honeypot stack might do. Each entry records: what exists today, whether the -published campaign's technique applies here, and the evidence. +published campaign's technique applies here, and the evidence. (The original +pass read Go and Python source; the Go tier was deleted at #1628 on +2026-08-22, so a re-read today should be against the Rust modules named in +the status banner above.) --- diff --git a/docs/dashboard-manual-ip-block-design.md b/docs/dashboard-manual-ip-block-design.md index 2a2d979a..3689aa7d 100644 --- a/docs/dashboard-manual-ip-block-design.md +++ b/docs/dashboard-manual-ip-block-design.md @@ -100,18 +100,31 @@ that already exists** (home is reachable at `10.8.0.2`, `docs/CGNAT-DEPLOYMENT.md`), the same "pull, don't get pushed to" posture `portbridge-blackhole-refresh.sh` already uses against GitHub. Concretely: -- The dashboard exposes `GET /export/portbridge-manual-blackhole.txt` - (then `dashboard/ip_block.go`'s `serveManualBlackholeExport`, now - `backend-service/src/ip_block.rs`'s `export`) — plain text, one +- The dashboard exposes the export as `GET /api/v1/ip-block-export` + (`backend-service/src/ip_block.rs`'s `export`) — plain text, one IPv4 address per line, the exact format `blackhole.go`'s existing parser already reads. No admin auth on the handler itself, the same posture every - other `/export/*.csv` GET already takes (access control is the network + other `/api/v1/export/*.csv` GET already takes (access control is the network boundary — WireGuard-only reachability — not a second app-layer secret); the data itself (a list of IPs an operator already chose to block) is no more sensitive than the maltrail feed it sits alongside. Reachable from the VPS at `10.8.0.2:19090` — the `dashboard` service's real published port (`arcane/home/honeypot-dashboard/compose.yml`, `${HP_BIND:-10.8.0.2}:19090:8080`), not an assumed default. + **The path moved at the Rust cutover and one caller was not carried across.** + This decision was written against the Go route + `GET /export/portbridge-manual-blackhole.txt` (then `dashboard/ip_block.go`'s + `serveManualBlackholeExport`); the Rust `export` keeps that handler's + *byte-compatible body* but is registered at `/api/v1/ip-block-export`, and + `backend-service/src/main.rs` has no `/export/portbridge-manual-blackhole.txt` + route at all. `vps/portbridge-manual-blackhole-refresh.sh` still defaults its + `MANUAL_BLACKHOLE_URL` to the **old** path + (`http://10.8.0.2:19090/export/portbridge-manual-blackhole.txt`), so on any + deployment that has not overridden that variable the sidecar is fetching a + 404. Either the default needs repointing at `/api/v1/ip-block-export` or the + VPS `.env` must set `MANUAL_BLACKHOLE_URL` explicitly — worth confirming + against the live host before trusting that manual blocks are reaching + portbridge. - A new sidecar, `vps/portbridge-manual-blackhole-refresh.sh`, is a near- verbatim copy of `portbridge-blackhole-refresh.sh` pointed at that URL instead of GitHub's maltrail mirror, writing to a second local file From 14008f9b47b10d8c5360913bae5d4901d486e134 Mon Sep 17 00:00:00 2001 From: Xore Date: Sun, 27 Sep 2026 15:08:28 +0200 Subject: [PATCH 13/17] docs(llm-worker): name the deployed captured-data entry point from the manifest --- docs/llm-worker/README.md | 14 ++++++++++++++ 1 file changed, 14 insertions(+) diff --git a/docs/llm-worker/README.md b/docs/llm-worker/README.md index df0e476e..72e1ce2c 100644 --- a/docs/llm-worker/README.md +++ b/docs/llm-worker/README.md @@ -122,6 +122,20 @@ inline scripts read-only. Version 1 accepts regular text files no larger than 1 MiB, refuses symlinks and NUL-containing/binary data, and hashes content itself instead of trusting a filename. +**The deployed entry point is a third file, not this one.** +`arcane/manifests/home-production.json` points the `llm-worker` stack at +`llm-worker/docker-compose.captured-data-deploy.yml` (#1751), not at +`docker-compose.captured-data.yml`. The deploy file is a thin `include:` +wrapper listing `docker-compose.yml` then `docker-compose.captured-data.yml` +in that order, so the behaviour described above is what actually runs — but the +network/mount/volume grant lives in the included file, not the deployed one. +It exists because the captured-data authorization had been applied by hand and +was in no tracked file, so an Arcane sync silently reverted the container to +`synthetic-only` while `LLM_ALLOW_CAPTURED_DATA=true` stayed set (#1751); the +`include` list is a single two-file entry rather than two entries because +`include` does not override the way repeated `-f` does (#2225). Deleting the +file returns the deployment to synthetic-only by design. + ## Guardrails - strict pydantic schemas reject extra keys, invalid enums, malformed ATT&CK From 0446bc77dfc490b5fce70fdb70a6998a671ed29c Mon Sep 17 00:00:00 2001 From: xore Date: Sun, 27 Sep 2026 15:08:40 +0200 Subject: [PATCH 14/17] docs(ip-reporting): drop the Prometheus endpoint claim, make the test-coverage claim true - 'The reporter container' still promised a /metrics Prometheus endpoint for Grafana. metrics.go explicitly rejects that shape and writeMetricsLoop overwrites dataDir/metrics.json instead; the status banner 100 lines down already said so, so the doc contradicted itself - 'Every .go file has a matching _test.go' does not hold by filename: categorize.go is covered from event_test.go, and main.go/report.go from dryrun_test.go. Every function is still exercised, so the substance stands Every other claim verified against the tree: /data/reported.db, the reporter volumes, all thirteen .go files in the diagram, no .py, and the metrics.json/AUDIT_BLOCKLISTDE/GREYNOISE_ENABLED env contracts. --- docs/ip-reporting-plan.md | 7 +++++-- 1 file changed, 5 insertions(+), 2 deletions(-) diff --git a/docs/ip-reporting-plan.md b/docs/ip-reporting-plan.md index 443841c6..83594a28 100644 --- a/docs/ip-reporting-plan.md +++ b/docs/ip-reporting-plan.md @@ -76,7 +76,8 @@ flowchart TD The `reporter` container: - Watches the same log/event volume already mounted by the `analysis` and `ml-worker` containers - Maintains a local SQLite DB (`/data/reported.db`) to deduplicate IPs per service per 24 h -- Exposes a `/metrics` endpoint (Prometheus) so Grafana can graph reports-per-hour +- Exposes no HTTP listener at all: Phase 4 writes a `metrics.json` snapshot into the + data volume on an interval instead (see the status banner's Phase 4 note) --- @@ -262,7 +263,9 @@ flowchart TD DocsDir["docs/"] --> PlanMd["ip-reporting-plan.md
this file"] ``` -Every `.go` file has a matching `_test.go`. There is no `requirements.txt`, +Every function here is exercised by a test, though not always in a +same-named file: `categorize.go` is covered from `event_test.go` and `main.go` ++ `report.go` from `dryrun_test.go`. There is no `requirements.txt`, no `sources.py`/`apis.py`/`dedup.py`/`metrics.py`, and no Prometheus exporter — see the Phase 4 note in the status banner. From ca700c86b0d066a508c943fed7fa58f4b87355cc Mon Sep 17 00:00:00 2001 From: Xore Date: Sun, 27 Sep 2026 15:15:38 +0200 Subject: [PATCH 15/17] docs(ml-worker): correct the Tier 1 contract-check count to seven --- docs/ml-worker-evaluation.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/docs/ml-worker-evaluation.md b/docs/ml-worker-evaluation.md index 9da6a953..430f29b5 100644 --- a/docs/ml-worker-evaluation.md +++ b/docs/ml-worker-evaluation.md @@ -42,7 +42,7 @@ reported alongside accuracy rather than ignored. ### 2026-08-25 — Tier 1 harness landed; Tier 2 blocked -**Tier 1 (behaviour) is live.** `evaluate_detectors.py` runs six contract +**Tier 1 (behaviour) is live.** `evaluate_detectors.py` runs seven contract checks against a candidate over the per-sensor fixture corpus and emits a hashed JSON report. It is the first reusable acceptance bar `ml-worker` has had, and it makes "evaluate this candidate offline" answerable at all. From 7576ee357177fe1b70faf7631694100f3c75c370 Mon Sep 17 00:00:00 2001 From: xore Date: Sun, 27 Sep 2026 15:16:15 +0200 Subject: [PATCH 16/17] docs(ml-worker,writable-layer): seven contract checks, dated banner, drifted line ref - evaluate_detectors.py runs seven check_* functions (appended at lines 404-411), not six - the 2026-09-03 audit had no dated banner, so its live-host measurements (245.8 GB, 13.18 GB reclaimable, a '45+ hours up' buildkit container) read as current. Added one pointing at the same do-not-mirror principle security-fixes.md already states, and at the open issues #2915/#2904 - its install-homeserver.sh:431-500 pointer no longer lands on the builder-GC discussion; the reasoning is at 392-397 and the JSON block at 463-465 Everything else verified: all four quality.yml scripts now use 'docker rm -fv', rex86-eval is tracked but absent from the manifest and is the only such directory under arcane/home/, and the daemon.json gc values match exactly. --- docs/container-writable-layer-audit-2026-09-03.md | 13 +++++++++++-- 1 file changed, 11 insertions(+), 2 deletions(-) diff --git a/docs/container-writable-layer-audit-2026-09-03.md b/docs/container-writable-layer-audit-2026-09-03.md index 34874802..b20b9c61 100644 --- a/docs/container-writable-layer-audit-2026-09-03.md +++ b/docs/container-writable-layer-audit-2026-09-03.md @@ -1,5 +1,14 @@ # Container writable-layer and build-cache audit (#2859), 2026-09-03 +> **Dated record — 2026-09-03.** Every measurement below (245.8 GB, 13.18 GB +> reclaimable, the 45-hour leaked buildkit container, the dangling-volume +> census) was a snapshot of the homeserver on that date and is **not** a live +> status page. Per the same principle `security-fixes.md` states outright: do +> not mirror a live system's state into a markdown file. Re-run the commands +> before acting on any number. The attribution, the reasoning and the +> conclusions are the substance of this document and do not expire; the +> open items named here are tracked in their issues (#2915, #2904). + `docker system df` on the homeserver showed **245.8 GB in container writable layers**, invisible to the volume audit and the retention knob. This document is the attribution the issue asked for. @@ -78,8 +87,8 @@ failing** — GC only starts reclaiming once usage exceeds the 100 GB neither of which is close on this host. No config or timer change needed; a standing prune timer would be redundant with `builder.gc`, which already runs automatically as part of build activity per buildkit's own design -(the comment in `install-homeserver.sh:431-500` explains why a separate -timer isn't used). +(the comment in `install-homeserver.sh:381-465` — the reasoning at 392-397, +the JSON block itself at 463-465 — explains why a separate timer isn't used). One stray finding, not actioned: a leaked `buildx_buildkit_builder-` container (`docker-container` driver) has been running 45+ hours, From 18bf30e4f1956e7ea4c0d4c24b9fd3f41c990f6f Mon Sep 17 00:00:00 2001 From: Xore Date: Sun, 27 Sep 2026 15:20:44 +0200 Subject: [PATCH 17/17] docs(design-lab): correct how the read-only seam is enforced for the mounted backend --- docs/design-lab/README.md | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/docs/design-lab/README.md b/docs/design-lab/README.md index d658e325..160ab84b 100644 --- a/docs/design-lab/README.md +++ b/docs/design-lab/README.md @@ -75,8 +75,8 @@ It has since been rebuilt for `frontend-next` as playground on 19300. `frontend-next` has no nil-write-services handle — it reaches data over HTTP through two bases — so the read-only guarantee is made at that seam instead: `BACKEND_URL` goes through a gate that forwards -GET/HEAD and answers 405 to everything else, and `BACKEND_MOUNTED_URL` is left -unset so every request 503s. That is stronger than the original, which relied +GET/HEAD and answers 405 to everything else, and `BACKEND_MOUNTED_URL` is +pointed at a stub that answers 503 to every request. That is stronger than the original, which relied on remembering to pass nil. This directory is the redacted public copy and does not carry the harness