merge: docs reconciliation round 3, policy and host slice (#3399)

SECURITY.md

@@ -13,3 +13,12 @@ live malware, private keys, production `.env` files, packet captures containing
 private traffic, or unredacted logs to a public issue.
 
 Supported security fixes target the current `main` branch.
+
+`scripts/check-public-leaks.py` enforces the "leak a real secret" half of this
+policy on every change, and fails CI on private keys, GitHub/AWS/Slack tokens,
+literal credential assignments, credentials embedded in URLs, deployment
+`.env` files, private-key and packet-capture binaries, and the
+deployment-specific addresses and default password this repository must never
+name. Exactly one tracked `.env` is exempt, and it is the decoy honeyfs file
+under `arcane/home/honeypot-cowrie/` that exists for attackers to find — not a
+credential, and not something to "fix" by deleting.

docs/HOST-TUNING.md

@@ -8,6 +8,39 @@ sudo ./scripts/tune-rocky10.sh --dry-run   # show what would change
 sudo ./scripts/tune-rocky10.sh             # apply
 ```
 
+> **Status (2026-09-27): the script is not fully applied on the live homeserver.**
+> The host was re-provisioned onto Rocky Linux 10.2 on 2026-09-03, and the
+> install it now has differs from the one this script was written against. Read
+> the checklist in "Verifying afterwards" as *the intended end state*, not as a
+> description of the box — re-measured read-only over `ssh homeserver` on
+> 2026-09-27, it currently fails two of its six lines (the two `zramctl` /
+> `swapon --show` lines; the other four commands pass). The bullets below are
+> keyed to the **tuning** numbers in the table, not to those six command lines,
+> so they list three tunings rather than two: a tuning can be inapplicable as a
+> durable change while the one command that observes it still reads correct
+> today. That is exactly the case for #2 and #4.
+>
+> - **#1 (zram) is not in effect.** `zram-generator` is not installed, no zram
+>   module is loaded, `zramctl` prints nothing, and `swapon --show` lists only
+>   the 32G LVM swap. An `/etc/systemd/zram-generator.conf` *does* exist, but it
+>   is not the file this script writes: it has no `# Managed by` header and
+>   sizes the device with `min(ram / 2, 32768)` rather than the computed MB
+>   value the script emits, so it was hand-written — and nothing consumes it
+>   while the package is missing.
+> - **#2 (scheduler rule) is not in effect.**
+>   `/etc/udev/rules.d/60-apiary-ioscheduler.rules` does not exist. The
+>   schedulers that *are* set came from elsewhere: `nvme0n1` is on `none` and
+>   the 8.7T rotational `sdb` is on `mq-deadline`, but the non-rotational `sda`
+>   is also on `none` where this script's rule would put it on `kyber`.
+> - **#4 (noatime) is half done.** `/var` carries `noatime` in `/etc/fstab`;
+>   `/home`, which is a separate mount on this install and did not exist in
+>   `/etc/fstab` before the re-provision, is still `defaults`.
+> - **#5 (CPU governor) is in effect** — `tuned-adm active` reports
+>   `throughput-performance` and cpu0's governor is `performance`.
+>
+> The 20 GB figure in the sizing note below is still right: the live
+> `NVIDIA RTX 4000 Ada Generation` reports 20475 MiB.
+
 These are the Rocky/RHEL-family equivalents of the tunings, not a
 transliteration of the Debian recipe. The differences are the point: the
 Debian instructions do not work here, and two of them do not survive a reboot

docs/KEYCLOAK-CUTOVER.md

@@ -2,6 +2,39 @@
 
 Status: accepted Phase 0 decision record for #976 and epic #986.
 
+> **Status, verified against the repository 2026-09-27: the cutover is
+> implemented, not pending.** The line above is the decision that was accepted
+> at Phase 0; the hard cutover it specifies has since been carried out, so read
+> this file as the contract that governs the live identity tier rather than as a
+> proposal. What the repository shows today:
+>
+> - `honeypot-keycloak` is a manifest entry
+>   (`arcane/manifests/home-production.json`) and a deployed stack
+>   (`arcane/home/honeypot-keycloak/compose.yml` — Keycloak plus its
+>   PostgreSQL), which is the "Fixed architecture" section's `honeypot-keycloak`
+>   stack.
+> - The six "Isolated gateway" rows in the matrix below correspond one-to-one to
+>   six `oauth2-proxy` services in `vps/docker-compose.yml`: `oidc-kibana`,
+>   `oidc-tanner`, `oidc-evebox`, `oidc-arkime`, `oidc-revdeck` and
+>   `oidc-traefik` (one `quay.io/oauth2-proxy/oauth2-proxy` image definition,
+>   six services sharing its `oidc-gateway-environment` anchor). `arcane.*` and
+>   the `dashboard.*`/`honeypot.*` row correctly have no gateway.
+> - Every identifier on the **Hard-cutover removal list** below is gone from
+>   live configuration. `auth-portal`, `strip-auth-identity`, `xore_sso`,
+>   `AUTH_INTROSPECTION_URL` and `AUTH_TARGET_HOST` now appear only in
+>   documentation (this file, `docs/TESTING.md` and the dated
+>   `docs/research/518-smoke-test-research.md` record), never in `vps/` or
+>   `arcane/`. No `forward-auth`/`forwardAuth` middleware reference remains
+>   under `vps/` or `arcane/`.
+> - The `Xore/auth-backend` line below is accurate as written and should not be
+>   read as a live dependency: the *runtime* is retired and only the
+>   presentation-only theme is still sourced from that repository, which is
+>   what `docs/SENSORS.md` and `docs/KEYCLOAK-OPERATIONS.md` already say.
+>
+> The contract text itself is unchanged and still accurate; this note records
+> only that it has shipped. Day-to-day administration lives in
+> [`KEYCLOAK-OPERATIONS.md`](KEYCLOAK-OPERATIONS.md).
+
 Implementation and operations are documented in
 [`KEYCLOAK-OPERATIONS.md`](KEYCLOAK-OPERATIONS.md). `Xore/auth-backend` owns
 exactly the presentation-only `themes/apiary` Keycloak theme, checked out
@@ -37,17 +70,30 @@ pinned Keycloak runtime does not provide a recovery-code required-action factory
 | Route | Consumer | Integration | Keycloak client | Required role | Trust boundary and special traffic |
 |---|---|---|---|---|---|
 | `dashboard.*`, `honeypot.*` | APIARY dashboard, APIs, SSE, exports, embedded settings | Native authorization-code OIDC with PKCE | `apiary-dashboard` | `user`; mutations require `admin` | Dashboard validates tokens and owns its session. No proxy identity headers. PDF routes use the same dashboard session. |
-| `kibana.*` | Kibana | Isolated gateway | `kibana` | `user` | Gateway is the only network peer allowed to reach Kibana; preserve WebSockets and base paths. |
-| `evebox.*` | EveBox | Isolated gateway | `evebox` | `user` | Gateway is the only upstream path; preserve API, stream, and download behavior. |
-| `arkime.*` | Arkime | Isolated gateway | `arkime` | `user` | Arkime trusts the gateway-injected `X-Forwarded-User` identity (`authMode=header+digest`, #979) -- safe only because the upstream is unreachable except through the gateway (verified) and oauth2-proxy strips any client-supplied copy of that header before injecting its own (`OAUTH2_PROXY_SKIP_AUTH_STRIP_HEADERS`, pinned). The pre-existing local "admin" digest account remains as a fallback. |
-| `tanner.*` | TANNER UI | Isolated gateway | `tanner` | `user` | Gateway-only upstream network; preserve API and static assets. |
-| `rev.*` | RevDeck/Ghidra UI | Isolated gateway | `revdeck` | `user` | Gateway-only upstream network; preserve long responses, downloads, and streams. |
+| `kibana.*` | Kibana | Isolated gateway | `kibana` | `access` | Gateway is the only network peer allowed to reach Kibana; preserve WebSockets and base paths. |
+| `evebox.*` | EveBox | Isolated gateway | `evebox` | `access` | Gateway is the only upstream path; preserve API, stream, and download behavior. |
+| `arkime.*` | Arkime | Isolated gateway | `arkime` | `access` | Arkime trusts the gateway-injected `X-Forwarded-User` identity (`authMode=header+digest`, #979) -- safe only because the upstream is unreachable except through the gateway (verified) and oauth2-proxy strips any client-supplied copy of that header before injecting its own (`OAUTH2_PROXY_SKIP_AUTH_STRIP_HEADERS`, pinned). The pre-existing local "admin" digest account remains as a fallback. |
+| `tanner.*` | TANNER UI | Isolated gateway | `tanner` | `access` | Gateway-only upstream network; preserve API and static assets. |
+| `rev.*` | RevDeck/Ghidra UI | Isolated gateway | `revdeck` | `access` | Gateway-only upstream network; preserve long responses, downloads, and streams. |
 | `traefik.*` | Traefik read-only dashboard | Isolated gateway | `traefik-dashboard` | `admin` | Gateway fronts `api@internal`; callback is excluded from recursive auth. |
 | `arcane.*` | Arcane administrator UI (#1185, Dockge's replacement -- Dockge decommissioned) | Native authorization-code OIDC | `arcane` | `admin` | No gateway: Arcane authenticates directly against Keycloak. Root-equivalent risk (`/var/run/docker.sock` mounted read-write) -- `admin` role is granted only to the `administrators` group. |
 | `auth.example.invalid` | OIDC login, discovery, JWKS, account console | Direct Keycloak | n/a | public protocol endpoints; authenticated account actions | Rate-limited edge route to the Keycloak WireGuard bridge. |
 | `auth.example.invalid/admin` | Keycloak administration | Direct Keycloak | n/a | Keycloak administrator + MFA | Same host as the issuer (#1028); a `PathPrefix(/admin)` router gives the SPA's bootstrap burst a larger rate limit. Keycloak owns authentication; HTTP Basic would conflict with the SPA's Bearer API calls. |
 | decoy/static/API/status/file/blog hosts currently lacking `forward-auth` | Public honeypot or explicitly application-owned auth | Public / unchanged | none | none | Never attach operator SSO merely because the hostname exists. Public collection must remain independent of IdP availability. |
 
+Role names differ per row on purpose. The realm's low-privilege client role is
+`access`, and every gateway enforces exactly `<client>:access` through
+`OAUTH2_PROXY_ALLOWED_ROLES` in `vps/docker-compose.yml`;
+`traefik-dashboard` and `arcane` deliberately use `admin` instead
+(#1014/#1185, root-equivalent). The dashboard row's `user` is
+*not* a Keycloak role name: `resource_access.apiary-dashboard.roles` carries
+`access` and `admin`, and the dashboard collapses them to its own
+`user`/`admin` session role. Which human carries which role is a realm
+provisioning decision, not a repository fact: the committed
+`keycloak/realm/apiary-realm.json` defines the `users` and `administrators`
+groups with empty `roleMappings`, so treat the group membership the matrix
+implies as an operator-side grant to be verified in the live realm.
+
 The deployment validator must fail when a protected router has neither native
 OIDC ownership nor its named gateway. A redirect alone is not evidence: each
 row must retain an authorized real-page/API check and an unauthorized denial

docs/ROCKY-10-MIGRATION.md

@@ -1,5 +1,33 @@
 # Rocky Linux 10 support in `install-homeserver.sh`
 
+> **Status (2026-09-27): the move is done — the homeserver now runs Rocky Linux
+> 10.2.** The RHEL path below is no longer a smoke-test target or a rehearsal
+> for a future migration: the host was re-provisioned onto Rocky 10.2
+> (2026-09-03, Anaconda) with a different disk layout, so the `rhel` branches in
+> `install-homeserver.sh` are the production path and `debian` is the fallback
+> that only a rebuild would exercise. Re-measured read-only over
+> `ssh homeserver` on 2026-09-27: `/etc/os-release` reports `ID=rocky` and
+> `VERSION="10.2 (Red Quartz)"`, and both conditions in
+> "Two things Rocky does that Ubuntu did not" are live — SELinux is `Enforcing`
+> and firewalld is `active`.
+>
+> The two things that stay open are unchanged and are the ones to read first:
+> the base OS is still installed by hand (there is still no kickstart artifact
+> in the tree), and the `:z`/`:Z` label gap below is still real — none of the
+> 35 compose files tracked under `arcane/home/` carries an SELinux relabel (34
+> stack-level `compose.yml` files, plus the decoy honeyfs compose nested inside
+> `arcane/home/honeypot-cowrie/cowrie/`, which is an attacker-facing artifact
+> rather than a deployed stack).
+>
+> Everything else in this document was re-checked against
+> `scripts/install-homeserver.sh` and `scripts/lib/install-common.sh` on
+> 2026-09-27 and still matches: the `pkg_update`/`pkg_install` shims, the
+> once-at-source-time `$DISTRO_FAMILY` resolution, the full package-name
+> table, the verbatim Docker `centos` repofile, the `cuda-rhel10` repo, the
+> `container_use_devices` boolean, and the non-fatal
+> `step_preflight_rhel_platform`. For the current disk layout see
+> [`HOMESERVER-DISK-LAYOUT.md`](HOMESERVER-DISK-LAYOUT.md).
+
 The homeserver is moving from Ubuntu to Rocky Linux 10. `scripts/install-homeserver.sh`
 now runs on both, so the reinstall smoke test in #1609 has a working installer.
 

docs/design-lab/design-notes.md

@@ -1,4 +1,19 @@
 # APIARY dashboard design review — dashboard.example — 2026-08-17
+
+> **Public, redacted copy — status as of 2026-09-27.** The address redaction in
+> this file is deliberate and correct as it stands: the only address literals
+> here are `127.0.0.1` (loopback) and `203.0.113.1` (RFC 5737 TEST-NET-3), and
+> the only hostname is a reserved `.example` domain. Do not substitute real
+> values for them, and do not restore anything redacted out of the source copy.
+>
+> The findings below are a **snapshot of a review session on 2026-08-17, not a
+> description of the current code**. Several cite the Go dashboard's static
+> assets and route table by name (`hp-app.js`, `hp-dynamic-nav.js`,
+> `routes.go`); that dashboard was deleted in #1628 on 2026-08-22, five days
+> after this review, and none of those files exist in the repository any more.
+> The current route authority is the Rust `axum` service at
+> `arcane/home/honeypot-dashboard/backend-service/src/main.rs`, so re-locate a
+> finding's code there before acting on it.
 ## Findings (running)
 - Overview (light): loads fast, authenticated. Heatmap "Activity — last 24h" dominates; lower rows (multipot, conpot-kamstrup, endlessh) appear near-empty/pale — visual weight wasted?
 - Theme toggle: monitor icon top-right (left of LIVE). Dark theme renders correctly on Overview.

docs/knowledge-store-design.md

@@ -4,6 +4,25 @@ Decision record for #2289, gating #2290–#2292. No code changes ship with
 this document — it is the design pass #1634 asked for before any ingest
 worker exists.
 
+> **Status, re-measured 2026-09-27: authored and tracked, not deployed.** The
+> ingest worker does exist in git — `vault-worker/` carries a `worker.py`, a
+> `sanitize.py`, a `Dockerfile` and two compose files, the shipped output of
+> #2289/#2290 — but nothing deploys it. It is absent from
+> `arcane/manifests/home-production.json`; no vault-worker container has ever
+> run on the homeserver, and there is no `vault-worker` stack directory among
+> the deployed ones; Elasticsearch carries no `knowledge-vault*` index, so the
+> `knowledge-vault-state-v1` checkpoint cited for this worker in
+> [PIPELINES.md](PIPELINES.md) does not exist either (that row is planned, the
+> same way); and the live APIARY worker runs
+> `WORKER_LOOPS=alert-notifier,attacker-identity,agent-intrusion,correlator,dashboard-rollups,threat-intel,zeek-proxy-attribution`
+> — there is no vault loop in it.
+>
+> So everything below is the design and implementation record of a **planned**
+> subsystem, kept because #2289/#2290 shipped real code against it. Read it as
+> intent that has not been switched on: none of the paths, indices or
+> checkpoints it names exist on a live host, and no deployment decision is
+> recorded anywhere in the repository.
+
 ## 1. Storage: plain markdown directory, carried by the existing off-host
    backup path, not git/Syncthing
 
@@ -39,9 +58,10 @@ stack, not a reuse of one, even though it is a trivial one to stand up
 (`git init` in a directory, a commit per note-write batch). The nearest real
 precedent for "git as a sync/deploy substrate" in this repo is Arcane's own
 GitOps machinery (`docs/ARCANE-GIT-SYNC.md`), which already runs a
-git-pull-and-apply loop against `main` with `auto_sync = 0` set deliberately
-on rows that must not auto-follow (`docs/ARCANE-GIT-SYNC.md:321` and
-`docs/ARCANE-GIT-SYNC.md:374`) — i.e.
+git-pull-and-apply loop against `main` and where every live row currently
+carries `auto_sync = 0`, with every deploy still a manual
+sync → build → redeploy (`docs/ARCANE-GIT-SYNC.md:425` and
+`docs/ARCANE-GIT-SYNC.md:478`) — i.e.
 this codebase's existing git-sync tooling defaults to *manual* triggers for
 anything sensitive, which is the posture this decision adopts too (see §4).
 
@@ -172,7 +192,7 @@ A persistent, curated, cross-referenced copy of (bounded, redacted) attacker
 material is qualitatively different from the raw per-event ES documents it's
 derived from: it's smaller, denser, and easier for a human or a script to
 sweep in one pass. Reading `analysis/backup-honeypot.sh`, its archive step
-(`backup-honeypot.sh:87`) already walks `./analysis ./dashboard ./personas
+(`backup-honeypot.sh:119`) already walks `./analysis ./dashboard ./personas
 ./state` by directory-existence check, unconditionally including anything
 found there. If the vault directory (§1: `state/knowledge-vault/`) is placed
 under `$stack_dir/state/`, it is **already** inside this glob and would start
@@ -189,16 +209,17 @@ above already bounds and strips what can land in a note, the vault's content
 is closer in sensitivity to the config material `backup-honeypot.sh` already
 carries than to the bulk payload/PCAP data it explicitly excludes — so
 extending that script's existing scope to include it is the correct call,
-not an oversight to patch around later. This is a decision to record
-verbatim in `analysis/backup-honeypot.sh`'s own comment block when #2290
-lands the directory, so a future reader sees it was deliberate rather than
-inferring it from a directory glob matching by accident.
+not an oversight to patch around later. That decision is already recorded
+verbatim in `analysis/backup-honeypot.sh`'s own comment block
+(`backup-honeypot.sh:107-114`, which names #2289, #2290 and this document), so
+a future reader sees it was deliberate rather than inferring it from a
+directory glob matching by accident.
 
 ### Worker authorization gate
 
 The vault-ingest worker (#2290) must gate non-dry-run writes the same way
-`llm-worker` gates captured-data mode. Reading `llm-worker/worker.py:200-202`
-and `llm-worker/worker.py:254-264`:
+`llm-worker` gates captured-data mode. Reading `llm-worker/worker.py:246-248`
+and `llm-worker/worker.py:314-318`:
 non-dry-run requires `LLM_ENABLED=true` **and** `LLM_ALLOW_CAPTURED_DATA=true`
 together, with the error message naming both. The vault worker adopts the
 same two-flag shape (its own env var names, e.g. `VAULT_ENABLED` /