hp-snare has been crash-looping since 2026-09-07 — PermissionError on /opt/snare/snare.pid, and no alarm caught it

[Xore]

What is happening

hp-snare has been in a crash loop on the homeserver since roughly 2026-09-07 12:44. docker ps -a shows Restarting (1) and it never stays up long enough to serve.

snare-entrypoint: serving page-dir 'portal.meridian.example'
Traceback (most recent call last):
  File "/usr/local/bin/snare", line 159, in <module>
    snare_uuid = snare_setup(base_path)
                 ^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/bin/snare", line 54, in snare_setup
    with open(os.path.join(base_path, 'snare.pid'), 'wb') as pid_fh:
         ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
PermissionError: [Errno 13] Permission denied: '/opt/snare/snare.pid'

No issue tracks this. A title search for snare returns only closed issues (#89, #124, #128, #557), none of which describe this failure.

Observed state

/opt/snare in the container is the bind mount /opt/stacks/apiary/logs/snare (arcane/home/honeypot-tanner/compose.yml:276). On the host:

drwxr-xrwx.  3 nobody nobody     133 Sep  4 01:34 .
-rw-r--r--.  1 nobody nobody      31 Sep  4 01:34 snare.cfg
-rw-r--r--.  1 nobody nobody     887 Sep  7 11:26 snare.err
-rw-r--r--.  1 nobody nobody 6216944 Sep  7 12:44 snare.log
-rw-r--r--.  1 nobody nobody       1 Sep  6 13:52 snare.pid
-rw-r--r--.  1 nobody nobody      36 Sep  4 01:34 snare.uuid

The directory is 0757 (other has write), so creating a new file there works. snare.pid already exists at 0644 owned by 65534:65534, so truncating that existing file requires either being uid 65534 or holding CAP_DAC_OVERRIDE.

The container has neither:

$ docker inspect hp-snare --format 'capdrop={{.HostConfig.CapDrop}} capadd={{.HostConfig.CapAdd}} user={{.Config.User}}'
capdrop=[ALL] capadd=[CAP_SETGID CAP_SETUID] user=

Config.User is empty and the image sets no USER, so the process is uid 0 at the point snare_setup() runs — SNARE drops privileges later, which is exactly why SETUID/SETGID are granted. Root without CAP_DAC_OVERRIDE cannot write another user's 0644 file. That is a consistent explanation for EACCES here, though it has not been proven by a direct test yet (see below).

Note that the compose comment immediately above the cap list records the opposite conclusion:

# SETUID+SETGID is the measured minimum; CHOWN and DAC_OVERRIDE were
# tested on top and changed nothing, so they are not granted.

That measurement was presumably taken on a first boot, when snare.pid did not yet exist and the other-writable directory let root create it. It does not hold once the file exists owned by another uid, which is the steady state after honeypot-init's chown -R 65534:65534 /logs/snare.

SELinux is Enforcing but ausearch -m avc -ts recent shows no snare denials, and the directory carries system_u:object_r:var_t:s0 identically to logs/tanner, whose containers run fine — so SELinux is not currently indicated as the cause. read_only: true is set on the service, but /opt/snare is a bind mount and is unaffected by it.

Blast radius

hp-snare is the HTTP front half of the Tanner stack — it serves the cloned persona pages and forwards every request to hp-tanner for emulation. While it loops, that sensor path captures nothing. The five sibling Tanner containers (hp-tanner, -web, -api, -redis, -docker, -phpox) are all Up 16 hours and healthy, so nothing else in the stack signals a problem — this is silent to every existing alarm.

Suggested fix direction

Two candidates, smallest first:

1. Have the entrypoint remove the stale pid file before exec'ing snare (rm -f /opt/snare/snare.pid) — creating it fresh works under the other-writable directory. Cheapest, no cap change.
2. Grant CAP_DAC_OVERRIDE and correct the compose comment that says it was tested and changed nothing.

Option 1 keeps the hardened cap set intact and is preferable if it holds. Either way the fix should be verified against a container that has been through at least one full restart cycle, not just a clean first boot — that is the exact gap that produced the misleading measurement in the comment.

Acceptance

• hp-snare stays Up across a deliberate docker restart and a second one after it has written its pid file
• Root cause confirmed by a direct test rather than inferred from file modes (e.g. reproduce the EACCES, then show the chosen fix clears it)
• The SETUID+SETGID is the measured minimum comment in arcane/home/honeypot-tanner/compose.yml reflects whatever the retest actually shows
• One request to the snare port returns a persona page

Found during the round-10 issue audit.

[Xore]

Coding started 2026-09-10 18:35 CEST (round-10 stage 1, lane A). Worktree isolation used; grit bypassed — compose YAML does not index.

Update (Europe/Berlin): root cause reproduced by direct capsh test — uid 0 without CAP_DAC_OVERRIDE cannot truncate-open an existing 0644 file owned by another uid; the source of the nobody ownership is honeypot-init's log-init chown (honeypot-init/compose.yml:149), which re-arms the trap on every init pass. Fixed by clearing snare's state files in the container entrypoint before exec.

Live verification: status=running restarts=0 started=2026-09-10T17:09:26Z; two deliberate restarts clean; the decisive case — files re-chowned to 65534:65534 then restarted — came up running restarts=0 with all three files recreated and HTTP 200 (<title>Meridian Customer Portal</title>) at 19:21:36 CEST. Fresh access line in snare.log same second. Coder handover + review evidence in the PR.

[Xore]

Closed by #3164 (squash 3f24a4b). Live verified 2026-09-10 19:21-19:36 CEST: restarts=0, re-armed-trap restart clean, HTTP 200. [Non-blocking → followups: N1 log-history wipe (no ES loss — no JSON sink), N2 STACK-REBUILD.md/install-homeserver chown remnants, N3 resource-limit drift, N4 plain image tags, N7 comment clause, N8 pro bative restarts, N9 live comment text, N10 handover filename, N11 commit — all tracked or trivially stale. Merged is not deployed: Arcane sync for honeypot-tanner pending, see #3161-adjacent.]