ops: filebeat crashes every 1-2 min with Go thread exhaustion (10000-thread limit)

[Xore]

What happens

hp-filebeat crashes every one to two minutes with a Go runtime fatal error:

runtime: program exceeds 10000-thread limit
fatal error: thread exhaustion

Docker restarts it each time. The restart count went from 1401 (2026-09-24 20:26 UTC) to 1774 (2026-09-25 05:38 UTC), about 40 an hour.

Observed (2026-09-25, homeserver)

• Between crashes the process is ordinary: 12 threads, 41 open fds, about 250 MB RSS. It runs 70–90 s, then the thread count explodes past 10,000 within the runtime's own check (checkmcount via sysmon/handoffp in the stack). That looks like a burst of goroutines blocked in syscalls, not a slow leak.
• The last log line before a crash is often an elasticsearch.deadLetterIndexForPolicy warning.
• No OOM kill (OOMKilled=false). The container has no healthcheck and no autoheal label, so these are crashes, not health restarts.
• Ingest still works between crashes, and each restart resumes from the registry: 75,326 events indexed into honeypot-v2-* in 30 min, with no Failed to index since the shard limit was raised (ops: Elasticsearch at 1000/1000 shards, all sensor ingest dead-lettered since 2026-09-21 #3283). So nothing is lost, but events arrive in bursts, and every restart re-scans all inputs.

This predates #3283's fix. It was already restarting (count 1401) while the shard limit was the visible problem.

Leads, not yet checked

• Which input's harvesters spike: /logs/zeek*/*.log, /logs/enriched/*.json, suricata eve-*.json and the others are all filestream inputs.
• Whether writes to the event_data/dead-letter log (logging.event_data) block on disk I/O during the dead-letter storm and pile up threads.
• Setting GODEBUG / runtime/debug.SetMaxThreads only hides it. Better: harvester_limit per input, or close.on_state_change.inactive, and checking the file count under /logs/zeek*.

[Xore]

Verified fixed (2026-09-26 13:38 UTC)

Fixed by #3335 (merged, deployed). It merged with Refs, so it didn't auto-close this.

• hp-filebeat: RestartCount 0, running continuously since 11:20:41 UTC (2h18m), against ~40 restarts/hour before. docker logs --since 11:20: 0 thread exhaustion lines, 0 Failed to index.
• VPS: /root/vps/zeek-log-maintenance.sh is running and wired to @reboot; /opt/stacks/apiary/logs/zeek holds 1,885 files (4.6 GB, 13,717 before), none older than 3 days.
• Ingest: 147,669 events into honeypot-v2-* in the last hour; 0 new dead-letter docs in the last hour.

The index consolidation, replicas and dead-letter backfill remain on #3283.