ops: Elasticsearch at 1000/1000 shards, all sensor ingest dead-lettered since 2026-09-21

[Xore]

What happened

Every sensor in honeypot-v2-* stopped indexing at about 2026-09-21 18:19 UTC (cowrie, multipot, cisco-asa, hellpot, sentrypeer, beelzebub, http, endlessh, citrix, rdp, api, dnp3, dns, elasticpot, dicompot, mailoney, galah all show the same last @timestamp). Dionaea incidents are affected the same way.

Cause (observed on the homeserver, 2026-09-24)

• _cluster/health: 991 active shards plus 9 unassigned, which is the 1000 default of cluster.max_shards_per_node on this single-node cluster. _cluster/settings has no override.
• The newest dead-letter documents all carry the same error:
  validation_exception: this action would add [1] shards, but this cluster currently has [1000]/[1000] maximum normal shards open.
  Each new daily index or data-stream backing index is refused, so Filebeat routes every event to the dead-letter index.
• dead-letter-honeypot now holds ~1.73 billion documents, 600 GB. Those documents contain the missing events, so they can be replayed and should not be deleted before the gap is backfilled.
• hp-filebeat shows a RestartCount of 1401 and logs Failed to index N events in last 10s: tried dead letter index every 10 seconds.
• Disk is not the cause: 68% used, 2.7 TB free.

Where the shards go

The biggest consumers are many tiny daily indices: about 30 zeek-proxy-v1-<log type>-* and zeek-v1-* families with 11 to 14 days each (several of them hold under 100 docs in total), arkime_sessions3-*, dashboard-backend-v1-* and dionaea-incidents-v1-*.

Fix options

1. Immediate unblock (reversible): raise the limit, e.g. set cluster.max_shards_per_node to 1500 as a persistent setting. This buys headroom, not a fix.
2. Real fix: consolidate the zeek per-log-type daily indices (one data stream, or weekly/monthly rollover), add ILM or retention for arkime_sessions3-*, dashboard-backend-v1-* and dashboard-bff-v1-*, and set replicas to 0 on the single node (the 9 unassigned shards).
3. Backfill: replay the dead-letter documents from 2026-09-21 18:19 onward through the honeypot pipeline, then trim dead-letter-honeypot.
4. Alerting: the source-health page and the live toasts should flag "all sensors silent" and a growing dead-letter rate; this went unnoticed for three days.

Found incidentally while auditing the dashboard mock-data coverage.

[Xore]

Immediate unblock applied 2026-09-24 22:15 UTC: cluster.max_shards_per_node raised to 1500 (persistent cluster setting).

Verified after the change:

• New daily indices were created: active shards 991 → 1004.
• honeypot-v2-* receives events again. In the five minutes after the change: multipot 243, cowrie 48, dionaea 27, conpot 8, sentrypeer 2.
• hp-filebeat: no Failed to index lines in the 45 s after the retry backlog drained, and the restart count stayed at 1474.

Still open, so this issue stays open:

• Backfill: replay dead-letter-honeypot (about 1.73 billion docs, 600 GB) from 2026-09-21 18:19 onward through the honeypot pipeline, then trim it.
• Real fix: consolidate the per-log-type daily zeek-* indices, and add retention for arkime_sessions3-*, dashboard-backend-v1-* and dashboard-bff-v1-*. The cluster sits at 1004 of 1500 now, and the ceiling will be reached again in weeks without it.
• Replicas to 0 on the single node: 9 shards remain unassigned, so the cluster is yellow.
• An "all sensors silent" alert.

[Xore]

Picking this up (#3284 closed as verified). Order: unassigned replicas → retention/consolidation of the shard-heavy families → dead-letter backfill plan → alerting.

[Xore]

Replicas done, cluster green

Root cause of the 12 unassigned shards: every arkime_sessions3-* index got Elasticsearch's default 1 replica. Arkime's legacy template says 0, but legacy templates never apply once a composable template (arkime-sessions3-ip-fix) matches. Fixed live (template + the 12 existing indices) and in the repo via #3344. _cluster/health: green, 0 unassigned (1078 active of the 1500 limit).

The same shadowing means Arkime's own mappings never apply (firstPacket mapped long, not date): filed as #3343.

#3284 is closed as verified: 0 Filebeat restarts in 2h18m after #3335.

Still open here: retention/consolidation of the shard-heavy families (zeek history, arkime_sessions3-*, dashboard-backend-v1-*, dashboard-bff-v1-*) and the dead-letter backfill (1.73 B docs, 600 GB; the last hour added 0 new dead letters). Pausing this issue to work through the newest issues first, per the owner.

[Xore]

Shard limit would have been hit again within days: fixed live, PR #3362

• Measured: by the date in index names, the node creates ~90 new primary shards a day (09-24: 64, 09-25: 90, 09-26: 87). 84 of 99 daily families are Zeek log types (zeek-v1-<type>-<day>). Nearly every family already has an ILM policy; only arkime_sessions3-* is unmanaged.
• Projection: live retention is 21 days (HONEYPOT_RETENTION_DAYS=21), so steady state is ~90×21 + ~89 static ≈ 1,980 shards against the 1,500 cap. Worse, 09-14 to 09-21 created almost no indices (Filebeat's crash loop, ops: filebeat crashes every 1-2 min with Go thread exhaustion (10000-thread limit) #3284), so deletions pause for about a week around 10-05 while creation continues.
• Fix (live since 16:26Z, repo in fix(elasticsearch): zeek writes one index per log type per month, not per day #3362): Zeek now writes one index per type per month, named …-yyyy.MM.01 so every existing pattern and the backend's date regex keep working, under new 42-day policies (2× retention, so each month is kept 21–42 days). Existing daily indices keep their 21-day policy and expire on schedule. That's ~84 new shards a month instead of a day.
• Side note: the 09-14 to 09-21 gap in Zeek indices is probably lost Zeek data from that period, not just late indices. Worth checking alongside the dead-letter backfill.

Still open here: the dead-letter backfill (1.73 B docs, 600 GB) and arkime_sessions3-* retention.