fix(arkime): un-shadow db.pl, and make #3346's translation actually run (#3343) (#3392)

* fix(arkime): un-shadow db.pl and make the translation actually run (#3343)

into composable ones, and left the refusal it could not get past open:
Elasticsearch refuses to create a legacy template whose patterns match an
existing composable one, and single-node-replica-default matches "*", so
db.pl could never install or refresh any of Arkime's templates again. On an
Arkime version bump that needed a template upgrade, arkime-init's db.pl would
fail under `set -e`, the done-marker would not be written, and capture and
viewer would wait on it forever.

It was worse than that. arkime-init's command is a YAML `>` folded block
scalar, and a folded scalar joins every run of adjacent non-empty lines with
a single space. The node invocation sat directly under its comment with no
blank line between, so the whole thing rendered as one `fi` line with a
trailing comment:

  fi # #3343: translate ... See arkime/composable-templates.js. /opt/arkime/bin/node /opt/arkime/composable-templates.js

composable-templates.js had therefore never once run, and #3346's fix was
inert while still reading as correct in the YAML. bash -n accepts the
result either way and nothing else exercised the rendering.

composable-templates.js gains a `shadow` subcommand, run before db.pl and
restored after: it deletes every composable template that covers an Arkime
index family, stashing each body first, so db.pl has nothing to collide
with. Coverage is decided by Elasticsearch's own simpleMatch glob semantics
rather than by name, so a renamed or second catch-all is shadowed too --
the previous approach of testing one invented probe name missed even
arkime_sessions3-2*, which overlaps arkime_sessions3-* just as hard. The
restore runs in a `finally`, because the shadowed set includes the
replicas catch-all for every template-less index family in the stack and
leaving it deleted would turn one failure into a yellow cluster.

compose.yml wraps db.pl in a bounded three-attempt retry, which covers the
one race a shadow cannot: a template created *while* db.pl runs, by
elasticsearch-setup's catch-all landing mid-run. Both db.pl verbs are safe
to re-run, and the loop exits non-zero without writing the done marker
rather than wedging the deploy permanently.

elasticsearch-setup.sh's comment claimed the catch-all "explicitly
excluded" Arkime, which was never true -- composable templates have no
exclusion syntax, so the "-arkime_..." entries were literal index names
matching nothing. The inert entries are gone and the comment now names the
mechanism that actually works. Its 60s wait for Arkime's legacy template
is kept: it is the common case and needs no shadow/restore cycle, but it
is no longer load-bearing.

Refs #3343

* fix(arkime): keep the template stash out of world-readable /tmp (#3343)

CodeQL flagged js/insecure-temp-file on the shadow file: it was written to a
fixed path directly in /tmp, so anything on the host could have pre-created
it as a symlink or read the stash. The stash holds the full body of every
Arkime template this script deletes from the cluster, so that is worth
closing.

The path stays fixed because the shadow and generate passes are separate
processes that must share it -- a fresh mkdtemp per process would hand each
one its own empty stash and the generate pass would have nothing to restore.
The fix is the permissions instead: the parent directory is created 0700 and
the file 0600, and neither is widened on rewrite. os.tmpdir() honours TMPDIR
and SHADOW_FILE still overrides the location outright.

Asserted in tests/docs/test_3343_fix.py so a later change cannot quietly drop
the modes back to the mkdir/write defaults.

* test(arkime): teach the #3343 stub the routes #3283 added to generate()

Conflict resolution follow-up for the rebase onto main. #3391 (the #3283
retention work) landed in composable-templates.js between this branch being
cut and the rebase, and it widened the contract generate() has with
Elasticsearch: it now installs _ilm/policy/arkime-sessions-30d immediately
before the template that names it, verifies index.lifecycle.name through
_simulate_index, and adopts the arkime_sessions3-* indices already on disk.

The #3343 suite drives the real script against a stub Elasticsearch, so two
of its tests failed on the rebase with an AssertionError naming an
"unexpected" PUT /_ilm/policy/arkime-sessions-30d. No assertion changed --
the stub is what was stale, and it is what had to learn the new routes.

Its _simulate_index answer changes from a canned
{"number_of_replicas": "0"} to a composition of the templates the script
really PUT, which is what Elasticsearch does and what the script's own
post-install verification depends on. That is strictly stronger than what it
replaces: the canned reply reported 0 replicas whatever the script
installed, so a generated template that dropped number_of_replicas would
still have passed. Composed, it fails. Dropping the retention line from
build() now fails 18 tests across both suites rather than passing silently.

The adoption routes answer with an empty index list, so the #3343
assertions stay about templates and the shadow; the adoption contract
itself remains asserted in tests/docs/test_3283_fix.py, which owns it.

Refs #3343, #3283

* test(arkime): drive #3283's adoption from the #3343 stub, and pin the ordering

Follow-up to 7a80c9c, which taught this stub the routes #3283 added. That
commit answered them, but answered _cat/indices with an empty list and
_settings with a constant, so the adoption branch was never actually entered
from this suite -- the routes were registered and dead. tests/docs/
test_3283_fix.py owns the adoption contract and pins it properly; this change
is about the *combined* run, which is what this suite owns.

The stub now models the one field the script reads: a name maps to whatever
index.lifecycle.name it carries, or None for unmanaged. _stubbed() seeds one
unmanaged sessions index, so the functional tests drive the real path, and
put_ilm_policies records the body so the ordering can be asserted rather than
assumed.

New test: the run that restores the catch-all is the same run that installs
arkime-sessions-30d and adopts what predates it, and the policy is installed
before the template that names it. That ordering is the part that breaks
silently -- an index template naming an ILM policy that does not exist yet
fails index creation outright, and Elasticsearch validates it at index
creation rather than here, so nothing upstream would report it.

Non-vacuous, checked by mutating the script and reverting:
  * moving ensurePolicy after the template PUT fails the new test
    ("the policy was installed after the template naming it")
  * deleting the adoption loop fails it
    ("arkime_sessions3-2026.09.01 was not adopted onto the policy: None")

Local: pytest tests/docs/ 521 passed, 1 xfailed (main: 497 passed, same
xfail); test_3343 + test_3283 together 46 passed. scripts/tests unchanged
from main: the same 2 pre-existing test_compose_drift_watch_sweep failures
that fail on a pristine main checkout on this host.

---------

Co-authored-by: CI-fix lane <ci-fix@users.noreply.github.com>

Author: Xore

arcane/home/honeypot-init/analysis/elasticsearch-setup.sh

@@ -1556,48 +1556,46 @@ curl -fsS -X PUT "$es_url/_all/_settings?expand_wildcards=all" \
 # number_of_replicas explicitly and outranks this one on priority, so this
 # only ever applies to an index nothing more specific already covers.
 #
-# EXCEPT arkime_sessions3-*/arkime_history_v1-*, explicitly excluded below.
-# Elasticsearch's own documented precedence rule: when ANY composable index
-# template (the modern _index_template API, what this whole script and
-# every "priority": N template above uses) matches an index, EVERY legacy
-# template (the old _template API) is ignored outright for that index, not
-# merged -- even a priority-1 catch-all like this one wins outright over a
-# legacy template with no priority concept at all. Arkime's own db.pl
-# still creates its real field-typing templates (arkime_sessions3_template/
-# _ecs_template, arkime_history_v1_template) via that legacy API, and this
-# catch-all's original "*" pattern silently shadowed them completely --
-# confirmed live: every arkime_sessions3-* index's source.ip/destination.ip
-# fell through to Elasticsearch's own dynamic string default (text +
-# .keyword) instead of a real `ip`-typed field, breaking session-detail
-# lookups outright ("TypeError: Cannot create property 'keyword' on
-# string" in Arkime's own viewer, since it assumes an object-typed IP
-# field it can attach a .keyword accessor to). Both excluded index
-# families already set their own number_of_replicas: 0 in their real
-# legacy templates, so excluding them here doesn't reintroduce the
-# yellow-cluster problem this template exists to prevent -- it only lets
-# their own already-correct settings apply uncontested again. Every OTHER
-# arkime_* index (dstats, files, stats, users, etc) has no legacy template
-# of its own and still needs this catch-all, so the exclusion is scoped to
-# exactly these two, not arkime_* broadly.
-# Wait for Arkime's own legacy template before adding ANY composable template that overlaps it.
-# Elasticsearch allows a composable template to overlap an existing legacy one
-# (it only warns), but REFUSES the reverse:
+# #3343: the "EXCEPT arkime_sessions3-*/arkime_history_v1-*, explicitly
+# excluded below" claim that used to stand here was simply false. Composable
+# index templates have no exclusion syntax, so the "-arkime_..." entries this
+# template carried matched nothing but literal index names -- the "*" really
+# did match every arkime_sessions3-* index. The inert entries are gone, and so
+# is any pretence that this catch-all leaves Arkime alone. It does not.
+#
+# That is not only a mapping wart, it is why Arkime could not be installed at
+# all. Elasticsearch applies NO legacy template to an index once ANY
+# composable template matches it (even a priority-1 catch-all like this one
+# wins outright over a legacy template, which has no priority concept at all),
+# and it refuses the reverse direction outright:
 #
 #   illegal_argument_exception: legacy template [arkime_sessions3_template] has
 #   index patterns [arkime_sessions3-*] matching patterns from existing
-#   composable templates [arkime-sessions3-ip-fix, ...] -- use composable
+#   composable templates [single-node-replica-default, ...] -- use composable
 #   templates (/_index_template) instead
 #
-# arkime-init and elasticsearch-setup are both honeypot-init one-shots with no
-# depends_on between them, so they race. On the 2026-09-04 rebuild this script
-# won, and `db.pl init` then failed with the above and exited 255 -- Arkime got
-# no session indices at all, while hp-arkime-capture and -viewer both looked
-# healthy. Ordering the composable templates behind it makes Arkime win deterministically,
-# without coupling the whole of this script to arkime-init succeeding (a hard
-# depends_on would let one Arkime failure block every template here).
+# `db.pl init` DELETEs and re-creates all three of Arkime's legacy templates,
+# so against this catch-all it fails and exits 255, and Arkime ends up with no
+# session indices at all while hp-arkime-capture and -viewer both look healthy.
+#
+# arkime-init now owns Arkime's mappings properly: it deletes this catch-all
+# for the duration of db.pl, puts it straight back afterwards, and translates
+# Arkime's own legacy templates into full composable equivalents at priority 11
+# -- above this priority-1 one -- so the real field typing wins. See
+# arkime/composable-templates.js. That is what replaced the
+# arkime-sessions3-ip-fix fragment which used to stand below: it decided every
+# sessions index alone (firstPacket long, no wordSplit analyzer, 1 replica) and
+# left source.ip/destination.ip as text + .keyword, breaking session-detail
+# lookups outright ("TypeError: Cannot create property 'keyword' on string" in
+# Arkime's own viewer, since it assumes an object-typed IP field it can attach
+# a .keyword accessor to).
 #
-# Bounded, and non-fatal on timeout: a deployment with arkime-init disabled
-# entirely should still get this mapping fix rather than hang.
+# Waiting for Arkime's own legacy template before adding this catch-all is
+# still worth doing -- it is the common case and needs no shadow/restore cycle
+# at all -- but it is no longer load-bearing, since arkime-init now handles
+# either order itself. It stays bounded and non-fatal: this script must not
+# depend on arkime-init succeeding, or one Arkime failure would block every
+# template here.
 arkime_legacy_wait=60
 while (( arkime_legacy_wait > 0 )); do
   if curl -fsS -o /dev/null "$es_url/_template/arkime_sessions3_template" 2>/dev/null; then
@@ -1609,38 +1607,31 @@ while (( arkime_legacy_wait > 0 )); do
 done
 if (( arkime_legacy_wait <= 0 )); then
   echo "elasticsearch-setup: Arkime legacy template did not appear within 60s --" \
-       "adding the composable templates anyway (arkime-init may be disabled; if it" \
-       "runs later it will fail its own template creation, see #2961)"
+       "adding the composable catch-all anyway (arkime-init may be disabled; if it" \
+       "runs later it shadows this template and regenerates Arkime's own, so the" \
+       "worst case is a needless shadow/restore cycle, not a failure -- #3343)"
 fi
 
 curl -fsS -X PUT "$es_url/_index_template/single-node-replica-default" \
   -H 'Content-Type: application/json' \
-  --data-binary '{"index_patterns":["*","-arkime_sessions3-*","-arkime_history_v1-*"],"priority":1,"template":{"settings":{"index.number_of_replicas":0}}}' >/dev/null
+  --data-binary '{"index_patterns":["*"],"priority":1,"template":{"settings":{"index.number_of_replicas":0}}}' >/dev/null
 
-# #3343: no Arkime template is created here any more. What the comment that
-# used to stand here called "under-documented legacy-template merge behavior"
-# was plain shadowing: Elasticsearch applies NO legacy template to an index
-# once any composable template matches it, and single-node-replica-default
-# above matches "*" (composable templates have no exclusion syntax -- the
-# "-arkime_..." entries are literal names). So neither of Arkime's legacy
-# templates ever applied, and the arkime-sessions3-ip-fix fragment that stood
-# here decided every sessions index alone (firstPacket long, no analyzer, 1
-# replica). arkime-init now translates Arkime's legacy templates into full
-# composable ones right after db.pl (arkime/composable-templates.js) and
-# deletes arkime-sessions3-ip-fix.
-#
 # #3283: arkime_sessions3-*'s retention is NOT here, with the other twelve
 # policies, and that placement is the fix rather than an omission. This
-# script's own catch-all cannot reach that family (it is excluded above, and
-# the generated composable template is what creates the indices), and an
-# index template naming an ILM policy that does not exist yet fails index
-# creation outright. This job and arkime-init are independent one-shots that
-# race -- the wait above exists because of exactly that -- and arkime-capture
-# waits only on arkime-init.done, so a policy created here could not be
-# relied on to exist before the first sessions index is created.
-# composable-templates.js therefore installs arkime-sessions-30d immediately
-# before the template that names it, and adopts the indices already on disk
-# in the same run. One definition, one owner, an ordering that cannot lose.
+# script's own catch-all cannot reach that family: since #3343 its pattern is
+# a bare "*", so it really does match arkime_sessions3-* -- what keeps it off
+# that family is that the composable template arkime-init generates for it
+# outranks this one on priority, and composable templates replace rather than
+# merge, so nothing set here would survive to govern an index. The generated
+# template is also what creates those indices. An index template naming an
+# ILM policy that does not exist yet fails index creation outright. This job
+# and arkime-init are independent one-shots that race -- the wait above exists
+# because of exactly that -- and arkime-capture waits only on arkime-init.done,
+# so a policy created here could not be relied on to exist before the first
+# sessions index is created. composable-templates.js therefore installs
+# arkime-sessions-30d immediately before the template that names it, and
+# adopts the indices already on disk in the same run. One definition, one
+# owner, an ordering that cannot lose.
 
 echo
 echo "elasticsearch-setup: GeoIP, retention policies, and event templates installed"