Enable unattended-upgrades fleet-wide; keep Elasticsearch undisturbed

[akuzminsky]

Problem

Two conflicting goals:

• Want: automatic security updates on every host.
• Don't want: unattended-upgrades restarting Elasticsearch and turning the cluster yellow/red.

Current state met neither: the base AMI ships unattended-upgrades masked (so e.g. bookstack got no security fixes), and Elasticsearch nodes additionally forced the service stopped.

Approach

Separate "apply updates" from "restart services".

Fleet-wide (profile::unattended_upgrades, included from profile::base):

• Unmask + enable unattended-upgrades.service and the apt-daily / apt-daily-upgrade timers. The timers' trigger .service units must be unmasked too, or the timers refuse to start (unit to trigger not loaded).
• Manage /etc/apt/apt.conf.d/20auto-upgrades and a 52- override; Automatic-Reboot "false".
• Package blacklist is Hiera-driven (profile::unattended_upgrades::package_blacklist).

Elasticsearch nodes (profile::elastic::service):

• Drop the blunt ensure => stopped.
• Blacklist the elasticsearch package from automatic upgrades (Hiera) — ES version changes flow through AMI rebuild + instance refresh, guarded by the existing decommission-node cron.
• Set needrestart to list-only ($nrconf{restart} = 'l') so library security fixes (openssl/glibc) land on disk without bouncing the ES process. The restart is deferred to the controlled rolling replacement.

Also normalizes some dev profile file modes (755 → 644) to match global.

Verification

Applied on a production node:

• unmask-* execs run once, then idempotent.
• apt-daily.timer / apt-daily-upgrade.timer → running.
• unattended-upgrades.service → active (running) (the expected --wait-for-signal shutdown applier).

🤖 Generated with Claude Code

[akuzminsky]

Live verification (ih-puppet apply on production nodes)

Hostnames, internal domain, ops email, and username redacted.

1. Non-Elasticsearch node — timers fix

First apply failed because apt-daily.timer refused to start while its trigger
apt-daily.service was still masked:

Error: Systemd start for apt-daily.timer failed!
apt-daily.timer: Refusing to start, unit apt-daily.service to trigger not loaded.

After also unmasking the trigger .service units, a clean apply:

Notice: /Stage[main]/Profile::Unattended_upgrades/Exec[unmask-apt-daily.service]/returns: executed successfully
Notice: /Stage[main]/Profile::Unattended_upgrades/Exec[unmask-apt-daily-upgrade.service]/returns: executed successfully
Notice: /Stage[main]/Profile::Unattended_upgrades/Service[apt-daily.timer]/ensure: ensure changed 'stopped' to 'running'
Notice: /Stage[main]/Profile::Unattended_upgrades/Service[apt-daily-upgrade.timer]/ensure: ensure changed 'stopped' to 'running'
Notice: Applied catalog in 10.83 seconds

systemctl status unattended-upgrades.service → active (running) (the expected
--wait-for-signal shutdown applier).

2. Elasticsearch node — blacklist + needrestart baked into the profile

Notice: Compiled catalog for <es-node> in environment production in 0.75 seconds
Notice: /Stage[main]/Profile::Unattended_upgrades/Service[unattended-upgrades.service]/ensure: ensure changed 'stopped' to 'running'
Notice: /Stage[main]/Profile::Unattended_upgrades/File[/etc/apt/apt.conf.d/20auto-upgrades]/content: content changed ...
Notice: /Stage[main]/Profile::Unattended_upgrades/File[/etc/apt/apt.conf.d/52unattended-upgrades-infrahouse]/ensure: defined content ...
Notice: /Stage[main]/Profile::Elastic::Service/File[/etc/apt/apt.conf.d/53-elasticsearch-blacklist]/ensure: defined content ...
Notice: /Stage[main]/Profile::Elastic::Service/File[/etc/needrestart/conf.d/90-elastic-no-auto-restart.conf]/ensure: defined content ...
Notice: Applied catalog in 76.83 seconds

Both drop-ins land from profile::elastic::service — no Hiera involved.

3. Idempotency

Immediate second apply on the same node makes no profile changes (only the
unconditional apt_update runs):

Notice: Compiled catalog for <es-node> in environment production in 1.52 seconds
Notice: /Stage[init]/Apt::Update/Exec[apt_update]/returns: executed successfully
Notice: Applied catalog in 9.55 seconds
The run succeeded, and some resources were changed.

Note (unrelated to this PR)

On the ES node the apply also rewrote /etc/elasticsearch/elasticsearch.yml
(pre-existing config drift) which triggered an Elasticsearch refresh. That is
not caused by this change — unattended-upgrades did not restart ES.