From 3fea28a5b5eb5cdce45b20cca048bcba0c1926b2 Mon Sep 17 00:00:00 2001 From: Oleksandr Kuzminskyi Date: Fri, 3 Jul 2026 17:27:27 -0700 Subject: [PATCH] Converge InfraHouse APT repo + signing key from Puppet (development) Add profile::infrahouse_repo (wired into profile::base) so already-running instances pick up a rotated GPG signing key without being reprovisioned. Today cloud-init seeds /etc/apt/keyrings/infrahouse.gpg and /etc/apt/sources.list.d/50-infrahouse.list only at first boot, so a long-lived instance never receives a new key -- when the current noble signing key expires (2026-07-20) apt-get update would fail fleet-wide with EXPKEYSIG. The class converges both artifacts every Puppet run: - Keyring: files/converge-infrahouse-keyring.sh fetches the published (possibly multi-key) bundle over TLS from release-.infrahouse.com, dearmors it, and installs atomically. A 'check' verb guards the exec so steady state is a no-op; a fetch/dearmor failure leaves the prior keyring intact. - Source list: rendered byte-identical to cloud-init's seed (codename from os.distro.codename fills both host and suite), so Puppet and cloud-init never fight. cloud-init must keep seeding at first boot (ih-secrets has to be installable before Puppet runs), so shared ownership is intentional. - Both notify a single refreshonly 'apt-get update' that fires only on change. Trust is anchored on the TLS channel (no fingerprint pinning), consistent with cloud-init 2.4.0. Validated real-time on jumphost-sincere-crawdad (development): in-sync no-op, source list byte-identical (md5 unchanged), keyring == published bundle, and the rewrite path (staled keyring -> reconverged byte-identical, apt refreshed once). Dev-first; promotes to sandbox then global. Note: the live release-noble bundle is still single-key -- the upstream rotation must publish the new key into the bundle for the fleet to actually converge onto it. Co-Authored-By: Claude Opus 4.8 (1M context) --- .../plans/infrahouse-keyring-convergence.md | 119 ++++++++++++++++++ debian/changelog | 6 + .../files/converge-infrahouse-keyring.sh | 72 +++++++++++ .../modules/profile/manifests/base.pp | 1 + .../profile/manifests/infrahouse_repo.pp | 67 ++++++++++ 5 files changed, 265 insertions(+) create mode 100644 .claude/plans/infrahouse-keyring-convergence.md create mode 100644 environments/development/modules/profile/files/converge-infrahouse-keyring.sh create mode 100644 environments/development/modules/profile/manifests/infrahouse_repo.pp diff --git a/.claude/plans/infrahouse-keyring-convergence.md b/.claude/plans/infrahouse-keyring-convergence.md new file mode 100644 index 0000000..a46e5a7 --- /dev/null +++ b/.claude/plans/infrahouse-keyring-convergence.md @@ -0,0 +1,119 @@ +# InfraHouse APT Keyring Convergence + +Manage `/etc/apt/keyrings/infrahouse.gpg` from Puppet so **already-running** instances pick up +rotated GPG signing keys. This is the client-side "long pole" of the InfraHouse APT GPG key +rotation (noble signing key expires **2026-07-20**). + +## Why + +Today cloud-init installs the keyring **only at first boot** (`bootcmd.sh`, guarded by +`if ! test -f $REPO_LIST`). `profile::repos` sets up the `apt` module but **does not** manage the +InfraHouse keyring. So a long-lived instance that isn't reprovisioned never receives a new key — +when the current signing key expires, `apt-get update` fails fleet-wide with `EXPKEYSIG`. + +Puppet runs every cycle (`apt` update frequency is already `always`), so it's the right place to +**converge** the keyring: fetch the published (possibly multi-key) bundle each run and rewrite the +keyring idempotently. Rotation then needs no per-key Puppet change — new keys just appear in the +bundle. + +## Design + +Mirror cloud-init 2.4.0's approach (it dropped fingerprint pinning; trust is anchored on **TLS** +to `release-.infrahouse.com`): + +- Fetch `https://release-${codename}.infrahouse.com/DEB-GPG-KEY-release-${codename}.infrahouse.com` + each run. +- `gpg --dearmor` → `/etc/apt/keyrings/infrahouse.gpg` (handles concatenated multi-key bundles). +- Write idempotently (only rewrite when content changes); `notify` an `apt-get update`. +- **No fingerprint pin** — stay consistent with cloud-init; trust is the TLS channel to our own + repo host. (If we ever want defense-in-depth, pin the *set* of fingerprints, but do it in both + places or neither.) +- Only manage the keyring for codenames the node actually uses (derive from `$facts['os']` / + `lsb` codename); don't assume noble. + +Likely shape: a new `profile::infrahouse_repo` class (or extend `profile::repos`) using an +`exec`/`file` pair, or `apt::keyring` if the module version supports fetching. Keep the sources +list ownership consistent with what cloud-init writes (`/etc/apt/sources.list.d/50-infrahouse.list`) +so the two don't fight. + +## Progress Tracker + +### Prerequisites (done elsewhere) +- [x] Repo module publishes a multi-key bundle — `terraform-aws-debian-repo` 4.0.0 (`gpg_public_keys`) +- [x] cloud-init consumes the bundle over TLS — `terraform-aws-cloud-init` 2.4.0 +- [x] Package-neutral re-sign tool — `ih-s3-reprepro export` (infrahouse-toolkit #249) + +### Phase 0: Implement +Implemented in `environments/development/modules/profile` (dev-first; promotes to sandbox then +global). New class `profile::infrahouse_repo` (wired into `profile::base`) + deployed script +`files/converge-infrahouse-keyring.sh`. Class converges **both** the signing key and the repo +source line — they're one concern. +- [x] Add convergence class: keyring (fetch bundle → dearmor → `/etc/apt/keyrings/infrahouse.gpg`) + **and** source list (`/etc/apt/sources.list.d/50-infrahouse.list`). +- [x] Idempotent; codename derived from facts (`os.distro.codename`, overridable param) and fills + both host (`release-`) and suite. Keyring `exec` guarded by a `check` verb; source list + is a native `file`. Both notify a single shared refreshonly `apt-get update`, so steady state is a + no-op and apt only refreshes on an actual change. +- [x] Shared ownership with cloud-init is **intentional**, not a conflict: cloud-init must seed both + at first boot (ih-secrets has to be installable before Puppet runs — secret *names* live in + user-data, values don't). Puppet renders the source line **byte-identical** to cloud-init's seed + so the two never fight; Puppet just keeps it converged afterward. (So: no follow-up to remove the + cloud-init write — the seed is load-bearing.) +- [x] Fetch failure leaves prior keyring intact (atomic write to temp, install only on success). +- [~] ~~Unit/rspec-puppet coverage~~ — skipped: repo has no existing class specs / test harness + provisioned. Gates used instead: `puppet-lint --fail-on-warnings` (pass), `shellcheck` (pass), + real-node validation on `jumphost-sincere-crawdad` (Phase 1). + +### Phase 1: Development +Test node: `jumphost-sincere-crawdad.rmdbkn.ci-cd.infrahouse.com` +(`/etc/puppetlabs/facter/facts.d/puppet.yaml` → `puppet_environment: development`, `puppet_role: jumphost`). + +**1a. Real-time apply (uncommitted code, iterate live)** — done 2026-07-04 on the test node. +- [x] Sync the working `environments/development/modules/profile` to the node and `ih-puppet apply` +- [x] `50-infrahouse.list` unchanged vs cloud-init's seed — md5 `627951c5b24fde2de0feb9e93700c0b6` + before **and** after (byte-identical → no rewrite/churn) +- [x] `apt-get update` clean against the InfraHouse repo (`Hit … noble InRelease`, no `EXPKEYSIG`) +- [x] Second run is a clean no-op (guarded `exec` skips; no spurious `apt-get update`) +- [x] Keyring == published bundle verified (`curl … | gpg --dearmor | cmp` matches) +- [x] **Rewrite path** exercised: staled the keyring (`truncate -s 0`), re-applied → + `Exec[…::converge]` ran, `Exec[…::apt_update]` triggered once, `cmp` vs backup byte-identical. + Second pass no-op. This also covers the drifted first-boot-only node (same divergence → rewrite). + Genuine *new-key* pickup still awaits the upstream bundle rotation (see Finding). + +> **Finding (2026-07-04):** the live `release-noble` bundle is still **single-key** — only the old +> signing key `A627B776…689AD619` (`[expires: 2026-07-20]`). The **new rotated key is not published +> yet**. So end-to-end multi-key convergence can't be validated here until upstream adds the new key +> to `gpg_public_keys` and re-exports. Deploying this class is **necessary but not sufficient** for +> the rotation: someone must also publish the new key into the live bundle (dual-key) — see Deadline +> note. `gpg --show-keys` "lists every key in the bundle" check deferred until then. + +**1b. Cut a release** (once the live test looks good) +- [ ] Merge to `main` → CD publishes the `.deb` to the APT repo (dev picks it up on next Puppet run) + +**1c. Zero-bootstrap check via instance-refresh** +- [ ] `instance-refresh` the dev jumphost ASG so a brand-new instance provisions from scratch +- [ ] On the fresh instance confirm cloud-init first-boot + Puppet convergence agree; keyring valid, + `apt-get update` clean — i.e. Puppet bootstraps an instance from zero, not just an existing one + +### Phase 2: Sandbox +- [ ] Promote: copy the change into `environments/sandbox/modules/profile` +- [ ] Cut a release (merge → CD) +- [ ] Watch sandbox nodes across roles: keyring converges, `apt-get update` clean, no drift + +### Phase 3: Production (global modules) +- [ ] Promote: move the change into global `modules/profile` (production has no env-local override) +- [ ] Cut a release (merge → CD); deploy fleet-wide +- [ ] Watch production; spot-check long-lived instances across services (openvpn, jumphost, etc.) +- [ ] **GATE:** fleet trusts the current bundle before any signing-key retire happens upstream + +## Deadline note + +This gates the upstream rotation: `terraform-aws-debian-repo` cannot switch signing to the new key +(dual-sign → retire old) until running instances trust it (the GATE above), and that must complete +**before 2026-07-20**. If Phases 1–3 can't be deployed and verified with margin, a **one-off fleet +keyring refresh** (SSM run-command / ansible) is the deadline fallback, with this convergence class +as the durable follow-up. + +## Cross-references +- Full rotation design/runbook: `terraform-aws-debian-repo/.claude/plans/gpg-key-rotation-design.local.md` +- cloud-init client behavior: `terraform-aws-cloud-init/files/bootcmd.sh`, issue infrahouse/terraform-aws-cloud-init#89 diff --git a/debian/changelog b/debian/changelog index 768ee74..2c8f995 100644 --- a/debian/changelog +++ b/debian/changelog @@ -1,3 +1,9 @@ +puppet-code (0.1.0-1build311) noble; urgency=medium + + * commit event. see changes history in git log + + -- root Sat, 04 Jul 2026 00:27:29 +0000 + puppet-code (0.1.0-1build310) noble; urgency=medium * commit event. see changes history in git log diff --git a/environments/development/modules/profile/files/converge-infrahouse-keyring.sh b/environments/development/modules/profile/files/converge-infrahouse-keyring.sh new file mode 100644 index 0000000..d16d476 --- /dev/null +++ b/environments/development/modules/profile/files/converge-infrahouse-keyring.sh @@ -0,0 +1,72 @@ +#!/bin/bash +# +# Managed by Puppet (profile::infrahouse_repo). Do not edit locally. +# +# Converge /etc/apt/keyrings/infrahouse.gpg from the published InfraHouse +# signing-key bundle so that long-lived instances pick up rotated GPG keys +# without being reprovisioned. Trust is anchored on the TLS channel to +# release-.infrahouse.com (no fingerprint pinning), consistent with +# cloud-init. The (possibly concatenated multi-key) armored bundle is fetched +# each run and dearmored into the keyring. +# +# Usage: +# converge-infrahouse-keyring.sh check +# exit 0 if the installed keyring already matches the published bundle, +# exit 1 otherwise (including when the bundle cannot be fetched, so the +# caller re-runs 'apply' and surfaces the underlying error). +# converge-infrahouse-keyring.sh apply +# fetch + dearmor + install the keyring atomically. A fetch/dearmor +# failure leaves the existing keyring intact. Refreshing apt is left to +# the Puppet-managed 'apt-get update' that this exec notifies. +# +set -euo pipefail + +export PATH=/usr/sbin:/usr/bin:/sbin:/bin + +KEYRING=/etc/apt/keyrings/infrahouse.gpg + +usage() { + echo "usage: $0 {check|apply} " >&2 + exit 2 +} + +[ "$#" -eq 2 ] || usage +action=$1 +codename=$2 +url="https://release-${codename}.infrahouse.com/DEB-GPG-KEY-release-${codename}.infrahouse.com" + +# Fetch the armored bundle over TLS and dearmor it into $1. Returns non-zero +# without touching $KEYRING on any failure (fetch, dearmor, or empty result). +fetch_dearmored() { + local out=$1 armored + armored=$(mktemp) + # shellcheck disable=SC2064 + trap "rm -f '${armored}'" RETURN + curl --fail --silent --show-error --location \ + --retry 3 --retry-delay 2 --max-time 30 \ + --output "${armored}" "${url}" + gpg --dearmor <"${armored}" >"${out}" + # Refuse to install an empty/garbage keyring. + test -s "${out}" +} + +candidate=$(mktemp) +trap 'rm -f "${candidate}"' EXIT + +case "${action}" in + check) + fetch_dearmored "${candidate}" || exit 1 + # 0 = already converged, 1 = differs or keyring missing. + cmp -s "${candidate}" "${KEYRING}" + ;; + apply) + fetch_dearmored "${candidate}" + if cmp -s "${candidate}" "${KEYRING}"; then + exit 0 + fi + install -D -o root -g root -m 0644 "${candidate}" "${KEYRING}" + ;; + *) + usage + ;; +esac \ No newline at end of file diff --git a/environments/development/modules/profile/manifests/base.pp b/environments/development/modules/profile/manifests/base.pp index 5790001..1f84f07 100644 --- a/environments/development/modules/profile/manifests/base.pp +++ b/environments/development/modules/profile/manifests/base.pp @@ -5,6 +5,7 @@ include 'profile::ntp' include 'profile::repos' + include 'profile::infrahouse_repo' include 'profile::packages' include 'profile::infrahouse_toolkit' include 'profile::puppet_apply' diff --git a/environments/development/modules/profile/manifests/infrahouse_repo.pp b/environments/development/modules/profile/manifests/infrahouse_repo.pp new file mode 100644 index 0000000..ee65810 --- /dev/null +++ b/environments/development/modules/profile/manifests/infrahouse_repo.pp @@ -0,0 +1,67 @@ +# @summary: Converges the InfraHouse APT repository (source list + signing key). +# +# cloud-init seeds both `/etc/apt/sources.list.d/50-infrahouse.list` and +# `/etc/apt/keyrings/infrahouse.gpg` at **first boot only** — it must, because the +# InfraHouse toolkit (ih-secrets) has to be installable before Puppet ever runs. +# This class **converges** both on every Puppet run so long-lived instances pick +# up a rotated signing key (the urgent case: the key expires) or a changed repo +# line without being reprovisioned. +# +# Ownership is intentionally shared: cloud-init writes the seed, Puppet keeps it +# in sync. The rendered source line is kept byte-identical to cloud-init's seed so +# the two never fight. Trust is anchored on the TLS channel to +# release-.infrahouse.com (no fingerprint pinning), consistent with +# cloud-init. On a fetch failure the previously installed keyring is left intact. +# +# @param codename APT codename whose repo/bundle to converge; derived from the +# node's OS facts by default. +class profile::infrahouse_repo ( + String[1] $codename = $facts['os']['distro']['codename'], +) { + $script = '/usr/local/sbin/converge-infrahouse-keyring.sh' + $keyring = '/etc/apt/keyrings/infrahouse.gpg' + $source_list = '/etc/apt/sources.list.d/50-infrahouse.list' + + stdlib::ensure_packages(['curl', 'gnupg', 'ca-certificates']) + + file { $script: + ensure => file, + owner => 'root', + group => 'root', + mode => '0755', + source => 'puppet:///modules/profile/converge-infrahouse-keyring.sh', + } + + # Converge the keyring: 'check' short-circuits when already in sync, so steady + # state is a no-op and the shared 'apt-get update' only fires on a key change. + exec { 'profile::infrahouse_repo::converge': + command => "${script} apply ${codename}", + unless => "${script} check ${codename}", + path => ['/usr/bin', '/usr/sbin', '/bin', '/sbin'], + require => [ + File[$script], + Package['curl'], + Package['gnupg'], + ], + notify => Exec['profile::infrahouse_repo::apt_update'], + } + + # Converge the repo line. Kept byte-identical to cloud-init's first-boot seed; + # $codename fills both the host (release-) and the suite. + file { $source_list: + ensure => file, + owner => 'root', + group => 'root', + mode => '0644', + content => "deb [signed-by=${keyring}] https://release-${codename}.infrahouse.com/ ${codename} main\n", + require => Exec['profile::infrahouse_repo::converge'], + notify => Exec['profile::infrahouse_repo::apt_update'], + } + + # Single refresh point for both the keyring and the source list. + exec { 'profile::infrahouse_repo::apt_update': + command => 'apt-get update', + path => ['/usr/bin', '/usr/sbin', '/bin', '/sbin'], + refreshonly => true, + } +} \ No newline at end of file