Skip to content
Closed
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
1 change: 1 addition & 0 deletions docs/src/SUMMARY.md
Original file line number Diff line number Diff line change
Expand Up @@ -89,6 +89,7 @@

- [syncthing](roles/syncthing.md)
- [systemd](roles/systemd.md)
- [systemd_hive_services](roles/systemd_hive_services.md)
- [proxy (Caddy)](roles/proxy.md)
- [tailscale_cert](roles/tailscale_cert.md)
- [bst_dashboard](roles/bst_dashboard.md)
Expand Down
1 change: 1 addition & 0 deletions docs/src/architecture.md
Original file line number Diff line number Diff line change
Expand Up @@ -83,6 +83,7 @@ Every role in `site.yml` carries one or more tags. Tags fall into two kinds: **p
| Desktop apps | `pipewire_audio` | `desktop`, `audio`, `pipewire` |
| Services | `syncthing` | `services`, `syncthing` |
| Services | `systemd` | `services`, `systemd` |
| Services | `systemd_hive_services` | `services`, `systemd`, `systemd_hive_services` |
| Services | `bst_dashboard` | `services`, `bst_dashboard` |
| Services | `kirocrew` | `services`, `kirocrew` |
| Services | `proxy` | `services`, `proxy`, `caddy` |
Expand Down
2 changes: 1 addition & 1 deletion docs/src/index.md
Original file line number Diff line number Diff line change
Expand Up @@ -68,7 +68,7 @@ The `llm` group is dormant (commented out in `inventory.yml`).
| 1 — System + packages + dotfiles | sudo, sshd, apk_packages, homebrew, termux_packages, bitwarden, shell_fonts, shell_dotfiles, omp, hive_ops, git, neovim | `system`, `packages`, `dotfiles` |
| 2 — Secrets + auth | shell_atuin, shell_ai, ssh_keys, ssh_mesh, github, tailscale, kube, forgejo_registry | `secrets` |
| 3 — Desktop apps | flatpak, bluefin_common, gnome, zen_browser (+ browser_fxa), pipewire_audio | `desktop` |
| 4 — Services | syncthing, systemd, bst_dashboard, kirocrew, proxy, tailscale_cert, server_hardening | `services` |
| 4 — Services | syncthing, systemd, systemd_hive_services, bst_dashboard, kirocrew, proxy, tailscale_cert, server_hardening | `services` |

Only the `secrets`-tagged work needs Bitwarden. With the vault locked (or
`--skip-tags secrets`), phases 1, 3 and 4 run normally and BW-touching tasks
Expand Down
9 changes: 3 additions & 6 deletions docs/src/roles/systemd.md
Original file line number Diff line number Diff line change
Expand Up @@ -27,12 +27,9 @@ If podman is installed:
- Configures the podman socket for group access (mode 0660)
- Enables and starts `podman.socket`

### system-mitigator (himachal only)

A Python daemon that monitors for Lenovo EC firmware bugs:
- Detects false `BD_PROCHOT` signals that lock CPU cores to 400 MHz
- Watches for `systemd-journald` watchdog loops
- Sends desktop notifications when issues are detected
The host-specific fleet services run in `systemd_hive_services`, keeping
machine-specific monitoring and phone orchestration separate from the generic
systemd user services and conditional Podman socket setup.

## Notes

Expand Down
33 changes: 33 additions & 0 deletions docs/src/roles/systemd_hive_services.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,33 @@
# systemd_hive_services

**Tags:** `services`, `systemd`, `systemd_hive_services`<br>
**Secrets needed:** No<br>
**Runs on:** All machines; service tasks target `himachal`

Owns systemd user units that orchestrate fleet-specific work rather than the
common systemd services installed on every host.

## Services Deployed

### system-mitigator (`himachal` only)

Deploys and enables the Lenovo EC and journald health monitor. The daemon:

- Detects CPU cores stuck at their minimum frequency
- Watches for `systemd-journald` restart loops
- Sends desktop notifications when either condition needs attention

### phone-apply timer (`himachal` only)

Deploys a two-hourly timer that pulls the dotfiles checkout and applies the
no-secrets playbook to `termux` and `mumbai`. Those hosts are often asleep or
off the tailnet, so each remote apply is allowed to fail without failing the
local timer service.

## Relationship to `systemd`

The `systemd` role owns generic user services, the conditional Podman socket,
and automatic dotfiles updates. Both roles use the `systemd` tag and the
`skip_systemd` flag, so existing tagged and skipped applies keep the same
selection semantics while the fleet-specific units have an independent role
boundary.
72 changes: 0 additions & 72 deletions roles/systemd/tasks/main.yml
Original file line number Diff line number Diff line change
Expand Up @@ -132,75 +132,3 @@
scope: user
enabled: true
state: started

- name: Deploy system-mitigator script (Lenovo 400MHz/Journald health)
ansible.builtin.copy:
src: system-mitigator.py
dest: "{{ ansible_facts['env']['HOME'] }}/.local/bin/system-mitigator.py"
mode: "0755"
when: inventory_hostname == 'himachal'

- name: Deploy system-mitigator service template
ansible.builtin.template:
src: system-mitigator.service.j2
dest: "{{ ansible_facts['env']['HOME'] }}/.config/systemd/user/system-mitigator.service"
mode: "0644"
when: inventory_hostname == 'himachal'
notify: Reload systemd user daemon

- name: Enable and start system-mitigator daemon
ansible.builtin.systemd:
name: system-mitigator.service
scope: user
enabled: true
state: started
when: inventory_hostname == 'himachal'

# Phone hosts (termux, mumbai) can't reliably self-apply on a schedule —
# Termux can't run Ansible at all (no Rust target for the cryptography
# dependency), and neither host is always reachable (asleep, app killed,
# off tailnet). himachal periodically pushes to them instead, tolerating
# unreachable as the expected common case rather than an error.
- name: Deploy phone-apply service template
ansible.builtin.template:
src: phone-apply.service.j2
dest: "{{ ansible_facts['env']['HOME'] }}/.config/systemd/user/phone-apply.service"
mode: "0644"
when: inventory_hostname == 'himachal'
notify: Reload systemd user daemon

- name: Deploy phone-apply timer template
ansible.builtin.template:
src: phone-apply.timer.j2
dest: "{{ ansible_facts['env']['HOME'] }}/.config/systemd/user/phone-apply.timer"
mode: "0644"
when: inventory_hostname == 'himachal'
notify: Reload systemd user daemon

- name: Enable and start phone-apply timer
ansible.builtin.systemd:
name: phone-apply.timer
scope: user
enabled: true
state: started
daemon_reload: true
when: inventory_hostname == 'himachal' and phone_apply_enabled | default(true) | bool

- name: Stop and disable phone-apply timer when automatic pushes are off
ansible.builtin.systemd:
name: phone-apply.timer
scope: user
enabled: false
state: stopped
when: inventory_hostname == 'himachal' and not (phone_apply_enabled | default(true) | bool)
failed_when: false

- name: Remove disabled phone-apply units
ansible.builtin.file:
path: "{{ ansible_facts['env']['HOME'] }}/.config/systemd/user/{{ item }}"
state: absent
loop:
- phone-apply.service
- phone-apply.timer
when: inventory_hostname == 'himachal' and not (phone_apply_enabled | default(true) | bool)
notify: Reload systemd user daemon
5 changes: 5 additions & 0 deletions roles/systemd_hive_services/handlers/main.yml
Original file line number Diff line number Diff line change
@@ -0,0 +1,5 @@
---
- name: Reload systemd hive user daemon
ansible.builtin.systemd:
daemon_reload: true
scope: user
79 changes: 79 additions & 0 deletions roles/systemd_hive_services/tasks/main.yml
Original file line number Diff line number Diff line change
@@ -0,0 +1,79 @@
---
# Host-specific fleet services that do not belong to the generic systemd role.
- name: Ensure systemd user directory exists
ansible.builtin.file:
path: "{{ ansible_facts['env']['HOME'] }}/.config/systemd/user"
state: directory
mode: "0755"

- name: Deploy system-mitigator script (Lenovo 400MHz/Journald health)
ansible.builtin.copy:
src: system-mitigator.py
dest: "{{ ansible_facts['env']['HOME'] }}/.local/bin/system-mitigator.py"
mode: "0755"
when: inventory_hostname == 'himachal'

- name: Deploy system-mitigator service template
ansible.builtin.template:
src: system-mitigator.service.j2
dest: "{{ ansible_facts['env']['HOME'] }}/.config/systemd/user/system-mitigator.service"
mode: "0644"
when: inventory_hostname == 'himachal'
notify: Reload systemd hive user daemon

- name: Enable and start system-mitigator daemon
ansible.builtin.systemd:
name: system-mitigator.service
scope: user
enabled: true
state: started
when: inventory_hostname == 'himachal'

# Phone hosts (termux, mumbai) can't reliably self-apply on a schedule —
# Termux can't run Ansible at all (no Rust target for the cryptography
# dependency), and neither host is always reachable (asleep, app killed,
# off tailnet). himachal periodically pushes to them instead, tolerating
# unreachable as the expected common case rather than an error.
- name: Deploy phone-apply service template
ansible.builtin.template:
src: phone-apply.service.j2
dest: "{{ ansible_facts['env']['HOME'] }}/.config/systemd/user/phone-apply.service"
mode: "0644"
when: inventory_hostname == 'himachal'
notify: Reload systemd hive user daemon

- name: Deploy phone-apply timer template
ansible.builtin.template:
src: phone-apply.timer.j2
dest: "{{ ansible_facts['env']['HOME'] }}/.config/systemd/user/phone-apply.timer"
mode: "0644"
when: inventory_hostname == 'himachal'
notify: Reload systemd hive user daemon

- name: Enable and start phone-apply timer
ansible.builtin.systemd:
name: phone-apply.timer
scope: user
enabled: true
state: started
daemon_reload: true
when: inventory_hostname == 'himachal' and phone_apply_enabled | default(true) | bool

- name: Stop and disable phone-apply timer when automatic pushes are off
ansible.builtin.systemd:
name: phone-apply.timer
scope: user
enabled: false
state: stopped
when: inventory_hostname == 'himachal' and not (phone_apply_enabled | default(true) | bool)
failed_when: false

- name: Remove disabled phone-apply units
ansible.builtin.file:
path: "{{ ansible_facts['env']['HOME'] }}/.config/systemd/user/{{ item }}"
state: absent
loop:
- phone-apply.service
- phone-apply.timer
when: inventory_hostname == 'himachal' and not (phone_apply_enabled | default(true) | bool)
notify: Reload systemd hive user daemon
3 changes: 3 additions & 0 deletions site.yml
Original file line number Diff line number Diff line change
Expand Up @@ -130,6 +130,9 @@
- role: systemd
tags: [services, systemd]
when: not skip_systemd | default(false)
- role: systemd_hive_services
tags: [services, systemd, systemd_hive_services]
when: not skip_systemd | default(false)
- role: bst_dashboard
tags: [services, bst_dashboard]
- role: kirocrew
Expand Down
Loading