Skip to content

Add fleet-wide host health monitoring - #42

Merged
Nickfost merged 6 commits into
mainfrom
feat/fleet-health-monitoring
Jul 23, 2026
Merged

Add fleet-wide host health monitoring#42
Nickfost merged 6 commits into
mainfrom
feat/fleet-health-monitoring

Conversation

@Nickfost

Copy link
Copy Markdown
Member

Closes #34.

Adds one redacted local health command, persisted machine-readable results, scheduled warning semantics, provider-neutral HTTPS heartbeats, external missed-heartbeat evaluation, host-local threshold/backup hooks, installer rollback integration, and operator recovery guidance.

Safety: no repair or global prune behavior; monitoring credentials remain host-local; warning exit 1 does not fail systemd or installer activation; critical exit 2 does.

Validated locally (the current workstation has no Docker daemon, so image/Compose validation is delegated to GitHub Actions):

  • planner tests: 7
  • desired-state tests: 11
  • health tests: 7
  • capacity preflight tests
  • installer/adoption/rollback tests
  • Python compilation, shell syntax, diff check, and committed-secret scan

@Nickfost

Copy link
Copy Markdown
Member Author

@codex review

Please review the exact current head for secret exposure, unsafe cleanup or mutation, false healthy/false unhealthy states, threshold and lifecycle handling, heartbeat trust-boundary validation, installer rollback behavior, timer semantics, and deterministic regression coverage.

@Nickfost

Copy link
Copy Markdown
Member Author

@codex review

Please review exact head 19ff666 after the non-root installer-fixture ownership fix. Recheck the complete diff and especially secret handling, stale-resource classification, timer age parsing, lifecycle states, warning/critical installer semantics, heartbeat configuration versus delivery failure, rollback/uninstall scope, and test coverage.

@chatgpt-codex-connector chatgpt-codex-connector Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: 19ff666a1f

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

Comment thread scripts/health.py Outdated
Comment thread scripts/health.py Outdated
Comment thread scripts/health.py Outdated
Comment thread scripts/install-worker-controller.sh Outdated
Comment thread scripts/health.py Outdated
@Nickfost

Copy link
Copy Markdown
Member Author

@codex review

Please review the complete PR at exact head 80f37d12683edbd681b2c8bc429eae6dae4eb8c1. Recheck secret handling, heartbeat trust boundaries, stale-resource classification, lifecycle semantics, sustained pressure/load collection, timer/service state, installer rollback/uninstall behavior, and deterministic regression coverage.

@chatgpt-codex-connector chatgpt-codex-connector Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: 80f37d1268

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

Comment thread scripts/health.py Outdated
Comment thread scripts/health.py Outdated
Comment thread scripts/health.py Outdated
Comment thread scripts/health.py Outdated
Comment thread scripts/health.py
Comment thread scripts/health.py Outdated
Comment thread scripts/health.py Outdated
@Nickfost

Copy link
Copy Markdown
Member Author

@codex review

Please review the complete PR at exact head cad423225ef75d262f48d9cf6664b6564365d970 after resolving all five prior findings with regression coverage. Recheck probe failure persistence, missing service units, future heartbeat skew, pre-health engine rollback compatibility, full systemd time-span parsing, and the complete security/safety scope.

@Nickfost

Copy link
Copy Markdown
Member Author

@codex review

Please review exact head cad423225ef75d262f48d9cf6664b6564365d970. The branch remains frozen. The earlier 80f37d1 review arrived after this head was pushed; this head fixes its timeout, future-heartbeat, and systemd-span findings, while the remaining self-service, non-Debian update, controller-identity, and expired-active-container findings are acknowledged and awaiting this exact-head review before the next edit cycle.

@chatgpt-codex-connector chatgpt-codex-connector Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: cad423225e

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

Comment thread scripts/test-install-worker-controller.sh Outdated
Comment thread scripts/install-worker-controller.sh
Comment thread scripts/healthcheck.sh
Comment thread scripts/health.py
@Nickfost

Copy link
Copy Markdown
Member Author

@codex review

Please review exact head 3c892d194260d5ad5a96ebad61a7335a7724ac25. Recheck the resolved findings: self-observation, portable update checks, heartbeat identity, expired active resources, source-export-safe installer tests, stopped capacity, manual environment loading, managed-image labels, and merge integration. The branch is frozen pending this exact-head review.

@chatgpt-codex-connector

Copy link
Copy Markdown

Codex Review: Didn't find any major issues. Hooray!

Reviewed commit: 3c892d1942

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

@Nickfost
Nickfost merged commit 9a257bc into main Jul 23, 2026
1 check passed
@Nickfost
Nickfost deleted the feat/fleet-health-monitoring branch July 23, 2026 05:10
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Add fleet-wide host health monitoring and missed-heartbeat alerts

1 participant