Skip to content

ops: 0 compose service(s) drifted out of existence (#2747 watch) + 1 container(s) unhealthy for 1.0h+ (#3030 watch) #3220

Description

@github-actions

Sweep at 2026-09-18T00:36:32Z on supermicro: 0 compose-defined, always-on service(s) have no container at all (not even stopped), 0 live stack(s) no longer exist in the repo manifest, 1 container(s) have had a healthcheck failing streak continuously for at least 1.0h, and 0 running container(s) have a cpus/memory limit that diverges from the repo's declared value.

Healthcheck failing streak

Context: #3030 — a container that exists and is Up but keeps failing its own healthcheck (FailingStreak from docker inspect) continuously for at least 1.0h. Tracked as a duration across sweeps, not a raw streak count — hp-autoheal restarts a container the moment Docker marks it unhealthy, which resets FailingStreak to 0 well before any fixed count threshold is reached (#3023 observed exactly FailingStreak=3, never higher). This is a distinct failure mode from the sections above: the container was never missing, so restart: policies never get a chance to help — the thing that's broken outlives a container restart (a wedged dependency, a resource the healthcheck needs that a plain restart doesn't free). Check docker inspect <name> --format '{{json .State.Health}}' for the actual probe failure before restarting anything.

  • hp-arkime-capture — unhealthy for 2.3h (10 consecutive healthcheck failures, status: starting)

Activity

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    compose-drift-alarmCompose-defined service missing all containers while a sibling runs (scripts/compose-drift-watch.py)

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions