Skip to content

fix(broker): send start alerts at the same time, so a stop stays clean - #70

Merged
groupsky merged 1 commit into
mainfrom
fix/parallel-start-alerts
Oct 15, 2026
Merged

groupsky merged 1 commit into
mainfrom
fix/parallel-start-alerts

Conversation

@groupsky

Copy link
Copy Markdown
Contributor

What

Start alerts (a torn audit line found at start, a failed store) were sent one after another, and each first try can take up to 30 s. With 3 queued alerts and a Telegram that does not answer, a SIGTERM right after the start hit the broker's 80 s stop limit: the broker exited 1 without writing broker_stopped.

  • Each queued alert now has its own goroutine, so the first tries run at the same time. Every alert still gets at least one full try, and none starts after the stop begins, so the alert step ends within 30 s of the stop.
  • Each alert keeps its own growing wait between tries; failed-try lines carry alert=N. A panic in one alert does not stop the others, and its value is not logged.
  • An alert stuck in a long outage no longer blocks the alerts after it.

Tests

TestStartAlertsAtTheSameTime (synctest with the real 30 s and 80 s limits: a clean stop after exactly 30 s; the old code gave up at 1m20s), TestStartAlertsTryAgain, TestStartAlertPanics, and two tests with the real broker in a child process: a SIGTERM with three unanswered alerts stops cleanly with broker_stopped, and each alert goes out exactly once when Telegram comes back. All fail with the old code.

Docs

DESIGN.md section 15.

Review

One round with the security-reviewer agent: "ready to commit".

Closes #59

@groupsky groupsky added this to the v1 milestone Oct 15, 2026
@groupsky
groupsky merged commit 131611b into main Oct 15, 2026
7 checks passed
@groupsky
groupsky deleted the fix/parallel-start-alerts branch October 15, 2026 08:12
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Broker: send start alerts at the same time, so a stop during an outage stays clean

1 participant