Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
29 changes: 19 additions & 10 deletions DESIGN.md
Original file line number Diff line number Diff line change
Expand Up @@ -2066,11 +2066,15 @@ it started before.
broker again later, and no alert goes out each time.
8. It writes the `broker_started` entry. No later step of the start can
fail.
9. It starts to poll Telegram, and sends the alerts that waited. After a
power cut, the host can be up before Telegram can be reached. So when
an alert does not go out, the broker tries again: after 5 seconds,
then twice as long each time, at most 5 minutes, until the alert is
out or the broker stops. Each failed try gets one log line.
9. It starts to poll Telegram, and sends the alerts that waited. It sends
them at the same time, each on its own, so a slow try of one alert
does not delay the others. They can reach the phone in any order.
After a power cut, the host can be up before Telegram can be reached.
So when an alert does not go out, the broker tries again: after 5
seconds, then twice as long each time, at most 5 minutes, until the
alert is out or the broker stops. Each alert has its own tries. Each
failed try gets one log line, with the number of the alert, such as
`alert=2` for the second one that waited.
10. Only then does it read the connections on the `listen` address and
take requests.

Expand All @@ -2086,15 +2090,20 @@ order:
the `foreign_tap` entries that still wait.
3. The server: it closes the `listen` address and every connection.
4. The alerts of the start: each one gets its first try, also when the
stop began right after the start, but no try again. An alert that did
not go out is lost; the journal says so.
stop began right after the start, but no try again. The tries run at
the same time, and a try takes at most 30 seconds. So they end at
most 30 seconds after the stop began, however many alerts wait. An
alert that did not go out is lost; the journal says so.
5. The stores: they clear their tokens and session keys.
6. The `broker_stopped` entry. Then the audit log is closed.

The stop takes at most 80 seconds. A request may need up to 30 seconds
to take its answer, and a try of an alert up to 30 seconds. After 80 seconds the
broker ends without the rest of the stop, and exits with 1. The example
unit gives it 90 seconds (`TimeoutStopSec=`); then systemd kills it.
to take its answer, and the alert about an audit log that cannot be
written up to 30 seconds more. The tries of the alerts of the start run
at the same time as these steps, and end within 30 seconds. After 80
seconds the broker ends without the rest of the stop, and exits with 1.
The example unit gives it 90 seconds (`TimeoutStopSec=`); then systemd
kills it.

`nodkey broker` exits with:

Expand Down
77 changes: 47 additions & 30 deletions cmd/nodkey/broker.go
Original file line number Diff line number Diff line change
Expand Up @@ -64,8 +64,10 @@ const (
// stopTimeout bounds the whole stop. The service waits for the
// requests in progress, which may write an answer for up to 30
// seconds, and for its alert messages, up to 30 seconds more. The
// example unit sets TimeoutStopSec=90s, so the broker ends on its own
// first and logs why.
// tries of the alert messages of the start that are in progress run at
// the same time, for at most alertTimeout (sendAlerts). The example
// unit sets TimeoutStopSec=90s, so the broker ends on its own first and
// logs why.
stopTimeout = 80 * time.Second
)

Expand Down Expand Up @@ -222,8 +224,9 @@ type brokerParts struct {
// alerts are the alert messages that wait until the approver can send
// them: a torn line in the audit log, and stores that did not start.
alerts []string
// alertsDone is closed when the alerts went out, or the sender gave
// up because the stop began. stop closes alertsStop when it begins.
// alertsDone is closed when each alert went out, or its sender gave
// up because the stop began. stop closes alertsStop when it begins;
// the senders of the alerts read it.
alertsDone chan struct{}
alertsStop chan struct{}

Expand Down Expand Up @@ -352,7 +355,7 @@ func (b *brokerParts) start(ctx context.Context, path string) error {
}
b.started = true
b.startRun(core)
b.sendAlerts()
b.sendAlerts(tg.Alert)
b.serve(srv, ln)
b.log.Info("nodkey broker: listening; the broker is ready", slog.String("address", ln.Addr().String()))
return nil
Expand Down Expand Up @@ -449,29 +452,42 @@ func (b *brokerParts) startRun(core *approval.Core) {
b.log.Info("nodkey broker: the approver runs: it polls Telegram for the answers")
}

// sendAlerts sends the alert messages that waited for the approver, one
// after the other, in a goroutine of its own. Each one gets a first try,
// also when the stop has begun; the stop waits for it. After a failed
// try, sendAlert tries again (see alertFirstWait) until the alert is out
// or the stop begins. Each failed try gets one log line; the growing wait
// limits them to one line in 5 seconds at first and one in 5 minutes
// later. An alert that did not go out before the stop is lost, and the
// journal says so.
func (b *brokerParts) sendAlerts() {
// sendAlerts sends the alert messages that waited for the approver with
// send, each in a goroutine of its own. So their tries run at the same
// time: a slow try of one alert does not delay the others, and the stop
// waits for them at most as long as one try takes, however many alerts
// wait. Each alert gets a first try, also when the stop has begun; the
// stop waits for it. After a failed try, sendAlert tries again (see
// alertFirstWait) until the alert is out or the stop begins. Each failed
// try gets one log line, with the number of the alert: its place in the
// queue, from 1. The growing wait limits the lines of each alert to one
// in 5 seconds at first and one in 5 minutes later. An alert that did not
// go out before the stop is lost, and the journal says so.
func (b *brokerParts) sendAlerts(send broker.AlertFunc) {
msgs := b.alerts
b.alerts = nil
b.alertsDone = make(chan struct{})
done := make(chan struct{})
b.alertsDone = done
b.alertsStop = make(chan struct{})
sent := make([]bool, len(msgs))
var wg sync.WaitGroup
for i, msg := range msgs {
wg.Go(func() {
defer func() {
if recover() != nil {
b.log.Error("nodkey broker: sending an alert message panicked; the panic value is not logged",
slog.Int("alert", i+1))
}
}()
sent[i] = b.sendAlert(send, i+1, msg)
})
}
go func() {
defer close(b.alertsDone)
defer func() {
if recover() != nil {
b.log.Error("nodkey broker: sending an alert message panicked; the panic value is not logged")
}
}()
defer close(done)
wg.Wait()
lost := 0
for _, msg := range msgs {
if !b.sendAlert(msg) {
for _, ok := range sent {
if !ok {
lost++
}
}
Expand All @@ -482,26 +498,27 @@ func (b *brokerParts) sendAlerts() {
}()
}

// sendAlert tries to send msg until it is out, and reports true. After the
// stop began, it makes no new try, and reports false.
func (b *brokerParts) sendAlert(msg string) bool {
// sendAlert tries to send msg, the alert with the number n, until it is
// out, and reports true. After the stop began, it makes no new try, and
// reports false.
func (b *brokerParts) sendAlert(send broker.AlertFunc, n int, msg string) bool {
wait := alertFirstWait
for try := 1; ; try++ {
ctx, cancel := context.WithTimeout(context.Background(), alertTimeout)
err := b.tg.Alert(ctx, msg)
err := send(ctx, msg)
cancel()
if err == nil {
return true
}
select {
case <-b.alertsStop:
// Telegram's errors never hold the token or Go's error text.
b.log.Warn("nodkey broker: an alert message did not go out", slog.String("error", err.Error()),
slog.Int("try", try))
b.log.Warn("nodkey broker: an alert message did not go out", slog.Int("alert", n),
slog.String("error", err.Error()), slog.Int("try", try))
return false
default:
}
b.log.Warn("nodkey broker: an alert message did not go out; the broker tries again",
b.log.Warn("nodkey broker: an alert message did not go out; the broker tries again", slog.Int("alert", n),
slog.String("error", err.Error()), slog.Int("try", try), slog.Duration("next_try_in", wait))
t := time.NewTimer(wait)
select {
Expand Down
107 changes: 107 additions & 0 deletions cmd/nodkey/broker_linux_test.go
Original file line number Diff line number Diff line change
Expand Up @@ -475,6 +475,13 @@ func tearAuditLog(t *testing.T, e *brokerEnv) {
t.Helper()
e.signal(syscall.SIGTERM)
e.mustExit(exitOK)
appendTornLine(t, e)
}

// appendTornLine adds a torn last line to the audit log of e, as a power
// cut during a write would. The broker must have stopped.
func appendTornLine(t *testing.T, e *brokerEnv) {
t.Helper()
f, err := os.OpenFile(e.auditPath, os.O_WRONLY|os.O_APPEND, 0)
if err != nil {
t.Fatal(err)
Expand Down Expand Up @@ -671,6 +678,106 @@ func TestBrokerAlertTriesEndAtTheStop(t *testing.T) {
}
}

// threeAlertsAtTheNextStart makes the next start of the broker of e, with
// seamStoreStartFails, queue three alerts: it appends a log_recovered
// entry, as from a start that recorded a torn line and stopped before its
// broker_started entry, and then a torn line. It returns a part of the
// text of each alert: about the old torn line, the new torn line and the
// store. The broker must have stopped.
func threeAlertsAtTheNextStart(t *testing.T, e *brokerEnv) []string {
t.Helper()
appendForgedRecovery(t, e, 10, strings.Repeat("ab", 32))
entries := e.auditEntries()
seq, ok := entries[len(entries)-1]["seq"].(float64)
if !ok {
t.Fatal("the log_recovered entry has no seq")
}
appendTornLine(t, e)
return []string{
fmt.Sprintf("Entry %d (log_recovered)", int(seq)),
fmt.Sprintf("Entry %d (log_recovered)", int(seq)+1),
"could not start",
}
}

// TestBrokerStopWithThreeAlertsAndNoAnswer: three alerts wait at the start,
// and Telegram takes each alert message but never answers, as in a long
// outage, so each try runs into its time limit. The first tries of the
// three alerts run at the same time. A SIGTERM right after the start: the
// stop waits for these tries, as long as one of them takes, then stops
// cleanly within its limit and writes broker_stopped. Each alert got at
// least one try, and the log says that all three are lost. Tried one after
// the other, the three tries took longer than the stop limit (issue #59);
// TestStartAlertsAtTheSameTime checks the times of the real limits.
func TestBrokerStopWithThreeAlertsAndNoAnswer(t *testing.T) {
t.Parallel()
e := startListening(t, brokerOptions{holdAlerts: true})
e.stderr.wait(t, lineListening)
e.signal(syscall.SIGTERM)
e.mustExit(exitOK)
alerts := threeAlertsAtTheNextStart(t, e)
e.start(seamStoreStartFails, seamFastAlertTries)
e.stderr.wait(t, lineListening)
e.tg.waitHeld(len(alerts)) // the first tries, at the same time
e.signal(syscall.SIGTERM)
e.mustExit(exitOK) // a clean stop, in time
for _, alert := range alerts {
if e.tg.alertCount(alert) == 0 {
t.Errorf("the alert with %q got no try", alert)
}
}
if !e.stderr.has(`they are lost" lost=3`) {
t.Errorf("the log does not say that the three alerts are lost; its log:\n%s", e.stderr.text())
}
got := eventNames(e.auditEntries())
want := []string{
"broker_started", "broker_stopped", "log_recovered", "(torn line)", "log_recovered", "broker_started",
"broker_stopped",
}
if !slices.Equal(got, want) {
t.Errorf("audit entries %q, want %q", got, want)
}
}

// TestBrokerThreeAlertsGoOutOnce: three alerts wait at the start, and
// Telegram cannot be reached. Each alert tries again on its own. Once
// Telegram can be reached, the approver gets each alert exactly once, and
// none is lost.
func TestBrokerThreeAlertsGoOutOnce(t *testing.T) {
t.Parallel()
e := startListening(t, brokerOptions{})
e.stderr.wait(t, lineListening)
e.signal(syscall.SIGTERM)
e.mustExit(exitOK)
alerts := threeAlertsAtTheNextStart(t, e)
e.tg.setOffline(true)
e.start(seamStoreStartFails, seamFastAlertTries)
e.stderr.wait(t, lineListening)
for n := range len(alerts) {
e.stderr.waitCount(t, fmt.Sprintf(`%s" alert=%d `, lineAlertAgain, n+1), 2)
}
if n := e.tg.alertCount(""); n != 0 {
t.Fatalf("%d alerts while Telegram cannot be reached", n)
}
e.tg.setOffline(false)
for _, alert := range alerts {
e.tg.waitAlert(alert)
}
e.signal(syscall.SIGTERM)
e.mustExit(exitOK)
for _, alert := range alerts {
if n := e.tg.alertCount(alert); n != 1 {
t.Errorf("%d alerts with %q, want 1", n, alert)
}
}
if n := e.tg.alertCount(""); n != len(alerts) {
t.Errorf("%d alerts, want %d", n, len(alerts))
}
if e.stderr.has("they are lost") {
t.Errorf("the log says that alerts are lost; its log:\n%s", e.stderr.text())
}
}

// appendForgedRecovery appends a log_recovered entry that chains right
// onto the last entry of the audit log of e, as someone with write access
// to the log could: it names a torn line that never was.
Expand Down
Loading
Loading