Skip to content

broker should lock its storage directory so two brokers cannot share sessions.log #179

Description

@fabracht

Since 0.42.0 all session writes go to one append-only file, <storage_dir>/sessions/sessions.log. Nothing stops two brokers, in one process or in two, from opening the same storage directory. Both then append to and compact the same log, so records interleave, one broker's compaction can overwrite the other's writes, and a failed flush can cut back records the other broker wrote. Clients see CONNACK 0x88 (Server Unavailable) when a session write fails. After a restart, a broker can resume stale sessions or lose some.

How it shows up

BrokerConfig::default() uses file storage in ./mqtt_storage relative to the working directory. Every integration test that starts a broker with the default config shares crates/mqtt5/mqtt_storage, and cargo runs them in parallel. On main (eb9b051), broker_quic_integration failed 3 of 8 local runs:

  • client must connect to the broker: ConnectionRefused(ServerUnavailable)
  • one run with 5 failures
  • Failed to bind QUIC endpoint: Address already in use on the in-process restart test, which went away once the QUIC tests used in-memory storage

The same happens in production when two mqttv5 broker processes are started from the same working directory, or share --storage-dir.

Proposed fix

  • At startup, take an exclusive advisory lock on a file in the storage directory (for example <storage_dir>/.lock, via flock/LockFileEx). Hold it for the broker's lifetime; the OS releases it on exit or crash.
  • If the lock is held, fail startup with a clear error naming the directory. Do not fall back to sharing it.
  • Integration tests that call run() should use in-memory storage or a temporary directory rather than ./mqtt_storage. The QUIC tests move to in-memory storage in add broker QUIC flow-control flags and stats, apply client stream limit #178.
  • Wasm uses in-memory storage and is not affected.

Activity

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions