From 5219b14340219f9e4bf021c9038de5359ff2ef18 Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?Fabr=C3=ADcio=20Bracht?= Date: Thu, 24 Sep 2026 13:57:30 -0300 Subject: [PATCH 1/2] cancel pending delayed will on reconnect and bound it by session expiry --- CHANGELOG.md | 21 + crates/mqtt5-conformance/CONFORMANCE_DIARY.md | 12 + crates/mqtt5-conformance/conformance.toml | 2 +- .../section3_connect_extended.rs | 54 ++- crates/mqtt5-wasm/Cargo.toml | 2 +- crates/mqtt5-wasm/src/broker.rs | 98 +++++ .../src/client_handler/lifecycle.rs | 117 +++--- crates/mqtt5-wasm/src/client_handler/mod.rs | 24 +- crates/mqtt5-wasm/tests/broker_will_delay.rs | 332 +++++++++++++++ crates/mqtt5/Cargo.toml | 2 +- .../src/broker/client_handler/lifecycle.rs | 144 +++---- crates/mqtt5/src/broker/client_handler/mod.rs | 69 +++- crates/mqtt5/src/broker/router.rs | 165 +++++++- crates/mqtt5/src/broker/storage/mod.rs | 7 + crates/mqtt5/src/broker/storage/tests.rs | 29 ++ crates/mqtt5/tests/will_delay.rs | 390 ++++++++++++++++++ 16 files changed, 1317 insertions(+), 151 deletions(-) create mode 100644 crates/mqtt5-wasm/tests/broker_will_delay.rs create mode 100644 crates/mqtt5/tests/will_delay.rs diff --git a/CHANGELOG.md b/CHANGELOG.md index f6d721af..4075884c 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -5,6 +5,27 @@ All notable changes to this project will be documented in this file. The format is based on [Keep a Changelog](https://keepachangelog.com/en/1.0.0/), and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0.html). +## [mqtt5 0.41.1] - 2026-09-24 + +### Fixed + +- **A delayed Will Message is cancelled when a new connection for the same ClientID opens before the Will Delay Interval elapses** (`[MQTT-3.1.3-9]`, `[MQTT-3.1.2-8]`). This applies whether the new connection resumes the session, starts clean, or takes over a live connection. Before, the broker always published the Will after the delay, so a client that dropped and reconnected got a spurious Will. Reported in #154. +- **The Will is published when the Will Delay Interval elapses or the session ends, whichever comes first.** A Session Expiry Interval of 0 publishes it immediately at disconnect, and a Session Expiry shorter than the Will Delay publishes it when the session ends. Before, the full delay was always waited out. +- **A published Will, or one deleted by DISCONNECT 0x00, is removed from the stored session state** (`[MQTT-3.1.2-10]`). +- The conformance test for `[MQTT-3.1.3-9]` passed vacuously: it stopped watching before the Will Delay elapsed. It now waits past the delay, and a new test checks that the Will is published when nobody reconnects. + +### Added + +- `MessageRouter::arm_will`, `MessageRouter::claim_will`, `MessageRouter::clear_stored_will`, `MessageRouter::owns_client` and `ClientSession::will_publish_delay`, which back the pending-Will tracking. + +## [mqtt5-wasm 2.0.1] - 2026-09-24 + +### Fixed + +- **The in-browser broker cancels a delayed Will when the client reconnects within the Will Delay Interval**, and publishes it no later than session end. This is the same fix as mqtt5 0.41.1. +- **The in-browser broker detects a client closing its MessagePort** (the port's `close` event) and treats it as an abnormal disconnect, so the Will is published. Before, a closed port went unnoticed until keep-alive expiry, and never with a keep-alive of 0, which also leaked the connection handler. Environments that don't raise `close` on MessagePort still rely on keep-alive expiry. +- **A published Will, or one deleted by DISCONNECT 0x00, is removed from the in-browser broker's stored session** (`[MQTT-3.1.2-10]`). + ## [mqtt5 0.41.0] - 2026-09-23 A client-side conformance audit drove the real `MqttClient` against a raw-byte fake broker for each of the 149 normative statements in MQTT v5.0 that apply to a client. About 40 MUST statements failed. All are fixed here, and each is pinned by a test named after its OASIS statement ID in `crates/mqtt5/tests/conf_client_{a,b,c,d}.rs`. diff --git a/crates/mqtt5-conformance/CONFORMANCE_DIARY.md b/crates/mqtt5-conformance/CONFORMANCE_DIARY.md index edbe93ec..2fd5da10 100644 --- a/crates/mqtt5-conformance/CONFORMANCE_DIARY.md +++ b/crates/mqtt5-conformance/CONFORMANCE_DIARY.md @@ -38,6 +38,18 @@ ## Diary Entries +### Delayed Will was never cancelled, and the MQTT-3.1.3-9 test could not see it (2026-09-24) + +**Trigger**: issue #154. With a Will Delay Interval above zero, the broker spawned a detached task that slept for the delay and then published the Will unconditionally. A client that reconnected inside the delay still had its Will published, which violates `[MQTT-3.1.3-9]` and the "new Network Connection ... before the Will Delay Interval has elapsed" clause of `[MQTT-3.1.2-8]`. + +**Why the suite passed anyway**: `will_delay_reconnect_suppresses_will` used a 5 s delay but stopped watching about 2.3 s after the drop, so the stale Will always arrived after the assertion. The test was vacuous. It now uses a 2 s delay and waits 4 s after the reconnect. On the unfixed broker it fails with the Will received. A positive control, `will_delay_elapsed_publishes_will`, runs the same setup without a reconnect and asserts that nothing arrives at 1.2 s and that the Will arrives once the delay has elapsed. It passes on both the old and the fixed broker, which shows the negative test fails for the right reason and not because Wills are never delivered. + +**Fix**: the router keeps one pending delayed Will per client id, tagged with the generation of the connection that armed it. A connection arms its Will before it releases its router entry, and only if it still owns that entry. `register_session` removes any pending Will for the client id while it holds the clients write lock, so any new connection (Clean Start 0 or 1, takeover included) cancels it. When the timer fires, the task has to claim the entry by generation before it publishes. Claim and cancel both remove the entry under one mutex, so exactly one of them wins. The Will fires at min(Will Delay Interval, Session Expiry Interval), so a Session Expiry of 0 publishes at once and a shorter expiry publishes when the session ends (`[MQTT-3.1.2-8]`, ยง3.1.3.2.2). A published Will, and a Will deleted by DISCONNECT 0x00, is also removed from the stored session (`[MQTT-3.1.2-10]`). + +**Manifest**: both tests are listed under MQTT-3.1.3-9, whose manifest text matches `mqtt-v5.0-statement-texts.txt`. The manifest entry labelled MQTT-3.1.2-8 carries the Will Retain text ("If the Will Flag is set to 0, then Will Retain MUST be set to 0"), not the Will publication statement, so neither test is cited there. That drift is left as it was. + +**Broker-side coverage**: `crates/mqtt5/tests/will_delay.rs` covers resume and clean-start reconnects, no reconnect, Session Expiry 0 and 2 against longer delays, reconnect-then-drop, DISCONNECT 0x00 and 0x04, and takeover with and without a delay. Six of these fail on the unfixed broker. The other four are regression guards for behaviour that was already correct. + ### Quorum review of the client fixes, and a TLA+-verified outcome model for the offline queue (2026-09-23) **Trigger**: a five-reviewer quorum review of PR #164 before merge. Most findings came with a failing test. The worst was a regression in the wasm client: a v3.1.1 persistent session could never reconnect, because the session-lifetime check used Session Expiry (always 0 in 3.1.1) and the new strict `[MQTT-3.2.2-4]` check then rejected the broker's Session Present=1 forever. Other findings: QUIC teardown left the connection open after the client's own DISCONNECT; a publish waiting across a reconnect was sent under the old server's limits; and the offline queue dropped messages silently after `publish()` had returned success. All were fixed in the same PR. diff --git a/crates/mqtt5-conformance/conformance.toml b/crates/mqtt5-conformance/conformance.toml index 6861dd8c..4bfb2c74 100644 --- a/crates/mqtt5-conformance/conformance.toml +++ b/crates/mqtt5-conformance/conformance.toml @@ -519,7 +519,7 @@ level = "MustNot" applies_to = "Server" text = "If a new Network Connection to this Session is made before the Will Delay Interval has passed, the Server MUST NOT send the Will Message" status = "Tested" -test_names = ["will_delay_reconnect_suppresses_will"] +test_names = ["will_delay_reconnect_suppresses_will", "will_delay_elapsed_publishes_will"] [[sections."3.1".statements]] id = "MQTT-3.1.3-10" diff --git a/crates/mqtt5-conformance/src/conformance_tests/section3_connect_extended.rs b/crates/mqtt5-conformance/src/conformance_tests/section3_connect_extended.rs index a9550833..338cc22a 100644 --- a/crates/mqtt5-conformance/src/conformance_tests/section3_connect_extended.rs +++ b/crates/mqtt5-conformance/src/conformance_tests/section3_connect_extended.rs @@ -379,7 +379,7 @@ async fn will_delay_reconnect_suppresses_will(sut: SutHandle) { .await .unwrap(); raw.send_raw(&RawPacketBuilder::connect_with_will_delay( - &client_id, 5, 60, + &client_id, 2, 60, )) .await .unwrap(); @@ -398,7 +398,7 @@ async fn will_delay_reconnect_suppresses_will(sut: SutHandle) { .await .expect("reconnect failed"); - tokio::time::sleep(Duration::from_secs(2)).await; + tokio::time::sleep(Duration::from_secs(4)).await; assert_eq!( subscription.count(), @@ -410,6 +410,56 @@ async fn will_delay_reconnect_suppresses_will(sut: SutHandle) { subscriber.disconnect().await.expect("disconnect failed"); } +/// Positive control for `[MQTT-3.1.3-9]`: without a reconnect, the delayed +/// Will is published once the Will Delay Interval elapses and not before. +#[conformance_test( + ids = ["MQTT-3.1.3-9"], + requires = ["transport.tcp"], +)] +async fn will_delay_elapsed_publishes_will(sut: SutHandle) { + let client_id = unique_client_id("wdpub"); + let will_topic = format!("will/{client_id}"); + + let subscriber = TestClient::connect_with_prefix(&sut, "wdpub-sub") + .await + .unwrap(); + let subscription = subscriber + .subscribe(&will_topic, SubscribeOptions::default()) + .await + .expect("subscribe failed"); + tokio::time::sleep(Duration::from_millis(100)).await; + + let mut raw = RawMqttClient::connect_tcp(sut.expect_tcp_addr()) + .await + .unwrap(); + raw.send_raw(&RawPacketBuilder::connect_with_will_delay( + &client_id, 2, 60, + )) + .await + .unwrap(); + let connack = raw.expect_connack(TIMEOUT).await; + assert!(connack.is_some(), "Must receive CONNACK"); + let (_, reason) = connack.unwrap(); + assert_eq!(reason, 0x00, "Connection must succeed"); + + drop(raw); + tokio::time::sleep(Duration::from_millis(1200)).await; + + assert_eq!( + subscription.count(), + 0, + "Will message must not be sent before the Will Delay Interval elapses" + ); + assert!( + subscription + .wait_for_messages(1, Duration::from_secs(4)) + .await, + "Will message must be sent once the Will Delay Interval elapses without a reconnect" + ); + + subscriber.disconnect().await.expect("disconnect failed"); +} + /// `[MQTT-3.1.3-10]` The User Property is part of the Will Properties and /// the Server MUST maintain the order of User Properties when publishing the /// Will Message. diff --git a/crates/mqtt5-wasm/Cargo.toml b/crates/mqtt5-wasm/Cargo.toml index bfc28f82..21013946 100644 --- a/crates/mqtt5-wasm/Cargo.toml +++ b/crates/mqtt5-wasm/Cargo.toml @@ -1,6 +1,6 @@ [package] name = "mqtt5-wasm" -version = "2.0.0" +version = "2.0.1" edition.workspace = true rust-version.workspace = true authors.workspace = true diff --git a/crates/mqtt5-wasm/src/broker.rs b/crates/mqtt5-wasm/src/broker.rs index 14d4308c..0fc956ec 100644 --- a/crates/mqtt5-wasm/src/broker.rs +++ b/crates/mqtt5-wasm/src/broker.rs @@ -806,3 +806,101 @@ impl WasmBroker { ) } } + +#[cfg(all(test, target_arch = "wasm32"))] +mod tests { + use super::{WasmBroker, WasmBrokerConfig}; + use bytes::BytesMut; + use mqtt5::broker::storage::StorageBackend; + use mqtt5_protocol::packet::connect::ConnectPacket; + use mqtt5_protocol::packet::disconnect::DisconnectPacket; + use mqtt5_protocol::packet::MqttPacket; + use mqtt5_protocol::protocol::v5::reason_codes::ReasonCode; + use mqtt5_protocol::types::{ConnectOptions, WillMessage}; + use std::cell::RefCell; + use std::rc::Rc; + use wasm_bindgen::prelude::*; + use wasm_bindgen::JsCast; + use wasm_bindgen_test::wasm_bindgen_test; + use web_sys::{MessageEvent, MessagePort}; + + async fn sleep(ms: u32) { + gloo_timers::future::TimeoutFuture::new(ms).await; + } + + fn send(port: &MessagePort, packet: &impl MqttPacket) { + let mut buf = BytesMut::new(); + packet.encode(&mut buf).unwrap(); + port.post_message(&js_sys::Uint8Array::from(&buf[..]).buffer()) + .unwrap(); + } + + async fn connect_with_will(broker: &WasmBroker, client_id: &str) -> MessagePort { + let port = broker.create_client_port().unwrap(); + let inbox = Rc::new(RefCell::new(Vec::new())); + let inbox_in = Rc::clone(&inbox); + let on_message = Closure::::new(move |event: MessageEvent| { + inbox_in + .borrow_mut() + .extend(js_sys::Uint8Array::new(&event.data()).to_vec()); + }); + port.add_event_listener_with_callback("message", on_message.as_ref().unchecked_ref()) + .unwrap(); + on_message.forget(); + port.start(); + let options = ConnectOptions::new(client_id) + .with_session_expiry_interval(60) + .with_will(WillMessage::new(format!("will/{client_id}"), "offline")); + send(&port, &ConnectPacket::new(options)); + for _ in 0..200 { + if inbox.borrow().len() >= 4 { + break; + } + sleep(5).await; + } + assert_eq!(inbox.borrow().first(), Some(&0x20), "expected CONNACK"); + port + } + + async fn stored_will_present(broker: &WasmBroker, client_id: &str) -> bool { + broker + .storage + .get_session(client_id) + .await + .unwrap() + .expect("session is kept for its expiry interval") + .will_message + .is_some() + } + + fn broker() -> WasmBroker { + let mut config = WasmBrokerConfig::new(); + config.set_allow_anonymous(true); + WasmBroker::with_config(config).unwrap() + } + + #[wasm_bindgen_test] + async fn normal_disconnect_removes_stored_will() { + let broker = broker(); + let port = connect_with_will(&broker, "stored-will-normal").await; + assert!(stored_will_present(&broker, "stored-will-normal").await); + + send(&port, &DisconnectPacket::new(ReasonCode::Success)); + port.close(); + sleep(300).await; + + assert!(!stored_will_present(&broker, "stored-will-normal").await); + } + + #[wasm_bindgen_test] + async fn published_will_is_removed_from_stored_session() { + let broker = broker(); + let port = connect_with_will(&broker, "stored-will-published").await; + assert!(stored_will_present(&broker, "stored-will-published").await); + + port.close(); + sleep(300).await; + + assert!(!stored_will_present(&broker, "stored-will-published").await); + } +} diff --git a/crates/mqtt5-wasm/src/client_handler/lifecycle.rs b/crates/mqtt5-wasm/src/client_handler/lifecycle.rs index 46f659c7..718c1928 100644 --- a/crates/mqtt5-wasm/src/client_handler/lifecycle.rs +++ b/crates/mqtt5-wasm/src/client_handler/lifecycle.rs @@ -1,3 +1,4 @@ +use futures::future::{select, Either}; use mqtt5::broker::auth::EnhancedAuthStatus; use mqtt5_protocol::error::{MqttError, Result}; use mqtt5_protocol::packet::auth::AuthPacket; @@ -190,59 +191,73 @@ impl WasmClientHandler { true } + pub(super) async fn clear_owned_stored_will(&self, client_id: &str) { + if self.router.owns_client(client_id, self.generation).await { + self.router.clear_stored_will(client_id).await; + } + } + pub(super) async fn publish_will_message(&self, client_id: &str) { - if let Some(ref session) = self.session { - if let Some(ref will) = session.will_message { - debug!("Publishing will message for client {}", client_id); - - let mut publish = - PublishPacket::new(will.topic.clone(), will.payload.clone(), will.qos); - publish.retain = will.retain; - - will.properties - .apply_to_publish_properties(&mut publish.properties); - publish.properties.inject_sender(self.user_id.as_deref()); - publish.properties.inject_client_id(Some(client_id)); - - if let Some(delay) = session.will_delay_interval { - if delay > 0 { - debug!("Spawning task to publish will after {} seconds", delay); - let router = Arc::clone(&self.router); - let auth_provider = Arc::clone(&self.auth_provider); - let user_id = self.user_id.clone(); - let publish_clone = publish.clone(); - let client_id_clone = client_id.to_string(); - spawn_local(async move { - gloo_timers::future::sleep(std::time::Duration::from_secs(u64::from( - delay, - ))) - .await; - - let authorized = auth_provider - .authorize_publish( - &client_id_clone, - user_id.as_deref(), - &publish_clone.topic_name, - ) - .await; - if !authorized { - warn!( - "Delayed will for {} denied for topic {}", - client_id_clone, publish_clone.topic_name - ); - return; - } - - debug!("Publishing delayed will message for {}", client_id_clone); - router.route_message(&publish_clone, None).await; - }); - } else if self.authorize_will(client_id, &publish).await { - self.router.route_message(&publish, None).await; - } - } else if self.authorize_will(client_id, &publish).await { - self.router.route_message(&publish, None).await; - } + let Some(session) = self.session.as_ref() else { + return; + }; + let (Some(will), Some(delay)) = (&session.will_message, session.will_publish_delay()) + else { + return; + }; + + let mut publish = PublishPacket::new(will.topic.clone(), will.payload.clone(), will.qos); + publish.retain = will.retain; + will.properties + .apply_to_publish_properties(&mut publish.properties); + publish.properties.inject_sender(self.user_id.as_deref()); + publish.properties.inject_client_id(Some(client_id)); + + if delay == 0 { + if self.authorize_will(client_id, &publish).await { + self.router.route_message(&publish, None).await; } + self.clear_owned_stored_will(client_id).await; + return; } + + let Some(cancelled) = self.router.arm_will(client_id, self.generation).await else { + debug!( + client_id, + "Delayed will dropped: a new connection for the client id was opened" + ); + return; + }; + + debug!(client_id, delay, "Scheduling delayed will"); + let router = Arc::clone(&self.router); + let auth_provider = Arc::clone(&self.auth_provider); + let user_id = self.user_id.clone(); + let client_id = client_id.to_string(); + let generation = self.generation; + spawn_local(async move { + let timer = + gloo_timers::future::sleep(std::time::Duration::from_secs(u64::from(delay))); + if let Either::Right(_) = select(timer, cancelled).await { + debug!(client_id, "Delayed will cancelled by a new connection"); + return; + } + if !router.claim_will(&client_id, generation).await { + debug!(client_id, "Delayed will cancelled by a new connection"); + return; + } + let authorized = auth_provider + .authorize_publish(&client_id, user_id.as_deref(), &publish.topic_name) + .await; + if !authorized { + warn!( + "Delayed will for {client_id} denied for topic {}", + publish.topic_name + ); + return; + } + debug!(client_id, "Publishing delayed will"); + router.route_message(&publish, None).await; + }); } } diff --git a/crates/mqtt5-wasm/src/client_handler/mod.rs b/crates/mqtt5-wasm/src/client_handler/mod.rs index 3a2cdb2f..29ebc055 100644 --- a/crates/mqtt5-wasm/src/client_handler/mod.rs +++ b/crates/mqtt5-wasm/src/client_handler/mod.rs @@ -158,13 +158,26 @@ impl WasmClientHandler { move |e: web_sys::MessageEvent| { if let Ok(abuf) = e.data().dyn_into::() { let array = js_sys::Uint8Array::new(&abuf); - let vec = array.to_vec(); - let _ = msg_tx_clone.unbounded_send(vec); + if msg_tx_clone.unbounded_send(array.to_vec()).is_err() { + debug!("Dropping message received after the client port closed"); + } } }, ); let js_fn: js_sys::Function = handler_fn.into_js_value().unchecked_into(); - let _ = port.add_event_listener_with_callback("message", &js_fn); + if let Err(e) = port.add_event_listener_with_callback("message", &js_fn) { + error!("Failed to listen for client port messages: {e:?}"); + } + let close_fn = wasm_bindgen::closure::Closure::::new( + move |_: web_sys::Event| { + debug!("Client port closed"); + msg_tx.close_channel(); + }, + ); + let close_js: js_sys::Function = close_fn.into_js_value().unchecked_into(); + if let Err(e) = port.add_event_listener_with_callback("close", &close_js) { + error!("Failed to listen for client port close: {e:?}"); + } port.start(); let handler = Self { @@ -238,7 +251,9 @@ impl WasmClientHandler { let result = self.packet_loop(&mut reader, writer, disconnect_rx).await; - if !self.normal_disconnect { + if self.normal_disconnect { + self.clear_owned_stored_will(&client_id).await; + } else { self.publish_will_message(&client_id).await; } @@ -291,6 +306,7 @@ impl WasmClientHandler { let result = self.packet_loop(&mut reader, writer, disconnect_rx).await; let (reason, unexpected) = if self.normal_disconnect { + self.clear_owned_stored_will(&client_id).await; ("client disconnected", false) } else { self.publish_will_message(&client_id).await; diff --git a/crates/mqtt5-wasm/tests/broker_will_delay.rs b/crates/mqtt5-wasm/tests/broker_will_delay.rs new file mode 100644 index 00000000..c53ffa9e --- /dev/null +++ b/crates/mqtt5-wasm/tests/broker_will_delay.rs @@ -0,0 +1,332 @@ +#![cfg(all(target_arch = "wasm32", feature = "broker"))] + +use bytes::BytesMut; +use mqtt5_protocol::packet::connect::ConnectPacket; +use mqtt5_protocol::packet::disconnect::DisconnectPacket; +use mqtt5_protocol::packet::MqttPacket; +use mqtt5_protocol::protocol::v5::reason_codes::ReasonCode; +use mqtt5_protocol::types::{ConnectOptions, WillMessage}; +use mqtt5_wasm::{ + WasmBroker, WasmBrokerConfig, WasmConnectOptions, WasmMqttClient, WasmSubscribeOptions, +}; +use std::cell::{Cell, RefCell}; +use std::rc::Rc; +use std::time::Duration; +use wasm_bindgen::prelude::*; +use wasm_bindgen::JsCast; +use wasm_bindgen_futures::JsFuture; +use wasm_bindgen_test::wasm_bindgen_test; +use web_sys::{MessageEvent, MessagePort}; + +const SESSION_EXPIRY: u32 = 60; + +async fn sleep(ms: i32) { + let promise = js_sys::Promise::new(&mut |resolve, _| { + let set_timeout = js_sys::Reflect::get(&js_sys::global(), &JsValue::from_str("setTimeout")) + .unwrap() + .unchecked_into::(); + set_timeout + .call2(&JsValue::NULL, &resolve, &JsValue::from(ms)) + .unwrap(); + }); + JsFuture::from(promise).await.unwrap(); +} + +fn now_ms() -> f64 { + js_sys::Date::now() +} + +async fn sleep_until(deadline_ms: f64) { + while now_ms() < deadline_ms { + sleep(10).await; + } +} + +fn broker() -> WasmBroker { + let mut config = WasmBrokerConfig::new(); + config.set_allow_anonymous(true); + WasmBroker::with_config(config).unwrap() +} + +fn will_topic(client_id: &str) -> String { + format!("will/{client_id}") +} + +fn delayed_will(client_id: &str, delay: u32) -> WillMessage { + let mut will = WillMessage::new(will_topic(client_id), "offline"); + will.properties.will_delay_interval = Some(delay); + will +} + +struct RawClient { + port: MessagePort, + inbox: Rc>>, + on_message: Closure, +} + +impl RawClient { + async fn connect( + broker: &WasmBroker, + client_id: &str, + clean_start: bool, + session_expiry: u32, + will: Option, + ) -> Self { + Self::connect_with_keep_alive(broker, client_id, clean_start, session_expiry, will, 60) + .await + } + + async fn connect_with_keep_alive( + broker: &WasmBroker, + client_id: &str, + clean_start: bool, + session_expiry: u32, + will: Option, + keep_alive_secs: u64, + ) -> Self { + let port = broker.create_client_port().unwrap(); + let inbox = Rc::new(RefCell::new(Vec::new())); + let inbox_in = Rc::clone(&inbox); + let on_message = Closure::::new(move |event: MessageEvent| { + let data = js_sys::Uint8Array::new(&event.data()); + inbox_in.borrow_mut().extend(data.to_vec()); + }); + port.add_event_listener_with_callback("message", on_message.as_ref().unchecked_ref()) + .unwrap(); + port.start(); + let client = Self { + port, + inbox, + on_message, + }; + + let options = ConnectOptions::new(client_id) + .with_clean_start(clean_start) + .with_session_expiry_interval(session_expiry) + .with_keep_alive(Duration::from_secs(keep_alive_secs)); + let options = match will { + Some(will) => options.with_will(will), + None => options, + }; + client.send(&ConnectPacket::new(options)); + + for _ in 0..200 { + if client.inbox.borrow().len() >= 4 { + break; + } + sleep(5).await; + } + let inbox = client.inbox.borrow().clone(); + assert!(inbox.len() >= 4, "no CONNACK for {client_id}"); + assert_eq!(inbox[0], 0x20, "expected CONNACK"); + assert_eq!(inbox[3], 0x00, "CONNACK reason must be Success"); + client + } + + fn send(&self, packet: &impl MqttPacket) { + let mut buf = BytesMut::new(); + packet.encode(&mut buf).unwrap(); + let array = js_sys::Uint8Array::from(&buf[..]); + self.port.post_message(&array.buffer()).unwrap(); + } + + fn close_without_disconnect(self) { + self.port.close(); + } + + fn disconnect(self, reason: ReasonCode) { + self.send(&DisconnectPacket::new(reason)); + self.port.close(); + } +} + +impl Drop for RawClient { + fn drop(&mut self) { + self.port + .remove_event_listener_with_callback( + "message", + self.on_message.as_ref().unchecked_ref(), + ) + .unwrap(); + } +} + +async fn watch_will(broker: &WasmBroker, client_id: &str) -> (WasmMqttClient, Rc>) { + let watcher = WasmMqttClient::new(format!("{client_id}-watcher")); + watcher + .connect_message_port_with_options( + broker.create_client_port().unwrap(), + &WasmConnectOptions::new(), + ) + .await + .unwrap(); + let count = Rc::new(Cell::new(0usize)); + let sink = Rc::clone(&count); + let callback = Closure::::new( + move |_: JsValue, _: JsValue, _: JsValue| sink.set(sink.get() + 1), + ); + watcher + .subscribe_with_options( + &will_topic(client_id), + callback.into_js_value().unchecked_into(), + &WasmSubscribeOptions::new(), + ) + .await + .unwrap(); + sleep(100).await; + (watcher, count) +} + +async fn wait_for_will(count: &Rc>, timeout_ms: i32) -> bool { + let deadline = now_ms() + f64::from(timeout_ms); + while now_ms() < deadline { + if count.get() > 0 { + return true; + } + sleep(20).await; + } + count.get() > 0 +} + +async fn reconnect_within_delay_cancels_will(client_id: &str, clean_start: bool) { + let broker = broker(); + let (watcher, count) = watch_will(&broker, client_id).await; + + let first = RawClient::connect( + &broker, + client_id, + true, + SESSION_EXPIRY, + Some(delayed_will(client_id, 1)), + ) + .await; + let dropped_at = now_ms(); + first.close_without_disconnect(); + sleep(300).await; + + let second = RawClient::connect(&broker, client_id, clean_start, SESSION_EXPIRY, None).await; + + sleep_until(dropped_at + 3000.0).await; + assert_eq!( + count.get(), + 0, + "a reconnect within the Will Delay Interval must cancel the Will" + ); + + second.disconnect(ReasonCode::Success); + watcher.disconnect().await.unwrap(); +} + +#[wasm_bindgen_test] +async fn will_published_after_delay_without_reconnect() { + let broker = broker(); + let client_id = "wasm-wd-no-reconnect"; + let (watcher, count) = watch_will(&broker, client_id).await; + + let conn = RawClient::connect( + &broker, + client_id, + true, + SESSION_EXPIRY, + Some(delayed_will(client_id, 2)), + ) + .await; + let dropped_at = now_ms(); + conn.close_without_disconnect(); + + sleep_until(dropped_at + 1200.0).await; + assert_eq!( + count.get(), + 0, + "the Will must not be published before the Will Delay Interval" + ); + assert!( + wait_for_will(&count, 3000).await, + "the Will must be published once the Will Delay Interval elapses" + ); + sleep(300).await; + assert_eq!(count.get(), 1, "the Will is published once"); + + watcher.disconnect().await.unwrap(); +} + +#[wasm_bindgen_test] +async fn resume_within_delay_cancels_will() { + reconnect_within_delay_cancels_will("wasm-wd-resume", false).await; +} + +#[wasm_bindgen_test] +async fn clean_start_within_delay_cancels_will() { + reconnect_within_delay_cancels_will("wasm-wd-clean", true).await; +} + +#[wasm_bindgen_test] +async fn session_expiry_zero_publishes_will_immediately() { + let broker = broker(); + let client_id = "wasm-wd-expiry-zero"; + let (watcher, count) = watch_will(&broker, client_id).await; + + let conn = RawClient::connect( + &broker, + client_id, + true, + 0, + Some(delayed_will(client_id, 5)), + ) + .await; + conn.close_without_disconnect(); + + assert!( + wait_for_will(&count, 1500).await, + "the Session ends at disconnect, so the Will must not wait for the delay" + ); + + watcher.disconnect().await.unwrap(); +} + +#[wasm_bindgen_test] +async fn normal_disconnect_then_close_discards_will() { + let broker = broker(); + let client_id = "wasm-wd-normal"; + let (watcher, count) = watch_will(&broker, client_id).await; + + let conn = RawClient::connect( + &broker, + client_id, + true, + SESSION_EXPIRY, + Some(delayed_will(client_id, 0)), + ) + .await; + conn.disconnect(ReasonCode::Success); + + sleep(1500).await; + assert_eq!(count.get(), 0, "DISCONNECT 0x00 must delete the Will"); + + watcher.disconnect().await.unwrap(); +} + +#[wasm_bindgen_test] +async fn keep_alive_expiry_publishes_will() { + let broker = broker(); + let client_id = "wasm-wd-keepalive"; + let (watcher, count) = watch_will(&broker, client_id).await; + + let silent = RawClient::connect_with_keep_alive( + &broker, + client_id, + true, + SESSION_EXPIRY, + Some(delayed_will(client_id, 0)), + 1, + ) + .await; + + assert!( + wait_for_will(&count, 4000).await, + "a client silent past 1.5x its Keep Alive is disconnected and its Will published" + ); + + drop(silent); + watcher.disconnect().await.unwrap(); +} diff --git a/crates/mqtt5/Cargo.toml b/crates/mqtt5/Cargo.toml index 4c6d9ee5..830099c1 100644 --- a/crates/mqtt5/Cargo.toml +++ b/crates/mqtt5/Cargo.toml @@ -1,6 +1,6 @@ [package] name = "mqtt5" -version = "0.41.0" +version = "0.41.1" edition.workspace = true rust-version.workspace = true authors.workspace = true diff --git a/crates/mqtt5/src/broker/client_handler/lifecycle.rs b/crates/mqtt5/src/broker/client_handler/lifecycle.rs index e7b24eb6..9de25343 100644 --- a/crates/mqtt5/src/broker/client_handler/lifecycle.rs +++ b/crates/mqtt5/src/broker/client_handler/lifecycle.rs @@ -4,6 +4,7 @@ use crate::packet::publish::PublishPacket; use crate::protocol::v5::reason_codes::ReasonCode; use crate::time::Duration; use std::sync::Arc; +use tokio::sync::oneshot; use tracing::{debug, warn}; use super::ClientHandler; @@ -29,77 +30,82 @@ impl ClientHandler { self.write_to_client(crate::packet::Packet::PingResp).await } - pub(super) async fn publish_will_message(&self, client_id: &str) { - if let Some(ref session) = self.session { - if let Some(ref will) = session.will_message { - debug!("Publishing will message for client {}", client_id); - - let mut publish = - PublishPacket::new(will.topic.clone(), will.payload.clone(), will.qos); - publish.retain = will.retain; - - will.properties - .apply_to_publish_properties(&mut publish.properties); - publish.properties.inject_sender(self.user_id.as_deref()); - publish.properties.inject_client_id(Some(client_id)); - - if let Some(delay) = session.will_delay_interval { - debug!("Using will delay from session: {} seconds", delay); - if delay > 0 { - debug!("Spawning task to publish will after {} seconds", delay); - let router = Arc::clone(&self.router); - let auth_provider = Arc::clone(&self.auth_provider); - let user_id = self.user_id.clone(); - let publish_clone = publish.clone(); - let client_id_clone = client_id.to_string(); - let skip_bridges = self.skip_bridge_forwarding; - tokio::spawn(async move { - debug!( - "Task started: waiting {} seconds before publishing will for {}", - delay, client_id_clone - ); - tokio::time::sleep(Duration::from_secs(u64::from(delay))).await; - - let authorized = auth_provider - .authorize_publish( - &client_id_clone, - user_id.as_deref(), - &publish_clone.topic_name, - ) - .await; - if !authorized { - warn!( - "Delayed will for {} denied for topic {}", - client_id_clone, publish_clone.topic_name - ); - return; - } - - debug!( - "Task completed: publishing delayed will message for {}", - client_id_clone - ); - if skip_bridges { - router.route_message_local_only(&publish_clone, None).await; - } else { - router.route_message(&publish_clone, None).await; - } - }); - debug!("Spawned delayed will task for {}", client_id); - } else { - debug!("Publishing will immediately (delay = 0)"); - if self.authorize_will(client_id, &publish).await { - self.route_publish(&publish, None).await; - } - } - } else { - debug!("Publishing will immediately (no delay specified)"); - if self.authorize_will(client_id, &publish).await { - self.route_publish(&publish, None).await; - } - } + pub(super) async fn publish_will_message( + &self, + client_id: &str, + session_taken_over: bool, + armed_will: Option>, + ) { + let Some(session) = self.session.as_ref() else { + return; + }; + let (Some(will), Some(delay)) = (&session.will_message, session.will_publish_delay()) + else { + return; + }; + + let mut publish = PublishPacket::new(will.topic.clone(), will.payload.clone(), will.qos); + publish.retain = will.retain; + will.properties + .apply_to_publish_properties(&mut publish.properties); + publish.properties.inject_sender(self.user_id.as_deref()); + publish.properties.inject_client_id(Some(client_id)); + + if delay == 0 { + debug!(client_id, "Publishing will immediately"); + if self.authorize_will(client_id, &publish).await { + self.route_publish(&publish, None).await; + } + if !session_taken_over { + self.router.clear_stored_will(client_id).await; } + return; } + + let Some(cancelled) = armed_will else { + debug!( + client_id, + "Delayed will dropped: a new connection for the client id was opened" + ); + return; + }; + + debug!(client_id, delay, "Scheduling delayed will"); + let router = Arc::clone(&self.router); + let auth_provider = Arc::clone(&self.auth_provider); + let user_id = self.user_id.clone(); + let client_id = client_id.to_string(); + let generation = self.generation; + let skip_bridges = self.skip_bridge_forwarding; + tokio::spawn(async move { + tokio::select! { + () = tokio::time::sleep(Duration::from_secs(u64::from(delay))) => {} + _ = cancelled => { + debug!(client_id, "Delayed will cancelled by a new connection"); + return; + } + } + if !router.claim_will(&client_id, generation).await { + debug!(client_id, "Delayed will cancelled by a new connection"); + return; + } + let authorized = auth_provider + .authorize_publish(&client_id, user_id.as_deref(), &publish.topic_name) + .await; + if !authorized { + warn!( + "Delayed will for {client_id} denied for topic {}", + publish.topic_name + ); + return; + } + debug!(client_id, "Publishing delayed will"); + if skip_bridges { + router.route_message_local_only(&publish, None).await; + } else { + router.route_message(&publish, None).await; + } + }); } async fn authorize_will(&self, client_id: &str, publish: &PublishPacket) -> bool { diff --git a/crates/mqtt5/src/broker/client_handler/mod.rs b/crates/mqtt5/src/broker/client_handler/mod.rs index 3cbdb165..9feee776 100644 --- a/crates/mqtt5/src/broker/client_handler/mod.rs +++ b/crates/mqtt5/src/broker/client_handler/mod.rs @@ -360,20 +360,17 @@ impl ClientHandler { Err(e) => (Err(e), LoopExit::Closed), }; - let taken_over = if let LoopExit::TakenOver(notice) = exit { + let (taken_over, armed_will) = if let LoopExit::TakenOver(notice) = exit { self.hand_off(&queue, notice).await; self.release_router_entry(&client_id).await; - true + (true, None) } else { - // Move this connection's unfinished deliveries back to (or off) the queue BEFORE - // releasing the router entry, so a reconnect that races the release still sees this - // entry, is handed a notice, and waits for the hand-off instead of binding onto - // half-torn-down state and re-delivering. if self.session_preserved() { self.requeue_unsent(&queue).await; } else { self.drop_unsent().await; } + let armed_will = self.arm_delayed_will(&client_id).await; match self.release_router_entry(&client_id).await { Release::Owned => { queue.finish_drain(); @@ -382,29 +379,29 @@ impl ClientHandler { } else { queue.clear(None); } - false + (false, armed_will) } Release::Displaced => { - // A successor registered during the requeue above. Complete its hand-off - // protocol: this connection's messages are already back on the queue, so - // just release the successor and let its guard balance the count. if let Ok(notice) = disconnect_rx.try_recv() { queue.finish_drain(); let TakeoverNotice { released, guard, .. } = notice; drop(guard); - let _ = released.send(()); + if released.send(()).is_err() { + debug!("New session handler went away before the hand-off completed"); + } queue.notify(); } else { queue.finish_drain(); } - true + (true, armed_will) } } }; - self.handle_disconnect_cleanup(&client_id, taken_over).await; + self.handle_disconnect_cleanup(&client_id, taken_over, armed_will) + .await; info!("Client {} disconnected", client_id); @@ -567,7 +564,23 @@ impl ClientHandler { } } - async fn handle_disconnect_cleanup(&mut self, client_id: &str, session_taken_over: bool) { + async fn arm_delayed_will(&self, client_id: &str) -> Option> { + if self.normal_disconnect { + return None; + } + let delay = self.session.as_ref()?.will_publish_delay()?; + if delay == 0 { + return None; + } + self.router.arm_will(client_id, self.generation).await + } + + async fn handle_disconnect_cleanup( + &mut self, + client_id: &str, + session_taken_over: bool, + armed_will: Option>, + ) { #[cfg(feature = "opentelemetry")] { use tracing::Instrument; @@ -575,16 +588,21 @@ impl ClientHandler { "mqtt.disconnect", mqtt.client_id = %client_id, ); - self.handle_disconnect_cleanup_inner(client_id, session_taken_over) + self.handle_disconnect_cleanup_inner(client_id, session_taken_over, armed_will) .instrument(span) .await; } #[cfg(not(feature = "opentelemetry"))] - self.handle_disconnect_cleanup_inner(client_id, session_taken_over) + self.handle_disconnect_cleanup_inner(client_id, session_taken_over, armed_will) .await; } - async fn handle_disconnect_cleanup_inner(&mut self, client_id: &str, session_taken_over: bool) { + async fn handle_disconnect_cleanup_inner( + &mut self, + client_id: &str, + session_taken_over: bool, + armed_will: Option>, + ) { self.resource_monitor .unregister_connection(client_id, self.client_addr.ip()) .await; @@ -608,16 +626,21 @@ impl ClientHandler { mqtt.client_id = %client_id, mqtt.topic = %will.topic, ); - self.publish_will_message(client_id).instrument(span).await; + self.publish_will_message(client_id, session_taken_over, armed_will) + .instrument(span) + .await; } else { - self.publish_will_message(client_id).await; + self.publish_will_message(client_id, session_taken_over, armed_will) + .await; } } else { - self.publish_will_message(client_id).await; + self.publish_will_message(client_id, session_taken_over, armed_will) + .await; } } #[cfg(not(feature = "opentelemetry"))] - self.publish_will_message(client_id).await; + self.publish_will_message(client_id, session_taken_over, armed_will) + .await; } self.fire_disconnect_event(client_id).await; @@ -644,6 +667,10 @@ impl ClientHandler { match storage.get_session(client_id).await { Ok(Some(mut stored_session)) => { stored_session.touch(); + if self.normal_disconnect { + stored_session.will_message = None; + stored_session.will_delay_interval = None; + } if let Err(e) = storage.store_session(stored_session).await { warn!("Failed to store session for {client_id}: {e}"); } diff --git a/crates/mqtt5/src/broker/router.rs b/crates/mqtt5/src/broker/router.rs index 3ab74b4f..4667d466 100644 --- a/crates/mqtt5/src/broker/router.rs +++ b/crates/mqtt5/src/broker/router.rs @@ -17,7 +17,7 @@ use std::sync::Arc; use std::sync::Weak; use tokio::sync::{mpsc, oneshot, RwLock}; use tokio::time::{Duration, Instant}; -use tracing::{debug, error, info, trace}; +use tracing::{debug, error, info, trace, warn}; /// Upper bound on how long one publish may wait for slow subscribers' delivery channels. pub const ROUTE_BUDGET_MAX: Duration = Duration::from_secs(2); @@ -180,6 +180,12 @@ pub struct MessageRouter { max_outbound_rate: AtomicU32, fallback_queues: QueueRegistry, next_generation: std::sync::atomic::AtomicU64, + pending_wills: parking_lot::Mutex>, +} + +struct PendingWill { + generation: u64, + cancel: oneshot::Sender<()>, } /// Information about a connected client @@ -342,6 +348,7 @@ impl MessageRouter { max_outbound_rate: AtomicU32::new(0), fallback_queues: QueueRegistry::new(QueueLimits::default(), None), next_generation: std::sync::atomic::AtomicU64::new(0), + pending_wills: parking_lot::Mutex::new(HashMap::new()), } } @@ -375,6 +382,7 @@ impl MessageRouter { max_outbound_rate: AtomicU32::new(0), fallback_queues: QueueRegistry::new(QueueLimits::default(), None), next_generation: std::sync::atomic::AtomicU64::new(0), + pending_wills: parking_lot::Mutex::new(HashMap::new()), } } @@ -500,6 +508,16 @@ impl MessageRouter { .next_generation .fetch_add(1, std::sync::atomic::Ordering::Relaxed) + 1; + if let Some(pending) = self.pending_wills.lock().remove(&client_id) { + debug!( + client_id = %client_id, + armed_by = pending.generation, + "New connection cancelled pending delayed will" + ); + if pending.cancel.send(()).is_err() { + debug!(client_id = %client_id, "Delayed will timer already gone"); + } + } let released = match clients.remove(&client_id) { Some(old_client) => { info!("Client ID takeover: {}", client_id); @@ -571,6 +589,65 @@ impl MessageRouter { Release::Owned } + pub async fn arm_will( + &self, + client_id: &str, + generation: u64, + ) -> Option> { + let clients = self.clients.read().await; + if clients + .get(client_id) + .is_none_or(|info| info.generation != generation) + { + return None; + } + let (cancel, cancelled) = oneshot::channel(); + self.pending_wills + .lock() + .insert(client_id.to_string(), PendingWill { generation, cancel }); + drop(clients); + Some(cancelled) + } + + pub async fn owns_client(&self, client_id: &str, generation: u64) -> bool { + self.clients + .read() + .await + .get(client_id) + .is_some_and(|info| info.generation == generation) + } + + pub async fn claim_will(&self, client_id: &str, generation: u64) -> bool { + let claimed = { + let mut pending = self.pending_wills.lock(); + let armed_here = pending + .get(client_id) + .is_some_and(|will| will.generation == generation); + armed_here && pending.remove(client_id).is_some() + }; + if claimed { + self.clear_stored_will(client_id).await; + } + claimed + } + + pub async fn clear_stored_will(&self, client_id: &str) { + let Some(storage) = &self.storage else { + return; + }; + match storage.get_session(client_id).await { + Ok(Some(mut session)) if session.will_message.is_some() => { + session.will_message = None; + session.will_delay_interval = None; + if let Err(e) = storage.store_session(session).await { + warn!("Failed to clear stored will for {client_id}: {e}"); + } + } + Ok(_) => {} + Err(e) => warn!("Failed to load session for {client_id} to clear its will: {e}"), + } + } + pub async fn is_connected(&self, client_id: &str) -> bool { self.clients.read().await.contains_key(client_id) } @@ -2338,4 +2415,90 @@ mod tests { assert_eq!(routable.target_flow, Some(5)); assert!(rx.try_recv().is_err()); } + + async fn register(router: &MessageRouter, client_id: &str, lanes: &TestLanes) -> u64 { + let (dtx, _drx) = tokio::sync::oneshot::channel(); + router + .register_client( + client_id.to_string(), + lanes.lanes(), + router.queue_handle(client_id), + dtx, + ) + .await + .generation + } + + #[tokio::test] + async fn armed_will_is_claimed_exactly_once() { + let router = MessageRouter::new(); + let lanes = TestLanes::new(10); + let generation = register(&router, "w1", &lanes).await; + + let cancelled = router.arm_will("w1", generation).await; + assert!(cancelled.is_some()); + router.release_client("w1", generation, true).await; + + assert!(router.claim_will("w1", generation).await); + assert!(!router.claim_will("w1", generation).await); + assert!(router.pending_wills.lock().is_empty()); + } + + #[tokio::test] + async fn new_connection_cancels_armed_will() { + let router = MessageRouter::new(); + let lanes = TestLanes::new(10); + let generation = register(&router, "w2", &lanes).await; + let cancelled = router + .arm_will("w2", generation) + .await + .expect("owner can arm its will"); + router.release_client("w2", generation, true).await; + + register(&router, "w2", &lanes).await; + + assert!(cancelled.await.is_ok(), "the timer is woken by the cancel"); + assert!(!router.claim_will("w2", generation).await); + assert!(router.pending_wills.lock().is_empty()); + } + + #[tokio::test] + async fn displaced_connection_cannot_arm_will() { + let router = MessageRouter::new(); + let lanes = TestLanes::new(10); + let displaced = register(&router, "w3", &lanes).await; + register(&router, "w3", &lanes).await; + + assert!(router.arm_will("w3", displaced).await.is_none()); + assert!(router.pending_wills.lock().is_empty()); + } + + #[tokio::test] + async fn claimed_will_is_removed_from_stored_session() { + let storage = Arc::new(DynamicStorage::Memory( + crate::broker::storage::MemoryBackend::new(), + )); + let router = MessageRouter::with_storage(Arc::clone(&storage)); + let will = crate::types::WillMessage::new("will/w4", "offline"); + storage + .store_session(crate::broker::storage::ClientSession::new_with_will( + "w4", + true, + Some(60), + Some(will), + )) + .await + .unwrap(); + let lanes = TestLanes::new(10); + let generation = register(&router, "w4", &lanes).await; + let armed = router.arm_will("w4", generation).await; + assert!(armed.is_some()); + router.release_client("w4", generation, true).await; + + assert!(router.claim_will("w4", generation).await); + + let stored = storage.get_session("w4").await.unwrap().unwrap(); + assert!(stored.will_message.is_none()); + assert!(stored.will_delay_interval.is_none()); + } } diff --git a/crates/mqtt5/src/broker/storage/mod.rs b/crates/mqtt5/src/broker/storage/mod.rs index fc0fc5bd..85f022f9 100644 --- a/crates/mqtt5/src/broker/storage/mod.rs +++ b/crates/mqtt5/src/broker/storage/mod.rs @@ -884,6 +884,13 @@ impl ClientSession { false } } + + #[must_use] + pub fn will_publish_delay(&self) -> Option { + self.will_message.as_ref()?; + let will_delay = self.will_delay_interval.unwrap_or(0); + Some(will_delay.min(self.expiry_interval.unwrap_or(u32::MAX))) + } } impl QueuedMessage { diff --git a/crates/mqtt5/src/broker/storage/tests.rs b/crates/mqtt5/src/broker/storage/tests.rs index b6c5885f..80f86fc4 100644 --- a/crates/mqtt5/src/broker/storage/tests.rs +++ b/crates/mqtt5/src/broker/storage/tests.rs @@ -714,3 +714,32 @@ async fn test_dynamic_storage_inflight() { let msgs = dynamic.get_inflight_messages("client1").await.unwrap(); assert!(msgs.is_empty()); } + +#[test] +fn will_publish_delay_is_bounded_by_session_expiry() { + let delayed_will = |delay: Option| { + let mut will = crate::types::WillMessage::new("will/t", "gone"); + will.properties.will_delay_interval = delay; + will + }; + + let no_will = ClientSession::new_with_will("c", true, Some(60), None); + assert_eq!(no_will.will_publish_delay(), None); + + let undelayed = ClientSession::new_with_will("c", true, Some(60), Some(delayed_will(None))); + assert_eq!(undelayed.will_publish_delay(), Some(0)); + + let delayed = ClientSession::new_with_will("c", true, Some(60), Some(delayed_will(Some(5)))); + assert_eq!(delayed.will_publish_delay(), Some(5)); + + let session_ends = + ClientSession::new_with_will("c", false, Some(0), Some(delayed_will(Some(5)))); + assert_eq!(session_ends.will_publish_delay(), Some(0)); + + let short_session = + ClientSession::new_with_will("c", true, Some(2), Some(delayed_will(Some(10)))); + assert_eq!(short_session.will_publish_delay(), Some(2)); + + let never_expires = ClientSession::new_with_will("c", true, None, Some(delayed_will(Some(10)))); + assert_eq!(never_expires.will_publish_delay(), Some(10)); +} diff --git a/crates/mqtt5/tests/will_delay.rs b/crates/mqtt5/tests/will_delay.rs new file mode 100644 index 00000000..bcd161f7 --- /dev/null +++ b/crates/mqtt5/tests/will_delay.rs @@ -0,0 +1,390 @@ +#![cfg(feature = "broker")] +mod common; + +use common::{MessageCollector, TestBroker}; +use mqtt5::time::Duration; +use mqtt5::MqttClient; +use mqtt5_protocol::packet::connect::ConnectPacket; +use mqtt5_protocol::packet::disconnect::DisconnectPacket; +use mqtt5_protocol::packet::MqttPacket; +use mqtt5_protocol::protocol::v5::reason_codes::ReasonCode; +use mqtt5_protocol::types::{ConnectOptions, WillMessage}; +use tokio::io::{AsyncReadExt, AsyncWriteExt}; +use tokio::net::TcpStream; +use tokio::time::{sleep, Instant}; + +const SESSION_EXPIRY: u32 = 60; + +fn will_topic(client_id: &str) -> String { + format!("will/{client_id}") +} + +fn delayed_will(client_id: &str, delay: u32) -> WillMessage { + let mut will = WillMessage::new(will_topic(client_id), "offline"); + will.properties.will_delay_interval = Some(delay); + will +} + +async fn raw_connect( + broker: &TestBroker, + client_id: &str, + clean_start: bool, + session_expiry: u32, + will: Option, +) -> TcpStream { + let addr = broker.address().trim_start_matches("mqtt://"); + let mut stream = TcpStream::connect(addr).await.expect("connect tcp"); + let options = ConnectOptions::new(client_id) + .with_clean_start(clean_start) + .with_session_expiry_interval(session_expiry); + let options = match will { + Some(will) => options.with_will(will), + None => options, + }; + let mut buf = Vec::new(); + ConnectPacket::new(options) + .encode(&mut buf) + .expect("encode CONNECT"); + stream.write_all(&buf).await.expect("write CONNECT"); + + let mut connack = [0u8; 64]; + let read = tokio::time::timeout(Duration::from_secs(5), stream.read(&mut connack)) + .await + .expect("CONNACK timed out") + .expect("read CONNACK"); + assert!(read >= 4, "broker closed before CONNACK"); + assert_eq!(connack[0], 0x20, "expected CONNACK"); + assert_eq!(connack[3], 0x00, "CONNACK reason must be Success"); + stream +} + +async fn send_disconnect(mut stream: TcpStream, reason: ReasonCode) { + let mut buf = Vec::new(); + DisconnectPacket::new(reason) + .encode(&mut buf) + .expect("encode DISCONNECT"); + stream.write_all(&buf).await.expect("write DISCONNECT"); + stream.flush().await.expect("flush DISCONNECT"); + let mut drain = [0u8; 16]; + let closed = tokio::time::timeout(Duration::from_secs(5), stream.read(&mut drain)).await; + assert!( + matches!(closed, Ok(Ok(0))), + "broker must close after DISCONNECT" + ); +} + +async fn watch_will(broker: &TestBroker, client_id: &str) -> (MqttClient, MessageCollector) { + let watcher = MqttClient::new(format!("{client_id}-watcher")); + watcher + .connect(broker.address()) + .await + .expect("watcher connect"); + let collector = MessageCollector::new(); + watcher + .subscribe(&will_topic(client_id), collector.callback()) + .await + .expect("watcher subscribe"); + sleep(Duration::from_millis(100)).await; + (watcher, collector) +} + +async fn assert_no_will_until(collector: &MessageCollector, deadline: Instant, context: &str) { + sleep(deadline.saturating_duration_since(Instant::now())).await; + assert_eq!(collector.count().await, 0, "{context}"); +} + +async fn reconnect_within_delay_cancels_will(clean_start: bool) { + let broker = TestBroker::start().await; + let client_id = format!("wd-reconnect-{clean_start}"); + let (watcher, collector) = watch_will(&broker, &client_id).await; + + let first = raw_connect( + &broker, + &client_id, + true, + SESSION_EXPIRY, + Some(delayed_will(&client_id, 2)), + ) + .await; + let dropped_at = Instant::now(); + drop(first); + sleep(Duration::from_millis(300)).await; + + let second = raw_connect(&broker, &client_id, clean_start, SESSION_EXPIRY, None).await; + + assert_no_will_until( + &collector, + dropped_at + Duration::from_secs(4), + "a reconnect within the Will Delay Interval must cancel the Will", + ) + .await; + + send_disconnect(second, ReasonCode::Success).await; + watcher.disconnect().await.expect("watcher disconnect"); +} + +#[tokio::test] +async fn resume_within_delay_cancels_will() { + Box::pin(reconnect_within_delay_cancels_will(false)).await; +} + +#[tokio::test] +async fn clean_start_within_delay_cancels_will() { + Box::pin(reconnect_within_delay_cancels_will(true)).await; +} + +#[tokio::test] +async fn will_published_after_delay_without_reconnect() { + let broker = TestBroker::start().await; + let client_id = "wd-no-reconnect"; + let (watcher, collector) = watch_will(&broker, client_id).await; + + let conn = raw_connect( + &broker, + client_id, + true, + SESSION_EXPIRY, + Some(delayed_will(client_id, 2)), + ) + .await; + let dropped_at = Instant::now(); + drop(conn); + + assert_no_will_until( + &collector, + dropped_at + Duration::from_millis(1500), + "the Will must not be published before the Will Delay Interval", + ) + .await; + assert!( + collector.wait_for_messages(1, Duration::from_secs(3)).await, + "the Will must be published once the Will Delay Interval elapses" + ); + sleep(Duration::from_millis(500)).await; + assert_eq!(collector.count().await, 1, "the Will is published once"); + + watcher.disconnect().await.expect("watcher disconnect"); +} + +#[tokio::test] +async fn session_expiry_zero_publishes_will_immediately() { + let broker = TestBroker::start().await; + let client_id = "wd-expiry-zero"; + let (watcher, collector) = watch_will(&broker, client_id).await; + + let conn = raw_connect( + &broker, + client_id, + true, + 0, + Some(delayed_will(client_id, 5)), + ) + .await; + drop(conn); + + assert!( + collector + .wait_for_messages(1, Duration::from_millis(1500)) + .await, + "the Session ends at disconnect, so the Will must be published without waiting for the delay" + ); + + watcher.disconnect().await.expect("watcher disconnect"); +} + +#[tokio::test] +async fn session_expiry_shorter_than_delay_publishes_at_session_end() { + let broker = TestBroker::start().await; + let client_id = "wd-expiry-short"; + let (watcher, collector) = watch_will(&broker, client_id).await; + + let conn = raw_connect( + &broker, + client_id, + true, + 2, + Some(delayed_will(client_id, 10)), + ) + .await; + let dropped_at = Instant::now(); + drop(conn); + + assert_no_will_until( + &collector, + dropped_at + Duration::from_millis(1500), + "the Will must not be published before the Session ends", + ) + .await; + assert!( + collector.wait_for_messages(1, Duration::from_secs(3)).await, + "the Will must be published when the Session expires, before the Will Delay Interval" + ); + + watcher.disconnect().await.expect("watcher disconnect"); +} + +#[tokio::test] +async fn reconnect_then_drop_again_publishes_one_will_after_second_delay() { + let broker = TestBroker::start().await; + let client_id = "wd-redrop"; + let (watcher, collector) = watch_will(&broker, client_id).await; + + let first = raw_connect( + &broker, + client_id, + true, + SESSION_EXPIRY, + Some(delayed_will(client_id, 2)), + ) + .await; + let first_dropped_at = Instant::now(); + drop(first); + sleep(Duration::from_millis(300)).await; + + let second = raw_connect( + &broker, + client_id, + false, + SESSION_EXPIRY, + Some(delayed_will(client_id, 2)), + ) + .await; + sleep(Duration::from_millis(500)).await; + let second_dropped_at = Instant::now(); + drop(second); + + assert_no_will_until( + &collector, + first_dropped_at + Duration::from_millis(2400), + "the first connection's Will was cancelled by the reconnect", + ) + .await; + assert_no_will_until( + &collector, + second_dropped_at + Duration::from_millis(1500), + "the second Will must wait for its own delay", + ) + .await; + assert!( + collector.wait_for_messages(1, Duration::from_secs(3)).await, + "the second connection's Will must be published after its delay" + ); + sleep(Duration::from_millis(700)).await; + assert_eq!(collector.count().await, 1, "exactly one Will is published"); + + watcher.disconnect().await.expect("watcher disconnect"); +} + +#[tokio::test] +async fn normal_disconnect_discards_will() { + let broker = TestBroker::start().await; + let client_id = "wd-normal"; + let (watcher, collector) = watch_will(&broker, client_id).await; + + let conn = raw_connect( + &broker, + client_id, + true, + SESSION_EXPIRY, + Some(delayed_will(client_id, 1)), + ) + .await; + send_disconnect(conn, ReasonCode::Success).await; + + assert_no_will_until( + &collector, + Instant::now() + Duration::from_millis(2500), + "DISCONNECT 0x00 must delete the Will", + ) + .await; + + watcher.disconnect().await.expect("watcher disconnect"); +} + +#[tokio::test] +async fn disconnect_with_will_message_follows_delay() { + let broker = TestBroker::start().await; + let client_id = "wd-disconnect-with-will"; + let (watcher, collector) = watch_will(&broker, client_id).await; + + let conn = raw_connect( + &broker, + client_id, + true, + SESSION_EXPIRY, + Some(delayed_will(client_id, 2)), + ) + .await; + let disconnected_at = Instant::now(); + send_disconnect(conn, ReasonCode::DisconnectWithWillMessage).await; + + assert_no_will_until( + &collector, + disconnected_at + Duration::from_millis(1500), + "DISCONNECT 0x04 must still honour the Will Delay Interval", + ) + .await; + assert!( + collector.wait_for_messages(1, Duration::from_secs(3)).await, + "DISCONNECT 0x04 must publish the Will after the delay" + ); + + watcher.disconnect().await.expect("watcher disconnect"); +} + +#[tokio::test] +async fn takeover_cancels_delayed_will() { + let broker = TestBroker::start().await; + let client_id = "wd-takeover-delayed"; + let (watcher, collector) = watch_will(&broker, client_id).await; + + let first = raw_connect( + &broker, + client_id, + true, + SESSION_EXPIRY, + Some(delayed_will(client_id, 2)), + ) + .await; + let taken_over_at = Instant::now(); + let second = raw_connect(&broker, client_id, false, SESSION_EXPIRY, None).await; + drop(first); + + assert_no_will_until( + &collector, + taken_over_at + Duration::from_secs(4), + "the taking-over connection opened before the Will Delay Interval elapsed", + ) + .await; + + send_disconnect(second, ReasonCode::Success).await; + watcher.disconnect().await.expect("watcher disconnect"); +} + +#[tokio::test] +async fn takeover_publishes_undelayed_will() { + let broker = TestBroker::start().await; + let client_id = "wd-takeover-immediate"; + let (watcher, collector) = watch_will(&broker, client_id).await; + + let first = raw_connect( + &broker, + client_id, + true, + SESSION_EXPIRY, + Some(delayed_will(client_id, 0)), + ) + .await; + let second = raw_connect(&broker, client_id, false, SESSION_EXPIRY, None).await; + + assert!( + collector + .wait_for_messages(1, Duration::from_millis(1500)) + .await, + "the displaced connection's Will without a delay must be published on takeover" + ); + + drop(first); + send_disconnect(second, ReasonCode::Success).await; + watcher.disconnect().await.expect("watcher disconnect"); +} From 0503dbc419f7661318182ce607120ab4b4e62b09 Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?Fabr=C3=ADcio=20Bracht?= Date: Sun, 27 Sep 2026 16:15:55 -0300 Subject: [PATCH 2/2] harden session log, batch subscribe writes, fix enhanced auth replies --- ARCHITECTURE.md | 2 +- CHANGELOG.md | 108 +- WASM_USAGE.md | 2 +- crates/mqtt5-conformance/CONFORMANCE_DIARY.md | 16 + crates/mqtt5-conformance/conformance.toml | 6 +- .../section3_connect_extended.rs | 73 + .../conformance_tests/section3_disconnect.rs | 135 +- crates/mqtt5-wasm/Cargo.toml | 4 +- crates/mqtt5-wasm/src/broker.rs | 419 ++++- .../mqtt5-wasm/src/client_handler/connect.rs | 225 +-- .../src/client_handler/lifecycle.rs | 145 +- crates/mqtt5-wasm/src/client_handler/mod.rs | 164 +- .../src/client_handler/subscribe.rs | 94 +- crates/mqtt5-wasm/src/config.rs | 4 + crates/mqtt5-wasm/tests/broker_will_delay.rs | 512 +++++- crates/mqtt5/Cargo.toml | 2 +- .../mqtt5/src/broker/client_handler/auth.rs | 57 +- .../src/broker/client_handler/connect.rs | 446 +++-- .../src/broker/client_handler/lifecycle.rs | 105 +- crates/mqtt5/src/broker/client_handler/mod.rs | 315 ++-- .../src/broker/client_handler/subscribe.rs | 314 ++-- crates/mqtt5/src/broker/config/mod.rs | 7 +- crates/mqtt5/src/broker/mod.rs | 3 + crates/mqtt5/src/broker/router.rs | 705 ++++++-- crates/mqtt5/src/broker/server.rs | 46 +- .../mqtt5/src/broker/session_fault_tests.rs | 470 +++++ crates/mqtt5/src/broker/session_slot.rs | 121 ++ .../mqtt5/src/broker/storage/file_backend.rs | 365 ++-- .../src/broker/storage/memory_backend.rs | 91 +- crates/mqtt5/src/broker/storage/mod.rs | 199 ++- .../mqtt5/src/broker/storage/session_log.rs | 1532 +++++++++++++++++ crates/mqtt5/src/broker/storage/tests.rs | 97 +- crates/mqtt5/src/types.rs | 10 + crates/mqtt5/tests/cli_e2e.rs | 3 +- crates/mqtt5/tests/cli_features.rs | 14 +- crates/mqtt5/tests/common/cli_helpers.rs | 5 +- crates/mqtt5/tests/enhanced_auth.rs | 273 ++- .../mqtt5/tests/integration_mqtt5_features.rs | 1 + crates/mqtt5/tests/persistence.rs | 55 +- crates/mqtt5/tests/session_crash.rs | 386 +++++ crates/mqtt5/tests/session_lifecycle.rs | 557 ++++++ crates/mqtt5/tests/session_ownership.rs | 950 ++++++++++ .../tests/subscription_options_persistence.rs | 50 +- crates/mqtt5/tests/will_delay.rs | 416 ++++- crates/mqttv5-cli/CLI_USAGE.md | 16 +- crates/mqttv5-cli/Cargo.toml | 4 +- crates/mqttv5-cli/src/commands/broker_cmd.rs | 13 +- crates/mqttv5-cli/src/commands/client_args.rs | 49 +- crates/mqttv5-cli/src/commands/pub_cmd.rs | 2 +- crates/mqttv5-cli/src/commands/sub_cmd.rs | 2 +- specs/tla/session-ownership/README.md | 506 ++++++ .../session-ownership/SessionOwnership.cfg | 33 + .../session-ownership/SessionOwnership.tla | 818 +++++++++ .../SessionOwnership_ABL_claim.cfg | 33 + .../SessionOwnership_ABL_disc.cfg | 33 + .../SessionOwnership_ABL_rebuild.cfg | 33 + .../SessionOwnership_ABL_release.cfg | 33 + .../SessionOwnership_ABL_release_core.cfg | 32 + .../SessionOwnership_ABL_stamp.cfg | 33 + .../SessionOwnership_ABL_sub.cfg | 33 + .../SessionOwnership_ABL_sub_core.cfg | 32 + .../SessionOwnership_ABL_sweep.cfg | 33 + .../SessionOwnership_ABL_wt_claim.cfg | 33 + ...essionOwnership_ABL_wt_claim_resurrect.cfg | 21 + .../SessionOwnership_ABL_wt_disc.cfg | 33 + .../SessionOwnership_ABL_wt_remove.cfg | 33 + .../SessionOwnership_ABL_wt_remove_core.cfg | 32 + .../SessionOwnership_ABL_wt_sub.cfg | 33 + .../SessionOwnership_CASSPLIT_agree.cfg | 21 + .../SessionOwnership_CASSPLIT_resurrect.cfg | 21 + .../SessionOwnership_CAS_agree.cfg | 21 + .../SessionOwnership_CAS_mirrors.cfg | 21 + .../SessionOwnership_CAS_rest.cfg | 31 + .../SessionOwnership_CUR_AckedSubsDurable.cfg | 21 + .../SessionOwnership_CUR_CleanStart.cfg | 21 + .../SessionOwnership_CUR_ExpiryExact.cfg | 21 + ...nOwnership_CUR_FailedHandshakeHarmless.cfg | 21 + ...sionOwnership_CUR_LiveOwnerSessionKept.cfg | 21 + ...SessionOwnership_CUR_NoLeakedConnected.cfg | 21 + .../SessionOwnership_CUR_NoResurrection.cfg | 21 + .../SessionOwnership_CUR_OwnershipAgree.cfg | 21 + .../SessionOwnership_CUR_Progress.cfg | 21 + ...ssionOwnership_CUR_QuiescentConsistent.cfg | 21 + .../SessionOwnership_CUR_RouterMirrors.cfg | 21 + .../SessionOwnership_CUR_SessionPresent.cfg | 21 + .../SessionOwnership_CUR_live.cfg | 18 + .../SessionOwnership_GC_ABL_ackearly.cfg | 35 + .../SessionOwnership_GC_ABL_ackearly_core.cfg | 33 + .../SessionOwnership_GC_ABL_noryw.cfg | 35 + .../SessionOwnership_GC_ABL_noryw_acked.cfg | 21 + .../SessionOwnership_GC_ABL_noryw_core.cfg | 33 + .../SessionOwnership_GC_ABL_noryw_subs.cfg | 21 + .../SessionOwnership_GC_ABL_perwriter.cfg | 35 + ...SessionOwnership_GC_ABL_perwriter_core.cfg | 33 + .../SessionOwnership_GC_perwriter_mono.cfg | 35 + .../SessionOwnership_GC_prefix.cfg | 35 + .../SessionOwnership_GC_prefix_boots2.cfg | 35 + .../SessionOwnership_GC_prefix_live.cfg | 18 + .../SessionOwnership_GC_whole.cfg | 35 + .../SessionOwnership_NEG_nosweepfair.cfg | 18 + ...essionOwnership_NEG_resumeafterrestart.cfg | 20 + .../SessionOwnership_NEG_resumingtakeover.cfg | 20 + .../SessionOwnership_NEG_timeneverends.cfg | 18 + .../SessionOwnership_boots2.cfg | 33 + .../SessionOwnership_live.cfg | 18 + 105 files changed, 11170 insertions(+), 1173 deletions(-) create mode 100644 crates/mqtt5/src/broker/session_fault_tests.rs create mode 100644 crates/mqtt5/src/broker/session_slot.rs create mode 100644 crates/mqtt5/src/broker/storage/session_log.rs create mode 100644 crates/mqtt5/tests/session_crash.rs create mode 100644 crates/mqtt5/tests/session_lifecycle.rs create mode 100644 crates/mqtt5/tests/session_ownership.rs create mode 100644 specs/tla/session-ownership/README.md create mode 100644 specs/tla/session-ownership/SessionOwnership.cfg create mode 100644 specs/tla/session-ownership/SessionOwnership.tla create mode 100644 specs/tla/session-ownership/SessionOwnership_ABL_claim.cfg create mode 100644 specs/tla/session-ownership/SessionOwnership_ABL_disc.cfg create mode 100644 specs/tla/session-ownership/SessionOwnership_ABL_rebuild.cfg create mode 100644 specs/tla/session-ownership/SessionOwnership_ABL_release.cfg create mode 100644 specs/tla/session-ownership/SessionOwnership_ABL_release_core.cfg create mode 100644 specs/tla/session-ownership/SessionOwnership_ABL_stamp.cfg create mode 100644 specs/tla/session-ownership/SessionOwnership_ABL_sub.cfg create mode 100644 specs/tla/session-ownership/SessionOwnership_ABL_sub_core.cfg create mode 100644 specs/tla/session-ownership/SessionOwnership_ABL_sweep.cfg create mode 100644 specs/tla/session-ownership/SessionOwnership_ABL_wt_claim.cfg create mode 100644 specs/tla/session-ownership/SessionOwnership_ABL_wt_claim_resurrect.cfg create mode 100644 specs/tla/session-ownership/SessionOwnership_ABL_wt_disc.cfg create mode 100644 specs/tla/session-ownership/SessionOwnership_ABL_wt_remove.cfg create mode 100644 specs/tla/session-ownership/SessionOwnership_ABL_wt_remove_core.cfg create mode 100644 specs/tla/session-ownership/SessionOwnership_ABL_wt_sub.cfg create mode 100644 specs/tla/session-ownership/SessionOwnership_CASSPLIT_agree.cfg create mode 100644 specs/tla/session-ownership/SessionOwnership_CASSPLIT_resurrect.cfg create mode 100644 specs/tla/session-ownership/SessionOwnership_CAS_agree.cfg create mode 100644 specs/tla/session-ownership/SessionOwnership_CAS_mirrors.cfg create mode 100644 specs/tla/session-ownership/SessionOwnership_CAS_rest.cfg create mode 100644 specs/tla/session-ownership/SessionOwnership_CUR_AckedSubsDurable.cfg create mode 100644 specs/tla/session-ownership/SessionOwnership_CUR_CleanStart.cfg create mode 100644 specs/tla/session-ownership/SessionOwnership_CUR_ExpiryExact.cfg create mode 100644 specs/tla/session-ownership/SessionOwnership_CUR_FailedHandshakeHarmless.cfg create mode 100644 specs/tla/session-ownership/SessionOwnership_CUR_LiveOwnerSessionKept.cfg create mode 100644 specs/tla/session-ownership/SessionOwnership_CUR_NoLeakedConnected.cfg create mode 100644 specs/tla/session-ownership/SessionOwnership_CUR_NoResurrection.cfg create mode 100644 specs/tla/session-ownership/SessionOwnership_CUR_OwnershipAgree.cfg create mode 100644 specs/tla/session-ownership/SessionOwnership_CUR_Progress.cfg create mode 100644 specs/tla/session-ownership/SessionOwnership_CUR_QuiescentConsistent.cfg create mode 100644 specs/tla/session-ownership/SessionOwnership_CUR_RouterMirrors.cfg create mode 100644 specs/tla/session-ownership/SessionOwnership_CUR_SessionPresent.cfg create mode 100644 specs/tla/session-ownership/SessionOwnership_CUR_live.cfg create mode 100644 specs/tla/session-ownership/SessionOwnership_GC_ABL_ackearly.cfg create mode 100644 specs/tla/session-ownership/SessionOwnership_GC_ABL_ackearly_core.cfg create mode 100644 specs/tla/session-ownership/SessionOwnership_GC_ABL_noryw.cfg create mode 100644 specs/tla/session-ownership/SessionOwnership_GC_ABL_noryw_acked.cfg create mode 100644 specs/tla/session-ownership/SessionOwnership_GC_ABL_noryw_core.cfg create mode 100644 specs/tla/session-ownership/SessionOwnership_GC_ABL_noryw_subs.cfg create mode 100644 specs/tla/session-ownership/SessionOwnership_GC_ABL_perwriter.cfg create mode 100644 specs/tla/session-ownership/SessionOwnership_GC_ABL_perwriter_core.cfg create mode 100644 specs/tla/session-ownership/SessionOwnership_GC_perwriter_mono.cfg create mode 100644 specs/tla/session-ownership/SessionOwnership_GC_prefix.cfg create mode 100644 specs/tla/session-ownership/SessionOwnership_GC_prefix_boots2.cfg create mode 100644 specs/tla/session-ownership/SessionOwnership_GC_prefix_live.cfg create mode 100644 specs/tla/session-ownership/SessionOwnership_GC_whole.cfg create mode 100644 specs/tla/session-ownership/SessionOwnership_NEG_nosweepfair.cfg create mode 100644 specs/tla/session-ownership/SessionOwnership_NEG_resumeafterrestart.cfg create mode 100644 specs/tla/session-ownership/SessionOwnership_NEG_resumingtakeover.cfg create mode 100644 specs/tla/session-ownership/SessionOwnership_NEG_timeneverends.cfg create mode 100644 specs/tla/session-ownership/SessionOwnership_boots2.cfg create mode 100644 specs/tla/session-ownership/SessionOwnership_live.cfg diff --git a/ARCHITECTURE.md b/ARCHITECTURE.md index 13801168..dcd99d37 100644 --- a/ARCHITECTURE.md +++ b/ARCHITECTURE.md @@ -127,7 +127,7 @@ The client validates acknowledgment reason codes: PUBACK (QoS 1) returns `MqttEr Each accepted connection spawns a **ClientHandler** that directly reads and writes packets, manages client session state, and handles the MQTT protocol. Outbound control packets (CONNACK, SUBACK, PUBACK, PUBREC, AUTH, DISCONNECT) are written through a choke point that honours the client's advertised Maximum Packet Size: if a packet would exceed it, the Reason String is omitted and the packet re-encoded, and only if it still does not fit is it discarded, so the broker never exceeds the client's limit (`[MQTT-3.2.2-19]`, `[MQTT-3.1.2-24]`). A CONNECT advertising a Maximum Packet Size of 0 is rejected as a Protocol Error. The **MessageRouter** performs subscription matching using MQTT-compliant topic wildcards (`+`, `#`), protects system topics (`$SYS/#` excluded from `#`), and supports shared subscriptions (`$share/group/topic`). -The **storage backend** persists sessions, retained messages, queued messages, and inflight messages. The file-based backend uses percent-encoded filenames with atomic writes and fsync; the memory backend stores everything in-process. +The **storage backend** persists sessions, retained messages, queued messages, and inflight messages. The memory backend stores everything in-process. The file-based backend stores retained, queued and inflight messages in percent-encoded files with atomic writes, and all sessions in one append-only log, `sessions/sessions.log`, with group commit: concurrent session writes are visible to later readers at once, one flush appends and fsyncs every pending write, and each write is acknowledged only after the flush that covers it and every earlier write. Each record is one line carrying a CRC-32 of its body and an explicit record type (put or remove), so a damaged record is detected rather than misread. A failed flush rejects every pending write and restores the last durable state in memory, and the bytes it may have left are truncated away before the failure is reported; if that truncation fails, the log is rewritten, and no later write is appended until the repair succeeds. The log is compacted into a new file (fsynced, renamed, directory fsynced) when it is larger than both 1 MB and twice its live size. At startup an incomplete last line from an unfinished write is discarded; a damaged complete record is skipped and replay continues, since every record carries the whole session, and the original log is first copied to `sessions.log.corrupt-`. Startup then rewrites the log; if that fails (full disk, read-only directory), the broker serves the replayed sessions and refuses session writes until a later write succeeds in rewriting it. Storage version 1 directories (one file per session) are migrated on open. On Windows, which has no directory fsync, appends are durable through `FlushFileBuffers`, but the rename that installs a compacted log relies on NTFS metadata journaling and may be lost by a power failure right after a compaction. ### Broker Data Flow diff --git a/CHANGELOG.md b/CHANGELOG.md index 4075884c..dd763b6a 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -5,26 +5,118 @@ All notable changes to this project will be documented in this file. The format is based on [Keep a Changelog](https://keepachangelog.com/en/1.0.0/), and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0.html). -## [mqtt5 0.41.1] - 2026-09-24 +## [mqtt5 0.42.0] - 2026-09-25 + +### Breaking + +- **An absent Session Expiry Interval in an MQTT v5 CONNECT now means 0**, as ยง3.1.2.11.2 requires: the session ends when the network connection closes. Before, the broker kept such sessions, with their subscriptions and queued messages, forever. Clients that resume with Clean Start 0 must set a non-zero Session Expiry Interval (`ConnectOptions::with_session_expiry_interval`). For MQTT v3.1.1, CleanSession=0 keeps the session and CleanSession=1 now ends it at disconnect. Reported in #171. +- **`BrokerConfig::session_expiry_interval` is now the maximum Session Expiry the broker grants.** A client that asks for more, or an MQTT v3.1.1 CleanSession=0 session, gets the maximum. MQTT v5 clients are told the granted value in the CONNACK Session Expiry Interval (ยง3.2.2.3.2); an MQTT v3.1.1 CONNACK carries no properties. A Session Expiry sent on DISCONNECT is capped the same way. The setting was previously unused. Its default is `u32::MAX` seconds (no limit), so nothing is capped unless you set it. +- **With persistence disabled, every connection now has an in-memory session that ends at disconnect.** Will Messages are now published; before, no Will was ever sent without persistence. Because the session ends at disconnect: + - a delayed Will is published at disconnect; + - CONNACK returns Session Expiry Interval 0 when the client asked for more; + - Session Present is always 0; + - a connection that takes over a ClientID never inherits the previous connection's subscriptions. +- **Session ownership is now one protocol**, verified with TLA+ in `specs/tla/session-ownership/`. + - **The session slot.** Each ClientID has one. Claiming a session (before CONNACK), applying a SUBSCRIBE, UNSUBSCRIBE or QUIC flow close, releasing a session, and sweeping it each run as one critical section over the router and storage. + - **What the claim does.** It decides Session Present, writes the session, and sets the router's subscriptions for the ClientID to exactly the session's. A displaced connection changes nothing. + - **What this fixes:** + - a SUBSCRIBE racing a takeover left a session that never expired; + - a failed or aborted handshake overwrote or orphaned a live connection's session; + - a resumed session kept routing a subscription that is no longer authorized (#173); + - a takeover or clean start inherited subscriptions; + - a new connection could claim a session while it was still being released; + - the sweep could delete a session a new connection had just claimed. +- **The file storage backend group-commits session writes.** Sessions live in one append-only log, `sessions/sessions.log`. + - A write is visible to later readers at once. + - One flush appends and fsyncs every pending write. + - CONNACK, SUBACK/UNSUBACK and the end of DISCONNECT processing wait for the flush that covers their write and every earlier one, so a crash can no longer lose an acknowledged session or bring back a discarded one. + - A failed flush rejects all pending writes and restores the last durable state. + - The log is compacted when it is larger than both 1 MB and twice its live size. + - A SUBSCRIBE or UNSUBSCRIBE is stored in one write however many filters it carries, so it waits for one flush. + - 1000 concurrent CONNECTs are acknowledged in about 70 ms, where a per-write fsync under a global lock needed about 9 s. + - The ordering rule (no acknowledgement before the flush covering the write and every earlier one) is model-checked in `specs/tla/session-ownership/`. The failure path (rejecting pending writes, restoring the durable state, repairing the log) is not in the model; it is covered by tests. +- **The session log is robust against damage and failed writes.** + - Every record carries a CRC-32 and an explicit type (put or remove), so a damaged record is detected instead of being misread, and a damaged update can no longer be replayed as a removal. + - A damaged record is skipped and replay continues with the records after it. Before, replay stopped at the first damaged record and startup compaction deleted every later session. When anything other than an incomplete last line is discarded, the original log is kept as `sessions/sessions.log.corrupt-` and the broker logs an error. + - A write whose flush failed is truncated out of the log before the failure is reported, so a restart cannot bring back a write the client was told had failed. If the truncation fails, the log is rewritten; until that succeeds, every session write fails. + - If the log cannot be rewritten at startup (full disk, read-only directory), the broker starts with the replayed sessions and refuses session writes until a rewrite succeeds, instead of refusing to start. + - Leftover temporary files from an interrupted write are removed at startup. A legacy session file that cannot be read or parsed during migration is kept as `.corrupt-`, and an unreadable log is moved to `sessions.log.unreadable-`; neither overwrites an earlier copy. + - The `.storage_version` file is replaced atomically (temp file, fsync, rename, directory fsync). +- **Storage format version 2.** Version 1 directories are migrated on open; older brokers refuse a version 2 directory. **Back up the storage directory before upgrading.** Rolling back means restoring that backup, because older builds refuse version 2. A broker that finds a storage version newer than it supports refuses to start and says to run the version that wrote the directory or restore the backup; it no longer suggests a `mqttv5 storage backup` command, which does not exist. On Windows, the compaction rename relies on NTFS metadata journaling. `FileBackend::flush_sessions`, `FileBackend::start_flush_task`, `DynamicStorage::flush_sessions` and the 5 s flush task are removed. `MessageRouter::recover_sessions` takes the maximum Session Expiry. +- **Sessions survive a broker restart as the protocol requires.** Reported in #172. + - The disconnect time is persisted (`ClientSession::disconnected_at`), so an ended session expires at its disconnect time plus its expiry. + - A session that was connected when the broker stopped is treated as disconnected at boot. + - Expiry-0 sessions are dropped at startup. + - The router's subscriptions are rebuilt from the persisted sessions, so an offline persistent session receives and queues messages before its client reconnects. + - Session files from earlier versions still load. +- **`ClientSession` has new public fields.** Code that builds it with a struct literal must add them: + - `connection_token`, which now holds the router generation of the owning connection; + - `connected`; + - `disconnected_at`. + Router generations are now epoch-qualified, so they don't repeat across restarts. +- **Expired sessions are removed only by the sweep.** `StorageBackend::get_session` and `StorageBackend::cleanup_expired` no longer remove them; `MessageRouter::sweep_sessions` does, and it replaces `MessageRouter::cleanup_stale_subscriptions`. +- `MessageRouter::release_client` reports a missing registration as `Release::Displaced`. `MessageRouter::arm_will` and `MessageRouter::clear_stored_will` identify the connection by its generation. ### Fixed -- **A delayed Will Message is cancelled when a new connection for the same ClientID opens before the Will Delay Interval elapses** (`[MQTT-3.1.3-9]`, `[MQTT-3.1.2-8]`). This applies whether the new connection resumes the session, starts clean, or takes over a live connection. Before, the broker always published the Will after the delay, so a client that dropped and reconnected got a spurious Will. Reported in #154. -- **The Will is published when the Will Delay Interval elapses or the session ends, whichever comes first.** A Session Expiry Interval of 0 publishes it immediately at disconnect, and a Session Expiry shorter than the Will Delay publishes it when the session ends. Before, the full delay was always waited out. +- **A delayed Will Message is cancelled when a new connection for the same ClientID opens before the Will Delay Interval elapses** (`[MQTT-3.1.3-9]`, `[MQTT-3.1.2-8]`). This holds whether the new connection resumes the session, starts clean, or takes over a live connection. Before, the broker always published the Will after the delay. The pending Will is cancelled when the new session is claimed, before CONNACK. Reported in #154. +- **The Will is published when the Will Delay Interval elapses or the session ends, whichever comes first.** A Session Expiry Interval of 0 publishes it at disconnect. +- **Session Expiry is counted from disconnect**, as ยง3.1.2.11.2 requires. A connected client's session never expires. +- **A resumed session takes its Session Expiry Interval from the resuming CONNECT** instead of keeping the value from the connection that created it. A live expiry-0 session that is taken over is never resumed. +- **A Session Expiry Interval sent on DISCONNECT is applied** (ยง3.14.2.2.2). A non-zero value after a CONNECT value of 0 is a Protocol Error: the broker sends DISCONNECT 0x82, closes the connection, publishes the Will, and ends the session. Reported in #171. +- **The CONNACK sent after enhanced authentication carries the same properties as a plain CONNACK**, including a capped Session Expiry Interval. Enhanced authentication now completes before the connection starts its normal packet loop. - **A published Will, or one deleted by DISCONNECT 0x00, is removed from the stored session state** (`[MQTT-3.1.2-10]`). -- The conformance test for `[MQTT-3.1.3-9]` passed vacuously: it stopped watching before the Will Delay elapsed. It now waits past the delay, and a new test checks that the Will is published when nobody reconnects. +- **The connect timeout no longer cancels a claim in progress.** Before, it could leave a router owner with no connection and a session stuck connected. +- **A claim whose write fails gets CONNACK 0x88** and no longer disconnects the live owner. A SUBSCRIBE or UNSUBSCRIBE whose write fails is not acknowledged, and its route change is undone; the broker sends DISCONNECT 0x80 and closes the connection, and the Will is published. A session left connected by a failed release write is ended by the sweep. +- **The DISCONNECT Session Expiry is stored when the DISCONNECT is processed.** +- **Multi-step enhanced authentication applies the Will QoS/Retain checks and the client Receive Maximum**, like a plain CONNECT. +- **A failed multi-step enhanced authentication is refused with CONNACK** carrying the provider's reason code (0x87 Not authorized), or with DISCONNECT during re-authentication. Before, the broker sent an AUTH packet with that reason code, which AUTH may not carry (`[MQTT-3.15.2-1]`). +- **The server's final Authentication Data reaches the client.** It is sent in the CONNACK after enhanced authentication, and in the AUTH 0x00 that ends a re-authentication. Before, it was dropped, so a SCRAM client never received the server signature. +- **Startup recovery is concurrent and caps stored expiry to the configured maximum.** Before, 500 sessions left connected by a crash delayed the first CONNACK by over 5 s. Measured on a Mac release build, the first CONNACK now comes about 44 ms after start with 500 such sessions, and about 0.5 s with 5000. An unreadable session log or legacy session file no longer blocks startup or a ClientID. +- **The connected-clients statistic no longer underflows when a handshake fails.** In debug builds, the underflow made a later handler panic. Reported in #175. +- The conformance test for `[MQTT-3.1.3-9]` passed vacuously. It now waits past the delay, with a positive control. New conformance tests cover an absent Session Expiry and a Session Expiry sent on DISCONNECT. ### Added -- `MessageRouter::arm_will`, `MessageRouter::claim_will`, `MessageRouter::clear_stored_will`, `MessageRouter::owns_client` and `ClientSession::will_publish_delay`, which back the pending-Will tracking. +- `broker::session_slot` (`SessionSlots`, `SessionSlotGuard`), `MessageRouter::lock_session`, `MessageRouter::set_client_subscriptions`, `MessageRouter::sweep_sessions`, `MessageRouter::recover_sessions`, `MessageRouter::is_current_owner`, `MessageRouter::stored_subscription_request`, `MessageRouter::claim_will`, `MessageRouter::cancel_pending_will`, `MessageRouter::allocate_generation`, `MessageRouter::register_session_as` and `MqttBroker::storage`. +- `StorageBackend::update_session`, `StorageBackend::remove_owned_session`, `StorageBackend::remove_expired_session` and `StorageBackend::session_client_ids`, with default implementations. The first three are atomic in the memory and file backends. +- `ClientSession::expiry_from_connect`, `ClientSession::granted_expiry`, `ClientSession::will_publish_delay`, `ClientSession::mark_connected`, `ClientSession::mark_disconnected` and `broker::storage::unix_millis_now`. + +## [mqttv5-cli 0.29.0] - 2026-09-25 + +### Breaking + +- **The broker's file storage is migrated to version 2 on first start**, and the migration is one-way: sessions move into `sessions/sessions.log`, and earlier mqttv5 versions refuse a migrated directory. **Back up the storage directory (`--storage-dir`, default `./mqtt_storage`) before upgrading.** Rolling back means stopping the broker and restoring that backup. Version 1 directories are migrated automatically. +- **`broker --session-expiry` is now an optional maximum Session Expiry the broker grants** (default: no limit). Before, it defaulted to 3600 and had no effect. `broker generate-config` no longer writes a `session_expiry_interval`; add one to set a maximum. +- **The broker ends a session at disconnect when an MQTT v5 client sends no Session Expiry Interval**, as mqtt5 0.42.0 does. Clients that resume with Clean Start 0 must send a non-zero Session Expiry Interval. + +### Changed + +- `--no-clean-start` without `--session-expiry` now sends a Session Expiry Interval of 1 hour, so the resumed session also survives this run. An explicit `--session-expiry`, including 0, still takes precedence. +- Requires mqtt5 0.42. + +## [mqtt5-wasm 2.1.0] - 2026-09-25 + +### Changed + +- **An absent Session Expiry Interval in an MQTT v5 CONNECT now means 0 in the in-browser broker**: the session ends when the connection closes. Clients that resume with Clean Start 0 must set `sessionExpiryInterval`. +- **`BrokerConfig.sessionExpiryIntervalSecs` is now the maximum Session Expiry the broker grants.** It applies to CONNECT and DISCONNECT values and to MQTT v3.1.1 persistent sessions. It is returned in CONNACK only when it caps an MQTT v5 client's value. Its default is 4294967295 (no limit). Before, the broker put 3600 in every CONNACK without applying it. +- **The in-browser broker uses the same session ownership protocol as mqtt5 0.42.0.** A clean start, or a takeover of a live expiry-0 session, never resumes or inherits the previous session's subscriptions (#174). SUBSCRIBE and UNSUBSCRIBE from a displaced connection change nothing. Expired sessions are swept periodically. +- Requires mqtt5 0.42. + +### Added -## [mqtt5-wasm 2.0.1] - 2026-09-24 +- `BrokerConfig.sessionSweepIntervalSecs` (default 3600 s). ### Fixed -- **The in-browser broker cancels a delayed Will when the client reconnects within the Will Delay Interval**, and publishes it no later than session end. This is the same fix as mqtt5 0.41.1. +- **The in-browser broker cancels a delayed Will when the client reconnects within the Will Delay Interval**, and publishes it no later than session end. This is the same fix as mqtt5 0.42.0. +- **Session Expiry is counted from disconnect.** A resumed session takes the resuming CONNECT's Session Expiry, and a Session Expiry sent on DISCONNECT is applied, as in mqtt5 0.42.0. A non-zero value after 0 is answered with DISCONNECT 0x82. +- **Will Delay Intervals longer than about 24.8 days no longer fire immediately or throw.** The timer now sleeps in chunks. - **The in-browser broker detects a client closing its MessagePort** (the port's `close` event) and treats it as an abnormal disconnect, so the Will is published. Before, a closed port went unnoticed until keep-alive expiry, and never with a keep-alive of 0, which also leaked the connection handler. Environments that don't raise `close` on MessagePort still rely on keep-alive expiry. -- **A published Will, or one deleted by DISCONNECT 0x00, is removed from the in-browser broker's stored session** (`[MQTT-3.1.2-10]`). +- **Enhanced authentication completes before the connection starts serving**, and its CONNACK carries the capped Session Expiry. +- **The claimed session is stored before the connection is registered**, so a failed write can't disconnect the live owner; the client gets CONNACK Server Unavailable. Failing to discard queued or inflight messages on Clean Start is only logged. +- **A published Will, or one deleted by DISCONNECT 0x00, is removed from the stored session** (`[MQTT-3.1.2-10]`). ## [mqtt5 0.41.0] - 2026-09-23 diff --git a/WASM_USAGE.md b/WASM_USAGE.md index 0270817a..514b2c86 100644 --- a/WASM_USAGE.md +++ b/WASM_USAGE.md @@ -550,7 +550,7 @@ The JavaScript class is exported as `BrokerConfig` (Rust type: `WasmBrokerConfig const config = new BrokerConfig(); config.maxClients = 1000; // default: 1000 -config.sessionExpiryIntervalSecs = 3600; // default: 3600 +config.sessionExpiryIntervalSecs = 3600; // maximum granted; default: 4294967295 (no limit) config.maxPacketSize = 268435456; // default: 268435456 (256MB) config.topicAliasMaximum = 65535; // default: 65535 config.retainAvailable = true; // default: true diff --git a/crates/mqtt5-conformance/CONFORMANCE_DIARY.md b/crates/mqtt5-conformance/CONFORMANCE_DIARY.md index 2fd5da10..56b79073 100644 --- a/crates/mqtt5-conformance/CONFORMANCE_DIARY.md +++ b/crates/mqtt5-conformance/CONFORMANCE_DIARY.md @@ -38,6 +38,22 @@ ## Diary Entries +### Absent Session Expiry now means 0, and DISCONNECT can change the Session Expiry (2026-09-24) + +**Trigger**: the quorum review of PR #170 and issue #171, which was folded into it. The broker stored an absent CONNECT Session Expiry Interval as "never expires", although ยง3.1.2.11.2 says an absent value is 0. A client that left the property out kept its session, subscriptions and queued messages forever. It also broke the Will timing from #154: with the Will bounded by session end, "never" meant the delay was always honoured instead of the Will going out at disconnect. The broker also ignored a Session Expiry Interval sent on DISCONNECT (ยง3.14.2.2.2), and a resumed session kept the Session Expiry of the connection that created it instead of taking the resuming CONNECT's value. + +**Fix**: the Session Expiry is worked out once, from the CONNECT, when a session is created or resumed. MQTT v5: the property's value, or 0 when absent. MQTT v3.1.1 has no property, so CleanSession=1 gives 0 and CleanSession=0 keeps the session with no expiry, as before. A Session Expiry on DISCONNECT replaces the stored value before the session-end and Will logic runs. If CONNECT had 0 and DISCONNECT sends a non-zero value, the server sends DISCONNECT 0x82 and closes. The spec says such a DISCONNECT is not valid, so it is not a normal disconnection: the Will is published, and the session still ends because its expiry stays 0. + +**New tests**, all failing on the tree before the fix: +- `absent_session_expiry_discards_session_at_disconnect` (`[MQTT-4.1.0-2]`): connect without the property, subscribe at QoS 1, disconnect, publish while offline, then reconnect with Clean Start 0. It expects Session Present 0 and no delivery of the offline message. +- `disconnect_session_expiry_zero_discards_session` (`[MQTT-4.1.0-2]`): CONNECT 300, DISCONNECT 0, then Session Present 0. +- `disconnect_session_expiry_extends_session` (`[MQTT-3.1.2-23]`): CONNECT 1, DISCONNECT 300, wait 2.5 s, then Session Present 1. +- `disconnect_session_expiry_after_zero_is_protocol_error` (`[MQTT-4.13.1-1]`): CONNECT 0, DISCONNECT 300, then DISCONNECT 0x82 and the connection closes. + +**IDs**: neither "absent means 0" nor the DISCONNECT 0 to non-zero rule has its own normative statement in `mqtt-v5.0-statement-texts.txt`; both are prose in ยง3.1.2.11.2 and ยง3.14.2.2.2. The tests are filed under the statements they exercise: session discard after the interval (MQTT-4.1.0-2), session storage when the interval is above 0 (MQTT-3.1.2-23), and closing the connection on a Protocol Error (MQTT-4.13.1-1). The manifest text for all three matches the statement file. + +**Knock-on**: tests elsewhere in the workspace had resumed sessions without setting a Session Expiry. Several of them only asserted inside `if session_present`, so after the fix they would have passed without checking anything. They now set an explicit expiry and assert Session Present. + ### Delayed Will was never cancelled, and the MQTT-3.1.3-9 test could not see it (2026-09-24) **Trigger**: issue #154. With a Will Delay Interval above zero, the broker spawned a detached task that slept for the delay and then published the Will unconditionally. A client that reconnected inside the delay still had its Will published, which violates `[MQTT-3.1.3-9]` and the "new Network Connection ... before the Will Delay Interval has elapsed" clause of `[MQTT-3.1.2-8]`. diff --git a/crates/mqtt5-conformance/conformance.toml b/crates/mqtt5-conformance/conformance.toml index 4bfb2c74..d1cc7908 100644 --- a/crates/mqtt5-conformance/conformance.toml +++ b/crates/mqtt5-conformance/conformance.toml @@ -408,7 +408,7 @@ level = "Must" applies_to = "Both" text = "The Client and Server MUST store the Session State after the Network Connection is closed if the Session Expiry Interval is greater than 0" status = "Tested" -test_names = ["session_stored_when_expiry_positive"] +test_names = ["session_stored_when_expiry_positive", "disconnect_session_expiry_extends_session"] [[sections."3.1".statements]] id = "MQTT-3.1.2-24" @@ -1795,7 +1795,7 @@ level = "Must" applies_to = "Server" text = "The Server MUST discard the Session State when the Network Connection is closed and the Session Expiry Interval has passed" status = "Tested" -test_names = ["session_discarded_after_expiry"] +test_names = ["session_discarded_after_expiry", "absent_session_expiry_discards_session_at_disconnect", "disconnect_session_expiry_zero_discards_session"] # =========================================================================== # Section 4.2 -- Network Connections @@ -2359,7 +2359,7 @@ level = "Must" applies_to = "Server" text = "When a Server detects a Malformed Packet or Protocol Error, and a Reason Code is given in the specification, it MUST close the Network Connection" status = "Tested" -test_names = ["malformed_packet_closes_connection"] +test_names = ["malformed_packet_closes_connection", "disconnect_session_expiry_after_zero_is_protocol_error"] [[sections."4.13".statements]] id = "MQTT-4.13.2-1" diff --git a/crates/mqtt5-conformance/src/conformance_tests/section3_connect_extended.rs b/crates/mqtt5-conformance/src/conformance_tests/section3_connect_extended.rs index 338cc22a..40013e9e 100644 --- a/crates/mqtt5-conformance/src/conformance_tests/section3_connect_extended.rs +++ b/crates/mqtt5-conformance/src/conformance_tests/section3_connect_extended.rs @@ -592,6 +592,79 @@ async fn session_discarded_after_expiry(sut: SutHandle) { client2.disconnect().await.expect("disconnect failed"); } +/// `[MQTT-4.1.0-2]` An absent Session Expiry Interval means 0 (ยง3.1.2.11.2), +/// so the Session ends when the Network Connection closes and the Server +/// discards its Session State: subscriptions are gone and messages published +/// while the client was offline are not delivered. +#[conformance_test( + ids = ["MQTT-4.1.0-2"], + requires = ["transport.tcp", "max_qos>=1"], +)] +async fn absent_session_expiry_discards_session_at_disconnect(sut: SutHandle) { + let client_id = unique_client_id("sess-absent"); + let topic = format!("sess/{client_id}"); + + let opts = ConnectOptions::new(&client_id).with_clean_start(true); + let client1 = TestClient::connect_with_options(&sut, opts) + .await + .expect("first connect failed"); + client1 + .subscribe( + &topic, + SubscribeOptions { + qos: mqtt5_protocol::QoS::AtLeastOnce, + ..SubscribeOptions::default() + }, + ) + .await + .expect("subscribe failed"); + client1.disconnect().await.expect("disconnect failed"); + tokio::time::sleep(Duration::from_millis(200)).await; + + let publisher = TestClient::connect_with_prefix(&sut, "sess-absent-pub") + .await + .unwrap(); + publisher + .publish_with_options( + &topic, + b"offline", + mqtt5_protocol::types::PublishOptions { + qos: mqtt5_protocol::QoS::AtLeastOnce, + ..Default::default() + }, + ) + .await + .expect("publish failed"); + + let mut raw = RawMqttClient::connect_tcp(sut.expect_tcp_addr()) + .await + .unwrap(); + let reconnect = ConnectOptions::new(&client_id) + .with_clean_start(false) + .with_session_expiry_interval(300); + let mut buf = bytes::BytesMut::new(); + mqtt5_protocol::packet::MqttPacket::encode( + &mqtt5_protocol::packet::connect::ConnectPacket::new(reconnect), + &mut buf, + ) + .expect("CONNECT encodes"); + raw.send_raw(&buf).await.unwrap(); + let connack = raw + .expect_connack_packet(TIMEOUT) + .await + .expect("Must receive CONNACK"); + assert!( + !connack.session_present, + "[MQTT-4.1.0-2] an absent Session Expiry Interval is 0, so no session survives the disconnect" + ); + assert!( + raw.expect_publish(Duration::from_secs(1)).await.is_none(), + "[MQTT-4.1.0-2] a message published while offline must not be delivered to a discarded session" + ); + + publisher.disconnect().await.expect("disconnect failed"); +} + /// `[MQTT-2.1.3-1]` Where a flag bit is marked as Reserved, it is reserved /// for future use and MUST be set to the value listed. #[conformance_test( diff --git a/crates/mqtt5-conformance/src/conformance_tests/section3_disconnect.rs b/crates/mqtt5-conformance/src/conformance_tests/section3_disconnect.rs index 3c4a3eb3..82742c6c 100644 --- a/crates/mqtt5-conformance/src/conformance_tests/section3_disconnect.rs +++ b/crates/mqtt5-conformance/src/conformance_tests/section3_disconnect.rs @@ -5,7 +5,11 @@ use crate::harness::unique_client_id; use crate::raw_client::{RawMqttClient, RawPacketBuilder}; use crate::sut::SutHandle; use crate::test_client::TestClient; -use mqtt5_protocol::types::SubscribeOptions; +use bytes::BytesMut; +use mqtt5_protocol::packet::disconnect::DisconnectPacket; +use mqtt5_protocol::packet::MqttPacket; +use mqtt5_protocol::protocol::v5::reason_codes::ReasonCode; +use mqtt5_protocol::types::{ConnectOptions, SubscribeOptions}; use std::time::Duration; const TIMEOUT: Duration = Duration::from_secs(10); @@ -266,3 +270,132 @@ async fn no_packets_after_client_disconnect(sut: SutHandle) { "[MQTT-3.14.4-1] no PINGRESP should be received after client sent DISCONNECT" ); } + +fn disconnect_with_session_expiry(session_expiry: u32) -> Vec { + let mut disconnect = DisconnectPacket::new(ReasonCode::Success); + disconnect + .properties + .set_session_expiry_interval(session_expiry); + let mut buf = BytesMut::new(); + disconnect.encode(&mut buf).expect("DISCONNECT encodes"); + buf.to_vec() +} + +async fn reconnect_session_present(sut: &SutHandle, client_id: &str) -> bool { + let opts = ConnectOptions::new(client_id) + .with_clean_start(false) + .with_session_expiry_interval(300); + let client = TestClient::connect_with_options(sut, opts) + .await + .expect("reconnect failed"); + let present = client.session_present(); + client.disconnect().await.expect("disconnect failed"); + present +} + +/// `[MQTT-4.1.0-2]` A Session Expiry Interval sent on DISCONNECT replaces the +/// one from CONNECT (ยง3.14.2.2.2). A value of 0 makes the session end when the +/// Network Connection closes, so the Server discards the Session State. +#[conformance_test( + ids = ["MQTT-4.1.0-2"], + requires = ["transport.tcp"], +)] +async fn disconnect_session_expiry_zero_discards_session(sut: SutHandle) { + let client_id = unique_client_id("disc-sei0"); + + let mut raw = RawMqttClient::connect_tcp(sut.expect_tcp_addr()) + .await + .unwrap(); + raw.send_raw(&RawPacketBuilder::connect_with_session_expiry( + &client_id, 300, + )) + .await + .unwrap(); + raw.expect_connack(TIMEOUT).await.expect("expected CONNACK"); + raw.send_raw(&disconnect_with_session_expiry(0)) + .await + .unwrap(); + assert!( + raw.expect_disconnect(TIMEOUT).await, + "server must close after DISCONNECT" + ); + tokio::time::sleep(Duration::from_millis(200)).await; + + assert!( + !reconnect_session_present(&sut, &client_id).await, + "[MQTT-4.1.0-2] Session Expiry 0 on DISCONNECT must end the session at disconnect" + ); +} + +/// `[MQTT-3.1.2-23]` The Server MUST store the Session State after the +/// Network Connection closes if the Session Expiry Interval is greater than 0. +/// A non-zero value sent on DISCONNECT replaces a shorter CONNECT value. +#[conformance_test( + ids = ["MQTT-3.1.2-23"], + requires = ["transport.tcp"], +)] +async fn disconnect_session_expiry_extends_session(sut: SutHandle) { + let client_id = unique_client_id("disc-seiext"); + + let mut raw = RawMqttClient::connect_tcp(sut.expect_tcp_addr()) + .await + .unwrap(); + raw.send_raw(&RawPacketBuilder::connect_with_session_expiry( + &client_id, 1, + )) + .await + .unwrap(); + raw.expect_connack(TIMEOUT).await.expect("expected CONNACK"); + raw.send_raw(&disconnect_with_session_expiry(300)) + .await + .unwrap(); + assert!( + raw.expect_disconnect(TIMEOUT).await, + "server must close after DISCONNECT" + ); + tokio::time::sleep(Duration::from_millis(2500)).await; + + assert!( + reconnect_session_present(&sut, &client_id).await, + "[MQTT-3.1.2-23] the DISCONNECT Session Expiry of 300s replaces the CONNECT value of 1s" + ); +} + +/// `[MQTT-4.13.1-1]` If CONNECT set a Session Expiry Interval of 0, a +/// non-zero value on DISCONNECT is a Protocol Error (ยง3.14.2.2.2): the Server +/// sends DISCONNECT 0x82 and MUST close the Network Connection. +#[conformance_test( + ids = ["MQTT-4.13.1-1"], + requires = ["transport.tcp"], +)] +async fn disconnect_session_expiry_after_zero_is_protocol_error(sut: SutHandle) { + let client_id = unique_client_id("disc-seierr"); + + let mut raw = RawMqttClient::connect_tcp(sut.expect_tcp_addr()) + .await + .unwrap(); + raw.send_raw(&RawPacketBuilder::connect_with_session_expiry( + &client_id, 0, + )) + .await + .unwrap(); + raw.expect_connack(TIMEOUT).await.expect("expected CONNACK"); + raw.send_raw(&disconnect_with_session_expiry(300)) + .await + .unwrap(); + + assert_eq!( + raw.expect_disconnect_packet(TIMEOUT).await, + Some(0x82), + "[MQTT-4.13.1-1] non-zero Session Expiry on DISCONNECT after 0 on CONNECT is a Protocol Error" + ); + assert!( + raw.expect_disconnect(TIMEOUT).await, + "[MQTT-4.13.1-1] Server must close the connection after the Protocol Error" + ); + tokio::time::sleep(Duration::from_millis(200)).await; + assert!( + !reconnect_session_present(&sut, &client_id).await, + "the session keeps its Session Expiry of 0 and ends at disconnect" + ); +} diff --git a/crates/mqtt5-wasm/Cargo.toml b/crates/mqtt5-wasm/Cargo.toml index 21013946..d646565a 100644 --- a/crates/mqtt5-wasm/Cargo.toml +++ b/crates/mqtt5-wasm/Cargo.toml @@ -1,6 +1,6 @@ [package] name = "mqtt5-wasm" -version = "2.0.1" +version = "2.1.0" edition.workspace = true rust-version.workspace = true authors.workspace = true @@ -28,7 +28,7 @@ codec = ["client", "dep:miniz_oxide"] [dependencies] mqtt5-protocol = "0.15.2" -mqtt5 = { version = "0.41", optional = true, default-features = false, features = [ +mqtt5 = { version = "0.42", optional = true, default-features = false, features = [ "tokio", ] } diff --git a/crates/mqtt5-wasm/src/broker.rs b/crates/mqtt5-wasm/src/broker.rs index 0fc956ec..913fccdc 100644 --- a/crates/mqtt5-wasm/src/broker.rs +++ b/crates/mqtt5-wasm/src/broker.rs @@ -50,6 +50,7 @@ struct ConfigHashFields { max_message_rate_per_client: u32, max_bandwidth_per_client: u64, load_balancer_backends: Vec, + session_sweep_interval_secs: u32, } #[wasm_bindgen(js_name = "BrokerConfig")] @@ -74,6 +75,7 @@ pub struct WasmBrokerConfig { max_message_rate_per_client: u32, max_bandwidth_per_client: u64, load_balancer_backends: Vec, + session_sweep_interval_secs: u32, } #[wasm_bindgen(js_class = "BrokerConfig")] @@ -83,7 +85,7 @@ impl WasmBrokerConfig { pub fn new() -> Self { Self { max_clients: 1000, - session_expiry_interval_secs: 3600, + session_expiry_interval_secs: u32::MAX, max_packet_size: 268_435_456, topic_alias_maximum: 65535, retain_available: true, @@ -101,6 +103,7 @@ impl WasmBrokerConfig { max_message_rate_per_client: 0, max_bandwidth_per_client: 0, load_balancer_backends: Vec::new(), + session_sweep_interval_secs: 3600, } } @@ -209,8 +212,13 @@ impl WasmBrokerConfig { self.load_balancer_backends.clear(); } + #[wasm_bindgen(setter, js_name = "sessionSweepIntervalSecs")] + pub fn set_session_sweep_interval_secs(&mut self, value: u32) { + self.session_sweep_interval_secs = value; + } + fn to_broker_config(&self) -> BrokerConfig { - BrokerConfig { + let mut config = BrokerConfig { max_clients: self.max_clients as usize, session_expiry_interval: Duration::from_secs(u64::from( self.session_expiry_interval_secs, @@ -245,7 +253,10 @@ impl WasmBrokerConfig { Some(LoadBalancerConfig::new(self.load_balancer_backends.clone())) }, ..Default::default() - } + }; + config.storage_config.cleanup_interval = + Duration::from_secs(u64::from(self.session_sweep_interval_secs.max(1))); + config } fn calculate_hash(&self) -> u64 { @@ -269,6 +280,7 @@ impl WasmBrokerConfig { max_message_rate_per_client: self.max_message_rate_per_client, max_bandwidth_per_client: self.max_bandwidth_per_client, load_balancer_backends: self.load_balancer_backends.clone(), + session_sweep_interval_secs: self.session_sweep_interval_secs, }; let mut hasher = DefaultHasher::new(); fields.hash(&mut hasher); @@ -292,6 +304,8 @@ fn resource_limits_from(config: &BrokerConfig) -> ResourceLimits { } } +const SESSION_SWEEP_INTERVAL_MS: u32 = 60_000; + #[wasm_bindgen(js_name = "Broker")] pub struct WasmBroker { config: Arc>, @@ -376,10 +390,38 @@ impl WasmBroker { }; broker.setup_bridge_callback(); + broker.spawn_session_sweep(); Ok(broker) } + fn spawn_session_sweep(&self) { + let router = Arc::downgrade(&self.router); + let storage = Arc::downgrade(&self.storage); + let interval = self + .config + .read() + .map_or(SESSION_SWEEP_INTERVAL_MS, |config| { + u32::try_from(config.storage_config.cleanup_interval.as_millis()) + .unwrap_or(SESSION_SWEEP_INTERVAL_MS) + .max(1) + }); + wasm_bindgen_futures::spawn_local(async move { + loop { + gloo_timers::future::TimeoutFuture::new(interval).await; + let (Some(router), Some(storage)) = (router.upgrade(), storage.upgrade()) else { + break; + }; + router.sweep_sessions().await; + if let Err(e) = + mqtt5::broker::storage::StorageBackend::cleanup_expired(&*storage).await + { + tracing::warn!("Storage cleanup error: {e}"); + } + } + }); + } + /// # Errors /// Returns an error if adding the user fails. #[wasm_bindgen(js_name = "addUser")] @@ -800,7 +842,7 @@ impl WasmBroker { web_sys::console::warn_1( &"Config read failed, using default session_expiry_interval".into(), ); - 3600 + u32::MAX }, |c| u64_to_u32_saturating(c.session_expiry_interval.as_secs()), ) @@ -810,15 +852,24 @@ impl WasmBroker { #[cfg(all(test, target_arch = "wasm32"))] mod tests { use super::{WasmBroker, WasmBrokerConfig}; + use crate::client_handler::WasmClientHandler; use bytes::BytesMut; + use mqtt5::broker::auth::{AuthProvider, AuthResult, EnhancedAuthResult}; use mqtt5::broker::storage::StorageBackend; + use mqtt5_protocol::error::Result; + use mqtt5_protocol::packet::auth::AuthPacket; use mqtt5_protocol::packet::connect::ConnectPacket; use mqtt5_protocol::packet::disconnect::DisconnectPacket; use mqtt5_protocol::packet::MqttPacket; use mqtt5_protocol::protocol::v5::reason_codes::ReasonCode; use mqtt5_protocol::types::{ConnectOptions, WillMessage}; + use mqtt5_protocol::QoS; use std::cell::RefCell; + use std::future::Future; + use std::net::SocketAddr; + use std::pin::Pin; use std::rc::Rc; + use std::sync::Arc; use wasm_bindgen::prelude::*; use wasm_bindgen::JsCast; use wasm_bindgen_test::wasm_bindgen_test; @@ -903,4 +954,364 @@ mod tests { assert!(!stored_will_present(&broker, "stored-will-published").await); } + + struct Client { + port: MessagePort, + inbox: Rc>>, + } + + impl Client { + fn open(port: MessagePort) -> Self { + let inbox = Rc::new(RefCell::new(Vec::new())); + let inbox_in = Rc::clone(&inbox); + let on_message = Closure::::new(move |event: MessageEvent| { + inbox_in + .borrow_mut() + .extend(js_sys::Uint8Array::new(&event.data()).to_vec()); + }); + port.add_event_listener_with_callback("message", on_message.as_ref().unchecked_ref()) + .unwrap(); + on_message.forget(); + port.start(); + Self { port, inbox } + } + + fn send(&self, packet: &impl MqttPacket) { + send(&self.port, packet); + } + + fn send_raw(&self, bytes: &[u8]) { + self.port + .post_message(&js_sys::Uint8Array::from(bytes).buffer()) + .unwrap(); + } + + async fn wait_for(&self, len: usize) -> Vec { + for _ in 0..300 { + if self.inbox.borrow().len() >= len { + break; + } + sleep(10).await; + } + self.inbox.borrow().clone() + } + + fn received(&self) -> Vec { + self.inbox.borrow().clone() + } + + fn clear(&self) { + self.inbox.borrow_mut().clear(); + } + } + + fn connect_options(client_id: &str, clean_start: bool, expiry: u32) -> ConnectOptions { + ConnectOptions::new(client_id) + .with_clean_start(clean_start) + .with_session_expiry_interval(expiry) + } + + fn start_connect(broker: &WasmBroker, options: ConnectOptions) -> Client { + let client = Client::open(broker.create_client_port().unwrap()); + client.send(&ConnectPacket::new(options)); + client + } + + async fn connected(broker: &WasmBroker, options: ConnectOptions) -> Client { + let client = start_connect(broker, options); + let connack = client.wait_for(4).await; + assert_eq!(connack.first(), Some(&0x20), "expected CONNACK"); + assert_eq!(connack[3], 0x00, "CONNACK must be Success"); + client.clear(); + client + } + + fn subscribe_bytes(topic: &str) -> Vec { + let topic_len = u8::try_from(topic.len()).unwrap(); + let mut bytes = vec![0x82, 6 + topic_len, 0x00, 0x01, 0x00, 0x00, topic_len]; + bytes.extend_from_slice(topic.as_bytes()); + bytes.push(0x01); + bytes + } + + fn unsubscribe_bytes(topic: &str) -> Vec { + let topic_len = u8::try_from(topic.len()).unwrap(); + let mut bytes = vec![0xA2, 5 + topic_len, 0x00, 0x02, 0x00, 0x00, topic_len]; + bytes.extend_from_slice(topic.as_bytes()); + bytes + } + + async fn subscribed(client: &Client, topic: &str) { + client.send_raw(&subscribe_bytes(topic)); + assert_eq!( + client.wait_for(1).await.first(), + Some(&0x90), + "expected SUBACK" + ); + client.clear(); + } + + #[wasm_bindgen_test] + async fn claim_waits_for_the_session_slot() { + let broker = broker(); + let slot = broker.router.lock_session("slot-claim").await; + let client = start_connect(&broker, connect_options("slot-claim", true, 60)); + sleep(200).await; + assert!( + client.received().is_empty(), + "CONNACK sent while the slot was held" + ); + assert!( + broker + .storage + .get_session("slot-claim") + .await + .unwrap() + .is_none(), + "the claim wrote the session without holding the slot" + ); + assert!( + !broker.router.is_connected("slot-claim").await, + "the claim registered without holding the slot" + ); + drop(slot); + assert_eq!(client.wait_for(4).await.first(), Some(&0x20)); + assert!(broker.router.is_connected("slot-claim").await); + } + + #[wasm_bindgen_test] + async fn displaced_subscribe_installs_no_route() { + let broker = broker(); + let first = connected(&broker, connect_options("dsub", true, 60)).await; + let slot = broker.router.lock_session("dsub").await; + let second = start_connect(&broker, connect_options("dsub", true, 60)); + sleep(50).await; + first.send_raw(&subscribe_bytes("dsub/t")); + sleep(50).await; + drop(slot); + assert_eq!(second.wait_for(4).await.first(), Some(&0x20)); + sleep(200).await; + assert!( + !broker.router.has_subscription("dsub", "dsub/t").await, + "a displaced connection's SUBSCRIBE installed a route for the successor" + ); + let stored = broker.storage.get_session("dsub").await.unwrap().unwrap(); + assert!(!stored.subscriptions.contains_key("dsub/t")); + } + + #[wasm_bindgen_test] + async fn displaced_unsubscribe_keeps_the_successors_route() { + let broker = broker(); + let first = connected(&broker, connect_options("dunsub", true, 60)).await; + subscribed(&first, "keep/x").await; + let slot = broker.router.lock_session("dunsub").await; + let second = start_connect(&broker, connect_options("dunsub", false, 60)); + sleep(50).await; + first.send_raw(&unsubscribe_bytes("keep/x")); + sleep(50).await; + drop(slot); + let connack = second.wait_for(4).await; + assert_eq!(connack.first(), Some(&0x20)); + assert_eq!(connack[2] & 1, 1, "session present"); + sleep(200).await; + assert!( + broker.router.has_subscription("dunsub", "keep/x").await, + "a displaced connection's UNSUBSCRIBE removed the successor's route" + ); + let stored = broker.storage.get_session("dunsub").await.unwrap().unwrap(); + assert!(stored.subscriptions.contains_key("keep/x")); + } + + #[wasm_bindgen_test] + async fn resume_drops_a_revoked_subscription_from_session_and_routes() { + let broker = broker(); + let first = connected(&broker, connect_options("revoked", true, 3600)).await; + subscribed(&first, "secret/x").await; + first.send(&DisconnectPacket::new(ReasonCode::Success)); + first.port.close(); + sleep(200).await; + assert!(broker.router.has_subscription("revoked", "secret/x").await); + + broker.set_acl_default_deny().await; + let second = start_connect(&broker, connect_options("revoked", false, 3600)); + let connack = second.wait_for(4).await; + assert_eq!(connack.first(), Some(&0x20)); + assert_eq!(connack[2] & 1, 1, "session present"); + assert!( + !broker.router.has_subscription("revoked", "secret/x").await, + "a subscription no longer authorized must not stay routed after resume" + ); + let stored = broker + .storage + .get_session("revoked") + .await + .unwrap() + .unwrap(); + assert!(!stored.subscriptions.contains_key("secret/x")); + } + + #[wasm_bindgen_test] + async fn sweep_removes_an_expired_offline_session() { + let mut config = WasmBrokerConfig::new(); + config.set_allow_anonymous(true); + config.set_session_sweep_interval_secs(1); + let broker = WasmBroker::with_config(config).unwrap(); + let client = connected(&broker, connect_options("swept", true, 1)).await; + subscribed(&client, "swept/t").await; + client.send(&DisconnectPacket::new(ReasonCode::Success)); + client.port.close(); + sleep(200).await; + assert!(broker.router.has_subscription("swept", "swept/t").await); + sleep(3500).await; + assert_eq!( + broker.router.subscription_count_for_client("swept").await, + 0, + "the periodic sweep must strip an expired session's routes" + ); + assert!(broker.storage.get_session("swept").await.unwrap().is_none()); + } + + #[wasm_bindgen_test] + async fn release_waits_for_the_session_slot() { + let broker = broker(); + let client = connected(&broker, connect_options("slot-release", true, 60)).await; + let slot = broker.router.lock_session("slot-release").await; + client.port.close(); + sleep(200).await; + assert!( + broker.router.is_connected("slot-release").await, + "the router entry was released without holding the slot" + ); + drop(slot); + sleep(200).await; + assert!(!broker.router.is_connected("slot-release").await); + } + + #[wasm_bindgen_test] + async fn disconnect_expiry_is_stored_before_the_release() { + let broker = broker(); + let client = connected(&broker, connect_options("disc-zero", true, 3600)).await; + let slot = broker.router.lock_session("disc-zero").await; + let mut disconnect = DisconnectPacket::new(ReasonCode::Success); + disconnect.properties.set_session_expiry_interval(0); + client.send(&disconnect); + sleep(100).await; + let next = broker.router.lock_session("disc-zero"); + futures::pin_mut!(next); + assert!(futures::poll!(next.as_mut()).is_pending()); + drop(slot); + let observed = next.await; + let owner_live = broker.router.is_connected("disc-zero").await; + let stored = broker + .storage + .get_session("disc-zero") + .await + .unwrap() + .map(|session| session.expiry_interval); + drop(observed); + assert!( + owner_live, + "the release ran before the DISCONNECT expiry reached the store" + ); + assert_eq!( + stored, + Some(Some(0)), + "between DISCONNECT and release a claim must already see expiry 0" + ); + let resumed = start_connect(&broker, connect_options("disc-zero", false, 3600)); + let connack = resumed.wait_for(4).await; + assert_eq!(connack[2] & 1, 0, "an ended session must not be resumed"); + } + + struct Challenge; + + impl AuthProvider for Challenge { + fn authenticate<'a>( + &'a self, + _connect: &'a ConnectPacket, + _client_addr: SocketAddr, + ) -> Pin> + Send + 'a>> { + Box::pin(async move { Ok(AuthResult::success()) }) + } + + fn authorize_publish<'a>( + &'a self, + _client_id: &str, + _user_id: Option<&'a str>, + _topic: &'a str, + ) -> Pin + Send + 'a>> { + Box::pin(async move { true }) + } + + fn authorize_subscribe<'a>( + &'a self, + _client_id: &str, + _user_id: Option<&'a str>, + _topic_filter: &'a str, + ) -> Pin + Send + 'a>> { + Box::pin(async move { true }) + } + + fn supports_enhanced_auth(&self) -> bool { + true + } + + fn authenticate_enhanced<'a>( + &'a self, + auth_method: &'a str, + auth_data: Option<&'a [u8]>, + _client_id: &'a str, + ) -> Pin> + Send + 'a>> { + let method = auth_method.to_string(); + Box::pin(async move { + Ok(match auth_data { + None => EnhancedAuthResult::continue_auth(method, Some(b"c".to_vec())), + Some(_) => EnhancedAuthResult::success(method), + }) + }) + } + } + + #[wasm_bindgen_test] + async fn multi_step_enhanced_auth_rejects_an_unsupported_will_qos() { + let mut config = WasmBrokerConfig::new(); + config.set_maximum_qos(1); + let broker = WasmBroker::with_config(config).unwrap(); + let channel = web_sys::MessageChannel::new().unwrap(); + WasmClientHandler::new( + channel.port2(), + Arc::clone(&broker.config), + Arc::clone(&broker.router), + Arc::new(Challenge), + Arc::clone(&broker.storage), + Arc::clone(&broker.stats), + Arc::clone(&broker.resource_monitor), + broker.event_callbacks.clone(), + ); + let client = Client::open(channel.port1()); + let will = WillMessage::new("will/scram", "offline").with_qos(QoS::ExactlyOnce); + client.send(&ConnectPacket::new( + ConnectOptions::new("scram") + .with_authentication_method("X") + .with_will(will), + )); + assert_eq!( + client.wait_for(1).await.first(), + Some(&0xF0), + "expected AUTH" + ); + client.clear(); + client.send( + &AuthPacket::continue_authentication("X".to_string(), Some(b"r".to_vec())).unwrap(), + ); + let connack = client.wait_for(4).await; + assert_eq!(connack.first(), Some(&0x20), "expected CONNACK"); + assert_eq!( + connack[3], + u8::from(ReasonCode::QoSNotSupported), + "a Will QoS above the server maximum must be rejected after multi-step auth too" + ); + assert!(!broker.router.is_connected("scram").await); + assert!(broker.storage.get_session("scram").await.unwrap().is_none()); + } } diff --git a/crates/mqtt5-wasm/src/client_handler/connect.rs b/crates/mqtt5-wasm/src/client_handler/connect.rs index 87750a9c..3194d0ec 100644 --- a/crates/mqtt5-wasm/src/client_handler/connect.rs +++ b/crates/mqtt5-wasm/src/client_handler/connect.rs @@ -1,5 +1,5 @@ use mqtt5::broker::auth::{EnhancedAuthResult, EnhancedAuthStatus}; -use mqtt5::broker::router::SubscriptionRequest; +use mqtt5::broker::router::DeliveryLanes; use mqtt5::broker::storage::{ClientSession, StorageBackend}; use mqtt5_protocol::error::{MqttError, Result}; use mqtt5_protocol::packet::auth::AuthPacket; @@ -7,7 +7,6 @@ use mqtt5_protocol::packet::connack::ConnAckPacket; use mqtt5_protocol::packet::connect::ConnectPacket; use mqtt5_protocol::packet::Packet; use mqtt5_protocol::protocol::v5::reason_codes::ReasonCode; -use mqtt5_protocol::types::ProtocolVersion; use mqtt5_protocol::{u64_to_u32_saturating, usize_to_u32_saturating}; use std::net::{IpAddr, Ipv4Addr, SocketAddr}; use tracing::{debug, error, info, warn}; @@ -25,14 +24,24 @@ impl WasmClientHandler { ) -> Result<()> { let packet = read_packet(reader).await?; - if let Packet::Connect(connect) = packet { - self.handle_connect(*connect, writer).await - } else { + let Packet::Connect(connect) = packet else { error!("First packet must be CONNECT"); - Err(MqttError::ProtocolError( + return Err(MqttError::ProtocolError( "First packet must be CONNECT".to_string(), - )) + )); + }; + self.handle_connect(*connect, writer).await?; + while self.pending_connect.is_some() { + match read_packet(reader).await? { + Packet::Auth(auth) => self.handle_auth(auth, writer).await?, + _ => { + return Err(MqttError::ProtocolError( + "Only AUTH may follow CONNECT before CONNACK".to_string(), + )) + } + } } + Ok(()) } pub(super) async fn handle_connect( @@ -147,113 +156,130 @@ impl WasmClientHandler { Ok(()) } + pub(super) fn maximum_session_expiry(&self) -> u32 { + self.config.read().map_or(u32::MAX, |config| { + u64_to_u32_saturating(config.session_expiry_interval.as_secs()) + }) + } + pub(super) async fn handle_session( &mut self, connect: &ConnectPacket, writer: &mut WasmWriter, ) -> Result { - let client_id = &connect.client_id; - let session_expiry = connect.properties.get_session_expiry_interval(); - - if connect.clean_start { - self.storage.remove_session(client_id).await.ok(); - self.storage.remove_queued_messages(client_id).await.ok(); - self.storage - .remove_all_inflight_messages(client_id) - .await - .ok(); - - let mut session = ClientSession::new_with_will( - client_id.clone(), - session_expiry != Some(0), - session_expiry, - connect.will.clone(), - ); - session.user_id.clone_from(&self.user_id); - self.storage.store_session(session.clone()).await.ok(); - self.session = Some(session); - Ok(false) - } else { - match self.storage.get_session(client_id).await { - Ok(Some(session)) => { - self.restore_existing_session(connect, session, writer) - .await - } - Ok(None) => { - let mut session = ClientSession::new_with_will( - client_id.clone(), - session_expiry != Some(0), - session_expiry, - connect.will.clone(), - ); - session.user_id.clone_from(&self.user_id); - self.storage.store_session(session.clone()).await.ok(); - self.session = Some(session); - Ok(false) - } - Err(e) => Err(e), + let client_id = connect.client_id.clone(); + let requested = ClientSession::expiry_from_connect(connect); + let granted = ClientSession::granted_expiry(requested, self.maximum_session_expiry()); + self.connect_session_expiry = requested; + self.advertised_session_expiry = (requested != Some(granted)).then_some(granted); + + let slot = self.router.lock_session(&client_id).await; + let resumable = self + .storage + .get_session(&client_id) + .await? + .filter(|session| !connect.clean_start && session.expiry_interval != Some(0)); + if let Some(session) = resumable.as_ref() { + if session.user_id.as_deref() != self.user_id.as_deref() { + drop(slot); + warn!( + client_id = %client_id, + session_user = ?session.user_id, + current_user = ?self.user_id, + "Session user mismatch, rejecting connection" + ); + let connack = ConnAckPacket::new(false, ReasonCode::NotAuthorized); + self.write_packet(&Packet::ConnAck(connack), writer)?; + return Err(MqttError::AuthenticationFailed); } } - } + let resume = resumable.is_some(); + let mut session = match resumable { + Some(mut session) => { + self.drop_unauthorized_subscriptions(&mut session).await; + session.will_message.clone_from(&connect.will); + session.will_delay_interval = connect + .will + .as_ref() + .and_then(|will| will.properties.will_delay_interval); + session + } + None => ClientSession::new_with_will( + client_id.clone(), + granted != 0, + Some(granted), + connect.will.clone(), + ), + }; + session.expiry_interval = Some(granted); + session.persistent = granted != 0; + session.user_id.clone_from(&self.user_id); - async fn restore_existing_session( - &mut self, - connect: &ConnectPacket, - mut session: ClientSession, - writer: &mut WasmWriter, - ) -> Result { - let client_id = &connect.client_id; - - if session.user_id.as_deref() != self.user_id.as_deref() { - warn!( - client_id = %client_id, - session_user = ?session.user_id, - current_user = ?self.user_id, - "Session user mismatch, rejecting connection" - ); - let connack = ConnAckPacket::new(false, ReasonCode::NotAuthorized); + let generation = self.router.allocate_generation(); + session.mark_connected(generation); + if let Err(e) = self.storage.store_session(session.clone()).await { + drop(slot); + warn!(client_id = %client_id, "Failed to store the claimed session: {e}"); + let mut connack = ConnAckPacket::new(false, ReasonCode::ServerUnavailable); + connack.protocol_version = self.protocol_version; self.write_packet(&Packet::ConnAck(connack), writer)?; - return Err(MqttError::AuthenticationFailed); + return Err(e); + } + + let (disconnect_tx, disconnect_rx) = tokio::sync::oneshot::channel(); + let queue = self.router.queue_handle(&client_id); + self.router + .register_session_as( + generation, + client_id.clone(), + DeliveryLanes { + qos1_tx: self.qos1_tx.clone(), + qos0_tx: self.qos0_tx.clone(), + }, + queue.clone(), + disconnect_tx, + !resume, + ) + .await; + self.generation = generation; + self.disconnect_rx = Some(disconnect_rx); + self.queue = Some(queue); + + if !resume { + if let Err(e) = self.storage.remove_queued_messages(&client_id).await { + warn!(client_id = %client_id, "Failed to discard queued messages on clean start: {e}"); + } + if let Err(e) = self.storage.remove_all_inflight_messages(&client_id).await { + warn!(client_id = %client_id, "Failed to discard inflight messages on clean start: {e}"); + } + } + if let Err(e) = self + .router + .set_client_subscriptions(&slot, Some(&session)) + .await + { + warn!(client_id = %client_id, "Failed to install session subscriptions: {e}"); } + drop(slot); - let mut unauthorized_filters = Vec::new(); - for (topic_filter, stored) in &session.subscriptions { + self.session = Some(session); + Ok(resume) + } + + async fn drop_unauthorized_subscriptions(&self, session: &mut ClientSession) { + let mut unauthorized = Vec::new(); + for topic_filter in session.subscriptions.keys() { let authorized = self .auth_provider - .authorize_subscribe(client_id, self.user_id.as_deref(), topic_filter) + .authorize_subscribe(&session.client_id, self.user_id.as_deref(), topic_filter) .await; if !authorized { - unauthorized_filters.push(topic_filter.clone()); - continue; + unauthorized.push(topic_filter.clone()); } - self.router - .subscribe( - SubscriptionRequest::new(client_id.clone(), topic_filter.clone(), stored.qos) - .with_subscription_id(stored.subscription_id) - .with_no_local(stored.no_local) - .with_retain_as_published(stored.retain_as_published) - .with_retain_handling(stored.retain_handling) - .with_protocol_version( - ProtocolVersion::try_from(self.protocol_version).unwrap_or_default(), - ) - .with_change_only(stored.change_only) - .with_flow_id(stored.flow_id), - ) - .await?; } - for filter in &unauthorized_filters { + for filter in &unauthorized { session.subscriptions.remove(filter); } - - session.will_message.clone_from(&connect.will); - session.will_delay_interval = connect - .will - .as_ref() - .and_then(|w| w.properties.will_delay_interval); - session.user_id.clone_from(&self.user_id); - self.storage.store_session(session.clone()).await.ok(); - self.session = Some(session); - Ok(true) } pub(super) async fn process_enhanced_auth_result( @@ -267,6 +293,7 @@ impl WasmClientHandler { self.user_id.clone_from(&result.user_id); if let Some(pending) = self.pending_connect.take() { + self.validate_will_capabilities(&pending.connect, writer)?; self.client_id = Some(pending.connect.client_id.clone()); self.keep_alive = mqtt5::time::Duration::from_secs(u64::from(pending.connect.keep_alive)); @@ -390,12 +417,10 @@ impl WasmClientHandler { } fn set_server_capability_properties(&self, connack: &mut ConnAckPacket) { + if let Some(granted) = self.advertised_session_expiry { + connack.properties.set_session_expiry_interval(granted); + } if let Ok(config) = self.config.read() { - connack - .properties - .set_session_expiry_interval(u64_to_u32_saturating( - config.session_expiry_interval.as_secs(), - )); if config.maximum_qos < 2 { connack.properties.set_maximum_qos(config.maximum_qos); } diff --git a/crates/mqtt5-wasm/src/client_handler/lifecycle.rs b/crates/mqtt5-wasm/src/client_handler/lifecycle.rs index 718c1928..dc9dd049 100644 --- a/crates/mqtt5-wasm/src/client_handler/lifecycle.rs +++ b/crates/mqtt5-wasm/src/client_handler/lifecycle.rs @@ -1,5 +1,7 @@ use futures::future::{select, Either}; use mqtt5::broker::auth::EnhancedAuthStatus; +use mqtt5::broker::router::Release; +use mqtt5::broker::storage::{unix_millis_now, ClientSession, StorageBackend}; use mqtt5_protocol::error::{MqttError, Result}; use mqtt5_protocol::packet::auth::AuthPacket; use mqtt5_protocol::packet::disconnect::DisconnectPacket; @@ -19,7 +21,47 @@ impl WasmClientHandler { self.write_packet(&Packet::PingResp, writer) } - pub(super) fn handle_disconnect(&mut self, disconnect: &DisconnectPacket) -> Result<()> { + pub(super) async fn handle_disconnect( + &mut self, + disconnect: &DisconnectPacket, + writer: &mut WasmWriter, + ) -> Result<()> { + if let Some(requested) = disconnect.properties.get_session_expiry_interval() { + if self.connect_session_expiry == Some(0) && requested != 0 { + warn!( + requested, + "Session Expiry Interval on DISCONNECT after 0 on CONNECT is a Protocol Error" + ); + self.write_packet( + &Packet::Disconnect(DisconnectPacket::new(ReasonCode::ProtocolError)), + writer, + )?; + return Err(MqttError::ProtocolError( + "Session Expiry Interval on DISCONNECT after 0 on CONNECT".to_string(), + )); + } + let granted = + ClientSession::granted_expiry(Some(requested), self.maximum_session_expiry()); + if let Some(session) = self.session.as_mut() { + session.expiry_interval = Some(granted); + session.persistent = granted != 0; + } + if let Some(client_id) = self.client_id.clone() { + let slot = self.router.lock_session(&client_id).await; + let stored = self + .storage + .update_session(&client_id, self.generation, move |session| { + session.expiry_interval = Some(granted); + session.persistent = granted != 0; + }) + .await; + drop(slot); + if let Err(e) = stored { + warn!(client_id = %client_id, "Failed to store the DISCONNECT Session Expiry: {e}"); + } + } + } + debug!("Client disconnected normally"); if disconnect.reason_code != ReasonCode::DisconnectWithWillMessage { @@ -191,13 +233,85 @@ impl WasmClientHandler { true } - pub(super) async fn clear_owned_stored_will(&self, client_id: &str) { - if self.router.owns_client(client_id, self.generation).await { - self.router.clear_stored_will(client_id).await; + pub(super) fn session_preserved(&self) -> bool { + self.session + .as_ref() + .is_some_and(|session| session.expiry_interval != Some(0)) + } + + pub(super) async fn release_ownership( + &self, + client_id: &str, + ) -> Option> { + let slot = self.router.lock_session(client_id).await; + let armed_will = if self.normal_disconnect || self.will_delay() == 0 { + None + } else { + self.router.arm_will(client_id, self.generation).await + }; + let release = self + .router + .release_client(client_id, self.generation, self.session_preserved()) + .await; + if matches!(release, Release::Owned) { + self.persist_session_end(client_id).await; + } + drop(slot); + armed_will + } + + fn will_delay(&self) -> u32 { + self.session + .as_ref() + .and_then(ClientSession::will_publish_delay) + .unwrap_or(0) + } + + async fn persist_session_end(&self, client_id: &str) { + let Some(session) = self.session.as_ref() else { + return; + }; + if session.expiry_interval == Some(0) { + match self + .storage + .remove_owned_session(client_id, self.generation) + .await + { + Ok(true) => { + self.storage.queue_handle(client_id).clear(None); + if let Err(e) = self.storage.remove_all_inflight_messages(client_id).await { + warn!("Failed to remove inflight messages for {client_id}: {e}"); + } + } + Ok(false) => debug!(client_id, "Stored session is not this connection's"), + Err(e) => warn!("Failed to remove session for {client_id}: {e}"), + } + return; + } + let expiry_interval = session.expiry_interval; + let discard_will = self.normal_disconnect; + let disconnected_at = unix_millis_now(); + let updated = self + .storage + .update_session(client_id, self.generation, |stored| { + stored.mark_disconnected(disconnected_at); + stored.expiry_interval = expiry_interval; + if discard_will { + stored.will_message = None; + stored.will_delay_interval = None; + } + }) + .await; + if let Err(e) = updated { + warn!("Failed to update session for {client_id}: {e}"); } } - pub(super) async fn publish_will_message(&self, client_id: &str) { + pub(super) async fn publish_will_message( + &self, + client_id: &str, + armed_will: Option>, + ) { let Some(session) = self.session.as_ref() else { return; }; @@ -217,11 +331,13 @@ impl WasmClientHandler { if self.authorize_will(client_id, &publish).await { self.router.route_message(&publish, None).await; } - self.clear_owned_stored_will(client_id).await; + self.router + .clear_stored_will(client_id, self.generation) + .await; return; } - let Some(cancelled) = self.router.arm_will(client_id, self.generation).await else { + let Some(cancelled) = armed_will else { debug!( client_id, "Delayed will dropped: a new connection for the client id was opened" @@ -236,9 +352,7 @@ impl WasmClientHandler { let client_id = client_id.to_string(); let generation = self.generation; spawn_local(async move { - let timer = - gloo_timers::future::sleep(std::time::Duration::from_secs(u64::from(delay))); - if let Either::Right(_) = select(timer, cancelled).await { + if let Either::Right(_) = select(Box::pin(sleep_secs(delay)), cancelled).await { debug!(client_id, "Delayed will cancelled by a new connection"); return; } @@ -261,3 +375,14 @@ impl WasmClientHandler { }); } } + +const MAX_TIMER_MS: u32 = 2_147_483_647; + +async fn sleep_secs(secs: u32) { + let mut remaining_ms = u64::from(secs) * 1000; + while remaining_ms > 0 { + let chunk = u32::try_from(remaining_ms).map_or(MAX_TIMER_MS, |ms| ms.min(MAX_TIMER_MS)); + gloo_timers::future::TimeoutFuture::new(chunk).await; + remaining_ms -= u64::from(chunk); + } +} diff --git a/crates/mqtt5-wasm/src/client_handler/mod.rs b/crates/mqtt5-wasm/src/client_handler/mod.rs index 29ebc055..5c1604e5 100644 --- a/crates/mqtt5-wasm/src/client_handler/mod.rs +++ b/crates/mqtt5-wasm/src/client_handler/mod.rs @@ -5,7 +5,6 @@ mod publish; mod subscribe; use crate::broker::WasmEventCallbacks; -use crate::transport::message_port::MessagePortTransport; use crate::transport::{WasmReader, WasmWriter}; use bytes::BytesMut; use events::{fire_event, set_prop}; @@ -13,7 +12,7 @@ use mqtt5::broker::auth::AuthProvider; use mqtt5::broker::config::BrokerConfig; use mqtt5::broker::resource_monitor::ResourceMonitor; use mqtt5::broker::router::MessageRouter; -use mqtt5::broker::router::{DeliveryLanes, RoutableMessage}; +use mqtt5::broker::router::RoutableMessage; use mqtt5::broker::storage::{ClientSession, DynamicStorage, QueueHandle}; use mqtt5::broker::sys_topics::BrokerStats; use mqtt5_protocol::error::{MqttError, Result}; @@ -22,7 +21,6 @@ use mqtt5_protocol::packet::MqttPacket; use mqtt5_protocol::packet::Packet; use mqtt5_protocol::KeepaliveConfig; use mqtt5_protocol::QoS; -use mqtt5_protocol::Transport; use std::cell::{Cell, RefCell}; use std::collections::HashMap; use std::rc::Rc; @@ -56,6 +54,9 @@ pub struct WasmClientHandler { pub(super) stats: Arc, pub(super) resource_monitor: Arc, pub(super) session: Option, + pub(super) connect_session_expiry: Option, + pub(super) advertised_session_expiry: Option, + disconnect_rx: Option>, qos1_rx: tokio::sync::mpsc::Receiver, qos1_tx: tokio::sync::mpsc::Sender, qos0_rx: tokio::sync::mpsc::Receiver, @@ -76,59 +77,6 @@ pub struct WasmClientHandler { static HANDLER_COUNTER: std::sync::atomic::AtomicU32 = std::sync::atomic::AtomicU32::new(1); impl WasmClientHandler { - #[allow(dead_code, clippy::too_many_arguments)] - pub fn start_deferred( - port: MessagePort, - config: Arc>, - router: Arc, - auth_provider: Arc, - storage: Arc, - stats: Arc, - resource_monitor: Arc, - event_callbacks: WasmEventCallbacks, - ) { - use wasm_bindgen::JsCast; - - let port = Rc::new(RefCell::new(Some(port))); - let port_clone = Rc::clone(&port); - - let config = Rc::new(config); - let router = Rc::new(router); - let auth_provider: Rc> = Rc::new(auth_provider); - let storage = Rc::new(storage); - let stats = Rc::new(stats); - let resource_monitor = Rc::new(resource_monitor); - let event_callbacks = Rc::new(event_callbacks); - - let callback = wasm_bindgen::closure::Closure::::new(move || { - if let Some(p) = port_clone.borrow_mut().take() { - let config = (*config).clone(); - let router = (*router).clone(); - let auth_provider = (*auth_provider).clone(); - let storage = (*storage).clone(); - let stats = (*stats).clone(); - let resource_monitor = (*resource_monitor).clone(); - let event_callbacks = (*event_callbacks).clone(); - - Self::new( - p, - config, - router, - auth_provider, - storage, - stats, - resource_monitor, - event_callbacks, - ); - } - }); - - if let Some(window) = web_sys::window() { - let _ = window.set_timeout_with_callback(callback.as_ref().unchecked_ref()); - } - callback.forget(); - } - #[allow( clippy::must_use_candidate, clippy::new_ret_no_self, @@ -192,6 +140,9 @@ impl WasmClientHandler { stats, resource_monitor, session: None, + connect_session_expiry: None, + advertised_session_expiry: None, + disconnect_rx: None, qos1_rx, qos1_tx, qos0_rx, @@ -216,53 +167,6 @@ impl WasmClientHandler { }); } - #[allow(dead_code)] - async fn run(mut self, port: MessagePort) -> Result<()> { - let mut transport = MessagePortTransport::new(port); - transport.connect().await?; - - let (reader, writer) = transport.into_split()?; - let mut reader = WasmReader::MessagePort(reader); - let mut writer = WasmWriter::MessagePort(writer); - - self.wait_for_connect(&mut reader, &mut writer).await?; - - let client_id = self.client_id.clone().unwrap(); - let (disconnect_tx, disconnect_rx) = tokio::sync::oneshot::channel(); - - let queue = self.router.queue_handle(&client_id); - self.queue = Some(queue.clone()); - let registration = self - .router - .register_client( - client_id.clone(), - DeliveryLanes { - qos1_tx: self.qos1_tx.clone(), - qos0_tx: self.qos0_tx.clone(), - }, - queue.clone(), - disconnect_tx, - ) - .await; - self.generation = registration.generation; - queue.notify(); - - self.stats.client_connected(); - - let result = self.packet_loop(&mut reader, writer, disconnect_rx).await; - - if self.normal_disconnect { - self.clear_owned_stored_will(&client_id).await; - } else { - self.publish_will_message(&client_id).await; - } - - self.release_router_entry(&client_id).await; - self.stats.client_disconnected(); - - result - } - async fn run_with_receiver( mut self, port: MessagePort, @@ -279,62 +183,38 @@ impl WasmClientHandler { let mut reader = WasmReader::MessagePort(reader); let mut writer = WasmWriter::MessagePort(writer); - self.wait_for_connect(&mut reader, &mut writer).await?; - - let client_id = self.client_id.clone().unwrap(); - let (disconnect_tx, disconnect_rx) = tokio::sync::oneshot::channel(); - - let queue = self.router.queue_handle(&client_id); - self.queue = Some(queue.clone()); - let registration = self - .router - .register_client( - client_id.clone(), - DeliveryLanes { - qos1_tx: self.qos1_tx.clone(), - qos0_tx: self.qos0_tx.clone(), - }, - queue.clone(), - disconnect_tx, - ) - .await; - self.generation = registration.generation; + let handshake = self.wait_for_connect(&mut reader, &mut writer).await; + let (Some(client_id), Some(disconnect_rx), Some(queue)) = ( + self.client_id.clone().filter(|_| self.generation != 0), + self.disconnect_rx.take(), + self.queue.clone(), + ) else { + return handshake; + }; + if let Err(e) = handshake { + self.release_ownership(&client_id).await; + return Err(e); + } queue.notify(); self.stats.client_connected(); let result = self.packet_loop(&mut reader, writer, disconnect_rx).await; + let armed_will = self.release_ownership(&client_id).await; let (reason, unexpected) = if self.normal_disconnect { - self.clear_owned_stored_will(&client_id).await; ("client disconnected", false) } else { - self.publish_will_message(&client_id).await; + self.publish_will_message(&client_id, armed_will).await; ("connection lost", true) }; self.fire_client_disconnect(&client_id, reason, unexpected); - - self.release_router_entry(&client_id).await; self.stats.client_disconnected(); result } - /// Releases this connection's router entry, but only if it still owns it. A newer - /// connection with the same client id takes over the entry (a higher generation); an - /// unconditional unregister here would evict that live successor and strip its - /// subscriptions. - async fn release_router_entry(&self, client_id: &str) { - let preserve_session = self - .session - .as_ref() - .is_some_and(|session| session.expiry_interval != Some(0)); - self.router - .release_client(client_id, self.generation, preserve_session) - .await; - } - fn spawn_disconnect_watcher( running: &Rc>, disconnect_rx: tokio::sync::oneshot::Receiver, @@ -596,7 +476,7 @@ impl WasmClientHandler { Ok(()) } Packet::PingReq => self.handle_pingreq(writer), - Packet::Disconnect(ref disconnect) => self.handle_disconnect(disconnect), + Packet::Disconnect(ref disconnect) => self.handle_disconnect(disconnect, writer).await, Packet::Auth(auth) => self.handle_auth(auth, writer).await, _ => { warn!("Unexpected packet type"); diff --git a/crates/mqtt5-wasm/src/client_handler/subscribe.rs b/crates/mqtt5-wasm/src/client_handler/subscribe.rs index 0ee0c98d..14e61a0d 100644 --- a/crates/mqtt5-wasm/src/client_handler/subscribe.rs +++ b/crates/mqtt5-wasm/src/client_handler/subscribe.rs @@ -1,4 +1,4 @@ -use mqtt5::broker::router::SubscriptionRequest; +use mqtt5::broker::router::{Subscribed, SubscriptionRequest, Unsubscribed}; use mqtt5::broker::storage::{StorageBackend, StoredSubscription}; use mqtt5_protocol::error::{MqttError, Result}; use mqtt5_protocol::packet::disconnect::DisconnectPacket; @@ -90,8 +90,11 @@ impl WasmClientHandler { let subscription_id = subscribe.properties.get_subscription_identifier(); let change_only = self.is_change_only_filter(&filter.filter); - self.router - .subscribe( + let slot = self.router.lock_session(&client_id).await; + let outcome = self + .router + .subscribe_as( + Some(self.generation), SubscriptionRequest::new(client_id.clone(), filter.filter.clone(), granted_qos) .with_subscription_id(subscription_id) .with_no_local(filter.options.no_local) @@ -103,9 +106,14 @@ impl WasmClientHandler { .with_change_only(change_only), ) .await?; + if outcome == Subscribed::Fenced { + debug!("Ignoring SUBSCRIBE from a connection whose session was taken over"); + return Ok(()); + } self.persist_subscription(filter, granted_qos, subscription_id, change_only) - .await; + .await?; + drop(slot); self.deliver_retained_for_filter(filter, writer).await?; successful_subscriptions.push((filter.filter.clone(), granted_qos as u8)); @@ -157,21 +165,40 @@ impl WasmClientHandler { granted_qos: QoS, subscription_id: Option, change_only: bool, - ) { - if let Some(ref mut session) = self.session { - let stored = StoredSubscription { - qos: granted_qos, - no_local: filter.options.no_local, - retain_as_published: filter.options.retain_as_published, - retain_handling: filter.options.retain_handling as u8, - subscription_id, - protocol_version: self.protocol_version, - change_only, - flow_id: None, - }; - session.add_subscription(filter.filter.clone(), stored); - self.storage.store_session(session.clone()).await.ok(); + ) -> Result<()> { + let stored = StoredSubscription { + qos: granted_qos, + no_local: filter.options.no_local, + retain_as_published: filter.options.retain_as_published, + retain_handling: filter.options.retain_handling as u8, + subscription_id, + protocol_version: self.protocol_version, + change_only, + flow_id: None, + }; + if let Some(session) = self.session.as_mut() { + session.add_subscription(filter.filter.clone(), stored.clone()); } + let topic_filter = filter.filter.clone(); + self.update_stored_session(move |session| session.add_subscription(topic_filter, stored)) + .await + } + + async fn update_stored_session(&self, update: F) -> Result<()> + where + F: FnOnce(&mut mqtt5::broker::storage::ClientSession) + Send, + { + let Some(client_id) = self.client_id.as_ref() else { + return Ok(()); + }; + if !self + .storage + .update_session(client_id, self.generation, update) + .await? + { + debug!(client_id = %client_id, "Stored session is not this connection's"); + } + Ok(()) } async fn deliver_retained_for_filter( @@ -196,18 +223,39 @@ impl WasmClientHandler { unsubscribe: UnsubscribePacket, writer: &mut WasmWriter, ) -> Result<()> { - let client_id = self.client_id.as_ref().unwrap(); + let Some(client_id) = self.client_id.clone() else { + return Err(MqttError::ProtocolError( + "UNSUBSCRIBE before CONNECT".to_string(), + )); + }; let mut reason_codes = Vec::new(); for filter in &unsubscribe.filters { - let removed = self.router.unsubscribe(client_id, filter, None).await; + let slot = self.router.lock_session(&client_id).await; + let removed = match self + .router + .unsubscribe_as(Some(self.generation), &client_id, filter, None) + .await + { + Unsubscribed::Removed => true, + Unsubscribed::Absent => false, + Unsubscribed::Fenced => { + debug!("Ignoring UNSUBSCRIBE from a connection whose session was taken over"); + return Ok(()); + } + }; if removed { - if let Some(ref mut session) = self.session { + if let Some(session) = self.session.as_mut() { session.remove_subscription(filter); - self.storage.store_session(session.clone()).await.ok(); } + let topic_filter = filter.clone(); + self.update_stored_session(move |session| { + session.remove_subscription(&topic_filter); + }) + .await?; } + drop(slot); reason_codes.push(if removed { UnsubAckReasonCode::Success @@ -223,7 +271,7 @@ impl WasmClientHandler { self.write_packet(&Packet::UnsubAck(unsuback), writer)?; if !unsubscribe.filters.is_empty() { - self.fire_client_unsubscribe(client_id, &unsubscribe.filters); + self.fire_client_unsubscribe(&client_id, &unsubscribe.filters); } Ok(()) diff --git a/crates/mqtt5-wasm/src/config.rs b/crates/mqtt5-wasm/src/config.rs index 23287c30..755ae78e 100644 --- a/crates/mqtt5-wasm/src/config.rs +++ b/crates/mqtt5-wasm/src/config.rs @@ -216,6 +216,10 @@ impl WasmConnectOptions { /// for example after a page reload or crash. The client has nothing to resend in /// that case; the broker resumes delivery of the messages it holds. A CONNACK with /// Session Present set to 1 in reply to `cleanStart = true` is always rejected. + /// + /// The broker only keeps a session to resume if the previous connection set a non-zero + /// `sessionExpiryInterval`: under MQTT v5 an absent value means 0, so the session ends + /// when the connection closes. #[wasm_bindgen(getter = resumeExistingSession)] #[must_use] pub fn resume_existing_session(&self) -> bool { diff --git a/crates/mqtt5-wasm/tests/broker_will_delay.rs b/crates/mqtt5-wasm/tests/broker_will_delay.rs index c53ffa9e..e8756034 100644 --- a/crates/mqtt5-wasm/tests/broker_will_delay.rs +++ b/crates/mqtt5-wasm/tests/broker_will_delay.rs @@ -1,9 +1,10 @@ #![cfg(all(target_arch = "wasm32", feature = "broker"))] use bytes::BytesMut; +use mqtt5_protocol::packet::connack::ConnAckPacket; use mqtt5_protocol::packet::connect::ConnectPacket; use mqtt5_protocol::packet::disconnect::DisconnectPacket; -use mqtt5_protocol::packet::MqttPacket; +use mqtt5_protocol::packet::{FixedHeader, MqttPacket}; use mqtt5_protocol::protocol::v5::reason_codes::ReasonCode; use mqtt5_protocol::types::{ConnectOptions, WillMessage}; use mqtt5_wasm::{ @@ -48,6 +49,13 @@ fn broker() -> WasmBroker { WasmBroker::with_config(config).unwrap() } +fn capped_broker(maximum: u32) -> WasmBroker { + let mut config = WasmBrokerConfig::new(); + config.set_allow_anonymous(true); + config.set_session_expiry_interval_secs(maximum); + WasmBroker::with_config(config).unwrap() +} + fn will_topic(client_id: &str) -> String { format!("will/{client_id}") } @@ -84,6 +92,18 @@ impl RawClient { will: Option, keep_alive_secs: u64, ) -> Self { + let options = ConnectOptions::new(client_id) + .with_clean_start(clean_start) + .with_session_expiry_interval(session_expiry) + .with_keep_alive(Duration::from_secs(keep_alive_secs)); + let options = match will { + Some(will) => options.with_will(will), + None => options, + }; + Self::connect_with_options(broker, options).await + } + + async fn connect_with_options(broker: &WasmBroker, options: ConnectOptions) -> Self { let port = broker.create_client_port().unwrap(); let inbox = Rc::new(RefCell::new(Vec::new())); let inbox_in = Rc::clone(&inbox); @@ -100,14 +120,7 @@ impl RawClient { on_message, }; - let options = ConnectOptions::new(client_id) - .with_clean_start(clean_start) - .with_session_expiry_interval(session_expiry) - .with_keep_alive(Duration::from_secs(keep_alive_secs)); - let options = match will { - Some(will) => options.with_will(will), - None => options, - }; + let client_id = options.client_id.clone(); client.send(&ConnectPacket::new(options)); for _ in 0..200 { @@ -123,6 +136,29 @@ impl RawClient { client } + fn connack(&self) -> ConnAckPacket { + let inbox = self.inbox.borrow().clone(); + let mut cursor = &inbox[..]; + let header = FixedHeader::decode(&mut cursor).unwrap(); + ConnAckPacket::decode_body(&mut cursor, &header).unwrap() + } + + fn session_present(&self) -> bool { + self.inbox.borrow()[2] & 0x01 == 0x01 + } + + async fn server_disconnect_reason(&self) -> Option { + for _ in 0..200 { + let inbox = self.inbox.borrow().clone(); + let after_connack = usize::from(inbox[1]) + 2; + if inbox.len() >= after_connack + 3 && inbox[after_connack] == 0xE0 { + return Some(inbox[after_connack + 2]); + } + sleep(10).await; + } + None + } + fn send(&self, packet: &impl MqttPacket) { let mut buf = BytesMut::new(); packet.encode(&mut buf).unwrap(); @@ -135,7 +171,11 @@ impl RawClient { } fn disconnect(self, reason: ReasonCode) { - self.send(&DisconnectPacket::new(reason)); + self.disconnect_with(&DisconnectPacket::new(reason)); + } + + fn disconnect_with(self, disconnect: &DisconnectPacket) { + self.send(disconnect); self.port.close(); } } @@ -330,3 +370,455 @@ async fn keep_alive_expiry_publishes_will() { drop(silent); watcher.disconnect().await.unwrap(); } + +fn disconnect_with_expiry(reason: ReasonCode, session_expiry: u32) -> DisconnectPacket { + let mut disconnect = DisconnectPacket::new(reason); + disconnect + .properties + .set_session_expiry_interval(session_expiry); + disconnect +} + +#[wasm_bindgen_test] +async fn resume_with_longer_expiry_keeps_full_will_delay() { + let broker = broker(); + let client_id = "wasm-wd-resume-longer"; + let (watcher, count) = watch_will(&broker, client_id).await; + + let first = RawClient::connect(&broker, client_id, true, 2, None).await; + first.disconnect(ReasonCode::Success); + sleep(300).await; + + let second = RawClient::connect( + &broker, + client_id, + false, + SESSION_EXPIRY, + Some(delayed_will(client_id, 5)), + ) + .await; + let dropped_at = now_ms(); + second.close_without_disconnect(); + + let published = wait_for_will(&count, 3500).await; + let elapsed = now_ms() - dropped_at; + assert!( + !published, + "Will published {elapsed}ms after drop, before the 5s Will Delay, though the resuming CONNECT set Session Expiry 60" + ); + + watcher.disconnect().await.unwrap(); +} + +#[wasm_bindgen_test] +async fn resume_with_expiry_zero_publishes_will_immediately() { + let broker = broker(); + let client_id = "wasm-wd-resume-zero"; + let (watcher, count) = watch_will(&broker, client_id).await; + + let first = RawClient::connect(&broker, client_id, true, SESSION_EXPIRY, None).await; + first.disconnect(ReasonCode::Success); + sleep(300).await; + + let second = RawClient::connect( + &broker, + client_id, + false, + 0, + Some(delayed_will(client_id, 5)), + ) + .await; + second.close_without_disconnect(); + + assert!( + wait_for_will(&count, 1500).await, + "Session Expiry 0 on the resuming CONNECT ends the session at disconnect, so the Will must not wait for the delay" + ); + + watcher.disconnect().await.unwrap(); +} + +#[wasm_bindgen_test] +async fn long_will_delay_is_not_published_early() { + let broker = broker(); + let client_id = "wasm-wd-long"; + let (watcher, count) = watch_will(&broker, client_id).await; + + let conn = RawClient::connect( + &broker, + client_id, + true, + 3_000_000, + Some(delayed_will(client_id, 2_200_000)), + ) + .await; + conn.close_without_disconnect(); + + assert!( + !wait_for_will(&count, 1500).await, + "a 2_200_000s (25 day) Will Delay must not publish the Will within 1.5s" + ); + + watcher.disconnect().await.unwrap(); +} + +#[wasm_bindgen_test] +async fn absent_session_expiry_publishes_will_immediately() { + let broker = broker(); + let client_id = "wasm-wd-absent-expiry"; + let (watcher, count) = watch_will(&broker, client_id).await; + + let options = ConnectOptions::new(client_id) + .with_clean_start(true) + .with_will(delayed_will(client_id, 3)); + let conn = RawClient::connect_with_options(&broker, options).await; + conn.close_without_disconnect(); + + assert!( + wait_for_will(&count, 1500).await, + "an absent Session Expiry Interval means 0, so the Will must be published immediately" + ); + + watcher.disconnect().await.unwrap(); +} + +#[wasm_bindgen_test] +async fn absent_session_expiry_session_is_not_resumed() { + let broker = broker(); + let client_id = "wasm-wd-absent-resume"; + + let first = RawClient::connect_with_options( + &broker, + ConnectOptions::new(client_id).with_clean_start(true), + ) + .await; + first.disconnect(ReasonCode::Success); + sleep(300).await; + + let second = RawClient::connect(&broker, client_id, false, SESSION_EXPIRY, None).await; + assert!( + !second.session_present(), + "an absent Session Expiry Interval means 0, so no session survives the disconnect" + ); + second.disconnect(ReasonCode::Success); +} + +#[wasm_bindgen_test] +async fn disconnect_session_expiry_zero_releases_delayed_will() { + let broker = broker(); + let client_id = "wasm-wd-disconnect-zero"; + let (watcher, count) = watch_will(&broker, client_id).await; + + let conn = RawClient::connect( + &broker, + client_id, + true, + SESSION_EXPIRY, + Some(delayed_will(client_id, 5)), + ) + .await; + conn.disconnect_with(&disconnect_with_expiry( + ReasonCode::DisconnectWithWillMessage, + 0, + )); + + assert!( + wait_for_will(&count, 1500).await, + "Session Expiry 0 on DISCONNECT ends the session now, so the Will must not wait for the delay" + ); + + let second = RawClient::connect(&broker, client_id, false, SESSION_EXPIRY, None).await; + assert!( + !second.session_present(), + "Session Expiry 0 on DISCONNECT must end the session" + ); + second.disconnect(ReasonCode::Success); + watcher.disconnect().await.unwrap(); +} + +#[wasm_bindgen_test] +async fn disconnect_session_expiry_after_zero_is_protocol_error() { + let broker = broker(); + let client_id = "wasm-wd-disconnect-error"; + let (watcher, count) = watch_will(&broker, client_id).await; + + let conn = RawClient::connect( + &broker, + client_id, + true, + 0, + Some(delayed_will(client_id, 0)), + ) + .await; + conn.send(&disconnect_with_expiry(ReasonCode::Success, SESSION_EXPIRY)); + + assert_eq!( + conn.server_disconnect_reason().await, + Some(0x82), + "a non-zero Session Expiry on DISCONNECT after CONNECT sent 0 is a Protocol Error" + ); + assert!( + wait_for_will(&count, 1500).await, + "an invalid DISCONNECT is not a normal disconnection, so the Will is published" + ); + conn.close_without_disconnect(); + + watcher.disconnect().await.unwrap(); +} + +#[wasm_bindgen_test] +async fn session_expiry_counts_from_disconnect() { + let broker = broker(); + let client_id = "wasm-long-lived"; + let first = RawClient::connect(&broker, client_id, true, 2, None).await; + sleep(3000).await; + first.disconnect(ReasonCode::Success); + sleep(200).await; + + let second = RawClient::connect(&broker, client_id, false, SESSION_EXPIRY, None).await; + let present = second.session_present(); + second.disconnect(ReasonCode::Success); + assert!( + present, + "a 2s Session Expiry counts from the disconnect, not from CONNECT" + ); +} + +#[wasm_bindgen_test] +async fn disconnect_expiry_extends_a_long_connection() { + let broker = broker(); + let client_id = "wasm-long-extend"; + let first = RawClient::connect(&broker, client_id, true, 1, None).await; + sleep(2500).await; + first.disconnect_with(&disconnect_with_expiry(ReasonCode::Success, SESSION_EXPIRY)); + sleep(1500).await; + + let second = RawClient::connect(&broker, client_id, false, SESSION_EXPIRY, None).await; + let present = second.session_present(); + second.disconnect(ReasonCode::Success); + assert!( + present, + "the DISCONNECT Session Expiry of 60s applies from the disconnect" + ); +} + +#[wasm_bindgen_test] +async fn session_expiry_above_the_broker_maximum_is_capped_and_advertised() { + let broker = capped_broker(1); + let client_id = "wasm-cap-high"; + let (watcher, count) = watch_will(&broker, client_id).await; + + let conn = RawClient::connect( + &broker, + client_id, + true, + 3600, + Some(delayed_will(client_id, 5)), + ) + .await; + assert_eq!( + conn.connack().properties.get_session_expiry_interval(), + Some(1) + ); + conn.close_without_disconnect(); + + assert!( + wait_for_will(&count, 2500).await, + "the session ends 1s after disconnect under the broker maximum, so the 5s Will fires then" + ); + watcher.disconnect().await.unwrap(); +} + +#[wasm_bindgen_test] +async fn session_expiry_within_the_broker_maximum_is_not_advertised() { + let broker = capped_broker(10); + let conn = RawClient::connect(&broker, "wasm-cap-low", true, 5, None).await; + assert_eq!( + conn.connack().properties.get_session_expiry_interval(), + None + ); + conn.disconnect(ReasonCode::Success); +} + +#[wasm_bindgen_test] +async fn connack_session_expiry_matches_the_session_kept() { + let broker = broker(); + let client_id = "wasm-connack-kept"; + let first = RawClient::connect_with_options( + &broker, + ConnectOptions::new(client_id).with_clean_start(true), + ) + .await; + let advertised = first.connack().properties.get_session_expiry_interval(); + first.disconnect(ReasonCode::Success); + sleep(300).await; + + let second = RawClient::connect(&broker, client_id, false, SESSION_EXPIRY, None).await; + let present = second.session_present(); + second.disconnect(ReasonCode::Success); + assert_eq!( + advertised, None, + "the broker keeps the client's Session Expiry, so CONNACK must not advertise another" + ); + assert!( + !present, + "an absent Session Expiry means 0: the session ended" + ); +} + +#[wasm_bindgen_test] +async fn reconnect_then_drop_again_publishes_one_will_after_second_delay() { + let broker = broker(); + let client_id = "wasm-wd-redrop"; + let (watcher, count) = watch_will(&broker, client_id).await; + + let first = RawClient::connect( + &broker, + client_id, + true, + SESSION_EXPIRY, + Some(delayed_will(client_id, 2)), + ) + .await; + let first_dropped_at = now_ms(); + first.close_without_disconnect(); + sleep(300).await; + + let second = RawClient::connect( + &broker, + client_id, + false, + SESSION_EXPIRY, + Some(delayed_will(client_id, 2)), + ) + .await; + sleep(500).await; + let second_dropped_at = now_ms(); + second.close_without_disconnect(); + + sleep_until(first_dropped_at + 2400.0).await; + assert_eq!( + count.get(), + 0, + "the first connection's Will was cancelled by the reconnect" + ); + sleep_until(second_dropped_at + 1500.0).await; + assert_eq!(count.get(), 0, "the second Will waits for its own delay"); + assert!( + wait_for_will(&count, 3000).await, + "the second connection's Will is published after its delay" + ); + sleep(700).await; + assert_eq!(count.get(), 1, "exactly one Will is published"); + + watcher.disconnect().await.unwrap(); +} + +#[wasm_bindgen_test] +async fn takeover_of_a_live_expiry_zero_session_is_not_resumed() { + let broker = broker(); + let client_id = "wasm-live-zero"; + let first = RawClient::connect(&broker, client_id, true, 0, None).await; + sleep(200).await; + let second = RawClient::connect(&broker, client_id, false, SESSION_EXPIRY, None).await; + let present = second.session_present(); + second.disconnect(ReasonCode::Success); + drop(first); + assert!( + !present, + "an expiry-0 session ends when its connection is taken over" + ); +} + +fn send_raw(client: &RawClient, bytes: &[u8]) { + let array = js_sys::Uint8Array::from(bytes); + client.port.post_message(&array.buffer()).unwrap(); +} + +fn send_subscribe(client: &RawClient, topic: &str) { + let topic_len = u8::try_from(topic.len()).unwrap(); + let mut subscribe = vec![0x82, 6 + topic_len, 0x00, 0x01, 0x00, 0x00, topic_len]; + subscribe.extend_from_slice(topic.as_bytes()); + subscribe.push(0x00); + send_raw(client, &subscribe); +} + +fn send_publish(client: &RawClient, topic: &str) { + let topic_len = u8::try_from(topic.len()).unwrap(); + let mut publish = vec![0x30, 5 + topic_len, 0x00, topic_len]; + publish.extend_from_slice(topic.as_bytes()); + publish.extend_from_slice(&[0x00, b'h', b'i']); + send_raw(client, &publish); +} + +fn received_publish(client: &RawClient) -> bool { + let inbox = client.inbox.borrow().clone(); + let mut i = usize::from(inbox[1]) + 2; + while i + 1 < inbox.len() { + if inbox[i] & 0xF0 == 0x30 { + return true; + } + i += usize::from(inbox[i + 1]) + 2; + } + false +} + +#[wasm_bindgen_test] +async fn clean_start_after_an_offline_session_drops_its_subscriptions() { + let broker = broker(); + let publisher = RawClient::connect(&broker, "wcs-pub", true, 0, None).await; + let first = RawClient::connect(&broker, "wcs-take", true, SESSION_EXPIRY, None).await; + send_subscribe(&first, "wcs/t"); + sleep(200).await; + first.disconnect(ReasonCode::Success); + sleep(200).await; + let second = RawClient::connect(&broker, "wcs-take", true, SESSION_EXPIRY, None).await; + let present = second.session_present(); + sleep(200).await; + send_publish(&publisher, "wcs/t"); + sleep(500).await; + let delivered = received_publish(&second); + second.disconnect(ReasonCode::Success); + publisher.disconnect(ReasonCode::Success); + assert!(!present); + assert!( + !delivered, + "a clean start must not keep the previous session's subscription" + ); +} + +#[wasm_bindgen_test] +async fn subscription_delivers_to_its_own_connection() { + let broker = broker(); + let publisher = RawClient::connect(&broker, "wpc-pub", true, 0, None).await; + let first = RawClient::connect(&broker, "wpc-sub", true, SESSION_EXPIRY, None).await; + send_subscribe(&first, "wpc/t"); + sleep(200).await; + send_publish(&publisher, "wpc/t"); + sleep(500).await; + let delivered = received_publish(&first); + first.disconnect(ReasonCode::Success); + publisher.disconnect(ReasonCode::Success); + assert!(delivered, "positive control for the clean-start test"); +} + +#[wasm_bindgen_test] +async fn takeover_does_not_resume_a_session_the_new_connection_starts_clean() { + let broker = broker(); + let publisher = RawClient::connect(&broker, "wtk-pub", true, 0, None).await; + let first = RawClient::connect(&broker, "wtk-sub", true, SESSION_EXPIRY, None).await; + send_subscribe(&first, "wtk/t"); + sleep(200).await; + let second = RawClient::connect(&broker, "wtk-sub", true, SESSION_EXPIRY, None).await; + sleep(200).await; + send_publish(&publisher, "wtk/t"); + sleep(500).await; + let delivered = received_publish(&second); + second.disconnect(ReasonCode::Success); + publisher.disconnect(ReasonCode::Success); + drop(first); + assert!( + !delivered, + "a clean-start takeover must not inherit the displaced connection's subscription" + ); +} diff --git a/crates/mqtt5/Cargo.toml b/crates/mqtt5/Cargo.toml index 830099c1..0b2edc7f 100644 --- a/crates/mqtt5/Cargo.toml +++ b/crates/mqtt5/Cargo.toml @@ -1,6 +1,6 @@ [package] name = "mqtt5" -version = "0.41.1" +version = "0.42.0" edition.workspace = true rust-version.workspace = true authors.workspace = true diff --git a/crates/mqtt5/src/broker/client_handler/auth.rs b/crates/mqtt5/src/broker/client_handler/auth.rs index d8ed2507..b923127e 100644 --- a/crates/mqtt5/src/broker/client_handler/auth.rs +++ b/crates/mqtt5/src/broker/client_handler/auth.rs @@ -1,7 +1,6 @@ use crate::broker::auth::EnhancedAuthStatus; use crate::error::{MqttError, Result}; use crate::packet::auth::AuthPacket; -use crate::packet::connack::ConnAckPacket; use crate::packet::disconnect::DisconnectPacket; use crate::packet::Packet; use crate::protocol::v5::reason_codes::ReasonCode; @@ -77,35 +76,10 @@ impl ClientHandler { self.user_id = result.user_id; if let Some(pending) = self.pending_connect.take() { - let session_present = self.handle_session(&pending.connect).await?; - - let mut connack = if self.protocol_version == 4 { - ConnAckPacket::new_v311(session_present, ReasonCode::Success) - } else { - ConnAckPacket::new(session_present, ReasonCode::Success) - }; - - if self.protocol_version == 5 { - if let Some(ref assigned_id) = pending.assigned_client_id { - connack - .properties - .set_assigned_client_identifier(assigned_id.clone()); - } - - connack - .properties - .set_topic_alias_maximum(self.config.topic_alias_maximum); - connack - .properties - .set_retain_available(self.config.retain_available); - connack.properties.set_maximum_packet_size( - u32::try_from(self.config.max_packet_size).unwrap_or(u32::MAX), - ); - } - - self.write_to_client(Packet::ConnAck(connack)).await?; + self.connack_auth_data = result.auth_data; + self.authenticated_connect = Some(pending); } else { - let success_auth = AuthPacket::success(result.auth_method)?; + let success_auth = Self::auth_success(result.auth_method, result.auth_data)?; self.write_to_client(Packet::Auth(success_auth)).await?; } } @@ -120,8 +94,19 @@ impl ClientHandler { } else { None }; - let failure_auth = AuthPacket::failure(result.reason_code, reason_string)?; - self.write_to_client(Packet::Auth(failure_auth)).await?; + if self.pending_connect.take().is_some() { + let mut connack = self.new_connack(false, result.reason_code); + if let Some(reason) = reason_string.filter(|_| self.protocol_version == 5) { + connack.properties.set_reason_string(reason); + } + self.write_to_client(Packet::ConnAck(connack)).await?; + } else if self.protocol_version == 5 { + let mut disconnect = DisconnectPacket::new(result.reason_code); + if let Some(reason) = reason_string { + disconnect.properties.set_reason_string(reason); + } + self.write_to_client(Packet::Disconnect(disconnect)).await?; + } return Err(MqttError::AuthenticationFailed); } } @@ -153,7 +138,7 @@ impl ClientHandler { match result.status { EnhancedAuthStatus::Success => { self.user_id = result.user_id; - let success_auth = AuthPacket::success(result.auth_method)?; + let success_auth = Self::auth_success(result.auth_method, result.auth_data)?; self.write_to_client(Packet::Auth(success_auth)).await?; } EnhancedAuthStatus::Continue => { @@ -171,4 +156,12 @@ impl ClientHandler { } Ok(()) } + + fn auth_success(auth_method: String, auth_data: Option>) -> Result { + let mut success = AuthPacket::success(auth_method)?; + if let Some(data) = auth_data { + success.properties.set_authentication_data(data.into()); + } + Ok(success) + } } diff --git a/crates/mqtt5/src/broker/client_handler/connect.rs b/crates/mqtt5/src/broker/client_handler/connect.rs index cb291387..e65d1a38 100644 --- a/crates/mqtt5/src/broker/client_handler/connect.rs +++ b/crates/mqtt5/src/broker/client_handler/connect.rs @@ -1,5 +1,5 @@ use crate::broker::auth::EnhancedAuthStatus; -use crate::broker::router::SubscriptionRequest; +use crate::broker::router::DeliveryLanes; use crate::broker::storage::{ClientSession, DynamicStorage, StorageBackend}; use crate::error::{MqttError, Result}; use crate::packet::auth::AuthPacket; @@ -8,8 +8,9 @@ use crate::packet::connect::ConnectPacket; use crate::packet::Packet; use crate::protocol::v5::reason_codes::ReasonCode; use crate::time::Duration; -use crate::types::ProtocolVersion; use std::sync::Arc; +use tokio::sync::oneshot; +use tokio::time::Instant; use tracing::{debug, info, trace, warn}; use super::{AuthState, ClientHandler, PendingConnect}; @@ -58,7 +59,10 @@ impl ClientHandler { } } - pub(super) async fn handle_connect(&mut self, mut connect: ConnectPacket) -> Result<()> { + pub(super) async fn handle_connect( + &mut self, + mut connect: ConnectPacket, + ) -> Result> { debug!( client_id = %connect.client_id, addr = %self.client_addr, @@ -72,7 +76,7 @@ impl ClientHandler { .await?; if let Some(redirect) = self.check_load_balancer_redirect(&connect).await? { - return redirect; + return redirect.map(|()| None); } self.request_problem_information = connect @@ -100,15 +104,24 @@ impl ClientHandler { let assigned_client_id = Self::assign_client_id_if_empty(&mut connect); self.validate_client_id(&connect).await?; - let connect = match self + match self .handle_authentication(connect, assigned_client_id.clone()) .await? { - AuthOutcome::Authenticated(connect) => *connect, - AuthOutcome::ConnectDeferred => return Ok(()), - AuthOutcome::Failed(err) => return Err(err), - }; + AuthOutcome::Authenticated(connect) => Ok(Some(PendingConnect { + connect: *connect, + assigned_client_id, + })), + AuthOutcome::ConnectDeferred => Ok(None), + AuthOutcome::Failed(err) => Err(err), + } + } + pub(super) async fn complete_connect(&mut self, accepted: PendingConnect) -> Result<()> { + let PendingConnect { + connect, + assigned_client_id, + } = accepted; self.validate_will_capabilities(&connect).await?; self.client_id = Some(connect.client_id.clone()); @@ -134,14 +147,15 @@ impl ClientHandler { #[cfg(not(feature = "opentelemetry"))] let session_present = self.handle_session(&connect).await?; - let mut connack = if self.protocol_version == 4 { - ConnAckPacket::new_v311(session_present, ReasonCode::Success) - } else { - ConnAckPacket::new(session_present, ReasonCode::Success) - }; - + let mut connack = self.new_connack(session_present, ReasonCode::Success); if self.protocol_version == 5 { self.build_connack_properties(&mut connack, assigned_client_id.as_ref()); + if let Some(method) = self.auth_method.clone() { + connack.properties.set_authentication_method(method); + if let Some(data) = self.connack_auth_data.take() { + connack.properties.set_authentication_data(data.into()); + } + } } debug!( @@ -252,6 +266,7 @@ impl ClientHandler { EnhancedAuthStatus::Success => { self.auth_state = AuthState::Completed; self.user_id = result.user_id; + self.connack_auth_data = result.auth_data; } EnhancedAuthStatus::Continue => { self.auth_state = AuthState::InProgress; @@ -319,7 +334,11 @@ impl ClientHandler { Ok(AuthOutcome::Authenticated(Box::new(connect))) } - fn new_connack(&self, session_present: bool, reason_code: ReasonCode) -> ConnAckPacket { + pub(super) fn new_connack( + &self, + session_present: bool, + reason_code: ReasonCode, + ) -> ConnAckPacket { if self.protocol_version == 4 { ConnAckPacket::new_v311(session_present, reason_code) } else { @@ -355,7 +374,7 @@ impl ClientHandler { Ok(()) } - fn build_connack_properties( + pub(super) fn build_connack_properties( &mut self, connack: &mut ConnAckPacket, assigned_client_id: Option<&String>, @@ -367,6 +386,10 @@ impl ClientHandler { .set_assigned_client_identifier(assigned_id.clone()); } + if let Some(granted) = self.advertised_session_expiry { + connack.properties.set_session_expiry_interval(granted); + } + connack .properties .set_topic_alias_maximum(self.config.topic_alias_maximum); @@ -414,142 +437,323 @@ impl ClientHandler { } } - /// Decides whether the connection resumes a session; a session that only lives as long - /// as its connection (expiry 0) cannot be resumed from a live connection, so taking one - /// over is a clean start. + pub(super) fn maximum_session_expiry(&self) -> u32 { + if self.storage.is_none() { + return 0; + } + u32::try_from(self.config.session_expiry_interval.as_secs()).unwrap_or(u32::MAX) + } + pub(super) async fn handle_session(&mut self, connect: &ConnectPacket) -> Result { - let mut session_present = false; - self.clean_start = connect.clean_start; - if let Some(storage) = self.storage.clone() { - let existing_session = storage.get_session(&connect.client_id).await?; - let connection_bound = existing_session - .as_ref() - .is_some_and(|session| session.expiry_interval == Some(0)); - if connection_bound && self.router.is_connected(&connect.client_id).await { - self.clean_start = true; + let client_id = connect.client_id.clone(); + let requested = ClientSession::expiry_from_connect(connect); + let granted = ClientSession::granted_expiry(requested, self.maximum_session_expiry()); + self.connect_session_expiry = requested; + self.advertised_session_expiry = (requested != Some(granted)).then_some(granted); + + let slot = self.router.lock_session(&client_id).await; + let stored = match self.storage.as_ref() { + Some(storage) => storage.get_session(&client_id).await?, + None => None, + }; + let resumable = + stored.filter(|session| !connect.clean_start && session.expiry_interval != Some(0)); + if let Some(session) = resumable.as_ref() { + if session.user_id.as_deref() != self.user_id.as_deref() { + drop(slot); + warn!( + client_id = %client_id, + session_user = ?session.user_id, + current_user = ?self.user_id, + "Session user mismatch, rejecting connection" + ); + let connack = ConnAckPacket::new(false, ReasonCode::NotAuthorized); + self.write_to_client(Packet::ConnAck(connack)).await?; + return Err(MqttError::AuthenticationFailed); } + } + let resume = resumable.is_some(); + let mut session = self.session_for_claim(connect, resumable, granted).await; - match existing_session { - Some(session) if !self.clean_start => { - session_present = true; - self.restore_existing_session(connect, session, &storage) - .await?; - } - _ => { - self.clean_start = true; - self.create_new_session(connect, &storage).await?; - } + let generation = self.router.allocate_generation(); + session.mark_connected(generation); + if let Some(storage) = self.storage.clone() { + if let Err(e) = Self::write_claim(&storage, &session, resume).await { + drop(slot); + warn!(client_id = %client_id, "Failed to store the claimed session: {e}"); + let connack = self.new_connack(false, ReasonCode::ServerUnavailable); + self.write_to_client(Packet::ConnAck(connack)).await?; + return Err(e); } } - Ok(session_present) - } - - async fn create_new_session( - &mut self, - connect: &ConnectPacket, - storage: &Arc, - ) -> Result<()> { - let session_expiry = connect.properties.get_session_expiry_interval(); - let will_message = connect.will.clone(); - if let Some(ref will) = will_message { - debug!( - "Will message present with delay: {:?}", - will.properties.will_delay_interval - ); + let (disconnect_tx, disconnect_rx) = oneshot::channel(); + let queue = self.router.queue_handle(&client_id); + let registration = self + .router + .register_session_as( + generation, + client_id.clone(), + DeliveryLanes { + qos1_tx: self.qos1_tx.clone(), + qos0_tx: self.qos0_tx.clone(), + }, + Arc::clone(&queue), + disconnect_tx, + !resume, + ) + .await; + self.generation = generation; + self.handoff_deadline = registration + .released + .as_ref() + .map(|_| Instant::now() + super::HANDOFF_BOUND); + self.released_rx = registration.released; + self.disconnect_rx = Some(disconnect_rx); + self.queue = Some(queue); + self.clean_start = !resume; + + if let Err(e) = self + .router + .set_client_subscriptions(&slot, Some(&session)) + .await + { + warn!(client_id = %client_id, "Failed to install session subscriptions: {e}"); } + if resume { + self.router + .load_change_only_state(&client_id, session.change_only_state.clone()) + .await; + } + drop(slot); - let mut session = ClientSession::new_with_will( - connect.client_id.clone(), - true, - session_expiry, - will_message, - ); - session.receive_maximum = self.client_receive_maximum; - session.user_id.clone_from(&self.user_id); debug!( - "Created new session with will_delay_interval: {:?}", - session.will_delay_interval + client_id = %client_id, + generation, + session_present = resume, + session_expiry = granted, + "Session claimed" ); - storage.store_session(session.clone()).await?; - storage - .remove_all_inflight_messages(&connect.client_id) - .await?; self.session = Some(session); - Ok(()) + Ok(resume) } - async fn restore_existing_session( - &mut self, + async fn session_for_claim( + &self, connect: &ConnectPacket, - mut session: ClientSession, - storage: &Arc, + resumable: Option, + granted: u32, + ) -> ClientSession { + let mut session = match resumable { + Some(mut session) => { + self.drop_unauthorized_subscriptions(&mut session).await; + session.will_message.clone_from(&connect.will); + session.will_delay_interval = connect + .will + .as_ref() + .and_then(|will| will.properties.will_delay_interval); + session + } + None => ClientSession::new_with_will( + connect.client_id.clone(), + true, + Some(granted), + connect.will.clone(), + ), + }; + session.expiry_interval = Some(granted); + session.receive_maximum = self.client_receive_maximum; + session.user_id.clone_from(&self.user_id); + session + } + + async fn write_claim( + storage: &DynamicStorage, + session: &ClientSession, + resume: bool, ) -> Result<()> { - if session.user_id.as_deref() != self.user_id.as_deref() { - warn!( - client_id = %connect.client_id, - session_user = ?session.user_id, - current_user = ?self.user_id, - "Session user mismatch, rejecting connection" - ); - let connack = ConnAckPacket::new(false, ReasonCode::NotAuthorized); - self.write_to_client(Packet::ConnAck(connack)).await?; - return Err(MqttError::AuthenticationFailed); + storage.store_session(session.clone()).await?; + if !resume { + if let Err(e) = storage + .remove_all_inflight_messages(&session.client_id) + .await + { + warn!( + client_id = %session.client_id, + "Failed to discard the inflight messages of the replaced session: {e}" + ); + } } + Ok(()) + } - let mut unauthorized_filters = Vec::new(); - for (topic_filter, stored) in &session.subscriptions { + async fn drop_unauthorized_subscriptions(&self, session: &mut ClientSession) { + let mut unauthorized = Vec::new(); + for topic_filter in session.subscriptions.keys() { let authorized = self .auth_provider - .authorize_subscribe(&connect.client_id, self.user_id.as_deref(), topic_filter) + .authorize_subscribe(&session.client_id, self.user_id.as_deref(), topic_filter) .await; if !authorized { warn!( - client_id = %connect.client_id, + client_id = %session.client_id, topic_filter = %topic_filter, "Dropping subscription on session restore: no longer authorized" ); - unauthorized_filters.push(topic_filter.clone()); - continue; + unauthorized.push(topic_filter.clone()); } - self.router - .subscribe( - SubscriptionRequest::new( - connect.client_id.clone(), - topic_filter.clone(), - stored.qos, - ) - .with_subscription_id(stored.subscription_id) - .with_no_local(stored.no_local) - .with_retain_as_published(stored.retain_as_published) - .with_retain_handling(stored.retain_handling) - .with_protocol_version( - ProtocolVersion::try_from(self.protocol_version).unwrap_or_default(), - ) - .with_change_only(stored.change_only), - ) - .await?; } - for filter in &unauthorized_filters { + for filter in &unauthorized { session.subscriptions.remove(filter); } + } +} - self.router - .load_change_only_state(&connect.client_id, session.change_only_state.clone()) - .await; +#[cfg(test)] +mod tests { + use super::super::ClientHandler; + use crate::broker::auth::AllowAllAuthProvider; + use crate::broker::config::BrokerConfig; + use crate::broker::resource_monitor::{ResourceLimits, ResourceMonitor}; + use crate::broker::router::{DeliveryLanes, MessageRouter}; + use crate::broker::storage::{DynamicStorage, MemoryBackend}; + use crate::broker::sys_topics::BrokerStats; + use crate::broker::transport::BrokerTransport; + use crate::packet::connect::ConnectPacket; + use crate::time::Duration; + use std::sync::Arc; + use tokio::net::{TcpListener, TcpStream}; + use tokio::sync::{broadcast, mpsc, oneshot}; + + #[tokio::test] + async fn accepted_session_cancels_pending_will_before_connack() { + let storage = Arc::new(DynamicStorage::Memory(MemoryBackend::new())); + let router = Arc::new(MessageRouter::with_storage(Arc::clone(&storage))); + + let (qos1_tx, _qos1_rx) = mpsc::channel(4); + let (qos0_tx, _qos0_rx) = mpsc::channel(4); + let (disconnect_tx, _disconnect_rx) = oneshot::channel(); + let departing = router + .register_client( + "early-cancel".to_string(), + DeliveryLanes { qos1_tx, qos0_tx }, + router.queue_handle("early-cancel"), + disconnect_tx, + ) + .await + .generation; + let cancelled = router + .arm_will("early-cancel", departing) + .await + .expect("the departing connection owns its entry"); + router.release_client("early-cancel", departing, true).await; + + let listener = TcpListener::bind("127.0.0.1:0").await.expect("bind"); + let addr = listener.local_addr().expect("local addr"); + let _client = TcpStream::connect(addr).await.expect("connect"); + let (server, peer) = listener.accept().await.expect("accept"); + let (_shutdown_tx, shutdown_rx) = broadcast::channel(1); + let mut handler = ClientHandler::new( + BrokerTransport::tcp(server), + peer, + Arc::new(BrokerConfig::default()), + Arc::clone(&router), + Arc::new(AllowAllAuthProvider), + Some(storage), + Arc::new(BrokerStats::new()), + Arc::new(ResourceMonitor::new(ResourceLimits::default())), + shutdown_rx, + ); - session.will_message.clone_from(&connect.will); - session.will_delay_interval = connect - .will - .as_ref() - .and_then(|w| w.properties.will_delay_interval); + let connect = ConnectPacket::new( + crate::types::ConnectOptions::new("early-cancel") + .with_clean_start(false) + .with_session_expiry_interval(60) + .protocol_options, + ); + handler + .handle_session(&connect) + .await + .expect("session accepted"); + + let woken = tokio::time::timeout(Duration::from_secs(1), cancelled).await; + assert!( + matches!(woken, Ok(Ok(()))), + "an accepted session must cancel the pending Will before CONNACK, not only at registration" + ); + } - session.receive_maximum = self.client_receive_maximum; - session.user_id.clone_from(&self.user_id); + async fn claim_with_inflight(clean_start: bool) -> usize { + use crate::broker::storage::{ + ClientSession, InflightDirection, InflightMessage, InflightPhase, StorageBackend, + }; + let storage = Arc::new(DynamicStorage::Memory(MemoryBackend::new())); + let mut stored = ClientSession::new("held-inflight", true, Some(60)); + stored.mark_disconnected(crate::broker::storage::unix_millis_now()); + storage.store_session(stored).await.expect("store session"); + let mut publish = crate::packet::publish::PublishPacket::new( + "inflight/t".to_string(), + b"unacked".to_vec(), + crate::QoS::AtLeastOnce, + ); + publish.packet_id = Some(7); + storage + .store_inflight_message(InflightMessage::from_publish( + &publish, + "held-inflight".to_string(), + InflightDirection::Outbound, + InflightPhase::AwaitingPubrec, + )) + .await + .expect("store inflight"); + let router = Arc::new(MessageRouter::with_storage(Arc::clone(&storage))); + + let listener = TcpListener::bind("127.0.0.1:0").await.expect("bind"); + let addr = listener.local_addr().expect("local addr"); + let _client = TcpStream::connect(addr).await.expect("connect"); + let (server, peer) = listener.accept().await.expect("accept"); + let (_shutdown_tx, shutdown_rx) = broadcast::channel(1); + let mut handler = ClientHandler::new( + BrokerTransport::tcp(server), + peer, + Arc::new(BrokerConfig::default()), + router, + Arc::new(AllowAllAuthProvider), + Some(Arc::clone(&storage)), + Arc::new(BrokerStats::new()), + Arc::new(ResourceMonitor::new(ResourceLimits::default())), + shutdown_rx, + ); + let connect = ConnectPacket::new( + crate::types::ConnectOptions::new("held-inflight") + .with_clean_start(clean_start) + .with_session_expiry_interval(60) + .protocol_options, + ); + let present = handler.handle_session(&connect).await.expect("claim"); + assert_eq!(present, !clean_start); + storage + .get_inflight_messages("held-inflight") + .await + .expect("read inflight") + .len() + } - session.touch(); - storage.store_session(session.clone()).await?; - self.session = Some(session); - Ok(()) + #[tokio::test] + async fn resuming_claim_keeps_the_persisted_inflight_messages() { + assert_eq!( + claim_with_inflight(false).await, + 1, + "a resumed session lost its unacknowledged messages at the claim" + ); + } + + #[tokio::test] + async fn clean_start_claim_discards_the_persisted_inflight_messages() { + assert_eq!( + claim_with_inflight(true).await, + 0, + "a clean start kept the old session's unacknowledged messages past its claim" + ); } } diff --git a/crates/mqtt5/src/broker/client_handler/lifecycle.rs b/crates/mqtt5/src/broker/client_handler/lifecycle.rs index 9de25343..959e7074 100644 --- a/crates/mqtt5/src/broker/client_handler/lifecycle.rs +++ b/crates/mqtt5/src/broker/client_handler/lifecycle.rs @@ -1,3 +1,4 @@ +use crate::broker::storage::ClientSession; use crate::error::{MqttError, Result}; use crate::packet::disconnect::DisconnectPacket; use crate::packet::publish::PublishPacket; @@ -10,7 +11,39 @@ use tracing::{debug, warn}; use super::ClientHandler; impl ClientHandler { - pub(super) fn handle_disconnect(&mut self, disconnect: &DisconnectPacket) -> Result<()> { + pub(super) async fn handle_disconnect(&mut self, disconnect: &DisconnectPacket) -> Result<()> { + if let Some(requested) = disconnect.properties.get_session_expiry_interval() { + if self.connect_session_expiry == Some(0) && requested != 0 { + warn!( + client_id = ?self.client_id, + requested, + "Session Expiry Interval on DISCONNECT after 0 on CONNECT is a Protocol Error" + ); + self.disconnect_reason = Some(ReasonCode::ProtocolError); + let reply = DisconnectPacket::new(ReasonCode::ProtocolError); + self.write_to_client(crate::packet::Packet::Disconnect(reply)) + .await?; + return Err(MqttError::ProtocolError( + "Session Expiry Interval on DISCONNECT after 0 on CONNECT".to_string(), + )); + } + let granted = + ClientSession::granted_expiry(Some(requested), self.maximum_session_expiry()); + if let Some(session) = self.session.as_mut() { + session.expiry_interval = Some(granted); + } + if let Some(client_id) = self.client_id.clone() { + let slot = self.router.lock_session(&client_id).await; + let stored = self + .update_stored_session(move |session| session.expiry_interval = Some(granted)) + .await; + drop(slot); + if let Err(e) = stored { + warn!(client_id = %client_id, "Failed to store the DISCONNECT Session Expiry: {e}"); + } + } + } + self.disconnect_reason = Some(disconnect.reason_code); if disconnect.reason_code == ReasonCode::DisconnectWithWillMessage { @@ -33,7 +66,6 @@ impl ClientHandler { pub(super) async fn publish_will_message( &self, client_id: &str, - session_taken_over: bool, armed_will: Option>, ) { let Some(session) = self.session.as_ref() else { @@ -56,9 +88,9 @@ impl ClientHandler { if self.authorize_will(client_id, &publish).await { self.route_publish(&publish, None).await; } - if !session_taken_over { - self.router.clear_stored_will(client_id).await; - } + self.router + .clear_stored_will(client_id, session.connection_token) + .await; return; } @@ -213,3 +245,66 @@ mod tests { ); } } + +#[cfg(test)] +mod disconnect_tests { + use super::super::ClientHandler; + use crate::broker::auth::AllowAllAuthProvider; + use crate::broker::config::BrokerConfig; + use crate::broker::resource_monitor::{ResourceLimits, ResourceMonitor}; + use crate::broker::router::MessageRouter; + use crate::broker::storage::{DynamicStorage, MemoryBackend, StorageBackend}; + use crate::broker::sys_topics::BrokerStats; + use crate::broker::transport::BrokerTransport; + use crate::packet::connect::ConnectPacket; + use crate::packet::disconnect::DisconnectPacket; + use crate::protocol::v5::reason_codes::ReasonCode; + use std::sync::Arc; + use tokio::net::{TcpListener, TcpStream}; + use tokio::sync::broadcast; + + #[tokio::test] + async fn disconnect_session_expiry_is_stored_when_the_disconnect_is_processed() { + let storage = Arc::new(DynamicStorage::Memory(MemoryBackend::new())); + let router = Arc::new(MessageRouter::with_storage(Arc::clone(&storage))); + let listener = TcpListener::bind("127.0.0.1:0").await.expect("bind"); + let addr = listener.local_addr().expect("local addr"); + let _client = TcpStream::connect(addr).await.expect("connect"); + let (server, peer) = listener.accept().await.expect("accept"); + let (_shutdown_tx, shutdown_rx) = broadcast::channel(1); + let mut handler = ClientHandler::new( + BrokerTransport::tcp(server), + peer, + Arc::new(BrokerConfig::default()), + Arc::clone(&router), + Arc::new(AllowAllAuthProvider), + Some(Arc::clone(&storage)), + Arc::new(BrokerStats::new()), + Arc::new(ResourceMonitor::new(ResourceLimits::default())), + shutdown_rx, + ); + let connect = ConnectPacket::new( + crate::types::ConnectOptions::new("ending") + .with_session_expiry_interval(3600) + .protocol_options, + ); + handler.protocol_version = 5; + handler.client_id = Some("ending".to_string()); + handler.handle_session(&connect).await.expect("claim"); + + let mut disconnect = DisconnectPacket::new(ReasonCode::Success); + disconnect.properties.set_session_expiry_interval(0); + assert!(handler.handle_disconnect(&disconnect).await.is_err()); + + let stored = storage + .get_session("ending") + .await + .expect("read") + .expect("session of the live connection"); + assert_eq!( + stored.expiry_interval, + Some(0), + "a claim racing the release must see that the session ended at the DISCONNECT" + ); + } +} diff --git a/crates/mqtt5/src/broker/client_handler/mod.rs b/crates/mqtt5/src/broker/client_handler/mod.rs index 9feee776..d0b86ea3 100644 --- a/crates/mqtt5/src/broker/client_handler/mod.rs +++ b/crates/mqtt5/src/broker/client_handler/mod.rs @@ -9,10 +9,11 @@ use crate::broker::config::BrokerConfig; use crate::broker::events::{ClientConnectEvent, ClientDisconnectEvent}; use crate::broker::resource_monitor::ResourceMonitor; use crate::broker::router::{ - DeliveryLanes, MessageRouter, Release, RoutableMessage, TakeoverNotice, ROUTE_BUDGET_MAX, + MessageRouter, Release, RoutableMessage, TakeoverNotice, ROUTE_BUDGET_MAX, }; use crate::broker::storage::{ - ClientSession, DynamicStorage, InflightDirection, QueueHandle, QueuedMessage, StorageBackend, + unix_millis_now, ClientSession, DynamicStorage, InflightDirection, QueueHandle, QueuedMessage, + StorageBackend, }; use crate::broker::sys_topics::BrokerStats; use crate::broker::transport::BrokerTransport; @@ -36,7 +37,7 @@ use tokio::time::{interval, timeout, timeout_at, Instant, Interval}; use tracing::{debug, info, warn}; /// Longest a new connection waits for the handler it displaced to hand the session over. -const HANDOFF_BOUND: Duration = Duration::from_secs(30); +pub(super) const HANDOFF_BOUND: Duration = Duration::from_secs(30); /// Why the packet loop returned. #[derive(Debug)] @@ -96,7 +97,9 @@ pub struct ClientHandler { pub(super) window: u16, pub(super) generation: u64, pub(super) bound: bool, + pub(super) counted_as_connected: bool, pub(super) released_rx: Option>, + pub(super) disconnect_rx: Option>, pub(super) handoff_deadline: Option, pub(super) handoff_waived: bool, pub(super) handoff_baseline: usize, @@ -106,14 +109,18 @@ pub struct ClientHandler { pub(super) inflight_order: VecDeque, pub(super) inflight_publishes: HashMap, pub(super) session: Option, + pub(super) connect_session_expiry: Option, + pub(super) advertised_session_expiry: Option, pub(super) next_packet_id: u16, pub(super) normal_disconnect: bool, pub(super) disconnect_reason: Option, pub(super) request_problem_information: bool, pub(super) request_response_information: bool, pub(super) auth_method: Option, + pub(super) connack_auth_data: Option>, pub(super) auth_state: AuthState, pub(super) pending_connect: Option, + pub(super) authenticated_connect: Option, pub(super) topic_aliases: HashMap, pub(super) external_packet_rx: Option)>>, pub(super) pending_external_flow_id: Option, @@ -208,7 +215,9 @@ impl ClientHandler { window, generation: 0, bound: false, + counted_as_connected: false, released_rx: None, + disconnect_rx: None, handoff_deadline: None, handoff_waived: false, handoff_baseline: 0, @@ -218,14 +227,18 @@ impl ClientHandler { inflight_order: VecDeque::new(), inflight_publishes: HashMap::new(), session: None, + connect_session_expiry: None, + advertised_session_expiry: None, next_packet_id: 1, normal_disconnect: false, disconnect_reason: None, request_problem_information: true, request_response_information: false, auth_method: None, + connack_auth_data: None, auth_state: AuthState::NotStarted, pending_connect: None, + authenticated_connect: None, topic_aliases: HashMap::new(), external_packet_rx, pending_external_flow_id: None, @@ -323,35 +336,23 @@ impl ClientHandler { /// /// Panics if `client_id` is None after successful connection pub async fn run(mut self) -> Result<()> { - let client_id = self.perform_connect_handshake().await?; - - let (disconnect_tx, mut disconnect_rx) = oneshot::channel(); + let handshake = self.perform_connect_handshake().await; + let (Some(client_id), Some(mut disconnect_rx), Some(queue)) = ( + self.client_id.clone().filter(|_| self.generation != 0), + self.disconnect_rx.take(), + self.queue.clone(), + ) else { + return handshake.map(|_| ()); + }; + if let Err(e) = handshake { + self.abandon_claim(&client_id, &queue).await; + return Err(e); + } - let queue = self.router.queue_handle(&client_id); - self.queue = Some(Arc::clone(&queue)); self.window = self .client_receive_maximum .min(self.config.max_outbound_inflight) .max(1); - let registration = self - .router - .register_session( - client_id.clone(), - DeliveryLanes { - qos1_tx: self.qos1_tx.clone(), - qos0_tx: self.qos0_tx.clone(), - }, - Arc::clone(&queue), - disconnect_tx, - self.clean_start, - ) - .await; - self.generation = registration.generation; - self.handoff_deadline = registration - .released - .as_ref() - .map(|_| Instant::now() + HANDOFF_BOUND); - self.released_rx = registration.released; self.fire_connect_event(&client_id).await; @@ -360,28 +361,27 @@ impl ClientHandler { Err(e) => (Err(e), LoopExit::Closed), }; - let (taken_over, armed_will) = if let LoopExit::TakenOver(notice) = exit { + let armed_will = if let LoopExit::TakenOver(notice) = exit { self.hand_off(&queue, notice).await; - self.release_router_entry(&client_id).await; - (true, None) + self.release_ownership(&client_id, false).await; + None } else { if self.session_preserved() { self.requeue_unsent(&queue).await; } else { self.drop_unsent().await; } - let armed_will = self.arm_delayed_will(&client_id).await; - match self.release_router_entry(&client_id).await { - Release::Owned => { + match self.release_ownership(&client_id, true).await { + (Release::Owned, armed_will) => { queue.finish_drain(); if self.session_preserved() { queue.notify(); } else { queue.clear(None); } - (false, armed_will) + armed_will } - Release::Displaced => { + (Release::Displaced, armed_will) => { if let Ok(notice) = disconnect_rx.try_recv() { queue.finish_drain(); let TakeoverNotice { @@ -395,19 +395,44 @@ impl ClientHandler { } else { queue.finish_drain(); } - (true, armed_will) + armed_will } } }; - self.handle_disconnect_cleanup(&client_id, taken_over, armed_will) - .await; + self.handle_disconnect_cleanup(&client_id, armed_will).await; info!("Client {} disconnected", client_id); result } + async fn abandon_claim(&mut self, client_id: &str, queue: &QueueHandle) { + let (release, _) = self.release_ownership(client_id, false).await; + if matches!(release, Release::Owned) && !self.session_preserved() { + queue.clear(None); + } + } + + async fn release_ownership( + &mut self, + client_id: &str, + arm_will: bool, + ) -> (Release, Option>) { + let slot = self.router.lock_session(client_id).await; + let armed_will = if arm_will { + self.arm_delayed_will(client_id).await + } else { + None + }; + let release = self.release_router_entry(client_id).await; + if matches!(release, Release::Owned) { + self.persist_session_end(client_id).await; + } + drop(slot); + (release, armed_will) + } + async fn serve( &mut self, disconnect_rx: &mut oneshot::Receiver, @@ -494,41 +519,47 @@ impl ClientHandler { "Waiting for CONNECT packet with {}s timeout", connect_timeout.as_secs() ); - match timeout(connect_timeout, self.wait_for_connect()).await { - Ok(Ok(())) => { - let client_id = self.client_id.as_ref().unwrap().clone(); - info!( - "Client {} connected from {} ({})", - client_id, - self.client_addr, - self.transport.transport_type() - ); - if let Some(cert_info) = self.transport.client_cert_info() { - debug!("Client certificate: {}", cert_info); - } - - self.resource_monitor - .register_connection(client_id.clone(), self.client_addr.ip()) - .await; - - self.stats.client_connected(); - Ok(client_id) - } - Ok(Err(e)) => { - if e.to_string().contains("Connection closed") { - info!("Client disconnected during connect phase: {e}"); - tracing::debug!("Connection closed error details: {:?}", e); - } else { - warn!("Connect error: {e}"); - tracing::debug!("Connect error details: {:?}", e); - } - Err(e) - } + let negotiated = timeout(connect_timeout, self.negotiate_connect()).await; + let completed = match negotiated { + Ok(Ok(accepted)) => self.complete_connect(accepted).await, + Ok(Err(e)) => Err(e), Err(_) => { warn!("Connect timeout from {}", self.client_addr); - Err(MqttError::Timeout) + return Err(MqttError::Timeout); + } + }; + if let Err(e) = completed { + if e.to_string().contains("Connection closed") { + info!("Client disconnected during connect phase: {e}"); + tracing::debug!("Connection closed error details: {:?}", e); + } else { + warn!("Connect error: {e}"); + tracing::debug!("Connect error details: {:?}", e); } + return Err(e); + } + let Some(client_id) = self.client_id.clone() else { + return Err(MqttError::ProtocolError( + "CONNECT completed without a client identifier".to_string(), + )); + }; + info!( + "Client {} connected from {} ({})", + client_id, + self.client_addr, + self.transport.transport_type() + ); + if let Some(cert_info) = self.transport.client_cert_info() { + debug!("Client certificate: {}", cert_info); } + + self.resource_monitor + .register_connection(client_id.clone(), self.client_addr.ip()) + .await; + + self.stats.client_connected(); + self.counted_as_connected = true; + Ok(client_id) } async fn fire_connect_event(&self, client_id: &str) { @@ -578,7 +609,6 @@ impl ClientHandler { async fn handle_disconnect_cleanup( &mut self, client_id: &str, - session_taken_over: bool, armed_will: Option>, ) { #[cfg(feature = "opentelemetry")] @@ -588,29 +618,24 @@ impl ClientHandler { "mqtt.disconnect", mqtt.client_id = %client_id, ); - self.handle_disconnect_cleanup_inner(client_id, session_taken_over, armed_will) + self.handle_disconnect_cleanup_inner(client_id, armed_will) .instrument(span) .await; } #[cfg(not(feature = "opentelemetry"))] - self.handle_disconnect_cleanup_inner(client_id, session_taken_over, armed_will) + self.handle_disconnect_cleanup_inner(client_id, armed_will) .await; } async fn handle_disconnect_cleanup_inner( &mut self, client_id: &str, - session_taken_over: bool, armed_will: Option>, ) { self.resource_monitor .unregister_connection(client_id, self.client_addr.ip()) .await; - if !session_taken_over { - self.cleanup_session_storage(client_id).await; - } - if let Some(ref user_id) = self.user_id { self.auth_provider.cleanup_session(user_id).await; } @@ -626,21 +651,18 @@ impl ClientHandler { mqtt.client_id = %client_id, mqtt.topic = %will.topic, ); - self.publish_will_message(client_id, session_taken_over, armed_will) + self.publish_will_message(client_id, armed_will) .instrument(span) .await; } else { - self.publish_will_message(client_id, session_taken_over, armed_will) - .await; + self.publish_will_message(client_id, armed_will).await; } } else { - self.publish_will_message(client_id, session_taken_over, armed_will) - .await; + self.publish_will_message(client_id, armed_will).await; } } #[cfg(not(feature = "opentelemetry"))] - self.publish_will_message(client_id, session_taken_over, armed_will) - .await; + self.publish_will_message(client_id, armed_will).await; } self.fire_disconnect_event(client_id).await; @@ -661,40 +683,44 @@ impl ClientHandler { release } - async fn cleanup_session_storage(&self, client_id: &str) { - if let Some(ref storage) = self.storage { - if let Some(ref session) = self.session { - match storage.get_session(client_id).await { - Ok(Some(mut stored_session)) => { - stored_session.touch(); - if self.normal_disconnect { - stored_session.will_message = None; - stored_session.will_delay_interval = None; - } - if let Err(e) = storage.store_session(stored_session).await { - warn!("Failed to store session for {client_id}: {e}"); - } - } - Ok(None) => {} - Err(e) => { - warn!("Failed to get session for {client_id}: {e}"); - } - } - - if session.expiry_interval == Some(0) { - if let Err(e) = storage.remove_session(client_id).await { - warn!("Failed to remove session for {client_id}: {e}"); - } + async fn persist_session_end(&self, client_id: &str) { + let (Some(storage), Some(session)) = (self.storage.as_ref(), self.session.as_ref()) else { + return; + }; + if session.expiry_interval == Some(0) { + match storage + .remove_owned_session(client_id, self.generation) + .await + { + Ok(true) => { storage.queue_handle(client_id).clear(None); if let Err(e) = storage.remove_all_inflight_messages(client_id).await { warn!("Failed to remove inflight messages for {client_id}: {e}"); } - debug!( - "Removed session, queued, and inflight messages for client {}", - client_id - ); + debug!(client_id, "Removed session, queued, and inflight messages"); } + Ok(false) => debug!(client_id, "Stored session is not this connection's"), + Err(e) => warn!("Failed to remove session for {client_id}: {e}"), } + return; + } + let expiry_interval = session.expiry_interval; + let discard_will = self.normal_disconnect; + let disconnected_at = unix_millis_now(); + let updated = storage + .update_session(client_id, self.generation, |stored| { + stored.mark_disconnected(disconnected_at); + stored.expiry_interval = expiry_interval; + if discard_will { + stored.will_message = None; + stored.will_delay_interval = None; + } + }) + .await; + match updated { + Ok(true) => {} + Ok(false) => debug!(client_id, "Stored session is not this connection's"), + Err(e) => warn!("Failed to update session for {client_id}: {e}"), } } @@ -769,31 +795,50 @@ impl ClientHandler { } } - async fn wait_for_connect(&mut self) -> Result<()> { + async fn negotiate_connect(&mut self) -> Result { let max_size = self.max_packet_size(); let packet = read_packet_reusing_buffer(&mut self.transport, 5, &mut self.read_buffer, max_size) .await?; - - match packet { - Packet::Connect(connect) => { - #[cfg(feature = "opentelemetry")] - { - use tracing::Instrument; - let span = tracing::info_span!( - "mqtt.connect", - mqtt.client_id = %connect.client_id, - mqtt.clean_start = connect.clean_start, - mqtt.protocol_version = connect.protocol_version, - ); - self.handle_connect(*connect).instrument(span).await - } - #[cfg(not(feature = "opentelemetry"))] - self.handle_connect(*connect).await - } - _ => Err(MqttError::ProtocolError( + let Packet::Connect(connect) = packet else { + return Err(MqttError::ProtocolError( "Expected CONNECT packet".to_string(), - )), + )); + }; + #[cfg(feature = "opentelemetry")] + let accepted = { + use tracing::Instrument; + let span = tracing::info_span!( + "mqtt.connect", + mqtt.client_id = %connect.client_id, + mqtt.clean_start = connect.clean_start, + mqtt.protocol_version = connect.protocol_version, + ); + self.handle_connect(*connect).instrument(span).await? + }; + #[cfg(not(feature = "opentelemetry"))] + let accepted = self.handle_connect(*connect).await?; + if let Some(accepted) = accepted { + return Ok(accepted); + } + loop { + let max_size = self.max_packet_size(); + let packet = read_packet_reusing_buffer( + &mut self.transport, + self.protocol_version, + &mut self.read_buffer, + max_size, + ) + .await?; + let Packet::Auth(auth) = packet else { + return Err(MqttError::ProtocolError( + "Only AUTH may follow CONNECT before CONNACK".to_string(), + )); + }; + self.handle_auth(auth).await?; + if let Some(accepted) = self.authenticated_connect.take() { + return Ok(accepted); + } } } @@ -1307,7 +1352,7 @@ impl ClientHandler { Ok(()) } Packet::PingReq => self.handle_pingreq().await, - Packet::Disconnect(disconnect) => self.handle_disconnect(&disconnect), + Packet::Disconnect(disconnect) => self.handle_disconnect(&disconnect).await, Packet::Auth(auth) => self.handle_auth(auth).await, _ => { warn!("Unexpected packet type"); @@ -1319,8 +1364,8 @@ impl ClientHandler { impl Drop for ClientHandler { fn drop(&mut self) { - if let Some(ref client_id) = self.client_id { - debug!("Client handler dropped for {}", client_id); + if self.counted_as_connected { + debug!(client_id = ?self.client_id, "Client handler dropped"); self.stats.client_disconnected(); } } diff --git a/crates/mqtt5/src/broker/client_handler/subscribe.rs b/crates/mqtt5/src/broker/client_handler/subscribe.rs index 3519ec4c..cd78b677 100644 --- a/crates/mqtt5/src/broker/client_handler/subscribe.rs +++ b/crates/mqtt5/src/broker/client_handler/subscribe.rs @@ -10,28 +10,105 @@ use crate::packet::unsuback::UnsubAckPacket; use crate::packet::unsubscribe::UnsubscribePacket; use crate::packet::Packet; use crate::protocol::v5::reason_codes::ReasonCode; -use crate::types::ProtocolVersion; use crate::validation::{parse_shared_subscription, topic_matches_filter, validate_topic_filter}; use crate::QoS; use tracing::{debug, warn}; -use crate::broker::router::{RoutableMessage, Subscribed, SubscriptionRequest, Unsubscribed}; +use crate::broker::router::{MessageRouter, RoutableMessage, Subscribed, Unsubscribed}; +use crate::broker::session_slot::SessionSlotGuard; use super::ClientHandler; +struct AcceptedFilter<'a> { + filter: &'a crate::packet::subscribe::TopicFilter, + stored: StoredSubscription, + is_new: bool, +} + impl ClientHandler { pub(super) async fn handle_subscribe(&mut self, subscribe: SubscribePacket) -> Result<()> { - let client_id = self.client_id.clone().unwrap(); - let mut reason_codes: Vec = Vec::new(); - + let client_id = self.connected_client_id()?; + let mut reason_codes = Vec::with_capacity(subscribe.filters.len()); for filter in &subscribe.filters { - if let Some(rc) = self.validate_subscribe_filter(filter, &client_id).await? { - reason_codes.push(rc); - continue; + reason_codes.push(self.validate_subscribe_filter(filter, &client_id).await?); + } + + let slot = self.router.lock_session(&client_id).await; + let routed = match self + .route_subscriptions(&subscribe, &client_id, &mut reason_codes) + .await + { + Ok(Some(routed)) => routed, + Ok(None) => { + debug!("Ignoring SUBSCRIBE from a connection whose session was taken over"); + return Ok(()); + } + Err(e) => { + self.restore_stored_routes(&slot).await?; + return Err(e); } + }; + if !routed.is_empty() { + let stored: Vec<(String, StoredSubscription)> = routed + .iter() + .map(|accepted| (accepted.filter.filter.clone(), accepted.stored.clone())) + .collect(); + let kept = stored.clone(); + self.persist_or_restore_routes(&slot, move |session| { + for (filter, subscription) in stored { + session.add_subscription(filter, subscription); + } + }) + .await?; + if let Some(session) = self.session.as_mut() { + for (filter, subscription) in kept { + session.add_subscription(filter, subscription); + } + } + } + drop(slot); + + let subscription_id = subscribe.properties.get_subscription_identifier(); + for accepted in routed { + #[cfg(all(not(target_arch = "wasm32"), feature = "transport-quic"))] + if let Some(fid) = accepted.stored.flow_id { + self.track_flow_subscription(fid, &accepted.filter.filter) + .await; + } + self.deliver_retained_for_filter( + &accepted.filter.filter, + &accepted.filter.options, + subscription_id, + accepted.is_new, + ) + .await?; + } - if let Some(rc) = self.check_subscription_quota(filter, &client_id).await { - reason_codes.push(rc); + let reason_codes: Vec = + reason_codes.into_iter().flatten().collect(); + self.build_and_send_suback(&subscribe, &reason_codes).await + } + + fn connected_client_id(&self) -> Result { + self.client_id + .clone() + .ok_or_else(|| MqttError::ProtocolError("packet received before CONNECT".to_string())) + } + + async fn route_subscriptions<'a>( + &self, + subscribe: &'a SubscribePacket, + client_id: &str, + reason_codes: &mut [Option], + ) -> Result>>> { + let subscription_id = subscribe.properties.get_subscription_identifier(); + let mut routed = Vec::new(); + for (filter, reason_code) in subscribe.filters.iter().zip(reason_codes.iter_mut()) { + if reason_code.is_some() { + continue; + } + if let Some(rc) = self.check_subscription_quota(filter, client_id).await { + *reason_code = Some(rc); continue; } @@ -49,64 +126,38 @@ impl ClientHandler { .iter() .any(|pattern| topic_matches_filter(&filter.filter, pattern)); - let flow_id = self.pending_external_flow_id; + let stored = StoredSubscription { + qos: QoS::from(granted_qos), + no_local: filter.options.no_local, + retain_as_published: filter.options.retain_as_published, + retain_handling: filter.options.retain_handling as u8, + subscription_id, + protocol_version: self.protocol_version, + change_only, + flow_id: self.pending_external_flow_id, + }; let outcome = self .router .subscribe_as( Some(self.generation), - SubscriptionRequest::new( - client_id.clone(), - filter.filter.clone(), - QoS::from(granted_qos), - ) - .with_subscription_id(subscribe.properties.get_subscription_identifier()) - .with_no_local(filter.options.no_local) - .with_retain_as_published(filter.options.retain_as_published) - .with_retain_handling(filter.options.retain_handling as u8) - .with_protocol_version( - ProtocolVersion::try_from(self.protocol_version).unwrap_or_default(), - ) - .with_change_only(change_only) - .with_flow_id(flow_id), + MessageRouter::stored_subscription_request(client_id, &filter.filter, &stored), ) .await?; let is_new = match outcome { Subscribed::New => true, Subscribed::Updated => false, - Subscribed::Fenced => { - debug!("Ignoring SUBSCRIBE from a connection whose session was taken over"); - return Ok(()); - } + Subscribed::Fenced => return Ok(None), }; - - self.persist_subscription( - &filter.filter, - &filter.options, - granted_qos, - subscribe.properties.get_subscription_identifier(), - change_only, - ) - .await; - - #[cfg(all(not(target_arch = "wasm32"), feature = "transport-quic"))] - if let Some(fid) = flow_id { - self.track_flow_subscription(fid, &filter.filter).await; - } - - self.deliver_retained_for_filter( - &filter.filter, - &filter.options, - subscribe.properties.get_subscription_identifier(), - is_new, - ) - .await?; - - reason_codes.push(crate::packet::suback::SubAckReasonCode::from_qos( - QoS::from(granted_qos), + *reason_code = Some(crate::packet::suback::SubAckReasonCode::from_qos( + stored.qos, )); + routed.push(AcceptedFilter { + filter, + stored, + is_new, + }); } - - self.build_and_send_suback(&subscribe, &reason_codes).await + Ok(Some(routed)) } async fn validate_subscribe_filter( @@ -199,32 +250,61 @@ impl ClientHandler { None } - async fn persist_subscription( + async fn persist_or_restore_routes( &mut self, - topic_filter: &str, - options: &crate::packet::subscribe::SubscriptionOptions, - granted_qos: u8, - subscription_id: Option, - change_only: bool, - ) { - if let Some(ref mut session) = self.session { - let stored = StoredSubscription { - qos: QoS::from(granted_qos), - no_local: options.no_local, - retain_as_published: options.retain_as_published, - retain_handling: options.retain_handling as u8, - subscription_id, - protocol_version: self.protocol_version, - change_only, - flow_id: self.pending_external_flow_id, - }; - session.add_subscription(topic_filter.to_string(), stored); - if let Some(ref storage) = self.storage { - if let Err(e) = storage.store_session(session.clone()).await { - warn!("Failed to store session: {e}"); - } + slot: &SessionSlotGuard, + update: F, + ) -> Result<()> + where + F: FnOnce(&mut crate::broker::storage::ClientSession) + Send, + { + let Err(e) = self.update_stored_session(update).await else { + return Ok(()); + }; + warn!( + client_id = slot.client_id(), + "Failed to store a subscription change; restoring the stored subscriptions: {e}" + ); + self.restore_stored_routes(slot).await?; + self.disconnect_reason = Some(ReasonCode::UnspecifiedError); + if self.protocol_version == 5 { + let disconnect = DisconnectPacket::new(ReasonCode::UnspecifiedError); + if let Err(write_error) = self.write_to_client(Packet::Disconnect(disconnect)).await { + debug!("Failed to send DISCONNECT after a failed session write: {write_error}"); } } + Err(e) + } + + async fn restore_stored_routes(&mut self, slot: &SessionSlotGuard) -> Result<()> { + let stored = match self.storage.as_ref() { + Some(storage) => storage.get_session(slot.client_id()).await?, + None => None, + }; + self.router + .set_client_subscriptions(slot, stored.as_ref()) + .await?; + if let (Some(session), Some(stored)) = (self.session.as_mut(), stored) { + session.subscriptions = stored.subscriptions; + } + Ok(()) + } + + pub(super) async fn update_stored_session(&self, update: F) -> Result<()> + where + F: FnOnce(&mut crate::broker::storage::ClientSession) + Send, + { + let (Some(storage), Some(client_id)) = (self.storage.as_ref(), self.client_id.as_ref()) + else { + return Ok(()); + }; + if !storage + .update_session(client_id, self.generation, update) + .await? + { + debug!(client_id = %client_id, "Stored session is not this connection's"); + } + Ok(()) } async fn deliver_retained_for_filter( @@ -305,7 +385,7 @@ impl ClientHandler { subscribe: &SubscribePacket, reason_codes: &[crate::packet::suback::SubAckReasonCode], ) -> Result<()> { - let client_id = self.client_id.clone().unwrap(); + let client_id = self.connected_client_id()?; let mut suback = if self.protocol_version == 4 { SubAckPacket::new_v311(subscribe.packet_id) } else { @@ -351,9 +431,11 @@ impl ClientHandler { return; }; + let client_id = client_id.clone(); + let slot = self.router.lock_session(&client_id).await; let removed_filters = self .router - .unsubscribe_by_flow(Some(self.generation), client_id, flow_id) + .unsubscribe_by_flow(Some(self.generation), &client_id, flow_id) .await; if removed_filters.is_empty() { return; @@ -366,15 +448,21 @@ impl ClientHandler { "Removed flow-bound subscriptions on flow close" ); - if let Some(ref mut session) = self.session { + if let Some(session) = self.session.as_mut() { for filter in &removed_filters { session.remove_subscription(filter); } - if let Some(ref storage) = self.storage { - if let Err(e) = storage.store_session(session.clone()).await { - warn!("Failed to store session after flow close: {e}"); + } + let persisted = self + .update_stored_session(move |session| { + for filter in &removed_filters { + session.remove_subscription(filter); } - } + }) + .await; + drop(slot); + if let Err(e) = persisted { + warn!("Failed to store session after flow close: {e}"); } #[cfg(all(not(target_arch = "wasm32"), feature = "transport-quic"))] @@ -409,11 +497,13 @@ impl ClientHandler { &mut self, unsubscribe: UnsubscribePacket, ) -> Result<()> { - let client_id = self.client_id.clone().unwrap(); - let mut reason_codes = Vec::new(); + let client_id = self.connected_client_id()?; + let mut reason_codes = Vec::with_capacity(unsubscribe.filters.len()); + let mut removed_filters = Vec::new(); + let slot = self.router.lock_session(&client_id).await; for topic_filter in &unsubscribe.filters { - let removed = match self + match self .router .unsubscribe_as( Some(self.generation), @@ -423,35 +513,39 @@ impl ClientHandler { ) .await { - Unsubscribed::Removed => true, - Unsubscribed::Absent => false, + Unsubscribed::Removed => { + removed_filters.push(topic_filter.clone()); + reason_codes.push(crate::packet::unsuback::UnsubAckReasonCode::Success); + } + Unsubscribed::Absent => reason_codes + .push(crate::packet::unsuback::UnsubAckReasonCode::NoSubscriptionExisted), Unsubscribed::Fenced => { debug!("Ignoring UNSUBSCRIBE from a connection whose session was taken over"); return Ok(()); } - }; - - if removed { - if let Some(ref mut session) = self.session { - session.remove_subscription(topic_filter); - if let Some(ref storage) = self.storage { - if let Err(e) = storage.store_session(session.clone()).await { - warn!("Failed to store session: {e}"); - } - } + } + } + if !removed_filters.is_empty() { + let removed = removed_filters.clone(); + self.persist_or_restore_routes(&slot, move |session| { + for filter in &removed { + session.remove_subscription(filter); } - - #[cfg(all(not(target_arch = "wasm32"), feature = "transport-quic"))] - if let Some(fid) = self.pending_external_flow_id { - self.untrack_flow_subscription(fid, topic_filter).await; + }) + .await?; + if let Some(session) = self.session.as_mut() { + for filter in &removed_filters { + session.remove_subscription(filter); } } + } + drop(slot); - reason_codes.push(if removed { - crate::packet::unsuback::UnsubAckReasonCode::Success - } else { - crate::packet::unsuback::UnsubAckReasonCode::NoSubscriptionExisted - }); + #[cfg(all(not(target_arch = "wasm32"), feature = "transport-quic"))] + if let Some(fid) = self.pending_external_flow_id { + for topic_filter in &removed_filters { + self.untrack_flow_subscription(fid, topic_filter).await; + } } let mut unsuback = if self.protocol_version == 4 { diff --git a/crates/mqtt5/src/broker/config/mod.rs b/crates/mqtt5/src/broker/config/mod.rs index fcf02638..e859a868 100644 --- a/crates/mqtt5/src/broker/config/mod.rs +++ b/crates/mqtt5/src/broker/config/mod.rs @@ -78,6 +78,7 @@ pub struct BrokerConfig { pub bind_addresses: Vec, pub max_clients: usize, #[cfg_attr(not(target_arch = "wasm32"), serde(with = "humantime_serde"))] + #[serde(skip_serializing_if = "is_unlimited_session_expiry")] pub session_expiry_interval: Duration, pub max_packet_size: usize, pub topic_alias_maximum: u16, @@ -213,7 +214,7 @@ impl Default for BrokerConfig { "[::]:1883".parse().unwrap(), ], max_clients: 10000, - session_expiry_interval: Duration::from_secs(3600), + session_expiry_interval: Duration::from_secs(u64::from(u32::MAX)), max_packet_size: 268_435_456, topic_alias_maximum: 65535, retain_available: true, @@ -496,6 +497,10 @@ impl BrokerConfig { } } +fn is_unlimited_session_expiry(interval: &Duration) -> bool { + *interval >= Duration::from_secs(u64::from(u32::MAX)) +} + #[cfg(test)] mod tests { use super::*; diff --git a/crates/mqtt5/src/broker/mod.rs b/crates/mqtt5/src/broker/mod.rs index c417e26d..80a69738 100644 --- a/crates/mqtt5/src/broker/mod.rs +++ b/crates/mqtt5/src/broker/mod.rs @@ -44,6 +44,9 @@ pub mod router; pub mod server; #[cfg(all(not(target_arch = "wasm32"), feature = "transport-quic"))] mod server_stream_manager; +#[cfg(all(test, not(target_arch = "wasm32")))] +mod session_fault_tests; +pub mod session_slot; pub mod storage; pub mod sys_topics; #[cfg(not(target_arch = "wasm32"))] diff --git a/crates/mqtt5/src/broker/router.rs b/crates/mqtt5/src/broker/router.rs index 4667d466..15713b02 100644 --- a/crates/mqtt5/src/broker/router.rs +++ b/crates/mqtt5/src/broker/router.rs @@ -1,9 +1,10 @@ #[cfg(not(target_arch = "wasm32"))] use crate::broker::bridge::BridgeManager; use crate::broker::events::{BrokerEventHandler, RetainedSetEvent}; +use crate::broker::session_slot::{SessionSlotGuard, SessionSlots}; use crate::broker::storage::{ - ChangeOnlyState, DynamicStorage, QueueHandle, QueueLimits, QueueRegistry, QueuedMessage, - RetainedMessage, StorageBackend, + unix_millis_now, ChangeOnlyState, ClientSession, DynamicStorage, QueueHandle, QueueLimits, + QueueRegistry, QueuedMessage, RetainedMessage, StorageBackend, StoredSubscription, }; use crate::packet::publish::PublishPacket; use crate::types::ProtocolVersion; @@ -181,6 +182,7 @@ pub struct MessageRouter { fallback_queues: QueueRegistry, next_generation: std::sync::atomic::AtomicU64, pending_wills: parking_lot::Mutex>, + session_slots: SessionSlots, } struct PendingWill { @@ -188,6 +190,13 @@ struct PendingWill { cancel: oneshot::Sender<()>, } +const GENERATIONS_PER_MILLISECOND: u64 = 1 << 20; +const RECOVERY_CONCURRENCY: usize = 1024; + +fn generation_epoch() -> u64 { + unix_millis_now().saturating_mul(GENERATIONS_PER_MILLISECOND) +} + /// Information about a connected client #[derive(Debug)] pub struct ClientInfo { @@ -347,8 +356,9 @@ impl MessageRouter { outbound_rates: parking_lot::RwLock::new(HashMap::new()), max_outbound_rate: AtomicU32::new(0), fallback_queues: QueueRegistry::new(QueueLimits::default(), None), - next_generation: std::sync::atomic::AtomicU64::new(0), + next_generation: std::sync::atomic::AtomicU64::new(generation_epoch()), pending_wills: parking_lot::Mutex::new(HashMap::new()), + session_slots: SessionSlots::default(), } } @@ -381,8 +391,9 @@ impl MessageRouter { outbound_rates: parking_lot::RwLock::new(HashMap::new()), max_outbound_rate: AtomicU32::new(0), fallback_queues: QueueRegistry::new(QueueLimits::default(), None), - next_generation: std::sync::atomic::AtomicU64::new(0), + next_generation: std::sync::atomic::AtomicU64::new(generation_epoch()), pending_wills: parking_lot::Mutex::new(HashMap::new()), + session_slots: SessionSlots::default(), } } @@ -492,6 +503,34 @@ impl MessageRouter { queue: QueueHandle, disconnect_tx: oneshot::Sender, clean_start: bool, + ) -> Registration { + let generation = self.allocate_generation(); + self.register_session_as( + generation, + client_id, + lanes, + queue, + disconnect_tx, + clean_start, + ) + .await + } + + #[must_use] + pub fn allocate_generation(&self) -> u64 { + self.next_generation + .fetch_add(1, std::sync::atomic::Ordering::Relaxed) + + 1 + } + + pub async fn register_session_as( + &self, + generation: u64, + client_id: String, + lanes: DeliveryLanes, + queue: QueueHandle, + disconnect_tx: oneshot::Sender, + clean_start: bool, ) -> Registration { let subscription_maps = if clean_start { let mut exact = self.exact_subscriptions.write().await; @@ -504,20 +543,7 @@ impl MessageRouter { }; let mut clients = self.clients.write().await; - let generation = self - .next_generation - .fetch_add(1, std::sync::atomic::Ordering::Relaxed) - + 1; - if let Some(pending) = self.pending_wills.lock().remove(&client_id) { - debug!( - client_id = %client_id, - armed_by = pending.generation, - "New connection cancelled pending delayed will" - ); - if pending.cancel.send(()).is_err() { - debug!(client_id = %client_id, "Delayed will timer already gone"); - } - } + self.cancel_pending_will(&client_id); let released = match clients.remove(&client_id) { Some(old_client) => { info!("Client ID takeover: {}", client_id); @@ -561,34 +587,69 @@ impl MessageRouter { } } - /// Removes the handler's router entry if it still owns it; a displaced handler's entry - /// already belongs to its successor and is left alone. pub async fn release_client( &self, client_id: &str, generation: u64, preserve_session: bool, ) -> Release { + let mut exact = self.exact_subscriptions.write().await; + let mut wildcard = self.wildcard_subscriptions.write().await; + let mut clients = self.clients.write().await; + if clients + .get(client_id) + .is_none_or(|info| info.generation != generation) { - let mut clients = self.clients.write().await; - match clients.get(client_id) { - Some(info) if info.generation == generation => { - clients.remove(client_id); - } - Some(_) => return Release::Displaced, - None => {} - } + return Release::Displaced; } + clients.remove(client_id); + if !preserve_session { + Self::strip_client(&mut exact, client_id); + Self::strip_client(&mut wildcard, client_id); + } + drop(clients); + drop(wildcard); + drop(exact); self.outbound_rates.write().remove(client_id); if preserve_session { debug!("Disconnected client (keeping subscriptions): {}", client_id); } else { - self.remove_client_subscriptions(client_id).await; debug!("Unregistered client: {}", client_id); } Release::Owned } + pub async fn lock_session(&self, client_id: &str) -> SessionSlotGuard { + self.session_slots.lock(client_id).await + } + + #[must_use] + pub fn session_slot_count(&self) -> usize { + self.session_slots.len() + } + + pub async fn is_current_owner(&self, client_id: &str, generation: u64) -> bool { + self.clients + .read() + .await + .get(client_id) + .is_some_and(|info| info.generation == generation) + } + + pub fn cancel_pending_will(&self, client_id: &str) { + let Some(pending) = self.pending_wills.lock().remove(client_id) else { + return; + }; + debug!( + client_id, + armed_by = pending.generation, + "New connection cancelled pending delayed will" + ); + if pending.cancel.send(()).is_err() { + debug!(client_id, "Delayed will timer already gone"); + } + } + pub async fn arm_will( &self, client_id: &str, @@ -609,14 +670,6 @@ impl MessageRouter { Some(cancelled) } - pub async fn owns_client(&self, client_id: &str, generation: u64) -> bool { - self.clients - .read() - .await - .get(client_id) - .is_some_and(|info| info.generation == generation) - } - pub async fn claim_will(&self, client_id: &str, generation: u64) -> bool { let claimed = { let mut pending = self.pending_wills.lock(); @@ -626,26 +679,255 @@ impl MessageRouter { armed_here && pending.remove(client_id).is_some() }; if claimed { - self.clear_stored_will(client_id).await; + self.clear_stored_will(client_id, generation).await; } claimed } - pub async fn clear_stored_will(&self, client_id: &str) { + pub async fn clear_stored_will(&self, client_id: &str, generation: u64) { let Some(storage) = &self.storage else { return; }; - match storage.get_session(client_id).await { - Ok(Some(mut session)) if session.will_message.is_some() => { + let cleared = storage + .update_session(client_id, generation, |session| { session.will_message = None; session.will_delay_interval = None; - if let Err(e) = storage.store_session(session).await { - warn!("Failed to clear stored will for {client_id}: {e}"); + }) + .await; + match cleared { + Ok(true) => debug!(client_id, "Removed will from stored session"), + Ok(false) => debug!( + client_id, + "Stored session belongs to a newer connection; left its will alone" + ), + Err(e) => warn!("Failed to clear stored will for {client_id}: {e}"), + } + } + + #[must_use] + pub fn stored_subscription_request( + client_id: &str, + topic_filter: &str, + stored: &StoredSubscription, + ) -> SubscriptionRequest { + SubscriptionRequest::new(client_id.to_string(), topic_filter.to_string(), stored.qos) + .with_subscription_id(stored.subscription_id) + .with_no_local(stored.no_local) + .with_retain_as_published(stored.retain_as_published) + .with_retain_handling(stored.retain_handling) + .with_protocol_version( + ProtocolVersion::try_from(stored.protocol_version).unwrap_or_default(), + ) + .with_change_only(stored.change_only) + .with_flow_id(stored.flow_id) + } + + /// # Errors + /// Returns an error if a stored subscription carries an invalid `retain_handling`. + pub async fn set_client_subscriptions( + &self, + slot: &SessionSlotGuard, + session: Option<&ClientSession>, + ) -> Result<()> { + let client_id = slot.client_id(); + let requests: Vec = session + .map(|session| { + session + .subscriptions + .iter() + .map(|(filter, stored)| { + Self::stored_subscription_request(client_id, filter, stored) + }) + .collect() + }) + .unwrap_or_default(); + let keep: HashSet<(String, Option)> = requests + .iter() + .map(|request| { + ( + parse_shared_subscription(&request.topic_filter) + .0 + .to_string(), + request.flow_id, + ) + }) + .collect(); + { + let mut exact = self.exact_subscriptions.write().await; + let mut wildcard = self.wildcard_subscriptions.write().await; + for map in [&mut *exact, &mut *wildcard] { + for (filter, subs) in map.iter_mut() { + subs.retain(|sub| { + sub.client_id != client_id || keep.contains(&(filter.clone(), sub.flow_id)) + }); } + map.retain(|_, subs| !subs.is_empty()); + } + } + for request in requests { + self.subscribe_as(None, request).await?; + } + Ok(()) + } + + pub async fn sweep_sessions(&self) { + let mut candidates: HashSet = { + let exact = self.exact_subscriptions.read().await; + let wildcard = self.wildcard_subscriptions.read().await; + exact + .values() + .flatten() + .chain(wildcard.values().flatten()) + .map(|sub| sub.client_id.clone()) + .collect() + }; + if let Some(storage) = &self.storage { + match storage.session_client_ids().await { + Ok(ids) => candidates.extend(ids), + Err(e) => warn!("Failed to list stored sessions for the expiry sweep: {e}"), + } + } + let connected: HashSet = self.clients.read().await.keys().cloned().collect(); + let mut swept = 0usize; + for client_id in candidates.difference(&connected) { + let slot = self.lock_session(client_id).await; + if self.sweep_session(&slot).await { + swept += 1; } - Ok(_) => {} - Err(e) => warn!("Failed to load session for {client_id} to clear its will: {e}"), } + self.session_slots.prune(); + for queue in self.fallback_queues.handles() { + queue.purge_expired(); + } + self.fallback_queues.evict_idle(); + if swept > 0 { + info!("Swept {swept} expired or ownerless session(s)"); + } + } + + async fn sweep_session(&self, slot: &SessionSlotGuard) -> bool { + let client_id = slot.client_id(); + if self.clients.read().await.contains_key(client_id) { + return false; + } + if let Some(storage) = &self.storage { + match storage.remove_expired_session(client_id).await { + Ok(removed) if removed => debug!(client_id, "Removed expired session"), + Ok(_) => {} + Err(e) => { + warn!("Failed to remove expired session for {client_id}: {e}"); + return false; + } + } + match storage.get_session(client_id).await { + Ok(Some(session)) => { + if session.connected { + self.end_stranded_session(storage, &session).await; + } + return false; + } + Ok(None) => {} + Err(e) => { + warn!("Failed to read session for {client_id} during the sweep: {e}"); + return false; + } + } + } + let had_subscriptions = self.subscription_count_for_client(client_id).await > 0; + self.remove_client_subscriptions(client_id).await; + self.change_only_states.write().await.remove(client_id); + self.queue_handle(client_id).clear(None); + had_subscriptions + } + + async fn end_stranded_session(&self, storage: &DynamicStorage, session: &ClientSession) { + let client_id = session.client_id.as_str(); + let disconnected_at = unix_millis_now(); + match storage + .update_session(client_id, session.connection_token, |stored| { + stored.mark_disconnected(disconnected_at); + }) + .await + { + Ok(true) => info!( + client_id, + "Ended a stored session left connected without an owner" + ), + Ok(false) => {} + Err(e) => warn!("Failed to end the ownerless session of {client_id}: {e}"), + } + } + + /// # Errors + /// Returns an error if the stored sessions cannot be listed. + pub async fn recover_sessions(&self, maximum_expiry: u32) -> Result { + use futures::StreamExt; + let Some(storage) = &self.storage else { + return Ok(0); + }; + let boot = unix_millis_now(); + let recovered = futures::stream::iter(storage.session_client_ids().await?) + .map(|client_id| self.recover_session(storage, client_id, boot, maximum_expiry)) + .buffer_unordered(RECOVERY_CONCURRENCY) + .filter(|recovered| std::future::ready(*recovered)) + .count() + .await; + self.session_slots.prune(); + if recovered > 0 { + info!("Recovered {recovered} persisted session(s)"); + } + Ok(recovered) + } + + async fn recover_session( + &self, + storage: &DynamicStorage, + client_id: String, + boot: u64, + maximum_expiry: u32, + ) -> bool { + let slot = self.lock_session(&client_id).await; + if let Err(e) = storage.remove_expired_session(&client_id).await { + warn!("Failed to remove expired session for {client_id}: {e}"); + return false; + } + let mut session = match storage.get_session(&client_id).await { + Ok(Some(session)) => session, + Ok(None) => return false, + Err(e) => { + warn!("Failed to read session for {client_id} during recovery: {e}"); + return false; + } + }; + let stored_expiry = session.expiry_interval; + let capped = Some(ClientSession::granted_expiry(stored_expiry, maximum_expiry)); + let stamp = session.disconnected_at.is_none(); + if stamp { + session.mark_disconnected(boot); + } + session.expiry_interval = capped; + if session.expiry_interval == Some(0) || session.is_expired() { + if let Err(e) = storage + .remove_owned_session(&client_id, session.connection_token) + .await + { + warn!("Failed to remove ended session for {client_id}: {e}"); + } + return false; + } + if stamp || capped != stored_expiry { + if let Err(e) = storage.store_session(session.clone()).await { + warn!("Failed to stamp the disconnect time of {client_id}: {e}"); + return false; + } + } + if let Err(e) = self.set_client_subscriptions(&slot, Some(&session)).await { + warn!("Failed to restore subscriptions for {client_id}: {e}"); + return false; + } + self.load_change_only_state(&client_id, session.change_only_state.clone()) + .await; + true } pub async fn is_connected(&self, client_id: &str) -> bool { @@ -687,78 +969,6 @@ impl MessageRouter { map.retain(|_, subs| !subs.is_empty()); } - pub async fn cleanup_stale_subscriptions(&self) { - for queue in self.fallback_queues.handles() { - queue.purge_expired(); - } - self.fallback_queues.evict_idle(); - let subscribed_ids: HashSet = { - let exact = self.exact_subscriptions.read().await; - let wildcard = self.wildcard_subscriptions.read().await; - exact - .values() - .flatten() - .chain(wildcard.values().flatten()) - .map(|sub| sub.client_id.clone()) - .collect() - }; - - let connected: HashSet = self.clients.read().await.keys().cloned().collect(); - - let mut stale: Vec = Vec::new(); - let storage = self.storage.as_ref(); - - for client_id in &subscribed_ids { - if connected.contains(client_id) { - continue; - } - let has_session = if let Some(storage) = storage { - matches!(storage.get_session(client_id).await, Ok(Some(s)) if !s.is_expired()) - } else { - false - }; - if !has_session { - stale.push(client_id.clone()); - } - } - - if stale.is_empty() { - return; - } - - { - let mut exact = self.exact_subscriptions.write().await; - for subs in exact.values_mut() { - subs.retain(|sub| !stale.contains(&sub.client_id)); - } - exact.retain(|_, subs| !subs.is_empty()); - } - - { - let mut wildcard = self.wildcard_subscriptions.write().await; - for subs in wildcard.values_mut() { - subs.retain(|sub| !stale.contains(&sub.client_id)); - } - wildcard.retain(|_, subs| !subs.is_empty()); - } - - { - let mut change_only = self.change_only_states.write().await; - for client_id in &stale { - change_only.remove(client_id); - } - } - - for client_id in &stale { - self.queue_handle(client_id).clear(None); - } - - info!( - "Cleaned up stale subscriptions for {} disconnected client(s)", - stale.len() - ); - } - /// Adds a subscription for a client. /// /// # Errors @@ -1542,6 +1752,7 @@ impl Default for MessageRouter { #[cfg(test)] mod tests { use super::*; + use crate::broker::storage::ClientSession; use bytes::Bytes; struct TestLanes { @@ -1769,7 +1980,7 @@ mod tests { let router = Arc::clone(&router); tokio::spawn(async move { for _ in 0..500 { - router.cleanup_stale_subscriptions().await; + router.sweep_sessions().await; tokio::task::yield_now().await; } }) @@ -2457,11 +2668,42 @@ mod tests { register(&router, "w2", &lanes).await; - assert!(cancelled.await.is_ok(), "the timer is woken by the cancel"); + let woken = tokio::time::timeout(Duration::from_secs(5), cancelled).await; + assert!( + matches!(woken, Ok(Ok(()))), + "the timer is woken by the cancel" + ); assert!(!router.claim_will("w2", generation).await); assert!(router.pending_wills.lock().is_empty()); } + #[tokio::test] + async fn claim_rejects_a_will_armed_by_an_older_connection() { + let router = MessageRouter::new(); + let lanes = TestLanes::new(10); + let first = register(&router, "w5", &lanes).await; + let first_armed = router.arm_will("w5", first).await; + assert!(first_armed.is_some()); + router.release_client("w5", first, true).await; + + let second = register(&router, "w5", &lanes).await; + router.release_client("w5", second, true).await; + let second_armed = router.arm_will("w5", second).await; + assert!(second_armed.is_none(), "a released connection cannot arm"); + + let third = register(&router, "w5", &lanes).await; + let third_armed = router.arm_will("w5", third).await; + assert!(third_armed.is_some()); + router.release_client("w5", third, true).await; + + assert!( + !router.claim_will("w5", first).await, + "the first connection's will was cancelled; its timer must not claim the newer will" + ); + assert!(router.claim_will("w5", third).await); + assert!(router.pending_wills.lock().is_empty()); + } + #[tokio::test] async fn displaced_connection_cannot_arm_will() { let router = MessageRouter::new(); @@ -2473,24 +2715,25 @@ mod tests { assert!(router.pending_wills.lock().is_empty()); } + fn session_with_will(client_id: &str, connection_token: u64) -> ClientSession { + let will = crate::types::WillMessage::new(format!("will/{client_id}"), "offline"); + let mut session = ClientSession::new_with_will(client_id, true, Some(60), Some(will)); + session.connection_token = connection_token; + session + } + #[tokio::test] async fn claimed_will_is_removed_from_stored_session() { let storage = Arc::new(DynamicStorage::Memory( crate::broker::storage::MemoryBackend::new(), )); let router = MessageRouter::with_storage(Arc::clone(&storage)); - let will = crate::types::WillMessage::new("will/w4", "offline"); + let lanes = TestLanes::new(10); + let generation = register(&router, "w4", &lanes).await; storage - .store_session(crate::broker::storage::ClientSession::new_with_will( - "w4", - true, - Some(60), - Some(will), - )) + .store_session(session_with_will("w4", generation)) .await .unwrap(); - let lanes = TestLanes::new(10); - let generation = register(&router, "w4", &lanes).await; let armed = router.arm_will("w4", generation).await; assert!(armed.is_some()); router.release_client("w4", generation, true).await; @@ -2501,4 +2744,210 @@ mod tests { assert!(stored.will_message.is_none()); assert!(stored.will_delay_interval.is_none()); } + + #[tokio::test] + async fn claimed_will_leaves_a_newer_stored_session_alone() { + let storage = Arc::new(DynamicStorage::Memory( + crate::broker::storage::MemoryBackend::new(), + )); + let router = MessageRouter::with_storage(Arc::clone(&storage)); + let lanes = TestLanes::new(10); + let generation = register(&router, "w6", &lanes).await; + storage + .store_session(session_with_will("w6", generation)) + .await + .unwrap(); + let armed = router.arm_will("w6", generation).await; + assert!(armed.is_some()); + router.release_client("w6", generation, true).await; + + let successor = register(&router, "w6", &lanes).await; + storage + .store_session(session_with_will("w6", successor)) + .await + .unwrap(); + + assert!(!router.claim_will("w6", generation).await); + + let stored = storage.get_session("w6").await.unwrap().unwrap(); + assert_eq!(stored.connection_token, successor); + assert!( + stored.will_message.is_some(), + "the successor's freshly stored will must survive" + ); + } + + #[tokio::test] + async fn generations_start_above_every_earlier_run() { + let earlier = MessageRouter::new(); + let lanes = TestLanes::new(10); + let earlier_generation = register(&earlier, "epoch", &lanes).await; + tokio::time::sleep(Duration::from_millis(5)).await; + let later = MessageRouter::new(); + let later_generation = register(&later, "epoch", &lanes).await; + assert!( + later_generation > earlier_generation, + "a restarted router must never hand out a generation a previous run used" + ); + } + + #[tokio::test] + async fn release_of_a_missing_entry_is_not_owned() { + let router = MessageRouter::new(); + assert!(matches!( + router.release_client("absent", 1, false).await, + Release::Displaced + )); + } + + #[tokio::test] + async fn sweep_leaves_a_client_id_claimed_while_it_waited_for_the_slot() { + let router = Arc::new(MessageRouter::new()); + router + .subscribe(SubscriptionRequest::new( + "late-owner", + "late/t", + QoS::AtMostOnce, + )) + .await + .unwrap(); + let slot = router.lock_session("late-owner").await; + let sweeping = tokio::spawn({ + let router = Arc::clone(&router); + async move { router.sweep_sessions().await } + }); + tokio::time::sleep(Duration::from_millis(100)).await; + let lanes = TestLanes::new(10); + register(&router, "late-owner", &lanes).await; + drop(slot); + sweeping.await.unwrap(); + assert!( + router.has_subscription("late-owner", "late/t").await, + "the sweep stripped the routes of a connection that claimed the ClientID while the sweep waited" + ); + } + + #[tokio::test] + async fn exact_subscription_set_replaces_rather_than_unions() { + let router = MessageRouter::new(); + let rx = TestLanes::new(10); + let (dtx, _drx) = tokio::sync::oneshot::channel(); + router + .register_client( + "exact".to_string(), + rx.lanes(), + router.queue_handle("exact"), + dtx, + ) + .await; + router + .subscribe(SubscriptionRequest::new( + "exact", + "old/topic", + QoS::AtMostOnce, + )) + .await + .unwrap(); + let mut session = ClientSession::new("exact", true, Some(60)); + session.add_subscription( + "new/topic", + StoredSubscription { + qos: QoS::AtLeastOnce, + no_local: false, + retain_as_published: false, + retain_handling: 0, + subscription_id: None, + protocol_version: 5, + change_only: false, + flow_id: None, + }, + ); + let slot = router.lock_session("exact").await; + router + .set_client_subscriptions(&slot, Some(&session)) + .await + .unwrap(); + drop(slot); + assert!(!router.has_subscription("exact", "old/topic").await); + assert!(router.has_subscription("exact", "new/topic").await); + + let slot = router.lock_session("exact").await; + router.set_client_subscriptions(&slot, None).await.unwrap(); + drop(slot); + assert_eq!(router.subscription_count_for_client("exact").await, 0); + assert_eq!(router.session_slot_count(), 0); + } + + fn ended_session(client_id: &str, expiry: u32, ended_ago_ms: u64) -> ClientSession { + let mut session = ClientSession::new(client_id, true, Some(expiry)); + session.add_subscription( + format!("{client_id}/t"), + crate::broker::storage::StoredSubscription::new(QoS::AtLeastOnce), + ); + session.mark_disconnected(unix_millis_now() - ended_ago_ms); + session + } + + #[tokio::test] + async fn session_expired_by_the_capped_expiry_is_not_routed_after_recovery() { + let dir = tempfile::tempdir().unwrap(); + let storage = Arc::new(DynamicStorage::File( + crate::broker::storage::FileBackend::new(dir.path()) + .await + .unwrap(), + )); + storage + .store_session(ended_session("stale", 3600, 120_000)) + .await + .unwrap(); + let router = MessageRouter::with_storage(Arc::clone(&storage)); + router.recover_sessions(60).await.unwrap(); + assert_eq!( + router.subscription_count_for_client("stale").await, + 0, + "a session that ended 120s ago outlived the 60s maximum, yet its routes were rebuilt" + ); + assert!(storage.session_client_ids().await.unwrap().is_empty()); + } + + #[tokio::test] + async fn capped_expiry_of_an_ended_session_is_stored_at_recovery() { + let dir = tempfile::tempdir().unwrap(); + { + let storage = Arc::new(DynamicStorage::File( + crate::broker::storage::FileBackend::new(dir.path()) + .await + .unwrap(), + )); + storage + .store_session(ended_session("idle", 3600, 10_000)) + .await + .unwrap(); + let router = MessageRouter::with_storage(Arc::clone(&storage)); + router.recover_sessions(60).await.unwrap(); + assert_eq!(router.subscription_count_for_client("idle").await, 1); + assert_eq!( + storage + .get_session("idle") + .await + .unwrap() + .unwrap() + .expiry_interval, + Some(60) + ); + } + let reopened = crate::broker::storage::FileBackend::new(dir.path()) + .await + .unwrap(); + assert_eq!( + reopened + .get_session("idle") + .await + .unwrap() + .unwrap() + .expiry_interval, + Some(60), + "the capped expiry must be stored, not only applied in memory" + ); + } } diff --git a/crates/mqtt5/src/broker/server.rs b/crates/mqtt5/src/broker/server.rs index cf7bab2b..f563f967 100644 --- a/crates/mqtt5/src/broker/server.rs +++ b/crates/mqtt5/src/broker/server.rs @@ -903,9 +903,6 @@ impl MqttBroker { shutdown_tx: &tokio::sync::broadcast::Sender<()>, task_handles: &mut Vec>, ) -> Result<()> { - // The periodic cleanup runs whether or not persistence is on: it also evicts the - // router's in-memory fallback queues, which otherwise leak one entry per distinct - // client id when storage is off. { if let Some(ref storage) = self.storage { storage.cleanup_expired().await?; @@ -925,36 +922,14 @@ impl MqttBroker { error!("Storage cleanup error: {e}"); } } - router_clone.cleanup_stale_subscriptions().await; + router_clone.sweep_sessions().await; } _ = shutdown_rx.recv() => { debug!("Storage cleanup task shutting down"); - break; - } - } - } - })); - } - - if let Some(ref storage) = self.storage { - let storage_clone = Arc::clone(storage); - let mut shutdown_rx = shutdown_tx.subscribe(); - let flush_interval = std::time::Duration::from_secs(5); - - task_handles.push(tokio::spawn(async move { - let mut interval = tokio::time::interval(flush_interval); - interval.set_missed_tick_behavior(tokio::time::MissedTickBehavior::Skip); - loop { - tokio::select! { - _ = interval.tick() => { - if let Err(e) = storage_clone.flush_sessions().await { - error!("Session flush error: {e}"); - } - } - _ = shutdown_rx.recv() => { - debug!("Flushing sessions before shutdown"); - if let Err(e) = storage_clone.shutdown().await { - error!("Session shutdown flush error: {e}"); + if let Some(ref storage) = storage_clone { + if let Err(e) = storage.shutdown().await { + error!("Storage shutdown error: {e}"); + } } break; } @@ -962,6 +937,7 @@ impl MqttBroker { } })); } + Ok(()) } @@ -1713,6 +1689,11 @@ impl MqttBroker { let mut task_handles: Vec> = Vec::new(); + self.router + .recover_sessions( + u32::try_from(self.config.session_expiry_interval.as_secs()).unwrap_or(u32::MAX), + ) + .await?; self.initialize_storage(&shutdown_tx, &mut task_handles) .await?; self.router.initialize().await?; @@ -1872,6 +1853,11 @@ impl MqttBroker { Arc::clone(&self.router) } + #[must_use] + pub fn storage(&self) -> Option> { + self.storage.clone() + } + #[must_use] pub fn local_addr(&self) -> Option { self.listeners.first()?.local_addr().ok() diff --git a/crates/mqtt5/src/broker/session_fault_tests.rs b/crates/mqtt5/src/broker/session_fault_tests.rs new file mode 100644 index 00000000..8d185097 --- /dev/null +++ b/crates/mqtt5/src/broker/session_fault_tests.rs @@ -0,0 +1,470 @@ +use crate::broker::config::{BrokerConfig, StorageBackend as BackendKind, StorageConfig}; +use crate::broker::router::MessageRouter; +use crate::broker::server::MqttBroker; +use crate::broker::storage::{DynamicStorage, FileBackend, StorageBackend}; +use crate::packet::connect::ConnectPacket; +use crate::packet::subscribe::SubscribePacket; +use crate::packet::unsubscribe::UnsubscribePacket; +use crate::packet::{MqttPacket, Packet}; +use crate::protocol::v5::reason_codes::ReasonCode; +use crate::time::Duration; +use crate::transport::packet_io::read_packet_from_stream; +use crate::types::ConnectOptions; +use crate::{QoS, WillMessage}; +use bytes::BytesMut; +use std::net::SocketAddr; +use std::path::Path; +use std::sync::Arc; +use tokio::io::AsyncWriteExt; +use tokio::net::TcpStream; +use tokio::time::sleep; + +struct Broker { + addr: String, + storage: Arc, + router: Arc, + handle: tokio::task::JoinHandle<()>, +} + +impl Broker { + async fn start(dir: &Path, cleanup_interval: Duration) -> Self { + let config = BrokerConfig::default() + .with_bind_address("127.0.0.1:0".parse::().unwrap()) + .with_storage(StorageConfig { + backend: BackendKind::File, + base_dir: dir.to_path_buf(), + enable_persistence: true, + cleanup_interval, + ..Default::default() + }); + let mut broker = MqttBroker::with_config(config).await.unwrap(); + let addr = broker.local_addr().unwrap().to_string(); + let storage = broker.storage().unwrap(); + let router = broker.router(); + let handle = tokio::spawn(async move { + if let Err(e) = broker.run().await { + tracing::debug!("broker stopped: {e}"); + } + }); + sleep(Duration::from_millis(100)).await; + Self { + addr, + storage, + router, + handle, + } + } + + fn file_backend(&self) -> &FileBackend { + let DynamicStorage::File(backend) = &*self.storage else { + panic!("file storage expected"); + }; + backend + } + + async fn break_next_session_write(&self) { + self.file_backend().break_next_session_write().await; + } + + async fn pause_session_writes(&self) -> impl Sized { + self.file_backend().pause_session_writes().await + } + + async fn token(&self, client_id: &str) -> u64 { + self.storage + .get_session(client_id) + .await + .unwrap() + .unwrap() + .connection_token + } +} + +impl Drop for Broker { + fn drop(&mut self) { + self.handle.abort(); + } +} + +struct Conn { + stream: TcpStream, + buffer: BytesMut, +} + +impl Conn { + async fn send(&mut self, packet: &impl MqttPacket) { + let mut bytes = Vec::new(); + packet.encode(&mut bytes).unwrap(); + self.stream.write_all(&bytes).await.unwrap(); + } + + async fn send_raw(&mut self, bytes: &[u8]) -> bool { + self.stream.write_all(bytes).await.is_ok() + } + + async fn next(&mut self, millis: u64) -> Option { + tokio::time::timeout( + Duration::from_millis(millis), + read_packet_from_stream(&mut self.stream, 5, &mut self.buffer, 1 << 20), + ) + .await + .ok()? + .ok() + } + + async fn closed(&mut self, millis: u64) -> bool { + tokio::time::timeout( + Duration::from_millis(millis), + read_packet_from_stream(&mut self.stream, 5, &mut self.buffer, 1 << 20), + ) + .await + .is_ok_and(|read| read.is_err()) + } + + async fn replies(&mut self, millis: u64) -> Vec { + let mut replies = Vec::new(); + while let Some(packet) = self.next(millis).await { + replies.push(packet); + } + replies + } + + async fn subscribed(&mut self, topic: &str) { + self.send(&SubscribePacket::new(1).add_filter(topic, QoS::AtLeastOnce)) + .await; + assert!(matches!(self.next(3000).await, Some(Packet::SubAck(_)))); + } + + async fn still_served(&mut self) -> bool { + self.send_raw(&[0xC0, 0x00]).await + && matches!(self.next(1000).await, Some(Packet::PingResp)) + } +} + +async fn connect_with(addr: &str, options: ConnectOptions) -> (Conn, Option) { + let mut conn = Conn { + stream: TcpStream::connect(addr).await.unwrap(), + buffer: BytesMut::new(), + }; + conn.send(&ConnectPacket::new(options.protocol_options)) + .await; + let connack = conn.next(3000).await; + (conn, connack) +} + +async fn open( + addr: &str, + client_id: &str, + clean_start: bool, + expiry: u32, +) -> (Conn, Option) { + connect_with( + addr, + ConnectOptions::new(client_id) + .with_clean_start(clean_start) + .with_session_expiry_interval(expiry), + ) + .await +} + +fn connack_reason(packet: Option<&Packet>) -> Option { + match packet { + Some(Packet::ConnAck(connack)) => Some(connack.reason_code), + _ => None, + } +} + +async fn publish(conn: &mut Conn, topic: &str) { + let topic_len = u8::try_from(topic.len()).unwrap(); + let mut bytes = vec![0x30, 5 + topic_len, 0x00, topic_len]; + bytes.extend_from_slice(topic.as_bytes()); + bytes.extend_from_slice(&[0x00, b'h', b'i']); + assert!(conn.send_raw(&bytes).await); +} + +#[tokio::test(flavor = "multi_thread", worker_threads = 4)] +async fn failed_claim_write_leaves_the_live_owner_in_place() { + let dir = tempfile::tempdir().unwrap(); + let broker = Broker::start(dir.path(), Duration::from_secs(3600)).await; + let (mut owner, connack) = open(&broker.addr, "held", true, 60).await; + assert_eq!(connack_reason(connack.as_ref()), Some(ReasonCode::Success)); + owner.subscribed("held/t").await; + let owner_token = broker.token("held").await; + + broker.break_next_session_write().await; + let (mut intruder, reply) = open(&broker.addr, "held", false, 60).await; + assert_eq!( + connack_reason(reply.as_ref()), + Some(ReasonCode::ServerUnavailable), + "the failed claim is answered with a CONNACK refusing it as Server unavailable" + ); + assert!(intruder.next(500).await.is_none()); + + assert!( + owner.still_served().await, + "a claim whose write failed must not displace the live owner" + ); + assert!(broker.router.is_current_owner("held", owner_token).await); + assert_eq!(broker.token("held").await, owner_token); + assert!(broker.router.has_subscription("held", "held/t").await); + + let (mut publisher, _) = open(&broker.addr, "held-pub", true, 0).await; + publish(&mut publisher, "held/t").await; + assert!(matches!(owner.next(1000).await, Some(Packet::Publish(_)))); +} + +#[tokio::test(flavor = "multi_thread", worker_threads = 4)] +async fn failed_subscribe_write_is_not_acknowledged_and_installs_no_route() { + let dir = tempfile::tempdir().unwrap(); + let broker = Broker::start(dir.path(), Duration::from_secs(3600)).await; + let (mut client, _) = open(&broker.addr, "subfail", true, 60).await; + client.subscribed("kept/t").await; + + broker.break_next_session_write().await; + client + .send( + &SubscribePacket::new(2) + .add_filter("lost/a", QoS::AtLeastOnce) + .add_filter("lost/b", QoS::AtLeastOnce) + .add_filter("kept/t", QoS::AtMostOnce), + ) + .await; + let replies = client.replies(1000).await; + assert!( + !replies + .iter() + .any(|packet| matches!(packet, Packet::SubAck(_))), + "a SUBSCRIBE whose write failed was acknowledged: {replies:?}" + ); + for lost in ["lost/a", "lost/b"] { + assert!(!broker.router.has_subscription("subfail", lost).await); + } + assert!(broker.router.has_subscription("subfail", "kept/t").await); + drop(client); + sleep(Duration::from_millis(200)).await; + + let stored = broker + .storage + .get_session("subfail") + .await + .unwrap() + .unwrap(); + assert!(!stored.subscriptions.contains_key("lost/a")); + assert!(!stored.subscriptions.contains_key("lost/b")); + assert_eq!( + stored.subscriptions.get("kept/t").map(|sub| sub.qos), + Some(QoS::AtLeastOnce), + "the failed SUBSCRIBE changed a stored subscription" + ); + assert!(!broker.router.has_subscription("subfail", "lost/a").await); +} + +#[tokio::test(flavor = "multi_thread", worker_threads = 4)] +async fn failed_unsubscribe_write_is_not_acknowledged_and_keeps_the_route() { + let dir = tempfile::tempdir().unwrap(); + let broker = Broker::start(dir.path(), Duration::from_secs(3600)).await; + let (mut client, _) = open(&broker.addr, "unsubfail", true, 60).await; + client.subscribed("kept/u").await; + + broker.break_next_session_write().await; + client + .send(&UnsubscribePacket::new(3).add_filter("kept/u")) + .await; + let replies = client.replies(1000).await; + assert!( + !replies + .iter() + .any(|packet| matches!(packet, Packet::UnsubAck(_))), + "an UNSUBSCRIBE whose write failed was acknowledged: {replies:?}" + ); + assert!(broker.router.has_subscription("unsubfail", "kept/u").await); + drop(client); + sleep(Duration::from_millis(200)).await; + let stored = broker + .storage + .get_session("unsubfail") + .await + .unwrap() + .unwrap(); + assert!(stored.subscriptions.contains_key("kept/u")); +} + +#[tokio::test(flavor = "multi_thread", worker_threads = 4)] +async fn failed_session_write_sends_disconnect_and_publishes_the_will() { + let dir = tempfile::tempdir().unwrap(); + let broker = Broker::start(dir.path(), Duration::from_secs(3600)).await; + let (mut watcher, _) = open(&broker.addr, "will-watcher", true, 0).await; + watcher.subscribed("wills/failed").await; + let (mut client, _) = connect_with( + &broker.addr, + ConnectOptions::new("will-failed") + .with_clean_start(true) + .with_session_expiry_interval(60) + .with_will(WillMessage::new("wills/failed", "gone")), + ) + .await; + + broker.break_next_session_write().await; + client + .send(&SubscribePacket::new(4).add_filter("never/t", QoS::AtMostOnce)) + .await; + match client.next(2000).await { + Some(Packet::Disconnect(disconnect)) => { + assert_eq!(disconnect.reason_code, ReasonCode::UnspecifiedError); + } + other => panic!("expected DISCONNECT before the close, got {other:?}"), + } + assert!(client.closed(2000).await); + assert!( + matches!(watcher.next(2000).await, Some(Packet::Publish(will)) if will.topic_name == "wills/failed"), + "a server-initiated close for an error must publish the Will" + ); +} + +#[tokio::test(flavor = "multi_thread", worker_threads = 4)] +async fn subscribe_with_many_filters_is_one_session_write() { + let dir = tempfile::tempdir().unwrap(); + let broker = Broker::start(dir.path(), Duration::from_secs(3600)).await; + let (mut client, _) = open(&broker.addr, "many", true, 60).await; + let before = broker.file_backend().session_flushes().await; + let subscribe = (0..20).fold(SubscribePacket::new(5), |packet, i| { + packet.add_filter(format!("many/{i}"), QoS::AtLeastOnce) + }); + client.send(&subscribe).await; + match client.next(3000).await { + Some(Packet::SubAck(suback)) => assert_eq!(suback.reason_codes.len(), 20), + other => panic!("expected SUBACK, got {other:?}"), + } + assert_eq!( + broker.file_backend().session_flushes().await - before, + 1, + "every filter of one SUBSCRIBE must be stored in one write" + ); + let stored = broker.storage.get_session("many").await.unwrap().unwrap(); + assert_eq!(stored.subscriptions.len(), 20); +} + +#[tokio::test(flavor = "multi_thread", worker_threads = 4)] +async fn unsubscribe_with_many_filters_is_one_session_write() { + let dir = tempfile::tempdir().unwrap(); + let broker = Broker::start(dir.path(), Duration::from_secs(3600)).await; + let (mut client, _) = open(&broker.addr, "many-un", true, 60).await; + let subscribe = (0..20).fold(SubscribePacket::new(6), |packet, i| { + packet.add_filter(format!("many/{i}"), QoS::AtLeastOnce) + }); + client.send(&subscribe).await; + assert!(matches!(client.next(3000).await, Some(Packet::SubAck(_)))); + let before = broker.file_backend().session_flushes().await; + let unsubscribe = (0..20).fold(UnsubscribePacket::new(7), |packet, i| { + packet.add_filter(format!("many/{i}")) + }); + client.send(&unsubscribe).await; + assert!(matches!(client.next(3000).await, Some(Packet::UnsubAck(_)))); + assert_eq!(broker.file_backend().session_flushes().await - before, 1); + let stored = broker + .storage + .get_session("many-un") + .await + .unwrap() + .unwrap(); + assert!(stored.subscriptions.is_empty()); +} + +#[tokio::test(flavor = "multi_thread", worker_threads = 4)] +async fn session_left_connected_by_a_failed_release_write_still_expires() { + let dir = tempfile::tempdir().unwrap(); + let broker = Broker::start(dir.path(), Duration::from_millis(200)).await; + let (mut client, _) = open(&broker.addr, "stranded", true, 1).await; + client.subscribed("stranded/t").await; + + broker.break_next_session_write().await; + assert!(client.send_raw(&[0xE0, 0x00]).await); + assert!(client.next(2000).await.is_none()); + sleep(Duration::from_millis(3000)).await; + + assert!( + broker + .storage + .get_session("stranded") + .await + .unwrap() + .is_none(), + "a session whose release write failed must still end and expire" + ); + assert_eq!( + broker + .router + .subscription_count_for_client("stranded") + .await, + 0 + ); +} + +#[tokio::test(flavor = "multi_thread", worker_threads = 4)] +async fn connack_waits_for_the_flush_that_covers_the_claim() { + let dir = tempfile::tempdir().unwrap(); + let broker = Broker::start(dir.path(), Duration::from_secs(3600)).await; + let paused = broker.pause_session_writes().await; + let mut conn = Conn { + stream: TcpStream::connect(&broker.addr).await.unwrap(), + buffer: BytesMut::new(), + }; + conn.send(&ConnectPacket::new( + ConnectOptions::new("covered") + .with_clean_start(true) + .with_session_expiry_interval(60) + .protocol_options, + )) + .await; + assert!( + conn.next(500).await.is_none(), + "CONNACK was sent before the claim was durable" + ); + drop(paused); + assert_eq!( + connack_reason(conn.next(3000).await.as_ref()), + Some(ReasonCode::Success) + ); +} + +#[tokio::test(flavor = "multi_thread", worker_threads = 4)] +async fn suback_waits_for_the_flush_that_covers_the_subscription() { + let dir = tempfile::tempdir().unwrap(); + let broker = Broker::start(dir.path(), Duration::from_secs(3600)).await; + let (mut client, _) = open(&broker.addr, "sub-covered", true, 60).await; + let paused = broker.pause_session_writes().await; + client + .send(&SubscribePacket::new(8).add_filter("covered/t", QoS::AtLeastOnce)) + .await; + assert!( + client.next(500).await.is_none(), + "SUBACK was sent before the subscription was durable" + ); + drop(paused); + assert!(matches!(client.next(3000).await, Some(Packet::SubAck(_)))); +} + +#[tokio::test(flavor = "multi_thread", worker_threads = 4)] +async fn disconnect_completes_only_after_its_writes_are_durable() { + let dir = tempfile::tempdir().unwrap(); + let broker = Broker::start(dir.path(), Duration::from_secs(3600)).await; + let (mut client, _) = open(&broker.addr, "disc-covered", true, 60).await; + let paused = broker.pause_session_writes().await; + assert!( + client + .send_raw(&[0xE0, 0x07, 0x00, 0x05, 0x11, 0x00, 0x00, 0x00, 0x00]) + .await + ); + assert!( + !client.closed(500).await, + "DISCONNECT processing completed before its writes were durable" + ); + drop(paused); + assert!(client.closed(3000).await); + assert!(broker + .storage + .session_client_ids() + .await + .unwrap() + .is_empty()); +} diff --git a/crates/mqtt5/src/broker/session_slot.rs b/crates/mqtt5/src/broker/session_slot.rs new file mode 100644 index 00000000..17cce49f --- /dev/null +++ b/crates/mqtt5/src/broker/session_slot.rs @@ -0,0 +1,121 @@ +use parking_lot::Mutex; +use std::collections::HashMap; +use std::sync::Arc; +use tokio::sync::{Mutex as AsyncMutex, OwnedMutexGuard}; + +type SlotMap = Arc>>>>; + +#[derive(Default)] +pub struct SessionSlots { + slots: SlotMap, +} + +impl SessionSlots { + pub async fn lock(&self, client_id: &str) -> SessionSlotGuard { + let slot = Arc::clone( + self.slots + .lock() + .entry(client_id.to_string()) + .or_insert_with(|| Arc::new(AsyncMutex::new(()))), + ); + let guard = slot.lock_owned().await; + SessionSlotGuard { + client_id: client_id.to_string(), + guard: Some(guard), + slots: Arc::clone(&self.slots), + } + } + + pub fn prune(&self) { + self.slots + .lock() + .retain(|_, slot| Arc::strong_count(slot) > 1); + } + + #[must_use] + pub fn len(&self) -> usize { + self.slots.lock().len() + } + + #[must_use] + pub fn is_empty(&self) -> bool { + self.slots.lock().is_empty() + } +} + +pub struct SessionSlotGuard { + client_id: String, + guard: Option>, + slots: SlotMap, +} + +impl SessionSlotGuard { + #[must_use] + pub fn client_id(&self) -> &str { + &self.client_id + } +} + +impl Drop for SessionSlotGuard { + fn drop(&mut self) { + drop(self.guard.take()); + let mut slots = self.slots.lock(); + if slots + .get(&self.client_id) + .is_some_and(|slot| Arc::strong_count(slot) == 1) + { + slots.remove(&self.client_id); + } + } +} + +#[cfg(test)] +mod tests { + use super::SessionSlots; + use std::sync::Arc; + use std::time::Duration; + + #[tokio::test] + async fn slot_is_removed_once_nobody_holds_or_waits_for_it() { + let slots = SessionSlots::default(); + let guard = slots.lock("a").await; + assert_eq!(slots.len(), 1); + drop(guard); + assert!(slots.is_empty()); + } + + #[tokio::test] + async fn waiter_keeps_the_slot_and_is_excluded_until_release() { + let slots = Arc::new(SessionSlots::default()); + let guard = slots.lock("a").await; + let waiter_slots = Arc::clone(&slots); + let waiter = tokio::spawn(async move { + let guard = waiter_slots.lock("a").await; + drop(guard); + }); + tokio::time::sleep(Duration::from_millis(50)).await; + assert!(!waiter.is_finished(), "the second holder must wait"); + drop(guard); + assert_eq!(slots.len(), 1, "the waiter still holds a reference"); + waiter.await.expect("waiter"); + assert!(slots.is_empty()); + } + + #[tokio::test] + async fn abandoned_waiter_is_pruned() { + let slots = Arc::new(SessionSlots::default()); + let guard = slots.lock("a").await; + let waiter_slots = Arc::clone(&slots); + let waiter = tokio::spawn(async move { + let guard = waiter_slots.lock("a").await; + drop(guard); + }); + tokio::time::sleep(Duration::from_millis(50)).await; + waiter.abort(); + let aborted = waiter.await; + assert!(aborted.is_err_and(|e| e.is_cancelled())); + drop(guard); + slots.prune(); + assert!(slots.is_empty()); + } +} diff --git a/crates/mqtt5/src/broker/storage/file_backend.rs b/crates/mqtt5/src/broker/storage/file_backend.rs index a7775c22..c4aa3e9c 100644 --- a/crates/mqtt5/src/broker/storage/file_backend.rs +++ b/crates/mqtt5/src/broker/storage/file_backend.rs @@ -2,6 +2,7 @@ //! //! Provides durable storage using organized file structure with atomic operations. +use super::session_log::{sync_directory, SessionChange, SessionLog}; use super::{ ClientSession, InflightDirection, InflightMessage, QueueHandle, QueueLimits, QueueOp, QueueRegistry, QueueWriter, QueuedMessage, RetainedMessage, StorageBackend, SEQ_FLOOR, @@ -11,11 +12,11 @@ use crate::validation::topic_matches_filter; use serde_json; use std::collections::{HashMap, HashSet}; use std::path::{Path, PathBuf}; -use std::sync::atomic::{AtomicBool, AtomicU64, Ordering}; +use std::sync::atomic::{AtomicU64, Ordering}; use std::sync::Arc; use tokio::fs::{self, File}; use tokio::io::AsyncWriteExt; -use tokio::sync::{mpsc, oneshot, RwLock}; +use tokio::sync::{mpsc, oneshot}; use tracing::{debug, info, warn}; /// How long an inflight or queued row may sit unwritten so that its remove can cancel it. @@ -173,18 +174,18 @@ static TEMP_FILE_COUNTER: AtomicU64 = AtomicU64::new(0); /// /// Version History: /// - 1: Initial version (0.10.0) -const STORAGE_VERSION: &str = "1"; +/// - 2: Sessions in one group-committed log, `sessions/sessions.log` (0.42.0); version 1 +/// directories are migrated on open +const STORAGE_VERSION: &str = "2"; +const LEGACY_STORAGE_VERSION: &str = "1"; -/// File-based storage backend with write-behind session caching +/// File-based storage backend; session writes are group-committed to one log pub struct FileBackend { _base_dir: PathBuf, retained_dir: PathBuf, - sessions_dir: PathBuf, queues_dir: PathBuf, inflight_dir: PathBuf, - sessions_cache: Arc>>, - dirty_sessions: Arc>>, - shutdown: Arc, + sessions: SessionLog, queues: QueueRegistry, queue_writer: QueueWriter, queue_flush: mpsc::Sender>, @@ -213,13 +214,17 @@ impl FileBackend { let queues_dir = base_dir.join("queues"); let inflight_dir = base_dir.join("inflight"); - Self::check_storage_version(&base_dir).await?; + let legacy = Self::check_storage_version(&base_dir).await?; for dir in [&retained_dir, &sessions_dir, &queues_dir, &inflight_dir] { fs::create_dir_all(dir).await.map_err(|e| { MqttError::Configuration(format!("Failed to create dir {}: {e}", dir.display())) })?; } + let sessions = SessionLog::open(sessions_dir, legacy).await?; + if legacy { + Self::write_storage_version(&base_dir).await?; + } let (writer_tx, writer_rx) = mpsc::unbounded_channel(); let (flush_tx, flush_rx) = mpsc::channel(4); @@ -233,12 +238,9 @@ impl FileBackend { let backend = Self { _base_dir: base_dir.clone(), retained_dir, - sessions_dir, queues_dir, inflight_dir, - sessions_cache: Arc::new(RwLock::new(HashMap::new())), - dirty_sessions: Arc::new(RwLock::new(HashSet::new())), - shutdown: Arc::new(AtomicBool::new(false)), + sessions, queues: QueueRegistry::new(limits, Some(writer_tx.clone())), queue_writer: writer_tx, queue_flush: flush_tx, @@ -450,72 +452,26 @@ impl FileBackend { .join(format!("{direction_tag}_{}.json", key.1)) } - /// # Errors - /// Returns an error if any session fails to persist. - pub async fn flush_sessions(&self) -> Result<()> { - let to_flush: Vec = self.dirty_sessions.read().await.iter().cloned().collect(); - - if to_flush.is_empty() { - return Ok(()); - } - - let cache = self.sessions_cache.read().await; - let mut failed = Vec::new(); - - for client_id in to_flush { - if let Some(session) = cache.get(&client_id) { - let filename = format!("{client_id}.json"); - let path = self.sessions_dir.join(filename); - if let Err(e) = self.write_file_atomic(path, session).await { - warn!("failed to persist session {}: {}", client_id, e); - failed.push(client_id); - } else { - self.dirty_sessions.write().await.remove(&client_id); - } - } else { - self.dirty_sessions.write().await.remove(&client_id); - } - } - - if failed.is_empty() { - Ok(()) - } else { - Err(MqttError::Io(format!( - "failed to persist {} sessions", - failed.len() - ))) - } + #[cfg(test)] + pub(crate) async fn pause_session_writes(&self) -> impl Sized { + self.sessions.pause_writes().await } - pub fn start_flush_task(self: &Arc, flush_interval: std::time::Duration) { - let backend = Arc::clone(self); - tokio::spawn(async move { - let mut interval = tokio::time::interval(flush_interval); - interval.set_missed_tick_behavior(tokio::time::MissedTickBehavior::Skip); - - loop { - interval.tick().await; - - if backend.shutdown.load(Ordering::Relaxed) { - if let Err(e) = backend.flush_sessions().await { - warn!("failed to flush sessions on shutdown: {e}"); - } - break; - } + #[cfg(test)] + pub(crate) async fn break_next_session_write(&self) { + self.sessions.break_next_write().await; + } - if let Err(e) = backend.flush_sessions().await { - warn!("failed to flush sessions: {e}"); - } - } - }); + #[cfg(test)] + pub(crate) async fn session_flushes(&self) -> u64 { + self.sessions.flushes().await } /// # Errors - /// Returns an error if flushing sessions fails. + /// Never fails; session writes are durable before they are acknowledged. pub async fn shutdown(&self) -> Result<()> { - self.shutdown.store(true, Ordering::Relaxed); self.flush_queue_writes().await; - self.flush_sessions().await + Ok(()) } fn send_queue_op(&self, op: QueueOp) -> Result<()> { @@ -532,52 +488,79 @@ impl FileBackend { } } - async fn check_storage_version(base_dir: &Path) -> Result<()> { + async fn check_storage_version(base_dir: &Path) -> Result { let version_file = base_dir.join(".storage_version"); - if version_file.exists() { - let stored_version = fs::read_to_string(&version_file).await.map_err(|e| { - MqttError::Configuration(format!("Failed to read storage version: {e}")) - })?; - - let stored_version = stored_version.trim(); - - if stored_version != STORAGE_VERSION { - return Err(MqttError::Configuration(format!( - "Storage version mismatch: found version {}, expected version {}.\n\ - \n\ - The storage format has changed and is incompatible.\n\ - \n\ - To resolve this issue:\n\ - 1. Backup your data: mqttv5 storage backup --dir {} --output backup.json\n\ - 2. Remove the storage directory: rm -rf {}\n\ - 3. Restart the broker (it will create a new storage with version {})\n\ - \n\ - Note: Without backup, all retained messages and session data will be lost.", - stored_version, - STORAGE_VERSION, - base_dir.display(), - base_dir.display(), - STORAGE_VERSION - ))); - } - - debug!("Storage version verified: {}", STORAGE_VERSION); - } else { + if !version_file.exists() { fs::create_dir_all(base_dir).await.map_err(|e| { MqttError::Configuration(format!("Failed to create storage dir: {e}")) })?; + Self::write_storage_version(base_dir).await?; + info!("Created new storage with version {STORAGE_VERSION}"); + return Ok(false); + } - fs::write(&version_file, STORAGE_VERSION) - .await - .map_err(|e| { - MqttError::Configuration(format!("Failed to write storage version: {e}")) - })?; + let stored_version = fs::read_to_string(&version_file).await.map_err(|e| { + MqttError::Configuration(format!("Failed to read storage version: {e}")) + })?; - info!("Created new storage with version {}", STORAGE_VERSION); + match stored_version.trim() { + STORAGE_VERSION => { + debug!("Storage version verified: {STORAGE_VERSION}"); + Ok(false) + } + LEGACY_STORAGE_VERSION => { + info!("Migrating storage from version {LEGACY_STORAGE_VERSION} to {STORAGE_VERSION}"); + Ok(true) + } + stored_version => Err(MqttError::Configuration(format!( + "Storage version mismatch: {} holds storage version {stored_version}, and this broker \ + reads versions {LEGACY_STORAGE_VERSION} and {STORAGE_VERSION} only.\n\ + \n\ + The directory was written by a newer broker, which changes the format in a way \ + this broker cannot read. To recover, do one of:\n\ + 1. Run the broker version that wrote this directory, or a newer one.\n\ + 2. Restore the backup of the storage directory taken before that upgrade, and \ + start this broker on it.\n\ + \n\ + The directory has not been modified.", + base_dir.display(), + ))), } + } - Ok(()) + async fn write_storage_version(base_dir: &Path) -> Result<()> { + let base_dir = base_dir.to_path_buf(); + tokio::task::spawn_blocking(move || Self::write_storage_version_blocking(&base_dir)) + .await + .map_err(|e| { + MqttError::Configuration(format!("Failed to write storage version: {e}")) + })? + } + + fn write_storage_version_blocking(base_dir: &Path) -> Result<()> { + use std::io::Write; + let target = base_dir.join(".storage_version"); + let temp = base_dir.join(format!( + ".storage_version.tmp.{}.{}", + std::process::id(), + TEMP_FILE_COUNTER.fetch_add(1, Ordering::Relaxed) + )); + let written = std::fs::File::create(&temp) + .and_then(|mut file| { + file.write_all(STORAGE_VERSION.as_bytes())?; + file.sync_all() + }) + .and_then(|()| std::fs::rename(&temp, &target)); + if let Err(e) = written { + if let Err(cleanup) = std::fs::remove_file(&temp) { + debug!("Could not remove {}: {cleanup}", temp.display()); + } + return Err(MqttError::Configuration(format!( + "Failed to write storage version: {e}" + ))); + } + sync_directory(base_dir) } fn topic_to_filename(topic: &str) -> String { @@ -877,55 +860,81 @@ impl StorageBackend for FileBackend { async fn store_session(&self, session: ClientSession) -> Result<()> { let client_id = session.client_id.clone(); - self.sessions_cache - .write() + self.sessions + .apply(&client_id, |_| { + (Some(SessionChange::Put(Box::new(session))), ()) + }) .await - .insert(client_id.clone(), session); - self.dirty_sessions.write().await.insert(client_id); - Ok(()) } - async fn get_session(&self, client_id: &str) -> Result> { - let cached = self.sessions_cache.read().await.get(client_id).cloned(); - if let Some(session) = cached { - if session.is_expired() { - self.remove_session(client_id).await?; - return Ok(None); - } - return Ok(Some(session)); - } - - let filename = format!("{client_id}.json"); - let path = self.sessions_dir.join(filename); - let session: Option = self.read_file(path).await?; + fn get_session( + &self, + client_id: &str, + ) -> impl std::future::Future>> + Send { + std::future::ready(Ok(self + .sessions + .get(client_id) + .filter(|session| !session.is_expired()))) + } - if let Some(ref sess) = session { - if sess.is_expired() { - self.remove_session(client_id).await?; - return Ok(None); - } - self.sessions_cache - .write() - .await - .insert(client_id.to_string(), sess.clone()); - } + async fn remove_expired_session(&self, client_id: &str) -> Result { + self.sessions + .apply(client_id, |current| { + if current.is_some_and(ClientSession::is_expired) { + (Some(SessionChange::Remove), true) + } else { + (None, false) + } + }) + .await + } - Ok(session) + fn session_client_ids(&self) -> impl std::future::Future>> + Send { + std::future::ready(Ok(self.sessions.client_ids())) } - async fn remove_session(&self, client_id: &str) -> Result<()> { - self.sessions_cache.write().await.remove(client_id); - self.dirty_sessions.write().await.remove(client_id); + async fn update_session( + &self, + client_id: &str, + connection_token: u64, + update: F, + ) -> Result + where + F: FnOnce(&mut ClientSession) + Send, + { + self.sessions + .apply(client_id, |current| { + match current.filter(|session| session.connection_token == connection_token) { + Some(current) => { + let mut updated = current.clone(); + update(&mut updated); + (Some(SessionChange::Put(Box::new(updated))), true) + } + None => (None, false), + } + }) + .await + } - let filename = format!("{client_id}.json"); - let path = self.sessions_dir.join(filename); - if path.exists() { - fs::remove_file(&path) - .await - .map_err(|e| MqttError::Io(format!("Failed to remove session file: {e}")))?; - debug!("Removed session for client: {}", client_id); - } + async fn remove_owned_session(&self, client_id: &str, connection_token: u64) -> Result { + self.sessions + .apply(client_id, |current| { + if current.is_some_and(|session| session.connection_token == connection_token) { + (Some(SessionChange::Remove), true) + } else { + (None, false) + } + }) + .await + } + async fn remove_session(&self, client_id: &str) -> Result<()> { + self.sessions + .apply(client_id, |current| { + (current.map(|_| SessionChange::Remove), ()) + }) + .await?; + debug!("Removed session for client: {client_id}"); Ok(()) } @@ -1025,20 +1034,6 @@ impl StorageBackend for FileBackend { } } - // Clean expired sessions - let session_files = self.list_files(&self.sessions_dir, "json").await?; - for file_path in session_files { - if let Some(session) = self.read_file::(file_path.clone()).await? { - if session.is_expired() { - if let Err(e) = fs::remove_file(&file_path).await { - warn!("Failed to remove expired session: {e}"); - } else { - removed_count += 1; - } - } - } - } - for queue in self.queues.handles() { // Scanned entries carry their expiry in memory (recorded during scan_queues), so // purge_expired covers them too; no per-tick re-read of every queued file. @@ -1054,10 +1049,6 @@ impl StorageBackend for FileBackend { Ok(()) } - - async fn flush_sessions(&self) -> Result<()> { - FileBackend::flush_sessions(self).await - } } #[cfg(test)] @@ -1308,4 +1299,46 @@ mod tests { assert!(!name.contains(".tmp"), "temp file left behind: {name}"); } } + + #[cfg(unix)] + #[tokio::test] + async fn storage_version_is_replaced_by_rename_not_rewritten_in_place() { + use std::os::unix::fs::PermissionsExt; + let dir = tempfile::tempdir().unwrap(); + let version = dir.path().join(".storage_version"); + std::fs::write(&version, "1").unwrap(); + std::fs::set_permissions(&version, std::fs::Permissions::from_mode(0o444)).unwrap(); + if std::fs::OpenOptions::new() + .write(true) + .open(&version) + .is_ok() + { + return; + } + let backend = FileBackend::new(dir.path()).await; + assert!(backend.is_ok(), "migration failed: {:?}", backend.err()); + assert_eq!(std::fs::read_to_string(&version).unwrap(), STORAGE_VERSION); + let leftovers: Vec<_> = std::fs::read_dir(dir.path()) + .unwrap() + .map(|entry| entry.unwrap().file_name()) + .filter(|name| name.to_string_lossy().contains(".tmp.")) + .collect(); + assert!( + leftovers.is_empty(), + "temp files left behind: {leftovers:?}" + ); + } + + #[tokio::test] + async fn newer_storage_version_names_the_recovery_steps() { + let dir = tempfile::tempdir().unwrap(); + std::fs::write(dir.path().join(".storage_version"), "3").unwrap(); + let Err(error) = FileBackend::new(dir.path()).await else { + panic!("a newer storage version was opened"); + }; + let message = error.to_string(); + assert!(message.contains("backup"), "{message}"); + assert!(!message.contains("rm -rf"), "{message}"); + assert!(!message.contains("mqttv5 storage"), "{message}"); + } } diff --git a/crates/mqtt5/src/broker/storage/memory_backend.rs b/crates/mqtt5/src/broker/storage/memory_backend.rs index bc1400ea..eaa98f57 100644 --- a/crates/mqtt5/src/broker/storage/memory_backend.rs +++ b/crates/mqtt5/src/broker/storage/memory_backend.rs @@ -112,20 +112,77 @@ impl StorageBackend for MemoryBackend { std::future::ready(Ok(())) } - async fn get_session(&self, client_id: &str) -> Result> { - let session = { - let sessions = self.sessions.lock(); - sessions.get(client_id).cloned() - }; + fn get_session( + &self, + client_id: &str, + ) -> impl std::future::Future>> + Send { + let session = self + .sessions + .lock() + .get(client_id) + .filter(|session| !session.is_expired()) + .cloned(); + std::future::ready(Ok(session)) + } - if let Some(ref sess) = session { - if sess.is_expired() { - self.remove_session(client_id).await?; - return Ok(None); - } + fn remove_expired_session( + &self, + client_id: &str, + ) -> impl std::future::Future> + Send { + let mut sessions = self.sessions.lock(); + let expired = sessions + .get(client_id) + .is_some_and(ClientSession::is_expired); + if expired { + sessions.remove(client_id); } + drop(sessions); + std::future::ready(Ok(expired)) + } + + fn session_client_ids(&self) -> impl std::future::Future>> + Send { + let ids = self.sessions.lock().keys().cloned().collect(); + std::future::ready(Ok(ids)) + } + + fn update_session( + &self, + client_id: &str, + connection_token: u64, + update: F, + ) -> impl std::future::Future> + Send + where + F: FnOnce(&mut ClientSession) + Send, + { + let updated = { + let mut sessions = self.sessions.lock(); + match sessions.get_mut(client_id) { + Some(session) if session.connection_token == connection_token => { + update(session); + true + } + _ => false, + } + }; + std::future::ready(Ok(updated)) + } - Ok(session) + fn remove_owned_session( + &self, + client_id: &str, + connection_token: u64, + ) -> impl std::future::Future> + Send { + let removed = { + let mut sessions = self.sessions.lock(); + let owned = sessions + .get(client_id) + .is_some_and(|session| session.connection_token == connection_token); + if owned { + sessions.remove(client_id); + } + owned + }; + std::future::ready(Ok(removed)) } fn remove_session( @@ -234,18 +291,6 @@ impl StorageBackend for MemoryBackend { }); } - { - let mut sessions = self.sessions.lock(); - sessions.retain(|_, session| { - if session.is_expired() { - removed_count += 1; - false - } else { - true - } - }); - } - for queue in self.queues.handles() { removed_count += queue.purge_expired(); } diff --git a/crates/mqtt5/src/broker/storage/mod.rs b/crates/mqtt5/src/broker/storage/mod.rs index 85f022f9..86acde40 100644 --- a/crates/mqtt5/src/broker/storage/mod.rs +++ b/crates/mqtt5/src/broker/storage/mod.rs @@ -9,6 +9,8 @@ pub mod file_backend; pub mod memory_backend; pub mod queue; pub mod retained; +#[cfg(not(target_arch = "wasm32"))] +mod session_log; pub mod sessions; pub use client_queue::{ @@ -276,6 +278,21 @@ pub struct ClientSession { pub change_only_state: ChangeOnlyState, #[serde(default)] pub user_id: Option, + #[serde(default)] + pub connection_token: u64, + #[serde(skip)] + pub connected: bool, + #[serde(default)] + pub disconnected_at: Option, +} + +#[must_use] +pub fn unix_millis_now() -> u64 { + SystemTime::now() + .duration_since(crate::time::UNIX_EPOCH) + .map_or(0, |elapsed| { + u64::try_from(elapsed.as_millis()).unwrap_or(u64::MAX) + }) } fn default_receive_maximum() -> u16 { @@ -427,6 +444,56 @@ pub trait StorageBackend: Send + Sync { fn flush_sessions(&self) -> impl std::future::Future> + Send { async { Ok(()) } } + + fn session_client_ids(&self) -> impl std::future::Future>> + Send { + async { Ok(Vec::new()) } + } + + fn remove_expired_session( + &self, + _client_id: &str, + ) -> impl std::future::Future> + Send { + async { Ok(false) } + } + + fn remove_owned_session( + &self, + client_id: &str, + connection_token: u64, + ) -> impl std::future::Future> + Send { + async move { + let owned = self + .get_session(client_id) + .await? + .is_some_and(|session| session.connection_token == connection_token); + if owned { + self.remove_session(client_id).await?; + } + Ok(owned) + } + } + + fn update_session( + &self, + client_id: &str, + connection_token: u64, + update: F, + ) -> impl std::future::Future> + Send + where + F: FnOnce(&mut ClientSession) + Send, + { + async move { + let Some(mut session) = self.get_session(client_id).await? else { + return Ok(false); + }; + if session.connection_token != connection_token { + return Ok(false); + } + update(&mut session); + self.store_session(session).await?; + Ok(true) + } + } } /// In-memory storage for fast access with persistent backing @@ -709,15 +776,7 @@ impl Storage { // Clean expired sessions from cache { let mut cache = self.sessions_cache.write().await; - cache.retain(|_, session| { - if let Some(expiry_interval) = session.expiry_interval { - let expiry_time = - session.last_seen + Duration::from_secs(u64::from(expiry_interval)); - expiry_time > now - } else { - true - } - }); + cache.retain(|_, session| !session.is_expired()); } // Clean expired data from backend @@ -825,6 +884,9 @@ impl ClientSession { receive_maximum: 65535, change_only_state: ChangeOnlyState::default(), user_id: None, + connection_token: 0, + connected: false, + disconnected_at: None, } } @@ -852,6 +914,9 @@ impl ClientSession { receive_maximum: 65535, change_only_state: ChangeOnlyState::default(), user_id: None, + connection_token: 0, + connected: false, + disconnected_at: None, } } @@ -877,14 +942,52 @@ impl ClientSession { /// Check if session has expired #[must_use] pub fn is_expired(&self) -> bool { - if let Some(expiry_interval) = self.expiry_interval { - let expiry_time = self.last_seen + Duration::from_secs(u64::from(expiry_interval)); - SystemTime::now() > expiry_time + if self.connected { + return false; + } + let Some(expiry_interval) = self.expiry_interval else { + return false; + }; + let since = self.disconnected_at.map_or(self.last_seen, |millis| { + crate::time::UNIX_EPOCH + Duration::from_millis(millis) + }); + SystemTime::now() > since + Duration::from_secs(u64::from(expiry_interval)) + } + + pub fn mark_connected(&mut self, connection_token: u64) { + self.connection_token = connection_token; + self.connected = true; + self.disconnected_at = None; + self.touch(); + } + + pub fn mark_disconnected(&mut self, disconnected_at: u64) { + self.connected = false; + self.disconnected_at = Some(disconnected_at); + self.touch(); + } + + #[must_use] + pub fn expiry_from_connect(connect: &crate::packet::connect::ConnectPacket) -> Option { + if connect.protocol_version == 5 { + Some( + connect + .properties + .get_session_expiry_interval() + .unwrap_or(0), + ) + } else if connect.clean_start { + Some(0) } else { - false + None } } + #[must_use] + pub fn granted_expiry(requested: Option, maximum: u32) -> u32 { + requested.map_or(maximum, |requested| requested.min(maximum)) + } + #[must_use] pub fn will_publish_delay(&self) -> Option { self.will_message.as_ref()?; @@ -1189,6 +1292,62 @@ impl StorageBackend for DynamicStorage { } } + async fn remove_expired_session(&self, client_id: &str) -> Result { + match self { + #[cfg(not(target_arch = "wasm32"))] + Self::File(backend) => backend.remove_expired_session(client_id).await, + Self::Memory(backend) => backend.remove_expired_session(client_id).await, + } + } + + async fn session_client_ids(&self) -> Result> { + match self { + #[cfg(not(target_arch = "wasm32"))] + Self::File(backend) => backend.session_client_ids().await, + Self::Memory(backend) => backend.session_client_ids().await, + } + } + + async fn remove_owned_session(&self, client_id: &str, connection_token: u64) -> Result { + match self { + #[cfg(not(target_arch = "wasm32"))] + Self::File(backend) => { + backend + .remove_owned_session(client_id, connection_token) + .await + } + Self::Memory(backend) => { + backend + .remove_owned_session(client_id, connection_token) + .await + } + } + } + + async fn update_session( + &self, + client_id: &str, + connection_token: u64, + update: F, + ) -> Result + where + F: FnOnce(&mut ClientSession) + Send, + { + match self { + #[cfg(not(target_arch = "wasm32"))] + Self::File(backend) => { + backend + .update_session(client_id, connection_token, update) + .await + } + Self::Memory(backend) => { + backend + .update_session(client_id, connection_token, update) + .await + } + } + } + async fn queue_message(&self, message: QueuedMessage) -> Result<()> { match self { #[cfg(not(target_arch = "wasm32"))] @@ -1268,20 +1427,6 @@ impl StorageBackend for DynamicStorage { } impl DynamicStorage { - /// # Errors - /// Returns an error if any session fails to persist. - #[cfg_attr( - target_arch = "wasm32", - allow(clippy::unused_async, clippy::unused_async_trait_impl) - )] - pub async fn flush_sessions(&self) -> Result<()> { - match self { - #[cfg(not(target_arch = "wasm32"))] - Self::File(backend) => backend.flush_sessions().await, - Self::Memory(_) => Ok(()), - } - } - /// # Errors /// Returns an error if flushing sessions fails. #[cfg_attr( diff --git a/crates/mqtt5/src/broker/storage/session_log.rs b/crates/mqtt5/src/broker/storage/session_log.rs new file mode 100644 index 00000000..ad70a037 --- /dev/null +++ b/crates/mqtt5/src/broker/storage/session_log.rs @@ -0,0 +1,1532 @@ +use super::{unix_millis_now, ClientSession}; +use crate::error::{MqttError, Result}; +use serde::{Deserialize, Serialize}; +use std::collections::HashMap; +use std::fs::{self, File, OpenOptions}; +use std::io::{BufWriter, Write}; +use std::path::{Path, PathBuf}; +use std::sync::atomic::{AtomicU64, Ordering}; +use std::sync::Arc; +use tokio::sync::{oneshot, Mutex as AsyncMutex, OwnedMutexGuard}; +use tracing::{debug, error, info, warn}; + +pub(super) const LOG_FILE: &str = "sessions.log"; +const COMPACT_FLOOR: u64 = 1 << 20; +const CHECKSUM_LEN: usize = 8; + +static TEMP_COUNTER: AtomicU64 = AtomicU64::new(0); + +const CRC_TABLE: [u32; 256] = { + let mut table = [0u32; 256]; + let mut index: u32 = 0; + while index < 256 { + let mut crc = index; + let mut bit = 0; + while bit < 8 { + crc = if crc & 1 == 1 { + 0xEDB8_8320 ^ (crc >> 1) + } else { + crc >> 1 + }; + bit += 1; + } + table[index as usize] = crc; + index += 1; + } + table +}; + +fn crc32(bytes: &[u8]) -> u32 { + !bytes.iter().fold(u32::MAX, |crc, byte| { + CRC_TABLE[((crc ^ u32::from(*byte)) & 0xFF) as usize] ^ (crc >> 8) + }) +} + +#[derive(Serialize, Deserialize, Clone, Copy, PartialEq, Eq)] +#[serde(rename_all = "snake_case")] +enum Op { + Put, + Remove, +} + +#[derive(Serialize)] +struct RecordRef<'a> { + op: Op, + id: &'a str, + #[serde(skip_serializing_if = "Option::is_none")] + session: Option<&'a ClientSession>, +} + +#[derive(Deserialize)] +struct Record { + op: Op, + id: String, + #[serde(default)] + session: Option, +} + +enum Replayed { + Put(String, Box), + Remove(String), +} + +pub(super) enum SessionChange { + Put(Box), + Remove, +} + +enum Entry { + Barrier, + Write { + client_id: String, + line: Vec, + live: bool, + undo: Option>, + }, +} + +struct Pending { + seq: u64, + entry: Entry, + done: oneshot::Sender>, +} + +struct LogState { + sessions: HashMap, + pending: Vec, + next_seq: u64, + settled_seq: u64, +} + +impl LogState { + fn all_settled(&self) -> bool { + self.settled_seq + 1 == self.next_seq + } + + fn enqueue(&mut self, entry: Entry) -> oneshot::Receiver> { + let (done, settled) = oneshot::channel(); + let seq = self.next_seq; + self.next_seq += 1; + self.pending.push(Pending { seq, entry, done }); + settled + } +} + +struct LogLine { + client_id: String, + line: Vec, + live: bool, +} + +struct LogWriter { + dir: PathBuf, + path: PathBuf, + file: Option, + len: u64, + durable: HashMap>, + live_bytes: u64, + health: Health, + flushes: u64, + #[cfg(test)] + gate: Option>, + #[cfg(test)] + faults: Faults, +} + +#[derive(Clone, Copy, PartialEq, Eq)] +enum Health { + Sound, + NeedsRepair, + DamagedOriginal, +} + +#[cfg(test)] +#[derive(Default)] +struct Faults { + sync: bool, + truncate: bool, +} + +impl LogWriter { + fn append(&mut self, lines: &[LogLine]) -> Result<()> { + #[cfg(test)] + if let Some(gate) = self.gate.take() { + gate.wait(); + } + if self.health != Health::Sound { + self.repair()?; + } + let bytes: Vec = lines + .iter() + .flat_map(|line| line.line.iter().copied()) + .collect(); + if !bytes.is_empty() { + if let Err(e) = self.write_and_sync(&bytes) { + self.discard_failed_append(); + return Err(e); + } + self.len += bytes.len() as u64; + self.flushes += 1; + debug!( + records = lines.len(), + flushes = self.flushes, + "Group-committed session writes" + ); + } + for line in lines { + self.record_durable(&line.client_id, &line.line, line.live); + } + Ok(()) + } + + fn write_and_sync(&mut self, bytes: &[u8]) -> Result<()> { + let file = self + .file + .as_mut() + .ok_or_else(|| MqttError::Io("session log is not open".to_string()))?; + file.write_all(bytes) + .map_err(|e| MqttError::Io(format!("Failed to append to the session log: {e}")))?; + #[cfg(test)] + if std::mem::take(&mut self.faults.sync) { + return Err(MqttError::Io("injected fsync failure".to_string())); + } + file.sync_data() + .map_err(|e| MqttError::Io(format!("Failed to sync the session log: {e}"))) + } + + fn discard_failed_append(&mut self) { + match self.truncate_to_durable() { + Ok(()) => { + debug!( + bytes = self.len, + "Truncated the session log back to its durable length" + ); + return; + } + Err(e) => warn!("Could not truncate the failed session write away: {e}"), + } + self.health = Health::NeedsRepair; + if let Err(e) = self.repair() { + error!("Session log repair failed; session writes are refused until it succeeds: {e}"); + } + } + + fn truncate_to_durable(&mut self) -> Result<()> { + #[cfg(test)] + if std::mem::take(&mut self.faults.truncate) { + return Err(MqttError::Io("injected truncate failure".to_string())); + } + let file = self + .file + .as_mut() + .ok_or_else(|| MqttError::Io("session log is not open".to_string()))?; + file.set_len(self.len) + .and_then(|()| file.sync_all()) + .map_err(|e| MqttError::Io(format!("Failed to truncate the session log: {e}"))) + } + + fn record_durable(&mut self, client_id: &str, record: &[u8], keep: bool) { + let previous = if keep { + self.durable.insert(client_id.to_string(), record.to_vec()) + } else { + self.durable.remove(client_id) + }; + if let Some(previous) = previous { + self.live_bytes = self.live_bytes.saturating_sub(previous.len() as u64); + } + if keep { + self.live_bytes += record.len() as u64; + } + } + + fn needs_compaction(&self) -> bool { + self.len > COMPACT_FLOOR.max(self.live_bytes.saturating_mul(2)) + } + + fn repair(&mut self) -> Result<()> { + if self.health == Health::DamagedOriginal { + let copy = preserve_copy(&self.dir, &self.path)?; + self.health = Health::NeedsRepair; + error!( + "Session log {} held damaged records; the original is kept as {}", + self.path.display(), + copy.display() + ); + } + self.compact() + } + + fn compact(&mut self) -> Result<()> { + let temp = self.dir.join(format!( + "{LOG_FILE}.tmp.{}.{}", + std::process::id(), + TEMP_COUNTER.fetch_add(1, Ordering::Relaxed) + )); + if let Err(e) = write_snapshot(&temp, self.durable.values()) { + remove_quietly(&temp); + self.mark_for_repair(); + return Err(e); + } + self.file = None; + if let Err(e) = fs::rename(&temp, &self.path) { + remove_quietly(&temp); + self.mark_for_repair(); + return Err(MqttError::Io(format!( + "Failed to install the compacted session log: {e}" + ))); + } + let reopened = sync_directory(&self.dir).and_then(|()| open_append(&self.path)); + match reopened { + Ok(file) => { + self.file = Some(file); + self.len = self.live_bytes; + self.health = Health::Sound; + debug!(bytes = self.len, "Compacted the session log"); + Ok(()) + } + Err(e) => { + self.mark_for_repair(); + Err(e) + } + } + } + + fn mark_for_repair(&mut self) { + if self.health == Health::Sound { + self.health = Health::NeedsRepair; + } + } +} + +struct Shared { + state: parking_lot::Mutex, + writer: Arc>, +} + +pub(super) struct SessionLog { + shared: Arc, +} + +impl SessionLog { + pub(super) async fn open(dir: PathBuf, import_legacy: bool) -> Result { + let (writer, sessions) = + tokio::task::spawn_blocking(move || open_blocking(&dir, import_legacy)) + .await + .map_err(|e| MqttError::Io(format!("Session log loader failed: {e}")))??; + Ok(Self { + shared: Arc::new(Shared { + state: parking_lot::Mutex::new(LogState { + sessions, + pending: Vec::new(), + next_seq: 1, + settled_seq: 0, + }), + writer: Arc::new(AsyncMutex::new(writer)), + }), + }) + } + + #[cfg(test)] + pub(super) async fn pause_writes(&self) -> impl Sized { + Arc::clone(&self.shared.writer).lock_owned().await + } + + #[cfg(test)] + pub(super) async fn break_next_write(&self) { + self.shared.writer.lock().await.file = None; + } + + #[cfg(test)] + pub(super) async fn flushes(&self) -> u64 { + self.shared.writer.lock().await.flushes + } + + pub(super) fn get(&self, client_id: &str) -> Option { + self.shared.state.lock().sessions.get(client_id).cloned() + } + + pub(super) fn client_ids(&self) -> Vec { + self.shared.state.lock().sessions.keys().cloned().collect() + } + + pub(super) async fn apply( + &self, + client_id: &str, + decide: impl FnOnce(Option<&ClientSession>) -> (Option, R), + ) -> Result { + let (settled, outcome) = { + let mut state = self.shared.state.lock(); + let (change, outcome) = decide(state.sessions.get(client_id)); + let entry = match change { + Some(SessionChange::Put(session)) => { + let line = encode(client_id, Some(&session))?; + let undo = state + .sessions + .insert(client_id.to_string(), *session) + .map(Box::new); + Entry::Write { + client_id: client_id.to_string(), + line, + live: true, + undo, + } + } + Some(SessionChange::Remove) => { + let line = encode(client_id, None)?; + let undo = state.sessions.remove(client_id).map(Box::new); + Entry::Write { + client_id: client_id.to_string(), + line, + live: false, + undo, + } + } + None if state.all_settled() => return Ok(outcome), + None => Entry::Barrier, + }; + (state.enqueue(entry), outcome) + }; + tokio::spawn(flush(Arc::clone(&self.shared))); + match settled.await { + Ok(Ok(())) => Ok(outcome), + Ok(Err(e)) => Err(e), + Err(_) => Err(MqttError::Io( + "session log writer stopped before the write settled".to_string(), + )), + } + } +} + +async fn flush(shared: Arc) { + let writer = Arc::clone(&shared.writer).lock_owned().await; + let batch = std::mem::take(&mut shared.state.lock().pending); + let Some(last_seq) = batch.last().map(|pending| pending.seq) else { + return; + }; + let lines: Vec = batch + .iter() + .filter_map(|pending| match &pending.entry { + Entry::Barrier => None, + Entry::Write { + client_id, + line, + live, + .. + } => Some(LogLine { + client_id: client_id.clone(), + line: line.clone(), + live: *live, + }), + }) + .collect(); + let appended = tokio::task::spawn_blocking(move || { + let mut writer = writer; + let result = writer.append(&lines); + (writer, result) + }) + .await; + let (writer, result) = match appended { + Ok((writer, result)) => (Some(writer), result), + Err(e) => ( + None, + Err(MqttError::Io(format!("Session log writer failed: {e}"))), + ), + }; + match result { + Ok(()) => settle(&shared, batch, last_seq), + Err(e) => { + warn!("Session log write failed; rejecting the pending session writes: {e}"); + fail(&shared, batch, &e); + } + } + if let Some(writer) = writer { + compact_if_needed(writer).await; + } +} + +fn settle(shared: &Shared, batch: Vec, last_seq: u64) { + shared.state.lock().settled_seq = last_seq; + for pending in batch { + if pending.done.send(Ok(())).is_err() { + debug!("Session write settled after its caller went away"); + } + } +} + +fn fail(shared: &Shared, batch: Vec, error: &MqttError) { + let mut state = shared.state.lock(); + let queued = std::mem::take(&mut state.pending); + let failed: Vec = batch.into_iter().chain(queued).collect(); + if let Some(last) = failed.last() { + state.settled_seq = last.seq; + } + for pending in failed.iter().rev() { + if let Entry::Write { + client_id, undo, .. + } = &pending.entry + { + match undo { + Some(previous) => { + state + .sessions + .insert(client_id.clone(), previous.as_ref().clone()); + } + None => { + state.sessions.remove(client_id); + } + } + } + } + drop(state); + for pending in failed { + if pending.done.send(Err(error.clone())).is_err() { + debug!("Session write failed after its caller went away"); + } + } +} + +async fn compact_if_needed(writer: OwnedMutexGuard) { + if writer.health != Health::Sound || !writer.needs_compaction() { + return; + } + let compacted = tokio::task::spawn_blocking(move || { + let mut writer = writer; + writer.compact() + }) + .await; + match compacted { + Ok(Ok(())) => {} + Ok(Err(e)) => warn!("Session log compaction failed; will retry: {e}"), + Err(e) => warn!("Session log compaction task failed: {e}"), + } +} + +fn encode(client_id: &str, session: Option<&ClientSession>) -> Result> { + let op = if session.is_some() { + Op::Put + } else { + Op::Remove + }; + let body = serde_json::to_vec(&RecordRef { + op, + id: client_id, + session, + }) + .map_err(|e| MqttError::Io(format!("Failed to encode session {client_id}: {e}")))?; + let mut line = Vec::with_capacity(CHECKSUM_LEN + body.len() + 2); + line.extend_from_slice(format!("{:08x} ", crc32(&body)).as_bytes()); + line.extend_from_slice(&body); + line.push(b'\n'); + Ok(line) +} + +fn decode(line: &[u8]) -> Option { + let (checksum, body) = line.split_at_checked(CHECKSUM_LEN + 1)?; + let (hex, separator) = checksum.split_at(CHECKSUM_LEN); + if separator != b" " { + return None; + } + let expected = u32::from_str_radix(std::str::from_utf8(hex).ok()?, 16).ok()?; + if crc32(body) != expected { + return None; + } + let record = serde_json::from_slice::(body).ok()?; + match (record.op, record.session) { + (Op::Put, Some(session)) if session.client_id == record.id => { + Some(Replayed::Put(record.id, Box::new(session))) + } + (Op::Remove, None) => Some(Replayed::Remove(record.id)), + _ => None, + } +} + +type Loaded = HashMap)>; + +#[derive(Default)] +struct ReplaySummary { + damaged: usize, + torn_bytes: usize, +} + +fn open_blocking( + dir: &Path, + import_legacy: bool, +) -> Result<(LogWriter, HashMap)> { + remove_temp_files(dir); + let path = dir.join(LOG_FILE); + let mut loaded = Loaded::new(); + let mut summary = ReplaySummary::default(); + match fs::read(&path) { + Ok(bytes) => summary = replay(&bytes, &mut loaded), + Err(e) if e.kind() == std::io::ErrorKind::NotFound => {} + Err(e) => { + let aside = unique_aside(&path, "unreadable"); + error!( + "Session log {} is unreadable ({e}); moving it to {} and starting without sessions", + path.display(), + aside.display() + ); + fs::rename(&path, &aside).map_err(|e| { + MqttError::Io(format!( + "Session log {} is unreadable and cannot be moved aside: {e}", + path.display() + )) + })?; + } + } + if summary.damaged > 0 { + error!( + damaged = summary.damaged, + "Session log {} has damaged records; they were skipped and every other record was replayed", + path.display() + ); + } + if summary.torn_bytes > 0 { + warn!( + discarded = summary.torn_bytes, + "Session log ends in an incomplete record from an unfinished write; discarding it" + ); + } + let legacy = if import_legacy { + import_legacy_files(dir, &mut loaded) + } else { + warn_about_legacy_files(dir); + Vec::new() + }; + + let durable: HashMap> = loaded + .iter() + .map(|(id, (_, line))| (id.clone(), line.clone())) + .collect(); + let live_bytes = durable.values().map(|line| line.len() as u64).sum(); + let mut writer = LogWriter { + dir: dir.to_path_buf(), + path, + file: None, + len: 0, + durable, + live_bytes, + health: if summary.damaged > 0 { + Health::DamagedOriginal + } else { + Health::NeedsRepair + }, + flushes: 0, + #[cfg(test)] + gate: None, + #[cfg(test)] + faults: Faults::default(), + }; + if let Err(e) = writer.repair() { + if import_legacy { + return Err(MqttError::Io(format!( + "Cannot migrate the sessions in {} to the session log: {e}", + dir.display() + ))); + } + warn!( + "Session log could not be rewritten at startup; serving the replayed sessions and refusing session writes until it can: {e}" + ); + } + for file in &legacy { + if let Err(e) = fs::remove_file(file) { + warn!( + "Imported legacy session file {} could not be removed: {e}", + file.display() + ); + } + } + if !legacy.is_empty() { + sync_directory(dir)?; + info!( + count = legacy.len(), + "Migrated legacy session files into the session log" + ); + } + let sessions = loaded + .into_iter() + .map(|(id, (session, _))| (id, session)) + .collect(); + Ok((writer, sessions)) +} + +fn replay(bytes: &[u8], loaded: &mut Loaded) -> ReplaySummary { + let mut summary = ReplaySummary::default(); + for segment in bytes.split_inclusive(|byte| *byte == b'\n') { + let Some(line) = segment.strip_suffix(b"\n") else { + summary.torn_bytes = segment.len(); + break; + }; + match decode(line) { + Some(Replayed::Put(id, session)) => { + loaded.insert(id, (*session, segment.to_vec())); + } + Some(Replayed::Remove(id)) => { + loaded.remove(&id); + } + None => summary.damaged += 1, + } + } + summary +} + +fn remove_temp_files(dir: &Path) { + let Ok(entries) = fs::read_dir(dir) else { + return; + }; + for path in entries + .filter_map(std::result::Result::ok) + .map(|e| e.path()) + { + let is_temp = path + .file_name() + .and_then(|name| name.to_str()) + .is_some_and(is_temp_name); + if is_temp && path.is_file() { + match fs::remove_file(&path) { + Ok(()) => info!("Removed leftover temporary file {}", path.display()), + Err(e) => warn!( + "Could not remove leftover temporary file {}: {e}", + path.display() + ), + } + } + } +} + +fn is_temp_name(name: &str) -> bool { + let mut parts = name.rsplit('.'); + let counter = parts.next(); + let pid = parts.next(); + let marker = parts.next(); + let numeric = |part: Option<&str>| { + part.is_some_and(|part| !part.is_empty() && part.bytes().all(|byte| byte.is_ascii_digit())) + }; + marker == Some("tmp") && numeric(pid) && numeric(counter) && parts.next().is_some() +} + +fn unique_aside(path: &Path, tag: &str) -> PathBuf { + let name = path + .file_name() + .and_then(|name| name.to_str()) + .unwrap_or(LOG_FILE); + let stamp = unix_millis_now(); + let mut attempt = 0u32; + loop { + let candidate = if attempt == 0 { + path.with_file_name(format!("{name}.{tag}-{stamp}")) + } else { + path.with_file_name(format!("{name}.{tag}-{stamp}-{attempt}")) + }; + if !candidate.exists() { + return candidate; + } + attempt += 1; + } +} + +fn preserve_copy(dir: &Path, path: &Path) -> Result { + let aside = unique_aside(path, "corrupt"); + let copied = fs::copy(path, &aside) + .and_then(|_| File::open(&aside)) + .and_then(|copy| copy.sync_all()); + if let Err(e) = copied { + remove_quietly(&aside); + return Err(MqttError::Io(format!( + "Failed to preserve the damaged session log as {}: {e}", + aside.display() + ))); + } + sync_directory(dir)?; + Ok(aside) +} + +fn legacy_files(dir: &Path) -> Vec { + let Ok(entries) = fs::read_dir(dir) else { + return Vec::new(); + }; + entries + .filter_map(std::result::Result::ok) + .map(|entry| entry.path()) + .filter(|path| path.is_file() && path.extension().is_some_and(|ext| ext == "json")) + .collect() +} + +fn warn_about_legacy_files(dir: &Path) { + let leftover = legacy_files(dir).len(); + if leftover > 0 { + warn!( + leftover, + dir = %dir.display(), + "Ignoring per-session files left beside the session log" + ); + } +} + +fn quarantine_legacy(path: &Path, reason: &str) { + let aside = unique_aside(path, "corrupt"); + match fs::rename(path, &aside) { + Ok(()) => error!( + "Legacy session file {} was not migrated ({reason}); kept as {}", + path.display(), + aside.display() + ), + Err(e) => error!( + "Legacy session file {} was not migrated ({reason}) and could not be moved aside: {e}", + path.display() + ), + } +} + +fn import_legacy_files(dir: &Path, loaded: &mut Loaded) -> Vec { + let mut imported = Vec::new(); + for path in legacy_files(dir) { + let bytes = match fs::read(&path) { + Ok(bytes) => bytes, + Err(e) => { + quarantine_legacy(&path, &e.to_string()); + continue; + } + }; + let session = match serde_json::from_slice::(&bytes) { + Ok(session) => session, + Err(e) => { + quarantine_legacy(&path, &e.to_string()); + continue; + } + }; + match encode(&session.client_id, Some(&session)) { + Ok(line) => { + loaded.insert(session.client_id.clone(), (session, line)); + imported.push(path); + } + Err(e) => quarantine_legacy(&path, &e.to_string()), + } + } + imported +} + +fn write_snapshot<'a>(path: &Path, lines: impl Iterator>) -> Result<()> { + let file = File::create(path) + .map_err(|e| MqttError::Io(format!("Failed to create {}: {e}", path.display())))?; + let mut out = BufWriter::new(file); + for line in lines { + out.write_all(line) + .map_err(|e| MqttError::Io(format!("Failed to write {}: {e}", path.display())))?; + } + let file = out + .into_inner() + .map_err(|e| MqttError::Io(format!("Failed to write {}: {e}", path.display())))?; + file.sync_all() + .map_err(|e| MqttError::Io(format!("Failed to sync {}: {e}", path.display()))) +} + +fn open_append(path: &Path) -> Result { + OpenOptions::new() + .append(true) + .open(path) + .map_err(|e| MqttError::Io(format!("Failed to open {}: {e}", path.display()))) +} + +fn remove_quietly(path: &Path) { + if let Err(e) = fs::remove_file(path) { + debug!("Could not remove {}: {e}", path.display()); + } +} + +#[cfg(unix)] +pub(super) fn sync_directory(dir: &Path) -> Result<()> { + File::open(dir) + .and_then(|handle| handle.sync_all()) + .map_err(|e| MqttError::Io(format!("Failed to sync {}: {e}", dir.display()))) +} + +#[cfg(not(unix))] +pub(super) fn sync_directory(dir: &Path) -> Result<()> { + fs::metadata(dir) + .map(|_| ()) + .map_err(|e| MqttError::Io(format!("Failed to stat {}: {e}", dir.display()))) +} + +#[cfg(test)] +mod tests { + use super::{SessionChange, SessionLog, LOG_FILE}; + use crate::broker::storage::ClientSession; + use std::path::Path; + use std::sync::Arc; + use std::time::Duration; + + async fn open(dir: &Path) -> Arc { + Arc::new(SessionLog::open(dir.to_path_buf(), false).await.unwrap()) + } + + async fn put(log: &SessionLog, session: ClientSession) -> crate::error::Result<()> { + let client_id = session.client_id.clone(); + log.apply(&client_id, |_| { + (Some(SessionChange::Put(Box::new(session))), ()) + }) + .await + } + + fn spawn_put( + log: &Arc, + session: ClientSession, + ) -> tokio::task::JoinHandle> { + let log = Arc::clone(log); + tokio::spawn(async move { put(&log, session).await }) + } + + fn on_disk(dir: &Path) -> String { + std::fs::read_to_string(dir.join(LOG_FILE)).unwrap() + } + + async fn wait_for_pending(log: &SessionLog, count: usize) { + for _ in 0..500 { + if log.shared.state.lock().pending.len() >= count { + return; + } + tokio::time::sleep(Duration::from_millis(2)).await; + } + panic!("writes never queued"); + } + + #[tokio::test] + async fn write_is_visible_at_once_and_acknowledged_only_once_durable() { + let dir = tempfile::tempdir().unwrap(); + let log = open(dir.path()).await; + let paused = Arc::clone(&log.shared.writer).lock_owned().await; + let write = spawn_put(&log, ClientSession::new("a", true, Some(60))); + wait_for_pending(&log, 1).await; + assert!(log.get("a").is_some(), "a later reader must see the write"); + tokio::time::sleep(Duration::from_millis(50)).await; + assert!( + !write.is_finished(), + "the write was acknowledged before it was durable" + ); + assert!(!on_disk(dir.path()).contains("\"id\":\"a\"")); + drop(paused); + write.await.unwrap().unwrap(); + assert!(on_disk(dir.path()).contains("\"id\":\"a\"")); + } + + #[tokio::test] + async fn removal_is_acknowledged_only_once_durable() { + let dir = tempfile::tempdir().unwrap(); + let log = open(dir.path()).await; + put(&log, ClientSession::new("a", true, Some(60))) + .await + .unwrap(); + let paused = Arc::clone(&log.shared.writer).lock_owned().await; + let removing = Arc::clone(&log); + let removal = tokio::spawn(async move { + removing + .apply("a", |current| (current.map(|_| SessionChange::Remove), ())) + .await + }); + wait_for_pending(&log, 1).await; + assert!(log.get("a").is_none()); + tokio::time::sleep(Duration::from_millis(50)).await; + assert!( + !removal.is_finished(), + "the removal was acknowledged before it was durable" + ); + drop(paused); + removal.await.unwrap().unwrap(); + drop(log); + assert!(open(dir.path()).await.get("a").is_none()); + } + + #[tokio::test] + async fn unchanged_outcome_waits_for_the_writes_it_observed() { + let dir = tempfile::tempdir().unwrap(); + let log = open(dir.path()).await; + let paused = Arc::clone(&log.shared.writer).lock_owned().await; + let write = spawn_put(&log, ClientSession::new("a", true, Some(60))); + wait_for_pending(&log, 1).await; + let reading = Arc::clone(&log); + let observed = tokio::spawn(async move { + reading + .apply("a", |current| (None, current.is_some())) + .await + }); + tokio::time::sleep(Duration::from_millis(50)).await; + assert!( + !observed.is_finished(), + "an outcome based on an unflushed write was acknowledged before that write" + ); + drop(paused); + write.await.unwrap().unwrap(); + assert!(observed.await.unwrap().unwrap()); + let idle = log.apply("b", |current| (None, current.is_some())).await; + assert!(!idle.unwrap()); + } + + #[tokio::test] + async fn concurrent_writes_share_one_flush() { + let dir = tempfile::tempdir().unwrap(); + let log = open(dir.path()).await; + let paused = Arc::clone(&log.shared.writer).lock_owned().await; + let before = paused.flushes; + let writes: Vec<_> = (0..200) + .map(|i| spawn_put(&log, ClientSession::new(format!("c{i}"), true, Some(60)))) + .collect(); + wait_for_pending(&log, 200).await; + drop(paused); + for write in writes { + write.await.unwrap().unwrap(); + } + assert_eq!(log.shared.writer.lock().await.flushes, before + 1); + drop(log); + let reopened = open(dir.path()).await; + assert_eq!(reopened.client_ids().len(), 200); + } + + #[tokio::test] + async fn failed_flush_fails_every_pending_write_and_restores_what_was_durable() { + let dir = tempfile::tempdir().unwrap(); + let log = open(dir.path()).await; + put(&log, ClientSession::new("a", true, Some(60))) + .await + .unwrap(); + + let mut paused = Arc::clone(&log.shared.writer).lock_owned().await; + paused.file = None; + let changed = spawn_put(&log, ClientSession::new("a", true, Some(99))); + let added = spawn_put(&log, ClientSession::new("b", true, Some(60))); + wait_for_pending(&log, 2).await; + assert_eq!(log.get("a").unwrap().expiry_interval, Some(99)); + drop(paused); + assert!(changed.await.unwrap().is_err()); + assert!(added.await.unwrap().is_err()); + assert_eq!(log.get("a").unwrap().expiry_interval, Some(60)); + assert!(log.get("b").is_none()); + + put(&log, ClientSession::new("c", true, Some(60))) + .await + .expect("the next write repairs the log"); + drop(log); + let reopened = open(dir.path()).await; + assert_eq!(reopened.get("a").unwrap().expiry_interval, Some(60)); + assert!(reopened.get("b").is_none()); + assert!(reopened.get("c").is_some()); + } + + #[tokio::test(flavor = "multi_thread", worker_threads = 2)] + async fn failed_flush_also_fails_writes_queued_behind_it() { + let dir = tempfile::tempdir().unwrap(); + let log = open(dir.path()).await; + put(&log, ClientSession::new("k", true, Some(1))) + .await + .unwrap(); + let gate = Arc::new(std::sync::Barrier::new(2)); + { + let mut writer = log.shared.writer.lock().await; + writer.file = None; + writer.gate = Some(Arc::clone(&gate)); + } + let first = spawn_put(&log, ClientSession::new("k", true, Some(2))); + for _ in 0..500 { + if log.get("k").unwrap().expiry_interval == Some(2) + && log.shared.state.lock().pending.is_empty() + { + break; + } + tokio::time::sleep(Duration::from_millis(2)).await; + } + let building = Arc::clone(&log); + let second = tokio::spawn(async move { + building + .apply("k", |current| { + let mut next = current.unwrap().clone(); + next.persistent = false; + (Some(SessionChange::Put(Box::new(next))), ()) + }) + .await + }); + wait_for_pending(&log, 1).await; + tokio::task::spawn_blocking(move || { + gate.wait(); + }) + .await + .unwrap(); + assert!(first.await.unwrap().is_err()); + assert!( + second.await.unwrap().is_err(), + "a write built on a failed write must fail with it" + ); + let restored = log.get("k").unwrap(); + assert_eq!(restored.expiry_interval, Some(1)); + assert!(restored.persistent); + put(&log, ClientSession::new("other", true, Some(1))) + .await + .unwrap(); + drop(log); + let reopened = open(dir.path()).await.get("k").unwrap(); + assert_eq!(reopened.expiry_interval, Some(1)); + assert!(reopened.persistent); + } + + #[tokio::test(flavor = "multi_thread", worker_threads = 2)] + async fn earlier_write_never_lands_over_a_later_one_for_the_same_client_id() { + let dir = tempfile::tempdir().unwrap(); + let log = open(dir.path()).await; + let gate = Arc::new(std::sync::Barrier::new(2)); + log.shared.writer.lock().await.gate = Some(Arc::clone(&gate)); + let older = spawn_put(&log, ClientSession::new("k", true, Some(1))); + for _ in 0..500 { + if log.get("k").is_some() && log.shared.state.lock().pending.is_empty() { + break; + } + tokio::time::sleep(Duration::from_millis(2)).await; + } + let newer = spawn_put(&log, ClientSession::new("k", true, Some(2))); + wait_for_pending(&log, 1).await; + assert!(!older.is_finished() && !newer.is_finished()); + tokio::task::spawn_blocking(move || { + gate.wait(); + }) + .await + .unwrap(); + older.await.unwrap().unwrap(); + newer.await.unwrap().unwrap(); + assert_eq!(log.get("k").unwrap().expiry_interval, Some(2)); + + let paused = Arc::clone(&log.shared.writer).lock_owned().await; + let third = spawn_put(&log, ClientSession::new("k", true, Some(3))); + wait_for_pending(&log, 1).await; + let fourth = spawn_put(&log, ClientSession::new("k", true, Some(4))); + wait_for_pending(&log, 2).await; + drop(paused); + third.await.unwrap().unwrap(); + fourth.await.unwrap().unwrap(); + drop(log); + assert_eq!( + open(dir.path()).await.get("k").unwrap().expiry_interval, + Some(4), + "the durable record went back to an older write" + ); + } + + #[tokio::test] + async fn torn_tail_is_discarded_and_later_writes_survive() { + let dir = tempfile::tempdir().unwrap(); + let log = open(dir.path()).await; + put(&log, ClientSession::new("a", true, Some(60))) + .await + .unwrap(); + drop(log); + let mut file = std::fs::OpenOptions::new() + .append(true) + .open(dir.path().join(LOG_FILE)) + .unwrap(); + std::io::Write::write_all(&mut file, b"{\"id\":\"x\",\"sess").unwrap(); + drop(file); + + let log = open(dir.path()).await; + assert!(log.get("a").is_some()); + assert!(log.get("x").is_none()); + put(&log, ClientSession::new("b", true, Some(60))) + .await + .unwrap(); + drop(log); + let reopened = open(dir.path()).await; + assert!(reopened.get("a").is_some()); + assert!(reopened.get("b").is_some()); + } + + #[tokio::test] + async fn unreadable_line_between_records_is_skipped() { + let dir = tempfile::tempdir().unwrap(); + let first = super::encode("a", Some(&ClientSession::new("a", true, Some(60)))).unwrap(); + let last = super::encode("b", Some(&ClientSession::new("b", true, Some(60)))).unwrap(); + let mut bytes = first; + bytes.extend_from_slice(b"\0\0\0garbage\n"); + bytes.extend_from_slice(&last); + std::fs::write(dir.path().join(LOG_FILE), bytes).unwrap(); + let log = open(dir.path()).await; + assert!(log.get("a").is_some()); + assert!( + log.get("b").is_some(), + "a record after a damaged one stands alone and must be replayed" + ); + } + + #[test] + fn checksum_is_crc32() { + assert_eq!(super::crc32(b"123456789"), 0xCBF4_3926); + } + + #[tokio::test] + async fn removal_is_replayed() { + let dir = tempfile::tempdir().unwrap(); + let log = open(dir.path()).await; + put(&log, ClientSession::new("a", true, Some(60))) + .await + .unwrap(); + log.apply("a", |current| (current.map(|_| SessionChange::Remove), ())) + .await + .unwrap(); + assert!(log.get("a").is_none()); + drop(log); + assert!(open(dir.path()).await.get("a").is_none()); + } + + #[tokio::test] + async fn compaction_bounds_the_log() { + let dir = tempfile::tempdir().unwrap(); + let log = open(dir.path()).await; + for round in 0..300u32 { + let mut session = ClientSession::new("big", true, Some(round)); + session.user_id = Some("u".repeat(10_000)); + put(&log, session).await.unwrap(); + } + let len = std::fs::metadata(dir.path().join(LOG_FILE)).unwrap().len(); + assert!( + len <= super::COMPACT_FLOOR + 20_000, + "log grew to {len} bytes" + ); + drop(log); + assert_eq!( + open(dir.path()).await.get("big").unwrap().expiry_interval, + Some(299) + ); + } + + #[cfg(unix)] + #[tokio::test] + async fn unreadable_log_is_moved_aside_instead_of_blocking_startup() { + use std::os::unix::fs::PermissionsExt; + let dir = tempfile::tempdir().unwrap(); + let log = open(dir.path()).await; + put(&log, ClientSession::new("a", true, Some(60))) + .await + .unwrap(); + drop(log); + let path = dir.path().join(LOG_FILE); + std::fs::set_permissions(&path, std::fs::Permissions::from_mode(0o000)).unwrap(); + if std::fs::read(&path).is_ok() { + return; + } + let log = open(dir.path()).await; + assert!(log.get("a").is_none()); + assert_eq!(siblings(dir.path(), "sessions.log.unreadable-").len(), 1); + put(&log, ClientSession::new("b", true, Some(60))) + .await + .unwrap(); + drop(log); + std::fs::set_permissions(&path, std::fs::Permissions::from_mode(0o000)).unwrap(); + drop(open(dir.path()).await); + assert_eq!( + siblings(dir.path(), "sessions.log.unreadable-").len(), + 2, + "a second unreadable log overwrote the first one moved aside" + ); + } + + #[cfg(unix)] + #[tokio::test] + async fn writes_are_refused_while_the_log_cannot_be_repaired() { + use std::os::unix::fs::PermissionsExt; + let dir = tempfile::tempdir().unwrap(); + let log = open(dir.path()).await; + put(&log, ClientSession::new("a", true, Some(60))) + .await + .unwrap(); + std::fs::set_permissions(dir.path(), std::fs::Permissions::from_mode(0o555)).unwrap(); + if std::fs::write(dir.path().join("probe"), b"").is_ok() { + std::fs::set_permissions(dir.path(), std::fs::Permissions::from_mode(0o755)).unwrap(); + return; + } + { + let mut writer = log.shared.writer.lock().await; + writer.faults.sync = true; + writer.faults.truncate = true; + } + let failed = put(&log, ClientSession::new("a", true, Some(99))).await; + let refused = put(&log, ClientSession::new("b", true, Some(60))).await; + std::fs::set_permissions(dir.path(), std::fs::Permissions::from_mode(0o755)).unwrap(); + assert!(failed.is_err()); + assert!( + refused.is_err(), + "a write was appended after a failed write that could not be removed" + ); + assert_eq!(log.get("a").unwrap().expiry_interval, Some(60)); + put(&log, ClientSession::new("c", true, Some(60))) + .await + .expect("the next write repairs the log"); + drop(log); + let reopened = open(dir.path()).await; + assert_eq!(reopened.get("a").unwrap().expiry_interval, Some(60)); + assert!(reopened.get("b").is_none()); + assert!(reopened.get("c").is_some()); + } + + #[cfg(unix)] + #[tokio::test] + async fn legacy_file_that_cannot_be_read_is_moved_aside() { + use std::os::unix::fs::PermissionsExt; + let dir = tempfile::tempdir().unwrap(); + let unreadable = dir.path().join("locked.json"); + std::fs::write( + &unreadable, + serde_json::to_vec(&ClientSession::new("locked", true, Some(60))).unwrap(), + ) + .unwrap(); + std::fs::set_permissions(&unreadable, std::fs::Permissions::from_mode(0o000)).unwrap(); + if std::fs::read(&unreadable).is_ok() { + return; + } + let garbled = dir.path().join("garbled.json"); + std::fs::write(&garbled, b"{not json").unwrap(); + let log = SessionLog::open(dir.path().to_path_buf(), true) + .await + .unwrap(); + assert!(log.get("locked").is_none()); + assert!( + !unreadable.exists(), + "an unreadable legacy file was left in place" + ); + assert!(!garbled.exists()); + assert_eq!(siblings(dir.path(), "locked.json.corrupt-").len(), 1); + assert_eq!(siblings(dir.path(), "garbled.json.corrupt-").len(), 1); + } + + fn siblings(dir: &Path, prefix: &str) -> Vec { + std::fs::read_dir(dir) + .unwrap() + .map(|entry| entry.unwrap().path()) + .filter(|path| { + path.file_name() + .and_then(|name| name.to_str()) + .is_some_and(|name| name.starts_with(prefix)) + }) + .collect() + } + + fn line_starts(bytes: &[u8]) -> Vec { + std::iter::once(0) + .chain( + bytes + .iter() + .enumerate() + .filter(|(_, byte)| **byte == b'\n') + .map(|(index, _)| index + 1), + ) + .filter(|start| *start < bytes.len()) + .collect() + } + + #[tokio::test] + async fn write_whose_fsync_failed_is_not_resurrected_on_restart() { + let dir = tempfile::tempdir().unwrap(); + let log = open(dir.path()).await; + put(&log, ClientSession::new("a", true, Some(60))) + .await + .unwrap(); + log.shared.writer.lock().await.faults.sync = true; + assert!(put(&log, ClientSession::new("a", true, Some(99))) + .await + .is_err()); + assert_eq!(log.get("a").unwrap().expiry_interval, Some(60)); + drop(log); + assert_eq!( + open(dir.path()).await.get("a").unwrap().expiry_interval, + Some(60), + "a write reported as failed came back after a restart" + ); + } + + #[tokio::test] + async fn removal_whose_fsync_failed_is_not_applied_on_restart() { + let dir = tempfile::tempdir().unwrap(); + let log = open(dir.path()).await; + put(&log, ClientSession::new("a", true, Some(60))) + .await + .unwrap(); + log.shared.writer.lock().await.faults.sync = true; + let removed = log + .apply("a", |current| (current.map(|_| SessionChange::Remove), ())) + .await; + assert!(removed.is_err()); + assert!(log.get("a").is_some()); + drop(log); + assert!( + open(dir.path()).await.get("a").is_some(), + "a removal reported as failed was applied after a restart" + ); + } + + #[tokio::test] + async fn failed_write_is_removed_from_the_log_before_it_is_reported() { + let dir = tempfile::tempdir().unwrap(); + let log = open(dir.path()).await; + put(&log, ClientSession::new("a", true, Some(60))) + .await + .unwrap(); + let durable = std::fs::read(dir.path().join(LOG_FILE)).unwrap(); + log.shared.writer.lock().await.faults.sync = true; + assert!(put(&log, ClientSession::new("b", true, Some(60))) + .await + .is_err()); + assert_eq!( + std::fs::read(dir.path().join(LOG_FILE)).unwrap(), + durable, + "the failed write is still in the log" + ); + } + + #[tokio::test] + async fn damaged_record_does_not_lose_the_records_after_it() { + let dir = tempfile::tempdir().unwrap(); + let log = open(dir.path()).await; + for i in 0..100 { + put(&log, ClientSession::new(format!("c{i}"), true, Some(60))) + .await + .unwrap(); + } + drop(log); + let path = dir.path().join(LOG_FILE); + let mut bytes = std::fs::read(&path).unwrap(); + let damaged = line_starts(&bytes)[1]; + bytes[damaged] = b'X'; + std::fs::write(&path, &bytes).unwrap(); + + let log = open(dir.path()).await; + assert_eq!( + log.client_ids().len(), + 99, + "records after a damaged one were discarded" + ); + let preserved = siblings(dir.path(), "sessions.log.corrupt-"); + assert_eq!(preserved.len(), 1, "the damaged log was not preserved"); + assert_eq!(std::fs::read(&preserved[0]).unwrap(), bytes); + drop(log); + assert_eq!(open(dir.path()).await.client_ids().len(), 99); + } + + #[tokio::test] + async fn damaged_session_field_is_not_read_as_a_removal() { + let dir = tempfile::tempdir().unwrap(); + let log = open(dir.path()).await; + put(&log, ClientSession::new("a", true, Some(60))) + .await + .unwrap(); + put(&log, ClientSession::new("a", true, Some(99))) + .await + .unwrap(); + drop(log); + let path = dir.path().join(LOG_FILE); + let mut bytes = std::fs::read(&path).unwrap(); + let key = b"\"session\""; + let at = bytes + .windows(key.len()) + .rposition(|window| window == key) + .unwrap(); + bytes[at + 3] ^= 0x01; + std::fs::write(&path, &bytes).unwrap(); + + let log = open(dir.path()).await; + assert!( + log.get("a").is_some(), + "a damaged update was replayed as a removal" + ); + } + + #[tokio::test] + async fn torn_tail_alone_is_not_treated_as_corruption() { + let dir = tempfile::tempdir().unwrap(); + let log = open(dir.path()).await; + put(&log, ClientSession::new("a", true, Some(60))) + .await + .unwrap(); + drop(log); + let mut file = std::fs::OpenOptions::new() + .append(true) + .open(dir.path().join(LOG_FILE)) + .unwrap(); + std::io::Write::write_all(&mut file, b"0123").unwrap(); + drop(file); + let log = open(dir.path()).await; + assert!(log.get("a").is_some()); + assert!(siblings(dir.path(), "sessions.log.corrupt-").is_empty()); + } + + #[tokio::test] + async fn removed_session_stays_removed_across_compaction_and_restart() { + let dir = tempfile::tempdir().unwrap(); + let log = open(dir.path()).await; + put(&log, ClientSession::new("a", true, Some(60))) + .await + .unwrap(); + put(&log, ClientSession::new("b", true, Some(60))) + .await + .unwrap(); + log.apply("a", |current| (current.map(|_| SessionChange::Remove), ())) + .await + .unwrap(); + log.shared.writer.lock().await.compact().unwrap(); + drop(log); + let reopened = open(dir.path()).await; + assert!( + reopened.get("a").is_none(), + "a removed session came back from the compacted log" + ); + assert!(reopened.get("b").is_some()); + } + + #[tokio::test] + async fn leftover_temp_files_are_removed_at_open() { + let dir = tempfile::tempdir().unwrap(); + let stale_log = dir.path().join(format!("{LOG_FILE}.tmp.1.2")); + let stale_v1 = dir.path().join("client.tmp.3.4"); + let session_named_like_temp = dir.path().join("a.tmp.3.4.json"); + std::fs::write(&stale_log, b"partial").unwrap(); + std::fs::write(&stale_v1, b"partial").unwrap(); + std::fs::write(&session_named_like_temp, b"{}").unwrap(); + let log = open(dir.path()).await; + assert!(!stale_log.exists()); + assert!(!stale_v1.exists()); + assert!( + session_named_like_temp.exists(), + "a session file whose ClientID looks like a temp name was deleted" + ); + drop(log); + } + + #[cfg(unix)] + #[tokio::test] + async fn startup_serves_the_replayed_sessions_when_compaction_fails() { + use std::os::unix::fs::PermissionsExt; + let dir = tempfile::tempdir().unwrap(); + let log = open(dir.path()).await; + put(&log, ClientSession::new("a", true, Some(60))) + .await + .unwrap(); + drop(log); + std::fs::set_permissions(dir.path(), std::fs::Permissions::from_mode(0o555)).unwrap(); + if std::fs::write(dir.path().join("probe"), b"").is_ok() { + std::fs::set_permissions(dir.path(), std::fs::Permissions::from_mode(0o755)).unwrap(); + return; + } + let reopened = SessionLog::open(dir.path().to_path_buf(), false).await; + let refused = match &reopened { + Ok(log) => Some(put(log, ClientSession::new("b", true, Some(60))).await), + Err(_) => None, + }; + std::fs::set_permissions(dir.path(), std::fs::Permissions::from_mode(0o755)).unwrap(); + let log = reopened.expect("startup must not fail when compaction fails"); + assert!(log.get("a").is_some()); + assert!( + refused.is_some_and(|result| result.is_err()), + "a write was accepted before the log was repaired" + ); + put(&log, ClientSession::new("c", true, Some(60))) + .await + .expect("the next write repairs the log"); + drop(log); + let reopened = open(dir.path()).await; + assert!(reopened.get("a").is_some()); + assert!(reopened.get("b").is_none()); + assert!(reopened.get("c").is_some()); + } +} diff --git a/crates/mqtt5/src/broker/storage/tests.rs b/crates/mqtt5/src/broker/storage/tests.rs index 80f86fc4..218bafcf 100644 --- a/crates/mqtt5/src/broker/storage/tests.rs +++ b/crates/mqtt5/src/broker/storage/tests.rs @@ -740,6 +740,99 @@ fn will_publish_delay_is_bounded_by_session_expiry() { ClientSession::new_with_will("c", true, Some(2), Some(delayed_will(Some(10)))); assert_eq!(short_session.will_publish_delay(), Some(2)); - let never_expires = ClientSession::new_with_will("c", true, None, Some(delayed_will(Some(10)))); - assert_eq!(never_expires.will_publish_delay(), Some(10)); + let v311_persistent = + ClientSession::new_with_will("c", true, None, Some(delayed_will(Some(10)))); + assert_eq!(v311_persistent.will_publish_delay(), Some(10)); +} + +fn connect_packet( + protocol_version: u8, + clean_start: bool, + session_expiry: Option, +) -> crate::packet::connect::ConnectPacket { + let mut options = mqtt5_protocol::types::ConnectOptions::new("c").with_clean_start(clean_start); + if let Some(expiry) = session_expiry { + options = options.with_session_expiry_interval(expiry); + } + let mut connect = crate::packet::connect::ConnectPacket::new(options); + connect.protocol_version = protocol_version; + connect +} + +#[test] +fn session_expiry_from_connect_treats_absent_v5_value_as_zero() { + assert_eq!( + ClientSession::expiry_from_connect(&connect_packet(5, false, None)), + Some(0) + ); + assert_eq!( + ClientSession::expiry_from_connect(&connect_packet(5, false, Some(120))), + Some(120) + ); + assert_eq!( + ClientSession::expiry_from_connect(&connect_packet(5, true, Some(0))), + Some(0) + ); +} + +#[test] +fn session_expiry_from_connect_keeps_v311_clean_session_semantics() { + assert_eq!( + ClientSession::expiry_from_connect(&connect_packet(4, true, None)), + Some(0), + "CleanSession=1 ends the session with the connection" + ); + assert_eq!( + ClientSession::expiry_from_connect(&connect_packet(4, false, None)), + None, + "CleanSession=0 keeps the session with no expiry" + ); +} + +async fn assert_update_session_is_token_guarded(backend: &impl StorageBackend) { + let mut session = ClientSession::new_with_will( + "guarded", + true, + Some(60), + Some(crate::types::WillMessage::new("will/guarded", "gone")), + ); + session.connection_token = 7; + backend.store_session(session).await.unwrap(); + + let stale = backend + .update_session("guarded", 6, |stored| stored.will_message = None) + .await + .unwrap(); + assert!( + !stale, + "a different connection token must not update the session" + ); + let stored = backend.get_session("guarded").await.unwrap().unwrap(); + assert!(stored.will_message.is_some()); + + let owned = backend + .update_session("guarded", 7, |stored| stored.will_message = None) + .await + .unwrap(); + assert!(owned); + let stored = backend.get_session("guarded").await.unwrap().unwrap(); + assert!(stored.will_message.is_none()); + + let missing = backend + .update_session("absent", 7, |stored| stored.will_message = None) + .await + .unwrap(); + assert!(!missing); +} + +#[tokio::test] +async fn memory_update_session_is_token_guarded() { + assert_update_session_is_token_guarded(&MemoryBackend::new()).await; +} + +#[tokio::test] +async fn file_update_session_is_token_guarded() { + let dir = tempfile::tempdir().unwrap(); + let backend = FileBackend::new(dir.path()).await.unwrap(); + assert_update_session_is_token_guarded(&backend).await; } diff --git a/crates/mqtt5/src/types.rs b/crates/mqtt5/src/types.rs index 3f3b66ee..2400a9cd 100644 --- a/crates/mqtt5/src/types.rs +++ b/crates/mqtt5/src/types.rs @@ -39,6 +39,10 @@ pub struct ConnectOptions { /// locally, so delivery across the restart is at-least-once: this is the deferred-ack /// crash-recovery pattern, where messages whose `AckToken` was never resolved are /// redelivered to the new process. It has no effect on `Clean Start = 1` connections. + /// + /// The broker only keeps a session to resume if the previous connection set a non-zero + /// Session Expiry Interval (`with_session_expiry_interval`): under MQTT v5 an absent + /// Session Expiry Interval means 0, so the session ends when the connection closes. pub resume_existing_session: bool, } @@ -137,6 +141,10 @@ impl ConnectOptions { self } + /// Sets Clean Start. With `false` the broker resumes the session it kept for this + /// client identifier, but under MQTT v5 it only keeps one if a connection set a non-zero + /// Session Expiry Interval (`with_session_expiry_interval`): an absent Session Expiry + /// Interval means 0, which ends the session when the network connection closes. #[must_use] pub fn with_clean_start(mut self, clean: bool) -> Self { self.protocol_options = self.protocol_options.with_clean_start(clean); @@ -159,6 +167,8 @@ impl ConnectOptions { self } + /// Sets how long, in seconds, the broker keeps the session after the network connection + /// closes. Leaving it unset means 0 under MQTT v5: the session ends at disconnect. #[must_use] pub fn with_session_expiry_interval(mut self, interval: u32) -> Self { self.protocol_options = self.protocol_options.with_session_expiry_interval(interval); diff --git a/crates/mqtt5/tests/cli_e2e.rs b/crates/mqtt5/tests/cli_e2e.rs index 3333e669..269cc7a5 100644 --- a/crates/mqtt5/tests/cli_e2e.rs +++ b/crates/mqtt5/tests/cli_e2e.rs @@ -32,7 +32,6 @@ async fn test_cli_session_persistence() { let client_id = "persist-test"; - // Test session persistence let result = verify_session_persistence(broker_url, client_id).await; match result { @@ -40,7 +39,7 @@ async fn test_cli_session_persistence() { println!("โœ… Session persistence verified - session was resumed"); } Ok(false) => { - println!("โš ๏ธ Session not resumed - broker may not support persistence"); + panic!("Session was not resumed although the first connection set a Session Expiry Interval"); } Err(e) => { println!("โŒ Session persistence test failed: {e}"); diff --git a/crates/mqtt5/tests/cli_features.rs b/crates/mqtt5/tests/cli_features.rs index 76734fb9..3f8220a4 100644 --- a/crates/mqtt5/tests/cli_features.rs +++ b/crates/mqtt5/tests/cli_features.rs @@ -98,7 +98,6 @@ async fn test_cli_clean_start() { let broker_url = broker.address(); let client_id = "test-clean-client"; - // First connection with default (clean_start=true) let pub1 = run_cli_command(&[ "pub", "--url", @@ -109,6 +108,8 @@ async fn test_cli_clean_start() { "test1", "--client-id", client_id, + "--session-expiry", + "60", "--non-interactive", ]) .await; @@ -119,7 +120,6 @@ async fn test_cli_clean_start() { "First connection should not resume session" ); - // Second connection with no-clean-start should resume session let pub2 = run_cli_command(&[ "pub", "--url", @@ -137,12 +137,10 @@ async fn test_cli_clean_start() { assert!(pub2.success, "Second publish should succeed"); - // Verify session resumption or check it didn't error - if pub2.stdout_contains("Resumed existing session") { - println!("โœ… Clean start functionality verified - session resumed"); - } else { - println!("โš ๏ธ Session resumption not confirmed in output"); - } + assert!( + pub2.stdout_contains("Resumed existing session"), + "--no-clean-start must resume the session the first connection kept with --session-expiry" + ); } /// Test session expiry interval diff --git a/crates/mqtt5/tests/common/cli_helpers.rs b/crates/mqtt5/tests/common/cli_helpers.rs index 2fbaaced..6ea10c15 100644 --- a/crates/mqtt5/tests/common/cli_helpers.rs +++ b/crates/mqtt5/tests/common/cli_helpers.rs @@ -194,7 +194,6 @@ pub async fn verify_pub_sub_delivery( } pub async fn verify_session_persistence(broker_url: &str, client_id: &str) -> Result { - // First connection with clean start let result1 = run_cli_command(&[ "pub", "--url", @@ -205,6 +204,8 @@ pub async fn verify_session_persistence(broker_url: &str, client_id: &str) -> Re "test", "--client-id", client_id, + "--session-expiry", + "60", "--non-interactive", ]) .await; @@ -213,7 +214,6 @@ pub async fn verify_session_persistence(broker_url: &str, client_id: &str) -> Re return Err(format!("First connection failed: {}", result1.stderr)); } - // Second connection without clean start - should resume session let result2 = run_cli_command(&[ "pub", "--url", @@ -233,7 +233,6 @@ pub async fn verify_session_persistence(broker_url: &str, client_id: &str) -> Re return Err(format!("Second connection failed: {}", result2.stderr)); } - // Check for session resumption message Ok(result2.stdout_contains("Resumed existing session") || result2.stdout_contains("Session present: true")) } diff --git a/crates/mqtt5/tests/enhanced_auth.rs b/crates/mqtt5/tests/enhanced_auth.rs index 1879e1fa..b8d9856c 100644 --- a/crates/mqtt5/tests/enhanced_auth.rs +++ b/crates/mqtt5/tests/enhanced_auth.rs @@ -5,9 +5,13 @@ use mqtt5::broker::{BrokerConfig, MqttBroker}; use mqtt5::client::{AuthHandler, AuthResponse}; use mqtt5::error::{MqttError, Result}; use mqtt5::packet::auth::AuthPacket; +use mqtt5::packet::connack::ConnAckPacket; use mqtt5::packet::connect::ConnectPacket; -use mqtt5::packet::MqttPacket; +use mqtt5::packet::publish::PublishPacket; +use mqtt5::packet::subscribe::SubscribePacket; +use mqtt5::packet::{MqttPacket, Packet}; use mqtt5::protocol::v5::reason_codes::ReasonCode; +use mqtt5::transport::packet_io::read_packet_from_stream; use mqtt5::types::ConnectOptions; use mqtt5::MqttClient; use std::future::Future; @@ -234,6 +238,7 @@ async fn test_auth_packet_large_data() { struct TestChallengeResponseAuthProvider { challenge: Vec, expected_response: Vec, + server_final: Option>, } impl AuthProvider for TestChallengeResponseAuthProvider { @@ -276,6 +281,7 @@ impl AuthProvider for TestChallengeResponseAuthProvider { let method = auth_method.to_string(); let challenge = self.challenge.clone(); let expected = self.expected_response.clone(); + let server_final = self.server_final.clone(); Box::pin(async move { if method != "CHALLENGE-RESPONSE" { @@ -287,7 +293,11 @@ impl AuthProvider for TestChallengeResponseAuthProvider { match auth_data { None => Ok(EnhancedAuthResult::continue_auth(method, Some(challenge))), - Some(response) if response == expected => Ok(EnhancedAuthResult::success(method)), + Some(response) if response == expected => { + let mut success = EnhancedAuthResult::success(method); + success.auth_data = server_final; + Ok(success) + } Some(_) => Ok(EnhancedAuthResult::fail(method, ReasonCode::NotAuthorized)), } }) @@ -336,6 +346,7 @@ async fn test_client_enhanced_auth_success() { let auth_provider = Arc::new(TestChallengeResponseAuthProvider { challenge: challenge.clone(), expected_response: response.clone(), + server_final: None, }); let mut broker = MqttBroker::with_config(test_broker_config()) @@ -381,6 +392,7 @@ async fn test_client_enhanced_auth_failure() { let auth_provider = Arc::new(TestChallengeResponseAuthProvider { challenge: challenge.clone(), expected_response: correct_response, + server_final: None, }); let mut broker = MqttBroker::with_config(test_broker_config()) @@ -419,6 +431,7 @@ async fn test_client_enhanced_auth_no_handler() { let auth_provider = Arc::new(TestChallengeResponseAuthProvider { challenge: b"challenge".to_vec(), expected_response: b"response".to_vec(), + server_final: None, }); let mut broker = MqttBroker::with_config(test_broker_config()) @@ -447,3 +460,259 @@ async fn test_client_enhanced_auth_no_handler() { broker_handle.abort(); } + +struct Wire { + stream: tokio::net::TcpStream, + buffer: bytes::BytesMut, +} + +impl Wire { + async fn open(addr: SocketAddr) -> Self { + Self { + stream: tokio::net::TcpStream::connect(addr).await.unwrap(), + buffer: bytes::BytesMut::new(), + } + } + + async fn send(&mut self, packet: &impl MqttPacket) { + use tokio::io::AsyncWriteExt; + let mut bytes = Vec::new(); + packet.encode(&mut bytes).unwrap(); + self.stream.write_all(&bytes).await.unwrap(); + } + + async fn next(&mut self, millis: u64) -> Option { + tokio::time::timeout( + Duration::from_millis(millis), + read_packet_from_stream(&mut self.stream, 5, &mut self.buffer, 1 << 20), + ) + .await + .ok()? + .ok() + } +} + +const METHOD: &str = "CHALLENGE-RESPONSE"; +const CHALLENGE: &[u8] = b"server-challenge-xyz"; +const RESPONSE: &[u8] = b"client-response-abc"; +const SERVER_FINAL: &[u8] = b"server-final-signature"; + +async fn start_challenge_broker(config: BrokerConfig) -> (SocketAddr, tokio::task::JoinHandle<()>) { + let auth_provider = Arc::new(TestChallengeResponseAuthProvider { + challenge: CHALLENGE.to_vec(), + expected_response: RESPONSE.to_vec(), + server_final: Some(SERVER_FINAL.to_vec()), + }); + let mut broker = MqttBroker::with_config(config) + .await + .unwrap() + .with_auth_provider(auth_provider); + let addr = broker.local_addr().unwrap(); + let handle = tokio::spawn(async move { + if let Err(e) = broker.run().await { + tracing::debug!("broker stopped: {e}"); + } + }); + tokio::time::sleep(Duration::from_millis(100)).await; + (addr, handle) +} + +async fn two_step_exchange( + addr: SocketAddr, + options: ConnectOptions, + response: &[u8], +) -> (Wire, Option) { + let mut wire = Wire::open(addr).await; + wire.send(&ConnectPacket::new( + options.with_authentication_method(METHOD).protocol_options, + )) + .await; + match wire.next(5000).await { + Some(Packet::Auth(auth)) => { + assert_eq!(auth.reason_code, ReasonCode::ContinueAuthentication); + assert_eq!(auth.authentication_data(), Some(CHALLENGE)); + } + other => panic!("expected AUTH continue, got {other:?}"), + } + wire.send( + &AuthPacket::continue_authentication(METHOD.to_string(), Some(response.to_vec())).unwrap(), + ) + .await; + let reply = wire.next(5000).await; + (wire, reply) +} + +async fn two_step_connect(addr: SocketAddr, options: ConnectOptions) -> (Wire, ConnAckPacket) { + match two_step_exchange(addr, options, RESPONSE).await { + (wire, Some(Packet::ConnAck(connack))) => (wire, connack), + (_, other) => panic!("expected CONNACK, got {other:?}"), + } +} + +#[tokio::test] +async fn enhanced_auth_connack_advertises_capped_session_expiry() { + let (addr, handle) = + start_challenge_broker(test_broker_config().with_session_expiry(Duration::from_secs(10))) + .await; + let (_wire, connack) = two_step_connect( + addr, + ConnectOptions::new("capped-enhanced").with_session_expiry_interval(3600), + ) + .await; + handle.abort(); + assert_eq!(connack.reason_code, ReasonCode::Success); + assert_eq!( + connack.properties.get_session_expiry_interval(), + Some(10), + "the broker granted its 10s maximum instead of the requested 3600s, so CONNACK must say so" + ); +} + +#[tokio::test] +async fn two_step_enhanced_auth_rejects_an_unsupported_will_qos() { + let (addr, handle) = start_challenge_broker(test_broker_config().with_maximum_qos(1)).await; + let will = mqtt5::types::WillMessage::new("will/t", b"gone".to_vec()) + .with_qos(mqtt5::QoS::ExactlyOnce); + let (_wire, connack) = + two_step_connect(addr, ConnectOptions::new("will-qos").with_will(will)).await; + handle.abort(); + assert_eq!( + connack.reason_code, + ReasonCode::QoSNotSupported, + "the Will QoS check must apply after multi-step authentication too" + ); +} + +#[tokio::test] +async fn two_step_enhanced_auth_honours_the_client_receive_maximum() { + let (addr, handle) = start_challenge_broker(test_broker_config()).await; + let (mut subscriber, connack) = + two_step_connect(addr, ConnectOptions::new("rm-one").with_receive_maximum(1)).await; + assert_eq!(connack.reason_code, ReasonCode::Success); + subscriber + .send(&SubscribePacket::new(1).add_filter("rm/t", mqtt5::QoS::AtLeastOnce)) + .await; + assert!(matches!( + subscriber.next(5000).await, + Some(Packet::SubAck(_)) + )); + + let mut publisher = Wire::open(addr).await; + publisher + .send(&ConnectPacket::new( + ConnectOptions::new("rm-pub").protocol_options, + )) + .await; + assert!(matches!( + publisher.next(5000).await, + Some(Packet::ConnAck(_)) + )); + for packet_id in 1..=3u16 { + let mut publish = + PublishPacket::new("rm/t".to_string(), b"hi".to_vec(), mqtt5::QoS::AtLeastOnce); + publish.packet_id = Some(packet_id); + publisher.send(&publish).await; + } + let mut delivered = 0; + while let Some(packet) = subscriber.next(1000).await { + if matches!(packet, Packet::Publish(_)) { + delivered += 1; + } + } + handle.abort(); + assert_eq!( + delivered, 1, + "Receive Maximum 1 allows one unacknowledged QoS 1 PUBLISH, yet {delivered} were sent" + ); +} + +#[tokio::test] +async fn two_step_enhanced_auth_success_returns_method_and_server_data_in_connack() { + let (addr, handle) = start_challenge_broker(test_broker_config()).await; + let (_wire, connack) = two_step_connect(addr, ConnectOptions::new("server-final")).await; + handle.abort(); + assert_eq!(connack.reason_code, ReasonCode::Success); + assert_eq!( + connack + .properties + .get_authentication_method() + .map(String::as_str), + Some(METHOD), + "CONNACK after enhanced authentication must carry the Authentication Method" + ); + assert_eq!( + connack.properties.get_authentication_data(), + Some(SERVER_FINAL), + "the server's final authentication data never reached the client" + ); +} + +#[tokio::test] +async fn failed_two_step_enhanced_auth_is_refused_by_connack() { + let (addr, handle) = start_challenge_broker(test_broker_config()).await; + let (mut wire, reply) = + two_step_exchange(addr, ConnectOptions::new("wrong-proof"), b"wrong").await; + let closed = wire.next(2000).await.is_none(); + handle.abort(); + match reply { + Some(Packet::ConnAck(connack)) => { + assert_eq!(connack.reason_code, ReasonCode::NotAuthorized); + } + other => { + panic!("a failed authentication must be answered with CONNACK 0x87, got {other:?}") + } + } + assert!(closed); +} + +async fn reauthenticate(wire: &mut Wire, response: &[u8]) -> Option { + wire.send(&AuthPacket::re_authenticate(METHOD.to_string(), None).unwrap()) + .await; + match wire.next(5000).await { + Some(Packet::Auth(auth)) => { + assert_eq!(auth.reason_code, ReasonCode::ContinueAuthentication); + } + other => panic!("expected AUTH continue, got {other:?}"), + } + wire.send( + &AuthPacket::continue_authentication(METHOD.to_string(), Some(response.to_vec())).unwrap(), + ) + .await; + wire.next(5000).await +} + +#[tokio::test] +async fn reauthentication_success_returns_the_server_data() { + let (addr, handle) = start_challenge_broker(test_broker_config()).await; + let (mut wire, connack) = two_step_connect(addr, ConnectOptions::new("reauth-ok")).await; + assert_eq!(connack.reason_code, ReasonCode::Success); + let reply = reauthenticate(&mut wire, RESPONSE).await; + handle.abort(); + match reply { + Some(Packet::Auth(auth)) => { + assert_eq!(auth.reason_code, ReasonCode::Success); + assert_eq!(auth.authentication_method(), Some(METHOD)); + assert_eq!(auth.authentication_data(), Some(SERVER_FINAL)); + } + other => panic!("expected AUTH success, got {other:?}"), + } +} + +#[tokio::test] +async fn failed_reauthentication_is_refused_by_disconnect() { + let (addr, handle) = start_challenge_broker(test_broker_config()).await; + let (mut wire, connack) = two_step_connect(addr, ConnectOptions::new("reauth-bad")).await; + assert_eq!(connack.reason_code, ReasonCode::Success); + let reply = reauthenticate(&mut wire, b"wrong").await; + handle.abort(); + match reply { + Some(Packet::Disconnect(disconnect)) => { + assert_eq!(disconnect.reason_code, ReasonCode::NotAuthorized); + } + other => { + panic!( + "a failed re-authentication must be answered with DISCONNECT 0x87, got {other:?}" + ) + } + } +} diff --git a/crates/mqtt5/tests/integration_mqtt5_features.rs b/crates/mqtt5/tests/integration_mqtt5_features.rs index b821daec..f78b14f1 100644 --- a/crates/mqtt5/tests/integration_mqtt5_features.rs +++ b/crates/mqtt5/tests/integration_mqtt5_features.rs @@ -178,6 +178,7 @@ async fn test_will_message_with_delay() { let connect_opts = ConnectOptions::new(will_client_id) .with_clean_start(true) + .with_session_expiry_interval(60) .with_will(will); let disconnect_time = Arc::new(Mutex::new(None::)); diff --git a/crates/mqtt5/tests/persistence.rs b/crates/mqtt5/tests/persistence.rs index 8fd1adb2..7b10a499 100644 --- a/crates/mqtt5/tests/persistence.rs +++ b/crates/mqtt5/tests/persistence.rs @@ -9,6 +9,8 @@ use std::sync::atomic::{AtomicBool, AtomicU32, Ordering}; use std::sync::Arc; use tokio::time::sleep; +const SESSION_EXPIRY: u32 = 300; + #[tokio::test] async fn test_clean_start_true() { let broker = TestBroker::start().await; @@ -54,7 +56,11 @@ async fn test_clean_start_false() { let client_id = "persist-test-1"; - let client1 = MqttClient::with_options(ConnectOptions::new(client_id).with_clean_start(true)); + let client1 = MqttClient::with_options( + ConnectOptions::new(client_id) + .with_clean_start(true) + .with_session_expiry_interval(SESSION_EXPIRY), + ); client1.connect(broker.address()).await.unwrap(); client1.subscribe("test/persist/1", |_| {}).await.unwrap(); @@ -65,6 +71,7 @@ async fn test_clean_start_false() { let client2 = MqttClient::with_options( ConnectOptions::new(client_id) .with_clean_start(false) + .with_session_expiry_interval(SESSION_EXPIRY) .with_resume_existing_session(true), ); @@ -73,6 +80,7 @@ async fn test_clean_start_false() { broker.address(), ConnectOptions::new(client_id) .with_clean_start(false) + .with_session_expiry_interval(SESSION_EXPIRY) .with_resume_existing_session(true), ), ) @@ -81,9 +89,10 @@ async fn test_clean_start_false() { let session_present_flag = session_present.session_present; println!("Session present: {session_present_flag}"); - if !session_present.session_present { - println!("Warning: Broker did not preserve session. This is broker-dependent behavior."); - } + assert!( + session_present.session_present, + "a session with a Session Expiry Interval must be resumed" + ); let received = Arc::new(AtomicU32::new(0)); let received_clone = received.clone(); @@ -170,6 +179,7 @@ async fn test_qos1_message_persistence() { let sub_options = ConnectOptions::new(sub_client_id) .with_clean_start(false) + .with_session_expiry_interval(SESSION_EXPIRY) .with_resume_existing_session(true); let sub_client = MqttClient::with_options(sub_options); @@ -205,6 +215,7 @@ async fn test_qos1_message_persistence() { let sub_client2 = MqttClient::with_options( ConnectOptions::new(sub_client_id) .with_clean_start(false) + .with_session_expiry_interval(SESSION_EXPIRY) .with_resume_existing_session(true), ); @@ -213,6 +224,7 @@ async fn test_qos1_message_persistence() { broker.address(), ConnectOptions::new(sub_client_id) .with_clean_start(false) + .with_session_expiry_interval(SESSION_EXPIRY) .with_resume_existing_session(true), ), ) @@ -223,9 +235,10 @@ async fn test_qos1_message_persistence() { "Session present after reconnect: {}", session_present.session_present ); - if !session_present.session_present { - println!("Warning: Broker did not restore session for QoS persistence test"); - } + assert!( + session_present.session_present, + "a session with a Session Expiry Interval must be resumed" + ); sub_client2 .subscribe_with_options( @@ -267,6 +280,7 @@ async fn test_qos2_message_persistence() { let sub_options = ConnectOptions::new(sub_client_id) .with_clean_start(false) + .with_session_expiry_interval(SESSION_EXPIRY) .with_resume_existing_session(true); let sub_client = MqttClient::with_options(sub_options); @@ -302,6 +316,7 @@ async fn test_qos2_message_persistence() { let sub_client2 = MqttClient::with_options( ConnectOptions::new(sub_client_id) .with_clean_start(false) + .with_session_expiry_interval(SESSION_EXPIRY) .with_resume_existing_session(true), ); @@ -349,7 +364,11 @@ async fn test_subscription_persistence() { let client_id = "sub-persist-test"; - let client1 = MqttClient::with_options(ConnectOptions::new(client_id).with_clean_start(true)); + let client1 = MqttClient::with_options( + ConnectOptions::new(client_id) + .with_clean_start(true) + .with_session_expiry_interval(SESSION_EXPIRY), + ); client1.connect(broker.address()).await.unwrap(); client1.subscribe("test/sub/1", |_| {}).await.unwrap(); @@ -364,6 +383,7 @@ async fn test_subscription_persistence() { let client2 = MqttClient::with_options( ConnectOptions::new(client_id) .with_clean_start(false) + .with_session_expiry_interval(SESSION_EXPIRY) .with_resume_existing_session(true), ); @@ -372,6 +392,7 @@ async fn test_subscription_persistence() { broker.address(), ConnectOptions::new(client_id) .with_clean_start(false) + .with_session_expiry_interval(SESSION_EXPIRY) .with_resume_existing_session(true), ), ) @@ -382,11 +403,10 @@ async fn test_subscription_persistence() { "Session present for subscription persistence: {}", session_present.session_present ); - if !session_present.session_present { - println!("Warning: Broker did not preserve session for subscription test"); - client2.disconnect().await.unwrap(); - return; - } + assert!( + session_present.session_present, + "a session with a Session Expiry Interval must be resumed" + ); client2 .subscribe("test/sub/+", move |msg| { @@ -444,6 +464,7 @@ async fn test_will_message_persistence() { let will_options = ConnectOptions::new(will_client_id) .with_clean_start(false) + .with_session_expiry_interval(SESSION_EXPIRY) .with_resume_existing_session(true) .with_will(will_msg); @@ -471,6 +492,7 @@ async fn test_packet_id_persistence() { let options = ConnectOptions::new(client_id) .with_clean_start(false) + .with_session_expiry_interval(SESSION_EXPIRY) .with_resume_existing_session(true); let client1 = MqttClient::with_options(options.clone()); @@ -515,6 +537,7 @@ async fn test_inflight_message_persistence() { let sub_client = MqttClient::with_options( ConnectOptions::new(sub_client_id) .with_clean_start(false) + .with_session_expiry_interval(SESSION_EXPIRY) .with_resume_existing_session(true), ); sub_client.connect(broker.address()).await.unwrap(); @@ -539,6 +562,7 @@ async fn test_inflight_message_persistence() { let pub_client = MqttClient::with_options( ConnectOptions::new(pub_client_id) .with_clean_start(false) + .with_session_expiry_interval(SESSION_EXPIRY) .with_resume_existing_session(true), ); pub_client.connect(broker.address()).await.unwrap(); @@ -559,6 +583,7 @@ async fn test_inflight_message_persistence() { let pub_client2 = MqttClient::with_options( ConnectOptions::new(pub_client_id) .with_clean_start(false) + .with_session_expiry_interval(SESSION_EXPIRY) .with_resume_existing_session(true), ); pub_client2.connect(broker.address()).await.unwrap(); @@ -587,6 +612,7 @@ async fn test_qos2_outbound_inflight_resend_on_reconnect() { let sub_client = MqttClient::with_options( ConnectOptions::new(sub_client_id) .with_clean_start(false) + .with_session_expiry_interval(SESSION_EXPIRY) .with_resume_existing_session(true), ); sub_client.connect(broker.address()).await.unwrap(); @@ -620,6 +646,7 @@ async fn test_qos2_outbound_inflight_resend_on_reconnect() { let sub_client2 = MqttClient::with_options( ConnectOptions::new(sub_client_id) .with_clean_start(false) + .with_session_expiry_interval(SESSION_EXPIRY) .with_resume_existing_session(true), ); @@ -628,6 +655,7 @@ async fn test_qos2_outbound_inflight_resend_on_reconnect() { broker.address(), ConnectOptions::new(sub_client_id) .with_clean_start(false) + .with_session_expiry_interval(SESSION_EXPIRY) .with_resume_existing_session(true), ), ) @@ -687,6 +715,7 @@ async fn test_clean_start_clears_inflight_state() { let sub_client = MqttClient::with_options( ConnectOptions::new(sub_client_id) .with_clean_start(false) + .with_session_expiry_interval(SESSION_EXPIRY) .with_resume_existing_session(true), ); sub_client.connect(broker.address()).await.unwrap(); diff --git a/crates/mqtt5/tests/session_crash.rs b/crates/mqtt5/tests/session_crash.rs new file mode 100644 index 00000000..9cb0b17c --- /dev/null +++ b/crates/mqtt5/tests/session_crash.rs @@ -0,0 +1,386 @@ +#![cfg(feature = "broker")] + +use bytes::BytesMut; +use mqtt5::broker::config::{BrokerConfig, StorageBackend as BackendKind, StorageConfig}; +use mqtt5::broker::server::MqttBroker; +use mqtt5::broker::storage::{FileBackend, StorageBackend}; +use mqtt5::time::Duration; +use mqtt5::transport::packet_io::read_packet_from_stream; +use mqtt5_protocol::packet::connack::ConnAckPacket; +use mqtt5_protocol::packet::connect::ConnectPacket; +use mqtt5_protocol::packet::{MqttPacket, Packet}; +use mqtt5_protocol::types::ConnectOptions; +use std::io::Write; +use std::net::SocketAddr; +use std::path::Path; +use std::process::Stdio; +use tokio::io::{AsyncBufReadExt, AsyncWriteExt, BufReader}; +use tokio::net::TcpStream; +use tokio::process::{Child, Command}; +use tokio::time::sleep; + +const HARNESS_DIR: &str = "MQTT5_CRASH_HARNESS_DIR"; +const HARNESS_CLEANUP_MS: &str = "MQTT5_CRASH_HARNESS_CLEANUP_MS"; +const LISTENING: &str = "CRASH-HARNESS-LISTENING "; + +#[tokio::test(flavor = "multi_thread", worker_threads = 4)] +async fn crash_harness_broker() { + let Ok(dir) = std::env::var(HARNESS_DIR) else { + return; + }; + let cleanup_ms = std::env::var(HARNESS_CLEANUP_MS) + .ok() + .and_then(|ms| ms.parse().ok()) + .unwrap_or(3_600_000); + let config = BrokerConfig::default() + .with_bind_address("127.0.0.1:0".parse::().expect("bind address")) + .with_storage(StorageConfig { + backend: BackendKind::File, + base_dir: dir.into(), + enable_persistence: true, + cleanup_interval: Duration::from_millis(cleanup_ms), + ..Default::default() + }); + let mut broker = MqttBroker::with_config(config).await.expect("start broker"); + let addr = broker.local_addr().expect("broker address"); + let mut stdout = std::io::stdout(); + writeln!(stdout, "{LISTENING}{addr}").expect("announce address"); + stdout.flush().expect("flush address"); + broker.run().await.expect("broker run"); +} + +struct CrashBroker { + addr: String, + child: Child, +} + +impl CrashBroker { + async fn start(dir: &Path, cleanup: Duration) -> Self { + let mut child = Command::new(std::env::current_exe().expect("test binary")) + .args([ + "crash_harness_broker", + "--exact", + "--nocapture", + "--test-threads", + "1", + ]) + .env(HARNESS_DIR, dir) + .env(HARNESS_CLEANUP_MS, cleanup.as_millis().to_string()) + .stdout(Stdio::piped()) + .stderr(Stdio::null()) + .kill_on_drop(true) + .spawn() + .expect("spawn broker process"); + let stdout = child.stdout.take().expect("broker stdout"); + let mut lines = BufReader::new(stdout).lines(); + let addr = tokio::time::timeout(Duration::from_secs(30), async { + while let Some(line) = lines.next_line().await.expect("read broker stdout") { + if let Some((_, addr)) = line.split_once(LISTENING) { + return addr.to_string(); + } + } + panic!("broker process exited before listening"); + }) + .await + .expect("broker process did not start"); + tokio::spawn(async move { while let Ok(Some(_)) = lines.next_line().await {} }); + Self { addr, child } + } + + async fn crash(mut self) { + self.child.start_kill().expect("kill broker process"); + self.child.wait().await.expect("reap broker process"); + } +} + +fn options(client_id: &str, clean_start: bool, session_expiry: Option) -> ConnectOptions { + let options = ConnectOptions::new(client_id).with_clean_start(clean_start); + match session_expiry { + Some(expiry) => options.with_session_expiry_interval(expiry), + None => options, + } +} + +struct Wire { + stream: TcpStream, + buffer: BytesMut, +} + +impl Wire { + async fn send(&mut self, bytes: &[u8]) { + self.stream.write_all(bytes).await.expect("write packet"); + } + + async fn next(&mut self, wait: Duration) -> Option { + tokio::time::timeout( + wait, + read_packet_from_stream(&mut self.stream, 5, &mut self.buffer, 1 << 20), + ) + .await + .ok()? + .ok() + } + + async fn closed(&mut self, wait: Duration) -> bool { + tokio::time::timeout( + wait, + read_packet_from_stream(&mut self.stream, 5, &mut self.buffer, 1 << 20), + ) + .await + .is_ok_and(|read| read.is_err()) + } +} + +async fn connect(addr: &str, options: ConnectOptions) -> (Wire, ConnAckPacket) { + let mut wire = Wire { + stream: TcpStream::connect(addr).await.expect("connect tcp"), + buffer: BytesMut::new(), + }; + let mut bytes = Vec::new(); + ConnectPacket::new(options) + .encode(&mut bytes) + .expect("encode CONNECT"); + wire.send(&bytes).await; + match wire.next(Duration::from_secs(10)).await { + Some(Packet::ConnAck(connack)) => (wire, connack), + other => panic!("expected CONNACK, got {other:?}"), + } +} + +async fn subscribe(wire: &mut Wire, topic: &str) { + let topic_len = u8::try_from(topic.len()).expect("short topic"); + let mut bytes = vec![0x82, 6 + topic_len, 0x00, 1, 0x00, 0x00, topic_len]; + bytes.extend_from_slice(topic.as_bytes()); + bytes.push(1); + wire.send(&bytes).await; + assert!( + matches!( + wire.next(Duration::from_secs(5)).await, + Some(Packet::SubAck(_)) + ), + "expected SUBACK" + ); +} + +async fn publish(wire: &mut Wire, topic: &str) { + let topic_len = u8::try_from(topic.len()).expect("short topic"); + let mut bytes = vec![0x32, 7 + topic_len, 0x00, topic_len]; + bytes.extend_from_slice(topic.as_bytes()); + bytes.extend_from_slice(&[0x00, 0x01, 0x00, b'h', b'i']); + wire.send(&bytes).await; + assert!( + matches!( + wire.next(Duration::from_secs(5)).await, + Some(Packet::PubAck(_)) + ), + "expected PUBACK" + ); +} + +async fn received_publish(wire: &mut Wire) -> bool { + matches!( + wire.next(Duration::from_millis(500)).await, + Some(Packet::Publish(_)) + ) +} + +async fn end(mut wire: Wire, disconnect: &[u8]) { + wire.send(disconnect).await; + assert!( + wire.closed(Duration::from_secs(5)).await, + "broker must close after DISCONNECT" + ); +} + +const DISCONNECT: [u8; 2] = [0xE0, 0x00]; +const DISCONNECT_EXPIRY_ZERO: [u8; 9] = [0xE0, 0x07, 0x00, 0x05, 0x11, 0x00, 0x00, 0x00, 0x00]; +const NEVER: Duration = Duration::from_secs(3600); + +#[tokio::test(flavor = "multi_thread", worker_threads = 4)] +async fn crashed_broker_serves_no_connection() { + let dir = tempfile::tempdir().expect("tempdir"); + let broker = CrashBroker::start(dir.path(), NEVER).await; + let (mut wire, _) = connect(&broker.addr, options("survivor", true, Some(60))).await; + broker.crash().await; + let pinged = wire.stream.write_all(&[0xC0, 0x00]).await; + assert!( + pinged.is_err() || wire.closed(Duration::from_secs(1)).await, + "a connection outlived the crash" + ); +} + +#[tokio::test(flavor = "multi_thread", worker_threads = 4)] +async fn subacked_subscription_survives_a_crash() { + let dir = tempfile::tempdir().expect("tempdir"); + let broker = CrashBroker::start(dir.path(), NEVER).await; + let (mut stream, _) = connect(&broker.addr, options("durable", true, Some(60))).await; + subscribe(&mut stream, "durable/t").await; + broker.crash().await; + drop(stream); + + let broker = CrashBroker::start(dir.path(), NEVER).await; + let (mut publisher, _) = connect(&broker.addr, options("durable-pub", true, None)).await; + let (mut stream, connack) = connect(&broker.addr, options("durable", false, Some(60))).await; + assert!( + connack.session_present, + "the session was claimed before CONNACK and must survive a crash" + ); + publish(&mut publisher, "durable/t").await; + assert!( + received_publish(&mut stream).await, + "a SUBACKed subscription must survive a crash" + ); + broker.crash().await; +} + +#[tokio::test(flavor = "multi_thread", worker_threads = 4)] +async fn clean_start_discard_is_not_undone_by_a_crash() { + let dir = tempfile::tempdir().expect("tempdir"); + let broker = CrashBroker::start(dir.path(), NEVER).await; + let (mut stream, _) = connect(&broker.addr, options("discard", true, Some(60))).await; + subscribe(&mut stream, "discard/old").await; + end(stream, &DISCONNECT).await; + let (_stream, connack) = connect(&broker.addr, options("discard", true, Some(60))).await; + assert!(!connack.session_present); + broker.crash().await; + + let broker = CrashBroker::start(dir.path(), NEVER).await; + let (mut publisher, _) = connect(&broker.addr, options("discard-pub", true, None)).await; + let (mut stream, connack) = connect(&broker.addr, options("discard", false, Some(60))).await; + assert!( + connack.session_present, + "the clean-start session was acknowledged and must survive" + ); + publish(&mut publisher, "discard/old").await; + assert!( + !received_publish(&mut stream).await, + "the discarded session's subscription came back after a crash" + ); + broker.crash().await; +} + +#[tokio::test(flavor = "multi_thread", worker_threads = 4)] +async fn ended_session_expires_at_its_disconnect_time_after_a_crash() { + let dir = tempfile::tempdir().expect("tempdir"); + let broker = CrashBroker::start(dir.path(), NEVER).await; + let (stream, _) = connect(&broker.addr, options("ended", true, Some(1))).await; + end(stream, &DISCONNECT).await; + sleep(Duration::from_millis(2500)).await; + broker.crash().await; + + let broker = CrashBroker::start(dir.path(), NEVER).await; + let (stream, connack) = connect(&broker.addr, options("ended", false, Some(60))).await; + end(stream, &DISCONNECT).await; + broker.crash().await; + assert!( + !connack.session_present, + "the session ended 2.5s before the crash with a 1s expiry, so it must not be resumed" + ); +} + +#[tokio::test(flavor = "multi_thread", worker_threads = 4)] +async fn offline_session_queues_messages_published_after_a_crash() { + let dir = tempfile::tempdir().expect("tempdir"); + let broker = CrashBroker::start(dir.path(), NEVER).await; + let (mut stream, _) = connect(&broker.addr, options("offline", true, Some(60))).await; + subscribe(&mut stream, "offline/t").await; + end(stream, &DISCONNECT).await; + broker.crash().await; + + let broker = CrashBroker::start(dir.path(), NEVER).await; + let (mut publisher, _) = connect(&broker.addr, options("offline-pub", true, None)).await; + publish(&mut publisher, "offline/t").await; + let (mut stream, connack) = connect(&broker.addr, options("offline", false, Some(60))).await; + assert!(connack.session_present); + assert!( + received_publish(&mut stream).await, + "a message published after the restart, before the client came back, must be queued for it" + ); + broker.crash().await; +} + +#[tokio::test(flavor = "multi_thread", worker_threads = 4)] +async fn session_connected_at_a_crash_expires_from_the_restart() { + let dir = tempfile::tempdir().expect("tempdir"); + let broker = CrashBroker::start(dir.path(), NEVER).await; + let (stream, _) = connect(&broker.addr, options("crashed", true, Some(1))).await; + broker.crash().await; + drop(stream); + + let broker = CrashBroker::start(dir.path(), NEVER).await; + sleep(Duration::from_millis(2500)).await; + let (_stream, connack) = connect(&broker.addr, options("crashed", false, Some(60))).await; + broker.crash().await; + assert!( + !connack.session_present, + "a session connected at the crash with a 1s expiry must be gone 2.5s after the restart" + ); +} + +#[tokio::test(flavor = "multi_thread", worker_threads = 4)] +async fn expiry_of_a_session_connected_at_a_crash_survives_a_second_crash() { + let dir = tempfile::tempdir().expect("tempdir"); + let broker = CrashBroker::start(dir.path(), NEVER).await; + let (stream, _) = connect(&broker.addr, options("twice", true, Some(2))).await; + broker.crash().await; + drop(stream); + + let broker = CrashBroker::start(dir.path(), NEVER).await; + sleep(Duration::from_millis(2500)).await; + broker.crash().await; + + let broker = CrashBroker::start(dir.path(), NEVER).await; + let (_stream, connack) = connect(&broker.addr, options("twice", false, Some(60))).await; + broker.crash().await; + assert!( + !connack.session_present, + "the session ended at the first restart; a second crash must not restart its expiry" + ); +} + +#[tokio::test(flavor = "multi_thread", worker_threads = 4)] +async fn expiry_zero_session_is_not_rebuilt_after_a_crash() { + let dir = tempfile::tempdir().expect("tempdir"); + let broker = CrashBroker::start(dir.path(), NEVER).await; + let (mut stream, _) = connect(&broker.addr, options("zero-crash", true, Some(0))).await; + subscribe(&mut stream, "zc/t").await; + broker.crash().await; + drop(stream); + + let backend = FileBackend::new(dir.path()).await.expect("file backend"); + let stored_at_crash = backend.session_client_ids().await.expect("list sessions"); + drop(backend); + assert_eq!( + stored_at_crash, + ["zero-crash"], + "the claim is durable before CONNACK" + ); + + let broker = CrashBroker::start(dir.path(), NEVER).await; + let (mut publisher, _) = connect(&broker.addr, options("zc-pub", true, None)).await; + let (mut stream, connack) = connect(&broker.addr, options("zero-crash", false, Some(60))).await; + assert!(!connack.session_present); + publish(&mut publisher, "zc/t").await; + assert!(!received_publish(&mut stream).await); + broker.crash().await; +} + +#[tokio::test(flavor = "multi_thread", worker_threads = 4)] +async fn ended_expiry_zero_session_is_removed_before_the_connection_closes() { + let dir = tempfile::tempdir().expect("tempdir"); + let broker = CrashBroker::start(dir.path(), NEVER).await; + let (mut stream, _) = connect(&broker.addr, options("gone", true, Some(60))).await; + subscribe(&mut stream, "gone/t").await; + end(stream, &DISCONNECT_EXPIRY_ZERO).await; + broker.crash().await; + + let backend = FileBackend::new(dir.path()).await.expect("file backend"); + assert!( + backend + .session_client_ids() + .await + .expect("list sessions") + .is_empty(), + "the session ended by DISCONNECT with expiry 0 is still stored after a crash" + ); +} diff --git a/crates/mqtt5/tests/session_lifecycle.rs b/crates/mqtt5/tests/session_lifecycle.rs new file mode 100644 index 00000000..e6219c88 --- /dev/null +++ b/crates/mqtt5/tests/session_lifecycle.rs @@ -0,0 +1,557 @@ +#![cfg(feature = "broker")] +mod common; + +use bytes::BytesMut; +use common::MessageCollector; +use mqtt5::broker::config::{BrokerConfig, StorageBackend as BackendKind, StorageConfig}; +use mqtt5::broker::server::MqttBroker; +use mqtt5::broker::storage::{ClientSession, DynamicStorage, FileBackend, StorageBackend}; +use mqtt5::time::Duration; +use mqtt5::transport::packet_io::read_packet_from_stream; +use mqtt5::MqttClient; +use mqtt5_protocol::packet::connack::ConnAckPacket; +use mqtt5_protocol::packet::connect::ConnectPacket; +use mqtt5_protocol::packet::disconnect::DisconnectPacket; +use mqtt5_protocol::packet::{MqttPacket, Packet}; +use mqtt5_protocol::protocol::v5::reason_codes::ReasonCode; +use mqtt5_protocol::types::{ConnectOptions, WillMessage}; +use std::net::SocketAddr; +use std::sync::Arc; +use tokio::io::AsyncWriteExt; +use tokio::net::TcpStream; +use tokio::time::sleep; + +struct Broker { + addr: String, + storage: Option>, + handle: tokio::task::JoinHandle<()>, +} + +impl Broker { + async fn start(config: BrokerConfig) -> Self { + let config = + config.with_bind_address("127.0.0.1:0".parse::().expect("bind address")); + let mut broker = MqttBroker::with_config(config).await.expect("start broker"); + let addr = broker.local_addr().expect("broker address").to_string(); + let storage = broker.storage(); + let handle = tokio::spawn(async move { + if let Err(e) = broker.run().await { + tracing::debug!("broker stopped: {e}"); + } + }); + sleep(Duration::from_millis(100)).await; + Self { + addr, + storage, + handle, + } + } + + async fn persistent() -> Self { + Self::start(BrokerConfig::default().with_storage(memory_storage())).await + } + + fn url(&self) -> String { + format!("mqtt://{}", self.addr) + } + + fn storage(&self) -> &DynamicStorage { + self.storage.as_deref().expect("persistence is enabled") + } +} + +impl Drop for Broker { + fn drop(&mut self) { + self.handle.abort(); + } +} + +fn memory_storage() -> StorageConfig { + StorageConfig { + backend: BackendKind::Memory, + enable_persistence: true, + ..Default::default() + } +} + +fn options(client_id: &str, clean_start: bool, session_expiry: Option) -> ConnectOptions { + let options = ConnectOptions::new(client_id).with_clean_start(clean_start); + match session_expiry { + Some(expiry) => options.with_session_expiry_interval(expiry), + None => options, + } +} + +struct Wire { + stream: TcpStream, + buffer: BytesMut, +} + +impl Wire { + fn new(stream: TcpStream) -> Self { + Self { + stream, + buffer: BytesMut::new(), + } + } + + async fn write_all(&mut self, bytes: &[u8]) -> std::io::Result<()> { + self.stream.write_all(bytes).await + } + + async fn next(&mut self, wait: Duration) -> Option { + tokio::time::timeout( + wait, + read_packet_from_stream(&mut self.stream, 5, &mut self.buffer, 1 << 20), + ) + .await + .ok()? + .ok() + } + + async fn closed(&mut self, wait: Duration) -> bool { + tokio::time::timeout( + wait, + read_packet_from_stream(&mut self.stream, 5, &mut self.buffer, 1 << 20), + ) + .await + .is_ok_and(|read| read.is_err()) + } +} + +async fn read_connack(wire: &mut Wire) -> ConnAckPacket { + match wire.next(Duration::from_secs(5)).await { + Some(Packet::ConnAck(connack)) => { + assert_eq!(connack.reason_code, ReasonCode::Success); + connack + } + other => panic!("expected CONNACK, got {other:?}"), + } +} + +async fn connect(addr: &str, options: ConnectOptions) -> (Wire, ConnAckPacket) { + connect_packet(addr, ConnectPacket::new(options)).await +} + +async fn connect_packet(addr: &str, packet: ConnectPacket) -> (Wire, ConnAckPacket) { + let mut wire = Wire::new(TcpStream::connect(addr).await.expect("connect tcp")); + let mut buf = Vec::new(); + packet.encode(&mut buf).expect("encode CONNECT"); + wire.write_all(&buf).await.expect("write CONNECT"); + let connack = read_connack(&mut wire).await; + (wire, connack) +} + +async fn disconnect(mut wire: Wire, disconnect: DisconnectPacket) { + let mut buf = Vec::new(); + disconnect.encode(&mut buf).expect("encode DISCONNECT"); + wire.write_all(&buf).await.expect("write DISCONNECT"); + assert!( + wire.closed(Duration::from_secs(5)).await, + "broker must close after DISCONNECT" + ); +} + +fn disconnect_with_expiry(session_expiry: u32) -> DisconnectPacket { + let mut packet = DisconnectPacket::new(ReasonCode::Success); + packet + .properties + .set_session_expiry_interval(session_expiry); + packet +} + +async fn resumes(addr: &str, client_id: &str) -> bool { + let (stream, connack) = connect(addr, options(client_id, false, Some(60))).await; + disconnect(stream, DisconnectPacket::new(ReasonCode::Success)).await; + connack.session_present +} + +#[tokio::test] +async fn session_expiry_counts_from_disconnect() { + let broker = Broker::persistent().await; + let (stream, _) = connect(&broker.addr, options("long-lived", true, Some(2))).await; + sleep(Duration::from_secs(3)).await; + disconnect(stream, DisconnectPacket::new(ReasonCode::Success)).await; + + assert!( + resumes(&broker.addr, "long-lived").await, + "a 2s Session Expiry counts from the disconnect, not from CONNECT" + ); +} + +#[tokio::test] +async fn disconnect_expiry_extends_a_long_connection() { + let broker = Broker::persistent().await; + let (stream, _) = connect(&broker.addr, options("long-extend", true, Some(1))).await; + sleep(Duration::from_millis(2500)).await; + disconnect(stream, disconnect_with_expiry(60)).await; + sleep(Duration::from_millis(1500)).await; + + assert!( + resumes(&broker.addr, "long-extend").await, + "the DISCONNECT Session Expiry of 60s applies from the disconnect" + ); +} + +#[tokio::test] +async fn expiry_sweep_keeps_the_session_of_a_connected_client() { + let mut storage = memory_storage(); + storage.cleanup_interval = Duration::from_millis(200); + let broker = Broker::start(BrokerConfig::default().with_storage(storage)).await; + let (stream, _) = connect(&broker.addr, options("swept", true, Some(1))).await; + sleep(Duration::from_millis(2500)).await; + + assert!( + broker + .storage() + .get_session("swept") + .await + .expect("read session") + .is_some(), + "the session of a connected client must not expire" + ); + disconnect(stream, DisconnectPacket::new(ReasonCode::Success)).await; + assert!(resumes(&broker.addr, "swept").await); +} + +#[tokio::test(flavor = "multi_thread", worker_threads = 4)] +async fn expiry_zero_departure_keeps_the_successor_session() { + let broker = Broker::persistent().await; + let rounds = 100; + let mut lost = 0; + for round in 0..rounds { + let client_id = format!("race-{round}"); + let (mut first, _) = connect(&broker.addr, options(&client_id, true, None)).await; + let mut second = Wire::new(TcpStream::connect(&broker.addr).await.expect("connect tcp")); + let mut connect_bytes = Vec::new(); + ConnectPacket::new(options(&client_id, true, Some(3600))) + .encode(&mut connect_bytes) + .expect("encode CONNECT"); + let (first_sent, second_sent) = tokio::join!( + first.write_all(&[0xE0, 0x00]), + second.write_all(&connect_bytes) + ); + first_sent.expect("first DISCONNECT"); + second_sent.expect("second CONNECT"); + read_connack(&mut second).await; + sleep(Duration::from_millis(20)).await; + second + .write_all(&[0xE0, 0x00]) + .await + .expect("second DISCONNECT"); + sleep(Duration::from_millis(20)).await; + if broker + .storage() + .get_session(&client_id) + .await + .expect("read session") + .is_none() + { + lost += 1; + } + } + assert_eq!( + lost, 0, + "{lost}/{rounds} successor sessions with Session Expiry 3600 were deleted by a departing expiry-0 connection" + ); +} + +#[tokio::test(flavor = "multi_thread", worker_threads = 4)] +async fn takeover_of_an_expiry_zero_session_keeps_the_successor_session() { + let broker = Broker::persistent().await; + let rounds = 50; + let mut lost = 0; + for round in 0..rounds { + let client_id = format!("takeover-{round}"); + let (first, _) = connect(&broker.addr, options(&client_id, true, None)).await; + let (mut second, _) = connect(&broker.addr, options(&client_id, true, Some(3600))).await; + sleep(Duration::from_millis(20)).await; + second.write_all(&[0xE0, 0x00]).await.expect("DISCONNECT"); + sleep(Duration::from_millis(50)).await; + drop(first); + if broker + .storage() + .get_session(&client_id) + .await + .expect("read session") + .is_none() + { + lost += 1; + } + } + assert_eq!( + lost, 0, + "{lost}/{rounds} successor sessions lost after takeover" + ); +} + +#[tokio::test(flavor = "multi_thread", worker_threads = 4)] +async fn resume_racing_a_disconnect_with_expiry_zero_keeps_its_subscriptions() { + let broker = Broker::persistent().await; + let (mut publisher, _) = connect(&broker.addr, options("rr-pub", true, None)).await; + let rounds = 60; + let mut resumed = 0; + let mut inconsistent = 0; + for round in 0..rounds { + let client_id = format!("rr-{round}"); + let topic = format!("t/{round:03}"); + let (mut first, _) = connect(&broker.addr, options(&client_id, true, Some(3600))).await; + let mut subscribe = vec![0x82, 0x0B, 0x00, 0x01, 0x00, 0x00, 0x05]; + subscribe.extend_from_slice(topic.as_bytes()); + subscribe.push(0x00); + first.write_all(&subscribe).await.expect("SUBSCRIBE"); + assert!( + matches!( + first.next(Duration::from_secs(5)).await, + Some(Packet::SubAck(_)) + ), + "expected SUBACK" + ); + + let mut second = Wire::new(TcpStream::connect(&broker.addr).await.expect("connect tcp")); + let mut connect_bytes = Vec::new(); + ConnectPacket::new(options(&client_id, false, Some(3600))) + .encode(&mut connect_bytes) + .expect("encode CONNECT"); + let mut end_session = Vec::new(); + disconnect_with_expiry(0) + .encode(&mut end_session) + .expect("encode DISCONNECT"); + let (first_sent, second_sent) = tokio::join!( + first.write_all(&end_session), + second.write_all(&connect_bytes) + ); + first_sent.expect("first DISCONNECT"); + second_sent.expect("second CONNECT"); + let session_present = read_connack(&mut second).await.session_present; + sleep(Duration::from_millis(50)).await; + + let mut publish = vec![0x30, 0x0A, 0x00, 0x05]; + publish.extend_from_slice(topic.as_bytes()); + publish.extend_from_slice(&[0x00, b'h', b'i']); + publisher.write_all(&publish).await.expect("PUBLISH"); + let delivered = matches!( + second.next(Duration::from_millis(300)).await, + Some(Packet::Publish(_)) + ); + let stored = broker + .storage() + .get_session(&client_id) + .await + .expect("read session"); + if session_present { + resumed += 1; + let stored_has_subscription = stored + .as_ref() + .is_some_and(|session| session.subscriptions.contains_key(&topic)); + if !delivered || !stored_has_subscription { + inconsistent += 1; + } + } + second.write_all(&[0xE0, 0x00]).await.expect("DISCONNECT"); + } + assert_eq!( + inconsistent, 0, + "{inconsistent}/{resumed} resumed sessions lost their subscription" + ); +} + +#[tokio::test] +async fn version_one_session_files_migrate_into_the_session_log() { + let dir = tempfile::tempdir().expect("tempdir"); + let sessions = dir.path().join("sessions"); + std::fs::create_dir_all(&sessions).expect("sessions dir"); + std::fs::write(dir.path().join(".storage_version"), "1").expect("version"); + let mut value = + serde_json::to_value(ClientSession::new("legacy", true, Some(3600))).expect("json"); + value + .as_object_mut() + .expect("object") + .remove("connection_token") + .expect("connection token field"); + std::fs::write( + sessions.join("legacy.json"), + serde_json::to_string(&value).expect("json"), + ) + .expect("write legacy file"); + std::fs::write(sessions.join("broken.json"), b"{\"client_id\":").expect("write broken file"); + + let backend = FileBackend::new(dir.path()).await.expect("file backend"); + let loaded = backend + .get_session("legacy") + .await + .expect("read") + .expect("migrated session"); + assert_eq!(loaded.connection_token, 0); + assert_eq!(loaded.expiry_interval, Some(3600)); + assert!(backend.get_session("broken").await.expect("read").is_none()); + drop(backend); + + assert!(!sessions.join("legacy.json").exists()); + assert!(!sessions.join("broken.json").exists()); + assert!( + std::fs::read_dir(&sessions) + .expect("list sessions dir") + .filter_map(std::result::Result::ok) + .any(|entry| entry + .file_name() + .to_string_lossy() + .starts_with("broken.json.corrupt-")), + "the unreadable legacy file was not kept aside" + ); + assert_eq!( + std::fs::read_to_string(dir.path().join(".storage_version")).expect("version"), + "2" + ); + let reopened = FileBackend::new(dir.path()).await.expect("file backend"); + assert!(reopened + .get_session("legacy") + .await + .expect("read") + .is_some()); +} + +fn no_persistence() -> BrokerConfig { + BrokerConfig::default().with_storage(StorageConfig { + enable_persistence: false, + ..Default::default() + }) +} + +#[tokio::test] +async fn will_is_published_without_persistence() { + let broker = Broker::start(no_persistence()).await; + let watcher = MqttClient::new("np-will-watcher"); + watcher + .connect(&broker.url()) + .await + .expect("watcher connect"); + let collector = MessageCollector::new(); + watcher + .subscribe("will/np-will", collector.callback()) + .await + .expect("subscribe"); + sleep(Duration::from_millis(100)).await; + + let mut will = WillMessage::new("will/np-will", "offline"); + will.properties.will_delay_interval = Some(5); + let (stream, _) = connect( + &broker.addr, + options("np-will", true, Some(60)).with_will(will), + ) + .await; + drop(stream); + + assert!( + collector + .wait_for_messages(1, Duration::from_millis(1500)) + .await, + "without persistence the session ends at disconnect, so the Will is published at once" + ); + watcher.disconnect().await.expect("watcher disconnect"); +} + +#[tokio::test] +async fn disconnect_expiry_after_zero_is_a_protocol_error_without_persistence() { + let broker = Broker::start(no_persistence()).await; + let (mut stream, _) = connect(&broker.addr, options("np-error", true, Some(0))).await; + let mut buf = Vec::new(); + disconnect_with_expiry(60) + .encode(&mut buf) + .expect("encode DISCONNECT"); + stream.write_all(&buf).await.expect("write DISCONNECT"); + match stream.next(Duration::from_secs(5)).await { + Some(Packet::Disconnect(reply)) => { + assert_eq!(reply.reason_code, ReasonCode::ProtocolError); + } + other => panic!("the broker must send DISCONNECT, got {other:?}"), + } +} + +#[tokio::test] +async fn sessions_do_not_survive_disconnect_without_persistence() { + let broker = Broker::start(no_persistence()).await; + let (stream, connack) = connect(&broker.addr, options("np-resume", true, Some(60))).await; + assert_eq!( + connack.properties.get_session_expiry_interval(), + Some(0), + "the broker cannot keep the session, so it grants Session Expiry 0" + ); + disconnect(stream, DisconnectPacket::new(ReasonCode::Success)).await; + assert!(!resumes(&broker.addr, "np-resume").await); +} + +fn capped(maximum: u64) -> BrokerConfig { + BrokerConfig::default() + .with_storage(memory_storage()) + .with_session_expiry(Duration::from_secs(maximum)) +} + +#[tokio::test] +async fn session_expiry_above_the_broker_maximum_is_capped_and_advertised() { + let broker = Broker::start(capped(10)).await; + let (stream, connack) = connect(&broker.addr, options("cap-high", true, Some(3600))).await; + assert_eq!(connack.properties.get_session_expiry_interval(), Some(10)); + disconnect(stream, DisconnectPacket::new(ReasonCode::Success)).await; + let stored = broker + .storage() + .get_session("cap-high") + .await + .expect("read session") + .expect("session"); + assert_eq!(stored.expiry_interval, Some(10)); +} + +#[tokio::test] +async fn session_expiry_within_the_broker_maximum_is_not_advertised() { + let broker = Broker::start(capped(10)).await; + let (stream, connack) = connect(&broker.addr, options("cap-low", true, Some(5))).await; + assert_eq!(connack.properties.get_session_expiry_interval(), None); + disconnect(stream, DisconnectPacket::new(ReasonCode::Success)).await; + let stored = broker + .storage() + .get_session("cap-low") + .await + .expect("read session") + .expect("session"); + assert_eq!(stored.expiry_interval, Some(5)); +} + +#[tokio::test] +async fn persistent_v311_session_is_capped_by_the_broker_maximum() { + let broker = Broker::start(capped(10)).await; + let mut packet = ConnectPacket::new(options("cap-v311", false, None)); + packet.protocol_version = 4; + let (stream, _) = connect_packet(&broker.addr, packet).await; + drop(stream); + sleep(Duration::from_millis(200)).await; + let stored = broker + .storage() + .get_session("cap-v311") + .await + .expect("read session") + .expect("session"); + assert_eq!(stored.expiry_interval, Some(10)); +} + +#[tokio::test] +async fn disconnect_expiry_is_capped_by_the_broker_maximum() { + let broker = Broker::start(capped(10)).await; + let (stream, _) = connect(&broker.addr, options("cap-disconnect", true, Some(5))).await; + disconnect(stream, disconnect_with_expiry(3600)).await; + let stored = broker + .storage() + .get_session("cap-disconnect") + .await + .expect("read session") + .expect("session"); + assert_eq!(stored.expiry_interval, Some(10)); +} + +#[tokio::test] +async fn default_broker_does_not_cap_or_advertise_session_expiry() { + let broker = Broker::persistent().await; + let (stream, connack) = connect(&broker.addr, options("uncapped", true, Some(u32::MAX))).await; + assert_eq!(connack.properties.get_session_expiry_interval(), None); + disconnect(stream, DisconnectPacket::new(ReasonCode::Success)).await; +} diff --git a/crates/mqtt5/tests/session_ownership.rs b/crates/mqtt5/tests/session_ownership.rs new file mode 100644 index 00000000..07e0f481 --- /dev/null +++ b/crates/mqtt5/tests/session_ownership.rs @@ -0,0 +1,950 @@ +#![cfg(feature = "broker")] + +use bytes::BytesMut; +use mqtt5::broker::auth::{AuthProvider, AuthResult}; +use mqtt5::broker::config::{BrokerConfig, StorageBackend as BackendKind, StorageConfig}; +use mqtt5::broker::router::MessageRouter; +use mqtt5::broker::server::MqttBroker; +use mqtt5::broker::storage::{ + ClientSession, DynamicStorage, FileBackend, StorageBackend, StoredSubscription, +}; +use mqtt5::error::Result; +use mqtt5::time::Duration; +use mqtt5::transport::packet_io::read_packet_from_stream; +use mqtt5_protocol::packet::connack::ConnAckPacket; +use mqtt5_protocol::packet::connect::ConnectPacket; +use mqtt5_protocol::packet::{MqttPacket, Packet}; +use mqtt5_protocol::protocol::v5::reason_codes::ReasonCode; +use mqtt5_protocol::types::ConnectOptions; +use std::future::Future; +use std::net::SocketAddr; +use std::path::Path; +use std::pin::Pin; +use std::sync::atomic::{AtomicBool, Ordering}; +use std::sync::Arc; +use tokio::io::AsyncWriteExt; +use tokio::net::TcpStream; +use tokio::time::sleep; + +struct Provider { + slow_subscribe: Duration, + deny_secret: AtomicBool, +} + +impl AuthProvider for Provider { + fn authenticate<'a>( + &'a self, + _connect: &'a ConnectPacket, + _client_addr: SocketAddr, + ) -> Pin> + Send + 'a>> { + Box::pin(async move { Ok(AuthResult::success()) }) + } + + fn authorize_publish<'a>( + &'a self, + _client_id: &str, + _user_id: Option<&'a str>, + _topic: &'a str, + ) -> Pin + Send + 'a>> { + Box::pin(async move { true }) + } + + fn authorize_subscribe<'a>( + &'a self, + _client_id: &str, + _user_id: Option<&'a str>, + topic_filter: &'a str, + ) -> Pin + Send + 'a>> { + Box::pin(async move { + if topic_filter.starts_with("slow/") { + sleep(self.slow_subscribe).await; + } + !(topic_filter.starts_with("secret/") && self.deny_secret.load(Ordering::SeqCst)) + }) + } +} + +struct Broker { + addr: String, + storage: Option>, + router: Arc, + provider: Arc, + handle: tokio::task::JoinHandle<()>, +} + +impl Broker { + async fn start(config: BrokerConfig, slow_subscribe: Duration) -> Self { + let config = + config.with_bind_address("127.0.0.1:0".parse::().expect("bind address")); + let provider = Arc::new(Provider { + slow_subscribe, + deny_secret: AtomicBool::new(false), + }); + let mut broker = MqttBroker::with_config(config) + .await + .expect("start broker") + .with_auth_provider(Arc::clone(&provider) as Arc); + let addr = broker.local_addr().expect("broker address").to_string(); + let storage = broker.storage(); + let router = broker.router(); + let handle = tokio::spawn(async move { + if let Err(e) = broker.run().await { + tracing::debug!("broker stopped: {e}"); + } + }); + sleep(Duration::from_millis(100)).await; + Self { + addr, + storage, + router, + provider, + handle, + } + } + + async fn stop(self) { + self.handle.abort(); + let aborted = self.handle.await; + assert!(aborted.is_err_and(|e| e.is_cancelled())); + } + + fn storage(&self) -> &DynamicStorage { + self.storage.as_deref().expect("persistence is enabled") + } +} + +fn memory(cleanup_interval: Duration) -> BrokerConfig { + BrokerConfig::default().with_storage(StorageConfig { + backend: BackendKind::Memory, + enable_persistence: true, + cleanup_interval, + ..Default::default() + }) +} + +fn file(dir: &Path, cleanup_interval: Duration) -> BrokerConfig { + BrokerConfig::default().with_storage(StorageConfig { + backend: BackendKind::File, + base_dir: dir.to_path_buf(), + enable_persistence: true, + cleanup_interval, + ..Default::default() + }) +} + +fn no_persistence() -> BrokerConfig { + BrokerConfig::default().with_storage(StorageConfig { + enable_persistence: false, + ..Default::default() + }) +} + +const SWEEP: Duration = Duration::from_millis(200); +const NEVER: Duration = Duration::from_secs(3600); + +fn options(client_id: &str, clean_start: bool, session_expiry: Option) -> ConnectOptions { + let options = ConnectOptions::new(client_id).with_clean_start(clean_start); + match session_expiry { + Some(expiry) => options.with_session_expiry_interval(expiry), + None => options, + } +} + +fn connect_bytes(options: ConnectOptions) -> Vec { + let mut bytes = Vec::new(); + ConnectPacket::new(options) + .encode(&mut bytes) + .expect("encode CONNECT"); + bytes +} + +struct Wire { + stream: TcpStream, + buffer: BytesMut, +} + +impl Wire { + fn new(stream: TcpStream) -> Self { + Self { + stream, + buffer: BytesMut::new(), + } + } + + async fn write_all(&mut self, bytes: &[u8]) -> std::io::Result<()> { + self.stream.write_all(bytes).await + } + + async fn next(&mut self, wait: Duration) -> Option { + tokio::time::timeout( + wait, + read_packet_from_stream(&mut self.stream, 5, &mut self.buffer, 1 << 20), + ) + .await + .ok()? + .ok() + } + + async fn closed(&mut self, wait: Duration) -> bool { + tokio::time::timeout( + wait, + read_packet_from_stream(&mut self.stream, 5, &mut self.buffer, 1 << 20), + ) + .await + .is_ok_and(|read| read.is_err()) + } +} + +async fn read_connack(wire: &mut Wire) -> ConnAckPacket { + match wire.next(Duration::from_secs(5)).await { + Some(Packet::ConnAck(connack)) => connack, + other => panic!("expected CONNACK, got {other:?}"), + } +} + +async fn connect(addr: &str, options: ConnectOptions) -> (Wire, ConnAckPacket) { + let mut wire = Wire::new(TcpStream::connect(addr).await.expect("connect tcp")); + wire.write_all(&connect_bytes(options)) + .await + .expect("write CONNECT"); + let connack = read_connack(&mut wire).await; + (wire, connack) +} + +fn subscribe_bytes(packet_id: u8, topic: &str, qos: u8) -> Vec { + let topic_len = u8::try_from(topic.len()).expect("short topic"); + let mut bytes = vec![0x82, 6 + topic_len, 0x00, packet_id, 0x00, 0x00, topic_len]; + bytes.extend_from_slice(topic.as_bytes()); + bytes.push(qos); + bytes +} + +async fn subscribe(wire: &mut Wire, topic: &str, qos: u8) { + wire.write_all(&subscribe_bytes(1, topic, qos)) + .await + .expect("SUBSCRIBE"); + assert!( + matches!( + wire.next(Duration::from_secs(5)).await, + Some(Packet::SubAck(_)) + ), + "expected SUBACK" + ); +} + +async fn publish(wire: &mut Wire, topic: &str) { + let topic_len = u8::try_from(topic.len()).expect("short topic"); + let mut bytes = vec![0x30, 5 + topic_len, 0x00, topic_len]; + bytes.extend_from_slice(topic.as_bytes()); + bytes.extend_from_slice(&[0x00, b'h', b'i']); + wire.write_all(&bytes).await.expect("PUBLISH"); +} + +async fn received_publish(wire: &mut Wire) -> bool { + matches!( + wire.next(Duration::from_millis(500)).await, + Some(Packet::Publish(_)) + ) +} + +async fn end(mut wire: Wire) { + wire.write_all(&[0xE0, 0x00]).await.expect("DISCONNECT"); + assert!( + wire.closed(Duration::from_secs(5)).await, + "broker must close after DISCONNECT" + ); +} + +async fn still_served(wire: &mut Wire) -> bool { + wire.write_all(&[0xC0, 0x00]).await.is_ok() + && matches!( + wire.next(Duration::from_millis(300)).await, + Some(Packet::PingResp) + ) +} + +#[tokio::test] +async fn subscribe_from_a_displaced_connection_does_not_keep_the_session_alive() { + let broker = Broker::start(memory(SWEEP), Duration::from_millis(600)).await; + let (mut first, _) = connect(&broker.addr, options("leak", true, Some(1))).await; + first + .write_all(&subscribe_bytes(1, "slow/a", 1)) + .await + .expect("SUBSCRIBE"); + sleep(Duration::from_millis(100)).await; + + let (second, _) = connect(&broker.addr, options("leak", true, Some(1))).await; + sleep(Duration::from_millis(1000)).await; + end(second).await; + drop(first); + sleep(Duration::from_millis(3000)).await; + + let stored = broker.storage().get_session("leak").await.expect("read"); + assert!( + stored.is_none(), + "no connection is live and the 1s expiry passed, yet the session is still stored" + ); +} + +#[tokio::test] +async fn subscribe_from_a_displaced_connection_does_not_keep_an_expiry_zero_session() { + let broker = Broker::start(memory(SWEEP), Duration::from_millis(600)).await; + let (mut first, _) = connect(&broker.addr, options("leak0", true, Some(0))).await; + first + .write_all(&subscribe_bytes(1, "slow/a", 1)) + .await + .expect("SUBSCRIBE"); + sleep(Duration::from_millis(100)).await; + + let (second, _) = connect(&broker.addr, options("leak0", true, Some(0))).await; + sleep(Duration::from_millis(1000)).await; + end(second).await; + drop(first); + sleep(Duration::from_millis(1000)).await; + + assert!( + broker + .storage() + .get_session("leak0") + .await + .expect("read") + .is_none(), + "an expiry-0 session outlived every connection" + ); +} + +#[tokio::test(flavor = "multi_thread", worker_threads = 4)] +async fn subscribe_racing_a_takeover_leaks_no_session() { + let broker = Broker::start(memory(SWEEP), Duration::ZERO).await; + let rounds: u64 = 60; + for round in 0..rounds { + let client_id = format!("sr-{round}"); + let (mut first, _) = connect(&broker.addr, options(&client_id, true, Some(0))).await; + let mut burst = Vec::new(); + for i in 0..2000u16 { + let topic = format!("t/{i:05}"); + let packet_id = u8::try_from(i % 250 + 1).expect("small packet id"); + burst.extend_from_slice(&subscribe_bytes(packet_id, &topic, 0)); + } + let writer = tokio::spawn(async move { + let written = first.write_all(&burst).await; + sleep(Duration::from_millis(300)).await; + drop(first); + written + }); + sleep(Duration::from_micros((round % 10) * 200)).await; + let (mut second, _) = connect(&broker.addr, options(&client_id, true, Some(0))).await; + sleep(Duration::from_millis(50)).await; + second.write_all(&[0xE0, 0x00]).await.expect("DISCONNECT"); + drop(second); + if let Err(e) = writer.await.expect("writer task") { + tracing::debug!("burst ended early: {e}"); + } + } + sleep(Duration::from_millis(1500)).await; + let mut leaked = Vec::new(); + for round in 0..rounds { + let client_id = format!("sr-{round}"); + if let Some(session) = broker + .storage() + .get_session(&client_id) + .await + .expect("read") + { + leaked.push((client_id, session.connected, session.subscriptions.len())); + } + } + assert!( + leaked.is_empty(), + "{} expiry-0 sessions outlived all their connections: {leaked:?}", + leaked.len() + ); +} + +#[tokio::test] +async fn resumed_session_drops_a_revoked_subscription_from_routing() { + let broker = Broker::start(memory(NEVER), Duration::ZERO).await; + let (mut publisher, _) = connect(&broker.addr, options("rv-pub", true, Some(0))).await; + let (mut sub, _) = connect(&broker.addr, options("rv", true, Some(3600))).await; + subscribe(&mut sub, "secret/x", 1).await; + end(sub).await; + sleep(Duration::from_millis(200)).await; + + broker.provider.deny_secret.store(true, Ordering::SeqCst); + let (mut sub, connack) = connect(&broker.addr, options("rv", false, Some(3600))).await; + assert!(connack.session_present); + sleep(Duration::from_millis(100)).await; + publish(&mut publisher, "secret/x").await; + assert!( + !received_publish(&mut sub).await, + "a subscription dropped on restore as no longer authorized must not deliver" + ); +} + +#[tokio::test(flavor = "multi_thread", worker_threads = 4)] +async fn aborted_takeover_never_replaces_a_live_owners_session() { + let broker = Broker::start(memory(NEVER), Duration::ZERO).await; + let mut orphaned = 0; + let mut kept = 0; + for round in 0..40 { + let client_id = format!("abort-{round}"); + let (mut first, _) = connect(&broker.addr, options(&client_id, true, Some(60))).await; + subscribe(&mut first, "abort/t", 0).await; + let before = broker + .storage() + .get_session(&client_id) + .await + .expect("read") + .expect("session") + .connection_token; + + let mut intruder = TcpStream::connect(&broker.addr).await.expect("connect tcp"); + intruder.set_zero_linger().expect("linger"); + intruder + .write_all(&connect_bytes(options(&client_id, true, Some(60)))) + .await + .expect("write CONNECT"); + drop(intruder); + sleep(Duration::from_millis(100)).await; + + let after = broker + .storage() + .get_session(&client_id) + .await + .expect("read") + .map(|session| { + ( + session.connection_token, + session.subscriptions.contains_key("abort/t"), + ) + }); + if still_served(&mut first).await { + kept += 1; + if after != Some((before, true)) { + orphaned += 1; + } + } + } + assert_eq!( + orphaned, 0, + "{orphaned}/{kept} aborted takeovers changed the stored session of a connection that stayed live" + ); +} + +#[tokio::test] +async fn takeover_of_a_live_expiry_zero_session_is_not_resumed() { + let broker = Broker::start(memory(NEVER), Duration::ZERO).await; + let (first, _) = connect(&broker.addr, options("live-zero", true, Some(0))).await; + sleep(Duration::from_millis(200)).await; + let (second, connack) = connect(&broker.addr, options("live-zero", false, Some(60))).await; + drop(second); + drop(first); + assert!(!connack.session_present); +} + +#[tokio::test] +async fn takeover_without_persistence_does_not_inherit_subscriptions() { + let broker = Broker::start(no_persistence(), Duration::ZERO).await; + let (mut publisher, _) = connect(&broker.addr, options("np-pub", true, None)).await; + let (mut first, _) = connect(&broker.addr, options("np-take", true, Some(60))).await; + subscribe(&mut first, "np/t", 0).await; + let (mut second, connack) = connect(&broker.addr, options("np-take", false, Some(60))).await; + sleep(Duration::from_millis(200)).await; + publish(&mut publisher, "np/t").await; + let delivered = received_publish(&mut second).await; + drop(first); + assert!(!connack.session_present); + assert!( + !delivered, + "Session Present 0, yet the new connection receives the old connection's subscription" + ); +} + +#[tokio::test] +async fn takeover_of_an_expiry_zero_session_does_not_inherit_subscriptions() { + let broker = Broker::start(memory(NEVER), Duration::ZERO).await; + let (mut publisher, _) = connect(&broker.addr, options("p0-pub", true, None)).await; + let (mut first, _) = connect(&broker.addr, options("p0-take", true, Some(0))).await; + subscribe(&mut first, "p0/t", 0).await; + let (mut second, connack) = connect(&broker.addr, options("p0-take", false, Some(60))).await; + sleep(Duration::from_millis(200)).await; + publish(&mut publisher, "p0/t").await; + let delivered = received_publish(&mut second).await; + drop(first); + assert!(!connack.session_present); + assert!(!delivered); +} + +#[tokio::test] +async fn clean_start_after_an_offline_session_drops_its_subscriptions() { + let broker = Broker::start(memory(NEVER), Duration::ZERO).await; + let (mut publisher, _) = connect(&broker.addr, options("cs-pub", true, None)).await; + let (mut first, _) = connect(&broker.addr, options("cs-take", true, Some(60))).await; + subscribe(&mut first, "cs/t", 0).await; + end(first).await; + let (mut second, connack) = connect(&broker.addr, options("cs-take", true, Some(60))).await; + sleep(Duration::from_millis(200)).await; + publish(&mut publisher, "cs/t").await; + let delivered = received_publish(&mut second).await; + assert!(!connack.session_present); + assert!(!delivered); +} + +#[tokio::test(flavor = "multi_thread", worker_threads = 4)] +async fn session_from_a_previous_run_expires_after_restart() { + let dir = tempfile::tempdir().expect("tempdir"); + { + let backend = FileBackend::new(dir.path()).await.expect("file backend"); + let session = ClientSession::new("stale", true, Some(1)); + backend.store_session(session).await.expect("store"); + } + let broker = Broker::start(file(dir.path(), SWEEP), Duration::ZERO).await; + sleep(Duration::from_millis(2500)).await; + let stored = broker + .storage() + .session_client_ids() + .await + .expect("list sessions"); + broker.stop().await; + assert!( + !stored.iter().any(|id| id == "stale"), + "a 1s-expiry session written by a previous run must expire after the restart" + ); + let reopened = FileBackend::new(dir.path()).await.expect("file backend"); + assert!(reopened + .session_client_ids() + .await + .expect("list sessions") + .is_empty()); +} + +#[tokio::test(flavor = "multi_thread", worker_threads = 4)] +async fn file_backend_expiry_zero_departure_keeps_the_successor_session() { + let dir = tempfile::tempdir().expect("tempdir"); + let broker = Broker::start(file(dir.path(), NEVER), Duration::ZERO).await; + let rounds = 30; + let mut lost = 0; + for round in 0..rounds { + let client_id = format!("frace-{round}"); + let (mut first, _) = connect(&broker.addr, options(&client_id, true, None)).await; + let mut second = Wire::new(TcpStream::connect(&broker.addr).await.expect("connect tcp")); + let claim = connect_bytes(options(&client_id, true, Some(3600))); + let (first_sent, second_sent) = + tokio::join!(first.write_all(&[0xE0, 0x00]), second.write_all(&claim)); + first_sent.expect("first DISCONNECT"); + second_sent.expect("second CONNECT"); + read_connack(&mut second).await; + sleep(Duration::from_millis(20)).await; + second + .write_all(&[0xE0, 0x00]) + .await + .expect("second DISCONNECT"); + sleep(Duration::from_millis(30)).await; + if broker + .storage() + .get_session(&client_id) + .await + .expect("read session") + .is_none() + { + lost += 1; + } + } + broker.stop().await; + assert_eq!( + lost, 0, + "{lost}/{rounds} successor sessions were deleted by a departing expiry-0 connection" + ); +} + +#[tokio::test(flavor = "multi_thread", worker_threads = 4)] +async fn file_backend_displaced_subscribe_writes_nothing() { + let dir = tempfile::tempdir().expect("tempdir"); + let broker = Broker::start(file(dir.path(), SWEEP), Duration::from_millis(600)).await; + let (mut first, _) = connect(&broker.addr, options("fleak", true, Some(1))).await; + first + .write_all(&subscribe_bytes(1, "slow/a", 1)) + .await + .expect("SUBSCRIBE"); + sleep(Duration::from_millis(100)).await; + let (second, _) = connect(&broker.addr, options("fleak", true, Some(1))).await; + sleep(Duration::from_millis(1000)).await; + let stored = broker + .storage() + .get_session("fleak") + .await + .expect("read") + .expect("session of the live successor"); + assert!( + !stored.subscriptions.contains_key("slow/a"), + "the displaced connection's late SUBSCRIBE must not reach the successor's stored session" + ); + end(second).await; + drop(first); + sleep(Duration::from_millis(2500)).await; + let stored = broker + .storage() + .session_client_ids() + .await + .expect("list sessions"); + broker.stop().await; + assert!(!stored.iter().any(|id| id == "fleak")); +} + +fn unsubscribe_bytes(topic: &str) -> Vec { + let topic_len = u8::try_from(topic.len()).expect("short topic"); + let mut bytes = vec![0xA2, 5 + topic_len, 0x00, 2, 0x00, 0x00, topic_len]; + bytes.extend_from_slice(topic.as_bytes()); + bytes +} + +#[tokio::test(flavor = "multi_thread", worker_threads = 4)] +async fn concurrent_claims_agree_on_the_owner() { + let dir = tempfile::tempdir().expect("tempdir"); + let broker = Broker::start(file(dir.path(), NEVER), Duration::ZERO).await; + let rounds = 60; + let mut disagreements = Vec::new(); + for round in 0..rounds { + let client_id = format!("race-{round}"); + let claim = connect_bytes(options(&client_id, round % 2 == 0, Some(60))); + let mut streams = Vec::new(); + for _ in 0..3 { + streams.push(Wire::new( + TcpStream::connect(&broker.addr).await.expect("connect tcp"), + )); + } + futures::future::join_all(streams.iter_mut().map(|stream| stream.write_all(&claim))) + .await + .into_iter() + .for_each(|written| written.expect("write CONNECT")); + for stream in &mut streams { + read_connack(stream).await; + } + let stored = broker + .storage() + .get_session(&client_id) + .await + .expect("read") + .expect("claimed session"); + if !broker + .router + .is_current_owner(&client_id, stored.connection_token) + .await + { + disagreements.push(client_id); + } + } + broker.stop().await; + assert!( + disagreements.is_empty(), + "stored session token is not the router owner: {disagreements:?}" + ); +} + +#[tokio::test(flavor = "multi_thread", worker_threads = 4)] +async fn displaced_subscribe_installs_no_route_for_the_successor() { + let broker = Broker::start(memory(NEVER), Duration::from_millis(600)).await; + let (mut publisher, _) = connect(&broker.addr, options("dsub-pub", true, Some(0))).await; + let (mut first, _) = connect(&broker.addr, options("dsub", true, Some(60))).await; + first + .write_all(&subscribe_bytes(1, "slow/a", 1)) + .await + .expect("SUBSCRIBE"); + sleep(Duration::from_millis(100)).await; + let (mut second, _) = connect(&broker.addr, options("dsub", true, Some(60))).await; + sleep(Duration::from_millis(1000)).await; + let routed = broker.router.has_subscription("dsub", "slow/a").await; + publish(&mut publisher, "slow/a").await; + let delivered = received_publish(&mut second).await; + drop(first); + broker.stop().await; + assert!( + !routed, + "a displaced connection's SUBSCRIBE installed a route for the successor" + ); + assert!(!delivered); +} + +#[tokio::test(flavor = "multi_thread", worker_threads = 4)] +async fn displaced_unsubscribe_keeps_the_successors_route() { + let broker = Broker::start(memory(NEVER), Duration::from_millis(600)).await; + let (mut first, _) = connect(&broker.addr, options("dunsub", true, Some(60))).await; + subscribe(&mut first, "keep/x", 1).await; + let mut burst = subscribe_bytes(1, "slow/a", 1); + burst.extend_from_slice(&unsubscribe_bytes("keep/x")); + first + .write_all(&burst) + .await + .expect("SUBSCRIBE and UNSUBSCRIBE"); + sleep(Duration::from_millis(100)).await; + let (_second, connack) = connect(&broker.addr, options("dunsub", false, Some(60))).await; + assert!(connack.session_present); + sleep(Duration::from_millis(1000)).await; + let kept = broker.router.has_subscription("dunsub", "keep/x").await; + let stored = broker + .storage() + .get_session("dunsub") + .await + .expect("read") + .expect("session"); + drop(first); + broker.stop().await; + assert!( + kept, + "a displaced connection's UNSUBSCRIBE removed the successor's route" + ); + assert!(stored.subscriptions.contains_key("keep/x")); +} + +#[tokio::test(flavor = "multi_thread", worker_threads = 4)] +async fn displaced_unsubscribe_waiting_on_the_slot_is_fenced() { + let broker = Broker::start(memory(NEVER), Duration::ZERO).await; + let (mut first, _) = connect(&broker.addr, options("dunsub2", true, Some(60))).await; + subscribe(&mut first, "keep/x", 1).await; + let slot = broker.router.lock_session("dunsub2").await; + let mut second = Wire::new(TcpStream::connect(&broker.addr).await.expect("connect tcp")); + second + .write_all(&connect_bytes(options("dunsub2", false, Some(60)))) + .await + .expect("write CONNECT"); + sleep(Duration::from_millis(200)).await; + first + .write_all(&unsubscribe_bytes("keep/x")) + .await + .expect("UNSUBSCRIBE"); + sleep(Duration::from_millis(200)).await; + drop(slot); + let connack = read_connack(&mut second).await; + assert!(connack.session_present); + sleep(Duration::from_millis(300)).await; + let kept = broker.router.has_subscription("dunsub2", "keep/x").await; + let stored = broker + .storage() + .get_session("dunsub2") + .await + .expect("read") + .expect("session"); + drop(first); + broker.stop().await; + assert!( + kept, + "an UNSUBSCRIBE queued behind the successor's claim removed the successor's route" + ); + assert!(stored.subscriptions.contains_key("keep/x")); +} + +#[tokio::test(flavor = "multi_thread", worker_threads = 4)] +async fn sweep_without_persistence_keeps_a_live_connections_routes() { + let config = BrokerConfig::default().with_storage(StorageConfig { + enable_persistence: false, + cleanup_interval: Duration::from_millis(100), + ..Default::default() + }); + let broker = Broker::start(config, Duration::ZERO).await; + let (mut stream, _) = connect(&broker.addr, options("np-live", true, Some(0))).await; + subscribe(&mut stream, "np/t", 1).await; + sleep(Duration::from_millis(600)).await; + let kept = broker.router.has_subscription("np-live", "np/t").await; + broker.stop().await; + assert!(kept, "the sweep stripped a live connection's route"); +} + +#[tokio::test(flavor = "multi_thread", worker_threads = 4)] +async fn claim_waiting_past_the_connect_timeout_still_completes() { + let broker = Broker::start(memory(NEVER), Duration::ZERO).await; + let slot = broker.router.lock_session("late").await; + let mut stream = Wire::new(TcpStream::connect(&broker.addr).await.expect("connect tcp")); + stream + .write_all(&connect_bytes(options("late", true, Some(60)))) + .await + .expect("write CONNECT"); + sleep(Duration::from_millis(10_500)).await; + drop(slot); + let connack = read_connack(&mut stream).await; + assert_eq!(connack.reason_code, ReasonCode::Success); + let token = broker + .storage() + .get_session("late") + .await + .expect("read") + .expect("claimed session") + .connection_token; + let owner = broker.router.is_current_owner("late", token).await; + let served = still_served(&mut stream).await; + broker.stop().await; + assert!( + owner && served, + "the claim must finish once the CONNECT was read" + ); +} + +fn seeded(client_id: &str, expiry: u32, topic: Option<&str>) -> ClientSession { + let mut session = ClientSession::new(client_id, true, Some(expiry)); + session.mark_connected(7); + if let Some(topic) = topic { + session.add_subscription(topic, StoredSubscription::new(mqtt5::QoS::AtLeastOnce)); + } + session +} + +async fn seed(dir: &Path, sessions: Vec) { + let backend = FileBackend::new(dir).await.expect("file backend"); + futures::future::join_all( + sessions + .into_iter() + .map(|session| backend.store_session(session)), + ) + .await + .into_iter() + .for_each(|stored| stored.expect("store")); +} + +#[tokio::test(flavor = "multi_thread", worker_threads = 4)] +async fn session_left_connected_by_a_previous_run_expires_after_boot() { + let dir = tempfile::tempdir().expect("tempdir"); + seed(dir.path(), vec![seeded("seeded", 1, None)]).await; + let broker = Broker::start(file(dir.path(), NEVER), Duration::ZERO).await; + sleep(Duration::from_millis(2500)).await; + let (_stream, connack) = connect(&broker.addr, options("seeded", false, Some(60))).await; + broker.stop().await; + assert!( + !connack.session_present, + "a session connected when the broker stopped ends at boot" + ); +} + +#[tokio::test(flavor = "multi_thread", worker_threads = 4)] +async fn expiry_zero_session_from_a_previous_run_is_dropped_at_boot() { + let dir = tempfile::tempdir().expect("tempdir"); + seed(dir.path(), vec![seeded("seeded0", 0, Some("s0/t"))]).await; + let broker = Broker::start(file(dir.path(), NEVER), Duration::ZERO).await; + let routes = broker.router.subscription_count_for_client("seeded0").await; + let stored = broker + .storage() + .session_client_ids() + .await + .expect("list sessions"); + broker.stop().await; + assert_eq!(routes, 0); + assert!(stored.is_empty()); +} + +#[tokio::test(flavor = "multi_thread", worker_threads = 4)] +async fn recovery_caps_stored_expiry_to_the_configured_maximum() { + let dir = tempfile::tempdir().expect("tempdir"); + let mut long_ended = seeded("long-ended", 3600, Some("le/t")); + long_ended.mark_disconnected(mqtt5::broker::storage::unix_millis_now() - 120_000); + seed( + dir.path(), + vec![long_ended, seeded("capped", 3600, Some("c/t"))], + ) + .await; + let config = file(dir.path(), NEVER).with_session_expiry(Duration::from_secs(60)); + let broker = Broker::start(config, Duration::ZERO).await; + let capped = broker + .storage() + .get_session("capped") + .await + .expect("read") + .expect("capped session"); + let ended = broker + .storage() + .get_session("long-ended") + .await + .expect("read"); + let ended_routes = broker + .router + .subscription_count_for_client("long-ended") + .await; + broker.stop().await; + assert_eq!(capped.expiry_interval, Some(60)); + assert!( + ended.is_none() && ended_routes == 0, + "a session that ended 120s ago has outlived the 60s maximum" + ); +} + +#[tokio::test(flavor = "multi_thread", worker_threads = 4)] +async fn boot_with_many_sessions_left_connected_accepts_connections_promptly() { + let dir = tempfile::tempdir().expect("tempdir"); + seed( + dir.path(), + (0..500) + .map(|i| seeded(&format!("boot-{i}"), 3600, Some("boot/t"))) + .collect(), + ) + .await; + let started = std::time::Instant::now(); + let broker = Broker::start(file(dir.path(), NEVER), Duration::ZERO).await; + let (_stream, connack) = connect(&broker.addr, options("first", true, Some(0))).await; + let elapsed = started.elapsed(); + let routes = broker + .router + .subscription_count_for_client("boot-499") + .await; + broker.stop().await; + assert_eq!(connack.reason_code, ReasonCode::Success); + assert_eq!(routes, 1); + assert!( + elapsed < std::time::Duration::from_secs(2), + "recovering 500 sessions delayed the first CONNACK by {elapsed:?}" + ); +} + +async fn seed_unacked(dir: &Path, client_id: &str) { + use mqtt5::broker::storage::{InflightDirection, InflightMessage, InflightPhase}; + let backend = FileBackend::new(dir).await.expect("file backend"); + let mut session = ClientSession::new(client_id, true, Some(3600)); + session.mark_disconnected(mqtt5::broker::storage::unix_millis_now()); + backend.store_session(session).await.expect("store session"); + let mut publish = mqtt5::packet::publish::PublishPacket::new( + "unacked/t".to_string(), + b"unacked".to_vec(), + mqtt5::QoS::AtLeastOnce, + ); + publish.packet_id = Some(7); + backend + .store_inflight_message(InflightMessage::from_publish( + &publish, + client_id.to_string(), + InflightDirection::Outbound, + InflightPhase::AwaitingPubrec, + )) + .await + .expect("store inflight"); + backend.flush_queue_writes().await; +} + +#[tokio::test(flavor = "multi_thread", worker_threads = 4)] +async fn resume_after_restart_redelivers_the_persisted_inflight_message() { + let dir = tempfile::tempdir().expect("tempdir"); + seed_unacked(dir.path(), "unacked").await; + let broker = Broker::start(file(dir.path(), NEVER), Duration::ZERO).await; + let (mut wire, connack) = connect(&broker.addr, options("unacked", false, Some(3600))).await; + let redelivered = wire.next(Duration::from_secs(2)).await; + broker.stop().await; + assert!(connack.session_present); + match redelivered { + Some(Packet::Publish(publish)) => { + assert!(publish.dup); + assert_eq!(publish.topic_name, "unacked/t"); + } + other => panic!("the persisted unacknowledged message was not redelivered: {other:?}"), + } +} + +#[tokio::test(flavor = "multi_thread", worker_threads = 4)] +async fn clean_start_after_restart_drops_the_persisted_inflight_message() { + let dir = tempfile::tempdir().expect("tempdir"); + seed_unacked(dir.path(), "unacked-clean").await; + let broker = Broker::start(file(dir.path(), NEVER), Duration::ZERO).await; + let (mut wire, connack) = + connect(&broker.addr, options("unacked-clean", true, Some(3600))).await; + let delivered = wire.next(Duration::from_millis(500)).await; + broker.stop().await; + assert!(!connack.session_present); + assert!( + delivered.is_none(), + "a clean start delivered the old session's message: {delivered:?}" + ); +} diff --git a/crates/mqtt5/tests/subscription_options_persistence.rs b/crates/mqtt5/tests/subscription_options_persistence.rs index a58bdbde..681fc140 100644 --- a/crates/mqtt5/tests/subscription_options_persistence.rs +++ b/crates/mqtt5/tests/subscription_options_persistence.rs @@ -9,12 +9,18 @@ use std::sync::atomic::{AtomicU32, Ordering}; use std::sync::Arc; use tokio::time::sleep; +const SESSION_EXPIRY: u32 = 300; + #[tokio::test] async fn test_no_local_persists_after_reconnect() { let broker = TestBroker::start().await; let client_id = "no-local-persist-test"; - let client1 = MqttClient::with_options(ConnectOptions::new(client_id).with_clean_start(true)); + let client1 = MqttClient::with_options( + ConnectOptions::new(client_id) + .with_clean_start(true) + .with_session_expiry_interval(SESSION_EXPIRY), + ); client1.connect(broker.address()).await.unwrap(); client1 @@ -41,17 +47,17 @@ async fn test_no_local_persists_after_reconnect() { broker.address(), ConnectOptions::new(client_id) .with_clean_start(false) + .with_session_expiry_interval(SESSION_EXPIRY) .with_resume_existing_session(true), ), ) .await .unwrap(); - if !session.session_present { - println!("Session not preserved, skipping test"); - client2.disconnect().await.unwrap(); - return; - } + assert!( + session.session_present, + "a session with a Session Expiry Interval must be resumed" + ); client2 .subscribe_with_options( @@ -120,7 +126,11 @@ async fn test_retain_as_published_persists_after_reconnect() { .unwrap(); pub_client.disconnect().await.unwrap(); - let client1 = MqttClient::with_options(ConnectOptions::new(client_id).with_clean_start(true)); + let client1 = MqttClient::with_options( + ConnectOptions::new(client_id) + .with_clean_start(true) + .with_session_expiry_interval(SESSION_EXPIRY), + ); client1.connect(broker.address()).await.unwrap(); client1 @@ -147,17 +157,17 @@ async fn test_retain_as_published_persists_after_reconnect() { broker.address(), ConnectOptions::new(client_id) .with_clean_start(false) + .with_session_expiry_interval(SESSION_EXPIRY) .with_resume_existing_session(true), ), ) .await .unwrap(); - if !session.session_present { - println!("Session not preserved, skipping test"); - client2.disconnect().await.unwrap(); - return; - } + assert!( + session.session_present, + "a session with a Session Expiry Interval must be resumed" + ); client2 .subscribe_with_options( @@ -209,7 +219,11 @@ async fn test_subscription_options_all_preserved() { let broker = TestBroker::start().await; let client_id = "full-options-persist"; - let client1 = MqttClient::with_options(ConnectOptions::new(client_id).with_clean_start(true)); + let client1 = MqttClient::with_options( + ConnectOptions::new(client_id) + .with_clean_start(true) + .with_session_expiry_interval(SESSION_EXPIRY), + ); client1.connect(broker.address()).await.unwrap(); client1 @@ -237,17 +251,17 @@ async fn test_subscription_options_all_preserved() { broker.address(), ConnectOptions::new(client_id) .with_clean_start(false) + .with_session_expiry_interval(SESSION_EXPIRY) .with_resume_existing_session(true), ), ) .await .unwrap(); - if !session.session_present { - println!("Session not preserved, skipping test"); - client2.disconnect().await.unwrap(); - return; - } + assert!( + session.session_present, + "a session with a Session Expiry Interval must be resumed" + ); client2 .subscribe_with_options( diff --git a/crates/mqtt5/tests/will_delay.rs b/crates/mqtt5/tests/will_delay.rs index bcd161f7..b09f74ae 100644 --- a/crates/mqtt5/tests/will_delay.rs +++ b/crates/mqtt5/tests/will_delay.rs @@ -2,6 +2,9 @@ mod common; use common::{MessageCollector, TestBroker}; +use mqtt5::broker::config::{BrokerConfig, StorageBackend, StorageConfig}; +use mqtt5::broker::server::MqttBroker; +use mqtt5::broker::storage::{DynamicStorage, StorageBackend as _}; use mqtt5::time::Duration; use mqtt5::MqttClient; use mqtt5_protocol::packet::connect::ConnectPacket; @@ -9,6 +12,8 @@ use mqtt5_protocol::packet::disconnect::DisconnectPacket; use mqtt5_protocol::packet::MqttPacket; use mqtt5_protocol::protocol::v5::reason_codes::ReasonCode; use mqtt5_protocol::types::{ConnectOptions, WillMessage}; +use std::net::SocketAddr; +use std::sync::Arc; use tokio::io::{AsyncReadExt, AsyncWriteExt}; use tokio::net::TcpStream; use tokio::time::{sleep, Instant}; @@ -25,22 +30,8 @@ fn delayed_will(client_id: &str, delay: u32) -> WillMessage { will } -async fn raw_connect( - broker: &TestBroker, - client_id: &str, - clean_start: bool, - session_expiry: u32, - will: Option, -) -> TcpStream { - let addr = broker.address().trim_start_matches("mqtt://"); +async fn connect_with_options(addr: &str, options: ConnectOptions) -> (TcpStream, bool) { let mut stream = TcpStream::connect(addr).await.expect("connect tcp"); - let options = ConnectOptions::new(client_id) - .with_clean_start(clean_start) - .with_session_expiry_interval(session_expiry); - let options = match will { - Some(will) => options.with_will(will), - None => options, - }; let mut buf = Vec::new(); ConnectPacket::new(options) .encode(&mut buf) @@ -55,14 +46,48 @@ async fn raw_connect( assert!(read >= 4, "broker closed before CONNACK"); assert_eq!(connack[0], 0x20, "expected CONNACK"); assert_eq!(connack[3], 0x00, "CONNACK reason must be Success"); - stream + (stream, connack[2] & 0x01 == 0x01) +} + +fn connect_options( + client_id: &str, + clean_start: bool, + session_expiry: Option, + will: Option, +) -> ConnectOptions { + let options = ConnectOptions::new(client_id).with_clean_start(clean_start); + let options = match session_expiry { + Some(expiry) => options.with_session_expiry_interval(expiry), + None => options, + }; + match will { + Some(will) => options.with_will(will), + None => options, + } +} + +fn broker_addr(broker: &TestBroker) -> &str { + broker.address().trim_start_matches("mqtt://") +} + +async fn raw_connect( + broker: &TestBroker, + client_id: &str, + clean_start: bool, + session_expiry: u32, + will: Option, +) -> TcpStream { + let options = connect_options(client_id, clean_start, Some(session_expiry), will); + connect_with_options(broker_addr(broker), options).await.0 +} + +async fn send_disconnect(stream: TcpStream, reason: ReasonCode) { + send_disconnect_packet(stream, DisconnectPacket::new(reason)).await; } -async fn send_disconnect(mut stream: TcpStream, reason: ReasonCode) { +async fn send_disconnect_packet(mut stream: TcpStream, disconnect: DisconnectPacket) { let mut buf = Vec::new(); - DisconnectPacket::new(reason) - .encode(&mut buf) - .expect("encode DISCONNECT"); + disconnect.encode(&mut buf).expect("encode DISCONNECT"); stream.write_all(&buf).await.expect("write DISCONNECT"); stream.flush().await.expect("flush DISCONNECT"); let mut drain = [0u8; 16]; @@ -73,12 +98,21 @@ async fn send_disconnect(mut stream: TcpStream, reason: ReasonCode) { ); } +fn disconnect_with_expiry(reason: ReasonCode, session_expiry: u32) -> DisconnectPacket { + let mut disconnect = DisconnectPacket::new(reason); + disconnect + .properties + .set_session_expiry_interval(session_expiry); + disconnect +} + async fn watch_will(broker: &TestBroker, client_id: &str) -> (MqttClient, MessageCollector) { + watch_will_at(broker.address(), client_id).await +} + +async fn watch_will_at(address: &str, client_id: &str) -> (MqttClient, MessageCollector) { let watcher = MqttClient::new(format!("{client_id}-watcher")); - watcher - .connect(broker.address()) - .await - .expect("watcher connect"); + watcher.connect(address).await.expect("watcher connect"); let collector = MessageCollector::new(); watcher .subscribe(&will_topic(client_id), collector.callback()) @@ -388,3 +422,337 @@ async fn takeover_publishes_undelayed_will() { send_disconnect(second, ReasonCode::Success).await; watcher.disconnect().await.expect("watcher disconnect"); } + +#[tokio::test] +async fn resume_with_longer_expiry_keeps_full_will_delay() { + let broker = TestBroker::start().await; + let client_id = "wd-resume-longer"; + let (watcher, collector) = watch_will(&broker, client_id).await; + + let first = raw_connect(&broker, client_id, true, 2, None).await; + send_disconnect(first, ReasonCode::Success).await; + sleep(Duration::from_millis(300)).await; + + let second = raw_connect( + &broker, + client_id, + false, + SESSION_EXPIRY, + Some(delayed_will(client_id, 5)), + ) + .await; + let dropped_at = Instant::now(); + drop(second); + + assert_no_will_until( + &collector, + dropped_at + Duration::from_millis(3500), + "the resuming CONNECT set Session Expiry 60, so the 5s Will Delay must be honoured", + ) + .await; + watcher.disconnect().await.expect("watcher disconnect"); +} + +#[tokio::test] +async fn resume_with_expiry_zero_publishes_will_immediately() { + let broker = TestBroker::start().await; + let client_id = "wd-resume-zero"; + let (watcher, collector) = watch_will(&broker, client_id).await; + + let first = raw_connect(&broker, client_id, true, SESSION_EXPIRY, None).await; + send_disconnect(first, ReasonCode::Success).await; + sleep(Duration::from_millis(200)).await; + + let second = raw_connect( + &broker, + client_id, + false, + 0, + Some(delayed_will(client_id, 5)), + ) + .await; + drop(second); + + assert!( + collector + .wait_for_messages(1, Duration::from_millis(1500)) + .await, + "the resuming CONNECT set Session Expiry 0, so the session ends at close and the Will must not wait for the delay" + ); + watcher.disconnect().await.expect("watcher disconnect"); +} + +#[tokio::test] +async fn absent_session_expiry_publishes_will_immediately() { + let broker = TestBroker::start().await; + let client_id = "wd-absent-expiry"; + let (watcher, collector) = watch_will(&broker, client_id).await; + + let options = connect_options(client_id, true, None, Some(delayed_will(client_id, 3))); + let (stream, _) = connect_with_options(broker_addr(&broker), options).await; + drop(stream); + + assert!( + collector + .wait_for_messages(1, Duration::from_millis(1500)) + .await, + "an absent Session Expiry Interval means 0: the session ends at disconnect so the Will must be published immediately" + ); + watcher.disconnect().await.expect("watcher disconnect"); +} + +#[tokio::test] +async fn absent_session_expiry_session_is_not_resumed() { + let broker = TestBroker::start().await; + let client_id = "wd-absent-resume"; + + let (first, _) = connect_with_options( + broker_addr(&broker), + connect_options(client_id, true, None, None), + ) + .await; + send_disconnect(first, ReasonCode::Success).await; + sleep(Duration::from_millis(200)).await; + + let (second, session_present) = connect_with_options( + broker_addr(&broker), + connect_options(client_id, false, Some(SESSION_EXPIRY), None), + ) + .await; + assert!( + !session_present, + "an absent Session Expiry Interval means 0, so no session survives the disconnect" + ); + send_disconnect(second, ReasonCode::Success).await; +} + +#[tokio::test] +async fn disconnect_session_expiry_zero_ends_session_and_releases_delayed_will() { + let broker = TestBroker::start().await; + let client_id = "wd-disconnect-expiry-zero"; + let (watcher, collector) = watch_will(&broker, client_id).await; + + let conn = raw_connect( + &broker, + client_id, + true, + SESSION_EXPIRY, + Some(delayed_will(client_id, 5)), + ) + .await; + send_disconnect_packet( + conn, + disconnect_with_expiry(ReasonCode::DisconnectWithWillMessage, 0), + ) + .await; + + assert!( + collector + .wait_for_messages(1, Duration::from_millis(1500)) + .await, + "Session Expiry 0 on DISCONNECT ends the session now, so the Will must not wait for the delay" + ); + + let (second, session_present) = connect_with_options( + broker_addr(&broker), + connect_options(client_id, false, Some(SESSION_EXPIRY), None), + ) + .await; + assert!( + !session_present, + "Session Expiry 0 on DISCONNECT must end the session" + ); + send_disconnect(second, ReasonCode::Success).await; + watcher.disconnect().await.expect("watcher disconnect"); +} + +#[tokio::test] +async fn disconnect_session_expiry_extends_session() { + let broker = TestBroker::start().await; + let client_id = "wd-disconnect-expiry-extend"; + + let first = raw_connect(&broker, client_id, true, 1, None).await; + send_disconnect_packet( + first, + disconnect_with_expiry(ReasonCode::Success, SESSION_EXPIRY), + ) + .await; + sleep(Duration::from_millis(2500)).await; + + let (second, session_present) = connect_with_options( + broker_addr(&broker), + connect_options(client_id, false, Some(SESSION_EXPIRY), None), + ) + .await; + assert!( + session_present, + "the DISCONNECT Session Expiry of 60s replaces the CONNECT value of 1s" + ); + send_disconnect(second, ReasonCode::Success).await; +} + +#[tokio::test] +async fn disconnect_session_expiry_after_zero_is_protocol_error() { + let broker = TestBroker::start().await; + let client_id = "wd-disconnect-expiry-error"; + let (watcher, collector) = watch_will(&broker, client_id).await; + + let mut conn = raw_connect( + &broker, + client_id, + true, + 0, + Some(delayed_will(client_id, 0)), + ) + .await; + let mut buf = Vec::new(); + disconnect_with_expiry(ReasonCode::Success, SESSION_EXPIRY) + .encode(&mut buf) + .expect("encode DISCONNECT"); + conn.write_all(&buf).await.expect("write DISCONNECT"); + + let mut reply = [0u8; 64]; + let read = tokio::time::timeout(Duration::from_secs(5), conn.read(&mut reply)) + .await + .expect("no reply to the invalid DISCONNECT") + .expect("read reply"); + assert!(read >= 3, "the server must answer with DISCONNECT"); + assert_eq!(reply[0], 0xE0, "the server must send DISCONNECT"); + assert_eq!( + reply[2], 0x82, + "a non-zero Session Expiry on DISCONNECT after CONNECT sent 0 is a Protocol Error" + ); + let mut drain = [0u8; 16]; + let closed = tokio::time::timeout(Duration::from_secs(5), conn.read(&mut drain)).await; + assert!( + matches!(closed, Ok(Ok(0))), + "the server must close the connection" + ); + + assert!( + collector + .wait_for_messages(1, Duration::from_millis(1500)) + .await, + "an invalid DISCONNECT is not a normal disconnection, so the Will is published" + ); + + let (second, session_present) = connect_with_options( + broker_addr(&broker), + connect_options(client_id, false, Some(SESSION_EXPIRY), None), + ) + .await; + assert!( + !session_present, + "the session still ends because its Session Expiry stays 0" + ); + send_disconnect(second, ReasonCode::Success).await; + watcher.disconnect().await.expect("watcher disconnect"); +} + +struct StoredBroker { + address: String, + storage: Arc, + handle: tokio::task::JoinHandle<()>, +} + +impl StoredBroker { + async fn start() -> Self { + let storage_config = StorageConfig { + backend: StorageBackend::Memory, + enable_persistence: true, + ..Default::default() + }; + let config = BrokerConfig::default() + .with_bind_address("127.0.0.1:0".parse::().expect("bind address")) + .with_storage(storage_config); + let mut broker = MqttBroker::with_config(config).await.expect("start broker"); + let address = broker.local_addr().expect("broker address").to_string(); + let storage = broker.storage().expect("persistence is enabled"); + let handle = tokio::spawn(async move { + if let Err(e) = broker.run().await { + tracing::debug!("broker stopped: {e}"); + } + }); + sleep(Duration::from_millis(100)).await; + Self { + address, + storage, + handle, + } + } + + async fn stored_will_present(&self, client_id: &str) -> bool { + self.storage + .get_session(client_id) + .await + .expect("read session") + .expect("the session outlives the connection") + .will_message + .is_some() + } +} + +impl Drop for StoredBroker { + fn drop(&mut self) { + self.handle.abort(); + } +} + +#[tokio::test] +async fn published_will_is_removed_from_stored_session() { + let broker = StoredBroker::start().await; + let client_id = "wd-stored-published"; + let (watcher, collector) = + watch_will_at(&format!("mqtt://{}", broker.address), client_id).await; + + let (conn, _) = connect_with_options( + &broker.address, + connect_options( + client_id, + true, + Some(SESSION_EXPIRY), + Some(delayed_will(client_id, 0)), + ), + ) + .await; + assert!(broker.stored_will_present(client_id).await); + drop(conn); + + assert!( + collector + .wait_for_messages(1, Duration::from_millis(1500)) + .await, + "the Will is published at disconnect" + ); + sleep(Duration::from_millis(100)).await; + assert!( + !broker.stored_will_present(client_id).await, + "a published Will must be removed from the stored session" + ); + watcher.disconnect().await.expect("watcher disconnect"); +} + +#[tokio::test] +async fn normal_disconnect_removes_stored_will() { + let broker = StoredBroker::start().await; + let client_id = "wd-stored-normal"; + + let (conn, _) = connect_with_options( + &broker.address, + connect_options( + client_id, + true, + Some(SESSION_EXPIRY), + Some(delayed_will(client_id, 0)), + ), + ) + .await; + assert!(broker.stored_will_present(client_id).await); + send_disconnect(conn, ReasonCode::Success).await; + sleep(Duration::from_millis(200)).await; + + assert!( + !broker.stored_will_present(client_id).await, + "DISCONNECT 0x00 must remove the Will from the stored session" + ); +} diff --git a/crates/mqttv5-cli/CLI_USAGE.md b/crates/mqttv5-cli/CLI_USAGE.md index 608e4c43..2cc13828 100644 --- a/crates/mqttv5-cli/CLI_USAGE.md +++ b/crates/mqttv5-cli/CLI_USAGE.md @@ -125,7 +125,7 @@ mqttv5 broker generate-config [--output FILE] [--format json|toml] | `--storage-dir ` | Storage directory for persistence | `./mqtt_storage` | | `--storage-backend ` | Storage backend: `memory` or `file` | `file` | | `--no-persistence` | Disable message persistence | `false` | -| `--session-expiry ` | Default session expiry interval in seconds | `3600` | +| `--session-expiry ` | Optional maximum session expiry interval granted to clients; a client asking for more, or an MQTT 3.1.1 persistent session, gets this value. MQTT 5 clients are told the granted value in CONNACK; an MQTT 3.1.1 CONNACK carries no properties | no limit | | `--max-qos <0\|1\|2>` | Maximum QoS level | `2` | | `--keep-alive ` | Server keep-alive time in seconds | None | | `--response-information ` | Response information sent to clients that request it | None | @@ -135,6 +135,10 @@ mqttv5 broker generate-config [--output FILE] [--format json|toml] | `--sys-interval ` | `$SYS` topic publish interval (e.g., `10`, `10s`, `1m`) | `10` | | `--non-interactive` | Skip interactive prompts | `false` | +##### Upgrading the file storage + +mqttv5 0.29.0 stores sessions in a new format (storage version 2). On its first start it migrates a version 1 storage directory in place, and earlier mqttv5 versions refuse to open the migrated directory. Back up the storage directory (`--storage-dir`, default `./mqtt_storage`) with the broker stopped before upgrading. To roll back, stop the broker, restore that backup, and start the earlier version. + ##### Broker JWT Auth Flags | Flag | Description | Default | @@ -377,8 +381,8 @@ Publish an MQTT message to a broker. Supports all transport types (TCP, TLS, Web | `--auth-method ` | Authentication method: `password`, `scram`, `jwt` | `password` | | `--jwt-token ` | JWT token for JWT authentication | None | | `--client-id, -c ` | Client ID | Auto-generated | -| `--no-clean-start` | Resume existing session | `false` | -| `--session-expiry ` | Session expiry interval in seconds | `0` | +| `--no-clean-start` | Resume existing session; unless `--session-expiry` is given, a session expiry of 1 hour is requested so the session also survives this run | `false` | +| `--session-expiry ` | Session expiry interval in seconds (0 = the session ends at disconnect) | `0`, or `3600` with `--no-clean-start` | | `--keep-alive, -k ` | Keep-alive interval | `60` | | `--protocol-version ` | MQTT protocol version: `3.1.1`, `311`, `4`, `5`, `5.0` | `5` | | `--will-topic ` | Will message topic | None | @@ -579,8 +583,8 @@ Subscribe to one or more MQTT topics and print received messages. The subscriber | `--auth-method ` | Authentication method: `password`, `scram`, `jwt` | `password` | | `--jwt-token ` | JWT token for JWT authentication | None | | `--client-id, -c ` | Client ID | Auto-generated | -| `--no-clean-start` | Resume existing session | `false` | -| `--session-expiry ` | Session expiry interval in seconds | `0` | +| `--no-clean-start` | Resume existing session; unless `--session-expiry` is given, a session expiry of 1 hour is requested so the session also survives this run | `false` | +| `--session-expiry ` | Session expiry interval in seconds (0 = the session ends at disconnect) | `0`, or `3600` with `--no-clean-start` | | `--keep-alive, -k ` | Keep-alive interval | `60` | | `--protocol-version ` | MQTT protocol version: `3.1.1`, `311`, `4`, `5`, `5.0` | `5` | | `--will-topic ` | Will message topic | None | @@ -1111,7 +1115,7 @@ Every field is optional. Any field omitted from the file falls back to the defau | --- | --- | --- | --- | | `bind_addresses` | `string[]` | TCP listener addresses | `["0.0.0.0:1883", "[::]:1883"]` | | `max_clients` | `number` | Maximum concurrent client connections | `10000` | -| `session_expiry_interval` | `duration` | Default session expiry for clients | `"1h"` | +| `session_expiry_interval` | `duration` | Optional maximum session expiry granted to clients; larger requests, and MQTT 3.1.1 persistent sessions, are capped to it. MQTT 5 clients are told the granted value in CONNACK; an MQTT 3.1.1 CONNACK carries no properties. Omitted by `generate-config` | no limit | | `max_packet_size` | `number` | Maximum MQTT packet size in bytes | `268435456` (256 MB) | | `topic_alias_maximum` | `number` | Maximum number of topic aliases | `65535` | | `retain_available` | `boolean` | Enable retained messages | `true` | diff --git a/crates/mqttv5-cli/Cargo.toml b/crates/mqttv5-cli/Cargo.toml index 66b8161f..34c1ea91 100644 --- a/crates/mqttv5-cli/Cargo.toml +++ b/crates/mqttv5-cli/Cargo.toml @@ -1,6 +1,6 @@ [package] name = "mqttv5-cli" -version = "0.28.8" +version = "0.29.0" edition.workspace = true rust-version.workspace = true authors.workspace = true @@ -22,7 +22,7 @@ opentelemetry = ["mqtt5/opentelemetry"] codec = ["mqtt5/codec-all"] [dependencies] -mqtt5 = { path = "../mqtt5", version = "0.41" } +mqtt5 = { path = "../mqtt5", version = "0.42" } anyhow = "1.0.103" tracing = "0.1" serde = { version = "1.0", features = ["derive"] } diff --git a/crates/mqttv5-cli/src/commands/broker_cmd.rs b/crates/mqttv5-cli/src/commands/broker_cmd.rs index 77a040d4..29d09d3b 100644 --- a/crates/mqttv5-cli/src/commands/broker_cmd.rs +++ b/crates/mqttv5-cli/src/commands/broker_cmd.rs @@ -207,9 +207,10 @@ pub struct RunArgs { #[arg(long, env = "MQTT5_NO_PERSISTENCE")] pub no_persistence: bool, - /// Session expiry interval (e.g., 3600, 1h, 30m) - #[arg(long, default_value = "3600", value_parser = parse_duration_secs, env = "MQTT5_SESSION_EXPIRY")] - pub session_expiry: u64, + /// Maximum session expiry interval granted to clients (e.g., 3600, 1h, 30m); a client + /// asking for more, or an MQTT 3.1.1 persistent session, gets this value (default: no limit) + #[arg(long, value_parser = parse_duration_secs, env = "MQTT5_SESSION_EXPIRY")] + pub session_expiry: Option, /// Maximum `QoS` level supported (0, 1, or 2) #[arg(long, default_value = "2", env = "MQTT5_MAX_QOS")] @@ -279,7 +280,7 @@ fn build_example_config() -> BrokerConfig { "[::]:1883".parse().unwrap(), ], max_clients: 10000, - session_expiry_interval: std::time::Duration::from_secs(3600), + session_expiry_interval: BrokerConfig::default().session_expiry_interval, max_packet_size: 268_435_456, topic_alias_maximum: 65535, retain_available: true, @@ -1037,7 +1038,9 @@ async fn create_interactive_config(cmd: &mut RunArgs) -> Result { config = config.with_bind_addresses(bind_addrs?); config = config.with_max_clients(cmd.max_clients); - config.session_expiry_interval = std::time::Duration::from_secs(cmd.session_expiry); + if let Some(maximum) = cmd.session_expiry { + config.session_expiry_interval = std::time::Duration::from_secs(maximum); + } config.maximum_qos = cmd.max_qos; config.retain_available = !cmd.no_retain; config.wildcard_subscription_available = !cmd.no_wildcards; diff --git a/crates/mqttv5-cli/src/commands/client_args.rs b/crates/mqttv5-cli/src/commands/client_args.rs index 01d1539c..4ef2a1d5 100644 --- a/crates/mqttv5-cli/src/commands/client_args.rs +++ b/crates/mqttv5-cli/src/commands/client_args.rs @@ -23,11 +23,13 @@ pub fn parse_protocol_version(s: &str) -> Result { #[derive(Args)] pub struct SessionArgs { - /// Don't clean start (resume existing session) + /// Don't clean start (resume existing session); the session is kept for 1h after + /// disconnect unless --session-expiry says otherwise #[arg(long = "no-clean-start", env = "MQTT5_NO_CLEAN_START")] pub no_clean_start: bool, - /// Session expiry interval (e.g., 1h, 30m) (0 = expire on disconnect) + /// Session expiry interval (e.g., 1h, 30m) (0 = expire on disconnect; default: 0, or 1h + /// with --no-clean-start) #[arg(long, value_parser = parse_duration_secs, env = "MQTT5_SESSION_EXPIRY")] pub session_expiry: Option, @@ -40,6 +42,16 @@ pub struct SessionArgs { pub protocol_version: Option, } +pub const RESUMED_SESSION_EXPIRY_SECS: u64 = 3600; + +impl SessionArgs { + #[must_use] + pub fn session_expiry_secs(&self) -> Option { + self.session_expiry + .or(self.no_clean_start.then_some(RESUMED_SESSION_EXPIRY_SECS)) + } +} + #[derive(Args)] pub struct WillArgs { /// Will topic (last will and testament) @@ -135,3 +147,36 @@ pub struct QuicArgs { )] pub early_data: bool, } + +#[cfg(test)] +mod tests { + use super::{SessionArgs, RESUMED_SESSION_EXPIRY_SECS}; + + fn session(no_clean_start: bool, session_expiry: Option) -> SessionArgs { + SessionArgs { + no_clean_start, + session_expiry, + keep_alive: 60, + protocol_version: None, + } + } + + #[test] + fn resuming_without_expiry_keeps_the_session() { + assert_eq!( + session(true, None).session_expiry_secs(), + Some(RESUMED_SESSION_EXPIRY_SECS) + ); + } + + #[test] + fn explicit_expiry_wins() { + assert_eq!(session(true, Some(0)).session_expiry_secs(), Some(0)); + assert_eq!(session(false, Some(30)).session_expiry_secs(), Some(30)); + } + + #[test] + fn clean_start_without_expiry_sends_none() { + assert_eq!(session(false, None).session_expiry_secs(), None); + } +} diff --git a/crates/mqttv5-cli/src/commands/pub_cmd.rs b/crates/mqttv5-cli/src/commands/pub_cmd.rs index 56d66df9..4f199217 100644 --- a/crates/mqttv5-cli/src/commands/pub_cmd.rs +++ b/crates/mqttv5-cli/src/commands/pub_cmd.rs @@ -292,7 +292,7 @@ fn build_connect_options(cmd: &PubCommand, client_id: &str) -> ConnectOptions { options = options.with_protocol_version(version); } - if let Some(expiry) = cmd.session.session_expiry { + if let Some(expiry) = cmd.session.session_expiry_secs() { options = options.with_session_expiry_interval(duration_secs_to_u32(expiry)); } diff --git a/crates/mqttv5-cli/src/commands/sub_cmd.rs b/crates/mqttv5-cli/src/commands/sub_cmd.rs index 956c37ee..8ab966e1 100644 --- a/crates/mqttv5-cli/src/commands/sub_cmd.rs +++ b/crates/mqttv5-cli/src/commands/sub_cmd.rs @@ -201,7 +201,7 @@ fn build_connect_options(cmd: &SubCommand, client_id: &str) -> ConnectOptions { options = options.with_protocol_version(version); } - if let Some(expiry) = cmd.session.session_expiry { + if let Some(expiry) = cmd.session.session_expiry_secs() { options = options.with_session_expiry_interval(duration_secs_to_u32(expiry)); } diff --git a/specs/tla/session-ownership/README.md b/specs/tla/session-ownership/README.md new file mode 100644 index 00000000..26cdd20d --- /dev/null +++ b/specs/tla/session-ownership/README.md @@ -0,0 +1,506 @@ +# Session ownership: TLA+ model + +Model of how the `mqtt5` broker decides who owns a ClientID's session, and how it keeps the +router (live routes), the session store (cache plus file) and the MQTT-visible session state +(Session Present, Clean Start, Session Expiry, acknowledged subscriptions) in agreement while +connections race, handshakes fail, sweeps run and the broker crashes. + +It consolidates three independent models that reached the same verdict (see "Provenance"). The +chosen design is `MECH = "LOCK"`. Today's code is kept as `MECH = "CURRENT"`, the rejected +lock-free alternatives as `MECH = "CAS"` and `MECH = "CASSPLIT"`, and every element of the chosen +design can be switched off individually with `DROP` so that the spec shows why it is needed. + +## Files + +- `SessionOwnership.tla`: the spec. One module contains all four mechanisms. +- `SessionOwnership.cfg`: the chosen design, main safety run (3 connections, 2 filters, 1 restart). +- `SessionOwnership_boots2.cfg`: the chosen design with 2 restarts. +- `SessionOwnership_live.cfg`: liveness of the chosen design. +- `SessionOwnership_NEG_*.cfg`: negative controls. Each must fail. +- `SessionOwnership_ABL_.cfg`: the chosen design with one element dropped. Each must fail. + `_ABL__core.cfg` repeats an ablation without the property that fails first, to show the + deeper failure behind it. `_ABL_wt_claim_resurrect.cfg` checks only `InvNoResurrection`. +- `SessionOwnership_CUR_.cfg`: today's code, one cfg per property it breaks. +- `SessionOwnership_CAS_*.cfg`, `SessionOwnership_CASSPLIT_*.cfg`: the rejected lock-free designs. +- `SessionOwnership_GC_*.cfg`: the chosen design with group commit (`GC` other than `"OFF"`), see + "Group commit". `_GC_whole`, `_GC_prefix`, `_GC_prefix_boots2`, `_GC_prefix_live` and + `_GC_perwriter_mono` must pass; `_GC_ABL_*` are the group-commit ablations and must fail. + +Every cfg sets `GC` and `MaxPend`. All cfgs other than `_GC_*` use `GC = "OFF"` (every durable +write reaches the file in the step that makes it, as before group commit), and `MaxPend` is then +unused. + +## The chosen design + +All session state for one ClientID is guarded by one per-ClientID lock (the session slot). Every +step below runs as one critical section under that lock, so no other step for the same ClientID +can interleave with it. + +1. **Claim before CONNACK** (`LClaim`). In one critical section the new connection: + - reads the stored session (an expired one counts as absent and is removed), + - decides resume or discard: resume only if Clean Start = 0 and a live session with + Session Expiry > 0 exists; otherwise start empty [MQTT-3.1.2-4], + - writes the new session record durably, with a fresh connection token, + - registers itself as the router owner (displacing any previous owner), + - sets the router's routes for the ClientID to exactly the claimed session's subscriptions + (not a union with whatever the router held). + + CONNACK is sent only after the claim. Session Present is the claim's resume decision. + The model takes `LClaim` as one atomic step, so the order of these effects inside the + critical section is invisible to it: no other step for the ClientID can observe the state + between them. The code writes before it registers so that a claim whose write fails changes + nothing and the live owner stays registered; write failures are not modelled (see "Not + modelled") and are covered by tests. +2. **Owner-only SUBSCRIBE / UNSUBSCRIBE** (`LSub`). Under the lock, and only if this connection + is still the owner, the router routes and the stored subscriptions change together, and the + store write is durable before SUBACK / UNSUBACK. The model changes one filter per step. The + code applies every filter of one packet in one critical section and stores them in one + write; that is a run of consecutive `LSub` steps that nothing can interleave with (the lock + is held throughout), whose writes are merged into one record because a write replaces the + whole record, and whose acknowledgement waits for that single write. +3. **Owner-only DISCONNECT expiry update, written at the DISCONNECT** (`Close` with `disc`). + When the DISCONNECT packet is handled, in its own critical section under the lock and before + the release step, a DISCONNECT that changes Session Expiry writes the new value to storage if + the connection still owns the stored session. The release (step 4) then writes the disconnect + time or removes the session. Deferring the new expiry to the release is `DROP = {"disc"}` and + fails (`_ABL_disc`). +4. **Owner-only release, strip under the lock** (`LRelease`). When a connection ends it releases + ownership only if it is still the owner. In the same critical section it either removes the + session and strips its routes (expiry 0) or marks the session disconnected with the + disconnect time (expiry > 0). A displaced connection's release touches nothing. +5. **Sweeps only without an owner, atomic re-check and remove** (`LSweep`). A sweep takes the + lock, re-checks that there is no owner and that the session is expired (or absent while + routes remain), and removes the session and its routes in the same critical section. +6. **Persisted disconnect timestamp, boot stamping, router rebuild** (`stamp`, `rebuild`, + `Restart`). The disconnect time is stored with the session. At startup any session still + marked connected is stamped as disconnected at boot time, and the router's routes are + rebuilt from the stored sessions that have Session Expiry > 0. +7. **Every session write is durable** (`wt_claim`, `wt_sub`, `wt_disc`, `wt_remove`): the claim + (including the clean-start discard, which replaces the record), subscription changes, the + disconnect/expiry update and every removal (expiry-0 release, clean-start discard, sweeps). + With `GC = "OFF"` each write is durable in the step that makes it. With group commit the + same writes go through a pending buffer and every acknowledgement waits for the flush (see + "Group commit"). + +### Why a lock and not compare-and-swap + +The lock-free alternatives keep the router and the store as two separately updated objects and +fence stale writers with connection tokens: + +- `CAS`: the hardened version the modelers converged on. The storage claim allocates a token, + the router registration is fenced by a high-water mark, releases and sweeps are token- and + generation-fenced, and a displaced connection cleans up only what it still owns. With all of + that it satisfies every property except two: the router owner and the storage owner disagree + between the two steps of a claim (`InvOwnershipAgree`), and the router is not an exact mirror + of the session at every instant (`InvRouterMirrors`). Every consumer that reads router and + store together would have to tolerate that window. +- `CASSPLIT`: register at the router first (Clean Start strips the routes there), then claim + in storage with a token compare. A Clean Start that registered earlier can be undone by a + later connection's storage claim that resumes the pre-clean-start session: the discarded + session comes back (`InvNoResurrection`). This is the resurrection counterexample (see below). + +The lock gives the stronger properties with fewer moving parts: no tombstones, no router +high-water mark, no generation-fenced sweeps. + +## Model + +- **One ClientID.** `Conns` are successive connection attempts for the same ClientID. Each + attempt goes `idle`, claim, `connack`, `live`, `release`, `done`. A crash turns every attempt + in progress into `dead`. An attempt that never started stays `idle` and may start after the + crash. +- **Router.** `owner` (the registered connection) and `rsubs` (routes for the ClientID). +- **Session store.** `cache` (the in-memory record), `disk` (the file) and `dirty` (the cache + holds a write, or a removal, not yet on disk). `View` is what a reader sees: the cache when it + holds a record or a pending write, otherwise the file. A write is either write-through + (durable at once) or write-behind (flushed later by `Flush`). A record carries the connection + token `tok`, the subscriptions, the Session Expiry (`exp`: 0 or 1, meaning 0 or `E` ticks), + whether it is connected, `seen` (the disconnect time the expiry counts from), and two ghost + fields: `gdt` (the true disconnect time) and `born` (which fresh start created the session). +- **Time and crashes.** `now` advances to `MaxT`. `Restart` (up to `MaxBoots` times) loses the + cache, the router and every connection in progress; the file survives. +- **Ghost state.** `gEx`, `gEnd`, `gExp`, `gOwner`, `gSubs` track the logical session as MQTT + defines it: whether it exists, when it ended, its expiry, and the subscriptions acknowledged + to its current owner. `cleanEff` is the set of `born` values of every fresh start that took + effect. Per connection, `spOK`, `linOK`, `subOK` and `abortOK` record whether the claim's + decisions were correct. +- **Mechanisms.** `MECH` selects the mechanism and `DROP` removes elements of the chosen + design. Dropping an element reverts that element to today's behaviour: + +| Element (`DROP` name) | Chosen design | Behaviour when dropped (today's code) | +|---|---|---| +| `claim` | `LClaim`: one critical section before CONNACK | `HSRead`, `HSStore`, CONNACK, `Register`, `Install`: read, then write, then CONNACK, then router register, then route union, all separate | +| `sub` | `LSub`: per filter, owner-only, router and store together | `SubRouter` then `SubStore`: router step, then a whole-record store overwrite with the connection's own token, not owner-checked | +| `disc` | `Close` writes a changed Session Expiry owner-only | the new expiry reaches the store only at release | +| `release` | `LRelease`: owner-only, strip and store update together | `Release`, `Strip`, `Cleanup`, `RmOwned`: router release, route strip, store update and store removal as separate steps | +| `sweep` | `LSweep`: owner-less, re-check and remove together | `FileSweepRead`/`FileSweepApply` and `RouterSweepDecide`/`RouterSweepApply`: decide, then apply without re-checking | +| `stamp` | disconnect time persisted, connected records stamped at boot | the file has no disconnect time; a loaded record's expiry counts from the time it is read | +| `rebuild` | router routes rebuilt from the file at restart | router starts empty after restart | +| `wt_claim`, `wt_sub`, `wt_disc` | durable writes | write-behind (`store_session` caches and marks dirty) | +| `wt_remove` | durable removal | write-behind removal (a pending removal is lost in a crash) | + +`MECH = "CURRENT"` is every element dropped except `wt_remove`, because today's +`remove_session` / `remove_owned_session` already delete the file at once. + +## Properties in plain words + +All invariants below are checked in every run of the chosen design. + +| Property | Meaning | +|---|---| +| `TypeOK` | Variables stay within their bounded types. | +| `InvOwnershipAgree` | The router owner and the connection whose token the stored session carries are the same connection whenever both are running. | +| `InvNoLeakedConnected` | A stored session marked connected always belongs to a running connection. No session stays "connected" after its connection is gone. | +| `InvLiveOwnerSessionKept` | While a connection is the live owner, the stored session exists, is marked connected and carries its token (or a newer one). Nothing removes or disconnects the live owner's session under it. | +| `InvFailedHandshakeHarmless` | A connection whose handshake failed leaves the current live owner's session intact and never leaves its own token in it. | +| `InvSessionPresent` | Session Present is correct against the logical session: SP = 1 only if a session with expiry > 0 is alive (not ended, or ended less than its expiry ago), and SP = 0 on Clean Start = 0 only if no such session is alive. (SP = 0 on Clean Start = 1 holds by construction: no claim resumes when Clean Start = 1.) | +| `InvNoResurrection` | A resumed session was not discarded or superseded by any fresh start that took effect before the resume. This covers Clean Start = 1 discards [MQTT-3.1.2-4] and holds across crashes. It is the linearizability check for claims. | +| `InvAckedSubsDurable` | A resumed session carries exactly the subscriptions acknowledged (SUBACK / UNSUBACK) to the previous owner, including across a crash. | +| `InvAbortJustified` | A claim that aborts does so only because a newer claim exists (only the lock-free mechanisms can abort a claim; trivially true for LOCK and CURRENT). | +| `InvCleanStart` | A live owner that started empty has no routes and no stored subscriptions other than the ones it subscribed itself [MQTT-3.1.2-4]. | +| `InvRouterMirrors` | At every instant the router's routes equal the owner's subscriptions if there is an owner, otherwise the stored session's subscriptions if it has expiry > 0, otherwise nothing. | +| `InvQuiescentConsistent` | When no step is in flight, a live owner's routes, local subscriptions and stored subscriptions are equal. | +| `InvExpiryExact` | A stored disconnected session's expiry clock starts at its true disconnect time, including after a crash. | +| `InvProgress` | When time is exhausted and no broker step is enabled, no ended session (disconnected past its expiry, or marked connected with no running connection) and no owner-less stray route remains. This progress check is based on `ENABLED` and backs up the liveness result. | + +Liveness is checked under weak fairness of broker-side connection steps, sweeps, flushes and +time (`Spec`). Client actions (connect, subscribe, disconnect) and failures have no fairness. + +| Property | Meaning | +|---|---| +| `EventuallyClean == []<>(now < MaxT \/ ~Stale)` | Once time stops advancing, every ended session and every stray route is eventually cleaned up. | + +Negative controls. Each must fail. + +| Control | Why it must fail | +|---|---| +| `EventuallyClean` under `SpecNoSweepFair` | Without fairness on sweeps an expired session may never be removed. | +| `NegTimeNeverEnds == []<>(now < MaxT)` | Time reaches `MaxT` and stays there. Shows the checker reports `[]<>` violations under `Spec`. | +| `NegNoResumeAfterRestart` (invariant) | A connection that crashed with subscriptions is resumed, with those subscriptions, after the restart. Shows the chosen design reaches crash recovery. | +| `NegNoResumingTakeover` (invariant) | A new connection takes over a live session and resumes it with its subscriptions. Shows takeover is reached. | + +## Reductions and bounds + +- **Symmetry** over `Conns` (`SYMMETRY Sym`) in every safety run. Connection attempts are + interchangeable: `Init` is symmetric, no action or property names a particular connection, + and tokens are drawn from a counter rather than from connection identities. Liveness runs do + not use symmetry. +- **Filters.** The chosen design's main runs use two filters. The ablation, `CURRENT`, `CAS`, + `CASSPLIT` and liveness runs use one filter. Actions and properties treat filters pointwise + (add or remove one filter, set equality and inclusion), except the "routes non-empty" guard of + sweeps, so one filter is enough to exhibit each failure; a counterexample at one filter is a + real counterexample. The passing runs of the chosen design do not rely on this argument, + because they also pass at two filters. +- **Time.** `E = 1` (expiry 1 means one tick) and `MaxT` = 2 or 3. `Tick` is disabled while any + connection is in a release step. This models the disconnect time and the end of the logical + session as the same instant: the DISCONNECT and the release write the same timestamp. +- **Crashes.** `MaxBoots` = 1 (2 in `SessionOwnership_boots2.cfg`). Connection tokens are not + reset by a crash. +- **Expiry values.** Session Expiry is 0 or 1 (never "never expires"). Any positive value + behaves like 1 relative to the modelled time bound. + +## Group commit + +Making every session write durable with its own fsync under a global lock costs about two orders +of magnitude of throughput. Group commit keeps the guarantee (nothing is acknowledged before it +is durable) and makes concurrent writes durable together in one flush. The constant `GC` selects +it; `GC = "OFF"` is the model without group commit. + +### How it is modelled + +- **Pending buffer, read-your-writes** (`pbuf`, `Log`). Every session write of the chosen design + (claim, SUBSCRIBE / UNSUBSCRIBE, DISCONNECT expiry, release, sweep removal) still runs under the + per-ClientID lock. It updates the in-memory record (`cache`) at once and appends the new record + to the ordered pending buffer. Every reader under the lock sees the latest pending state + (`View` is the cache while anything is pending), so a claim reads a racing clean start that + is not yet on disk. +- **Flush** (`GFlushPrefix(k)`). A flush makes the first `k` pending entries durable: the file + becomes the record of entry `k` (a write replaces the ClientID's whole record, so the last entry + of the prefix is the file content). `GC = "WHOLE"` flushes only the whole buffer. + `GC = "PREFIX"` flushes any prefix, which is what a batch does when it snapshots the buffer + while later writes keep arriving. +- **Acknowledgements wait for the flush** (`AckReady`). `wpos[c]` is the position of connection + `c`'s latest write in the buffer, 0 once it has been flushed. CONNACK with Session Present + (`ConnackOK`), SUBACK / UNSUBACK (`SubAck`: `LSub` now ends in `suback`) and the end of + DISCONNECT processing (`RelDone`: `LRelease` now ends in `relwait`) are enabled only when + `wpos[c] = 0`, that is, when a flush has covered that write and every earlier pending write. + A claim that read another connection's unflushed write therefore cannot acknowledge before + that write is durable. +- **Crash** (`Restart`) loses the whole pending buffer. The file keeps what was flushed. +- **Ghost rollback.** The logical session changes at the write (so a claim that reads a pending + write is checked against it). Each entry records the logical session after its write, and `dg` + is the logical session of the file content. At a crash the logical session rolls back to `dg`: + the lost writes never happened, which is legitimate only because none of them was + acknowledged. If an acknowledged write is lost (possible only in the ablations), the rollback + stops at the newest acknowledged entry instead, so the logical session after the crash keeps + everything a client was told, and the existing invariants then compare the recovered store + with it. The end of a connection is a real event and applies to every history (`EndG`): the + session ended at that time even if the write that recorded it is lost, but a new Session + Expiry sent in the DISCONNECT is part of the DISCONNECT's write and is lost with it. +- **Time.** `Tick` is disabled while any write is unflushed: a flush completes within one unit + of Session Expiry time (flushes take milliseconds, expiry is counted in seconds). This extends + the existing "release and disconnect happen at the same instant" abstraction. +- **Bound.** `MaxPend` bounds the buffer; a write blocks while it is full. Every passing + group-commit run uses `MaxPend = 8` and checks `InvPendNotFull`, so the bound never blocked a + write there and the buffer's size comes only from the protocol. + +Added properties: + +| Property | Meaning | +|---|---| +| `InvAckedStateSurvivesCrash` | No crash loses an acknowledged write: at every `Restart`, no acknowledged entry is newer than the file content. The semantic consequences are checked by the existing invariants against the post-crash logical session: a lost claim or wrong Session Present (`InvSessionPresent`), a lost SUBACKed subscription (`InvAckedSubsDurable`), a lost clean-start discard (`InvNoResurrection`), a lost disconnect or expiry (`InvSessionPresent`, `InvExpiryExact`). The `_core` ablation runs leave this property out to show those consequences. With `WHOLE` and `PREFIX` it holds by construction of `AckReady`; it is the direct detector in the ablations. | +| `InvPendNotFull` | The pending buffer never reaches `MaxPend` (the bound is not a restriction). | +| `InvAckedClaimCorrect` (diagnostic) | Every connection that got its CONNACK had a correct Session Present, resume and subscription set. Used to show that a wrong decision in `NORYW` reaches the client. | + +Group-commit variants (`GC`): + +| `GC` | Flush | Ack rule | Reads | Must | +|---|---|---|---|---| +| `WHOLE` | whole buffer | after a flush covering the write and every earlier write | latest pending state | pass | +| `PREFIX` | any prefix | same | latest pending state | pass | +| `PERWRITER_MONO` | each entry on its own, any order; writing an entry drops every older pending entry, which is then never written (the newer record already carries it) | after the connection's own entry is flushed or dropped as covered | latest pending state | pass | +| `PERWRITER` (ablation a) | each entry on its own, any order; the file gets whatever was flushed last | after the connection's own entry is flushed, earlier entries ignored | latest pending state | fail | +| `ACKEARLY` (ablation b) | any prefix | no wait | latest pending state | fail | +| `NORYW` (ablation c) | any prefix | same as `PREFIX` | the file only | fail | + +### Ordering rule for the implementation + +1. Under the per-ClientID lock, a session write updates the in-memory session (so every later + reader under the lock sees it) and is given the next sequence number in lock order. +2. A flush writes and fsyncs every pending write up to some sequence number `N` and only then + publishes "durable through `N`". The file for a ClientID never goes back to an older version: + a record is never written over a newer one. +3. CONNACK, SUBACK / UNSUBACK and the completion of DISCONNECT processing for a write with + sequence number `s` wait until "durable through" is at least `s`. Because `s` was assigned + after everything the connection read under the lock, this also covers every write the + decision depended on (for example a racing clean start). A step that acknowledges after a + pure read would have to wait for the sequence number of the newest write it could see (not + exercised by the model: every acknowledgement here follows a write of its own). +4. Reads under the lock use the in-memory state, never only the file. + +Within one ClientID a write replaces the whole record, so the record of write `s` already +carries every earlier write it read (a claim that resumed after a racing clean start stores the +clean start's result). Flushing only the connection's own write is therefore not unsafe by +itself: `PERWRITER_MONO` passes, and it is the same thing as a prefix flush, since writing entry +`s` with no regression makes everything up to `s` durable. What fails in `PERWRITER` is the +missing no-regression half of rule 2: an earlier write still in flight lands after the later, +acknowledged one and overwrites it. Waiting for "durable through `s`" over a prefix (rule 3) +rules this out by construction, because nothing up to `s` is still in flight when the +acknowledgement is sent. The model has one ClientID; under the "Not modelled" assumption that +different ClientIDs never read each other's records, no order between ClientIDs is needed. + +## Runs + +Every run was executed with TLC. Every passing run explored its full reachable state space and +every failing run stopped at a counterexample; no run hit a limit or was cut short. Command: + +``` +java -XX:+UseParallelGC -Xmx6g -cp tla2tools.jar tlc2.TLC -workers 4 \ + -metadir /states/ -config SessionOwnership.tla +``` + +Common constants: `E = 1`, `None = none`. "Conns" is the number of connection attempts. Every +run except the `_GC_*` runs uses `GC = "OFF"`, `MaxPend = 1`. All runs below were repeated +against the group-commit spec: every passing run has the same distinct-state count as before +group commit was added, and every failing run fails on the same property except +`_ABL_wt_disc` (see the note under the negative controls). + +### Chosen design (`MECH = "LOCK"`, `DROP = {}`) + +| cfg | Conns | Filters | MaxT | MaxBoots | Reductions | Checks | Result | Distinct states | Depth | Time | +|---|---|---|---|---|---|---|---|---|---|---| +| `SessionOwnership.cfg` | 3 | 2 | 3 | 1 | symmetry | all 14 invariants | pass | 4,138,304 | 24 | 91 s | +| `SessionOwnership_boots2.cfg` | 3 | 2 | 3 | 2 | symmetry | all 14 invariants | pass | 7,649,114 | 25 | 176 s | +| `SessionOwnership_live.cfg` | 3 | 1 | 2 | 1 | none | `EventuallyClean` under `Spec` | pass | 2,654,334 | 20 | 145 s | + +### Negative controls + +| cfg | Conns | Filters | MaxT | MaxBoots | Reductions | Expected | Result | Distinct states | Depth | Time | Counterexample | +|---|---|---|---|---|---|---|---|---|---|---|---| +| `SessionOwnership_NEG_nosweepfair.cfg` | 3 | 1 | 2 | 1 | none | liveness failure | `EventuallyClean` violated | 81,717 | 10 | 3 s | c3's expiry-0 session is still connected when the broker crashes. It is stamped as ended at boot, expires one tick later, and without sweep fairness it is never removed. | +| `SessionOwnership_NEG_timeneverends.cfg` | 3 | 1 | 2 | 1 | none | liveness failure | `NegTimeNeverEnds` violated | 85,406 | 10 | 3 s | Time reaches `MaxT`. | +| `SessionOwnership_NEG_resumeafterrestart.cfg` | 3 | 1 | 2 | 1 | symmetry | violation | `NegNoResumeAfterRestart` violated | 3,824 | 10 | 1 s | c1 claims with expiry 1 and subscribes, the broker crashes, c2 claims and resumes c1's session with its subscription. | +| `SessionOwnership_NEG_resumingtakeover.cfg` | 3 | 1 | 2 | 1 | symmetry | violation | `NegNoResumingTakeover` violated | 1,241 | 9 | 1 s | c2 is live with a subscription, c3 takes over and resumes it. | + +For a failing run, the state count is what TLC had explored when it found the violation. With 4 +workers it varies slightly between runs, and when two counterexamples have the same length the +one reported first can differ. For example, `_ABL_wt_disc` has reported both `InvSessionPresent` and `InvExpiryExact` (the +final run reports `InvExpiryExact`). + +### Ablations of the chosen design (`MECH = "LOCK"`, 3 Conns, 1 filter, `MaxT = 2`, `MaxBoots = 1`, symmetry) + +Each ablation checks all 14 invariants (unless noted). BFS gives the shortest counterexample, so +"first violated" is the property that breaks earliest. + +| cfg (`DROP`) | Expected | First violated | Distinct states | Depth | Time | Counterexample | +|---|---|---|---|---|---|---| +| `_ABL_claim` (`claim`) | Session Present or ownership | `InvSessionPresent` | 62 | 7 | 0 s | c1 writes its session but has not registered or sent CONNACK. c3 reads it and resumes (SP = 1) a session with expiry 0 that belongs to a connection still in its handshake. Without the claim, "who owns the session" and "what did the session look like" are decided at different moments. | +| `_ABL_sub` (`sub`) | router mirror | `InvRouterMirrors` | 101 | 7 | 1 s | The router route is added before the store write. | +| `_ABL_sub_core` (`sub`, without `InvRouterMirrors`) | ownership | `InvOwnershipAgree` | 1,474 | 9 | 0 s | c2 adds a route, c3 takes over, then c2's delayed store write overwrites the whole session with c2's token: the store now names a displaced connection. | +| `_ABL_disc` (`disc`) | Session Present | `InvSessionPresent` | 1,069 | 8 | 1 s | c1 (expiry 1) disconnects with expiry 0. Before its release runs, c2 claims and resumes the stored session, which still says expiry 1, although the session ended at the DISCONNECT. | +| `_ABL_release` (`release`) | router mirror | `InvRouterMirrors` | 834 | 8 | 1 s | Router release and route strip are separate steps. | +| `_ABL_release_core` (`release`, without `InvRouterMirrors`) | quiescent consistency | `InvQuiescentConsistent` | 71,829 | 13 | 2 s | c1's handshake fails and it releases the router. c2 claims and subscribes. c1's delayed strip then deletes c2's routes. | +| `_ABL_sweep` (`sweep`) | live owner intact | `InvLiveOwnerSessionKept` | 14,263 | 11 | 1 s | After a restart the file sweep decides a session is expired. c3 claims a new session. The sweep then deletes c3's live session without re-checking. | +| `_ABL_stamp` (`stamp`) | exact expiry | `InvExpiryExact` | 120 | 7 | 1 s | After a crash the loaded session's expiry counts from the time it is read, not from when it ended. | +| `_ABL_rebuild` (`rebuild`) | router mirror | `InvRouterMirrors` | 810 | 8 | 0 s | c3 (expiry 1) subscribes, the broker crashes, and the router comes back without the stored session's routes, so offline messages are not routed. | +| `_ABL_wt_claim` (`wt_claim`) | Session Present | `InvSessionPresent` | 453 | 7 | 1 s | c1's claim with expiry 1 is acknowledged but still in the write-behind cache when the broker crashes. c2 gets SP = 0 for a session that should still exist. | +| `_ABL_wt_claim_resurrect` (`wt_claim`, only `InvNoResurrection`) | resurrection across a crash | `InvNoResurrection` | 6,163 | 10 | 1 s | c1's session (expiry 1) is flushed. c2 connects with Clean Start = 1; the discard is still in the cache when the broker crashes. c3 resumes c1's discarded session. | +| `_ABL_wt_sub` (`wt_sub`) | acked subscriptions durable | `InvAckedSubsDurable` | 3,542 | 9 | 1 s | c1's SUBACK is sent, the broker crashes before the flush, and c2 resumes without the subscription. | +| `_ABL_wt_disc` (`wt_disc`) | Session Present or exact expiry | `InvExpiryExact` | 2,696 | 9 | 0 s | c3 (expiry 1) fails its CONNACK; its release marks the session disconnected at time 0 in the cache only. Time advances and the broker crashes: the file still says connected, so the boot stamps the disconnect at time 1 and the expiry clock starts late. | +| `_ABL_wt_remove` (`wt_remove`) | resurrection of a removed session | `InvExpiryExact` | 896 | 8 | 1 s | c3's expiry-0 session (its CONNACK failed) is removed in the cache only. After a crash the file brings it back with a wrong disconnect time. | +| `_ABL_wt_remove_core` (`wt_remove`, without `InvExpiryExact`) | ended session never cleaned | `InvProgress` | 2,582 | 9 | 1 s | The same lost removal: the resurrected session outlives its true end and nothing removes it in time. | + +### Today's code (`MECH = "CURRENT"`, 3 Conns, 1 filter, `MaxT = 2`, `MaxBoots = 1`, symmetry) + +Each cfg checks `TypeOK` and one property. Liveness is checked without symmetry. + +| cfg | Result | Distinct states | Depth | Time | Counterexample | +|---|---|---|---|---|---| +| `_CUR_OwnershipAgree` | `InvOwnershipAgree` violated | 513 | 8 | 0 s | c1 registers. c2, whose read happened earlier, writes its own session: the store names c2 while the router names c1. | +| `_CUR_NoLeakedConnected` | `InvNoLeakedConnected` violated | 23,722 | 12 | 1 s | c1 and c2 both write and register; c1 registers last. c2 disconnects with expiry 0 and, no longer owner, skips cleanup. c1's record was overwritten by c2's, which stays "connected" with no connection. | +| `_CUR_LiveOwnerSessionKept` | `InvLiveOwnerSessionKept` violated | 3,813 | 10 | 1 s | c1 is the live owner. c2 overwrites the store, fails its CONNACK and marks the record disconnected: the live owner's session is gone. | +| `_CUR_FailedHandshakeHarmless` | `InvFailedHandshakeHarmless` violated | 3,116 | 10 | 1 s | Same interleaving: a failed handshake leaves its own token in the live owner's session. | +| `_CUR_SessionPresent` | `InvSessionPresent` violated | 76 | 6 | 0 s | c2 resumes (SP = 1) the expiry-0 session c1 wrote during its handshake. | +| `_CUR_NoResurrection` | `InvNoResurrection` violated | 684 | 8 | 1 s | c3 reads c1's session. c2 then writes a fresh session. c3 writes back the resumed c1 session, undoing c2's fresh start. | +| `_CUR_AckedSubsDurable` | `InvAckedSubsDurable` violated | 36,779 | 12 | 1 s | c1 is the live owner and subscribes (SUBACK sent, stored). c2, still in its handshake, overwrites the store with the fresh empty session it decided on earlier. c3 then resumes that record, which lacks c1's acknowledged subscription. | +| `_CUR_CleanStart` | `InvCleanStart` violated | 2,643,432 | 18 | 26 s | c2 starts empty and subscribes. c3 resumes c2's session and registers, but its route install is still pending. c1, which wrote an empty session long before, now registers (clean, so it strips the routes). c3's late `Install` then unions c2's route into the router while c1, a clean owner with no subscriptions, owns it. | +| `_CUR_RouterMirrors` | `InvRouterMirrors` violated | 316 | 8 | 1 s | A route is added before the store write. | +| `_CUR_QuiescentConsistent` | `InvQuiescentConsistent` violated | 55,221 | 12 | 1 s | c1 subscribes and persists. c2, whose store write came earlier, registers: c2's router entry and stored session disagree. | +| `_CUR_ExpiryExact` | `InvExpiryExact` violated | 522 | 8 | 1 s | A connected session is flushed, the broker crashes, and the loaded session's expiry counts from the read time. | +| `_CUR_Progress` | `InvProgress` violated | 1,736 | 9 | 0 s | c1's expiry-0 session is flushed while connected and the broker crashes. With no stored disconnect time, the loaded session's expiry counts from the current time on every read, so it never expires and is never removed, although it ended at the crash. | +| `_CUR_live` (no symmetry) | `EventuallyClean` violated | 92,090 | 10 | 4 s | The same session is never cleaned up. | + +### Rejected lock-free designs (3 Conns, 1 filter, `MaxT = 2`, `MaxBoots = 1`, symmetry) + +| cfg | MECH | Checks | Result | Distinct states | Depth | Time | Counterexample | +|---|---|---|---|---|---|---|---| +| `_CAS_agree` | `CAS` | `InvOwnershipAgree` | violated | 27 | 5 | 0 s | c1, c2 and c3 claim in storage in turn. c1 registers at the router while the store holds c3's token. | +| `_CAS_mirrors` | `CAS` | `InvRouterMirrors` | violated | 130 | 7 | 1 s | The router step of a SUBSCRIBE runs before its fenced store step. | +| `_CAS_rest` | `CAS` | the other 12 invariants | pass | 1,887,418 | 30 | 31 s | none: hardened CAS is correct except for the two properties above. | +| `_CASSPLIT_agree` | `CASSPLIT` | `InvOwnershipAgree` | violated | 37 | 5 | 0 s | c1 and c2 register at the router in turn (c2 is owner). c1's storage claim then writes c1's token. | +| `_CASSPLIT_resurrect` | `CASSPLIT` | `InvNoResurrection` | violated | 5,542 | 8 | 1 s | **Resurrection.** c1 registers (expiry 1). c2 registers with Clean Start = 1, which strips the routes. c3 registers (Clean Start = 0). c1's storage claim writes c1's session. c3's storage claim passes the token compare (c1's token is older) and resumes c1's session, which c2's clean start, registered before c3, had discarded. | + +### Group commit (`MECH = "LOCK"`, `DROP = {}`, `MaxPend = 8`, symmetry except liveness) + +"All 16" is the 14 invariants of the chosen design plus `InvAckedStateSurvivesCrash` and +`InvPendNotFull`. + +| cfg | GC | Conns | Filters | MaxT | MaxBoots | Reductions | Checks | Result | Distinct states | Depth | Time | +|---|---|---|---|---|---|---|---|---|---|---|---| +| `SessionOwnership_GC_whole.cfg` | `WHOLE` | 3 | 2 | 3 | 1 | symmetry | all 16 invariants | pass | 26,003,764 | 42 | 519 s | +| `SessionOwnership_GC_prefix.cfg` | `PREFIX` | 3 | 2 | 3 | 1 | symmetry | all 16 invariants | pass | 26,003,764 | 42 | 544 s | +| `SessionOwnership_GC_prefix_boots2.cfg` | `PREFIX` | 3 | 2 | 3 | 2 | symmetry | all 16 invariants | pass | 47,933,876 | 43 | 1,006 s | +| `SessionOwnership_GC_prefix_live.cfg` | `PREFIX` | 3 | 1 | 2 | 1 | none | `EventuallyClean` under `Spec` | pass | 15,768,372 | 32 | 1,182 s | +| `SessionOwnership_GC_perwriter_mono.cfg` | `PERWRITER_MONO` | 3 | 1 | 2 | 1 | symmetry | all 16 invariants | pass | 2,628,548 | 32 | 50 s | + +`WHOLE` and `PREFIX` reach the same set of states: a partial flush leaves the same buffer as a +whole flush taken before the later writes were appended. + +Group-commit ablations (3 Conns, 1 filter, `MaxT = 2`, `MaxBoots = 1`, symmetry). Each must fail. +`_core` repeats the ablation with the 14 invariants of the chosen design only (without +`InvAckedStateSurvivesCrash`), to show the MQTT-visible consequence. + +| cfg | GC | Checks | First violated | Distinct states | Depth | Time | Counterexample | +|---|---|---|---|---|---|---|---| +| `_GC_ABL_perwriter` | `PERWRITER` (a) | all 16 | `InvAckedStateSurvivesCrash` | 1,219 | 8 | 1 s | c1, c2, c3 claim in turn (three pending entries). c2's entry is flushed and c2 gets its CONNACK. c1's older entry is flushed afterwards and overwrites the file with c1's record. The broker crashes: c2's acknowledged claim is gone. | +| `_GC_ABL_perwriter_core` | `PERWRITER` (a) | the 14 | `InvSessionPresent` | 10,036 | 9 | 1 s | c1 claims with expiry 1; c2 takes over (Clean Start = 0, expiry 0). c2's entry is flushed and c2 gets CONNACK. c1's older entry then lands on the file. After a crash c3 resumes c1's expiry-1 session (SP = 1), although the session c2 was told about had expiry 0 and ended at the crash. | +| `_GC_ABL_ackearly` | `ACKEARLY` (b) | all 16 | `InvAckedStateSurvivesCrash` | 112 | 6 | 0 s | c1, c2, c3 claim; c1 gets CONNACK while its entry is still pending; the broker crashes. | +| `_GC_ABL_ackearly_core` | `ACKEARLY` (b) | the 14 | `InvSessionPresent` | 1,401 | 8 | 1 s | c1 claims with expiry 1 and gets CONNACK before the flush; the broker crashes; c2 (Clean Start = 0) gets SP = 0 for the session c1 was told exists. | +| `_GC_ABL_noryw` | `NORYW` (c) | all 16 | `InvSessionPresent` | 68 | 6 | 0 s | c1 claims with expiry 1 (pending). c2 (Clean Start = 0) reads only the file, finds nothing and starts empty (SP = 0), discarding c1's session. No crash is involved. | +| `_GC_ABL_noryw_core` | `NORYW` (c) | the 14 | `InvSessionPresent` | 68 | 6 | 1 s | c2's expiry-1 claim is pending; c3 (Clean Start = 0) reads the file, starts empty and discards it. | +| `_GC_ABL_noryw_acked` | `NORYW` (c) | `TypeOK`, `InvAckedClaimCorrect` | `InvAckedClaimCorrect` | 793 | 8 | 0 s | The wrong decision reaches the client: after the stale claim above, both entries are flushed in order and c2 gets CONNACK with SP = 0 for a session that existed. | +| `_GC_ABL_noryw_subs` | `NORYW` (c) | `TypeOK`, `InvAckedSubsDurable` | `InvAckedSubsDurable` | 2,972 | 9 | 1 s | c2 (expiry 1) is live and subscribes; the SUBSCRIBE is pending. c3 takes over and resumes from the file, without the subscription. c2's entry precedes c3's, so c2's SUBACK can still be sent after the flush: an acknowledged subscription is lost without any crash. | + +Verdict on (c): reads that see only the file are unsafe, not merely stale. The lock orders the +claim after the pending write and the flush makes both durable in that order, so the stale +decision is final and it contradicts a write that precedes it (and that may be acknowledged). + +## Provenance + +Three modelers built independent models, checked exhaustively with TLC, and reached the same +verdict: the per-ClientID lock. + +- **Modeler 1** (`SessionMech`, `MECH` in `LOCK`, `CAS`, `CASSPLIT`, `CURRENT`, `CUSTOM`): + per-fix ablations and the router/store single-writer check. It found the `CASSPLIT` + resurrection (a clean start that registered first is undone by a later storage claim) and + that the clean-start discard must be durable (`durable_clean`), otherwise a crash resurrects + the discarded session (`InvNoResurrect`). +- **Modeler 2** (`SessionMech`, `SessionMechAnom`, `SessionMechDur`; `LOCK` vs hardened `CAS` vs + `FENCE`): Session Present linearizability (`InvSpLinearizable`), the always-mirror router + property (`InvRouterAlwaysMirrors`, which hardened CAS fails), the acked-clean-start-lost + anomaly of `FENCE` (`InvNoAckedCleanStartLost`), and a 16-way durability sweep over claim, + exit, remove and subscription writes showing that only "all durable" passes. +- **Modeler 3** (`SessionOwnership` first round, then `SessionMech` with `LOCK` vs + `CAS_RF` / `CAS_RF2` / `CAS_SF`): the file backend with cache, write-behind flush and restart, + exact expiry across restart (persisted timestamp plus boot stamping), claim linearizability + (`InvClaimLinearizable`), the `CURRENT` per-property counterexamples, and the hardening a CAS + design needs (token epochs, tombstones, displaced cleanup, fenced router sweeps). + +This consolidated spec is built on modeler 3's file-backend model (time, cache, write-behind, +restart, ghost logical session). Merged in: modeler 1's mechanism switch with per-fix ablation +and its `CASSPLIT` resurrection; modeler 2's always-mirror router property and its per-write +durability ablation (here `wt_claim`, `wt_sub`, `wt_disc`, `wt_remove`); and modeler 3's own +first-round `CURRENT` model as the "element dropped" behaviour. Names were unified: +`InvClaimLinearizable` / `InvNoResurrect` / `InvSpLinearizable` became `InvNoResurrection` and +`InvSessionPresent`; `InvRouterAlwaysMirrors` became `InvRouterMirrors`. The hardened `CAS` here +is modeler 3's `CAS_SF` with all hardening on; `CASSPLIT` is modeler 3's `CAS_RF` without the +storage tombstone, which is modeler 1's `CASSPLIT`. + +## Tool notes + +- All results come from the TLC command line. tla-mcp 0.10.1 was not used for any reported + result because: + - `P ~> Q` can pass vacuously, + - `[](P => <>Q)` is not supported, + - it ignores the cfg's `SPECIFICATION`, so a negative control that swaps in a spec without + sweep fairness would silently run with the fair spec. + + TLC honours `SPECIFICATION`, which the failing `SpecNoSweepFair` control demonstrates. +- Liveness is written only as `[]<>` over state predicates, never with `~>`. +- State directories were kept outside the repository and deleted after each run. A failing run + also makes TLC write `SessionOwnership_TTrace_*.tla` / `.bin` files next to the spec; those + were deleted and are not part of this directory. + +## Mapping from spec actions to code + +The code is being reworked alongside this model; names are as of this modelling pass. + +| Spec | Broker concept | +|---|---| +| per-ClientID critical section (every `L*` action) | the session slot lock: `SessionSlots` / `MessageRouter::lock_session`, held for the whole step | +| `LClaim` | the claim in the CONNECT path (`handle_session` in `client_handler/connect.rs`), under the session slot: `get_session`, then a durable session write (`write_claim`), then `register_session_as` and an exact router route set (`set_client_subscriptions`), all before CONNACK. If the write fails the claim stops before registering and the client gets CONNACK 0x88 | +| `tok` | `ClientSession::connection_token` (the router generation) | +| `Owns(c)` | the token compare in `update_session` / `remove_owned_session` | +| `LSub`, `SubRouter` / `SubStore` | `subscribe_as` / `unsubscribe_as` for every filter of the packet, then one owner-only `update_session` carrying all of them (`persist_or_restore_routes`), durable before SUBACK / UNSUBACK, versus today's router subscribe followed by a whole-session `store_session` | +| `Close` with `disc` | DISCONNECT packet handling: under the per-ClientID lock, at the DISCONNECT and before the release, if the DISCONNECT changes Session Expiry and the connection still owns the stored session (token compare), write the new expiry to storage (with group commit: append it to the pending buffer). The release (`LRelease`) is a separate, later critical section that writes the disconnect time or removes the session. | +| `LRelease`, `Release` / `Strip` / `Cleanup` / `RmOwned` | `release_client` / `release_ownership` / `persist_session_end` (owner-only, strip under the lock) versus today's `unregister_client`, route strip, `update_session` and `remove_owned_session` as separate awaits | +| `ConnackFail`, `HConnackFail` / `FailRel` | a CONNACK write that fails after the claim | +| `LSweep`, `FileSweepRead` / `FileSweepApply`, `RouterSweepDecide` / `RouterSweepApply` | `sweep_sessions` / `sweep_session` under the slot versus today's `cleanup_expired` (file sweep) and `cleanup_stale_subscriptions` (router sweep) | +| `Wr(_, TRUE, _)`, `Remove(TRUE)` | write-through session writes and removals | +| `Wr(_, FALSE, _)`, `Flush` | today's write-behind `store_session` and the periodic `flush_sessions` | +| `pbuf`, `Log` (group commit) | the group-commit pending buffer: each session write, under the per-ClientID lock, updates the in-memory cache and appends the new record with the next sequence number | +| `GFlushPrefix(k)` | one group-commit batch: write and fsync every record up to sequence number k, then publish "durable through k" | +| `AckReady`, `ConnackOK`, `SubAck`, `RelDone` | the wait before CONNACK, SUBACK / UNSUBACK and the end of DISCONNECT processing: the connection waits until "durable through" is at least the sequence number of its latest write | +| `seen`, `ts` | `disconnected_at` (`ClientSession::mark_disconnected`) | +| `Restart` with `stamp` and `rebuild` | `recover_sessions`: stamp still-connected sessions with the boot time, rebuild router routes from stored sessions with expiry > 0. The code also removes expired and expiry-0 sessions during recovery; the model leaves them to the sweep, whose re-check removes them the same way. | +| `SStore` / `SReg`, `RReg` / `RStore` / `RInstall`, `CRelease` / `CClean`, `CFileSweep` | the rejected lock-free designs; no code | + +## Not modelled + +- **Queues and inflight state.** Offline message queues, QoS 1/2 inflight and packet ids are not + modelled (see `specs/tla/offline-queue/` and `specs/tla/deferred-ack/`). The session record + here holds only ownership, subscriptions, expiry and timestamps. +- **Wills.** Will messages are not modelled. The implementation fences will claims with the same + connection generation that fences session writes (`claim_will`); that token compare is the + only part of will handling this model's `Owns` check stands for. +- **Multiple ClientIDs.** One ClientID only. Different ClientIDs use different locks and + different records and do not interact in this protocol. +- **Subscription options, shared subscriptions, retained messages, authentication.** +- **Memory backend.** Only the file backend is modelled. The memory backend is the file backend + without a file: nothing survives a restart. +- **Failed writes.** Every write in the model succeeds. What the code does when one fails is + covered by tests, not by the model: a failed flush rejects every write pending with it and + restores the last durable state, the log is repaired before the next append, a failed claim + write is refused with CONNACK 0x88 without displacing the live owner, and a failed SUBSCRIBE or + UNSUBSCRIBE write restores the routes and the stored subscriptions and ends the connection + with DISCONNECT 0x80. +- **Bounds.** Results hold for the constants listed, not in general. diff --git a/specs/tla/session-ownership/SessionOwnership.cfg b/specs/tla/session-ownership/SessionOwnership.cfg new file mode 100644 index 00000000..375e1135 --- /dev/null +++ b/specs/tla/session-ownership/SessionOwnership.cfg @@ -0,0 +1,33 @@ +SPECIFICATION Spec + +CONSTANTS + Conns = {c1, c2, c3} + Subs = {s1, s2} + None = none + MaxT = 3 + E = 1 + MaxBoots = 1 + MECH = "LOCK" + DROP = {} + GC = "OFF" + MaxPend = 1 + +SYMMETRY Sym + +INVARIANTS + TypeOK + InvOwnershipAgree + InvNoLeakedConnected + InvLiveOwnerSessionKept + InvFailedHandshakeHarmless + InvSessionPresent + InvNoResurrection + InvAckedSubsDurable + InvAbortJustified + InvCleanStart + InvRouterMirrors + InvQuiescentConsistent + InvExpiryExact + InvProgress + +CHECK_DEADLOCK FALSE diff --git a/specs/tla/session-ownership/SessionOwnership.tla b/specs/tla/session-ownership/SessionOwnership.tla new file mode 100644 index 00000000..9e333338 --- /dev/null +++ b/specs/tla/session-ownership/SessionOwnership.tla @@ -0,0 +1,818 @@ +--------------------------- MODULE SessionOwnership --------------------------- +EXTENDS Integers, FiniteSets, Sequences, TLC + +CONSTANTS Conns, Subs, None, MaxT, E, MaxBoots, MECH, DROP, GC, MaxPend + +VARIABLES pc, tok, clr, ex, loc, pend, cborn, startEff, + spOK, linOK, subOK, abortOK, failed, + owner, hwm, rsubs, rsPend, swPend, nextId, + cache, disk, dirty, + now, boots, bootT, + gEx, gEnd, gExp, gOwner, gSubs, cleanEff, + pbuf, dg, dpos, wpos, ackSafe + +connV == <> +routerV == <> +storeV == <> +timeV == <> +ghostV == <> +gcV == <> +vars == <> + +LockFixes == {"claim", "sub", "release", "disc", "sweep", "stamp", "rebuild", + "wt_claim", "wt_sub", "wt_disc", "wt_remove"} + +FixSet == + CASE MECH = "LOCK" -> LockFixes \ DROP + [] MECH = "CURRENT" -> {"wt_remove"} + [] MECH = "CAS" -> {"disc", "stamp", "rebuild", "wt_claim", "wt_sub", "wt_disc", "wt_remove"} + [] MECH = "CASSPLIT" -> {"disc", "stamp", "rebuild", "wt_claim", "wt_sub", "wt_disc", "wt_remove"} + +On(f) == f \in FixSet + +IsLock == MECH = "LOCK" +IsCas == MECH \in {"CAS", "CASSPLIT"} +Legacy(f) == MECH \in {"LOCK", "CURRENT"} /\ ~On(f) + +MaxTok == Cardinality(Conns) + 1 + +NoRec == [ex |-> FALSE, tok |-> 0, subs |-> {}, exp |-> 0, conn |-> FALSE, seen |-> 0, + gdt |-> 0, born |-> 0] +NoDisk == [ex |-> FALSE, tok |-> 0, subs |-> {}, exp |-> 0, ts |-> -1, gdt |-> 0, born |-> 0] + +Pcs == {"idle", "store", "hconnack", "register", "install", + "cstore", "cinstall", "creg", "connack", + "live", "substore", "unsubstore", "suback", "relwait", + "release", "strip", "cleanup", "rmowned", "failrel", "cclean", + "done", "dead"} +Closing == {"release", "strip", "cleanup", "rmowned", "failrel", "cclean"} +LivePC == {"live", "substore", "unsubstore", "suback"} +Ended == {"idle", "done", "dead"} + +Dur(e) == IF e = 0 THEN 0 ELSE E + +Expired(r) == r.ex /\ ~r.conn /\ now > r.seen + Dur(r.exp) + +Load(d) == [ex |-> TRUE, tok |-> d.tok, subs |-> d.subs, exp |-> d.exp, conn |-> FALSE, + seen |-> IF On("stamp") THEN (IF d.ts = -1 THEN bootT ELSE d.ts) ELSE now, + gdt |-> d.gdt, born |-> d.born] + +ToDisk(r) == [ex |-> TRUE, tok |-> r.tok, subs |-> r.subs, exp |-> r.exp, + ts |-> IF On("stamp") /\ ~r.conn THEN r.seen ELSE -1, + gdt |-> r.gdt, born |-> r.born] + +View == IF cache.ex \/ dirty THEN cache ELSE IF disk.ex THEN Load(disk) ELSE NoRec + +DiskView == + IF disk.ex + THEN [ex |-> TRUE, tok |-> disk.tok, subs |-> disk.subs, exp |-> disk.exp, + conn |-> disk.ts = -1, seen |-> IF disk.ts = -1 THEN now ELSE disk.ts, + gdt |-> disk.gdt, born |-> disk.born] + ELSE NoRec + +RV == IF GC = "NORYW" THEN DiskView ELSE View + +ReadExpired == RV.ex /\ Expired(RV) +ReadRes == IF ReadExpired THEN NoRec ELSE RV +DiskAfterRead == IF ReadExpired /\ On("wt_remove") THEN NoDisk ELSE disk + +Wr(r, wt, d0) == + /\ cache' = r + /\ IF wt THEN IF GC = "OFF" THEN disk' = ToDisk(r) /\ dirty' = FALSE + ELSE disk' = disk /\ dirty' = TRUE + ELSE disk' = d0 /\ dirty' = TRUE + +Remove(wt) == + /\ cache' = NoRec + /\ IF wt THEN IF GC = "OFF" THEN disk' = NoDisk /\ dirty' = FALSE + ELSE disk' = disk /\ dirty' = TRUE + ELSE disk' = disk /\ dirty' = TRUE + +ExpireOnRead == IF ReadExpired THEN Remove(On("wt_remove")) ELSE UNCHANGED storeV + +Owns(c) == tok[c] # 0 /\ RV.ex /\ RV.tok = tok[c] + +OwnsNow(c) == tok[c] # 0 /\ View.ex /\ View.tok = tok[c] + +Holder(t) == \E c \in Conns : pc[c] \notin Ended /\ tok[c] = t + +GAlive == gEx /\ gExp > 0 /\ (gEnd = -1 \/ now <= gEnd + E) + +GhostEnd(c, e) == + IF gOwner = c + THEN /\ gEnd' = now /\ gExp' = e /\ gEx' = (gEx /\ e > 0) + /\ UNCHANGED <> + ELSE UNCHANGED ghostV + +Disconnected(c) == [RV EXCEPT !.conn = FALSE, !.seen = now, !.gdt = now, !.exp = ex[c]] + +GRec(x, en, xp, o, sb, cl) == [ex |-> x, end |-> en, exp |-> xp, owner |-> o, subs |-> sb, clean |-> cl] + +GNow == GRec(gEx, gEnd, gExp, gOwner, gSubs, cleanEff) +GNext == GRec(gEx', gEnd', gExp', gOwner', gSubs', cleanEff') + +EndG(g, c) == IF g.owner = c THEN [g EXCEPT !.end = now, !.ex = (g.ex /\ g.exp > 0)] ELSE g + +Room == GC = "OFF" \/ Len(pbuf) < MaxPend + +Entry == [r |-> IF cache'.ex THEN ToDisk(cache') ELSE NoDisk, g |-> GNext, f |-> FALSE, a |-> FALSE] + +Log(c, w, ended) == + IF GC = "OFF" THEN UNCHANGED gcV + ELSE LET base == IF ended + THEN [i \in DOMAIN pbuf |-> [pbuf[i] EXCEPT !.g = EndG(@, c)]] + ELSE pbuf + IN /\ dg' = IF ended THEN EndG(dg, c) ELSE dg + /\ pbuf' = IF w THEN Append(base, Entry) ELSE base + /\ wpos' = IF w /\ c \in Conns THEN [wpos EXCEPT ![c] = Len(base) + 1] ELSE wpos + /\ UNCHANGED <> + +AckReady(c) == + CASE GC \in {"OFF", "ACKEARLY"} -> TRUE + [] GC \in {"PERWRITER", "PERWRITER_MONO"} -> IF wpos[c] = 0 THEN TRUE ELSE pbuf[wpos[c]].f + [] OTHER -> wpos[c] = 0 + +MarkAck(c) == + IF GC = "OFF" THEN UNCHANGED gcV + ELSE /\ pbuf' = IF wpos[c] = 0 THEN pbuf ELSE [pbuf EXCEPT ![wpos[c]].a = TRUE] + /\ UNCHANGED <> + +TypeOK == + /\ pc \in [Conns -> Pcs] + /\ tok \in [Conns -> 0..MaxTok] + /\ clr \in [Conns -> BOOLEAN] + /\ ex \in [Conns -> {0, 1}] + /\ loc \in [Conns -> SUBSET Subs] + /\ pend \in [Conns -> Subs] + /\ cborn \in [Conns -> 0..MaxTok] + /\ startEff \in [Conns -> SUBSET (1..MaxTok)] + /\ spOK \in [Conns -> BOOLEAN] + /\ linOK \in [Conns -> BOOLEAN] + /\ subOK \in [Conns -> BOOLEAN] + /\ abortOK \in [Conns -> BOOLEAN] + /\ failed \in [Conns -> BOOLEAN] + /\ owner \in Conns \cup {None} + /\ hwm \in 0..MaxTok + /\ rsubs \in SUBSET Subs + /\ rsPend \in 0..MaxTok + /\ swPend \in BOOLEAN + /\ nextId \in 1..MaxTok + /\ cache.ex \in BOOLEAN /\ cache.subs \subseteq Subs /\ cache.tok \in 0..MaxTok + /\ disk.ex \in BOOLEAN /\ disk.subs \subseteq Subs /\ disk.ts \in -1..MaxT + /\ dirty \in BOOLEAN + /\ now \in 0..MaxT /\ boots \in 0..MaxBoots /\ bootT \in 0..MaxT + /\ gEx \in BOOLEAN /\ gEnd \in -1..MaxT /\ gExp \in {0, 1} + /\ gOwner \in Conns \cup {None} /\ gSubs \subseteq Subs + /\ cleanEff \subseteq 1..MaxTok + /\ Len(pbuf) <= MaxPend + /\ \A i \in DOMAIN pbuf : pbuf[i].f \in BOOLEAN /\ pbuf[i].a \in BOOLEAN + /\ dpos \in 0..MaxPend + /\ wpos \in [Conns -> 0..MaxPend] + /\ ackSafe \in BOOLEAN + +Init == + /\ pc = [c \in Conns |-> "idle"] + /\ tok = [c \in Conns |-> 0] + /\ clr = [c \in Conns |-> FALSE] + /\ ex = [c \in Conns |-> 0] + /\ loc = [c \in Conns |-> {}] + /\ pend = [c \in Conns |-> CHOOSE s \in Subs : TRUE] + /\ cborn = [c \in Conns |-> 0] + /\ startEff = [c \in Conns |-> {}] + /\ spOK = [c \in Conns |-> TRUE] + /\ linOK = [c \in Conns |-> TRUE] + /\ subOK = [c \in Conns |-> TRUE] + /\ abortOK = [c \in Conns |-> TRUE] + /\ failed = [c \in Conns |-> FALSE] + /\ owner = None + /\ hwm = 0 + /\ rsubs = {} + /\ rsPend = 0 + /\ swPend = FALSE + /\ nextId = 1 + /\ cache = NoRec + /\ disk = NoDisk + /\ dirty = FALSE + /\ now = 0 + /\ boots = 0 + /\ bootT = 0 + /\ gEx = FALSE + /\ gEnd = -1 + /\ gExp = 0 + /\ gOwner = None + /\ gSubs = {} + /\ cleanEff = {} + /\ pbuf = <<>> + /\ dg = GRec(FALSE, -1, 0, None, {}, {}) + /\ dpos = 0 + /\ wpos = [c \in Conns |-> 0] + /\ ackSafe = TRUE + +SpCheck(cl, resume) == (resume => GAlive) /\ ((~cl /\ GAlive) => resume) + +LClaim(c, cl, e) == + /\ IsLock /\ On("claim") /\ pc[c] = "idle" + /\ Room + /\ LET r == ReadRes + t == nextId + resume == ~cl /\ r.ex /\ r.exp > 0 + rec == [ex |-> TRUE, tok |-> t, subs |-> IF resume THEN r.subs ELSE {}, exp |-> e, + conn |-> TRUE, seen |-> now, gdt |-> 0, born |-> IF resume THEN r.born ELSE t] + IN /\ Wr(rec, On("wt_claim"), DiskAfterRead) + /\ rsubs' = rec.subs + /\ loc' = [loc EXCEPT ![c] = rec.subs] + /\ cborn' = [cborn EXCEPT ![c] = rec.born] + /\ clr' = [clr EXCEPT ![c] = ~resume] + /\ spOK' = [spOK EXCEPT ![c] = SpCheck(cl, resume)] + /\ linOK' = [linOK EXCEPT ![c] = resume => \A x \in cleanEff : r.born >= x] + /\ subOK' = [subOK EXCEPT ![c] = resume => r.subs = gSubs] + /\ gSubs' = rec.subs + /\ cleanEff' = IF resume THEN cleanEff ELSE cleanEff \cup {t} + /\ owner' = c + /\ tok' = [tok EXCEPT ![c] = nextId] + /\ nextId' = nextId + 1 + /\ ex' = [ex EXCEPT ![c] = e] + /\ gEx' = TRUE /\ gEnd' = -1 /\ gExp' = e /\ gOwner' = c + /\ pc' = [pc EXCEPT ![c] = "connack"] + /\ UNCHANGED <> + /\ Log(c, TRUE, FALSE) + +HSRead(c, cl, e) == + /\ Legacy("claim") /\ pc[c] = "idle" + /\ LET r == ReadRes + bound == r.ex /\ r.exp = 0 /\ owner # None + resume == ~cl /\ ~bound /\ r.ex + IN /\ clr' = [clr EXCEPT ![c] = ~resume] + /\ loc' = [loc EXCEPT ![c] = IF resume THEN r.subs ELSE {}] + /\ cborn' = [cborn EXCEPT ![c] = IF resume THEN r.born ELSE 0] + /\ spOK' = [spOK EXCEPT ![c] = SpCheck(cl, resume)] + /\ subOK' = [subOK EXCEPT ![c] = resume => r.subs = gSubs] + /\ ExpireOnRead + /\ ex' = [ex EXCEPT ![c] = e] + /\ pc' = [pc EXCEPT ![c] = "store"] + /\ UNCHANGED <> + /\ UNCHANGED gcV + +HSStore(c) == + /\ pc[c] = "store" + /\ LET t == nextId + b == IF cborn[c] = 0 THEN t ELSE cborn[c] + rec == [ex |-> TRUE, tok |-> t, subs |-> loc[c], exp |-> ex[c], conn |-> TRUE, + seen |-> now, gdt |-> 0, born |-> b] + IN /\ Wr(rec, On("wt_claim"), disk) + /\ cborn' = [cborn EXCEPT ![c] = b] + /\ linOK' = [linOK EXCEPT ![c] = cborn[c] # 0 => \A x \in cleanEff : cborn[c] >= x] + /\ cleanEff' = IF cborn[c] = 0 THEN cleanEff \cup {t} ELSE cleanEff + /\ tok' = [tok EXCEPT ![c] = nextId] + /\ nextId' = nextId + 1 + /\ pc' = [pc EXCEPT ![c] = "hconnack"] + /\ UNCHANGED <> + /\ UNCHANGED gcV + +HConnackOK(c) == + /\ pc[c] = "hconnack" + /\ pc' = [pc EXCEPT ![c] = "register"] + /\ UNCHANGED <> + /\ UNCHANGED gcV + +HConnackFail(c) == + /\ pc[c] = "hconnack" + /\ failed' = [failed EXCEPT ![c] = TRUE] + /\ pc' = [pc EXCEPT ![c] = "failrel"] + /\ UNCHANGED <> + /\ UNCHANGED gcV + +FailRel(c) == + /\ pc[c] = "failrel" + /\ IF Owns(c) + THEN Wr([View EXCEPT !.conn = FALSE, !.seen = now, !.gdt = now], On("wt_disc"), disk) + ELSE UNCHANGED storeV + /\ pc' = [pc EXCEPT ![c] = "done"] + /\ UNCHANGED <> + /\ UNCHANGED gcV + +Register(c) == + /\ pc[c] = "register" + /\ owner' = c + /\ rsubs' = IF clr[c] THEN {} ELSE rsubs + /\ gEx' = TRUE /\ gEnd' = -1 /\ gExp' = ex[c] /\ gOwner' = c /\ gSubs' = loc[c] + /\ pc' = [pc EXCEPT ![c] = IF clr[c] THEN "live" ELSE "install"] + /\ UNCHANGED <> + /\ UNCHANGED gcV + +Install(c) == + /\ pc[c] = "install" + /\ rsubs' = rsubs \cup loc[c] + /\ pc' = [pc EXCEPT ![c] = "live"] + /\ UNCHANGED <> + /\ UNCHANGED gcV + +CasRec(c, r, t) == + LET resume == ~clr[c] /\ r.ex /\ r.exp > 0 + IN [resume |-> resume, + rec |-> [ex |-> TRUE, tok |-> t, subs |-> IF resume THEN r.subs ELSE {}, exp |-> ex[c], + conn |-> TRUE, seen |-> now, gdt |-> 0, born |-> IF resume THEN r.born ELSE t]] + +SStore(c, cl, e) == + /\ MECH = "CAS" /\ pc[c] = "idle" + /\ LET r == ReadRes + t == nextId + resume == ~cl /\ r.ex /\ r.exp > 0 + rec == [ex |-> TRUE, tok |-> t, subs |-> IF resume THEN r.subs ELSE {}, exp |-> e, + conn |-> TRUE, seen |-> now, gdt |-> 0, born |-> IF resume THEN r.born ELSE t] + IN /\ Wr(rec, On("wt_claim"), DiskAfterRead) + /\ loc' = [loc EXCEPT ![c] = rec.subs] + /\ clr' = [clr EXCEPT ![c] = ~resume] + /\ spOK' = [spOK EXCEPT ![c] = SpCheck(cl, resume)] + /\ linOK' = [linOK EXCEPT ![c] = resume => \A x \in cleanEff : r.born >= x] + /\ subOK' = [subOK EXCEPT ![c] = resume => r.subs = gSubs] + /\ gSubs' = rec.subs + /\ cleanEff' = IF resume THEN cleanEff ELSE cleanEff \cup {t} + /\ tok' = [tok EXCEPT ![c] = nextId] + /\ nextId' = nextId + 1 + /\ ex' = [ex EXCEPT ![c] = e] + /\ gEx' = TRUE /\ gEnd' = -1 /\ gExp' = e /\ gOwner' = c + /\ pc' = [pc EXCEPT ![c] = "creg"] + /\ UNCHANGED <> + /\ UNCHANGED gcV + +NewerActive(c) == \E d \in Conns : tok[d] > tok[c] + +SReg(c) == + /\ pc[c] = "creg" + /\ IF tok[c] > hwm + THEN /\ owner' = c /\ hwm' = tok[c] /\ rsubs' = loc[c] + /\ pc' = [pc EXCEPT ![c] = "connack"] + /\ UNCHANGED <> + ELSE /\ pc' = [pc EXCEPT ![c] = "release"] + /\ failed' = [failed EXCEPT ![c] = TRUE] + /\ abortOK' = [abortOK EXCEPT ![c] = NewerActive(c)] + /\ UNCHANGED <> + /\ UNCHANGED <> + /\ UNCHANGED gcV + +RReg(c, cl, e) == + /\ MECH = "CASSPLIT" /\ pc[c] = "idle" + /\ owner' = c + /\ tok' = [tok EXCEPT ![c] = nextId] + /\ nextId' = nextId + 1 + /\ clr' = [clr EXCEPT ![c] = cl] + /\ ex' = [ex EXCEPT ![c] = e] + /\ startEff' = [startEff EXCEPT ![c] = cleanEff] + /\ IF cl + THEN rsubs' = {} /\ cleanEff' = cleanEff \cup {nextId} + ELSE UNCHANGED <> + /\ pc' = [pc EXCEPT ![c] = "cstore"] + /\ UNCHANGED <> + /\ UNCHANGED gcV + +RStore(c) == + /\ pc[c] = "cstore" + /\ LET r == ReadRes + cr == CasRec(c, r, tok[c]) + IN IF r.ex /\ r.tok > tok[c] + THEN /\ pc' = [pc EXCEPT ![c] = "release"] + /\ failed' = [failed EXCEPT ![c] = TRUE] + /\ abortOK' = [abortOK EXCEPT ![c] = NewerActive(c)] + /\ UNCHANGED <> + ELSE /\ Wr(cr.rec, On("wt_claim"), DiskAfterRead) + /\ loc' = [loc EXCEPT ![c] = cr.rec.subs] + /\ spOK' = [spOK EXCEPT ![c] = SpCheck(clr[c], cr.resume)] + /\ linOK' = [linOK EXCEPT ![c] = cr.resume => \A x \in startEff[c] : r.born >= x] + /\ subOK' = [subOK EXCEPT ![c] = cr.resume => r.subs = gSubs] + /\ gEx' = TRUE /\ gEnd' = -1 /\ gExp' = ex[c] /\ gOwner' = c /\ gSubs' = cr.rec.subs + /\ cleanEff' = IF cr.resume THEN cleanEff ELSE cleanEff \cup {tok[c]} + /\ pc' = [pc EXCEPT ![c] = "cinstall"] + /\ UNCHANGED <> + /\ UNCHANGED <> + /\ UNCHANGED gcV + +RInstall(c) == + /\ pc[c] = "cinstall" + /\ rsubs' = IF owner = c THEN loc[c] ELSE rsubs + /\ pc' = [pc EXCEPT ![c] = "connack"] + /\ UNCHANGED <> + /\ UNCHANGED gcV + +ConnackOK(c) == + /\ pc[c] = "connack" + /\ AckReady(c) + /\ pc' = [pc EXCEPT ![c] = "live"] + /\ MarkAck(c) + /\ UNCHANGED <> + +ConnackFail(c) == + /\ pc[c] = "connack" + /\ failed' = [failed EXCEPT ![c] = TRUE] + /\ GhostEnd(c, ex[c]) + /\ pc' = [pc EXCEPT ![c] = "release"] + /\ UNCHANGED <> + /\ Log(c, FALSE, TRUE) + +Toggle(set, s, add) == IF add THEN set \cup {s} ELSE set \ {s} + +LSub(c, s, add) == + /\ IsLock /\ On("sub") /\ pc[c] = "live" /\ owner = c + /\ (add => s \notin loc[c]) /\ (~add => s \in loc[c]) + /\ rsubs' = Toggle(rsubs, s, add) + /\ loc' = [loc EXCEPT ![c] = Toggle(loc[c], s, add)] + /\ Owns(c) => Room + /\ IF Owns(c) + THEN /\ Wr([RV EXCEPT !.subs = Toggle(RV.subs, s, add)], On("wt_sub"), disk) + /\ gSubs' = IF gOwner = c THEN Toggle(gSubs, s, add) ELSE gSubs + ELSE UNCHANGED <> + /\ pc' = IF GC = "OFF" THEN pc ELSE [pc EXCEPT ![c] = "suback"] + /\ UNCHANGED <> + /\ Log(c, Owns(c), FALSE) + +SubAck(c) == + /\ pc[c] = "suback" + /\ AckReady(c) + /\ pc' = [pc EXCEPT ![c] = "live"] + /\ MarkAck(c) + /\ UNCHANGED <> + +SubRouter(c, s, add) == + /\ (IsCas \/ Legacy("sub")) /\ pc[c] = "live" /\ owner = c + /\ (add => s \notin loc[c]) /\ (~add => s \in loc[c]) + /\ rsubs' = Toggle(rsubs, s, add) + /\ pend' = [pend EXCEPT ![c] = s] + /\ pc' = [pc EXCEPT ![c] = IF add THEN "substore" ELSE "unsubstore"] + /\ UNCHANGED <> + /\ UNCHANGED gcV + +SubStore(c) == + /\ pc[c] \in {"substore", "unsubstore"} + /\ LET add == pc[c] = "substore" + nl == Toggle(loc[c], pend[c], add) + IN /\ loc' = [loc EXCEPT ![c] = nl] + /\ IF IsCas + THEN IF Owns(c) + THEN /\ Wr([View EXCEPT !.subs = Toggle(View.subs, pend[c], add)], On("wt_sub"), disk) + /\ gSubs' = IF gOwner = c THEN Toggle(gSubs, pend[c], add) ELSE gSubs + ELSE UNCHANGED <> + ELSE /\ Wr([ex |-> TRUE, tok |-> tok[c], subs |-> nl, exp |-> ex[c], conn |-> TRUE, + seen |-> now, gdt |-> 0, born |-> cborn[c]], On("wt_sub"), disk) + /\ gSubs' = IF gOwner = c THEN Toggle(gSubs, pend[c], add) ELSE gSubs + /\ pc' = [pc EXCEPT ![c] = "live"] + /\ UNCHANGED <> + /\ UNCHANGED gcV + +Close(c, e2) == + /\ pc[c] = "live" + /\ e2 \in IF ex[c] = 0 THEN {0} ELSE {0, ex[c]} + /\ LET w == On("disc") /\ e2 # ex[c] /\ Owns(c) + IN /\ w => Room + /\ ex' = [ex EXCEPT ![c] = e2] + /\ GhostEnd(c, e2) + /\ IF w THEN Wr([RV EXCEPT !.exp = e2], On("wt_disc"), disk) ELSE UNCHANGED storeV + /\ pc' = [pc EXCEPT ![c] = "release"] + /\ UNCHANGED <> + /\ Log(c, w, TRUE) + +LRelease(c) == + /\ IsLock /\ On("release") /\ pc[c] = "release" + /\ (owner = c /\ Owns(c)) => Room + /\ IF owner = c + THEN /\ owner' = None + /\ rsubs' = IF ex[c] = 0 THEN {} ELSE rsubs + /\ IF Owns(c) + THEN IF ex[c] = 0 THEN Remove(On("wt_remove")) + ELSE Wr(Disconnected(c), On("wt_disc"), disk) + ELSE UNCHANGED storeV + ELSE UNCHANGED <> + /\ pc' = [pc EXCEPT ![c] = IF GC = "OFF" THEN "done" ELSE "relwait"] + /\ UNCHANGED <> + /\ Log(c, owner = c /\ Owns(c), FALSE) + +RelDone(c) == + /\ pc[c] = "relwait" + /\ AckReady(c) + /\ pc' = [pc EXCEPT ![c] = "done"] + /\ MarkAck(c) + /\ UNCHANGED <> + +Release(c) == + /\ Legacy("release") /\ pc[c] = "release" + /\ IF owner = c + THEN /\ owner' = None + /\ pc' = [pc EXCEPT ![c] = IF ex[c] = 0 THEN "strip" ELSE "cleanup"] + ELSE /\ UNCHANGED owner + /\ pc' = [pc EXCEPT ![c] = "done"] + /\ UNCHANGED <> + /\ UNCHANGED gcV + +Strip(c) == + /\ pc[c] = "strip" + /\ rsubs' = {} + /\ pc' = [pc EXCEPT ![c] = "cleanup"] + /\ UNCHANGED <> + /\ UNCHANGED gcV + +Cleanup(c) == + /\ pc[c] = "cleanup" + /\ IF Owns(c) THEN Wr(Disconnected(c), On("wt_disc"), disk) ELSE UNCHANGED storeV + /\ pc' = [pc EXCEPT ![c] = IF ex[c] = 0 THEN "rmowned" ELSE "done"] + /\ UNCHANGED <> + /\ UNCHANGED gcV + +RmOwned(c) == + /\ pc[c] = "rmowned" + /\ IF Owns(c) THEN Remove(On("wt_remove")) ELSE UNCHANGED storeV + /\ pc' = [pc EXCEPT ![c] = "done"] + /\ UNCHANGED <> + /\ UNCHANGED gcV + +CRelease(c) == + /\ IsCas /\ pc[c] = "release" + /\ IF owner = c + THEN /\ owner' = None + /\ rsubs' = IF ex[c] = 0 THEN {} ELSE rsubs + ELSE UNCHANGED <> + /\ pc' = [pc EXCEPT ![c] = "cclean"] + /\ UNCHANGED <> + /\ UNCHANGED gcV + +CClean(c) == + /\ pc[c] = "cclean" + /\ IF Owns(c) + THEN IF ex[c] = 0 THEN Remove(On("wt_remove")) + ELSE Wr(Disconnected(c), On("wt_disc"), disk) + ELSE UNCHANGED storeV + /\ pc' = [pc EXCEPT ![c] = "done"] + /\ UNCHANGED <> + /\ UNCHANGED gcV + +LSweep == + /\ IsLock /\ On("sweep") + /\ owner = None + /\ \/ RV.ex /\ Expired(RV) + \/ ~RV.ex /\ rsubs # {} + /\ Room + /\ Remove(On("wt_remove")) + /\ rsubs' = {} + /\ UNCHANGED <> + /\ Log(None, TRUE, FALSE) + +FileSweepCond == disk.ex /\ IF cache.ex THEN Expired(cache) ELSE Expired(Load(disk)) + +FileSweepRead == + /\ Legacy("sweep") + /\ ~swPend /\ FileSweepCond + /\ swPend' = TRUE + /\ UNCHANGED <> + /\ UNCHANGED gcV + +FileSweepApply == + /\ swPend + /\ cache' = NoRec /\ disk' = NoDisk /\ dirty' = FALSE + /\ swPend' = FALSE + /\ UNCHANGED <> + /\ UNCHANGED gcV + +CFileSweep == + /\ IsCas + /\ View.ex /\ Expired(View) + /\ Remove(On("wt_remove")) + /\ UNCHANGED <> + /\ UNCHANGED gcV + +RouterSweepDecide == + /\ IsCas \/ Legacy("sweep") + /\ owner = None /\ rsubs # {} /\ rsPend = 0 + /\ ~ReadRes.ex + /\ ExpireOnRead + /\ rsPend' = nextId + /\ UNCHANGED <> + /\ UNCHANGED gcV + +RouterSweepApply == + /\ rsPend > 0 + /\ rsubs' = IF ~IsCas \/ (owner = None /\ nextId = rsPend) THEN {} ELSE rsubs + /\ rsPend' = 0 + /\ UNCHANGED <> + /\ UNCHANGED gcV + +Flush == + /\ GC = "OFF" /\ dirty + /\ disk' = IF cache.ex THEN ToDisk(cache) ELSE NoDisk + /\ dirty' = FALSE + /\ UNCHANGED <> + /\ UNCHANGED gcV + +GFlushPrefix(k) == + /\ GC \in {"WHOLE", "PREFIX", "ACKEARLY", "NORYW"} + /\ k \in 1..Len(pbuf) + /\ GC = "WHOLE" => k = Len(pbuf) + /\ disk' = pbuf[k].r + /\ dg' = pbuf[k].g + /\ pbuf' = SubSeq(pbuf, k + 1, Len(pbuf)) + /\ dirty' = (k < Len(pbuf)) + /\ wpos' = [c \in Conns |-> IF wpos[c] <= k THEN 0 ELSE wpos[c] - k] + /\ UNCHANGED <> + +FlushedPrefix(q, lim) == + CHOOSE j \in 0..lim : (\A l \in 1..j : q[l].f) /\ (j = lim \/ ~q[j + 1].f) + +GFlushOne(i) == + /\ GC \in {"PERWRITER", "PERWRITER_MONO"} + /\ i \in 1..Len(pbuf) /\ ~pbuf[i].f + /\ LET q == [pbuf EXCEPT ![i].f = TRUE] + m == IF GC = "PERWRITER_MONO" THEN i ELSE FlushedPrefix(q, i) + IN /\ disk' = pbuf[i].r + /\ dg' = pbuf[i].g + /\ pbuf' = SubSeq(q, m + 1, Len(q)) + /\ dpos' = i - m + /\ dirty' = \E l \in (m + 1)..Len(q) : ~q[l].f + /\ wpos' = [c \in Conns |-> IF wpos[c] <= m THEN 0 ELSE wpos[c] - m] + /\ UNCHANGED <> + +GFlush == (\E k \in 1..Len(pbuf) : GFlushPrefix(k)) \/ (\E i \in 1..Len(pbuf) : GFlushOne(i)) + +Persist == Flush \/ GFlush + +TrueEnd == + IF GC # "OFF" THEN (IF disk.ts = -1 THEN now ELSE disk.gdt) + ELSE IF cache.ex THEN (IF cache.conn THEN now ELSE cache.gdt) ELSE disk.gdt + +AckedLost == {i \in DOMAIN pbuf : pbuf[i].a /\ i > dpos} + +CrashBase == + IF GC = "OFF" THEN GNow + ELSE IF AckedLost = {} THEN dg + ELSE pbuf[CHOOSE i \in AckedLost : \A j \in AckedLost : j <= i].g + +Restart == + /\ boots < MaxBoots + /\ boots' = boots + 1 + /\ bootT' = now + /\ pc' = [c \in Conns |-> IF pc[c] \in {"idle", "done"} THEN pc[c] ELSE "dead"] + /\ owner' = None + /\ hwm' = 0 + /\ rsubs' = IF On("rebuild") /\ disk.ex /\ disk.exp > 0 THEN disk.subs ELSE {} + /\ rsPend' = 0 + /\ swPend' = FALSE + /\ cache' = NoRec + /\ dirty' = FALSE + /\ disk' = IF disk.ex + THEN [disk EXCEPT !.gdt = TrueEnd, + !.ts = IF On("stamp") /\ disk.ts = -1 THEN now ELSE disk.ts] + ELSE disk + /\ LET b == CrashBase + IN /\ gEnd' = IF b.ex /\ b.end = -1 THEN now ELSE b.end + /\ gEx' = (b.ex /\ b.exp > 0) + /\ gExp' = b.exp + /\ gSubs' = b.subs + /\ cleanEff' = b.clean + /\ gOwner' = None + /\ UNCHANGED <> + /\ IF GC = "OFF" + THEN UNCHANGED gcV + ELSE /\ pbuf' = <<>> + /\ dg' = GNext + /\ dpos' = 0 + /\ wpos' = [c \in Conns |-> 0] + /\ ackSafe' = (ackSafe /\ AckedLost = {}) + +Tick == + /\ now < MaxT + /\ \A c \in Conns : pc[c] \notin Closing + /\ \A i \in DOMAIN pbuf : pbuf[i].f + /\ now' = now + 1 + /\ UNCHANGED <> + /\ UNCHANGED gcV + +ReleaseStep(c) == + \/ LRelease(c) \/ Release(c) \/ Strip(c) \/ Cleanup(c) \/ RmOwned(c) \/ FailRel(c) + \/ CRelease(c) \/ CClean(c) + +Handler(c) == + \/ HSStore(c) \/ HConnackOK(c) \/ Register(c) \/ Install(c) + \/ SReg(c) \/ RStore(c) \/ RInstall(c) \/ ConnackOK(c) \/ SubStore(c) + \/ SubAck(c) \/ RelDone(c) + \/ ReleaseStep(c) + +Sweeps == + \/ LSweep \/ FileSweepRead \/ FileSweepApply \/ CFileSweep + \/ RouterSweepDecide \/ RouterSweepApply + +Next == + \/ \E c \in Conns, cl \in BOOLEAN, e \in {0, 1} : + LClaim(c, cl, e) \/ HSRead(c, cl, e) \/ SStore(c, cl, e) \/ RReg(c, cl, e) + \/ \E c \in Conns : Handler(c) \/ HConnackFail(c) \/ ConnackFail(c) + \/ \E c \in Conns, s \in Subs, add \in BOOLEAN : LSub(c, s, add) \/ SubRouter(c, s, add) + \/ \E c \in Conns, e2 \in {0, 1} : Close(c, e2) + \/ Sweeps \/ Persist \/ Restart \/ Tick + +SysNext == (\E c \in Conns : Handler(c)) \/ Sweeps \/ Persist + +Spec == Init /\ [][Next]_vars + /\ WF_vars(\E c \in Conns : Handler(c)) /\ WF_vars(Sweeps) + /\ WF_vars(Persist) /\ WF_vars(Tick) + +SpecNoSweepFair == Init /\ [][Next]_vars + /\ WF_vars(\E c \in Conns : Handler(c)) + /\ WF_vars(Persist) /\ WF_vars(Tick) + +Sym == Permutations(Conns) + +InvOwnershipAgree == + \A c, d \in Conns : + (pc[c] \notin Ended /\ pc[d] \notin Ended /\ owner = c /\ OwnsNow(d)) => c = d + +InvNoLeakedConnected == (View.ex /\ View.conn) => Holder(View.tok) + +InvLiveOwnerSessionKept == + \A c \in Conns : + (owner = c /\ pc[c] \in LivePC) => (View.ex /\ View.conn /\ View.tok >= tok[c]) + +InvFailedHandshakeHarmless == + \A c, d \in Conns : + (failed[c] /\ pc[c] = "done" /\ owner = d /\ pc[d] \in LivePC) + => (View.ex /\ View.conn /\ View.tok # tok[c]) + +InvSessionPresent == \A c \in Conns : spOK[c] + +InvNoResurrection == \A c \in Conns : linOK[c] + +InvAckedSubsDurable == \A c \in Conns : subOK[c] + +InvAbortJustified == \A c \in Conns : abortOK[c] + +InvCleanStart == + \A c \in Conns : + (owner = c /\ pc[c] = "live" /\ clr[c]) + => (rsubs \subseteq loc[c] /\ ((View.ex /\ View.tok = tok[c]) => View.subs \subseteq loc[c])) + +InvRouterMirrors == + rsubs = IF owner # None THEN loc[owner] + ELSE IF View.ex /\ View.exp > 0 THEN View.subs ELSE {} + +Quiet == /\ \A c \in Conns : pc[c] \in Ended \cup {"live"} + /\ rsPend = 0 /\ ~swPend + +InvQuiescentConsistent == + \A c \in Conns : + (Quiet /\ owner = c /\ pc[c] = "live") + => (rsubs = loc[c] /\ View.ex /\ View.subs = loc[c]) + +InvExpiryExact == (View.ex /\ ~View.conn) => View.seen = View.gdt + +StaleSession == + /\ View.ex + /\ ~Holder(View.tok) + /\ (View.conn \/ now > View.gdt + Dur(View.exp)) + +StaleRoutes == owner = None /\ ~View.ex /\ rsubs # {} + +Stale == StaleSession \/ StaleRoutes + +InvProgress == (now = MaxT /\ ~ENABLED SysNext) => ~Stale + +InvAckedStateSurvivesCrash == ackSafe + +InvPendNotFull == Len(pbuf) < MaxPend + +Acked(c) == ~failed[c] /\ pc[c] \in LivePC \cup {"release", "relwait", "done"} + +InvAckedClaimCorrect == \A c \in Conns : Acked(c) => (spOK[c] /\ linOK[c] /\ subOK[c]) + +EventuallyClean == []<>(now < MaxT \/ ~Stale) + +NegTimeNeverEnds == []<>(now < MaxT) + +NegNoResumeAfterRestart == + ~(\E c, d \in Conns : pc[c] = "connack" /\ ~clr[c] /\ loc[c] # {} + /\ pc[d] = "dead" /\ tok[d] = cborn[c]) + +NegNoResumingTakeover == + ~(\E c, d \in Conns : c # d /\ pc[c] = "live" /\ pc[d] = "connack" /\ owner = d /\ ~clr[d] + /\ loc[d] # {}) + +============================================================================= diff --git a/specs/tla/session-ownership/SessionOwnership_ABL_claim.cfg b/specs/tla/session-ownership/SessionOwnership_ABL_claim.cfg new file mode 100644 index 00000000..51e0f6fc --- /dev/null +++ b/specs/tla/session-ownership/SessionOwnership_ABL_claim.cfg @@ -0,0 +1,33 @@ +SPECIFICATION Spec + +CONSTANTS + Conns = {c1, c2, c3} + Subs = {s1} + None = none + MaxT = 2 + E = 1 + MaxBoots = 1 + MECH = "LOCK" + DROP = {"claim"} + GC = "OFF" + MaxPend = 1 + +SYMMETRY Sym + +INVARIANTS + TypeOK + InvOwnershipAgree + InvNoLeakedConnected + InvLiveOwnerSessionKept + InvFailedHandshakeHarmless + InvSessionPresent + InvNoResurrection + InvAckedSubsDurable + InvAbortJustified + InvCleanStart + InvRouterMirrors + InvQuiescentConsistent + InvExpiryExact + InvProgress + +CHECK_DEADLOCK FALSE diff --git a/specs/tla/session-ownership/SessionOwnership_ABL_disc.cfg b/specs/tla/session-ownership/SessionOwnership_ABL_disc.cfg new file mode 100644 index 00000000..63449fb5 --- /dev/null +++ b/specs/tla/session-ownership/SessionOwnership_ABL_disc.cfg @@ -0,0 +1,33 @@ +SPECIFICATION Spec + +CONSTANTS + Conns = {c1, c2, c3} + Subs = {s1} + None = none + MaxT = 2 + E = 1 + MaxBoots = 1 + MECH = "LOCK" + DROP = {"disc"} + GC = "OFF" + MaxPend = 1 + +SYMMETRY Sym + +INVARIANTS + TypeOK + InvOwnershipAgree + InvNoLeakedConnected + InvLiveOwnerSessionKept + InvFailedHandshakeHarmless + InvSessionPresent + InvNoResurrection + InvAckedSubsDurable + InvAbortJustified + InvCleanStart + InvRouterMirrors + InvQuiescentConsistent + InvExpiryExact + InvProgress + +CHECK_DEADLOCK FALSE diff --git a/specs/tla/session-ownership/SessionOwnership_ABL_rebuild.cfg b/specs/tla/session-ownership/SessionOwnership_ABL_rebuild.cfg new file mode 100644 index 00000000..d11df771 --- /dev/null +++ b/specs/tla/session-ownership/SessionOwnership_ABL_rebuild.cfg @@ -0,0 +1,33 @@ +SPECIFICATION Spec + +CONSTANTS + Conns = {c1, c2, c3} + Subs = {s1} + None = none + MaxT = 2 + E = 1 + MaxBoots = 1 + MECH = "LOCK" + DROP = {"rebuild"} + GC = "OFF" + MaxPend = 1 + +SYMMETRY Sym + +INVARIANTS + TypeOK + InvOwnershipAgree + InvNoLeakedConnected + InvLiveOwnerSessionKept + InvFailedHandshakeHarmless + InvSessionPresent + InvNoResurrection + InvAckedSubsDurable + InvAbortJustified + InvCleanStart + InvRouterMirrors + InvQuiescentConsistent + InvExpiryExact + InvProgress + +CHECK_DEADLOCK FALSE diff --git a/specs/tla/session-ownership/SessionOwnership_ABL_release.cfg b/specs/tla/session-ownership/SessionOwnership_ABL_release.cfg new file mode 100644 index 00000000..5e08cfb7 --- /dev/null +++ b/specs/tla/session-ownership/SessionOwnership_ABL_release.cfg @@ -0,0 +1,33 @@ +SPECIFICATION Spec + +CONSTANTS + Conns = {c1, c2, c3} + Subs = {s1} + None = none + MaxT = 2 + E = 1 + MaxBoots = 1 + MECH = "LOCK" + DROP = {"release"} + GC = "OFF" + MaxPend = 1 + +SYMMETRY Sym + +INVARIANTS + TypeOK + InvOwnershipAgree + InvNoLeakedConnected + InvLiveOwnerSessionKept + InvFailedHandshakeHarmless + InvSessionPresent + InvNoResurrection + InvAckedSubsDurable + InvAbortJustified + InvCleanStart + InvRouterMirrors + InvQuiescentConsistent + InvExpiryExact + InvProgress + +CHECK_DEADLOCK FALSE diff --git a/specs/tla/session-ownership/SessionOwnership_ABL_release_core.cfg b/specs/tla/session-ownership/SessionOwnership_ABL_release_core.cfg new file mode 100644 index 00000000..0587291f --- /dev/null +++ b/specs/tla/session-ownership/SessionOwnership_ABL_release_core.cfg @@ -0,0 +1,32 @@ +SPECIFICATION Spec + +CONSTANTS + Conns = {c1, c2, c3} + Subs = {s1} + None = none + MaxT = 2 + E = 1 + MaxBoots = 1 + MECH = "LOCK" + DROP = {"release"} + GC = "OFF" + MaxPend = 1 + +SYMMETRY Sym + +INVARIANTS + TypeOK + InvOwnershipAgree + InvNoLeakedConnected + InvLiveOwnerSessionKept + InvFailedHandshakeHarmless + InvSessionPresent + InvNoResurrection + InvAckedSubsDurable + InvAbortJustified + InvCleanStart + InvQuiescentConsistent + InvExpiryExact + InvProgress + +CHECK_DEADLOCK FALSE diff --git a/specs/tla/session-ownership/SessionOwnership_ABL_stamp.cfg b/specs/tla/session-ownership/SessionOwnership_ABL_stamp.cfg new file mode 100644 index 00000000..ac2d8c18 --- /dev/null +++ b/specs/tla/session-ownership/SessionOwnership_ABL_stamp.cfg @@ -0,0 +1,33 @@ +SPECIFICATION Spec + +CONSTANTS + Conns = {c1, c2, c3} + Subs = {s1} + None = none + MaxT = 2 + E = 1 + MaxBoots = 1 + MECH = "LOCK" + DROP = {"stamp"} + GC = "OFF" + MaxPend = 1 + +SYMMETRY Sym + +INVARIANTS + TypeOK + InvOwnershipAgree + InvNoLeakedConnected + InvLiveOwnerSessionKept + InvFailedHandshakeHarmless + InvSessionPresent + InvNoResurrection + InvAckedSubsDurable + InvAbortJustified + InvCleanStart + InvRouterMirrors + InvQuiescentConsistent + InvExpiryExact + InvProgress + +CHECK_DEADLOCK FALSE diff --git a/specs/tla/session-ownership/SessionOwnership_ABL_sub.cfg b/specs/tla/session-ownership/SessionOwnership_ABL_sub.cfg new file mode 100644 index 00000000..a1dbce2e --- /dev/null +++ b/specs/tla/session-ownership/SessionOwnership_ABL_sub.cfg @@ -0,0 +1,33 @@ +SPECIFICATION Spec + +CONSTANTS + Conns = {c1, c2, c3} + Subs = {s1} + None = none + MaxT = 2 + E = 1 + MaxBoots = 1 + MECH = "LOCK" + DROP = {"sub"} + GC = "OFF" + MaxPend = 1 + +SYMMETRY Sym + +INVARIANTS + TypeOK + InvOwnershipAgree + InvNoLeakedConnected + InvLiveOwnerSessionKept + InvFailedHandshakeHarmless + InvSessionPresent + InvNoResurrection + InvAckedSubsDurable + InvAbortJustified + InvCleanStart + InvRouterMirrors + InvQuiescentConsistent + InvExpiryExact + InvProgress + +CHECK_DEADLOCK FALSE diff --git a/specs/tla/session-ownership/SessionOwnership_ABL_sub_core.cfg b/specs/tla/session-ownership/SessionOwnership_ABL_sub_core.cfg new file mode 100644 index 00000000..1ce80415 --- /dev/null +++ b/specs/tla/session-ownership/SessionOwnership_ABL_sub_core.cfg @@ -0,0 +1,32 @@ +SPECIFICATION Spec + +CONSTANTS + Conns = {c1, c2, c3} + Subs = {s1} + None = none + MaxT = 2 + E = 1 + MaxBoots = 1 + MECH = "LOCK" + DROP = {"sub"} + GC = "OFF" + MaxPend = 1 + +SYMMETRY Sym + +INVARIANTS + TypeOK + InvOwnershipAgree + InvNoLeakedConnected + InvLiveOwnerSessionKept + InvFailedHandshakeHarmless + InvSessionPresent + InvNoResurrection + InvAckedSubsDurable + InvAbortJustified + InvCleanStart + InvQuiescentConsistent + InvExpiryExact + InvProgress + +CHECK_DEADLOCK FALSE diff --git a/specs/tla/session-ownership/SessionOwnership_ABL_sweep.cfg b/specs/tla/session-ownership/SessionOwnership_ABL_sweep.cfg new file mode 100644 index 00000000..8944e6ea --- /dev/null +++ b/specs/tla/session-ownership/SessionOwnership_ABL_sweep.cfg @@ -0,0 +1,33 @@ +SPECIFICATION Spec + +CONSTANTS + Conns = {c1, c2, c3} + Subs = {s1} + None = none + MaxT = 2 + E = 1 + MaxBoots = 1 + MECH = "LOCK" + DROP = {"sweep"} + GC = "OFF" + MaxPend = 1 + +SYMMETRY Sym + +INVARIANTS + TypeOK + InvOwnershipAgree + InvNoLeakedConnected + InvLiveOwnerSessionKept + InvFailedHandshakeHarmless + InvSessionPresent + InvNoResurrection + InvAckedSubsDurable + InvAbortJustified + InvCleanStart + InvRouterMirrors + InvQuiescentConsistent + InvExpiryExact + InvProgress + +CHECK_DEADLOCK FALSE diff --git a/specs/tla/session-ownership/SessionOwnership_ABL_wt_claim.cfg b/specs/tla/session-ownership/SessionOwnership_ABL_wt_claim.cfg new file mode 100644 index 00000000..67e3aa94 --- /dev/null +++ b/specs/tla/session-ownership/SessionOwnership_ABL_wt_claim.cfg @@ -0,0 +1,33 @@ +SPECIFICATION Spec + +CONSTANTS + Conns = {c1, c2, c3} + Subs = {s1} + None = none + MaxT = 2 + E = 1 + MaxBoots = 1 + MECH = "LOCK" + DROP = {"wt_claim"} + GC = "OFF" + MaxPend = 1 + +SYMMETRY Sym + +INVARIANTS + TypeOK + InvOwnershipAgree + InvNoLeakedConnected + InvLiveOwnerSessionKept + InvFailedHandshakeHarmless + InvSessionPresent + InvNoResurrection + InvAckedSubsDurable + InvAbortJustified + InvCleanStart + InvRouterMirrors + InvQuiescentConsistent + InvExpiryExact + InvProgress + +CHECK_DEADLOCK FALSE diff --git a/specs/tla/session-ownership/SessionOwnership_ABL_wt_claim_resurrect.cfg b/specs/tla/session-ownership/SessionOwnership_ABL_wt_claim_resurrect.cfg new file mode 100644 index 00000000..ffe2b3dd --- /dev/null +++ b/specs/tla/session-ownership/SessionOwnership_ABL_wt_claim_resurrect.cfg @@ -0,0 +1,21 @@ +SPECIFICATION Spec + +CONSTANTS + Conns = {c1, c2, c3} + Subs = {s1} + None = none + MaxT = 2 + E = 1 + MaxBoots = 1 + MECH = "LOCK" + DROP = {"wt_claim"} + GC = "OFF" + MaxPend = 1 + +SYMMETRY Sym + +INVARIANTS + TypeOK + InvNoResurrection + +CHECK_DEADLOCK FALSE diff --git a/specs/tla/session-ownership/SessionOwnership_ABL_wt_disc.cfg b/specs/tla/session-ownership/SessionOwnership_ABL_wt_disc.cfg new file mode 100644 index 00000000..6148448d --- /dev/null +++ b/specs/tla/session-ownership/SessionOwnership_ABL_wt_disc.cfg @@ -0,0 +1,33 @@ +SPECIFICATION Spec + +CONSTANTS + Conns = {c1, c2, c3} + Subs = {s1} + None = none + MaxT = 2 + E = 1 + MaxBoots = 1 + MECH = "LOCK" + DROP = {"wt_disc"} + GC = "OFF" + MaxPend = 1 + +SYMMETRY Sym + +INVARIANTS + TypeOK + InvOwnershipAgree + InvNoLeakedConnected + InvLiveOwnerSessionKept + InvFailedHandshakeHarmless + InvSessionPresent + InvNoResurrection + InvAckedSubsDurable + InvAbortJustified + InvCleanStart + InvRouterMirrors + InvQuiescentConsistent + InvExpiryExact + InvProgress + +CHECK_DEADLOCK FALSE diff --git a/specs/tla/session-ownership/SessionOwnership_ABL_wt_remove.cfg b/specs/tla/session-ownership/SessionOwnership_ABL_wt_remove.cfg new file mode 100644 index 00000000..ba1db419 --- /dev/null +++ b/specs/tla/session-ownership/SessionOwnership_ABL_wt_remove.cfg @@ -0,0 +1,33 @@ +SPECIFICATION Spec + +CONSTANTS + Conns = {c1, c2, c3} + Subs = {s1} + None = none + MaxT = 2 + E = 1 + MaxBoots = 1 + MECH = "LOCK" + DROP = {"wt_remove"} + GC = "OFF" + MaxPend = 1 + +SYMMETRY Sym + +INVARIANTS + TypeOK + InvOwnershipAgree + InvNoLeakedConnected + InvLiveOwnerSessionKept + InvFailedHandshakeHarmless + InvSessionPresent + InvNoResurrection + InvAckedSubsDurable + InvAbortJustified + InvCleanStart + InvRouterMirrors + InvQuiescentConsistent + InvExpiryExact + InvProgress + +CHECK_DEADLOCK FALSE diff --git a/specs/tla/session-ownership/SessionOwnership_ABL_wt_remove_core.cfg b/specs/tla/session-ownership/SessionOwnership_ABL_wt_remove_core.cfg new file mode 100644 index 00000000..fbba8aad --- /dev/null +++ b/specs/tla/session-ownership/SessionOwnership_ABL_wt_remove_core.cfg @@ -0,0 +1,32 @@ +SPECIFICATION Spec + +CONSTANTS + Conns = {c1, c2, c3} + Subs = {s1} + None = none + MaxT = 2 + E = 1 + MaxBoots = 1 + MECH = "LOCK" + DROP = {"wt_remove"} + GC = "OFF" + MaxPend = 1 + +SYMMETRY Sym + +INVARIANTS + TypeOK + InvOwnershipAgree + InvNoLeakedConnected + InvLiveOwnerSessionKept + InvFailedHandshakeHarmless + InvSessionPresent + InvNoResurrection + InvAckedSubsDurable + InvAbortJustified + InvCleanStart + InvRouterMirrors + InvQuiescentConsistent + InvProgress + +CHECK_DEADLOCK FALSE diff --git a/specs/tla/session-ownership/SessionOwnership_ABL_wt_sub.cfg b/specs/tla/session-ownership/SessionOwnership_ABL_wt_sub.cfg new file mode 100644 index 00000000..fa987b15 --- /dev/null +++ b/specs/tla/session-ownership/SessionOwnership_ABL_wt_sub.cfg @@ -0,0 +1,33 @@ +SPECIFICATION Spec + +CONSTANTS + Conns = {c1, c2, c3} + Subs = {s1} + None = none + MaxT = 2 + E = 1 + MaxBoots = 1 + MECH = "LOCK" + DROP = {"wt_sub"} + GC = "OFF" + MaxPend = 1 + +SYMMETRY Sym + +INVARIANTS + TypeOK + InvOwnershipAgree + InvNoLeakedConnected + InvLiveOwnerSessionKept + InvFailedHandshakeHarmless + InvSessionPresent + InvNoResurrection + InvAckedSubsDurable + InvAbortJustified + InvCleanStart + InvRouterMirrors + InvQuiescentConsistent + InvExpiryExact + InvProgress + +CHECK_DEADLOCK FALSE diff --git a/specs/tla/session-ownership/SessionOwnership_CASSPLIT_agree.cfg b/specs/tla/session-ownership/SessionOwnership_CASSPLIT_agree.cfg new file mode 100644 index 00000000..ab02baa2 --- /dev/null +++ b/specs/tla/session-ownership/SessionOwnership_CASSPLIT_agree.cfg @@ -0,0 +1,21 @@ +SPECIFICATION Spec + +CONSTANTS + Conns = {c1, c2, c3} + Subs = {s1} + None = none + MaxT = 2 + E = 1 + MaxBoots = 1 + MECH = "CASSPLIT" + DROP = {} + GC = "OFF" + MaxPend = 1 + +SYMMETRY Sym + +INVARIANTS + TypeOK + InvOwnershipAgree + +CHECK_DEADLOCK FALSE diff --git a/specs/tla/session-ownership/SessionOwnership_CASSPLIT_resurrect.cfg b/specs/tla/session-ownership/SessionOwnership_CASSPLIT_resurrect.cfg new file mode 100644 index 00000000..c1ef2ef2 --- /dev/null +++ b/specs/tla/session-ownership/SessionOwnership_CASSPLIT_resurrect.cfg @@ -0,0 +1,21 @@ +SPECIFICATION Spec + +CONSTANTS + Conns = {c1, c2, c3} + Subs = {s1} + None = none + MaxT = 2 + E = 1 + MaxBoots = 1 + MECH = "CASSPLIT" + DROP = {} + GC = "OFF" + MaxPend = 1 + +SYMMETRY Sym + +INVARIANTS + TypeOK + InvNoResurrection + +CHECK_DEADLOCK FALSE diff --git a/specs/tla/session-ownership/SessionOwnership_CAS_agree.cfg b/specs/tla/session-ownership/SessionOwnership_CAS_agree.cfg new file mode 100644 index 00000000..e63b6390 --- /dev/null +++ b/specs/tla/session-ownership/SessionOwnership_CAS_agree.cfg @@ -0,0 +1,21 @@ +SPECIFICATION Spec + +CONSTANTS + Conns = {c1, c2, c3} + Subs = {s1} + None = none + MaxT = 2 + E = 1 + MaxBoots = 1 + MECH = "CAS" + DROP = {} + GC = "OFF" + MaxPend = 1 + +SYMMETRY Sym + +INVARIANTS + TypeOK + InvOwnershipAgree + +CHECK_DEADLOCK FALSE diff --git a/specs/tla/session-ownership/SessionOwnership_CAS_mirrors.cfg b/specs/tla/session-ownership/SessionOwnership_CAS_mirrors.cfg new file mode 100644 index 00000000..f379c58b --- /dev/null +++ b/specs/tla/session-ownership/SessionOwnership_CAS_mirrors.cfg @@ -0,0 +1,21 @@ +SPECIFICATION Spec + +CONSTANTS + Conns = {c1, c2, c3} + Subs = {s1} + None = none + MaxT = 2 + E = 1 + MaxBoots = 1 + MECH = "CAS" + DROP = {} + GC = "OFF" + MaxPend = 1 + +SYMMETRY Sym + +INVARIANTS + TypeOK + InvRouterMirrors + +CHECK_DEADLOCK FALSE diff --git a/specs/tla/session-ownership/SessionOwnership_CAS_rest.cfg b/specs/tla/session-ownership/SessionOwnership_CAS_rest.cfg new file mode 100644 index 00000000..5d5ceaf3 --- /dev/null +++ b/specs/tla/session-ownership/SessionOwnership_CAS_rest.cfg @@ -0,0 +1,31 @@ +SPECIFICATION Spec + +CONSTANTS + Conns = {c1, c2, c3} + Subs = {s1} + None = none + MaxT = 2 + E = 1 + MaxBoots = 1 + MECH = "CAS" + DROP = {} + GC = "OFF" + MaxPend = 1 + +SYMMETRY Sym + +INVARIANTS + TypeOK + InvNoLeakedConnected + InvLiveOwnerSessionKept + InvFailedHandshakeHarmless + InvSessionPresent + InvNoResurrection + InvAckedSubsDurable + InvAbortJustified + InvCleanStart + InvQuiescentConsistent + InvExpiryExact + InvProgress + +CHECK_DEADLOCK FALSE diff --git a/specs/tla/session-ownership/SessionOwnership_CUR_AckedSubsDurable.cfg b/specs/tla/session-ownership/SessionOwnership_CUR_AckedSubsDurable.cfg new file mode 100644 index 00000000..f057a93b --- /dev/null +++ b/specs/tla/session-ownership/SessionOwnership_CUR_AckedSubsDurable.cfg @@ -0,0 +1,21 @@ +SPECIFICATION Spec + +CONSTANTS + Conns = {c1, c2, c3} + Subs = {s1} + None = none + MaxT = 2 + E = 1 + MaxBoots = 1 + MECH = "CURRENT" + DROP = {} + GC = "OFF" + MaxPend = 1 + +SYMMETRY Sym + +INVARIANTS + TypeOK + InvAckedSubsDurable + +CHECK_DEADLOCK FALSE diff --git a/specs/tla/session-ownership/SessionOwnership_CUR_CleanStart.cfg b/specs/tla/session-ownership/SessionOwnership_CUR_CleanStart.cfg new file mode 100644 index 00000000..4cc93e16 --- /dev/null +++ b/specs/tla/session-ownership/SessionOwnership_CUR_CleanStart.cfg @@ -0,0 +1,21 @@ +SPECIFICATION Spec + +CONSTANTS + Conns = {c1, c2, c3} + Subs = {s1} + None = none + MaxT = 2 + E = 1 + MaxBoots = 1 + MECH = "CURRENT" + DROP = {} + GC = "OFF" + MaxPend = 1 + +SYMMETRY Sym + +INVARIANTS + TypeOK + InvCleanStart + +CHECK_DEADLOCK FALSE diff --git a/specs/tla/session-ownership/SessionOwnership_CUR_ExpiryExact.cfg b/specs/tla/session-ownership/SessionOwnership_CUR_ExpiryExact.cfg new file mode 100644 index 00000000..57a48017 --- /dev/null +++ b/specs/tla/session-ownership/SessionOwnership_CUR_ExpiryExact.cfg @@ -0,0 +1,21 @@ +SPECIFICATION Spec + +CONSTANTS + Conns = {c1, c2, c3} + Subs = {s1} + None = none + MaxT = 2 + E = 1 + MaxBoots = 1 + MECH = "CURRENT" + DROP = {} + GC = "OFF" + MaxPend = 1 + +SYMMETRY Sym + +INVARIANTS + TypeOK + InvExpiryExact + +CHECK_DEADLOCK FALSE diff --git a/specs/tla/session-ownership/SessionOwnership_CUR_FailedHandshakeHarmless.cfg b/specs/tla/session-ownership/SessionOwnership_CUR_FailedHandshakeHarmless.cfg new file mode 100644 index 00000000..b0e56a0b --- /dev/null +++ b/specs/tla/session-ownership/SessionOwnership_CUR_FailedHandshakeHarmless.cfg @@ -0,0 +1,21 @@ +SPECIFICATION Spec + +CONSTANTS + Conns = {c1, c2, c3} + Subs = {s1} + None = none + MaxT = 2 + E = 1 + MaxBoots = 1 + MECH = "CURRENT" + DROP = {} + GC = "OFF" + MaxPend = 1 + +SYMMETRY Sym + +INVARIANTS + TypeOK + InvFailedHandshakeHarmless + +CHECK_DEADLOCK FALSE diff --git a/specs/tla/session-ownership/SessionOwnership_CUR_LiveOwnerSessionKept.cfg b/specs/tla/session-ownership/SessionOwnership_CUR_LiveOwnerSessionKept.cfg new file mode 100644 index 00000000..57439a78 --- /dev/null +++ b/specs/tla/session-ownership/SessionOwnership_CUR_LiveOwnerSessionKept.cfg @@ -0,0 +1,21 @@ +SPECIFICATION Spec + +CONSTANTS + Conns = {c1, c2, c3} + Subs = {s1} + None = none + MaxT = 2 + E = 1 + MaxBoots = 1 + MECH = "CURRENT" + DROP = {} + GC = "OFF" + MaxPend = 1 + +SYMMETRY Sym + +INVARIANTS + TypeOK + InvLiveOwnerSessionKept + +CHECK_DEADLOCK FALSE diff --git a/specs/tla/session-ownership/SessionOwnership_CUR_NoLeakedConnected.cfg b/specs/tla/session-ownership/SessionOwnership_CUR_NoLeakedConnected.cfg new file mode 100644 index 00000000..1af7930f --- /dev/null +++ b/specs/tla/session-ownership/SessionOwnership_CUR_NoLeakedConnected.cfg @@ -0,0 +1,21 @@ +SPECIFICATION Spec + +CONSTANTS + Conns = {c1, c2, c3} + Subs = {s1} + None = none + MaxT = 2 + E = 1 + MaxBoots = 1 + MECH = "CURRENT" + DROP = {} + GC = "OFF" + MaxPend = 1 + +SYMMETRY Sym + +INVARIANTS + TypeOK + InvNoLeakedConnected + +CHECK_DEADLOCK FALSE diff --git a/specs/tla/session-ownership/SessionOwnership_CUR_NoResurrection.cfg b/specs/tla/session-ownership/SessionOwnership_CUR_NoResurrection.cfg new file mode 100644 index 00000000..64a7a970 --- /dev/null +++ b/specs/tla/session-ownership/SessionOwnership_CUR_NoResurrection.cfg @@ -0,0 +1,21 @@ +SPECIFICATION Spec + +CONSTANTS + Conns = {c1, c2, c3} + Subs = {s1} + None = none + MaxT = 2 + E = 1 + MaxBoots = 1 + MECH = "CURRENT" + DROP = {} + GC = "OFF" + MaxPend = 1 + +SYMMETRY Sym + +INVARIANTS + TypeOK + InvNoResurrection + +CHECK_DEADLOCK FALSE diff --git a/specs/tla/session-ownership/SessionOwnership_CUR_OwnershipAgree.cfg b/specs/tla/session-ownership/SessionOwnership_CUR_OwnershipAgree.cfg new file mode 100644 index 00000000..b73c86db --- /dev/null +++ b/specs/tla/session-ownership/SessionOwnership_CUR_OwnershipAgree.cfg @@ -0,0 +1,21 @@ +SPECIFICATION Spec + +CONSTANTS + Conns = {c1, c2, c3} + Subs = {s1} + None = none + MaxT = 2 + E = 1 + MaxBoots = 1 + MECH = "CURRENT" + DROP = {} + GC = "OFF" + MaxPend = 1 + +SYMMETRY Sym + +INVARIANTS + TypeOK + InvOwnershipAgree + +CHECK_DEADLOCK FALSE diff --git a/specs/tla/session-ownership/SessionOwnership_CUR_Progress.cfg b/specs/tla/session-ownership/SessionOwnership_CUR_Progress.cfg new file mode 100644 index 00000000..a7220058 --- /dev/null +++ b/specs/tla/session-ownership/SessionOwnership_CUR_Progress.cfg @@ -0,0 +1,21 @@ +SPECIFICATION Spec + +CONSTANTS + Conns = {c1, c2, c3} + Subs = {s1} + None = none + MaxT = 2 + E = 1 + MaxBoots = 1 + MECH = "CURRENT" + DROP = {} + GC = "OFF" + MaxPend = 1 + +SYMMETRY Sym + +INVARIANTS + TypeOK + InvProgress + +CHECK_DEADLOCK FALSE diff --git a/specs/tla/session-ownership/SessionOwnership_CUR_QuiescentConsistent.cfg b/specs/tla/session-ownership/SessionOwnership_CUR_QuiescentConsistent.cfg new file mode 100644 index 00000000..5f65c8b0 --- /dev/null +++ b/specs/tla/session-ownership/SessionOwnership_CUR_QuiescentConsistent.cfg @@ -0,0 +1,21 @@ +SPECIFICATION Spec + +CONSTANTS + Conns = {c1, c2, c3} + Subs = {s1} + None = none + MaxT = 2 + E = 1 + MaxBoots = 1 + MECH = "CURRENT" + DROP = {} + GC = "OFF" + MaxPend = 1 + +SYMMETRY Sym + +INVARIANTS + TypeOK + InvQuiescentConsistent + +CHECK_DEADLOCK FALSE diff --git a/specs/tla/session-ownership/SessionOwnership_CUR_RouterMirrors.cfg b/specs/tla/session-ownership/SessionOwnership_CUR_RouterMirrors.cfg new file mode 100644 index 00000000..bb4e1f7c --- /dev/null +++ b/specs/tla/session-ownership/SessionOwnership_CUR_RouterMirrors.cfg @@ -0,0 +1,21 @@ +SPECIFICATION Spec + +CONSTANTS + Conns = {c1, c2, c3} + Subs = {s1} + None = none + MaxT = 2 + E = 1 + MaxBoots = 1 + MECH = "CURRENT" + DROP = {} + GC = "OFF" + MaxPend = 1 + +SYMMETRY Sym + +INVARIANTS + TypeOK + InvRouterMirrors + +CHECK_DEADLOCK FALSE diff --git a/specs/tla/session-ownership/SessionOwnership_CUR_SessionPresent.cfg b/specs/tla/session-ownership/SessionOwnership_CUR_SessionPresent.cfg new file mode 100644 index 00000000..2926db33 --- /dev/null +++ b/specs/tla/session-ownership/SessionOwnership_CUR_SessionPresent.cfg @@ -0,0 +1,21 @@ +SPECIFICATION Spec + +CONSTANTS + Conns = {c1, c2, c3} + Subs = {s1} + None = none + MaxT = 2 + E = 1 + MaxBoots = 1 + MECH = "CURRENT" + DROP = {} + GC = "OFF" + MaxPend = 1 + +SYMMETRY Sym + +INVARIANTS + TypeOK + InvSessionPresent + +CHECK_DEADLOCK FALSE diff --git a/specs/tla/session-ownership/SessionOwnership_CUR_live.cfg b/specs/tla/session-ownership/SessionOwnership_CUR_live.cfg new file mode 100644 index 00000000..2f296610 --- /dev/null +++ b/specs/tla/session-ownership/SessionOwnership_CUR_live.cfg @@ -0,0 +1,18 @@ +SPECIFICATION Spec + +CONSTANTS + Conns = {c1, c2, c3} + Subs = {s1} + None = none + MaxT = 2 + E = 1 + MaxBoots = 1 + MECH = "CURRENT" + DROP = {} + GC = "OFF" + MaxPend = 1 + +PROPERTIES + EventuallyClean + +CHECK_DEADLOCK FALSE diff --git a/specs/tla/session-ownership/SessionOwnership_GC_ABL_ackearly.cfg b/specs/tla/session-ownership/SessionOwnership_GC_ABL_ackearly.cfg new file mode 100644 index 00000000..fd3961cb --- /dev/null +++ b/specs/tla/session-ownership/SessionOwnership_GC_ABL_ackearly.cfg @@ -0,0 +1,35 @@ +SPECIFICATION Spec + +CONSTANTS + Conns = {c1, c2, c3} + Subs = {s1} + None = none + MaxT = 2 + E = 1 + MaxBoots = 1 + MECH = "LOCK" + DROP = {} + GC = "ACKEARLY" + MaxPend = 8 + +SYMMETRY Sym + +INVARIANTS + TypeOK + InvOwnershipAgree + InvNoLeakedConnected + InvLiveOwnerSessionKept + InvFailedHandshakeHarmless + InvSessionPresent + InvNoResurrection + InvAckedSubsDurable + InvAbortJustified + InvCleanStart + InvRouterMirrors + InvQuiescentConsistent + InvExpiryExact + InvProgress + InvAckedStateSurvivesCrash + InvPendNotFull + +CHECK_DEADLOCK FALSE diff --git a/specs/tla/session-ownership/SessionOwnership_GC_ABL_ackearly_core.cfg b/specs/tla/session-ownership/SessionOwnership_GC_ABL_ackearly_core.cfg new file mode 100644 index 00000000..805f1002 --- /dev/null +++ b/specs/tla/session-ownership/SessionOwnership_GC_ABL_ackearly_core.cfg @@ -0,0 +1,33 @@ +SPECIFICATION Spec + +CONSTANTS + Conns = {c1, c2, c3} + Subs = {s1} + None = none + MaxT = 2 + E = 1 + MaxBoots = 1 + MECH = "LOCK" + DROP = {} + GC = "ACKEARLY" + MaxPend = 8 + +SYMMETRY Sym + +INVARIANTS + TypeOK + InvOwnershipAgree + InvNoLeakedConnected + InvLiveOwnerSessionKept + InvFailedHandshakeHarmless + InvSessionPresent + InvNoResurrection + InvAckedSubsDurable + InvAbortJustified + InvCleanStart + InvRouterMirrors + InvQuiescentConsistent + InvExpiryExact + InvProgress + +CHECK_DEADLOCK FALSE diff --git a/specs/tla/session-ownership/SessionOwnership_GC_ABL_noryw.cfg b/specs/tla/session-ownership/SessionOwnership_GC_ABL_noryw.cfg new file mode 100644 index 00000000..85173d3e --- /dev/null +++ b/specs/tla/session-ownership/SessionOwnership_GC_ABL_noryw.cfg @@ -0,0 +1,35 @@ +SPECIFICATION Spec + +CONSTANTS + Conns = {c1, c2, c3} + Subs = {s1} + None = none + MaxT = 2 + E = 1 + MaxBoots = 1 + MECH = "LOCK" + DROP = {} + GC = "NORYW" + MaxPend = 8 + +SYMMETRY Sym + +INVARIANTS + TypeOK + InvOwnershipAgree + InvNoLeakedConnected + InvLiveOwnerSessionKept + InvFailedHandshakeHarmless + InvSessionPresent + InvNoResurrection + InvAckedSubsDurable + InvAbortJustified + InvCleanStart + InvRouterMirrors + InvQuiescentConsistent + InvExpiryExact + InvProgress + InvAckedStateSurvivesCrash + InvPendNotFull + +CHECK_DEADLOCK FALSE diff --git a/specs/tla/session-ownership/SessionOwnership_GC_ABL_noryw_acked.cfg b/specs/tla/session-ownership/SessionOwnership_GC_ABL_noryw_acked.cfg new file mode 100644 index 00000000..ae2466b1 --- /dev/null +++ b/specs/tla/session-ownership/SessionOwnership_GC_ABL_noryw_acked.cfg @@ -0,0 +1,21 @@ +SPECIFICATION Spec + +CONSTANTS + Conns = {c1, c2, c3} + Subs = {s1} + None = none + MaxT = 2 + E = 1 + MaxBoots = 1 + MECH = "LOCK" + DROP = {} + GC = "NORYW" + MaxPend = 8 + +SYMMETRY Sym + +INVARIANTS + TypeOK + InvAckedClaimCorrect + +CHECK_DEADLOCK FALSE diff --git a/specs/tla/session-ownership/SessionOwnership_GC_ABL_noryw_core.cfg b/specs/tla/session-ownership/SessionOwnership_GC_ABL_noryw_core.cfg new file mode 100644 index 00000000..e1eea666 --- /dev/null +++ b/specs/tla/session-ownership/SessionOwnership_GC_ABL_noryw_core.cfg @@ -0,0 +1,33 @@ +SPECIFICATION Spec + +CONSTANTS + Conns = {c1, c2, c3} + Subs = {s1} + None = none + MaxT = 2 + E = 1 + MaxBoots = 1 + MECH = "LOCK" + DROP = {} + GC = "NORYW" + MaxPend = 8 + +SYMMETRY Sym + +INVARIANTS + TypeOK + InvOwnershipAgree + InvNoLeakedConnected + InvLiveOwnerSessionKept + InvFailedHandshakeHarmless + InvSessionPresent + InvNoResurrection + InvAckedSubsDurable + InvAbortJustified + InvCleanStart + InvRouterMirrors + InvQuiescentConsistent + InvExpiryExact + InvProgress + +CHECK_DEADLOCK FALSE diff --git a/specs/tla/session-ownership/SessionOwnership_GC_ABL_noryw_subs.cfg b/specs/tla/session-ownership/SessionOwnership_GC_ABL_noryw_subs.cfg new file mode 100644 index 00000000..1d7d0ed1 --- /dev/null +++ b/specs/tla/session-ownership/SessionOwnership_GC_ABL_noryw_subs.cfg @@ -0,0 +1,21 @@ +SPECIFICATION Spec + +CONSTANTS + Conns = {c1, c2, c3} + Subs = {s1} + None = none + MaxT = 2 + E = 1 + MaxBoots = 1 + MECH = "LOCK" + DROP = {} + GC = "NORYW" + MaxPend = 8 + +SYMMETRY Sym + +INVARIANTS + TypeOK + InvAckedSubsDurable + +CHECK_DEADLOCK FALSE diff --git a/specs/tla/session-ownership/SessionOwnership_GC_ABL_perwriter.cfg b/specs/tla/session-ownership/SessionOwnership_GC_ABL_perwriter.cfg new file mode 100644 index 00000000..83f18d70 --- /dev/null +++ b/specs/tla/session-ownership/SessionOwnership_GC_ABL_perwriter.cfg @@ -0,0 +1,35 @@ +SPECIFICATION Spec + +CONSTANTS + Conns = {c1, c2, c3} + Subs = {s1} + None = none + MaxT = 2 + E = 1 + MaxBoots = 1 + MECH = "LOCK" + DROP = {} + GC = "PERWRITER" + MaxPend = 8 + +SYMMETRY Sym + +INVARIANTS + TypeOK + InvOwnershipAgree + InvNoLeakedConnected + InvLiveOwnerSessionKept + InvFailedHandshakeHarmless + InvSessionPresent + InvNoResurrection + InvAckedSubsDurable + InvAbortJustified + InvCleanStart + InvRouterMirrors + InvQuiescentConsistent + InvExpiryExact + InvProgress + InvAckedStateSurvivesCrash + InvPendNotFull + +CHECK_DEADLOCK FALSE diff --git a/specs/tla/session-ownership/SessionOwnership_GC_ABL_perwriter_core.cfg b/specs/tla/session-ownership/SessionOwnership_GC_ABL_perwriter_core.cfg new file mode 100644 index 00000000..7b037295 --- /dev/null +++ b/specs/tla/session-ownership/SessionOwnership_GC_ABL_perwriter_core.cfg @@ -0,0 +1,33 @@ +SPECIFICATION Spec + +CONSTANTS + Conns = {c1, c2, c3} + Subs = {s1} + None = none + MaxT = 2 + E = 1 + MaxBoots = 1 + MECH = "LOCK" + DROP = {} + GC = "PERWRITER" + MaxPend = 8 + +SYMMETRY Sym + +INVARIANTS + TypeOK + InvOwnershipAgree + InvNoLeakedConnected + InvLiveOwnerSessionKept + InvFailedHandshakeHarmless + InvSessionPresent + InvNoResurrection + InvAckedSubsDurable + InvAbortJustified + InvCleanStart + InvRouterMirrors + InvQuiescentConsistent + InvExpiryExact + InvProgress + +CHECK_DEADLOCK FALSE diff --git a/specs/tla/session-ownership/SessionOwnership_GC_perwriter_mono.cfg b/specs/tla/session-ownership/SessionOwnership_GC_perwriter_mono.cfg new file mode 100644 index 00000000..cad1f552 --- /dev/null +++ b/specs/tla/session-ownership/SessionOwnership_GC_perwriter_mono.cfg @@ -0,0 +1,35 @@ +SPECIFICATION Spec + +CONSTANTS + Conns = {c1, c2, c3} + Subs = {s1} + None = none + MaxT = 2 + E = 1 + MaxBoots = 1 + MECH = "LOCK" + DROP = {} + GC = "PERWRITER_MONO" + MaxPend = 8 + +SYMMETRY Sym + +INVARIANTS + TypeOK + InvOwnershipAgree + InvNoLeakedConnected + InvLiveOwnerSessionKept + InvFailedHandshakeHarmless + InvSessionPresent + InvNoResurrection + InvAckedSubsDurable + InvAbortJustified + InvCleanStart + InvRouterMirrors + InvQuiescentConsistent + InvExpiryExact + InvProgress + InvAckedStateSurvivesCrash + InvPendNotFull + +CHECK_DEADLOCK FALSE diff --git a/specs/tla/session-ownership/SessionOwnership_GC_prefix.cfg b/specs/tla/session-ownership/SessionOwnership_GC_prefix.cfg new file mode 100644 index 00000000..5b173227 --- /dev/null +++ b/specs/tla/session-ownership/SessionOwnership_GC_prefix.cfg @@ -0,0 +1,35 @@ +SPECIFICATION Spec + +CONSTANTS + Conns = {c1, c2, c3} + Subs = {s1, s2} + None = none + MaxT = 3 + E = 1 + MaxBoots = 1 + MECH = "LOCK" + DROP = {} + GC = "PREFIX" + MaxPend = 8 + +SYMMETRY Sym + +INVARIANTS + TypeOK + InvOwnershipAgree + InvNoLeakedConnected + InvLiveOwnerSessionKept + InvFailedHandshakeHarmless + InvSessionPresent + InvNoResurrection + InvAckedSubsDurable + InvAbortJustified + InvCleanStart + InvRouterMirrors + InvQuiescentConsistent + InvExpiryExact + InvProgress + InvAckedStateSurvivesCrash + InvPendNotFull + +CHECK_DEADLOCK FALSE diff --git a/specs/tla/session-ownership/SessionOwnership_GC_prefix_boots2.cfg b/specs/tla/session-ownership/SessionOwnership_GC_prefix_boots2.cfg new file mode 100644 index 00000000..f4940745 --- /dev/null +++ b/specs/tla/session-ownership/SessionOwnership_GC_prefix_boots2.cfg @@ -0,0 +1,35 @@ +SPECIFICATION Spec + +CONSTANTS + Conns = {c1, c2, c3} + Subs = {s1, s2} + None = none + MaxT = 3 + E = 1 + MaxBoots = 2 + MECH = "LOCK" + DROP = {} + GC = "PREFIX" + MaxPend = 8 + +SYMMETRY Sym + +INVARIANTS + TypeOK + InvOwnershipAgree + InvNoLeakedConnected + InvLiveOwnerSessionKept + InvFailedHandshakeHarmless + InvSessionPresent + InvNoResurrection + InvAckedSubsDurable + InvAbortJustified + InvCleanStart + InvRouterMirrors + InvQuiescentConsistent + InvExpiryExact + InvProgress + InvAckedStateSurvivesCrash + InvPendNotFull + +CHECK_DEADLOCK FALSE diff --git a/specs/tla/session-ownership/SessionOwnership_GC_prefix_live.cfg b/specs/tla/session-ownership/SessionOwnership_GC_prefix_live.cfg new file mode 100644 index 00000000..4add5329 --- /dev/null +++ b/specs/tla/session-ownership/SessionOwnership_GC_prefix_live.cfg @@ -0,0 +1,18 @@ +SPECIFICATION Spec + +CONSTANTS + Conns = {c1, c2, c3} + Subs = {s1} + None = none + MaxT = 2 + E = 1 + MaxBoots = 1 + MECH = "LOCK" + DROP = {} + GC = "PREFIX" + MaxPend = 8 + +PROPERTIES + EventuallyClean + +CHECK_DEADLOCK FALSE diff --git a/specs/tla/session-ownership/SessionOwnership_GC_whole.cfg b/specs/tla/session-ownership/SessionOwnership_GC_whole.cfg new file mode 100644 index 00000000..9ab05e3d --- /dev/null +++ b/specs/tla/session-ownership/SessionOwnership_GC_whole.cfg @@ -0,0 +1,35 @@ +SPECIFICATION Spec + +CONSTANTS + Conns = {c1, c2, c3} + Subs = {s1, s2} + None = none + MaxT = 3 + E = 1 + MaxBoots = 1 + MECH = "LOCK" + DROP = {} + GC = "WHOLE" + MaxPend = 8 + +SYMMETRY Sym + +INVARIANTS + TypeOK + InvOwnershipAgree + InvNoLeakedConnected + InvLiveOwnerSessionKept + InvFailedHandshakeHarmless + InvSessionPresent + InvNoResurrection + InvAckedSubsDurable + InvAbortJustified + InvCleanStart + InvRouterMirrors + InvQuiescentConsistent + InvExpiryExact + InvProgress + InvAckedStateSurvivesCrash + InvPendNotFull + +CHECK_DEADLOCK FALSE diff --git a/specs/tla/session-ownership/SessionOwnership_NEG_nosweepfair.cfg b/specs/tla/session-ownership/SessionOwnership_NEG_nosweepfair.cfg new file mode 100644 index 00000000..4090997f --- /dev/null +++ b/specs/tla/session-ownership/SessionOwnership_NEG_nosweepfair.cfg @@ -0,0 +1,18 @@ +SPECIFICATION SpecNoSweepFair + +CONSTANTS + Conns = {c1, c2, c3} + Subs = {s1} + None = none + MaxT = 2 + E = 1 + MaxBoots = 1 + MECH = "LOCK" + DROP = {} + GC = "OFF" + MaxPend = 1 + +PROPERTIES + EventuallyClean + +CHECK_DEADLOCK FALSE diff --git a/specs/tla/session-ownership/SessionOwnership_NEG_resumeafterrestart.cfg b/specs/tla/session-ownership/SessionOwnership_NEG_resumeafterrestart.cfg new file mode 100644 index 00000000..ef72bfab --- /dev/null +++ b/specs/tla/session-ownership/SessionOwnership_NEG_resumeafterrestart.cfg @@ -0,0 +1,20 @@ +SPECIFICATION Spec + +CONSTANTS + Conns = {c1, c2, c3} + Subs = {s1} + None = none + MaxT = 2 + E = 1 + MaxBoots = 1 + MECH = "LOCK" + DROP = {} + GC = "OFF" + MaxPend = 1 + +SYMMETRY Sym + +INVARIANTS + NegNoResumeAfterRestart + +CHECK_DEADLOCK FALSE diff --git a/specs/tla/session-ownership/SessionOwnership_NEG_resumingtakeover.cfg b/specs/tla/session-ownership/SessionOwnership_NEG_resumingtakeover.cfg new file mode 100644 index 00000000..06104b72 --- /dev/null +++ b/specs/tla/session-ownership/SessionOwnership_NEG_resumingtakeover.cfg @@ -0,0 +1,20 @@ +SPECIFICATION Spec + +CONSTANTS + Conns = {c1, c2, c3} + Subs = {s1} + None = none + MaxT = 2 + E = 1 + MaxBoots = 1 + MECH = "LOCK" + DROP = {} + GC = "OFF" + MaxPend = 1 + +SYMMETRY Sym + +INVARIANTS + NegNoResumingTakeover + +CHECK_DEADLOCK FALSE diff --git a/specs/tla/session-ownership/SessionOwnership_NEG_timeneverends.cfg b/specs/tla/session-ownership/SessionOwnership_NEG_timeneverends.cfg new file mode 100644 index 00000000..7399ce63 --- /dev/null +++ b/specs/tla/session-ownership/SessionOwnership_NEG_timeneverends.cfg @@ -0,0 +1,18 @@ +SPECIFICATION Spec + +CONSTANTS + Conns = {c1, c2, c3} + Subs = {s1} + None = none + MaxT = 2 + E = 1 + MaxBoots = 1 + MECH = "LOCK" + DROP = {} + GC = "OFF" + MaxPend = 1 + +PROPERTIES + NegTimeNeverEnds + +CHECK_DEADLOCK FALSE diff --git a/specs/tla/session-ownership/SessionOwnership_boots2.cfg b/specs/tla/session-ownership/SessionOwnership_boots2.cfg new file mode 100644 index 00000000..8ffb92ca --- /dev/null +++ b/specs/tla/session-ownership/SessionOwnership_boots2.cfg @@ -0,0 +1,33 @@ +SPECIFICATION Spec + +CONSTANTS + Conns = {c1, c2, c3} + Subs = {s1, s2} + None = none + MaxT = 3 + E = 1 + MaxBoots = 2 + MECH = "LOCK" + DROP = {} + GC = "OFF" + MaxPend = 1 + +SYMMETRY Sym + +INVARIANTS + TypeOK + InvOwnershipAgree + InvNoLeakedConnected + InvLiveOwnerSessionKept + InvFailedHandshakeHarmless + InvSessionPresent + InvNoResurrection + InvAckedSubsDurable + InvAbortJustified + InvCleanStart + InvRouterMirrors + InvQuiescentConsistent + InvExpiryExact + InvProgress + +CHECK_DEADLOCK FALSE diff --git a/specs/tla/session-ownership/SessionOwnership_live.cfg b/specs/tla/session-ownership/SessionOwnership_live.cfg new file mode 100644 index 00000000..8224baca --- /dev/null +++ b/specs/tla/session-ownership/SessionOwnership_live.cfg @@ -0,0 +1,18 @@ +SPECIFICATION Spec + +CONSTANTS + Conns = {c1, c2, c3} + Subs = {s1} + None = none + MaxT = 2 + E = 1 + MaxBoots = 1 + MECH = "LOCK" + DROP = {} + GC = "OFF" + MaxPend = 1 + +PROPERTIES + EventuallyClean + +CHECK_DEADLOCK FALSE