diff --git a/README.md b/README.md index 6aa8ed6..4a0b973 100644 --- a/README.md +++ b/README.md @@ -185,7 +185,7 @@ sponsored by PrintNode.* | [Recipes](docs/recipes.md) | print from n8n, Zapier, Make, curl in one HTTP node | | [E-commerce](docs/ecommerce.md) | WooCommerce plugin, Shopify webhook, `POST /orders` | | [Billing](docs/billing.md) | plans, checkout links, the signed provider webhook, org deletion | -| [Operator API](docs/operator.md) | for a hosted deployment: a service token, the audit log, reads and writes (quota, lock, create, delete, reset mail, key revoke, time-limited support access the customer can see) a separate console uses | +| [Operator API](docs/operator.md) | for a hosted deployment: a service token, the audit log, reads and writes (quota, lock, create, delete, reset mail, key revoke, time-limited support access the customer can see, live settings) a separate console uses | | [Migration](docs/migration.md) | upgrading from v2 (Python) to v3 (Rust): server, agents, changed defaults, rollback | | [Roadmap](docs/roadmap.md) | what shipped and what is next | diff --git a/crates/server/src/lib.rs b/crates/server/src/lib.rs index b6346d7..cb9f8f1 100644 --- a/crates/server/src/lib.rs +++ b/crates/server/src/lib.rs @@ -3,7 +3,7 @@ //! //! Module map (each mirrors the Python module it replaced, see git history before v3.0.0): //! - pure logic: `dispatch`, `auth`, `billing`, `cloudprnt`, `label`, `orders`, `packing_slip`, -//! `printnode`, `proxy` (client address), `mail` (env -> SMTP plan), `pyfmt` (Python-compatible value formatting) +//! `printnode`, `proxy` (client address), `mail` (env -> SMTP plan), `settings` (env parsers, live values), `pyfmt` (Python-compatible value formatting) //! - IO: `store` (SQLite), `server` (routing, sync), `http` (hyper glue, long-poll), `web` //! (embedded dashboard bundle) @@ -20,6 +20,7 @@ pub mod printnode; pub mod proxy; pub mod pyfmt; pub mod server; +pub mod settings; pub mod store; pub mod web; diff --git a/crates/server/src/main.rs b/crates/server/src/main.rs index 5fe487a..5e19252 100644 --- a/crates/server/src/main.rs +++ b/crates/server/src/main.rs @@ -15,7 +15,11 @@ //! accepted from), LOG_REQUESTS, SMTP_*. use printpapi_server::dispatch::{Destinations, HttpFetcher, OUTBOUND_IN_FLIGHT}; use printpapi_server::mail::{self, SmtpMailer}; -use printpapi_server::server::{AppState, Config, Terms}; +use printpapi_server::server::{AppState, Config}; +use printpapi_server::settings::{ + EnvFlags, Live, parse_fetch_cap, parse_quota, parse_retention, parse_terms, py_strip, + signup_env_locks, +}; use printpapi_server::store::Db; use printpapi_server::{billing, http}; use std::sync::Arc; @@ -25,99 +29,6 @@ fn env(key: &str) -> Option { std::env::var(key).ok() } -/// Python `str.strip()`: Unicode whitespace plus U+001C..U+001F. -fn py_strip(s: &str) -> &str { - s.trim_matches(|c: char| c.is_whitespace() || ('\u{1c}'..='\u{1f}').contains(&c)) -} - -/// Empty: no quota. Otherwise a non-negative integer (0 is a real quota: nothing may be printed). -fn parse_quota(raw: &str) -> Result, String> { - let t = py_strip(raw); - if t.is_empty() { - return Ok(None); - } - match t.parse::() { - Ok(n) if n >= 0 => Ok(Some(n)), - _ => Err(format!("must be a non-negative integer, not {t:?}")), - } -} - -/// Empty: unset (the default follows the signup mode). Otherwise a non-negative integer, 0 = no cap. -fn parse_fetch_cap(raw: &str) -> Result, String> { - let t = py_strip(raw); - if t.is_empty() { - return Ok(None); - } - t.parse::() - .map(Some) - .map_err(|_| format!("must be a non-negative integer, not {t:?}")) -} - -/// Empty or 0: keep for ever. Otherwise a positive number of days. -fn parse_retention(raw: &str) -> Result, String> { - let t = py_strip(raw); - if t.is_empty() { - return Ok(None); - } - match t.parse::() { - Ok(0) => Ok(None), - Ok(n) if n > 0 => Ok(Some(n)), - _ => Err(format!("must be a positive integer, not {t:?}")), - } -} - -/// The one link rule (`web/lib/terms.ts` `isHttpUrl` is its twin): `http://` or `https://`, any -/// case, then a non-empty host. Printable ASCII only (0x21..=0x7E): Rust's whitespace and JS's -/// `\s` disagree on U+0085 and U+FEFF, so neither is asked. -fn is_terms_url(u: &str) -> bool { - if !u.bytes().all(|b| (0x21..=0x7E).contains(&b)) { - return false; - } - let lower = u.to_ascii_lowercase(); - let Some(rest) = lower - .strip_prefix("https://") - .or_else(|| lower.strip_prefix("http://")) - else { - return false; - }; - !rest.split(['/', '?', '#']).next().unwrap_or("").is_empty() -} - -/// The terms a signup must accept. A version needs a terms URL. Every link that is set must be -/// an absolute `http(s)` URL with a host, version or not. -fn parse_terms( - version: &str, - terms_url: &str, - dpa_url: &str, - privacy_url: &str, -) -> Result, String> { - let version = py_strip(version); - let url = |name: &str, raw: &str| -> Result, String> { - let t = py_strip(raw); - if t.is_empty() { - Ok(None) - } else if is_terms_url(t) { - Ok(Some(t.to_string())) - } else { - Err(format!("{name} must be an http(s) URL, not {t:?}")) - } - }; - let terms_url = url("PRINTAPI_TERMS_URL", terms_url)?; - let dpa_url = url("PRINTAPI_DPA_URL", dpa_url)?; - let privacy_url = url("PRINTAPI_PRIVACY_URL", privacy_url)?; - if version.is_empty() { - return Ok(None); // no consent required: the links are only checked, not used - } - let terms_url = - terms_url.ok_or("PRINTAPI_TERMS_URL is required with PRINTAPI_TERMS_VERSION")?; - Ok(Some(Terms { - version: version.to_string(), - terms_url, - dpa_url, - privacy_url, - })) -} - /// The operator token and the peers it is accepted from. Unset or blank: the operator API is off. /// The token is at least 32 characters, never starts with `sess_` (a session's prefix, which the /// credential lookup treats specially) and is not the root token. The peer list is required: a @@ -151,22 +62,26 @@ fn parse_operator( Ok(Some((token.to_string(), from))) } -/// One stderr line per hosted-beta safeguard that is off while strangers can sign up. -fn startup_warnings(cfg: &Config) -> Vec<&'static str> { +/// One stderr line per hosted-beta safeguard that is off while strangers can sign up, judged on +/// the effective values (an override from the database counts). +fn startup_warnings(live: &Live) -> Vec<&'static str> { let mut out = Vec::new(); - if cfg.signup_open && cfg.signup_quota.is_none() { + if !live.signup.value { + return out; + } + if live.signup_quota.value.is_none() { out.push( - "warning: PRINTAPI_SIGNUP=open without PRINTAPI_SIGNUP_QUOTA: new orgs can print without limit", + "warning: signup is open (PRINTAPI_SIGNUP or the operator console) without PRINTAPI_SIGNUP_QUOTA: new orgs can print without limit", ); } - if cfg.signup_open && cfg.job_retention_days.is_none() { + if live.job_retention_days.value.is_none() { out.push( - "warning: PRINTAPI_SIGNUP=open without PRINTAPI_JOB_RETENTION_DAYS: job payloads are kept for ever", + "warning: signup is open (PRINTAPI_SIGNUP or the operator console) without PRINTAPI_JOB_RETENTION_DAYS: job payloads are kept for ever", ); } - if cfg.signup_open && cfg.terms.is_none() { + if live.terms.value.is_none() { out.push( - "warning: PRINTAPI_SIGNUP=open without PRINTAPI_TERMS_VERSION: signups do not accept any terms", + "warning: signup is open (PRINTAPI_SIGNUP or the operator console) without PRINTAPI_TERMS_VERSION: signups do not accept any terms", ); } out @@ -258,6 +173,16 @@ fn config_from(env: &dyn Fn(&str) -> Option) -> Result<(Config, String, &env("PRINTAPI_DPA_URL").unwrap_or_default(), &env("PRINTAPI_PRIVACY_URL").unwrap_or_default(), )?; + // Which of the live settings' variables are set at all: it decides a value's source and + // whether `PRINTAPI_SIGNUP` locks signup shut (any value but `open`). + let set = |k: &str| env(k).is_some_and(|v| !py_strip(&v).is_empty()); + cfg.env_flags = EnvFlags { + signup_set: env("PRINTAPI_SIGNUP").is_some(), + signup_explicit_closed: env("PRINTAPI_SIGNUP").is_some_and(|v| signup_env_locks(&v)), + signup_quota_set: set("PRINTAPI_SIGNUP_QUOTA"), + job_retention_days_set: set("PRINTAPI_JOB_RETENTION_DAYS"), + terms_set: set("PRINTAPI_TERMS_VERSION"), + }; cfg.open_enrollment = env("PRINTAPI_OPEN_ENROLLMENT").is_some_and(|s| py_strip(&s) == "1"); Ok((cfg, db_path, port)) } @@ -288,9 +213,6 @@ fn run() -> Result<(), String> { let (cfg, db_path, port) = config()?; let destinations = cfg.destinations; let hosted_locks_enforced = cfg.operator_token.is_some(); - for line in startup_warnings(&cfg) { - eprintln!("{line}"); - } if cfg.operator_token.is_some() { eprintln!("operator API on (hosted mode: private URLs denied unless allowed explicitly)"); } @@ -312,6 +234,9 @@ fn run() -> Result<(), String> { destinations, )), )); + for line in startup_warnings(&state.live()) { + eprintln!("{line}"); + } http::start_background(state.clone()); // The heavy lifting (SQLite, scrypt, outbound HTTP) runs on the blocking pool, so a few // workers are plenty and idle ones would only cost RSS. @@ -518,3 +443,124 @@ mod operator_tests { assert_eq!(c.fetch_cap(), 9); } } + +#[cfg(test)] +mod live_tests { + use super::{config_from, startup_warnings}; + use printpapi_server::server::Config; + use printpapi_server::settings::{Source, Sourced}; + + fn cfg(extra: &[(&str, &str)]) -> Config { + let mut vars: Vec<(String, String)> = vec![("PRINTAPI_TOKEN".into(), "root".into())]; + vars.extend(extra.iter().map(|(k, v)| (k.to_string(), v.to_string()))); + config_from(&|k| vars.iter().rfind(|(n, _)| n == k).map(|(_, v)| v.clone())) + .map(|(c, _, _)| c) + .unwrap() + } + + #[test] + fn nothing_set_means_no_flag_is_set() { + let f = cfg(&[]).env_flags; + assert!( + !f.signup_set + && !f.signup_explicit_closed + && !f.signup_quota_set + && !f.job_retention_days_set + && !f.terms_set, + "{f:?}" + ); + // blank is unset, as the parsers read it + let f = cfg(&[ + ("PRINTAPI_SIGNUP_QUOTA", " "), + ("PRINTAPI_JOB_RETENTION_DAYS", ""), + ]) + .env_flags; + assert!(!f.signup_quota_set && !f.job_retention_days_set, "{f:?}"); + } + + #[test] + fn the_flags_follow_the_variables() { + let f = cfg(&[("PRINTAPI_SIGNUP", "closed")]).env_flags; + assert!(f.signup_set && f.signup_explicit_closed); + let f = cfg(&[("PRINTAPI_SIGNUP", "open")]).env_flags; + assert!(f.signup_set && !f.signup_explicit_closed); + let f = cfg(&[ + ("PRINTAPI_SIGNUP_QUOTA", "5"), + ("PRINTAPI_JOB_RETENTION_DAYS", "0"), + ("PRINTAPI_TERMS_VERSION", "v1"), + ("PRINTAPI_TERMS_URL", "https://x.test/t"), + ]) + .env_flags; + assert!(f.signup_quota_set && f.job_retention_days_set && f.terms_set); + } + + #[test] + fn any_signup_value_but_open_locks_signup_shut() { + for v in [ + "closed", + "CLOSED", + "Closed", + "off", + "0", + "closd", + "\"closed\"", + "no", + ] { + let c = cfg(&[("PRINTAPI_SIGNUP", v)]); + assert!( + c.env_flags.signup_set && c.env_flags.signup_explicit_closed, + "{v:?}" + ); + assert!(!c.signup_open, "{v:?} never opens signup"); + } + // padding does not matter, blank is unset, `open` opens and never locks + let f = cfg(&[("PRINTAPI_SIGNUP", " closed ")]).env_flags; + assert!(f.signup_explicit_closed); + for v in ["", " "] { + let f = cfg(&[("PRINTAPI_SIGNUP", v)]).env_flags; + assert!(!f.signup_explicit_closed, "{v:?}"); + } + let c = cfg(&[("PRINTAPI_SIGNUP", "open")]); + assert!(c.signup_open && !c.env_flags.signup_explicit_closed); + } + + #[test] + fn a_terms_version_with_a_control_character_stops_the_server() { + let vars = [ + ("PRINTAPI_TERMS_VERSION", "v1\u{1b}[2J"), + ("PRINTAPI_TERMS_URL", "https://x.test/t"), + ]; + let mut all: Vec<(String, String)> = vec![("PRINTAPI_TOKEN".into(), "root".into())]; + all.extend(vars.iter().map(|(k, v)| (k.to_string(), v.to_string()))); + let e = config_from(&|k| all.iter().rfind(|(n, _)| n == k).map(|(_, v)| v.clone())) + .unwrap_err(); + assert!(e.contains("PRINTAPI_TERMS_VERSION"), "{e:?}"); + assert!(!e.contains('\u{1b}'), "{e:?}"); + } + + #[test] + fn the_open_signup_warnings_read_the_effective_values() { + let c = cfg(&[]); + let mut live = printpapi_server::settings::build_live(&c.env_start(), false, &[]).0; + assert!(startup_warnings(&live).is_empty()); + // the database opened signup: all three safeguards are missing + live.signup = Sourced { + value: true, + source: Source::Db, + }; + assert_eq!(startup_warnings(&live).len(), 3); + // the cause may be the console, so no warning blames the variable alone + for w in startup_warnings(&live) { + assert!(w.starts_with("warning: signup is open"), "{w}"); + assert!(!w.contains("PRINTAPI_SIGNUP=open"), "{w}"); + assert!(w.contains("operator console"), "{w}"); + } + // an override that supplies them silences the warning + live.signup_quota.value = Some(5); + live.job_retention_days.value = Some(30); + assert_eq!(startup_warnings(&live).len(), 1); + // a database close beats an env open: nothing to warn about + live.signup.value = false; + assert!(startup_warnings(&live).is_empty()); + } +} diff --git a/crates/server/src/proxy.rs b/crates/server/src/proxy.rs index 5e268ca..75994f2 100644 --- a/crates/server/src/proxy.rs +++ b/crates/server/src/proxy.rs @@ -109,6 +109,21 @@ impl TrustedProxies { pub fn contains(&self, ip: IpAddr) -> bool { self.0.iter().any(|n| n.contains(ip)) } + + /// The entries as text, for display: an address, with `/prefix` unless it names one host. + pub fn entries(&self) -> Vec { + self.0 + .iter() + .map(|n| { + let max = if n.addr.is_ipv4() { 32 } else { 128 }; + if n.prefix == max { + n.addr.to_string() + } else { + format!("{}/{}", n.addr, n.prefix) + } + }) + .collect() + } } /// One `X-Forwarded-For` entry: a plain IPv4/IPv6 address or `[IPv6]`. diff --git a/crates/server/src/server.rs b/crates/server/src/server.rs index f4fa6ef..5815e24 100644 --- a/crates/server/src/server.rs +++ b/crates/server/src/server.rs @@ -62,6 +62,8 @@ pub struct Config { pub operator_token: Option, /// PRINTAPI_OPERATOR_FROM: the socket peers the operator token is accepted from. pub operator_from: crate::proxy::TrustedProxies, + /// Which of the live settings' variables were set (`main` fills it): the source of a value. + pub env_flags: crate::settings::EnvFlags, } /// The terms a signup must accept (`PRINTAPI_TERMS_*`). The URLs are `http(s)` (checked at startup). @@ -100,6 +102,23 @@ impl Config { fetches_per_org: None, operator_token: None, operator_from: crate::proxy::TrustedProxies::default(), + env_flags: crate::settings::EnvFlags::default(), + } + } + + /// The startup values of the live settings and whether the environment set them. + pub fn env_start(&self) -> crate::settings::EnvStart { + let f = &self.env_flags; + crate::settings::EnvStart { + signup_open: self.signup_open, + signup_set: f.signup_set, + signup_explicit_closed: f.signup_explicit_closed, + signup_quota: self.signup_quota, + signup_quota_set: f.signup_quota_set, + job_retention_days: self.job_retention_days, + job_retention_days_set: f.job_retention_days_set, + terms: self.terms.clone(), + terms_set: f.terms_set, } } } @@ -206,6 +225,9 @@ pub struct AppState { last_purge: Mutex>, /// The operator overview, kept for `operator::OVERVIEW_TTL`. overview_cache: Mutex>, + /// The effective live settings. The lock is held only to clone or replace the `Arc`; each + /// request clones it once, at its start (`AppState::live`). + live: Mutex>, } /// Concurrent image renders (see `AppState::render_slots`). ponytail: one global cap, not per @@ -362,6 +384,7 @@ impl AppState { fetcher: Arc, ) -> AppState { let limiter = LoginLimiter::new(cfg.max_login_fails, cfg.login_window_s); + let live = initial_live(&cfg, &db); let dummy_hash = crate::auth::hash_password(Some("no such user, no such password")) .expect("the dummy password is long enough"); AppState { @@ -381,8 +404,78 @@ impl AppState { started: std::time::Instant::now(), last_purge: Mutex::new(None), overview_cache: Mutex::new(None), + live: Mutex::new(Arc::new(live)), } } + + /// The live settings as they are now. A request calls this once and keeps the snapshot. + pub fn live(&self) -> Arc { + self.live.lock().unwrap_or_else(|e| e.into_inner()).clone() + } + + /// Write a validated change (and its audit row), re-read the rows, rebuild the live values + /// from them and swap them in, all inside one `db.with`: two writers cannot commit in one + /// order and swap in the other. The live values come from the rows, never from the request. + /// Only the operator API calls this (it is a 404 without the token). + pub fn apply_settings( + &self, + change: &crate::settings::Change, + reason: Option<&str>, + ) -> store::Result> { + let out = self.try_apply_settings( + |_| Ok::<_, std::convert::Infallible>(change.clone()), + reason, + )?; + Ok(out.unwrap_or_else(|never| match never {})) + } + + /// `apply_settings`, with the change decided inside the critical section: `decide` sees the + /// live values as they are at that moment, so a second writer cannot slip in between the check + /// and the write. `Err` from `decide` writes nothing. + pub fn try_apply_settings( + &self, + decide: impl FnOnce(&crate::settings::Live) -> Result, + reason: Option<&str>, + ) -> store::Result, E>> { + self.db.with(|c| { + let change = match decide(&self.live()) { + Ok(change) => change, + Err(e) => return Ok(Err(e)), + }; + store::operator_update_settings(c, &change, reason)?; + let rows = store::settings_rows(c)?; + let (live, _) = crate::settings::build_live( + &self.cfg.env_start(), + self.cfg.destinations == crate::dispatch::Destinations::Any, + &rows, + ); + let live = Arc::new(live); + *self.live.lock().unwrap_or_else(|e| e.into_inner()) = live.clone(); + Ok(Ok(live)) + }) + } +} + +/// The live values at startup: the env ones, and with the operator token set the override rows +/// on top (one stderr line per applied or ignored row). Without the token the table is never read. +fn initial_live(cfg: &Config, db: &Db) -> crate::settings::Live { + let rows = if cfg.operator_token.is_some() { + db.with(store::settings_rows).unwrap_or_else(|e| { + eprintln!("settings: cannot read the overrides ({e}); the environment values apply"); + Vec::new() + }) + } else { + Vec::new() + }; + let (live, notes) = crate::settings::build_live( + &cfg.env_start(), + cfg.destinations == crate::dispatch::Destinations::Any, + &rows, + ); + for n in notes { + eprintln!("{n}"); + } + live } /// The signup throttle key of a client address: IPv6 by /64 (`proxy::throttle_key`). Anything that @@ -810,6 +903,8 @@ struct Ctx<'a> { /// credential; only `enrollment_org` still needs to know, because open enrollment would /// otherwise take a request without a key. operator_cred: bool, + /// The live settings as they were when the request began: one snapshot for all of it. + live: Arc, } /// The org `key` may enroll an agent into, or why it may not (see `Ctx::agent_register`): the @@ -855,6 +950,7 @@ pub fn handle(state: &AppState, req: &Req) -> Resp { s: state, req, operator_cred: false, + live: state.live(), } .operator(); } @@ -873,6 +969,7 @@ fn route(state: &AppState, req: &Req, operator_cred: bool) -> Resp { s: state, req, operator_cred, + live: state.live(), }; let r = match req.method.as_str() { "GET" => cx.get(), @@ -1097,7 +1194,7 @@ impl Ctx<'_> { /// Self-serve: a new org, its first user, and a session in one call — the hosted deployment's /// front door. Off by default (`signup_open`). fn signup(&self) -> R { - if !self.cfg().signup_open { + if !self.live.signup.value { return Ok(jerr(403, "signup is disabled on this server")); } let body = gate!(self.read_obj()); @@ -1114,7 +1211,7 @@ impl Ctx<'_> { if !email.contains('@') { return Ok(jerr(400, "email required")); } - let terms_version = self.cfg().terms.as_ref().map(|t| t.version.as_str()); + let terms_version = self.live.terms.value.as_ref().map(|t| t.version.as_str()); if let Some(current) = terms_version { match body.get("accept_terms") { Some(Value::String(v)) if v == current => {} @@ -1140,7 +1237,7 @@ impl Ctx<'_> { .map(|n| printnode::py_trim(n).to_string()) .filter(|n| !n.is_empty()) .unwrap_or_else(|| email.split_once('@').map_or("", |(_, d)| d).to_string()); - let quota = self.cfg().signup_quota; + let quota = self.live.signup_quota.value; let new = match self.db(|c| store::signup(c, &name, &email, &pw_hash, quota, terms_version)) { Ok(n) => n, @@ -2086,10 +2183,10 @@ impl Ctx<'_> { // "create an account" and "forgot password" at all. Both are booleans about the // server's configuration, not about any account. "/health" => { - let mut h = json!({ "ok": true, "signup": if self.cfg().signup_open { "open" } else { "closed" }, + let mut h = json!({ "ok": true, "signup": if self.live.signup.value { "open" } else { "closed" }, "password_reset": self.cfg().reset_enabled }); // Only when consent is required, so a server without it answers as before. - if let Some(t) = &self.cfg().terms { + if let Some(t) = &self.live.terms.value { h["terms"] = json!({ "version": t.version, "terms_url": t.terms_url, "dpa_url": t.dpa_url, "privacy_url": t.privacy_url }); } @@ -2752,7 +2849,7 @@ const RETENTION_EVERY_S: f64 = 3600.0; /// PRINTAPI_JOB_RETENTION_DAYS is set. `None`: not due, not configured, or the purge failed /// (logged; the next tick after the hour tries again). pub fn retention_pass(state: &AppState, now: f64) -> Option { - let days = state.cfg.job_retention_days?; + let days = state.live().job_retention_days.value?; { let mut last = state .last_retention diff --git a/crates/server/src/server/operator.rs b/crates/server/src/server/operator.rs index 495b701..35b2a1e 100644 --- a/crates/server/src/server/operator.rs +++ b/crates/server/src/server/operator.rs @@ -4,6 +4,7 @@ //! 404. The token is never a `Principal`: only `operator_gate` reads it, and on every other route //! `without_operator_credential` takes it out of the request before any lookup runs. use super::*; +use crate::settings; use std::net::IpAddr; use std::time::Duration; @@ -206,6 +207,8 @@ impl Ctx<'_> { let r = match (self.req.method.as_str(), path) { ("GET", "/operator/overview") => self.operator_overview(), ("GET", "/operator/audit") => self.operator_audit(), + ("GET", "/operator/settings") => self.operator_settings(), + ("PUT", "/operator/settings") => self.operator_settings_put(), ("GET", "/operator/orgs") => self.operator_orgs(), ("POST", "/operator/orgs") => self.operator_org_create(), (m, _) => match (m, org_route(path)) { @@ -473,7 +476,7 @@ impl Ctx<'_> { }; let tok = auth::new_session_token(); let ttl = self.cfg().reset_ttl_s; - let quota = self.cfg().signup_quota; + let quota = self.live.signup_quota.value; let new = match self.db(|c| { store::operator_create_org( c, @@ -805,6 +808,96 @@ impl Ctx<'_> { )) } + /// The live values with their source, and the startup-only values beside them (never a + /// secret: no token, no password, no key). + fn settings_json(&self, live: &settings::Live) -> Value { + let cfg = self.cfg(); + let terms = live.terms.value.as_ref().map(settings::terms_json); + // `ignored` (only when a row exists but is not applied) is the stored value, so a stale + // row can be seen and cleared. + let entry = |key: &str, value: Value, source: settings::Source| { + let mut m = serde_json::Map::new(); + m.insert("value".into(), value); + m.insert("source".into(), json!(source.as_str())); + if let Some((_, stored)) = live.ignored.iter().find(|(k, _)| *k == key) { + m.insert("ignored".into(), stored.clone()); + } + Value::Object(m) + }; + json!({ + "settings": { + "signup": entry( + "signup", + json!(if live.signup.value { "open" } else { "closed" }), + live.signup.source, + ), + "signup_quota": entry( + "signup_quota", json!(live.signup_quota.value), live.signup_quota.source, + ), + "job_retention_days": entry( + "job_retention_days", + json!(live.job_retention_days.value), + live.job_retention_days.source, + ), + "terms": entry("terms", json!(terms), live.terms.source), + }, + "env_only": { + "fetches_per_org": cfg.fetch_cap(), + "private_urls": if cfg.destinations == dispatch::Destinations::Any { + "allow" + } else { + "deny" + }, + "trusted_proxies": cfg.trusted_proxies.entries(), + "operator_from": cfg.operator_from.entries(), + "public_url": cfg.public_url, + "mail": cfg.reset_enabled, + }, + }) + } + + fn operator_settings(&self) -> R { + Ok(Resp::json(200, &self.settings_json(&self.live))) + } + + fn operator_settings_put(&self) -> R { + use settings::{Conflict, PutError}; + let body = gate!(self.read_obj()); + let reason = gate!(reason_of(&body, 0)); + let env = self.cfg().env_start(); + let any = self.cfg().destinations == dispatch::Destinations::Any; + let decide = |live: &settings::Live| { + let change = settings::validate_put(&body, reason.as_deref(), live, &env, any)?; + if settings::unset_needs_reason(&change, live, &env) + && !settings::reason_is_enough(reason.as_deref()) + { + return Err(PutError::ReasonNeeded); + } + Ok(change) + }; + Ok( + match self.s.try_apply_settings(decide, reason.as_deref())? { + Ok(live) => Resp::json(200, &self.settings_json(&live)), + Err(PutError::Invalid(m)) => jerr(400, m), + Err(PutError::ReasonNeeded) => jerr( + 400, + format!( + "reason of at least {} characters required: this scrubs payloads and titles for good", + settings::REASON_MIN + ), + ), + Err(PutError::Conflict(Conflict::SignupLocked)) => jerr( + 409, + "signup is locked closed by PRINTAPI_SIGNUP in the environment: only closing it is accepted", + ), + Err(PutError::Conflict(Conflict::OpenWithPrivateUrls)) => jerr( + 409, + "signup cannot be opened while private URLs are allowed (PRINTAPI_PRIVATE_URLS=allow)", + ), + }, + ) + } + fn operator_overview(&self) -> R { let mut cache = self .s @@ -849,7 +942,7 @@ impl Ctx<'_> { "deleted": purge.as_ref().map(|p| p.1.deleted), }, // the console shows a section only when the server reports its feature - "features": ["support"], + "features": ["support", "settings"], }); *cache = Some((std::time::Instant::now(), v.clone())); Ok(Resp::json(200, &v)) diff --git a/crates/server/src/settings.rs b/crates/server/src/settings.rs new file mode 100644 index 0000000..172c67a --- /dev/null +++ b/crates/server/src/settings.rs @@ -0,0 +1,1110 @@ +// printpapi — self-hosted PrintNode alternative. Elastic License 2.0 (see LICENSE). +//! Live settings, the pure side (spec A5): the env parsers `main` and the operator API share, the +//! effective values with their source, and the validation of a `PUT /operator/settings` body. No +//! IO: the rows come from `store`, the stderr notes are returned for `main` to print. +use crate::server::Terms; +use serde_json::{Map, Value as Json, json}; + +/// Python `str.strip()`: Unicode whitespace plus U+001C..U+001F. +pub fn py_strip(s: &str) -> &str { + s.trim_matches(|c: char| c.is_whitespace() || ('\u{1c}'..='\u{1f}').contains(&c)) +} + +/// Empty: no quota. Otherwise a non-negative integer (0 is a real quota: nothing may be printed). +pub fn parse_quota(raw: &str) -> Result, String> { + let t = py_strip(raw); + if t.is_empty() { + return Ok(None); + } + match t.parse::() { + Ok(n) if n >= 0 => Ok(Some(n)), + _ => Err(format!("must be a non-negative integer, not {t:?}")), + } +} + +/// Empty: unset (the default follows the signup mode). Otherwise a non-negative integer, 0 = no cap. +pub fn parse_fetch_cap(raw: &str) -> Result, String> { + let t = py_strip(raw); + if t.is_empty() { + return Ok(None); + } + t.parse::() + .map(Some) + .map_err(|_| format!("must be a non-negative integer, not {t:?}")) +} + +/// Empty or 0: keep for ever. Otherwise a positive number of days. +pub fn parse_retention(raw: &str) -> Result, String> { + let t = py_strip(raw); + if t.is_empty() { + return Ok(None); + } + match t.parse::() { + Ok(0) => Ok(None), + Ok(n) if n > 0 => Ok(Some(n)), + _ => Err(format!("must be a positive integer, not {t:?}")), + } +} + +/// The one link rule (`web/lib/terms.ts` `isHttpUrl` is its twin): `http://` or `https://`, any +/// case, then a non-empty host. Printable ASCII only (0x21..=0x7E): Rust's whitespace and JS's +/// `\s` disagree on U+0085 and U+FEFF, so neither is asked. +pub fn is_terms_url(u: &str) -> bool { + if !u.bytes().all(|b| (0x21..=0x7E).contains(&b)) { + return false; + } + let lower = u.to_ascii_lowercase(); + let Some(rest) = lower + .strip_prefix("https://") + .or_else(|| lower.strip_prefix("http://")) + else { + return false; + }; + !rest.split(['/', '?', '#']).next().unwrap_or("").is_empty() +} + +/// Longest terms version, in characters. +pub const TERMS_VERSION_MAX: usize = 100; + +/// The terms a signup must accept. A version needs a terms URL and has no control character (it +/// lands in stderr and `GET /health`) and at most `TERMS_VERSION_MAX` characters. Every link that +/// is set must be an absolute `http(s)` URL with a host, version or not (`is_terms_url` already +/// refuses control characters and whitespace). +pub fn parse_terms( + version: &str, + terms_url: &str, + dpa_url: &str, + privacy_url: &str, +) -> Result, String> { + let version = py_strip(version); + let url = |name: &str, raw: &str| -> Result, String> { + let t = py_strip(raw); + if t.is_empty() { + Ok(None) + } else if is_terms_url(t) { + Ok(Some(t.to_string())) + } else { + Err(format!("{name} must be an http(s) URL, not {t:?}")) + } + }; + if version.chars().any(char::is_control) { + return Err("PRINTAPI_TERMS_VERSION must not contain control characters".into()); + } + if version.chars().count() > TERMS_VERSION_MAX { + return Err(format!( + "PRINTAPI_TERMS_VERSION must be at most {TERMS_VERSION_MAX} characters" + )); + } + let terms_url = url("PRINTAPI_TERMS_URL", terms_url)?; + let dpa_url = url("PRINTAPI_DPA_URL", dpa_url)?; + let privacy_url = url("PRINTAPI_PRIVACY_URL", privacy_url)?; + if version.is_empty() { + return Ok(None); // no consent required: the links are only checked, not used + } + let terms_url = + terms_url.ok_or("PRINTAPI_TERMS_URL is required with PRINTAPI_TERMS_VERSION")?; + Ok(Some(Terms { + version: version.to_string(), + terms_url, + dpa_url, + privacy_url, + })) +} + +/// Does this `PRINTAPI_SIGNUP` value lock signup shut (the emergency brake)? Any present, non-blank +/// value that is not exactly `open` does, so a near miss (`CLOSED`, `off`, `closd`) fails closed. +pub fn signup_env_locks(raw: &str) -> bool { + let t = py_strip(raw); + !t.is_empty() && t != "open" +} + +/// The four keys an operator may change at runtime; everything else stays env-only. +pub const LIVE_KEYS: [&str; 4] = ["signup", "signup_quota", "job_retention_days", "terms"]; + +/// Where an effective value comes from. +#[derive(Debug, Clone, Copy, PartialEq, Eq)] +pub enum Source { + /// An override row. + Db, + /// The variable is set. + Env, + /// `PRINTAPI_SIGNUP` set to anything but `open`: the emergency brake, no override applies. + EnvLocked, + /// Neither a row nor a variable. + Default, +} + +impl Source { + pub fn as_str(self) -> &'static str { + match self { + Source::Db => "db", + Source::Env => "env", + Source::EnvLocked => "env-locked", + Source::Default => "default", + } + } +} + +#[derive(Debug, Clone, PartialEq)] +pub struct Sourced { + pub value: T, + pub source: Source, +} + +/// The effective live values. One of these is a snapshot; it is replaced whole, never edited. +#[derive(Debug, Clone, PartialEq)] +pub struct Live { + pub signup: Sourced, + pub signup_quota: Sourced>, + pub job_retention_days: Sourced>, + pub terms: Sourced>, + /// Keys whose stored row is not applied (env-locked, the private-URL rule, or a value that no + /// longer validates), with the row as stored (as the API shows it, else the raw text). + pub ignored: Vec<(&'static str, Json)>, +} + +/// The values the environment gave at startup and whether each variable was set at all. +#[derive(Debug, Clone, PartialEq, Default)] +pub struct EnvStart { + pub signup_open: bool, + pub signup_set: bool, + /// `PRINTAPI_SIGNUP` is present and not `open` (`signup_env_locks`): unset means closed by + /// default, not a lock. + pub signup_explicit_closed: bool, + pub signup_quota: Option, + pub signup_quota_set: bool, + pub job_retention_days: Option, + pub job_retention_days_set: bool, + pub terms: Option, + pub terms_set: bool, +} + +/// Which startup variables were set at all (a default and an explicit value can be equal). +#[derive(Debug, Clone, PartialEq, Default)] +pub struct EnvFlags { + pub signup_set: bool, + /// `PRINTAPI_SIGNUP` is present, not blank and not exactly `open` (`signup_env_locks`). + pub signup_explicit_closed: bool, + pub signup_quota_set: bool, + pub job_retention_days_set: bool, + pub terms_set: bool, +} + +/// One validated override value. +#[derive(Debug, Clone, PartialEq)] +pub enum Parsed { + Signup(bool), + SignupQuota(i64), + JobRetentionDays(i64), + Terms(Terms), +} + +impl Parsed { + pub fn key(&self) -> &'static str { + match self { + Parsed::Signup(_) => "signup", + Parsed::SignupQuota(_) => "signup_quota", + Parsed::JobRetentionDays(_) => "job_retention_days", + Parsed::Terms(_) => "terms", + } + } + + /// The value as the API shows it (and the audit detail holds it). + pub fn to_json(&self) -> Json { + match self { + Parsed::Signup(open) => json!(if *open { "open" } else { "closed" }), + Parsed::SignupQuota(n) | Parsed::JobRetentionDays(n) => json!(n), + Parsed::Terms(t) => terms_json(t), + } + } + + /// The text of the `settings.value` column. + pub fn encode(&self) -> String { + match self { + Parsed::Signup(open) => (if *open { "open" } else { "closed" }).to_string(), + Parsed::SignupQuota(n) | Parsed::JobRetentionDays(n) => n.to_string(), + Parsed::Terms(t) => terms_json(t).to_string(), + } + } + + /// A stored value, checked by the same rules as a request. + pub fn decode(key: &str, raw: &str) -> Result { + match key { + "signup" => signup_of(raw), + "signup_quota" | "job_retention_days" => Self::from_json( + key, + &json!( + raw.trim() + .parse::() + .map_err(|_| { "must be an integer".to_string() })? + ), + ), + "terms" => Self::from_json( + key, + &serde_json::from_str::(raw).map_err(|_| "not valid JSON".to_string())?, + ), + _ => Err(format!("unknown setting {key:?}")), + } + } + + /// A request value, checked by the env parsers' rules. + pub fn from_json(key: &str, v: &Json) -> Result { + let int = || { + v.as_i64() + .map(|n| n.to_string()) + .ok_or_else(|| "must be an integer".to_string()) + }; + match key { + "signup" => signup_of(v.as_str().ok_or("must be \"open\" or \"closed\"")?), + "signup_quota" => parse_quota(&int()?)? + .map(Parsed::SignupQuota) + .ok_or_else(|| "must be a non-negative integer".into()), + "job_retention_days" => parse_retention(&int()?)? + .map(Parsed::JobRetentionDays) + .ok_or_else(|| "must be a positive integer (unset the key to go back)".into()), + "terms" => { + let o = v.as_object().ok_or("must be an object")?; + if let Some(k) = o.keys().find(|k| { + !["version", "terms_url", "dpa_url", "privacy_url"].contains(&k.as_str()) + }) { + return Err(format!("unknown field {k:?}")); + } + let text = |name: &str| match o.get(name) { + None => Ok(""), + Some(Json::String(s)) => Ok(s.as_str()), + Some(_) => Err(format!("{name} must be a string")), + }; + let version = text("version")?; + if py_strip(version).is_empty() { + return Err("version must not be empty".into()); + } + parse_terms( + version, + text("terms_url")?, + text("dpa_url")?, + text("privacy_url")?, + )? + .map(Parsed::Terms) + .ok_or_else(|| "version must not be empty".into()) + } + _ => Err(format!("unknown setting {key:?}")), + } + } +} + +fn signup_of(s: &str) -> Result { + match s { + "open" => Ok(Parsed::Signup(true)), + "closed" => Ok(Parsed::Signup(false)), + _ => Err("must be \"open\" or \"closed\"".into()), + } +} + +/// The terms as the API shows them: absent links are left out. +pub fn terms_json(t: &Terms) -> Json { + let mut m = Map::new(); + m.insert("version".into(), json!(t.version)); + m.insert("terms_url".into(), json!(t.terms_url)); + if let Some(u) = &t.dpa_url { + m.insert("dpa_url".into(), json!(u)); + } + if let Some(u) = &t.privacy_url { + m.insert("privacy_url".into(), json!(u)); + } + Json::Object(m) +} + +/// The effective values from the startup env, the effective destinations policy and the override +/// rows (key, stored text), plus one stderr note per applied or ignored override. A row that no +/// longer validates, a `signup = open` while private URLs are allowed, and any signup row while +/// `PRINTAPI_SIGNUP` locks signup are ignored (and listed in `Live::ignored`). Notes never hold a +/// secret and print every user-supplied value with `{:?}`. +pub fn build_live( + env: &EnvStart, + destinations_any: bool, + rows: &[(String, String)], +) -> (Live, Vec) { + let src = |set: bool| if set { Source::Env } else { Source::Default }; + let mut live = Live { + signup: Sourced { + value: env.signup_open, + source: if env.signup_explicit_closed { + Source::EnvLocked + } else { + src(env.signup_set) + }, + }, + signup_quota: Sourced { + value: env.signup_quota, + source: src(env.signup_quota_set), + }, + job_retention_days: Sourced { + value: env.job_retention_days, + source: src(env.job_retention_days_set), + }, + terms: Sourced { + value: env.terms.clone(), + source: src(env.terms_set), + }, + ignored: Vec::new(), + }; + let opt = |v: Option| v.map_or("unset".to_string(), |n| n.to_string()); + let env_says = |key: &str| match key { + "signup" if env.signup_set => (if env.signup_open { "open" } else { "closed" }).to_string(), + "signup_quota" => opt(env.signup_quota), + "job_retention_days" => opt(env.job_retention_days), + "terms" => env + .terms + .as_ref() + .map_or("unset".to_string(), |t| format!("version {:?}", t.version)), + _ => "unset".to_string(), + }; + let mut notes = Vec::new(); + for key in LIVE_KEYS { + let Some((_, raw)) = rows.iter().find(|(k, _)| k == key) else { + continue; + }; + let parsed = match Parsed::decode(key, raw) { + Ok(p) => p, + Err(e) => { + notes.push(format!( + "setting {key}: ignoring the database value ({e}); the environment value applies" + )); + live.ignored.push((key, json!(raw))); + continue; + } + }; + if let Parsed::Signup(open) = &parsed { + if env.signup_explicit_closed { + notes.push( + "setting signup: ignoring the database value, PRINTAPI_SIGNUP locks signup closed" + .to_string(), + ); + live.ignored.push((key, parsed.to_json())); + continue; + } + if *open && destinations_any { + notes.push( + "setting signup: ignoring the database value open, private URLs are allowed" + .to_string(), + ); + live.ignored.push((key, parsed.to_json())); + continue; + } + } + let shown = match &parsed { + Parsed::Terms(t) => format!("version {:?}", t.version), + other => other.encode(), + }; + notes.push(format!( + "setting {key} = {shown} (from the database, env says {})", + env_says(key) + )); + match parsed { + Parsed::Signup(v) => { + live.signup = Sourced { + value: v, + source: Source::Db, + } + } + Parsed::SignupQuota(n) => { + live.signup_quota = Sourced { + value: Some(n), + source: Source::Db, + } + } + Parsed::JobRetentionDays(n) => { + live.job_retention_days = Sourced { + value: Some(n), + source: Source::Db, + } + } + Parsed::Terms(t) => { + live.terms = Sourced { + value: Some(t), + source: Source::Db, + } + } + } + } + (live, notes) +} + +/// A validated change: the values to write and the keys to remove, in `LIVE_KEYS` order. +#[derive(Debug, Clone, PartialEq, Default)] +pub struct Change { + pub set: Vec, + pub unset: Vec<&'static str>, +} + +/// A change that is well formed but not allowed right now (409 at the HTTP layer). +#[derive(Debug, Clone, Copy, PartialEq, Eq)] +pub enum Conflict { + /// The change would leave signup open while private URLs are allowed. + OpenWithPrivateUrls, + /// `PRINTAPI_SIGNUP` locks signup shut: only a change that closes it is accepted. + SignupLocked, +} + +#[derive(Debug, Clone, PartialEq)] +pub enum PutError { + /// Malformed or unknown (400). + Invalid(String), + /// A write that cannot be undone needs a reason of at least `REASON_MIN` characters (400). + ReasonNeeded, + Conflict(Conflict), +} + +/// Characters of reason an irreversible write needs. +pub const REASON_MIN: usize = 8; + +/// Check a `PUT /operator/settings` body (`{set, unset, reason?}`) against the effective values. +/// `reason` is the request's reason, already length-checked by the caller. Nothing is decided +/// from the body beyond the checks here; the caller writes `Change` and rebuilds `Live` from the +/// rows afterwards. +pub fn validate_put( + body: &Map, + reason: Option<&str>, + live: &Live, + env: &EnvStart, + destinations_any: bool, +) -> Result { + let bad = |m: String| PutError::Invalid(m); + if let Some(k) = body + .keys() + .find(|k| !["set", "unset", "reason"].contains(&k.as_str())) + { + return Err(bad(format!("unknown field {k:?}"))); + } + let empty = Map::new(); + let set = match body.get("set") { + None => &empty, + Some(Json::Object(m)) => m, + Some(_) => return Err(bad("set must be an object".into())), + }; + if let Some(k) = set.keys().find(|k| !LIVE_KEYS.contains(&k.as_str())) { + return Err(bad(format!("unknown setting {k:?}"))); + } + let mut unset: Vec<&'static str> = Vec::new(); + match body.get("unset") { + None => {} + Some(Json::Array(a)) => { + for v in a { + let name = v + .as_str() + .ok_or_else(|| bad("unset must be a list of keys".into()))?; + let key = LIVE_KEYS + .iter() + .find(|k| **k == name) + .ok_or_else(|| bad(format!("unknown setting {name:?}")))?; + unset.push(key); + } + } + Some(_) => return Err(bad("unset must be a list of keys".into())), + } + if set.is_empty() && unset.is_empty() { + return Err(bad("nothing to change: give set or unset".into())); + } + let mut parsed = Vec::new(); + for key in LIVE_KEYS { + let Some(v) = set.get(key) else { continue }; + if unset.contains(&key) { + return Err(bad(format!("{key} is both in set and in unset"))); + } + if v.is_null() { + return Err(bad(format!( + "{key}: null is not a value, unset the key to go back to the environment" + ))); + } + parsed.push(Parsed::from_json(key, v).map_err(|e| bad(format!("{key}: {e}")))?); + } + unset.sort_by_key(|k| LIVE_KEYS.iter().position(|l| l == k)); + unset.dedup(); + // well formed; now what the current state allows + // What signup ends up as, worked out as startup would (build_live): a set value, or with + // `unset` the env one. A PUT that does not touch signup is never refused here. + let new_signup = parsed.iter().find_map(|p| match p { + Parsed::Signup(open) => Some(*open), + _ => None, + }); + let touches_signup = new_signup.is_some() || unset.contains(&"signup"); + if touches_signup { + if live.signup.source == Source::EnvLocked { + // the brake: only changes that can close signup get through + if new_signup == Some(true) { + return Err(PutError::Conflict(Conflict::SignupLocked)); + } + } else if new_signup.unwrap_or(env.signup_open) && destinations_any { + return Err(PutError::Conflict(Conflict::OpenWithPrivateUrls)); + } + } + // the next retention pass scrubs payloads and titles for good: say why + let irreversible = parsed.iter().any(|p| match p { + Parsed::JobRetentionDays(n) => live.job_retention_days.value.is_none_or(|cur| *n < cur), + _ => false, + }); + if irreversible && !reason_is_enough(reason) { + return Err(PutError::ReasonNeeded); + } + Ok(Change { set: parsed, unset }) +} + +/// Would dropping the override of `job_retention_days` (`change.unset`) lower the effective +/// retention, or enable it over keep-for-ever? The same irreversible write as a `set` that does, +/// so it needs the same reason. `env` is the value it falls back to. +pub fn unset_needs_reason(change: &Change, live: &Live, env: &EnvStart) -> bool { + change.unset.contains(&"job_retention_days") + && live.job_retention_days.source == Source::Db + && env + .job_retention_days + .is_some_and(|e| live.job_retention_days.value.is_none_or(|cur| e < cur)) +} + +/// Does this reason satisfy `REASON_MIN`? +pub fn reason_is_enough(reason: Option<&str>) -> bool { + reason.is_some_and(|r| r.trim().chars().count() >= REASON_MIN) +} + +#[cfg(test)] +mod tests { + use super::*; + + fn terms(v: &str) -> Terms { + Terms { + version: v.into(), + terms_url: "https://x.test/t".into(), + dpa_url: None, + privacy_url: None, + } + } + + fn rows(r: &[(&str, &str)]) -> Vec<(String, String)> { + r.iter() + .map(|(k, v)| (k.to_string(), v.to_string())) + .collect() + } + + fn env() -> EnvStart { + EnvStart::default() + } + + // ---- build_live ---- + + #[test] + fn no_rows_means_the_env_values_with_env_or_default_as_source() { + let (live, notes) = build_live(&env(), false, &[]); + assert!(notes.is_empty()); + assert_eq!(live.signup.source, Source::Default); + assert_eq!(live.signup_quota.source, Source::Default); + let e = EnvStart { + signup_open: true, + signup_set: true, + signup_quota: Some(50), + signup_quota_set: true, + job_retention_days: Some(30), + job_retention_days_set: true, + terms: Some(terms("v1")), + terms_set: true, + ..env() + }; + let (live, _) = build_live(&e, false, &[]); + assert_eq!( + live.signup, + Sourced { + value: true, + source: Source::Env + } + ); + assert_eq!( + live.signup_quota, + Sourced { + value: Some(50), + source: Source::Env + } + ); + assert_eq!(live.job_retention_days.source, Source::Env); + assert_eq!(live.terms.value, Some(terms("v1"))); + assert_eq!(live.terms.source, Source::Env); + } + + #[test] + fn a_row_overrides_the_env_value_and_says_so() { + let e = EnvStart { + signup_quota: Some(50), + signup_quota_set: true, + ..env() + }; + let t = r#"{"version":"v2","terms_url":"https://x.test/t2","dpa_url":"https://x.test/d"}"#; + let (live, notes) = build_live( + &e, + false, + &rows(&[ + ("signup", "open"), + ("signup_quota", "5"), + ("job_retention_days", "14"), + ("terms", t), + ]), + ); + assert_eq!( + live.signup, + Sourced { + value: true, + source: Source::Db + } + ); + assert_eq!( + live.signup_quota, + Sourced { + value: Some(5), + source: Source::Db + } + ); + assert_eq!( + live.job_retention_days, + Sourced { + value: Some(14), + source: Source::Db + } + ); + let tv = live.terms.value.clone().unwrap(); + assert_eq!( + (tv.version.as_str(), tv.dpa_url.as_deref()), + ("v2", Some("https://x.test/d")) + ); + assert_eq!(live.terms.source, Source::Db); + assert_eq!(notes.len(), 4, "{notes:?}"); + assert!( + notes + .contains(&"setting signup = open (from the database, env says unset)".to_string()), + "{notes:?}" + ); + assert!( + notes + .iter() + .any(|n| n.contains("signup_quota = 5") && n.contains("env says 50")) + ); + } + + #[test] + fn an_explicit_env_closed_beats_a_db_open() { + let e = EnvStart { + signup_set: true, + signup_explicit_closed: true, + ..env() + }; + let (live, notes) = build_live(&e, false, &rows(&[("signup", "open")])); + assert_eq!( + live.signup, + Sourced { + value: false, + source: Source::EnvLocked + } + ); + assert_eq!(notes.len(), 1); + assert!( + notes[0].contains("ignor") && notes[0].contains("PRINTAPI_SIGNUP"), + "{notes:?}" + ); + // locked without any row, too + let (live, notes) = build_live(&e, false, &[]); + assert_eq!(live.signup.source, Source::EnvLocked); + assert!(notes.is_empty()); + } + + #[test] + fn an_unset_env_signup_is_closed_by_default_and_not_a_lock() { + let (live, _) = build_live(&env(), false, &rows(&[("signup", "open")])); + assert_eq!( + live.signup, + Sourced { + value: true, + source: Source::Db + } + ); + } + + #[test] + fn a_db_open_is_ignored_while_private_urls_are_allowed() { + let (live, notes) = build_live(&env(), true, &rows(&[("signup", "open")])); + assert_eq!( + live.signup, + Sourced { + value: false, + source: Source::Default + } + ); + assert_eq!(notes.len(), 1); + assert!( + notes[0].contains("ignor") && notes[0].contains("private"), + "{notes:?}" + ); + // closing is always fine + let e = EnvStart { + signup_open: true, + signup_set: true, + ..env() + }; + let (live, _) = build_live(&e, true, &rows(&[("signup", "closed")])); + assert_eq!( + live.signup, + Sourced { + value: false, + source: Source::Db + } + ); + } + + #[test] + fn a_row_that_no_longer_validates_is_ignored_with_a_note() { + let e = EnvStart { + signup_quota: Some(9), + signup_quota_set: true, + ..env() + }; + let (live, notes) = build_live( + &e, + false, + &rows(&[ + ("signup", "maybe"), + ("signup_quota", "-3"), + ("job_retention_days", "0"), + ("terms", r#"{"version":"v1","terms_url":"ftp://x"}"#), + ]), + ); + assert_eq!(live.signup.source, Source::Default); + assert_eq!( + live.signup_quota, + Sourced { + value: Some(9), + source: Source::Env + } + ); + assert_eq!(live.job_retention_days.source, Source::Default); + assert_eq!(live.terms.source, Source::Default); + assert_eq!(notes.len(), 4, "{notes:?}"); + assert!(notes.iter().all(|n| n.contains("ignor"))); + } + + #[test] + fn unknown_keys_in_the_rows_change_nothing() { + let (live, notes) = build_live(&env(), false, &rows(&[("fetches_per_org", "9")])); + assert_eq!(live, build_live(&env(), false, &[]).0); + assert!(notes.is_empty()); + } + + #[test] + fn ignored_rows_are_listed_with_their_stored_value() { + let locked = EnvStart { + signup_set: true, + signup_explicit_closed: true, + ..env() + }; + let l = build_live(&locked, false, &rows(&[("signup", "open")])).0; + assert_eq!(l.ignored, vec![("signup", json!("open"))]); + let l = build_live(&env(), true, &rows(&[("signup", "open")])).0; + assert_eq!(l.ignored, vec![("signup", json!("open"))]); + let l = build_live(&env(), false, &rows(&[("signup_quota", "-3")])).0; + assert_eq!(l.ignored, vec![("signup_quota", json!("-3"))]); + // an applied row is not ignored + let l = build_live(&env(), false, &rows(&[("signup", "open")])).0; + assert!(l.ignored.is_empty()); + } + + #[test] + fn the_terms_version_has_no_control_character_and_at_most_100_characters() { + let ok = |v: &str| parse_terms(v, "https://x.test/t", "", ""); + assert!(ok("v1").unwrap().is_some()); + assert!(ok(&"a".repeat(100)).is_ok()); + assert!(ok(&"a".repeat(101)).is_err()); + assert!(ok(&"é".repeat(100)).is_ok(), "characters, not bytes"); + for v in ["v1\u{1b}[2J", "v1\nx", "a\tb", "a\u{7f}b", "a\u{85}b"] { + let e = ok(v).unwrap_err(); + assert!(e.contains("control"), "{v:?}: {e}"); + } + // padding is stripped before the check, links with a control character are refused + assert!(ok(" v1\n").is_ok()); + assert!(parse_terms("v1", "https://x.test/\u{1b}", "", "").is_err()); + assert!(parse_terms("", "", "https://x.test/a\nb", "").is_err()); + } + + // ---- stored text and request values ---- + + #[test] + fn values_round_trip_through_the_stored_text() { + let t = Terms { + version: "v3".into(), + terms_url: "https://x.test/t".into(), + dpa_url: Some("https://x.test/d".into()), + privacy_url: Some("http://x.test/p".into()), + }; + for p in [ + Parsed::Signup(true), + Parsed::Signup(false), + Parsed::SignupQuota(0), + Parsed::JobRetentionDays(7), + Parsed::Terms(t), + ] { + assert_eq!(Parsed::decode(p.key(), &p.encode()), Ok(p.clone()), "{p:?}"); + assert_eq!( + Parsed::from_json(p.key(), &p.to_json()), + Ok(p.clone()), + "{p:?}" + ); + } + } + + #[test] + fn request_values_follow_the_env_parsers_rules() { + let ok = |k: &str, v: Json| Parsed::from_json(k, &v); + assert_eq!(ok("signup", json!("open")), Ok(Parsed::Signup(true))); + assert!(ok("signup", json!("Open")).is_err()); + assert!(ok("signup", json!(true)).is_err()); + assert_eq!(ok("signup_quota", json!(0)), Ok(Parsed::SignupQuota(0))); + assert!(ok("signup_quota", json!(-1)).is_err()); + assert!(ok("signup_quota", json!("5")).is_err()); + assert!(ok("signup_quota", json!(1.5)).is_err()); + assert_eq!( + ok("job_retention_days", json!(30)), + Ok(Parsed::JobRetentionDays(30)) + ); + assert!( + ok("job_retention_days", json!(0)).is_err(), + "0 is unset, not a value" + ); + assert!(ok("job_retention_days", json!(-2)).is_err()); + let t = |v: Json| ok("terms", v); + assert!(t(json!({"version":"v1","terms_url":"https://x.test/t"})).is_ok()); + assert!(t(json!({"version":"","terms_url":"https://x.test/t"})).is_err()); + assert!(t(json!({"version":" ","terms_url":"https://x.test/t"})).is_err()); + assert!(t(json!({"version":"v1"})).is_err(), "terms_url is required"); + assert!(t(json!({"version":"v1","terms_url":"ftp://x/t"})).is_err()); + assert!( + t(json!({"version":"v1","terms_url":"https://x.test/t","dpa_url":"javascript:x"})) + .is_err() + ); + assert!(t(json!({"version":"v1","terms_url":"https://x.test/t","extra":1})).is_err()); + assert!(t(json!({"version":1,"terms_url":"https://x.test/t"})).is_err()); + assert!(t(json!("v1")).is_err()); + } + + // ---- validate_put ---- + + fn put(body: Json, reason: Option<&str>, live: &Live, any: bool) -> Result { + put_env(body, reason, live, &env(), any) + } + + fn put_env( + body: Json, + reason: Option<&str>, + live: &Live, + env: &EnvStart, + any: bool, + ) -> Result { + validate_put(body.as_object().unwrap(), reason, live, env, any) + } + + fn base() -> Live { + build_live(&env(), false, &[]).0 + } + + fn invalid(r: Result) -> bool { + matches!(r, Err(PutError::Invalid(_))) + } + + #[test] + fn a_valid_change_comes_back_in_key_order() { + let c = put( + json!({"set": {"signup_quota": 10, "signup": "open"}, "unset": ["terms"]}), + None, + &base(), + false, + ) + .unwrap(); + assert_eq!(c.set, vec![Parsed::Signup(true), Parsed::SignupQuota(10)]); + assert_eq!(c.unset, vec!["terms"]); + } + + #[test] + fn malformed_requests_are_invalid() { + let l = base(); + assert!(invalid(put(json!({}), None, &l, false)), "empty"); + assert!( + invalid(put(json!({"set": {}, "unset": []}), None, &l, false)), + "empty lists" + ); + assert!( + invalid(put(json!({"set": {"fetches_per_org": 3}}), None, &l, false)), + "unknown key" + ); + assert!( + invalid(put(json!({"unset": ["nope"]}), None, &l, false)), + "unknown unset key" + ); + assert!( + invalid(put(json!({"set": {"signup": null}}), None, &l, false)), + "null" + ); + assert!( + invalid(put(json!({"set": {"signup_quota": "5"}}), None, &l, false)), + "wrong type" + ); + assert!( + invalid(put(json!({"set": [1]}), None, &l, false)), + "set not an object" + ); + assert!( + invalid(put(json!({"unset": "signup"}), None, &l, false)), + "unset not a list" + ); + assert!( + invalid(put(json!({"unset": [1]}), None, &l, false)), + "unset entry not a string" + ); + assert!( + invalid(put( + json!({"unset": ["terms"], "extra": 1}), + None, + &l, + false + )), + "unknown top key" + ); + assert!( + invalid(put( + json!({"set": {"signup_quota": 1}, "unset": ["signup_quota"]}), + None, + &l, + false + )), + "both set and unset" + ); + } + + #[test] + fn open_signup_is_refused_while_private_urls_are_allowed() { + let l = base(); + assert_eq!( + put(json!({"set": {"signup": "open"}}), None, &l, true), + Err(PutError::Conflict(Conflict::OpenWithPrivateUrls)) + ); + assert!(put(json!({"set": {"signup": "closed"}}), None, &l, true).is_ok()); + assert!(put(json!({"set": {"signup": "open"}}), None, &l, false).is_ok()); + } + + #[test] + fn unset_cannot_reopen_signup_while_private_urls_are_allowed() { + let e = EnvStart { + signup_open: true, + signup_set: true, + ..env() + }; + let rows = rows(&[("signup", "closed")]); + let l = build_live(&e, true, &rows).0; + assert!(!l.signup.value); + assert_eq!( + put_env(json!({"unset": ["signup"]}), None, &l, &e, true), + Err(PutError::Conflict(Conflict::OpenWithPrivateUrls)) + ); + // not touching signup, closing it, or an env that says closed: all fine + assert!(put_env(json!({"set": {"signup_quota": 2}}), None, &l, &e, true).is_ok()); + assert!(put_env(json!({"set": {"signup": "closed"}}), None, &l, &e, true).is_ok()); + assert!(put_env(json!({"unset": ["signup"]}), None, &l, &env(), true).is_ok()); + assert!(put_env(json!({"unset": ["signup"]}), None, &l, &e, false).is_ok()); + } + + #[test] + fn an_env_locked_signup_accepts_only_changes_that_close_it() { + let e = EnvStart { + signup_set: true, + signup_explicit_closed: true, + ..env() + }; + let l = build_live(&e, false, &[]).0; + for body in [ + json!({"set": {"signup": "open"}}), + json!({"set": {"signup": "open", "signup_quota": 1}}), + ] { + assert_eq!( + put(body.clone(), None, &l, false), + Err(PutError::Conflict(Conflict::SignupLocked)), + "{body}" + ); + } + assert!(put(json!({"set": {"signup": "closed"}}), None, &l, false).is_ok()); + assert!(put(json!({"unset": ["signup"]}), None, &l, false).is_ok()); + // the other keys stay writable + assert!(put(json!({"set": {"signup_quota": 1}}), None, &l, false).is_ok()); + } + + #[test] + fn malformed_input_is_reported_before_a_conflict() { + let e = EnvStart { + signup_set: true, + signup_explicit_closed: true, + ..env() + }; + let l = build_live(&e, false, &[]).0; + assert!(invalid(put( + json!({"set": {"signup": "open", "terms": 3}}), + None, + &l, + false + ))); + } + + #[test] + fn enabling_retention_over_keep_for_ever_needs_a_reason() { + let l = base(); // retention None + let body = json!({"set": {"job_retention_days": 30}}); + assert_eq!( + put(body.clone(), None, &l, false), + Err(PutError::ReasonNeeded) + ); + assert_eq!( + put(body.clone(), Some("short"), &l, false), + Err(PutError::ReasonNeeded) + ); + assert_eq!( + put(body.clone(), Some(" 1234567 "), &l, false), + Err(PutError::ReasonNeeded) + ); + assert!(put(body, Some("12345678"), &l, false).is_ok()); + } + + #[test] + fn lowering_retention_needs_a_reason_raising_or_keeping_does_not() { + let e = EnvStart { + job_retention_days: Some(30), + job_retention_days_set: true, + ..env() + }; + let l = build_live(&e, false, &[]).0; + let set = |n: i64| json!({"set": {"job_retention_days": n}}); + assert_eq!(put(set(29), None, &l, false), Err(PutError::ReasonNeeded)); + assert!(put(set(29), Some("ticket 4711"), &l, false).is_ok()); + assert!(put(set(30), None, &l, false).is_ok()); + assert!(put(set(90), None, &l, false).is_ok()); + // other keys never need one + assert!( + put( + json!({"set": {"signup_quota": 0}, "unset": ["job_retention_days"]}), + None, + &l, + false + ) + .is_ok() + ); + } + + #[test] + fn the_reason_rule_reads_the_effective_value_not_the_env_one() { + let rows = rows(&[("job_retention_days", "10")]); + let l = build_live(&env(), false, &rows).0; + assert_eq!(l.job_retention_days.value, Some(10)); + assert_eq!( + put(json!({"set": {"job_retention_days": 5}}), None, &l, false), + Err(PutError::ReasonNeeded) + ); + assert!(put(json!({"set": {"job_retention_days": 20}}), None, &l, false).is_ok()); + } +} diff --git a/crates/server/src/store.rs b/crates/server/src/store.rs index 429443f..6a8fafd 100644 --- a/crates/server/src/store.rs +++ b/crates/server/src/store.rs @@ -413,6 +413,12 @@ CREATE TABLE IF NOT EXISTS support_grants( reason TEXT NOT NULL, granted_at REAL NOT NULL, expires_at REAL NOT NULL, ended_at REAL, reads INTEGER NOT NULL DEFAULT 0); CREATE INDEX IF NOT EXISTS support_grants_org ON support_grants(org_id); +-- Live setting overrides the operator console writes (signup, signup_quota, job_retention_days, +-- terms), read only while the operator token is set; the env value applies without a row. `value` +-- is text (`open`, a number, or the terms as JSON). A table of its own, like operator_audit: the +-- name is free in the v2.5.1 schema, so the Python server ignores it. +CREATE TABLE IF NOT EXISTS settings( + key TEXT PRIMARY KEY, value TEXT NOT NULL, updated_at REAL NOT NULL); -- finds the audit rows of one grant (`grant:`) without a scan CREATE INDEX IF NOT EXISTS operator_audit_target ON operator_audit(target); "; @@ -3476,6 +3482,56 @@ pub fn db_sizes(conn: &Connection) -> Result<(i64, i64)> { Ok((pages * page_size, wal)) } +/// The override rows of the four live settings (key, stored text), by key. Anything else in the +/// table is not read. +pub fn settings_rows(conn: &Connection) -> Result> { + let keys = crate::settings::LIVE_KEYS + .map(|k| format!("'{k}'")) + .join(","); + let mut st = conn.prepare(&format!( + "SELECT key, value FROM settings WHERE key IN ({keys}) ORDER BY key" + ))?; + let rows = st + .query_map([], |r| Ok((r.get(0)?, r.get(1)?)))? + .collect::, _>>()?; + Ok(rows) +} + +/// `settings.update`: the validated change and its one audit row in one transaction. +pub fn operator_update_settings( + conn: &Connection, + change: &crate::settings::Change, + reason: Option<&str>, +) -> Result<()> { + let tx = begin(conn)?; + let now = crate::now(); + let mut set = Map::new(); + for p in &change.set { + tx.execute( + "INSERT INTO settings(key, value, updated_at) VALUES(?,?,?) \ + ON CONFLICT(key) DO UPDATE SET value=excluded.value, updated_at=excluded.updated_at", + params![p.key(), p.encode(), now], + )?; + set.insert(p.key().to_string(), p.to_json()); + } + for key in &change.unset { + tx.execute("DELETE FROM settings WHERE key=?", [key])?; + } + // setting values are allowed in the log (spec A2): ids, numbers, modes and links, no secret + let detail = serde_json::json!({ "set": set, "unset": change.unset }).to_string(); + insert_audit( + &tx, + &AuditRow { + action: "settings.update", + reason, + detail: Some(&detail), + ..Default::default() + }, + )?; + tx.commit()?; + Ok(()) +} + #[cfg(test)] mod tests { use super::*; diff --git a/crates/server/tests/http_operator.rs b/crates/server/tests/http_operator.rs index 3db328b..a295eef 100644 --- a/crates/server/tests/http_operator.rs +++ b/crates/server/tests/http_operator.rs @@ -461,7 +461,7 @@ fn overview_counts_orgs_jobs_and_failures() { assert_eq!(v["orgs"], 1); assert_eq!(v["signups_7d"], 0); assert_eq!(v["jobs_this_month"], 0); - assert_eq!(v["features"], json!(["support"])); + assert_eq!(v["features"], json!(["support", "settings"])); assert_eq!( v["jobs_today"], json!({"queued": 0, "claimed": 0, "done": 0, "failed": 0, "cancelled": 0}) diff --git a/crates/server/tests/http_operator_settings.rs b/crates/server/tests/http_operator_settings.rs new file mode 100644 index 0000000..5658d49 --- /dev/null +++ b/crates/server/tests/http_operator_settings.rs @@ -0,0 +1,644 @@ +// printpapi — self-hosted PrintNode alternative. Elastic License 2.0 (see LICENSE). +//! `GET` and `PUT /operator/settings` over HTTP (spec A5, "PR 3 decisions"): the live values with +//! their source, the validated write with its audit row, and the swap that follows it. +mod common; + +use common::*; +use printpapi_server::dispatch::Destinations; +use printpapi_server::proxy::TrustedProxies; +use printpapi_server::server::{Config, Terms}; +use printpapi_server::store::{self, Db}; +use serde_json::{Value, json}; + +const OP: &str = "operator-token-0123456789abcdef-xyz"; +const ROOT: &str = "root-token-0123456789abcdef-xyz"; +const BILLING: &str = "billing-secret-0123456789-xyz"; +const CONSOLE: &str = "10.9.0.2"; +const REASON: &str = "retention for the new terms"; + +/// Hosted the way `main` sets it up: the token, its peer and the private-URL deny. +fn hosted_cfg(c: &mut Config) { + c.token = ROOT.into(); + c.operator_token = Some(OP.into()); + c.operator_from = TrustedProxies::parse(CONSOLE).unwrap(); + c.destinations = Destinations::PublicOnly; + c.fetches_per_org = Some(4); + c.public_url = "https://print.example.test".into(); + c.reset_enabled = true; + c.billing_secret = Some(BILLING.into()); + c.trusted_proxies = TrustedProxies::parse("10.0.0.0/8, 192.168.1.5").unwrap(); +} + +fn app() -> App { + App::with(hosted_cfg) +} + +fn op<'a>(app: &'a App, method: &str, target: &str) -> Call<'a> { + app.call(method, target).bearer(OP).socket(CONSOLE) +} + +fn get(app: &App) -> Value { + let r = op(app, "GET", "/operator/settings").send(); + assert_eq!(r.status, 200, "{}", r.text()); + r.json() +} + +fn put(app: &App, body: Value) -> Reply { + op(app, "PUT", "/operator/settings").json(&body).send() +} + +fn put_ok(app: &App, body: Value) -> Value { + let r = put(app, body); + assert_eq!(r.status, 200, "{}", r.text()); + r.json() +} + +fn rows(db: &Db) -> Vec<(String, String)> { + db.with(|c| store::settings_rows(c).unwrap()) +} + +fn audit_of(app: &App, action: &str) -> Vec { + let r = op(app, "GET", "/operator/audit?limit=200").send(); + assert_eq!(r.status, 200, "{}", r.text()); + r.json()["entries"] + .as_array() + .unwrap() + .iter() + .filter(|e| e["action"] == action) + .cloned() + .collect() +} + +fn terms_body() -> Value { + json!({"version": "2026-11", "terms_url": "https://x.test/terms", + "privacy_url": "https://x.test/privacy"}) +} + +// --- GET ---------------------------------------------------------------------------------------- + +#[test] +fn get_shows_every_key_with_its_default_and_the_env_only_values() { + let app = app(); + let r = op(&app, "GET", "/operator/settings").send(); + assert_eq!(r.status, 200, "{}", r.text()); + assert_eq!(r.header("cache-control"), Some("no-store")); + let v = r.json(); + assert_eq!( + v["settings"], + json!({ + "signup": {"value": "closed", "source": "default"}, + "signup_quota": {"value": null, "source": "default"}, + "job_retention_days": {"value": null, "source": "default"}, + "terms": {"value": null, "source": "default"}, + }) + ); + assert_eq!( + v["env_only"], + json!({ + "fetches_per_org": 4, + "private_urls": "deny", + "trusted_proxies": ["10.0.0.0/8", "192.168.1.5"], + "operator_from": ["10.9.0.2"], + "public_url": "https://print.example.test", + "mail": true, + }) + ); +} + +#[test] +fn get_names_the_env_as_source_of_a_variable_that_is_set() { + let app = App::with(|c| { + hosted_cfg(c); + c.signup_open = true; + c.env_flags.signup_set = true; + c.signup_quota = Some(25); + c.env_flags.signup_quota_set = true; + c.job_retention_days = Some(14); + c.env_flags.job_retention_days_set = true; + c.terms = Some(Terms { + version: "v1".into(), + terms_url: "https://x.test/t".into(), + dpa_url: None, + privacy_url: None, + }); + c.env_flags.terms_set = true; + }); + let s = &get(&app)["settings"]; + assert_eq!(s["signup"], json!({"value": "open", "source": "env"})); + assert_eq!(s["signup_quota"], json!({"value": 25, "source": "env"})); + assert_eq!( + s["job_retention_days"], + json!({"value": 14, "source": "env"}) + ); + assert_eq!( + s["terms"], + json!({"value": {"version": "v1", "terms_url": "https://x.test/t"}, "source": "env"}) + ); +} + +#[test] +fn get_marks_an_explicit_env_closed_as_locked() { + let app = App::with(|c| { + hosted_cfg(c); + c.env_flags.signup_set = true; + c.env_flags.signup_explicit_closed = true; + }); + assert_eq!( + get(&app)["settings"]["signup"], + json!({"value": "closed", "source": "env-locked"}) + ); +} + +#[test] +fn get_never_shows_a_token_or_a_secret() { + let app = app(); + put_ok(&app, json!({"set": {"terms": terms_body()}})); + let body = op(&app, "GET", "/operator/settings").send().text(); + for secret in [OP, ROOT, BILLING] { + assert!(!body.contains(secret), "{secret} leaked: {body}"); + } + let put_body = put(&app, json!({"set": {"signup_quota": 3}})).text(); + for secret in [OP, ROOT, BILLING] { + assert!(!put_body.contains(secret), "{secret} leaked: {put_body}"); + } +} + +// --- PUT ---------------------------------------------------------------------------------------- + +#[test] +fn put_then_get_shows_the_values_as_db_and_answers_in_the_get_shape() { + let app = app(); + let v = put_ok( + &app, + json!({"set": {"signup": "open", "signup_quota": 7, "terms": terms_body()}}), + ); + assert_eq!(v, get(&app), "the PUT answers like the GET"); + let s = &v["settings"]; + assert_eq!(s["signup"], json!({"value": "open", "source": "db"})); + assert_eq!(s["signup_quota"], json!({"value": 7, "source": "db"})); + assert_eq!(s["terms"]["source"], "db"); + assert_eq!(s["terms"]["value"], terms_body()); + assert_eq!(s["job_retention_days"]["source"], "default"); + assert_eq!(v["env_only"]["private_urls"], "deny"); +} + +#[test] +fn the_new_values_apply_at_once() { + let app = app(); + assert_eq!(app.get("/health", None).1["signup"], "closed"); + put_ok(&app, json!({"set": {"signup": "open"}})); + assert_eq!(app.get("/health", None).1["signup"], "open"); + put_ok(&app, json!({"set": {"terms": terms_body()}})); + assert_eq!(app.get("/health", None).1["terms"]["version"], "2026-11"); +} + +#[test] +fn unset_goes_back_to_the_env_value_or_the_default() { + let app = App::with(|c| { + hosted_cfg(c); + c.signup_quota = Some(25); + c.env_flags.signup_quota_set = true; + }); + put_ok(&app, json!({"set": {"signup_quota": 7, "signup": "open"}})); + let v = put_ok(&app, json!({"unset": ["signup_quota", "signup"]})); + assert_eq!( + v["settings"]["signup_quota"], + json!({"value": 25, "source": "env"}) + ); + assert_eq!( + v["settings"]["signup"], + json!({"value": "closed", "source": "default"}) + ); + assert!(rows(&app.db).is_empty()); +} + +#[test] +fn a_set_and_an_unset_of_different_keys_go_in_one_request() { + let app = app(); + put_ok(&app, json!({"set": {"signup_quota": 7}})); + let v = put_ok( + &app, + json!({"set": {"terms": terms_body()}, "unset": ["signup_quota"]}), + ); + assert_eq!(v["settings"]["signup_quota"]["source"], "default"); + assert_eq!(v["settings"]["terms"]["source"], "db"); + assert_eq!(rows(&app.db).len(), 1); +} + +#[test] +fn every_malformed_request_is_a_400_that_writes_nothing() { + let app = app(); + let long = "x".repeat(501); + let bad = [ + json!({}), + json!({"set": {}}), + json!({"unset": []}), + json!({"set": {"nope": 1}}), + json!({"unset": ["nope"]}), + json!({"set": {"signup_quota": 3}, "unset": ["signup_quota"]}), + json!({"set": {"signup_quota": null}}), + json!({"set": {"signup": "maybe"}}), + json!({"set": {"signup": true}}), + json!({"set": {"signup_quota": -1}}), + json!({"set": {"signup_quota": "5"}}), + json!({"set": {"job_retention_days": 0}, "reason": REASON}), + json!({"set": {"terms": {"version": "", "terms_url": "https://x.test/t"}}}), + json!({"set": {"terms": {"version": "v1"}}}), + json!({"set": {"terms": {"version": "v1", "terms_url": "ftp://x.test/t"}}}), + json!({"set": {"terms": "v1"}}), + json!({"set": {"signup_quota": 3}, "reason": 5}), + json!({"set": {"signup_quota": 3}, "reason": long}), + json!({"set": [], "unset": []}), + json!({"set": {"signup_quota": 3}, "extra": 1}), + ]; + for body in bad { + let r = put(&app, body.clone()); + assert_eq!(r.status, 400, "{body}: {}", r.text()); + assert!( + r.json()["error"].as_str().is_some_and(|e| !e.is_empty()), + "{body}: a message" + ); + } + assert!(rows(&app.db).is_empty()); + assert!(audit_of(&app, "settings.update").is_empty()); + // not a JSON object at all + let r = op(&app, "PUT", "/operator/settings") + .raw(b"not json") + .send(); + assert_eq!(r.status, 400); +} + +#[test] +fn signup_open_is_refused_with_409_while_private_urls_are_allowed() { + let app = App::with(|c| { + hosted_cfg(c); + c.destinations = Destinations::Any; + }); + let r = put(&app, json!({"set": {"signup": "open"}})); + assert_eq!(r.status, 409, "{}", r.text()); + assert!(rows(&app.db).is_empty()); + // closing is fine, and so is everything else + put_ok( + &app, + json!({"set": {"signup": "closed", "signup_quota": 3}}), + ); +} + +#[test] +fn an_env_lock_refuses_every_open_and_leaves_the_other_keys_open() { + let app = App::with(|c| { + hosted_cfg(c); + c.env_flags.signup_set = true; + c.env_flags.signup_explicit_closed = true; + }); + for body in [ + json!({"set": {"signup": "open"}}), + json!({"set": {"signup": "open", "signup_quota": 3}}), + ] { + let r = put(&app, body.clone()); + assert_eq!(r.status, 409, "{body}: {}", r.text()); + } + assert!(rows(&app.db).is_empty(), "nothing written on a 409"); + // the other keys stay open to change + put_ok(&app, json!({"set": {"signup_quota": 3}})); +} + +/// Env open, private URLs allowed: signup may be closed but never reopened over the API. +fn open_with_private_urls(c: &mut Config) { + hosted_cfg(c); + c.signup_open = true; + c.env_flags.signup_set = true; + c.destinations = Destinations::Any; +} + +#[test] +fn unset_signup_cannot_reopen_signup_while_private_urls_are_allowed() { + let app = App::with(open_with_private_urls); + put_ok(&app, json!({"set": {"signup": "closed"}})); + let r = put(&app, json!({"unset": ["signup"]})); + assert_eq!(r.status, 409, "{}", r.text()); + assert_eq!( + get(&app)["settings"]["signup"], + json!({"value": "closed", "source": "db"}) + ); + assert_eq!( + rows(&app.db), + vec![("signup".to_string(), "closed".to_string())] + ); + // a PUT that leaves signup alone is not affected + put_ok(&app, json!({"set": {"signup_quota": 3}})); + // and unset is fine when it does not reopen: the env says closed + let app = App::with(|c| { + hosted_cfg(c); + c.destinations = Destinations::Any; + }); + put_ok(&app, json!({"set": {"signup": "closed"}})); + put_ok(&app, json!({"unset": ["signup"]})); +} + +fn locked(c: &mut Config) { + hosted_cfg(c); + c.env_flags.signup_set = true; + c.env_flags.signup_explicit_closed = true; +} + +fn seed_row(db: &Db, key: &str, value: &str) { + db.with(|c| { + c.execute( + "INSERT OR REPLACE INTO settings(key, value, updated_at) VALUES(?,?,1.0)", + rusqlite::params![key, value], + ) + .unwrap(); + }); +} + +#[test] +fn while_locked_only_closing_changes_are_accepted() { + let db = mem(); + seed_row(&db, "signup", "open"); + let app = App::on(db, locked); + let r = put(&app, json!({"set": {"signup": "open"}})); + assert_eq!(r.status, 409, "{}", r.text()); + assert_eq!(rows(&app.db).len(), 1, "nothing written on a 409"); + put_ok(&app, json!({"set": {"signup": "closed"}})); + assert_eq!( + rows(&app.db), + vec![("signup".to_string(), "closed".to_string())] + ); + let v = put_ok(&app, json!({"unset": ["signup"]})); + assert!(rows(&app.db).is_empty()); + assert_eq!( + v["settings"]["signup"], + json!({"value": "closed", "source": "env-locked"}) + ); +} + +#[test] +fn get_and_put_show_a_row_that_is_not_applied_as_ignored() { + // env-locked: the stale open row is visible and can be cleared + let db = mem(); + seed_row(&db, "signup", "open"); + let app = App::on(db, locked); + assert_eq!( + get(&app)["settings"]["signup"], + json!({"value": "closed", "source": "env-locked", "ignored": "open"}) + ); + // the PUT answer has the same shape + let v = put_ok(&app, json!({"set": {"signup_quota": 3}})); + assert_eq!(v["settings"]["signup"]["ignored"], "open"); + assert!(v["settings"]["signup_quota"].get("ignored").is_none()); + let v = put_ok(&app, json!({"unset": ["signup"]})); + assert!(v["settings"]["signup"].get("ignored").is_none()); + // the private-URL cross-rule + let db = mem(); + seed_row(&db, "signup", "open"); + let app = App::on(db, |c| { + hosted_cfg(c); + c.destinations = Destinations::Any; + }); + assert_eq!( + get(&app)["settings"]["signup"], + json!({"value": "closed", "source": "default", "ignored": "open"}) + ); + // a value that no longer validates; an applied row shows nothing + let db = mem(); + seed_row(&db, "signup_quota", "-3"); + seed_row(&db, "job_retention_days", "14"); + let app = App::on(db, hosted_cfg); + let v = get(&app); + assert_eq!(v["settings"]["signup_quota"]["ignored"], "-3"); + assert!(v["settings"]["job_retention_days"].get("ignored").is_none()); + assert_eq!(v["settings"]["job_retention_days"]["source"], "db"); +} + +#[test] +fn a_terms_version_with_control_characters_or_over_100_chars_is_refused() { + let app = app(); + for version in [ + "v1\u{1b}[2J".to_string(), + "v1\nsetting x".to_string(), + "v1\0".to_string(), + "a".repeat(101), + ] { + let r = put( + &app, + json!({"set": {"terms": {"version": version, "terms_url": "https://x.test/t"}}}), + ); + assert_eq!(r.status, 400, "{version:?}: {}", r.text()); + } + // a URL with a control character or a space inside is refused as well + for u in ["https://x.test/\u{1b}t", "https://x.test/a b"] { + let r = put( + &app, + json!({"set": {"terms": {"version": "v1", "terms_url": u}}}), + ); + assert_eq!(r.status, 400, "{u:?}"); + } + assert!(rows(&app.db).is_empty()); + put_ok( + &app, + json!({"set": {"terms": {"version": "a".repeat(100), "terms_url": "https://x.test/t"}}}), + ); +} + +#[test] +fn a_put_writes_one_settings_update_audit_row_without_an_email() { + let app = app(); + app.db.with(|c| { + store::signup(c, "Acme", "boss@acme.test", "PWHASH", None, None).unwrap(); + }); + put_ok( + &app, + json!({"set": {"signup_quota": 5, "terms": terms_body()}, + "unset": ["job_retention_days"], "reason": "new quota"}), + ); + let rows = audit_of(&app, "settings.update"); + assert_eq!(rows.len(), 1); + assert_eq!(rows[0]["reason"], "new quota"); + assert_eq!( + rows[0]["detail"], + json!({"set": {"signup_quota": 5, "terms": terms_body()}, + "unset": ["job_retention_days"]}) + ); + assert_eq!(rows[0]["org_id"], Value::Null); + assert!(!rows[0].to_string().contains('@'), "no e-mail address"); +} + +#[test] +fn two_puts_in_a_row_end_with_the_live_values_equal_to_the_rows() { + let app = app(); + put_ok(&app, json!({"set": {"signup": "open", "signup_quota": 3}})); + put_ok( + &app, + json!({"set": {"signup_quota": 9}, "unset": ["signup"]}), + ); + let v = get(&app); + assert_eq!( + v["settings"]["signup"], + json!({"value": "closed", "source": "default"}) + ); + assert_eq!( + v["settings"]["signup_quota"], + json!({"value": 9, "source": "db"}) + ); + assert_eq!(app.get("/health", None).1["signup"], "closed"); + assert_eq!(rows(&app.db), vec![("signup_quota".into(), "9".into())]); + // and a later GET (a new snapshot) agrees with the first + assert_eq!(get(&app), v); +} + +// --- the retention reason rule ------------------------------------------------------------------ + +#[test] +fn enabling_or_lowering_retention_needs_a_reason_of_eight_characters() { + let app = app(); + for body in [ + json!({"set": {"job_retention_days": 30}}), + json!({"set": {"job_retention_days": 30}, "reason": "short"}), + json!({"set": {"job_retention_days": 30}, "reason": " short "}), + ] { + let r = put(&app, body.clone()); + assert_eq!(r.status, 400, "{body}: {}", r.text()); + } + assert!(rows(&app.db).is_empty()); + assert!(audit_of(&app, "settings.update").is_empty()); + let v = put_ok( + &app, + json!({"set": {"job_retention_days": 30}, "reason": REASON}), + ); + assert_eq!( + v["settings"]["job_retention_days"], + json!({"value": 30, "source": "db"}) + ); + // raising it destroys nothing: no reason + put_ok(&app, json!({"set": {"job_retention_days": 60}})); + // lowering it again does + assert_eq!( + put(&app, json!({"set": {"job_retention_days": 10}})).status, + 400 + ); + put_ok( + &app, + json!({"set": {"job_retention_days": 10}, "reason": REASON}), + ); + assert_eq!(audit_of(&app, "settings.update").len(), 3); +} + +#[test] +fn unsetting_an_override_that_falls_back_to_a_shorter_env_value_needs_a_reason() { + let app = App::with(|c| { + hosted_cfg(c); + c.job_retention_days = Some(14); + c.env_flags.job_retention_days_set = true; + }); + put_ok(&app, json!({"set": {"job_retention_days": 90}})); + let r = put(&app, json!({"unset": ["job_retention_days"]})); + assert_eq!(r.status, 400, "{}", r.text()); + assert_eq!(rows(&app.db).len(), 1, "still overridden"); + let v = put_ok( + &app, + json!({"unset": ["job_retention_days"], "reason": REASON}), + ); + assert_eq!( + v["settings"]["job_retention_days"], + json!({"value": 14, "source": "env"}) + ); +} + +#[test] +fn unsetting_a_retention_override_that_falls_back_to_a_longer_value_needs_none() { + let app = App::with(|c| { + hosted_cfg(c); + c.job_retention_days = Some(90); + c.env_flags.job_retention_days_set = true; + }); + put_ok( + &app, + json!({"set": {"job_retention_days": 30}, "reason": REASON}), + ); + put_ok(&app, json!({"unset": ["job_retention_days"]})); + // and falling back to keep-for-ever destroys nothing either + let app = app_with_nothing(); + put_ok( + &app, + json!({"set": {"job_retention_days": 30}, "reason": REASON}), + ); + put_ok(&app, json!({"unset": ["job_retention_days"]})); +} + +fn app_with_nothing() -> App { + app() +} + +// --- who may call, and when --------------------------------------------------------------------- + +#[test] +fn the_endpoints_are_401_for_root_a_wrong_token_and_a_foreign_peer() { + let app = app(); + for (method, body) in [ + ("GET", None), + ("PUT", Some(json!({"set": {"signup_quota": 1}}))), + ] { + let mk = |token: &str, peer: &str| { + let mut c = app + .call(method, "/operator/settings") + .bearer(token) + .socket(peer); + if let Some(b) = &body { + c = c.json(b); + } + c.send() + }; + assert_eq!(mk(ROOT, CONSOLE).status, 401, "{method} root"); + assert_eq!(mk("wrong", CONSOLE).status, 401, "{method} wrong token"); + assert_eq!(mk(OP, "203.0.113.9").status, 401, "{method} foreign peer"); + assert_eq!(mk(OP, "127.0.0.1").status, 401, "{method} loopback"); + let mut anon = app.call(method, "/operator/settings").socket(CONSOLE); + if let Some(b) = &body { + anon = anon.json(b); + } + assert_eq!(anon.send().status, 401, "{method} no credential"); + } + assert!(rows(&app.db).is_empty()); +} + +#[test] +fn without_the_operator_token_the_endpoints_are_404_and_the_table_is_never_written() { + let app = App::with(|c| { + c.destinations = Destinations::PublicOnly; + }); + for (method, body) in [ + ("GET", None), + ("PUT", Some(json!({"set": {"signup_quota": 1}}))), + ] { + let mut c = app + .call(method, "/operator/settings") + .bearer(OP) + .socket(CONSOLE); + if let Some(b) = &body { + c = c.json(b); + } + assert_eq!(c.send().status, 404, "{method}"); + } + assert!(rows(&app.db).is_empty()); +} + +#[test] +fn other_methods_and_paths_under_settings_are_404() { + let app = app(); + for (method, path) in [ + ("DELETE", "/operator/settings"), + ("POST", "/operator/settings"), + ("GET", "/operator/settings/signup"), + ] { + assert_eq!(op(&app, method, path).send().status, 404, "{method} {path}"); + } +} + +#[test] +fn the_overview_reports_the_settings_feature() { + let app = app(); + let r = op(&app, "GET", "/operator/overview").send(); + assert_eq!(r.status, 200); + assert_eq!(r.json()["features"], json!(["support", "settings"])); +} diff --git a/crates/server/tests/http_operator_support.rs b/crates/server/tests/http_operator_support.rs index 9c4f30b..7ac3d7e 100644 --- a/crates/server/tests/http_operator_support.rs +++ b/crates/server/tests/http_operator_support.rs @@ -603,7 +603,7 @@ fn the_org_page_shows_the_active_grant_or_null() { fn the_overview_reports_the_support_feature() { let app = app(); let v = op(&app, "GET", "/operator/overview").send().json(); - assert_eq!(v["features"], json!(["support"])); + assert_eq!(v["features"], json!(["support", "settings"])); } // --- what the audit rows hold ------------------------------------------------------------------- diff --git a/crates/server/tests/live_settings.rs b/crates/server/tests/live_settings.rs new file mode 100644 index 0000000..a7253f5 --- /dev/null +++ b/crates/server/tests/live_settings.rs @@ -0,0 +1,385 @@ +// printpapi — self-hosted PrintNode alternative. Elastic License 2.0 (see LICENSE). +//! Live settings at runtime (spec A5, PR 3): the effective values are one snapshot per request, +//! read by the signup handler, `GET /health`, the operator org create and the retention pass +//! only, and only with the operator token set. +mod common; + +use common::*; +use printpapi_server::dispatch::{Destinations, Fetcher, HttpFetcher}; +use printpapi_server::proxy::TrustedProxies; +use printpapi_server::server::{AppState, Config, Terms, retention_pass}; +use printpapi_server::settings::{Change, Parsed, Source, signup_env_locks}; +use printpapi_server::store::{self, DEFAULT_ORG, Db}; +use serde_json::{Value, json}; +use std::sync::Arc; +use std::time::Duration; + +const OP: &str = "operator-token-0123456789abcdef-xyz"; +const CONSOLE: &str = "10.9.0.2"; + +/// Hosted the way `main` sets it up: the token, its peer and the private-URL deny. +fn hosted_cfg(c: &mut Config) { + c.operator_token = Some(OP.into()); + c.operator_from = TrustedProxies::parse(CONSOLE).unwrap(); + c.destinations = Destinations::PublicOnly; + c.fetches_per_org = Some(4); +} + +fn terms(version: &str) -> Terms { + Terms { + version: version.into(), + terms_url: "https://x.test/terms".into(), + dpa_url: None, + privacy_url: None, + } +} + +/// Write override rows straight into the table, as an earlier run of the operator API left them. +fn seed(db: &Db, rows: &[(&str, &str)]) { + db.with(|c| { + for (k, v) in rows { + c.execute( + "INSERT OR REPLACE INTO settings(key, value, updated_at) VALUES(?,?,1.0)", + rusqlite::params![k, v], + ) + .unwrap(); + } + }); +} + +fn rows(db: &Db) -> Vec<(String, String)> { + db.with(|c| store::settings_rows(c).unwrap()) +} + +fn change(set: Vec, unset: Vec<&'static str>) -> Change { + Change { set, unset } +} + +fn signup(app: &App, body: Value) -> (u16, Value) { + app.post("/signup", None, body) +} + +fn good() -> Value { + json!({"email": "a@b.test", "password": PW}) +} + +fn health(app: &App) -> Value { + app.get("/health", None).1 +} + +/// `POST /operator/orgs` as the console sends it (mail must be configured for it). +fn operator_create(app: &App, email: &str) -> (u16, Value) { + let r = app + .call("POST", "/operator/orgs") + .bearer(OP) + .socket(CONSOLE) + .json(&json!({"name": "Fresh", "email": email})) + .send(); + (r.status, r.json()) +} + +fn mail_cfg(c: &mut Config) { + hosted_cfg(c); + c.reset_enabled = true; + c.public_url = "https://print.example".into(); +} + +fn org_quota(db: &Db, org: i64) -> Option { + db.with(|c| store::get_org(c, org)) + .unwrap() + .unwrap() + .job_quota +} + +// --- the token gates the table ------------------------------------------------------------------ + +#[test] +fn a_database_open_with_the_token_opens_signup_and_health_says_so() { + let db = mem(); + seed(&db, &[("signup", "open")]); + let app = App::on(db, hosted_cfg); + assert_eq!(health(&app)["signup"], "open"); + let (st, v) = signup(&app, good()); + assert_eq!(st, 200, "{v}"); + assert_eq!(app.state.live().signup.source, Source::Db); +} + +#[test] +fn without_the_token_the_database_changes_nothing_and_the_table_is_left_alone() { + let db = mem(); + seed( + &db, + &[ + ("signup", "open"), + ("signup_quota", "5"), + ("job_retention_days", "7"), + ( + "terms", + r#"{"version":"v9","terms_url":"https://x.test/t"}"#, + ), + ], + ); + let before = rows(&db); + let app = App::on(db.clone(), |_| {}); + let h = health(&app); + assert_eq!(h["signup"], "closed"); + assert!(h.get("terms").is_none(), "{h}"); + assert_eq!(signup(&app, good()).0, 403); + assert_eq!(app.state.live().job_retention_days.value, None); + assert_eq!(app.state.live().signup_quota.value, None); + assert_eq!(rows(&db), before); +} + +// --- live changes leave the security defaults alone ------------------------------------------------- + +#[test] +fn opening_and_closing_signup_live_moves_no_security_default() { + let db = mem(); + let mail = Arc::new(Mailbox::default()); + let mut cfg = Config::new("t"); + hosted_cfg(&mut cfg); + let fetcher = Arc::new(HttpFetcher::with_limits_to( + Duration::from_secs(1), + Duration::from_secs(5), + 8, + cfg.destinations, + )); + let state = AppState::new(cfg, db.clone(), mail.clone(), fetcher.clone()); + let app = App { + state, + db: db.clone(), + mail, + fetch: Arc::new(FakeFetcher::default()), + }; + add_key(&app.db, "one", "k1", DEFAULT_ORG); + let pid = register(&app.db, "pc", "ak", &[decl("P", true)], DEFAULT_ORG).pid("P"); + let cap = app.state.cfg.fetch_cap(); + assert_eq!(app.state.cfg.destinations, Destinations::PublicOnly); + + let refused = |app: &App| { + let (st, v) = app.post( + "/jobs", + Some("k1"), + json!({"printer_id": pid, "type": "raw_uri", "url": "http://127.0.0.1/"}), + ); + assert_ne!(st, 200, "{v}"); + let e = fetcher.post_json("http://10.0.0.1/", &json!({"event": "x"})); + assert!(e.unwrap_err().contains("not allowed")); + }; + refused(&app); + assert_eq!(signup(&app, good()).0, 403); + app.state + .apply_settings(&change(vec![Parsed::Signup(true)], vec![]), None) + .unwrap(); + assert_eq!(health(&app)["signup"], "open"); + assert_eq!(signup(&app, good()).0, 200); + refused(&app); + app.state + .apply_settings(&change(vec![Parsed::Signup(false)], vec![]), None) + .unwrap(); + assert_eq!(health(&app)["signup"], "closed"); + assert_eq!( + signup(&app, json!({"email": "c@d.test", "password": PW})).0, + 403 + ); + refused(&app); + assert_eq!(app.state.cfg.fetch_cap(), cap); + assert_eq!(app.state.cfg.destinations, Destinations::PublicOnly); + assert!(!app.state.cfg.signup_open); +} + +#[test] +fn a_live_change_is_written_and_swapped_in_one_critical_section() { + let app = App::with(hosted_cfg); + let live = app + .state + .apply_settings( + &change(vec![Parsed::SignupQuota(3)], vec![]), + Some("because"), + ) + .unwrap(); + // the returned snapshot is the rebuilt one and is what the state serves from now on + assert_eq!(live.signup_quota.value, Some(3)); + assert_eq!(app.state.live().signup_quota.value, Some(3)); + assert_eq!( + rows(&app.db), + [("signup_quota".to_string(), "3".to_string())] + ); + app.state + .apply_settings(&change(vec![], vec!["signup_quota"]), None) + .unwrap(); + assert_eq!(app.state.live().signup_quota.value, None); + assert!(rows(&app.db).is_empty()); +} + +#[test] +fn a_snapshot_keeps_its_values_when_the_live_values_are_swapped() { + let app = App::with(hosted_cfg); + let before = app.state.live(); + assert!(!before.signup.value); + app.state + .apply_settings(&change(vec![Parsed::Signup(true)], vec![]), None) + .unwrap(); + assert!(!before.signup.value, "a request in flight keeps its view"); + assert!(app.state.live().signup.value); + assert!(!Arc::ptr_eq(&before, &app.state.live())); +} + +// --- the readers ---------------------------------------------------------------------------------- + +#[test] +fn a_live_signup_quota_applies_to_the_next_signup_and_to_operator_org_create() { + let db = mem(); + seed(&db, &[("signup", "open")]); + let app = App::on(db, |c| { + mail_cfg(c); + c.signup_quota = Some(25); + }); + let (_, v) = signup(&app, good()); + assert_eq!(org_quota(&app.db, v["org_id"].as_i64().unwrap()), Some(25)); + app.state + .apply_settings(&change(vec![Parsed::SignupQuota(7)], vec![]), None) + .unwrap(); + let (st, v) = signup(&app, json!({"email": "c@d.test", "password": PW})); + assert_eq!(st, 200, "{v}"); + assert_eq!(org_quota(&app.db, v["org_id"].as_i64().unwrap()), Some(7)); + let (st, v) = operator_create(&app, "e@f.test"); + assert_eq!(st, 200, "{v}"); + assert_eq!(org_quota(&app.db, v["org_id"].as_i64().unwrap()), Some(7)); + // unset: back to the env value + app.state + .apply_settings(&change(vec![], vec!["signup_quota"]), None) + .unwrap(); + let (_, v) = operator_create(&app, "g@h.test"); + assert_eq!(org_quota(&app.db, v["org_id"].as_i64().unwrap()), Some(25)); +} + +#[test] +fn live_terms_change_what_health_shows_and_what_signup_requires() { + let db = mem(); + seed(&db, &[("signup", "open")]); + let app = App::on(db, hosted_cfg); + assert!(health(&app).get("terms").is_none()); + assert_eq!(signup(&app, good()).0, 200); + app.state + .apply_settings(&change(vec![Parsed::Terms(terms("v2"))], vec![]), None) + .unwrap(); + assert_eq!(health(&app)["terms"]["version"], "v2"); + let body = |v: Option<&str>, mail: &str| { + let mut b = json!({"email": mail, "password": PW}); + if let Some(v) = v { + b["accept_terms"] = json!(v); + } + b + }; + assert_eq!(signup(&app, body(None, "c@d.test")).0, 400); + assert_eq!(signup(&app, body(Some("v1"), "c@d.test")).0, 409); + assert_eq!(signup(&app, body(Some("v2"), "c@d.test")).0, 200); + app.state + .apply_settings(&change(vec![], vec!["terms"]), None) + .unwrap(); + assert!(health(&app).get("terms").is_none()); +} + +#[test] +fn a_live_job_retention_is_used_by_the_next_retention_pass() { + let day = 86_400.0; + let now = store::month_start(printpapi_server::now()) + 25.0 * day; + let app = App::with(hosted_cfg); + let pid = agent_z(&app.db).pid("Z"); + let old = enqueue(&app.db, pid, b"secret", |j| j.title = Some("T".into())); + app.db.with(|c| { + c.execute( + "UPDATE jobs SET state='done', created_at=?1, finished_at=?1 WHERE id=?2", + rusqlite::params![now - 10.0 * day, old], + ) + .unwrap() + }); + // env keeps for ever: a pass does nothing + assert!(retention_pass(&app.state, now).is_none()); + app.state + .apply_settings( + &change(vec![Parsed::JobRetentionDays(7)], vec![]), + Some("keep a week"), + ) + .unwrap(); + assert_eq!(retention_pass(&app.state, now).unwrap().scrubbed, 1); +} + +// --- restart: the startup rules hold ------------------------------------------------------------------ + +#[test] +fn an_explicit_env_closed_beats_a_database_open_across_a_restart() { + let db = mem(); + seed(&db, &[("signup", "open")]); + let app = App::on(db.clone(), |c| { + hosted_cfg(c); + c.env_flags.signup_set = true; + c.env_flags.signup_explicit_closed = true; + }); + assert_eq!(health(&app)["signup"], "closed"); + assert_eq!(signup(&app, good()).0, 403); + assert_eq!(app.state.live().signup.source, Source::EnvLocked); + // the same DB with a plain (unset) env is open again: unset is not a lock + let app = App::on(db, hosted_cfg); + assert_eq!(health(&app)["signup"], "open"); +} + +#[test] +fn a_near_miss_of_closed_in_the_env_is_a_lock_across_a_restart() { + for value in ["CLOSED", "off", "closd"] { + let db = mem(); + seed(&db, &[("signup", "open")]); + let app = App::on(db, |c| { + hosted_cfg(c); + c.env_flags.signup_set = true; + c.env_flags.signup_explicit_closed = signup_env_locks(value); + }); + assert_eq!(health(&app)["signup"], "closed", "{value}"); + assert_eq!(signup(&app, good()).0, 403, "{value}"); + assert_eq!(app.state.live().signup.source, Source::EnvLocked, "{value}"); + } +} + +#[test] +fn a_terms_row_with_a_control_character_is_ignored_at_startup() { + let db = mem(); + seed( + &db, + &[( + "terms", + &json!({"version": "v1\u{1b}[2J", "terms_url": "https://x.test/t"}).to_string(), + )], + ); + let app = App::on(db, hosted_cfg); + assert_eq!(app.state.live().terms.value, None); + assert_eq!(app.state.live().terms.source, Source::Default); + assert_eq!(health(&app)["terms"], Value::Null); +} + +#[test] +fn a_database_open_is_ignored_once_private_urls_are_allowed() { + let db = mem(); + seed(&db, &[("signup", "open"), ("signup_quota", "4")]); + let app = App::on(db, |c| { + hosted_cfg(c); + c.destinations = Destinations::Any; + }); + assert_eq!(health(&app)["signup"], "closed"); + assert_eq!(signup(&app, good()).0, 403); + // only the open row is dropped, the others still apply + assert_eq!(app.state.live().signup_quota.value, Some(4)); +} + +#[test] +fn a_row_that_no_longer_validates_is_ignored_at_startup() { + let db = mem(); + seed(&db, &[("signup_quota", "-3"), ("job_retention_days", "0")]); + let app = App::on(db, |c| { + hosted_cfg(c); + c.signup_quota = Some(9); + }); + assert_eq!(app.state.live().signup_quota.value, Some(9)); + assert_eq!(app.state.live().signup_quota.source, Source::Default); + assert_eq!(app.state.live().job_retention_days.value, None); +} diff --git a/crates/server/tests/store_compat.rs b/crates/server/tests/store_compat.rs index ebbdc8e..c121d60 100644 --- a/crates/server/tests/store_compat.rs +++ b/crates/server/tests/store_compat.rs @@ -514,3 +514,43 @@ fn a_python_database_gets_the_operator_audit_table_and_the_python_tables_stay_as }); assert_eq!(grants, 1); } + +#[test] +fn a_python_database_gets_the_settings_table_and_keeps_it_across_a_restart() { + let dir = tempfile::tempdir().unwrap(); + let path = dir.path().join("python_v2.sqlite3"); + let fixture = concat!( + env!("CARGO_MANIFEST_DIR"), + "/tests/fixtures/python_v2.sqlite3" + ); + std::fs::copy(fixture, &path).unwrap(); + let table_columns = |path: &std::path::Path| -> Vec { + let c = rusqlite::Connection::open(path).unwrap(); + let mut st = c.prepare("PRAGMA table_info(settings)").unwrap(); + st.query_map([], |r| r.get::<_, String>(1)) + .unwrap() + .map(Result::unwrap) + .collect() + }; + // the name is free in the v2.5.1 schema: the fixture has no such table + assert!(table_columns(&path).is_empty()); + let db = Db::open(path.to_str().unwrap()).unwrap(); + assert_eq!(table_columns(&path), ["key", "value", "updated_at"]); + db.with(|c| { + operator_update_settings( + c, + &printpapi_server::settings::Change { + set: vec![printpapi_server::settings::Parsed::SignupQuota(7)], + unset: vec![], + }, + None, + ) + .unwrap() + }); + drop(db); + let db = Db::open(path.to_str().unwrap()).unwrap(); + assert_eq!( + db.with(settings_rows).unwrap(), + [("signup_quota".to_string(), "7".to_string())] + ); +} diff --git a/crates/server/tests/store_settings.rs b/crates/server/tests/store_settings.rs new file mode 100644 index 0000000..e78ecb1 --- /dev/null +++ b/crates/server/tests/store_settings.rs @@ -0,0 +1,171 @@ +// printpapi — self-hosted PrintNode alternative. Elastic License 2.0 (see LICENSE). +//! Live settings, the store side (spec A5): the override rows and the audited write. +use printpapi_server::settings::{Change, Parsed}; +use printpapi_server::store::*; +use rusqlite::Connection; +use serde_json::{Value, json}; + +fn db() -> Connection { + let conn = connect(":memory:").unwrap(); + init_db(&conn).unwrap(); + conn +} + +fn rows(conn: &Connection) -> Vec<(String, String)> { + settings_rows(conn).unwrap() +} + +fn set(conn: &Connection, p: Vec, unset: Vec<&'static str>, reason: Option<&str>) { + operator_update_settings(conn, &Change { set: p, unset }, reason).unwrap(); +} + +fn audit(conn: &Connection) -> Vec { + list_audit(conn, None, Some("settings.update"), None, 100).unwrap() +} + +#[test] +fn a_fresh_database_has_the_table_and_no_rows() { + let conn = db(); + let cols: Vec = conn + .prepare("PRAGMA table_info(settings)") + .unwrap() + .query_map([], |r| r.get(1)) + .unwrap() + .map(Result::unwrap) + .collect(); + assert_eq!(cols, ["key", "value", "updated_at"]); + assert!(rows(&conn).is_empty()); +} + +#[test] +fn a_write_stores_the_rows_and_one_audit_row() { + let conn = db(); + set( + &conn, + vec![Parsed::Signup(true), Parsed::SignupQuota(25)], + vec![], + Some("opening the beta"), + ); + assert_eq!( + rows(&conn), + [ + ("signup".to_string(), "open".to_string()), + ("signup_quota".to_string(), "25".to_string()) + ] + ); + let a = audit(&conn); + assert_eq!(a.len(), 1); + assert_eq!(a[0].actor, "operator"); + assert_eq!(a[0].reason.as_deref(), Some("opening the beta")); + assert_eq!(a[0].org_id, None); + let detail: Value = serde_json::from_str(a[0].detail.as_deref().unwrap()).unwrap(); + assert_eq!( + detail, + json!({"set": {"signup": "open", "signup_quota": 25}, "unset": []}) + ); + let at: f64 = conn + .query_row( + "SELECT updated_at FROM settings WHERE key='signup'", + [], + |r| r.get(0), + ) + .unwrap(); + assert!(at > 1.0e9, "updated_at is a wall-clock time: {at}"); +} + +#[test] +fn a_second_write_replaces_the_value_and_unset_removes_the_row() { + let conn = db(); + set(&conn, vec![Parsed::SignupQuota(25)], vec![], None); + set(&conn, vec![Parsed::SignupQuota(5)], vec![], None); + assert_eq!(rows(&conn), [("signup_quota".to_string(), "5".to_string())]); + set(&conn, vec![], vec!["signup_quota"], None); + assert!(rows(&conn).is_empty()); + // unset of a key that has no row is fine, and still audited + set(&conn, vec![], vec!["terms"], None); + let a = audit(&conn); + assert_eq!(a.len(), 4); + let detail: Value = serde_json::from_str(a[0].detail.as_deref().unwrap()).unwrap(); + assert_eq!(detail, json!({"set": {}, "unset": ["terms"]})); + assert_eq!(a[0].reason, None); +} + +#[test] +fn set_and_unset_in_one_change_share_one_audit_row() { + let conn = db(); + set(&conn, vec![Parsed::SignupQuota(25)], vec![], None); + set( + &conn, + vec![Parsed::JobRetentionDays(30)], + vec!["signup_quota"], + Some("tightening"), + ); + assert_eq!( + rows(&conn), + [("job_retention_days".to_string(), "30".to_string())] + ); + assert_eq!(audit(&conn).len(), 2); +} + +#[test] +fn only_the_four_live_keys_are_read() { + let conn = db(); + conn.execute( + "INSERT INTO settings(key, value, updated_at) VALUES('fetches_per_org', '99', 1.0)", + [], + ) + .unwrap(); + assert!(rows(&conn).is_empty()); + set(&conn, vec![Parsed::SignupQuota(1)], vec![], None); + assert_eq!(rows(&conn).len(), 1); +} + +#[test] +fn terms_are_stored_as_json_text() { + let conn = db(); + let t = printpapi_server::server::Terms { + version: "v2".into(), + terms_url: "https://x.test/t".into(), + dpa_url: None, + privacy_url: Some("https://x.test/p".into()), + }; + set(&conn, vec![Parsed::Terms(t.clone())], vec![], None); + let r = rows(&conn); + assert_eq!(r[0].0, "terms"); + assert_eq!(Parsed::decode("terms", &r[0].1), Ok(Parsed::Terms(t))); +} + +#[test] +fn a_failing_audit_insert_leaves_no_settings_row() { + let conn = db(); + set(&conn, vec![Parsed::SignupQuota(25)], vec![], None); + conn.execute_batch( + "CREATE TRIGGER audit_refuses BEFORE INSERT ON operator_audit \ + BEGIN SELECT RAISE(ABORT, 'audit down'); END;", + ) + .unwrap(); + let r = operator_update_settings( + &conn, + &Change { + set: vec![Parsed::Signup(true), Parsed::SignupQuota(1)], + unset: vec!["terms"], + }, + None, + ); + assert!(r.is_err()); + // nothing of the change survived, the old row is untouched + assert_eq!( + rows(&conn), + [("signup_quota".to_string(), "25".to_string())] + ); + let r = operator_update_settings( + &conn, + &Change { + set: vec![], + unset: vec!["signup_quota"], + }, + None, + ); + assert!(r.is_err()); + assert_eq!(rows(&conn).len(), 1); +} diff --git a/docs/operator.md b/docs/operator.md index 85c0b34..ed1bd3f 100644 --- a/docs/operator.md +++ b/docs/operator.md @@ -7,8 +7,8 @@ as numbers, an audit log) and nothing in it is specific to one business. **It is `404`, as if the API did not exist. This page covers the gate, hosted mode, the audit log, the read endpoints (overview, orgs and their -jobs) and the write endpoints (change, lock, delete and create an org, send a reset mail, revoke a -key). +jobs), the write endpoints (change, lock, delete and create an org, send a reset mail, revoke a +key), support access and live settings. ## The gate @@ -103,7 +103,7 @@ callback, idempotency key, key secret or password hash. "server": {"version": "3.5.0", "uptime_s": 86400}, "db": {"size_bytes": 12582912, "wal_bytes": 0}, "retention": {"last_pass_at": 1760000000.0, "abandoned": 0, "scrubbed": 14, "deleted": 0}, - "features": ["support"] + "features": ["support", "settings"] } ``` @@ -279,6 +279,112 @@ generation). Agents enrolled with it end their long-poll with `401`. Removing a single user is not offered here: customers remove their own accounts, and root still can. +## Live settings + +Four settings can be changed while the server runs, so the console does not need a restart or +shell access. They are start values read from the environment; with `PRINTAPI_OPERATOR_TOKEN` set, +an override stored in the database applies on top of them. + +| Key | Value | Env variable it overrides | +|---|---|---| +| `signup` | `"open"` or `"closed"` | `PRINTAPI_SIGNUP` | +| `signup_quota` | integer, 0 or more (0 blocks printing) | `PRINTAPI_SIGNUP_QUOTA` | +| `job_retention_days` | integer, 1 or more | `PRINTAPI_JOB_RETENTION_DAYS` | +| `terms` | `{"version", "terms_url", "dpa_url"?, "privacy_url"?}`, a non-empty version without control characters, at most 100 characters, `http(s)` links without spaces, `terms_url` required | `PRINTAPI_TERMS_VERSION` and the three link variables | + +The values follow the rules of the environment variables. They apply to the next request, with no +restart: the signup form, `GET /health`, `POST /operator/orgs` (the quota of the new org) and the +next retention pass read them. + +### `GET /operator/settings` + +```json +{ + "settings": { + "signup": {"value": "closed", "source": "env-locked"}, + "signup_quota": {"value": 50, "source": "db"}, + "job_retention_days": {"value": null, "source": "default"}, + "terms": {"value": {"version": "2026-11", "terms_url": "https://example.com/terms"}, "source": "env"} + }, + "env_only": { + "fetches_per_org": 4, "private_urls": "deny", "trusted_proxies": ["10.0.0.0/8"], + "operator_from": ["10.9.0.2"], "public_url": "https://print.example.com", "mail": true + } +} +``` + +`source` says where the effective value comes from: + +- `db`: an override of the operator API. +- `env`: the environment variable is set. +- `default`: neither (`value` is `"closed"` for `signup`, `null` for the others). +- `env-locked`: `signup` only, see [the emergency brake](#the-emergency-brake). + +A key also has `ignored`, the stored row value, when a row exists but is not applied: signup is +env-locked, a `signup` row of `open` while private URLs are allowed, or a value that no longer +validates. It is absent otherwise. `unset` clears such a row. + +`env_only` shows the settings that stay in the environment, as they were at startup. It never +holds a token, a password or a key. + +### `PUT /operator/settings` + +`{"set": {key: value}, "unset": [key], "reason"?}`. `set` changes a key, `unset` removes the +override so the environment value (or the default) applies again. There are no `null` values. +Everything is checked first and written in one transaction with one audit row; the answer is the +same as `GET`. `reason` is optional, at most 500 characters. + +| Status | Why | +|---|---| +| `400` | an unknown key or field, a key in both `set` and `unset`, an empty request, a `null`, a value that fails the rules above, a bad `reason`, or a missing reason (see [retention](#retention-needs-a-reason)) | +| `409` | `signup` is locked by the environment and the change sets `open`, or the change (a `set` or an `unset` of `signup`) would leave signup open while private URLs are allowed. A PUT that does not touch `signup` is never refused for this | +| `401` | not the operator token, or not from an address in `PRINTAPI_OPERATOR_FROM` (root does not pass) | +| `404` | `PRINTAPI_OPERATOR_TOKEN` is not set | + +Audit: `settings.update`, `detail` `{"set": {key: value}, "unset": [key]}`. Setting values are +allowed in the log (a mode, numbers, links); no e-mail address is. + +### The emergency brake + +While the operator token is set, any value of `PRINTAPI_SIGNUP` other than `open` (`closed`, but +also `CLOSED`, `off` or a typo) locks signup shut and beats every override: `source` is +`env-locked`. Only changes that close signup are accepted then: `set` to `closed` and `unset`; +setting `open` answers `409`. Leaving the variable out, or blank, means closed by default and is +not a lock. To shut signup at once even if the console is compromised, set the variable to +`closed` and restart. + +A stored `signup` row is not applied while locked, but it stays in the database and shows as +`ignored`. Clear it (`unset` or `set` to `closed`) before lifting the brake, or it applies again +once the variable is removed. + +### Retention needs a reason + +Retention is the one irreversible setting: the next pass scrubs payloads and titles for good, and +at once if retention was off. A `set` of `job_retention_days` that lowers the effective value, or +turns it on while jobs were kept for ever, needs a `reason` of at least 8 characters (`400` +otherwise). So does an `unset` that falls back to a lower or enabled environment value. Raising +it needs none. + +### What stays in the environment, and why + +- `fetches_per_org`, private URLs (`PRINTAPI_PRIVATE_URLS`) and trusted proxies decide what the + server may reach and whose address it believes. They are fixed at startup, and the live + `signup` value never moves them: hosted mode keeps the private-URL deny whether signup is open + or closed. A change that leaves signup open is refused (`409`) while private URLs are allowed. +- `PRINTAPI_OPERATOR_FROM` and the token are the gate itself: a console must not be able to widen + who can reach it. +- SMTP and `PUBLIC_URL` hold credentials and the address links in mails point to. + +### Startup and rollback + +Only with the operator token set does the server read the `settings` table; without it every +value is the environment one, as in 3.5.0, and a database carried over from a hosted box changes +nothing. At startup each applied override prints one line to stderr (never a secret). A stored +value that no longer validates is ignored with a line, and so is `signup = open` while private +URLs are allowed or `PRINTAPI_SIGNUP` locks signup shut. Values in these lines are quoted. An older +version ignores the table, so the environment values apply again after a rollback; the table is +left in place. + ## Support access Support access lets the operator read what the customer's jobs contain, for a limited time, with diff --git a/docs/server.md b/docs/server.md index c3e34e7..5fc1a53 100644 --- a/docs/server.md +++ b/docs/server.md @@ -31,7 +31,7 @@ Environment variables: | `PRINT_DB` | `printpapi.db` | SQLite database path | | `PRINT_PORT` | `3460` | Listen port | | `LOG_REQUESTS` | *(off)* | Set to log every HTTP request | -| `PRINTAPI_SIGNUP` | `closed` | `open` lets anyone create an org via `POST /signup` — for a hosted deployment, not a private box | +| `PRINTAPI_SIGNUP` | `closed` | `open` lets anyone create an org via `POST /signup` — for a hosted deployment, not a private box. `PRINTAPI_SIGNUP`, `PRINTAPI_SIGNUP_QUOTA`, `PRINTAPI_JOB_RETENTION_DAYS` and `PRINTAPI_TERMS_VERSION` (with its links) are start values: while `PRINTAPI_OPERATOR_TOKEN` is set, the operator console can override them at runtime ([live settings](operator.md#live-settings)). With the token set, any value other than `open` (`closed`, `CLOSED`, `off`, a typo) locks signup shut and no override can open it; blank means unset | | `PUBLIC_URL` | *(unset)* | This server's own address, e.g. `https://print.example`. Only then does a password-reset mail carry a link (see [api.md](api.md#password-reset)) | | `SMTP_HOST` | *(unset)* | Mail server for password resets. Unset ⇒ the reset token is printed to stderr instead | | `SMTP_PORT` | `587` (`465` with `SMTP_SSL`) | | @@ -41,9 +41,9 @@ Environment variables: | `SMTP_STARTTLS` | `1` | `0` to disable STARTTLS (plaintext relay on a trusted network) | | `PRINTAPI_PLANS` | *(unset)* | [Billing](billing.md) catalogue: a JSON array of plans, or a path to a file holding one. Unset ⇒ no plans, no billing routes | | `PRINTAPI_BILLING_SECRET` | *(unset)* | Shared secret for the `POST /billing/webhook` signature. Unset ⇒ the webhook answers `503` | -| `PRINTAPI_SIGNUP_QUOTA` | *(unset: no quota)* | Non-negative integer: the monthly [`job_quota`](api.md#quotas) of every org created through `POST /signup`, so a free signup cannot print without limit. `0` blocks printing. Orgs the root token creates (`POST /orgs`) are untouched, and a [billing](billing.md) plan still overrides it per org. Not an integer stops the server at startup. With `PRINTAPI_SIGNUP=open` and no quota, startup prints a warning | -| `PRINTAPI_JOB_RETENTION_DAYS` | *(unset or `0`: keep for ever)* | Positive integer N: see [Retention](#retention). Not a positive integer stops the server at startup. With `PRINTAPI_SIGNUP=open` and no retention, startup prints a warning | -| `PRINTAPI_TERMS_VERSION` | *(unset: no consent asked)* | A non-empty version string, e.g. `2026-11-13`. Set: `POST /signup` must carry `accept_terms` equal to it, `GET /health` reports it with the links below, and the dashboard's sign-up form shows a checkbox for the terms and the data processing agreement. Each acceptance (org, user, version, time) is stored in the `terms_acceptances` table. See [Self-signup](api.md#self-signup). With `PRINTAPI_SIGNUP=open` and no version, startup prints a warning | +| `PRINTAPI_SIGNUP_QUOTA` | *(unset: no quota)* | Non-negative integer: the monthly [`job_quota`](api.md#quotas) of every org created through `POST /signup`, so a free signup cannot print without limit. `0` blocks printing. Orgs the root token creates (`POST /orgs`) are untouched, and a [billing](billing.md) plan still overrides it per org. Not an integer stops the server at startup. With signup open and no quota, startup prints a warning | +| `PRINTAPI_JOB_RETENTION_DAYS` | *(unset or `0`: keep for ever)* | Positive integer N: see [Retention](#retention). Not a positive integer stops the server at startup. With signup open and no retention, startup prints a warning | +| `PRINTAPI_TERMS_VERSION` | *(unset: no consent asked)* | A non-empty version string of at most 100 characters without control characters, e.g. `2026-11-13`. Set: `POST /signup` must carry `accept_terms` equal to it, `GET /health` reports it with the links below, and the dashboard's sign-up form shows a checkbox for the terms and the data processing agreement. Each acceptance (org, user, version, time) is stored in the `terms_acceptances` table. See [Self-signup](api.md#self-signup). With signup open and no version, startup prints a warning | | `PRINTAPI_TERMS_URL` | *(required with the version)* | `http(s)` link to the terms. Anything else, or a version without it, stops the server at startup | | `PRINTAPI_DPA_URL`, `PRINTAPI_PRIVACY_URL` | *(unset)* | Optional `http(s)` links to the data processing agreement and the privacy policy, shown next to the checkbox. Anything else stops the server at startup | | `PRINTAPI_FETCHES_PER_ORG` | `4` with open signup or an [operator token](operator.md#hosted-mode), else `0` (no cap) | Payload downloads (`*_uri`, `*_uri_post`, `label_uri`) one org may have in flight at once; the next one answers `503`. A non-negative integer, `0` = no cap; anything else stops the server at startup. Keeps one hosted tenant from using every download slot; a single-org box has no reason for it, since four hung upstream hosts would then block every other fetching job |