diff --git a/CHANGELOG.md b/CHANGELOG.md index af5caf27..2f6484fb 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -8,6 +8,7 @@ ### Added +- **`[privacy]` in `uteke.toml` (#1464, part 2b-2a)** - `mode` (`redact`, the default, `strict` or `off`) and `allowlist` (regexes) now configure the secret filter in the CLI, `uteke-serve` and `uteke-mcp`; `UTEKE_PRIVACY_MODE` overrides the mode. The section is read from the global file only (a project file could otherwise switch the filter off), and a bad value falls back to the default instead of disabling it. - **`author` on every write path (#1427, phase 2)** - a memory records who wrote it (the agent, user or tool name), with or without a room. `POST /remember`, MCP `uteke_remember` and `uteke remember --author` accept it (free text, trimmed, 1 to 128 characters; an invalid value is a 400, a tool error or a CLI error before anything is stored). A room write stamps the named writer on the memory as well as on the room link; the `anonymous` / `user` fallback of the room link is not an author. `author` is returned on the memory object by recall, search, list and get (omitted when unset), echoed by `POST /remember`, and kept by export and import. Stores migrate to schema v22 on first open (see Internal below): back up a production store first. Filtering by author is not part of this phase. Contract: uteke-cloud#219 (cloud already has the field). ### Fixed diff --git a/crates/uteke-cli/src/main.rs b/crates/uteke-cli/src/main.rs index 891b1b4d..7f8d2b49 100644 --- a/crates/uteke-cli/src/main.rs +++ b/crates/uteke-cli/src/main.rs @@ -247,6 +247,8 @@ fn main() { }); // #928: apply lifecycle configuration from config u.set_lifecycle_config(config.lifecycle.to_core()); + // #1464: secret filter on the write path ([privacy] in uteke.toml) + u.set_privacy_config(uteke_core::privacy::load_config()); u } Err(e) => { diff --git a/crates/uteke-core/src/config_layers.rs b/crates/uteke-core/src/config_layers.rs index 63e0afe3..0867736f 100644 --- a/crates/uteke-core/src/config_layers.rs +++ b/crates/uteke-core/src/config_layers.rs @@ -42,6 +42,9 @@ pub const UNTRUSTED_PROJECT_KEYS: &[&str] = &[ "extraction.api_key", "extraction.base_url", "extraction.endpoint_path", + // A cloned repository must not switch the secret filter off or allowlist + // everything. + "privacy", "server", ]; diff --git a/crates/uteke-core/src/privacy.rs b/crates/uteke-core/src/privacy.rs index 0420f473..ce8c77d8 100644 --- a/crates/uteke-core/src/privacy.rs +++ b/crates/uteke-core/src/privacy.rs @@ -97,6 +97,89 @@ pub struct PrivacyConfig { pub allowlist: Vec, } +/// `[privacy]` in `uteke.toml`. +#[derive(Debug, Clone, Default, serde::Deserialize)] +#[serde(default)] +pub struct PrivacySettings { + /// `redact` (default), `strict` or `off`. + pub mode: Option, + /// Regexes; a value matching any of them is kept as it is. + pub allowlist: Vec, +} + +#[derive(Debug, Default, serde::Deserialize)] +#[serde(default)] +struct PrivacyFile { + privacy: PrivacySettings, +} + +impl PrivacyConfig { + /// Build the config from `[privacy]` settings. An unknown mode or an + /// invalid regex is an error; callers that must keep running use + /// [`load_config`], which falls back to the safe default. + pub fn from_settings(settings: &PrivacySettings) -> Result { + Ok(Self { + mode: match settings.mode.as_deref() { + Some(m) => PrivacyMode::parse(m)?, + None => PrivacyMode::default(), + }, + allowlist: compile_allowlist(&settings.allowlist)?, + }) + } +} + +/// The privacy config of this installation: `[privacy]` from the global +/// `uteke.toml` (the project-local file is never trusted for it, so a cloned +/// repository cannot switch the filter off), then `UTEKE_PRIVACY_MODE`. +/// A bad value is logged and replaced by the default (`redact`): the filter +/// never turns itself off because of a typo. +pub fn load_config() -> PrivacyConfig { + let global = crate::uteke_home().ok().map(|h| h.join("uteke.toml")); + let project = std::env::current_dir() + .ok() + .map(|cwd| cwd.join(".uteke").join("uteke.toml")); + let mut settings = + crate::config_layers::resolve::(&crate::config_layers::Layers { + global: global.as_deref(), + project: project.as_deref(), + trust_project: matches!( + std::env::var("UTEKE_TRUST_PROJECT_CONFIG").as_deref(), + Ok("1") | Ok("true") + ), + }) + .value + .privacy; + if let Ok(mode) = std::env::var("UTEKE_PRIVACY_MODE") { + settings.mode = Some(mode); + } + config_or_default(&settings) +} + +fn config_or_default(settings: &PrivacySettings) -> PrivacyConfig { + // Each setting falls back on its own: a bad allowlist entry must not + // lose a `strict` mode, and a bad mode must not drop the allowlist. + let mode = match settings.mode.as_deref().map(PrivacyMode::parse) { + Some(Ok(m)) => m, + Some(Err(e)) => { + tracing::warn!("Invalid [privacy] mode, using redact: {e}"); + PrivacyMode::default() + } + None => PrivacyMode::default(), + }; + let allowlist = settings + .allowlist + .iter() + .filter_map(|p| match compile_allowlist(std::slice::from_ref(p)) { + Ok(mut v) => v.pop(), + Err(e) => { + tracing::warn!("Ignoring [privacy] allowlist entry: {e}"); + None + } + }) + .collect(); + PrivacyConfig { mode, allowlist } +} + /// Content and metadata ready to store, plus what the filter did. #[derive(Debug, Clone, PartialEq)] pub struct Prepared { @@ -1545,4 +1628,59 @@ mod tests { let comment = format!("# {b}\n# {run}\n# {e}"); assert_eq!(redact(&comment, &[]).text, "# [REDACTED:private-key]"); } + + #[test] + fn settings_become_a_config_and_bad_values_fall_back_one_by_one() { + let ok = PrivacyConfig::from_settings(&PrivacySettings { + mode: Some("Strict".into()), + allowlist: vec!["^dev-".into()], + }) + .unwrap(); + assert_eq!(ok.mode, PrivacyMode::Strict); + assert_eq!(ok.allowlist.len(), 1); + assert!( + PrivacyConfig::from_settings(&PrivacySettings { + mode: Some("maybe".into()), + allowlist: vec![], + }) + .is_err() + ); + + // A bad mode keeps the allowlist and never turns the filter off; a bad + // regex keeps the mode. + let c = config_or_default(&PrivacySettings { + mode: Some("none".into()), + allowlist: vec!["^dev-".into()], + }); + assert_eq!((c.mode, c.allowlist.len()), (PrivacyMode::Redact, 1)); + let c = config_or_default(&PrivacySettings { + mode: Some("strict".into()), + allowlist: vec!["(".into(), "^dev-".into()], + }); + assert_eq!((c.mode, c.allowlist.len()), (PrivacyMode::Strict, 1)); + } + + #[test] + fn privacy_is_read_from_the_global_file_only() { + use crate::config_layers::{Layers, resolve}; + let dir = std::env::temp_dir().join(format!("uteke-privacy-cfg-{}", std::process::id())); + std::fs::create_dir_all(&dir).unwrap(); + let global = dir.join("global.toml"); + let project = dir.join("project.toml"); + std::fs::write(&global, "[privacy]\nmode = \"strict\"\n").unwrap(); + std::fs::write( + &project, + "[privacy]\nmode = \"off\"\nallowlist = [\".*\"]\n", + ) + .unwrap(); + let r = resolve::(&Layers { + global: Some(&global), + project: Some(&project), + trust_project: false, + }); + assert_eq!(r.value.privacy.mode.as_deref(), Some("strict")); + assert!(r.value.privacy.allowlist.is_empty()); + assert!(r.ignored_untrusted.contains(&"privacy".to_string())); + std::fs::remove_dir_all(&dir).ok(); + } } diff --git a/crates/uteke-mcp/src/main.rs b/crates/uteke-mcp/src/main.rs index dd7ab833..de0f4a47 100644 --- a/crates/uteke-mcp/src/main.rs +++ b/crates/uteke-mcp/src/main.rs @@ -37,7 +37,11 @@ fn main() { }); let uteke = match Uteke::open(&store_path) { - Ok(u) => u, + Ok(mut u) => { + // #1464: secret filter on the write path ([privacy] in uteke.toml) + u.set_privacy_config(uteke_core::privacy::load_config()); + u + } Err(e) => { eprintln!("Failed to open uteke store: {e}"); std::process::exit(1); diff --git a/crates/uteke-server/src/main.rs b/crates/uteke-server/src/main.rs index 3ac9e429..aab4b2aa 100644 --- a/crates/uteke-server/src/main.rs +++ b/crates/uteke-server/src/main.rs @@ -285,6 +285,10 @@ fn main() { Ok(mut u) => { // [recall] jaccard_weight (#719, #1355 step B); the core clamps to 0..=1. u.set_jaccard_weight(recall_tuning.jaccard_weight); + // [privacy] (#1464): secret filter on the write path. + let privacy = uteke_core::privacy::load_config(); + info!("Privacy filter: {:?}", privacy.mode); + u.set_privacy_config(privacy); // [embed_fallback] + UTEKE_EMBED_FALLBACK_* (#1355): only when complete. if let Some(fallback) = resolve_embed_fallback(&config, |k| std::env::var(k).ok()) { info!("Embedding fallback: enabled ({})", fallback.base_url); diff --git a/docs/configuration.md b/docs/configuration.md index ede91cbd..ce7e1757 100644 --- a/docs/configuration.md +++ b/docs/configuration.md @@ -309,6 +309,7 @@ Resolution order (highest priority first): | Env Var | Config Equivalent | Default | Description | |---------|-------------------|---------|-------------| | `UTEKE_HOME` | — | `~/.codecora/uteke` | Data directory | +| `UTEKE_PRIVACY_MODE` | `[privacy] mode` | `redact` | Secret filter mode: `redact`, `strict`, `off` | | `UTEKE_NAMESPACE` | `[store] namespace` | `default` | Default namespace (applied in CLI) | | `UTEKE_AUTH_TOKEN` | — | — | Server auth token (applied in server) | | `UTEKE_LOG_LEVEL` | `[logging] level` | `warn` | Log level (trace/debug/info/warn/error) | @@ -400,10 +401,27 @@ A project directory can be an untrusted clone, so `.uteke/uteke.toml` is **not** - `[embedding]`: `backend`, `api_key`, `base_url`, `endpoint_path` - `[embed_fallback]` and `[extraction]`: `api_key`, `base_url`, `endpoint_path` +- the whole `[privacy]` section (a cloned repository must not switch the secret filter off) - the whole `[server]` section (`enabled`, `host`, `port`, `auth_token`, `read_only_token`, `cors_origins`, `allowed_hosts`) Everything else (store path, namespace, logging, recall tuning, lifecycle, ...) can be set per project. To trust a project file completely, set `UTEKE_TRUST_PROJECT_CONFIG=1`. The CLI and `uteke-serve` apply the same rules and the same per-key merge. +## Privacy Filter (#1464) + +Secrets in the content of a memory are masked before it is embedded or stored (`ghp_[REDACTED:github-token]`, `API_KEY=[REDACTED:env-secret]`). The memory is flagged with `metadata.redacted` and `metadata.redactions` (kinds and counts, never values) and gets a `redacted` timeline event. The original text is not kept anywhere. + +```toml +[privacy] +mode = "redact" # redact (default) | strict | off +allowlist = ["^dev-"] # regexes; a value matching any of them is kept as it is +``` + +- `redact`: mask and store. +- `strict`: refuse the write (`secret_detected`; import skips the row). +- `off`: store as given. + +`UTEKE_PRIVACY_MODE` overrides `mode`. An unknown mode or an invalid regex is logged and replaced by the default (`redact`) for that setting; the filter never turns itself off because of a typo. The filter covers the content of `remember`, room remember, `update` and `import`; metadata, tags and `source` are not filtered yet. + ## CLI Flag Override CLI flags always take precedence over config file values: