diff --git a/CHANGELOG.md b/CHANGELOG.md index 5b9df7c5..de81a0f9 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -8,6 +8,7 @@ ### Added +- **The privacy filter now covers tags, metadata strings and `source` (#1464, part 2c)** - a secret in a tag, in any string inside the metadata or in the `source` is masked (or refused in strict mode) on `remember`, room remember, `update`, `import`, `set_source` and room consolidation, and counted in `redactions`. `update_memory_detailed` returns what an update masked; `PUT /memory` returns `redacted` and `uteke_update` reports it. Importing a structural export (`uteke export --full`) restores a backup as it was and is not filtered. - **Privacy filter reporting in the CLI and MCP (#1464, part 2b-2c)** - `uteke remember` prints which kinds of secret were masked and its `--json` output gains `redacted`; `uteke import` reports masked and rejected rows; `uteke_remember` over MCP says so in its result, and in strict mode `uteke_remember` and `uteke_update` fail with a tool error that starts with `secret_detected:` and names the kinds, never the value. - **`[privacy]` in `uteke.toml` (#1464, part 2b-2a)** - `mode` (`redact`, the default, `strict` or `off`) and `allowlist` (regexes) now configure the secret filter in the CLI, `uteke-serve` and `uteke-mcp`; `UTEKE_PRIVACY_MODE` overrides the mode. The section is read from the global file only (a project file could otherwise switch the filter off), and a bad value falls back to the default instead of disabling it. - **`redacted` on the HTTP write responses and `secret_detected` in strict mode (#1464, part 2b-2b)** - `POST /remember` and `POST /room/remember` return `redacted: true` when the privacy filter masked a secret; a strict-mode write is refused with `400 {"error": "secret_detected", "message", "kinds"}` (also on `PUT /memory`); `POST /import` reports `redacted` and `rejected_secret`. Additive; agreed with cloud in uteke-cloud#218. diff --git a/crates/uteke-cli/src/commands/remember.rs b/crates/uteke-cli/src/commands/remember.rs index 3cf5f8fb..e9b36973 100644 --- a/crates/uteke-cli/src/commands/remember.rs +++ b/crates/uteke-cli/src/commands/remember.rs @@ -96,6 +96,15 @@ pub(crate) fn run( }) .transpose()?; + // #1464: the source is stored after the memory, so strict mode refuses a + // source with a secret before anything is written. + if let Some(src) = source { + uteke + .privacy_config() + .mask_text(src) + .map_err(|e| format!("Failed to store memory: {e}"))?; + } + // Read from stdin when content argument is "-" (#620). // Standard Unix convention: pipe content via echo "text" | uteke remember - --tags "test" let content = if content == "-" { diff --git a/crates/uteke-core/src/consolidate.rs b/crates/uteke-core/src/consolidate.rs index a831c302..bfa67753 100644 --- a/crates/uteke-core/src/consolidate.rs +++ b/crates/uteke-core/src/consolidate.rs @@ -73,9 +73,14 @@ impl crate::Uteke { crate::validate_input(content, tags)?; let ns = namespace.unwrap_or(DEFAULT_NAMESPACE); // Privacy filter (#1464), before the content is embedded or stored. - let prepared = self.privacy_config.prepare(content, metadata)?; + let prepared = self + .privacy_config + .prepare_with_tags(content, tags, metadata)?; let content = prepared.content.as_str(); let metadata = prepared.metadata.clone(); + let tag_owned = prepared.tags.clone(); + let tags: Vec<&str> = tag_owned.iter().map(String::as_str).collect(); + let tags = tags.as_slice(); // Embed first to check for contradictions before persisting self.ensure_embedder()?; diff --git a/crates/uteke-core/src/consolidation_api.rs b/crates/uteke-core/src/consolidation_api.rs index 48528a94..5c5c6c9f 100644 --- a/crates/uteke-core/src/consolidation_api.rs +++ b/crates/uteke-core/src/consolidation_api.rs @@ -26,6 +26,22 @@ impl ConsolidationStore for Uteke { } fn insert_memory(&self, memory: &mut Memory) -> Result<(), Error> { + // Privacy filter (#1464): the consolidated text comes from an LLM and + // is stored like any other write. (The embedding was computed from the + // text before this point; it does not carry the secret as text.) + let tag_refs: Vec<&str> = memory.tags.iter().map(String::as_str).collect(); + let prepared = self.privacy_config().prepare_with_tags( + &memory.content, + &tag_refs, + Some(memory.metadata.clone()), + )?; + memory.content = prepared.content; + memory.tags = prepared.tags; + if let Some(m) = prepared.metadata { + memory.metadata = m; + } + let redactions = prepared.redactions; + let room_id = memory .metadata .get("room_id") @@ -34,6 +50,7 @@ impl ConsolidationStore for Uteke { // Persist (transactional insert, mirrors crud flow). self.store().insert(memory)?; + self.record_redaction(&memory.id, &redactions); // Insert into the vector index so recall finds it; roll back the row // if indexing fails so we never leave an un-searchable memory behind. @@ -205,4 +222,31 @@ mod tests { "deprecated memory must not appear in room recall" ); } + + #[test] + fn a_consolidated_memory_goes_through_the_privacy_filter() { + let mut uteke = Uteke::open(":memory:").unwrap(); + uteke.set_privacy_config(crate::privacy::PrivacyConfig::default()); + let token = format!("{}{}", "ghp_", "aB3dE5gH7jK9mN1pQ3rS5tU7vW9xY1zA3bC5"); + let mut m = mem("m1", &format!("the LLM wrote {token}")); + m.metadata = serde_json::json!({"summary": format!("see {token}")}); + ConsolidationStore::insert_memory(&uteke, &mut m).unwrap(); + let stored = uteke.store.get_by_id("m1").unwrap().unwrap(); + assert!( + !stored.content.contains("aB3dE5gH7jK9"), + "{}", + stored.content + ); + assert!(!stored.metadata.to_string().contains("aB3dE5gH7jK9")); + assert_eq!(stored.metadata["redacted"], true); + + let mut strict = Uteke::open(":memory:").unwrap(); + strict.set_privacy_config(crate::privacy::PrivacyConfig { + mode: crate::privacy::PrivacyMode::Strict, + allowlist: Vec::new(), + }); + let mut m = mem("m2", &format!("the LLM wrote {token}")); + assert!(ConsolidationStore::insert_memory(&strict, &mut m).is_err()); + assert!(strict.store.get_by_id("m2").unwrap().is_none()); + } } diff --git a/crates/uteke-core/src/import_export.rs b/crates/uteke-core/src/import_export.rs index 94c984ac..4605b62b 100644 --- a/crates/uteke-core/src/import_export.rs +++ b/crates/uteke-core/src/import_export.rs @@ -91,10 +91,17 @@ impl crate::Uteke { // Privacy filter (#1464), before the content is embedded or stored. // Strict mode skips the row (counted as skipped) instead of // aborting the whole import. - let prepared = match self - .privacy_config - .prepare(&entry.content, Some(std::mem::take(&mut entry.metadata))) - { + // The tags and the source are filtered with the content (the + // source rides along as the last "tag" and is taken off again). + let mut fields: Vec<&str> = entry.tags.iter().map(String::as_str).collect(); + if let Some(src) = entry.source.as_deref() { + fields.push(src); + } + let prepared = match self.privacy_config.prepare_with_tags( + &entry.content, + &fields, + Some(std::mem::take(&mut entry.metadata)), + ) { Ok(p) => p, Err(e) => { tracing::warn!("Skipping import entry: {e}"); @@ -106,6 +113,11 @@ impl crate::Uteke { } }; entry.content = prepared.content; + let mut masked_fields = prepared.tags; + if entry.source.is_some() { + entry.source = masked_fields.pop(); + } + entry.tags = masked_fields; if let Some(m) = prepared.metadata { entry.metadata = m; } diff --git a/crates/uteke-core/src/lib.rs b/crates/uteke-core/src/lib.rs index aa2cecd6..8f00341f 100644 --- a/crates/uteke-core/src/lib.rs +++ b/crates/uteke-core/src/lib.rs @@ -37,6 +37,8 @@ mod operations; pub mod pack_mode; pub mod privacy; #[cfg(test)] +mod privacy_field_tests; +#[cfg(test)] mod privacy_write_tests; pub use forget_request::{ @@ -47,7 +49,7 @@ pub use list_request::{ LIST_DEFAULT_LIMIT, LIST_HTTP_DEFAULT_LIMIT, LIST_HTTP_MAX_LIMIT, ListInput, ListPageMeta, ListPolicy, ListRequest, ListRequestError, }; -pub use operations::RememberOutcome; +pub use operations::{RememberOutcome, UpdateOutcome}; pub use recall_request::{ RECALL_DEFAULT_BUDGET_CHARS, RECALL_DEFAULT_LIMIT, RECALL_DEFAULT_MIN_SCORE, RECALL_HTTP_MAX_LIMIT, RECALL_STRICT_MIN_SCORE, RecallInput, RecallPolicy, RecallRequest, @@ -1437,7 +1439,18 @@ impl Uteke { source: Option<&str>, source_type: &str, ) -> Result { - self.store.set_source(id, source, source_type) + // #1464: the source text is filtered like the content. It is set after + // the memory is stored, so in strict mode the rejection leaves the + // memory without a source. + let masked = source + .map(|s| self.privacy_config.mask_text(s)) + .transpose()?; + let source = masked.as_ref().map(|(text, _)| text.as_str()); + let found = self.store.set_source(id, source, source_type)?; + if let Some((_, redactions)) = &masked { + self.record_redaction(id, redactions); + } + Ok(found) } /// Set the content hash recorded at write time (#1172 Fase 1). diff --git a/crates/uteke-core/src/operations.rs b/crates/uteke-core/src/operations.rs index 4559110e..212e9761 100644 --- a/crates/uteke-core/src/operations.rs +++ b/crates/uteke-core/src/operations.rs @@ -92,6 +92,36 @@ pub struct RememberOutcome { pub redactions: Vec, } +/// What an update did (#1464): whether the memory was found and updated, and +/// what the privacy filter masked in the new content, tags or metadata. +#[derive(Debug, Clone, Default)] +pub struct UpdateOutcome { + pub updated: bool, + pub redactions: Vec, +} + +impl UpdateOutcome { + /// Whether the privacy filter masked anything in this update. + pub fn redacted(&self) -> bool { + !self.redactions.is_empty() + } + + /// What was masked, for a human (see [`RememberOutcome::redaction_summary`]). + pub fn redaction_summary(&self) -> Option { + redaction_summary(&self.redactions) + } +} + +fn redaction_summary(redactions: &[crate::privacy::RedactionCount]) -> Option { + (!redactions.is_empty()).then(|| { + redactions + .iter() + .map(|r| format!("{} x{}", r.kind, r.count)) + .collect::>() + .join(", ") + }) +} + impl RememberOutcome { /// Whether the privacy filter masked anything in the stored content. pub fn redacted(&self) -> bool { @@ -101,13 +131,7 @@ impl RememberOutcome { /// What was masked, for a human: `github-token x1, jwt x2`. `None` when /// nothing was. pub fn redaction_summary(&self) -> Option { - self.redacted().then(|| { - self.redactions - .iter() - .map(|r| format!("{} x{}", r.kind, r.count)) - .collect::>() - .join(", ") - }) + redaction_summary(&self.redactions) } } @@ -289,9 +313,14 @@ impl crate::Uteke { })?; // Privacy filter (#1464): before the content is embedded, indexed or // stored, so a secret never reaches the vector, FTS5 or the row. - let prepared = self.privacy_config.prepare(content, metadata)?; + let prepared = self + .privacy_config + .prepare_with_tags(content, tags, metadata)?; let content = prepared.content.as_str(); let metadata = prepared.metadata.clone(); + let tag_owned = prepared.tags.clone(); + let tags: Vec<&str> = tag_owned.iter().map(String::as_str).collect(); + let tags = tags.as_slice(); // Detect JSON content and use flattened text for embedding let content_type = crate::memory::crud::detect_content_type(content); let embed_text = if content_type == "json" { @@ -1865,7 +1894,9 @@ impl crate::Uteke { /// Only provided fields are changed. If `content` is changed, the /// embedding is regenerated and the vector index is updated. /// Returns `Ok(true)` if the memory was found and updated, - /// `Ok(false)` if the memory ID doesn't exist. + /// `Ok(false)` if the memory ID doesn't exist. Use + /// [`update_memory_detailed`](Self::update_memory_detailed) to also learn + /// what the privacy filter masked. /// /// Acceptance criteria: /// - Partial update semantics (only provided fields changed) @@ -1882,6 +1913,24 @@ impl crate::Uteke { pinned: Option, memory_type: Option<&str>, ) -> Result { + self.update_memory_detailed(id, content, tags, metadata, importance, pinned, memory_type) + .map(|o| o.updated) + } + + /// [`update_memory`](Self::update_memory) that returns an [`UpdateOutcome`]: + /// whether the memory was updated, and what the privacy filter masked in + /// the new content, tags or metadata (#1464). + #[allow(clippy::too_many_arguments)] + pub fn update_memory_detailed( + &self, + id: &str, + content: Option<&str>, + tags: Option<&[String]>, + metadata: Option<&serde_json::Value>, + importance: Option, + pinned: Option, + memory_type: Option<&str>, + ) -> Result { // Validate memory exists let existing = self .store @@ -1924,10 +1973,37 @@ impl crate::Uteke { Some(c) => Some(self.privacy_config.prepare(c, None)?), None => None, }; - let redactions = prepared - .as_ref() - .map(|p| p.redactions.clone()) - .unwrap_or_default(); + // The tags and the metadata strings are filtered too; what they hold + // counts with the content's redactions. + let (masked_metadata, meta_redactions) = match metadata { + Some(m) => { + let (v, r) = self.privacy_config.mask_metadata(m)?; + (Some(v), r) + } + None => (None, Vec::new()), + }; + let metadata: Option<&serde_json::Value> = masked_metadata.as_ref(); + let (masked_tags, tag_redactions) = match tags { + Some(t) => { + let (v, r) = self.privacy_config.mask_tags(t)?; + (Some(v), r) + } + None => (None, Vec::new()), + }; + let tags: Option<&[String]> = masked_tags.as_deref(); + let mut merged: std::collections::BTreeMap<&'static str, usize> = Default::default(); + for r in prepared + .iter() + .flat_map(|p| p.redactions.iter()) + .chain(&meta_redactions) + .chain(&tag_redactions) + { + *merged.entry(r.kind).or_default() += r.count; + } + let redactions: Vec = merged + .into_iter() + .map(|(kind, count)| crate::privacy::RedactionCount { kind, count }) + .collect(); let masked_content: Option = prepared.map(|p| p.content); let content: Option<&str> = masked_content.as_deref(); let flagged_metadata: Option = if redactions.is_empty() { @@ -1993,7 +2069,7 @@ impl crate::Uteke { )?; if !updated { - return Ok(false); + return Ok(UpdateOutcome::default()); } // Update vector index (insert handles dedup by removing old entry) @@ -2026,7 +2102,7 @@ impl crate::Uteke { now, )?; if !updated { - return Ok(false); + return Ok(UpdateOutcome::default()); } } @@ -2035,7 +2111,10 @@ impl crate::Uteke { // Invalidate recall cache for the memory's namespace self.recall_cache.invalidate_namespace(&existing.namespace); - Ok(true) + Ok(UpdateOutcome { + updated: true, + redactions, + }) } /// Recall memories that existed at a specific point in time. diff --git a/crates/uteke-core/src/privacy.rs b/crates/uteke-core/src/privacy.rs index ce8c77d8..270a9178 100644 --- a/crates/uteke-core/src/privacy.rs +++ b/crates/uteke-core/src/privacy.rs @@ -184,6 +184,8 @@ fn config_or_default(settings: &PrivacySettings) -> PrivacyConfig { #[derive(Debug, Clone, PartialEq)] pub struct Prepared { pub content: String, + /// The tags as given, with any secret masked. + pub tags: Vec, pub metadata: Option, /// Empty when nothing was masked. pub redactions: Vec, @@ -199,30 +201,127 @@ impl PrivacyConfig { &self, content: &str, metadata: Option, + ) -> Result { + self.prepare_with_tags(content, &[], metadata) + } + + /// [`prepare`](Self::prepare) for a whole write: the content, the tags and + /// every string value in the metadata are filtered, and the counts are + /// merged into one `redactions` list. + pub fn prepare_with_tags( + &self, + content: &str, + tags: &[&str], + metadata: Option, ) -> Result { let untouched = |metadata| Prepared { content: content.to_string(), + tags: tags.iter().map(|t| t.to_string()).collect(), metadata, redactions: Vec::new(), }; if self.mode == PrivacyMode::Off { return Ok(untouched(metadata)); } + let mut counts: BTreeMap<&'static str, usize> = BTreeMap::new(); let r = redact(content, &self.allowlist); - if r.is_clean() { + merge_counts(&mut counts, &r.redactions); + let tags_out: Vec = tags + .iter() + .map(|t| { + let r = redact(t, &self.allowlist); + merge_counts(&mut counts, &r.redactions); + r.text + }) + .collect(); + let metadata_out = metadata.as_ref().map(|m| self.mask_json(m, &mut counts)); + if counts.is_empty() { return Ok(untouched(metadata)); } + let redactions: Vec = counts + .into_iter() + .map(|(kind, count)| RedactionCount { kind, count }) + .collect(); if self.mode == PrivacyMode::Strict { return Err(Error::SecretDetected { - kinds: r.redactions.iter().map(|x| x.kind.to_string()).collect(), + kinds: redactions.iter().map(|x| x.kind.to_string()).collect(), }); } Ok(Prepared { content: r.text, - metadata: with_redaction_flags(metadata, &r.redactions), - redactions: r.redactions, + tags: tags_out, + metadata: with_redaction_flags(metadata_out, &redactions), + redactions, }) } + + /// Filter one free-text field (the `source` of a memory). Same modes as + /// [`prepare`](Self::prepare); returns the text to store and what was masked. + pub fn mask_text(&self, text: &str) -> Result<(String, Vec), Error> { + let p = self.prepare_with_tags("", &[text], None)?; + Ok((p.tags.into_iter().next().unwrap_or_default(), p.redactions)) + } + + /// Filter the metadata of an update that has no new content. Same modes as + /// [`prepare`](Self::prepare); the flags are not added here. + pub fn mask_metadata( + &self, + metadata: &serde_json::Value, + ) -> Result<(serde_json::Value, Vec), Error> { + if self.mode == PrivacyMode::Off { + return Ok((metadata.clone(), Vec::new())); + } + let mut counts = BTreeMap::new(); + let masked = self.mask_json(metadata, &mut counts); + let redactions: Vec = counts + .into_iter() + .map(|(kind, count)| RedactionCount { kind, count }) + .collect(); + if self.mode == PrivacyMode::Strict && !redactions.is_empty() { + return Err(Error::SecretDetected { + kinds: redactions.iter().map(|x| x.kind.to_string()).collect(), + }); + } + Ok((masked, redactions)) + } + + /// Filter the tags of an update. + pub fn mask_tags(&self, tags: &[String]) -> Result<(Vec, Vec), Error> { + let refs: Vec<&str> = tags.iter().map(String::as_str).collect(); + let p = self.prepare_with_tags("", &refs, None)?; + Ok((p.tags, p.redactions)) + } + + /// `value` with every string inside it filtered (keys are left alone). + fn mask_json( + &self, + value: &serde_json::Value, + counts: &mut BTreeMap<&'static str, usize>, + ) -> serde_json::Value { + use serde_json::Value; + match value { + Value::String(s) => { + let r = redact(s, &self.allowlist); + merge_counts(counts, &r.redactions); + Value::String(r.text) + } + Value::Array(items) => { + Value::Array(items.iter().map(|v| self.mask_json(v, counts)).collect()) + } + Value::Object(map) => Value::Object( + map.iter() + .map(|(k, v)| (k.clone(), self.mask_json(v, counts))) + .collect(), + ), + other => other.clone(), + } + } +} + +fn merge_counts(into: &mut BTreeMap<&'static str, usize>, from: &[RedactionCount]) { + for c in from { + *into.entry(c.kind).or_default() += c.count; + } } /// Merge `redacted: true` and `redactions` into `metadata`. `None` becomes a new diff --git a/crates/uteke-core/src/privacy_field_tests.rs b/crates/uteke-core/src/privacy_field_tests.rs new file mode 100644 index 00000000..c452c90f --- /dev/null +++ b/crates/uteke-core/src/privacy_field_tests.rs @@ -0,0 +1,170 @@ +//! The privacy filter on the fields next to the content (#1464): tags, the +//! strings inside metadata and the `source`, and what an update reports. + +use crate::privacy::{PrivacyConfig, PrivacyMode}; +use crate::test_support::open_keyword; +use crate::{Error, Uteke}; + +fn gh() -> String { + format!("{}{}", "ghp_", "aB3dE5gH7jK9mN1pQ3rS5tU7vW9xY1zA3bC5") +} + +fn store(mode: PrivacyMode) -> Uteke { + let mut u = open_keyword(); + u.set_privacy_config(PrivacyConfig { + mode, + allowlist: Vec::new(), + }); + u +} + +#[test] +fn remember_masks_secrets_in_tags_and_metadata_strings() { + let u = store(PrivacyMode::Redact); + let tag = format!("key-{}", gh()); + let meta = serde_json::json!({ + "note": format!("token {}", gh()), + "nested": {"list": [format!("x {}", gh()), 3]}, + "count": 7, + }); + let o = u + .remember_detailed("clean content", &[tag.as_str(), "ops"], Some(meta), None) + .unwrap(); + assert!(o.redacted()); + assert_eq!(o.redactions.iter().map(|r| r.count).sum::(), 3); + + let m = u.get_by_id(&o.id).unwrap().unwrap(); + assert_eq!(m.content, "clean content"); + let dump = format!("{:?} {}", m.tags, m.metadata); + assert!(!dump.contains("aB3dE5gH7jK9"), "{dump}"); + assert!(dump.contains("[REDACTED:github-token]")); + assert_eq!(m.metadata["count"], 7, "other values are untouched"); + assert_eq!(m.metadata["redacted"], true); + assert!(m.tags.contains(&"ops".to_string())); +} + +#[test] +fn strict_rejects_a_secret_in_a_tag_or_in_metadata() { + let u = store(PrivacyMode::Strict); + let tag = gh(); + let err = u + .remember_detailed("clean", &[tag.as_str()], None, None) + .unwrap_err(); + assert!(matches!(err, Error::SecretDetected { .. }), "{err}"); + let err = u + .remember_detailed("clean", &[], Some(serde_json::json!({"k": gh()})), None) + .unwrap_err(); + assert!(matches!(err, Error::SecretDetected { .. }), "{err}"); + assert!(u.list(None, 10, 0, None).unwrap().is_empty()); +} + +#[test] +fn an_update_filters_tags_and_metadata_and_reports_it() { + let u = store(PrivacyMode::Redact); + let id = u.remember_detailed("plain", &[], None, None).unwrap().id; + + // Metadata only: no new content, a secret in a metadata string. + let out = u + .update_memory_detailed( + &id, + None, + None, + Some(&serde_json::json!({"note": format!("k {}", gh())})), + None, + None, + None, + ) + .unwrap(); + assert!(out.updated && out.redacted()); + assert_eq!(out.redaction_summary().as_deref(), Some("github-token x1")); + let m = u.get_by_id(&id).unwrap().unwrap(); + assert!( + !m.metadata.to_string().contains("aB3dE5gH7jK9"), + "{}", + m.metadata + ); + assert_eq!(m.metadata["redacted"], true); + + // Tags. + let out = u + .update_memory_detailed(&id, None, Some(&[gh()]), None, None, None, None) + .unwrap(); + assert!(out.redacted()); + let m = u.get_by_id(&id).unwrap().unwrap(); + assert!( + !format!("{:?}", m.tags).contains("aB3dE5gH7jK9"), + "{:?}", + m.tags + ); + + // Nothing to mask, and the plain bool API still works. + let out = u + .update_memory_detailed(&id, Some("still plain"), None, None, None, None, None) + .unwrap(); + assert!(out.updated && !out.redacted() && out.redaction_summary().is_none()); + assert!( + u.update_memory(&id, None, None, None, Some(0.9), None, None) + .unwrap() + ); +} + +#[test] +fn strict_rejects_an_update_with_a_secret_in_metadata_or_tags() { + let u = store(PrivacyMode::Strict); + let id = u.remember_detailed("plain", &[], None, None).unwrap().id; + let err = u + .update_memory( + &id, + None, + None, + Some(&serde_json::json!({"k": gh()})), + None, + None, + None, + ) + .unwrap_err(); + assert!(matches!(err, Error::SecretDetected { .. }), "{err}"); + let err = u + .update_memory(&id, None, Some(&[gh()]), None, None, None, None) + .unwrap_err(); + assert!(matches!(err, Error::SecretDetected { .. }), "{err}"); +} + +#[test] +fn the_source_is_masked_or_refused() { + let u = store(PrivacyMode::Redact); + let id = u.remember_detailed("plain", &[], None, None).unwrap().id; + assert!( + u.set_source(&id, Some(&format!("https://x.example/?t={}", gh())), "user") + .unwrap() + ); + let m = u.get_by_id(&id).unwrap().unwrap(); + let source = m.source.unwrap(); + assert!( + source.contains("[REDACTED:github-token]") && !source.contains("aB3dE5gH7jK9"), + "{source}" + ); + + let s = store(PrivacyMode::Strict); + let id = s.remember_detailed("plain", &[], None, None).unwrap().id; + let err = s.set_source(&id, Some(&gh()), "user").unwrap_err(); + assert!(matches!(err, Error::SecretDetected { .. }), "{err}"); + assert!(s.get_by_id(&id).unwrap().unwrap().source.is_none()); +} + +#[test] +fn import_masks_tags_and_source() { + let u = store(PrivacyMode::Redact); + let row = serde_json::json!({ + "content": "imported row", + "tags": [gh(), "ops"], + "source": format!("file with {}", gh()), + }) + .to_string(); + let r = u.import(&row, None).unwrap(); + assert_eq!((r.imported, r.redacted), (1, 1)); + let m = &u.list(None, 10, 0, None).unwrap()[0]; + let dump = format!("{:?} {:?}", m.tags, m.source); + assert!(!dump.contains("aB3dE5gH7jK9"), "{dump}"); + assert!(m.tags.contains(&"ops".to_string())); +} diff --git a/crates/uteke-mcp/src/lib.rs b/crates/uteke-mcp/src/lib.rs index 0c30644c..7ede4dad 100644 --- a/crates/uteke-mcp/src/lib.rs +++ b/crates/uteke-mcp/src/lib.rs @@ -1661,8 +1661,8 @@ fn exec_update(uteke: &Uteke, args: &Value) -> Result { let tags_ref = tags.as_deref(); let metadata = args.get("metadata").filter(|m| !m.is_null()); - let updated = uteke - .update_memory( + let outcome = uteke + .update_memory_detailed( &id, content, tags_ref, @@ -1673,7 +1673,7 @@ fn exec_update(uteke: &Uteke, args: &Value) -> Result { ) .map_err(write_error)?; - if !updated { + if !outcome.updated { return Ok(ToolResult { content: vec![McpContent::Text { r#type: "text".to_string(), @@ -1703,7 +1703,11 @@ fn exec_update(uteke: &Uteke, args: &Value) -> Result { content: vec![McpContent::Text { r#type: "text".to_string(), text: format!( - "āœ“ Updated memory {id}{}", + "āœ“ Updated memory {id}{}{}", + outcome + .redaction_summary() + .map(|s| format!("\nāš  Secrets masked in the stored text (#1464): {s}")) + .unwrap_or_default(), namespace_deprecation_suffix(args["namespace"].as_str()) ), }], @@ -4290,6 +4294,21 @@ mod privacy_mcp_tests { assert!(!text(&clean).contains("Secrets masked")); } + #[test] + fn update_says_when_a_secret_was_masked() { + let u = store(PrivacyMode::Redact); + let ok = exec_remember(&u, &serde_json::json!({"content": "plain"})).unwrap(); + let id = text(&ok).rsplit(' ').next().unwrap().trim().to_string(); + let r = exec_update(&u, &serde_json::json!({"id": id, "tags": [token()]})).unwrap(); + let t = text(&r); + assert!( + t.contains("Secrets masked") && t.contains("github-token x1"), + "{t}" + ); + let r = exec_update(&u, &serde_json::json!({"id": id, "importance": 0.9})).unwrap(); + assert!(!text(&r).contains("Secrets masked")); + } + #[test] fn strict_mode_is_a_secret_detected_tool_error_for_remember_and_update() { let u = store(PrivacyMode::Strict); diff --git a/crates/uteke-server/src/handlers/memory.rs b/crates/uteke-server/src/handlers/memory.rs index e891f1ca..9b3d1728 100644 --- a/crates/uteke-server/src/handlers/memory.rs +++ b/crates/uteke-server/src/handlers/memory.rs @@ -18,6 +18,14 @@ pub(super) fn post_remember(c: &mut Call) -> Response>> { }; let tag_refs = req_data.tag_refs(); let metadata = req_data.metadata.clone(); + // #1464: the source is set after the memory is stored, so strict + // mode has to refuse it before anything is written. + if let Some(src) = req_data.source.as_deref() + && let Err(uteke_core::Error::SecretDetected { kinds }) = + uteke.privacy_config().mask_text(src) + { + return ctx.secret_detected_response(req, &kinds); + } let result: Result = if req_data.detect_contradiction { @@ -789,7 +797,7 @@ pub(super) fn put_memory(c: &mut Call) -> Response>> { } let tag_refs: Option> = req_data.tags; let tag_slice: Option<&[String]> = tag_refs.as_deref(); - match uteke.update_memory( + match uteke.update_memory_detailed( &req_data.id, req_data.content.as_deref(), tag_slice, @@ -798,7 +806,7 @@ pub(super) fn put_memory(c: &mut Call) -> Response>> { req_data.pinned, req_data.memory_type.as_deref(), ) { - Ok(true) => { + Ok(outcome) if outcome.updated => { // Namespace move (#1181): separate op after the field // update ā€” plain column change, no re-embed needed. if let Some(ns) = req_data.namespace.as_deref() { @@ -810,15 +818,20 @@ pub(super) fn put_memory(c: &mut Call) -> Response>> { return ctx.error_response_for(req, status, e.to_string()); } } + // #1464: true when the filter masked a secret in the new + // content, tags or metadata. ctx.ok_response_for( req, &with_namespace_warning( - serde_json::json!({"updated": req_data.id}), + serde_json::json!({ + "updated": req_data.id, + "redacted": outcome.redacted(), + }), req_data.namespace.as_deref(), ), ) } - Ok(false) => { + Ok(_) => { ctx.error_response_for(req, 404, format!("Memory not found: {}", req_data.id)) } Err(uteke_core::Error::SecretDetected { kinds }) => { diff --git a/crates/uteke-server/src/handlers/privacy_tests.rs b/crates/uteke-server/src/handlers/privacy_tests.rs index 1872dec8..47e481b7 100644 --- a/crates/uteke-server/src/handlers/privacy_tests.rs +++ b/crates/uteke-server/src/handlers/privacy_tests.rs @@ -143,3 +143,43 @@ fn room_remember_reports_redacted() { assert_eq!(status, 200, "{body}"); assert_eq!(body["redacted"], true); } + +#[test] +fn put_memory_reports_redacted_and_strict_covers_source_and_metadata() { + let app = App::new(PrivacyMode::Redact); + let (_, ok, _) = app.call(Method::Post, "/remember", Some(json!({"content": "plain"}))); + let id = ok["id"].as_str().unwrap().to_string(); + let (status, body, _) = app.call( + Method::Put, + "/memory", + Some(json!({"id": id, "metadata": {"note": format!("k {}", token())}})), + ); + assert_eq!(status, 200, "{body}"); + assert_eq!(body["redacted"], true); + let (_, body, _) = app.call( + Method::Put, + "/memory", + Some(json!({"id": id, "importance": 0.8})), + ); + assert_eq!(body["redacted"], false, "{body}"); + + let strict = App::new(PrivacyMode::Strict); + let (status, body, raw) = strict.call( + Method::Post, + "/remember", + Some(json!({"content": "plain", "source": token()})), + ); + assert_eq!((status, &body["error"]), (400, &json!("secret_detected"))); + assert!(!raw.contains("aB3dE5gH7jK9"), "{raw}"); + assert_eq!( + strict.count(), + 0, + "nothing is stored before the source check" + ); + let (status, body, _) = strict.call( + Method::Post, + "/remember", + Some(json!({"content": "plain", "metadata": {"k": token()}})), + ); + assert_eq!((status, &body["error"]), (400, &json!("secret_detected"))); +} diff --git a/docs/configuration.md b/docs/configuration.md index 5a4fba12..d343d63f 100644 --- a/docs/configuration.md +++ b/docs/configuration.md @@ -420,7 +420,7 @@ allowlist = ["^dev-"] # regexes; a value matching any of them is kept as it - `strict`: refuse the write (`secret_detected`; import skips the row). - `off`: store as given. -`UTEKE_PRIVACY_MODE` overrides `mode`. An unknown mode or an invalid regex is logged and replaced by the default (`redact`) for that setting; the filter never turns itself off because of a typo. The filter covers the content of `remember`, room remember, `update` and `import`; metadata, tags and `source` are not filtered yet. +`UTEKE_PRIVACY_MODE` overrides `mode`. An unknown mode or an invalid regex is logged and replaced by the default (`redact`) for that setting; the filter never turns itself off because of a typo. The filter covers the content, the tags, the strings inside the metadata and the `source` of `remember`, room remember, `update` and `import`, and the text written by room consolidation. Only the string values inside metadata are filtered, not its object keys. Importing a structural export (`uteke export --full`, restored by `uteke import`) puts a backup back as it was and does not filter; restore backups you trust. Over HTTP, `POST /remember` and `POST /room/remember` return `"redacted": true` when something was masked (the stored text then differs from the request). In `strict` mode a write that contains a secret is refused with `400`: @@ -428,7 +428,7 @@ Over HTTP, `POST /remember` and `POST /room/remember` return `"redacted": true` {"error": "secret_detected", "message": "...", "kinds": ["github-token"]} ``` -`PUT /memory` answers the same way when new content has a secret. `POST /import` reports `redacted` (rows masked) and `rejected_secret` (rows skipped in strict mode, also counted in `skipped`). +`PUT /memory` answers the same way when the new content, tags or metadata have a secret, and returns `redacted` when it masked one. `uteke_update` over MCP says so in its result. `POST /import` reports `redacted` (rows masked) and `rejected_secret` (rows skipped in strict mode, also counted in `skipped`). ## CLI Flag Override