diff --git a/CHANGELOG.md b/CHANGELOG.md index 35eac0b0..af5caf27 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -18,6 +18,7 @@ - Internal: privacy filter modes and flag helpers, part 2b-1a of #1464 (`uteke_core::privacy`): `PrivacyMode` (`redact`, `strict`, `off`), `PrivacyConfig::prepare`, `Error::SecretDetected` and the `metadata.redacted`/`redactions` helpers. Not wired to a write path yet. - Internal: privacy filter token rules and PEM blocks, part 2b-1a of #1464 (`uteke_core::privacy`): Stripe, `whsec_`, Slack, Google, Basic and `Token`/`Bot` auth, `curl -u`, password flags and URL passwords, and PEM private key blocks in their escaped, CRLF, collapsed, quoted and string-literal forms. Scan windows only bound the work. The shared vectors move to version 3 (113 cases so far). +- Internal: the privacy filter is wired into the core write paths, part 2b-1 of #1464: `remember`, `remember_with_contradiction`, room remember, `update_memory` and `import` mask secrets in the CONTENT before it is embedded, indexed or stored, so a masked secret is absent from the row, the FTS5 index, the timeline and the database file (checked by scanning the file bytes). `Uteke::set_privacy_config` takes `PrivacyMode` (`Redact`, the default, `Strict` or `Off`) and the allowlist; a masked memory gets `metadata.redacted = true` and `metadata.redactions` (`[{kind, count}]`, never a value), a `redacted` timeline event, and `RememberOutcome.redactions`. The flags follow the content: an update to clean content drops them, a masked round trip or a metadata-only update keeps them. `Strict` returns the new `Error::SecretDetected { kinds }` (an import skips that row). Detection was reworked after two adversarial reviews (on 330 ordinary texts the first version masked 139, on 147 leaks it missed 62; now 3 and 16): a bare `key`, `pass`, `token` or `secret` is an ordinary word and a name only when compound (`API_KEY`, `DB_PASS`, `apiKey`, `PGPASSWORD`) or written with `=`; a key or token value needs a digit, so type annotations and code references (`token: &str`, `config.jwtSecret`) are left alone; paths, ARNs, digests, `0x` addresses, versions and dates are not secrets; quoted values, URL query strings, glued pairs (`MODE=prod;API_KEY=...`), escaped JSON, table cells and CLI flags (`--password`, `docker login -p`, `curl -u`) are read whole; new kinds: Basic and `Token` authorization headers, passwords in URLs, Stripe, Slack, Google, npm and webhook secrets; a private key block needs its matching END (or is cut off) and key-like content. Known limits, next parts: metadata, tags and `source` are not scanned, the structural import (`uteke import` of a full export), room consolidation and the LLM extraction paths do not call the filter yet, a secret glued to letters or `_` (`MY_ghp_...`), a quoted passphrase with spaces and a value containing `(` can still pass, a field named `password` holding a hyphenated phrase is masked, and nothing reads `[privacy]` from `uteke.toml`, so until part 2b-2 the default `Redact` applies to every write. - Internal: privacy filter assignment scanner, part 2b-1a of #1464 (`uteke_core::privacy`): a secret-like name must end in a whole secret word (`API_KEY`, `DB_PASS`, `apiKey`; qualifiers such as `_PROD` are ignored), values are read whole (quotes, escaped JSON, `:=`, `=>`, table cells, `&`/`;` pair boundaries, passwords with `,` or `\`), and benign values (paths, ARNs, digests, versions, ULIDs, dates, emails, shell references, identifiers, placeholders) are left alone. On the 330-case benign and 147-case secret review corpus: 3 false positives, 16 misses (the first detector: 139 and 62). Shared vectors complete at version 3 (192 cases). A block of key lines is followed to its end up to 1 MiB. - Internal: `docs/privacy-filter-cases.json`, the shared test vectors of the privacy filter (35 cases: every kind and 23 inputs that must stay untouched; inputs are split into fragments so no complete token appears, and the file is excluded from Cora review in `.cora.yaml` because its secret-shaped assignments are the point of the test). `uteke-core` pins every case, and uteke-cloud pins parity against the same file (uteke-cloud#218). Part 2a of #1464. - Internal: `memories.author`, part 1 of #1427 phase 2 (schema v22): an optional, free-text author on the memory itself, usable without a room. The first open by a build with this change migrates the store in place (additive `ALTER TABLE`; a memory that belongs to exactly one room takes that room link's author, the rest stay unset). `Memory` gains `author` (omitted from JSON when unset) and `Uteke::set_author` validates it (trimmed, 1 to 128 characters, no control characters). No HTTP, MCP or CLI input accepts it yet: that follows once uteke-cloud#219 is answered. diff --git a/crates/uteke-cli/src/commands/remember.rs b/crates/uteke-cli/src/commands/remember.rs index 0420fc77..1d860938 100644 --- a/crates/uteke-cli/src/commands/remember.rs +++ b/crates/uteke-cli/src/commands/remember.rs @@ -66,7 +66,12 @@ pub(crate) fn run( source_type: Option<&str>, timestamp: Option<&str>, ) -> Result<(), String> { - tracing::debug!("Remembering: {content} (type: {type}, contradiction: {detect_contradiction})"); + // The content is not logged: it may hold a secret the privacy filter has + // not masked yet (#1464). + tracing::debug!( + "Remembering {} bytes (type: {type}, contradiction: {detect_contradiction})", + content.len() + ); // #1106: validate BEFORE any write so an invalid value fails loudly // without leaving a stored row behind (the post-store set would have diff --git a/crates/uteke-core/src/consolidate.rs b/crates/uteke-core/src/consolidate.rs index 8b6fb998..a831c302 100644 --- a/crates/uteke-core/src/consolidate.rs +++ b/crates/uteke-core/src/consolidate.rs @@ -72,6 +72,10 @@ impl crate::Uteke { ) -> Result<(RememberOutcome, ContradictionResult), Error> { crate::validate_input(content, tags)?; let ns = namespace.unwrap_or(DEFAULT_NAMESPACE); + // Privacy filter (#1464), before the content is embedded or stored. + let prepared = self.privacy_config.prepare(content, metadata)?; + let content = prepared.content.as_str(); + let metadata = prepared.metadata.clone(); // Embed first to check for contradictions before persisting self.ensure_embedder()?; @@ -98,7 +102,7 @@ impl crate::Uteke { let content_type = crate::memory::crud::detect_content_type(content); // Use remember_precomputed to avoid double-embedding - let outcome = self.remember_precomputed_detailed( + let mut outcome = self.remember_precomputed_detailed( content, tags, metadata, @@ -109,6 +113,8 @@ impl crate::Uteke { None, )?; let id = outcome.id.clone(); + self.record_redaction(&id, &prepared.redactions); + outcome.redactions = prepared.redactions.clone(); // Timeline: if this is a contradiction-aware remember that // supersedes an older memory, record a Consolidated event (#347) diff --git a/crates/uteke-core/src/import_export.rs b/crates/uteke-core/src/import_export.rs index 3ee2c953..bc5e3b08 100644 --- a/crates/uteke-core/src/import_export.rs +++ b/crates/uteke-core/src/import_export.rs @@ -80,12 +80,32 @@ impl crate::Uteke { let mut imported = 0; let mut deduped = 0; - for entry in entries { + for mut entry in entries { if entry.content.is_empty() { skipped += 1; continue; } + // Privacy filter (#1464), before the content is embedded or stored. + // Strict mode skips the row (counted as skipped) instead of + // aborting the whole import. + let prepared = match self + .privacy_config + .prepare(&entry.content, Some(std::mem::take(&mut entry.metadata))) + { + Ok(p) => p, + Err(e) => { + tracing::warn!("Skipping import entry: {e}"); + skipped += 1; + continue; + } + }; + entry.content = prepared.content; + if let Some(m) = prepared.metadata { + entry.metadata = m; + } + let redactions = prepared.redactions; + // Re-embed the content with retry (consistent with remember path, #1005). self.ensure_embedder()?; let embedding = crate::operations::retry_embed(&self.embedder, &entry.content)?; @@ -165,6 +185,8 @@ impl crate::Uteke { continue; } + self.record_redaction(&id, &redactions); + // Auto-link cosine edges (consistent with remember path, #1005). // Must run AFTER index.insert() so the new memory is searchable. self.auto_link_cosine(&id, &embedding, Some(memory.namespace.as_str())); diff --git a/crates/uteke-core/src/lib.rs b/crates/uteke-core/src/lib.rs index eaf00019..aa2cecd6 100644 --- a/crates/uteke-core/src/lib.rs +++ b/crates/uteke-core/src/lib.rs @@ -36,6 +36,8 @@ pub mod offline_extraction; mod operations; pub mod pack_mode; pub mod privacy; +#[cfg(test)] +mod privacy_write_tests; pub use forget_request::{ ForgetErrorKind, ForgetInput, ForgetPolicy, ForgetRequest, ForgetRequestError, FullIdForm, @@ -645,6 +647,9 @@ pub struct Uteke { /// Dream pipeline thresholds (#731). Used by contradiction scan, /// dedup/consolidate, and orphan detection phases. dream_config: DreamConfig, + /// Privacy filter applied to content before it is embedded or stored + /// (#1464). Default: mask secrets (`redact`), no allowlist. + privacy_config: privacy::PrivacyConfig, /// Memory lifecycle configuration (#928). Controls soft-delete behavior, /// dynamic deprecation caps, and TTL-based pruning. lifecycle_config: LifecycleConfig, @@ -1050,6 +1055,7 @@ impl Uteke { recall_cache: recall_cache::RecallCache::new(recall_cache::RecallCacheConfig::default()), jaccard_weight: 0.0, dream_config: DreamConfig::default(), + privacy_config: privacy::PrivacyConfig::default(), lifecycle_config: LifecycleConfig::default(), embed_cache_path, }) @@ -1093,6 +1099,18 @@ impl Uteke { self.dream_config = config; } + /// Set the privacy filter (#1464): `redact` (default), `strict` or `off`, + /// plus the allowlist of values to keep. Called once at startup from the + /// CLI and server config. + pub fn set_privacy_config(&mut self, config: privacy::PrivacyConfig) { + self.privacy_config = config; + } + + /// The privacy filter in force. + pub fn privacy_config(&self) -> &privacy::PrivacyConfig { + &self.privacy_config + } + /// Set memory lifecycle configuration (#928). /// /// Controls soft-delete behavior, dynamic deprecation caps, and diff --git a/crates/uteke-core/src/operations.rs b/crates/uteke-core/src/operations.rs index 40cd3ea7..e316011f 100644 --- a/crates/uteke-core/src/operations.rs +++ b/crates/uteke-core/src/operations.rs @@ -85,6 +85,18 @@ pub struct RememberOutcome { pub embedding_written: bool, /// Failure reason when the embedding could not be persisted. pub warning: Option, + /// Secrets the privacy filter masked in the content of THIS call (#1464): a + /// kind and a count each, never a value. Empty when nothing was masked. On a + /// dedup hit nothing new is stored, so the stored memory keeps its own flags. + #[serde(skip_serializing_if = "Vec::is_empty")] + pub redactions: Vec, +} + +impl RememberOutcome { + /// Whether the privacy filter masked anything in the stored content. + pub fn redacted(&self) -> bool { + !self.redactions.is_empty() + } } impl crate::Uteke { @@ -233,6 +245,20 @@ impl crate::Uteke { self.remember_embed_detailed(content, tags, metadata, namespace, &effective_type) } + /// Audit entry for a write whose content the privacy filter changed + /// (#1464): kinds and counts, never a value. Best-effort like every + /// timeline event. + pub(crate) fn record_redaction(&self, id: &str, redactions: &[crate::privacy::RedactionCount]) { + if redactions.is_empty() { + return; + } + self.try_timeline_event( + id, + crate::timeline::TimelineEventType::Redacted, + Some(&serde_json::json!({ "redactions": redactions })), + ); + } + /// Embed-then-store with honest embedding reporting (#1273). fn remember_embed_detailed( &self, @@ -249,6 +275,11 @@ impl crate::Uteke { "Unknown memory type '{memory_type}'. Valid types: fact, procedure, preference, decision, context, note, insight, reference, event" )) })?; + // Privacy filter (#1464): before the content is embedded, indexed or + // stored, so a secret never reaches the vector, FTS5 or the row. + let prepared = self.privacy_config.prepare(content, metadata)?; + let content = prepared.content.as_str(); + let metadata = prepared.metadata.clone(); // Detect JSON content and use flattened text for embedding let content_type = crate::memory::crud::detect_content_type(content); let embed_text = if content_type == "json" { @@ -300,11 +331,12 @@ impl crate::Uteke { id: existing_id, embedding_written: true, warning: None, + redactions: prepared.redactions.clone(), }); } } - self.remember_precomputed_detailed( + let mut outcome = self.remember_precomputed_detailed( content, tags, metadata, @@ -313,7 +345,10 @@ impl crate::Uteke { content_type, &embedding, generation_error, - ) + )?; + self.record_redaction(&outcome.id, &prepared.redactions); + outcome.redactions = prepared.redactions; + Ok(outcome) } /// Store a new memory with a pre-computed embedding. @@ -551,6 +586,7 @@ impl crate::Uteke { embedding_written, warning: vector_error, id, + redactions: Vec::new(), }) } @@ -1869,6 +1905,56 @@ impl crate::Uteke { } } + // Privacy filter (#1464) on the new content, before it is embedded or + // stored. A masked update also records `redacted` in the metadata: + // the caller's metadata when given, else the memory's current one. + let prepared = match content { + Some(c) => Some(self.privacy_config.prepare(c, None)?), + None => None, + }; + let redactions = prepared + .as_ref() + .map(|p| p.redactions.clone()) + .unwrap_or_default(); + let masked_content: Option = prepared.map(|p| p.content); + let content: Option<&str> = masked_content.as_deref(); + let flagged_metadata: Option = if redactions.is_empty() { + match (content, metadata) { + // Clean new content and no metadata from the caller: the flags + // a previous version left no longer describe the content, + // unless the edit kept a mask marker (a read-edit-write round + // trip of masked text). + (Some(c), None) if !c.contains("[REDACTED:") => { + crate::privacy::without_redaction_flags(&existing.metadata) + } + // A metadata-only update keeps the masked content, so it keeps + // the flags that describe it. + (None, Some(m)) => crate::privacy::carry_redaction_flags(m, &existing.metadata), + // New content and new metadata: the flags follow the content, + // never what the caller's metadata says about itself. + (Some(c), Some(m)) if c.contains("[REDACTED:") => { + let own = crate::privacy::without_redaction_flags(m); + crate::privacy::carry_redaction_flags( + own.as_ref().unwrap_or(m), + &existing.metadata, + ) + .or(own) + } + (Some(_), Some(m)) => crate::privacy::without_redaction_flags(m), + _ => None, + } + } else { + crate::privacy::with_redaction_flags( + Some( + metadata + .cloned() + .unwrap_or_else(|| existing.metadata.clone()), + ), + &redactions, + ) + }; + let metadata: Option<&serde_json::Value> = flagged_metadata.as_ref().or(metadata); + let now = chrono::Utc::now(); // Re-embed if content changed @@ -1932,6 +2018,8 @@ impl crate::Uteke { } } + self.record_redaction(id, &redactions); + // Invalidate recall cache for the memory's namespace self.recall_cache.invalidate_namespace(&existing.namespace); diff --git a/crates/uteke-core/src/privacy_write_tests.rs b/crates/uteke-core/src/privacy_write_tests.rs new file mode 100644 index 00000000..a8124203 --- /dev/null +++ b/crates/uteke-core/src/privacy_write_tests.rs @@ -0,0 +1,572 @@ +//! The privacy filter on the write paths (#1464, part 2b-1): `remember`, +//! `remember_with_contradiction`, room remember, `update_memory` and `import`. +//! A masked secret must be absent from the content, the FTS index, the +//! metadata, the timeline and the database file itself. + +use crate::memory::types::DEFAULT_NAMESPACE; +use crate::privacy::{PrivacyConfig, PrivacyMode, compile_allowlist}; +use crate::test_support::{open_keyword, open_keyword_at}; +use crate::{Error, Uteke}; + +/// The fake credential, assembled at run time so the repository holds no +/// string a scanner would flag. +fn gh() -> String { + format!("{}{}", "ghp_", "aB3dE5gH7jK9mN1pQ3rS5tU7vW9xY1zA3bC5") +} + +fn aws() -> String { + format!("{}{}", "AKIA", "IOSFODNN7EXAMPLQ") +} + +fn store(mode: PrivacyMode) -> Uteke { + let mut u = open_keyword(); + u.set_privacy_config(PrivacyConfig { + mode, + allowlist: Vec::new(), + }); + u +} + +fn kinds(o: &crate::operations::RememberOutcome) -> Vec<(&'static str, usize)> { + o.redactions.iter().map(|r| (r.kind, r.count)).collect() +} + +#[test] +fn redact_mode_masks_the_secret_and_flags_the_memory() { + let u = store(PrivacyMode::Redact); + let text = format!("deploy with {} then {}", gh(), aws()); + let o = u + .remember_typed_detailed(&text, &["ops"], None, None, "fact") + .unwrap(); + assert!(o.redacted()); + assert_eq!(kinds(&o), vec![("aws-access-key", 1), ("github-token", 1)]); + + let m = u.get_by_id(&o.id).unwrap().unwrap(); + assert_eq!( + m.content, + "deploy with ghp_[REDACTED:github-token] then AKIA[REDACTED:aws-access-key]" + ); + assert_eq!(m.metadata["redacted"], true); + assert_eq!( + m.metadata["redactions"], + serde_json::json!([ + {"kind": "aws-access-key", "count": 1}, + {"kind": "github-token", "count": 1} + ]) + ); + // Never the value, in the metadata or in the audit event. + let flags = m.metadata.to_string(); + assert!(!flags.contains(&gh()) && !flags.contains(&aws()), "{flags}"); + + // The audit entry names the kinds and nothing else. + let events = u.timeline(&o.id, 10).unwrap(); + let redacted: Vec<_> = events + .iter() + .filter(|e| e.event_type == "redacted") + .collect(); + assert_eq!(redacted.len(), 1, "{events:?}"); + let data = redacted[0].event_data.clone().unwrap(); + assert!( + data.contains("github-token") && !data.contains(&gh()), + "{data}" + ); + + // Recall and keyword search see the masked form only. + let hits = u.search("deploy", 10, None, None).unwrap(); + assert_eq!(hits[0].memory.content, m.content); + assert!(u.search(&gh(), 10, None, None).unwrap().is_empty()); +} + +#[test] +fn clean_content_and_caller_metadata_are_untouched_and_merged() { + let u = store(PrivacyMode::Redact); + let clean = u + .remember_typed_detailed( + "nothing secret here", + &[], + Some(serde_json::json!({"k": "v"})), + None, + "fact", + ) + .unwrap(); + assert!(!clean.redacted()); + let m = u.get_by_id(&clean.id).unwrap().unwrap(); + assert_eq!( + m.metadata, + serde_json::json!({"k": "v"}), + "no flags on clean content" + ); + assert!( + u.timeline(&clean.id, 10) + .unwrap() + .iter() + .all(|e| e.event_type != "redacted") + ); + + // With a secret, the caller's keys stay next to the flags. + let o = u + .remember_typed_detailed( + &format!("key {}", gh()), + &[], + Some(serde_json::json!({"k": "v"})), + None, + "fact", + ) + .unwrap(); + let m = u.get_by_id(&o.id).unwrap().unwrap(); + assert_eq!(m.metadata["k"], "v"); + assert_eq!(m.metadata["redacted"], true); +} + +#[test] +fn strict_mode_refuses_the_write_and_stores_nothing() { + let u = store(PrivacyMode::Strict); + let err = u + .remember_typed_detailed(&format!("key {}", gh()), &[], None, None, "fact") + .unwrap_err(); + match &err { + Error::SecretDetected { kinds } => assert_eq!(kinds, &vec!["github-token".to_string()]), + other => panic!("expected SecretDetected, got {other:?}"), + } + assert!(err.to_string().contains("github-token"), "{err}"); + assert!( + !err.to_string().contains(&gh()), + "the error never carries the value" + ); + assert_eq!(u.count(None).unwrap(), 0); + // Clean content is still accepted in strict mode. + u.remember("plain note", &[], None, None).unwrap(); + assert_eq!(u.count(None).unwrap(), 1); +} + +#[test] +fn off_mode_stores_the_content_as_given() { + let u = store(PrivacyMode::Off); + let o = u + .remember_typed_detailed(&format!("key {}", gh()), &[], None, None, "fact") + .unwrap(); + assert!(!o.redacted()); + let m = u.get_by_id(&o.id).unwrap().unwrap(); + assert!(m.content.contains(&gh())); + assert!(m.metadata.get("redacted").is_none()); +} + +#[test] +fn the_allowlist_keeps_a_matching_value() { + let mut u = open_keyword(); + u.set_privacy_config(PrivacyConfig { + mode: PrivacyMode::Redact, + allowlist: compile_allowlist(&["^ghp_aB3".to_string()]).unwrap(), + }); + let o = u + .remember_typed_detailed(&format!("key {}", gh()), &[], None, None, "fact") + .unwrap(); + assert!(!o.redacted()); + assert!(u.get_by_id(&o.id).unwrap().unwrap().content.contains(&gh())); +} + +#[test] +fn the_contradiction_and_room_paths_are_filtered_too() { + let u = store(PrivacyMode::Redact); + let (o, _) = u + .remember_with_contradiction(&format!("key {}", gh()), &[], None, None, None, true, 0.9) + .unwrap(); + assert_eq!(kinds(&o), vec![("github-token", 1)]); + assert!(!u.get_by_id(&o.id).unwrap().unwrap().content.contains(&gh())); + + let room = u + .remember_in_room( + &format!("room {}", aws()), + &[], + None, + None, + "fact", + "r1", + "alice", + ) + .unwrap(); + assert_eq!(kinds(&room), vec![("aws-access-key", 1)]); + assert!( + !u.get_by_id(&room.id) + .unwrap() + .unwrap() + .content + .contains(&aws()) + ); + + let strict = store(PrivacyMode::Strict); + assert!(matches!( + strict.remember_with_contradiction( + &format!("key {}", gh()), + &[], + None, + None, + None, + true, + 0.9 + ), + Err(Error::SecretDetected { .. }) + )); + assert!(matches!( + strict.remember_in_room( + &format!("room {}", aws()), + &[], + None, + None, + "fact", + "r1", + "alice" + ), + Err(Error::SecretDetected { .. }) + )); + assert_eq!(strict.count(None).unwrap(), 0); +} + +#[test] +fn update_memory_masks_new_content_and_flags_the_metadata() { + let u = store(PrivacyMode::Redact); + let id = u + .remember_typed_detailed( + "first version", + &[], + Some(serde_json::json!({"k": "v"})), + None, + "fact", + ) + .unwrap() + .id; + assert!( + u.update_memory( + &id, + Some(&format!("now with {}", gh())), + None, + None, + None, + None, + None + ) + .unwrap() + ); + let m = u.get_by_id(&id).unwrap().unwrap(); + assert_eq!(m.content, "now with ghp_[REDACTED:github-token]"); + assert_eq!(m.metadata["k"], "v", "the existing metadata is kept"); + assert_eq!(m.metadata["redacted"], true); + assert_eq!( + u.timeline(&id, 20) + .unwrap() + .iter() + .filter(|e| e.event_type == "redacted") + .count(), + 1 + ); + // An update without content leaves the flags alone. + u.update_memory(&id, None, None, None, Some(0.9), None, None) + .unwrap(); + assert_eq!( + u.get_by_id(&id).unwrap().unwrap().metadata["redacted"], + true + ); + + // Strict: the update is refused and the memory keeps its old content. + let s = store(PrivacyMode::Strict); + let sid = s.remember("original", &[], None, None).unwrap(); + let err = s + .update_memory( + &sid, + Some(&format!("now {}", gh())), + None, + None, + None, + None, + None, + ) + .unwrap_err(); + assert!(matches!(err, Error::SecretDetected { .. }), "{err:?}"); + assert_eq!(s.get_by_id(&sid).unwrap().unwrap().content, "original"); +} + +/// The flags describe the current content: an update to clean content drops +/// the ones an earlier version left (the timeline keeps the history), and a +/// caller who supplies metadata owns it. +#[test] +fn a_clean_update_drops_the_stale_flags_and_keeps_the_other_metadata() { + let u = store(PrivacyMode::Redact); + let id = u + .remember_typed_detailed( + &format!("key {}", gh()), + &[], + Some(serde_json::json!({"k": "v"})), + None, + "fact", + ) + .unwrap() + .id; + assert_eq!( + u.get_by_id(&id).unwrap().unwrap().metadata["redacted"], + true + ); + + u.update_memory( + &id, + Some("rotated, nothing secret now"), + None, + None, + None, + None, + None, + ) + .unwrap(); + let m = u.get_by_id(&id).unwrap().unwrap(); + assert_eq!(m.content, "rotated, nothing secret now"); + assert_eq!( + m.metadata, + serde_json::json!({"k": "v"}), + "flags dropped, k kept" + ); + let events = u.timeline(&id, 20).unwrap(); + assert_eq!( + events.iter().filter(|e| e.event_type == "redacted").count(), + 1, + "the history of the first masking stays" + ); + + // An update that does not touch the content keeps whatever flags exist. + let id2 = u + .remember_typed_detailed(&format!("key {}", gh()), &[], None, None, "fact") + .unwrap() + .id; + u.update_memory(&id2, None, Some(&["t".to_string()]), None, None, None, None) + .unwrap(); + assert_eq!( + u.get_by_id(&id2).unwrap().unwrap().metadata["redacted"], + true + ); + + // Metadata supplied by the caller replaces it as before, untouched. + u.update_memory( + &id2, + Some("clean again"), + None, + Some(&serde_json::json!({"mine": 1})), + None, + None, + None, + ) + .unwrap(); + assert_eq!( + u.get_by_id(&id2).unwrap().unwrap().metadata, + serde_json::json!({"mine": 1}) + ); +} + +/// Flags follow the content: a read-edit-write round trip of masked text and a +/// metadata-only update both keep them; null import metadata still gets them. +#[test] +fn flags_survive_a_masked_round_trip_a_metadata_only_update_and_null_metadata() { + let u = store(PrivacyMode::Redact); + let id = u + .remember_typed_detailed( + &format!("key {}", gh()), + &[], + Some(serde_json::json!({"k": "v"})), + None, + "fact", + ) + .unwrap() + .id; + + // An agent reads the masked memory, fixes a word and writes it back: the + // content still carries the mask marker, so the flags stay. + u.update_memory( + &id, + Some("key ghp_[REDACTED:github-token] (rotated)"), + None, + None, + None, + None, + None, + ) + .unwrap(); + assert_eq!( + u.get_by_id(&id).unwrap().unwrap().metadata["redacted"], + true + ); + + // A metadata-only update replaces the metadata but keeps the flags that + // describe the unchanged content. + u.update_memory( + &id, + None, + None, + Some(&serde_json::json!({"new": 1})), + None, + None, + None, + ) + .unwrap(); + let m = u.get_by_id(&id).unwrap().unwrap(); + assert_eq!(m.metadata["new"], 1); + assert_eq!(m.metadata["redacted"], true); + assert!( + m.metadata.get("k").is_none(), + "the caller's metadata replaced the old keys" + ); + // Metadata that sets its own flags is left as given. + u.update_memory( + &id, + None, + None, + Some(&serde_json::json!({"redacted": false})), + None, + None, + None, + ) + .unwrap(); + assert_eq!( + u.get_by_id(&id).unwrap().unwrap().metadata["redacted"], + false + ); + + // An import row with `"metadata": null` is still flagged. + let file = format!("{{\"content\":\"export {}\",\"metadata\":null}}", gh()); + let imp = store(PrivacyMode::Redact); + imp.import(&file, None).unwrap(); + let row = &imp.list(None, 10, 0, None).unwrap()[0]; + assert_eq!(row.metadata["redacted"], true, "{row:?}"); +} + +#[test] +fn import_masks_rows_and_strict_skips_them() { + let file = format!( + "{{\"content\":\"export {}\"}}\n{{\"content\":\"plain row\"}}", + gh() + ); + let u = store(PrivacyMode::Redact); + assert_eq!(u.import(&file, None).unwrap().imported, 2); + let rows = u.list(None, 10, 0, None).unwrap(); + assert!(rows.iter().all(|m| !m.content.contains(&gh())), "{rows:?}"); + let masked = rows + .iter() + .find(|m| m.content.contains("[REDACTED:github-token]")) + .unwrap(); + assert_eq!(masked.metadata["redacted"], true); + assert!( + rows.iter() + .find(|m| m.content == "plain row") + .unwrap() + .metadata + .get("redacted") + .is_none() + ); + + let s = store(PrivacyMode::Strict); + let r = s.import(&file, None).unwrap(); + assert_eq!( + (r.imported, r.skipped), + (1, 1), + "the row with a secret is skipped" + ); + assert_eq!(s.list(None, 10, 0, None).unwrap()[0].content, "plain row"); +} + +/// The strongest check: after a masked write, the secret is nowhere in the +/// database files: not in the row, the FTS index, the metadata or the timeline. +#[test] +fn the_secret_is_not_in_the_database_files() { + let dir = tempfile::tempdir().unwrap(); + let path = dir.path().join("uteke.db"); + { + let u = open_keyword_at(&path); + let o = u + .remember_typed_detailed( + &format!("deploy {} now", gh()), + &["ops"], + None, + Some(DEFAULT_NAMESPACE), + "fact", + ) + .unwrap(); + assert!(o.redacted()); + assert!( + u.update_memory( + &o.id, + Some(&format!("again {}", aws())), + None, + None, + None, + None, + None, + ) + .unwrap() + ); + u.import(&format!("{{\"content\":\"imported {}\"}}", gh()), None) + .unwrap(); + } + let mut scanned = 0; + for entry in std::fs::read_dir(dir.path()).unwrap().flatten() { + let bytes = std::fs::read(entry.path()).unwrap(); + scanned += bytes.len(); + for secret in [gh(), aws()] { + let found = bytes.windows(secret.len()).any(|w| w == secret.as_bytes()); + assert!(!found, "{secret} found in {:?}", entry.file_name()); + } + } + assert!(scanned > 0, "the database files were scanned"); +} + +#[test] +fn the_mode_parses_and_defaults_to_redact() { + assert_eq!(PrivacyMode::default(), PrivacyMode::Redact); + assert_eq!(PrivacyMode::parse(" Strict ").unwrap(), PrivacyMode::Strict); + assert_eq!(PrivacyMode::parse("OFF").unwrap(), PrivacyMode::Off); + assert!(PrivacyMode::parse("paranoid").is_err()); + // A fresh store masks by default. + let u = open_keyword(); + assert_eq!(u.privacy_config().mode, PrivacyMode::Redact); +} + +/// New content together with new metadata: the flags follow the content, not +/// the caller's metadata and not the previous version. +#[test] +fn an_update_with_content_and_metadata_flags_by_the_content() { + let u = store(PrivacyMode::Redact); + let id = u + .remember_typed_detailed(&format!("key {}", gh()), &[], None, None, "fact") + .unwrap() + .id; + + // Clean content: a flag the caller put in the metadata is dropped. + u.update_memory( + &id, + Some("nothing secret"), + None, + Some(&serde_json::json!({"k": "v", "redacted": true})), + None, + None, + None, + ) + .unwrap(); + assert_eq!( + u.get_by_id(&id).unwrap().unwrap().metadata, + serde_json::json!({"k": "v"}) + ); + + // Content that still carries a mask marker keeps the earlier flags. + let id2 = u + .remember_typed_detailed(&format!("key {}", gh()), &[], None, None, "fact") + .unwrap() + .id; + u.update_memory( + &id2, + Some("edited, still [REDACTED:github-token]"), + None, + Some(&serde_json::json!({"k": "v"})), + None, + None, + None, + ) + .unwrap(); + let m = u.get_by_id(&id2).unwrap().unwrap(); + assert_eq!(m.metadata["k"], "v"); + assert_eq!(m.metadata["redacted"], true); +} diff --git a/crates/uteke-core/src/test_support.rs b/crates/uteke-core/src/test_support.rs index d8e3c332..00c27083 100644 --- a/crates/uteke-core/src/test_support.rs +++ b/crates/uteke-core/src/test_support.rs @@ -35,7 +35,12 @@ impl crate::embed::Embedder for KeywordEmbedder { } pub fn open_keyword() -> crate::Uteke { - let (_db, store) = crate::Uteke::open_store(":memory:").expect("open_store"); + open_keyword_at(":memory:") +} + +/// [`open_keyword`] on a database file, for tests that inspect the file. +pub fn open_keyword_at(path: impl AsRef) -> crate::Uteke { + let (_db, store) = crate::Uteke::open_store(path).expect("open_store"); crate::Uteke::finish_open_full( store, Some(Box::new(KeywordEmbedder)), diff --git a/crates/uteke-core/src/timeline.rs b/crates/uteke-core/src/timeline.rs index 11f19a57..91660e54 100644 --- a/crates/uteke-core/src/timeline.rs +++ b/crates/uteke-core/src/timeline.rs @@ -43,6 +43,9 @@ pub enum TimelineEventType { Superseded, /// A supersession was undone (#1172 Fase 2) — memory restored. SupersessionUndone, + /// The privacy filter masked secrets in the content (#1464). The event + /// data lists the kinds and counts, never the values. + Redacted, } impl TimelineEventType { @@ -57,6 +60,7 @@ impl TimelineEventType { Self::Deprecated => "deprecated", Self::Superseded => "superseded", Self::SupersessionUndone => "supersession_undone", + Self::Redacted => "redacted", } } @@ -71,6 +75,7 @@ impl TimelineEventType { "deprecated" => Some(Self::Deprecated), "superseded" => Some(Self::Superseded), "supersession_undone" => Some(Self::SupersessionUndone), + "redacted" => Some(Self::Redacted), _ => None, } }