feat(core): apply the privacy filter on remember, update and import (#1464, part 2b-1b)

CHANGELOG.md

@@ -18,6 +18,7 @@
 
 - Internal: privacy filter modes and flag helpers, part 2b-1a of #1464 (`uteke_core::privacy`): `PrivacyMode` (`redact`, `strict`, `off`), `PrivacyConfig::prepare`, `Error::SecretDetected` and the `metadata.redacted`/`redactions` helpers. Not wired to a write path yet.
 - Internal: privacy filter token rules and PEM blocks, part 2b-1a of #1464 (`uteke_core::privacy`): Stripe, `whsec_`, Slack, Google, Basic and `Token`/`Bot` auth, `curl -u`, password flags and URL passwords, and PEM private key blocks in their escaped, CRLF, collapsed, quoted and string-literal forms. Scan windows only bound the work. The shared vectors move to version 3 (113 cases so far).
+- Internal: the privacy filter is wired into the core write paths, part 2b-1 of #1464: `remember`, `remember_with_contradiction`, room remember, `update_memory` and `import` mask secrets in the CONTENT before it is embedded, indexed or stored, so a masked secret is absent from the row, the FTS5 index, the timeline and the database file (checked by scanning the file bytes). `Uteke::set_privacy_config` takes `PrivacyMode` (`Redact`, the default, `Strict` or `Off`) and the allowlist; a masked memory gets `metadata.redacted = true` and `metadata.redactions` (`[{kind, count}]`, never a value), a `redacted` timeline event, and `RememberOutcome.redactions`. The flags follow the content: an update to clean content drops them, a masked round trip or a metadata-only update keeps them. `Strict` returns the new `Error::SecretDetected { kinds }` (an import skips that row). Detection was reworked after two adversarial reviews (on 330 ordinary texts the first version masked 139, on 147 leaks it missed 62; now 3 and 16): a bare `key`, `pass`, `token` or `secret` is an ordinary word and a name only when compound (`API_KEY`, `DB_PASS`, `apiKey`, `PGPASSWORD`) or written with `=`; a key or token value needs a digit, so type annotations and code references (`token: &str`, `config.jwtSecret`) are left alone; paths, ARNs, digests, `0x` addresses, versions and dates are not secrets; quoted values, URL query strings, glued pairs (`MODE=prod;API_KEY=...`), escaped JSON, table cells and CLI flags (`--password`, `docker login -p`, `curl -u`) are read whole; new kinds: Basic and `Token` authorization headers, passwords in URLs, Stripe, Slack, Google, npm and webhook secrets; a private key block needs its matching END (or is cut off) and key-like content. Known limits, next parts: metadata, tags and `source` are not scanned, the structural import (`uteke import` of a full export), room consolidation and the LLM extraction paths do not call the filter yet, a secret glued to letters or `_` (`MY_ghp_...`), a quoted passphrase with spaces and a value containing `(` can still pass, a field named `password` holding a hyphenated phrase is masked, and nothing reads `[privacy]` from `uteke.toml`, so until part 2b-2 the default `Redact` applies to every write.
 - Internal: privacy filter assignment scanner, part 2b-1a of #1464 (`uteke_core::privacy`): a secret-like name must end in a whole secret word (`API_KEY`, `DB_PASS`, `apiKey`; qualifiers such as `_PROD` are ignored), values are read whole (quotes, escaped JSON, `:=`, `=>`, table cells, `&`/`;` pair boundaries, passwords with `,` or `\`), and benign values (paths, ARNs, digests, versions, ULIDs, dates, emails, shell references, identifiers, placeholders) are left alone. On the 330-case benign and 147-case secret review corpus: 3 false positives, 16 misses (the first detector: 139 and 62). Shared vectors complete at version 3 (192 cases). A block of key lines is followed to its end up to 1 MiB.
 - Internal: `docs/privacy-filter-cases.json`, the shared test vectors of the privacy filter (35 cases: every kind and 23 inputs that must stay untouched; inputs are split into fragments so no complete token appears, and the file is excluded from Cora review in `.cora.yaml` because its secret-shaped assignments are the point of the test). `uteke-core` pins every case, and uteke-cloud pins parity against the same file (uteke-cloud#218). Part 2a of #1464.
 - Internal: `memories.author`, part 1 of #1427 phase 2 (schema v22): an optional, free-text author on the memory itself, usable without a room. The first open by a build with this change migrates the store in place (additive `ALTER TABLE`; a memory that belongs to exactly one room takes that room link's author, the rest stay unset). `Memory` gains `author` (omitted from JSON when unset) and `Uteke::set_author` validates it (trimmed, 1 to 128 characters, no control characters). No HTTP, MCP or CLI input accepts it yet: that follows once uteke-cloud#219 is answered.

crates/uteke-cli/src/commands/remember.rs

@@ -66,7 +66,12 @@ pub(crate) fn run(
     source_type: Option<&str>,
     timestamp: Option<&str>,
 ) -> Result<(), String> {
-    tracing::debug!("Remembering: {content} (type: {type}, contradiction: {detect_contradiction})");
+    // The content is not logged: it may hold a secret the privacy filter has
+    // not masked yet (#1464).
+    tracing::debug!(
+        "Remembering {} bytes (type: {type}, contradiction: {detect_contradiction})",
+        content.len()
+    );
 
     // #1106: validate BEFORE any write so an invalid value fails loudly
     // without leaving a stored row behind (the post-store set would have

crates/uteke-core/src/consolidate.rs

@@ -72,6 +72,10 @@ impl crate::Uteke {
     ) -> Result<(RememberOutcome, ContradictionResult), Error> {
         crate::validate_input(content, tags)?;
         let ns = namespace.unwrap_or(DEFAULT_NAMESPACE);
+        // Privacy filter (#1464), before the content is embedded or stored.
+        let prepared = self.privacy_config.prepare(content, metadata)?;
+        let content = prepared.content.as_str();
+        let metadata = prepared.metadata.clone();
 
         // Embed first to check for contradictions before persisting
         self.ensure_embedder()?;
@@ -98,7 +102,7 @@ impl crate::Uteke {
         let content_type = crate::memory::crud::detect_content_type(content);
 
         // Use remember_precomputed to avoid double-embedding
-        let outcome = self.remember_precomputed_detailed(
+        let mut outcome = self.remember_precomputed_detailed(
             content,
             tags,
             metadata,
@@ -109,6 +113,8 @@ impl crate::Uteke {
             None,
         )?;
         let id = outcome.id.clone();
+        self.record_redaction(&id, &prepared.redactions);
+        outcome.redactions = prepared.redactions.clone();
 
         // Timeline: if this is a contradiction-aware remember that
         // supersedes an older memory, record a Consolidated event (#347)

crates/uteke-core/src/import_export.rs

@@ -80,12 +80,32 @@ impl crate::Uteke {
         let mut imported = 0;
         let mut deduped = 0;
 
-        for entry in entries {
+        for mut entry in entries {
             if entry.content.is_empty() {
                 skipped += 1;
                 continue;
             }
 
+            // Privacy filter (#1464), before the content is embedded or stored.
+            // Strict mode skips the row (counted as skipped) instead of
+            // aborting the whole import.
+            let prepared = match self
+                .privacy_config
+                .prepare(&entry.content, Some(std::mem::take(&mut entry.metadata)))
+            {
+                Ok(p) => p,
+                Err(e) => {
+                    tracing::warn!("Skipping import entry: {e}");
+                    skipped += 1;
+                    continue;
+                }
+            };
+            entry.content = prepared.content;
+            if let Some(m) = prepared.metadata {
+                entry.metadata = m;
+            }
+            let redactions = prepared.redactions;
+
             // Re-embed the content with retry (consistent with remember path, #1005).
             self.ensure_embedder()?;
             let embedding = crate::operations::retry_embed(&self.embedder, &entry.content)?;
@@ -165,6 +185,8 @@ impl crate::Uteke {
                 continue;
             }
 
+            self.record_redaction(&id, &redactions);
+
             // Auto-link cosine edges (consistent with remember path, #1005).
             // Must run AFTER index.insert() so the new memory is searchable.
             self.auto_link_cosine(&id, &embedding, Some(memory.namespace.as_str()));

crates/uteke-core/src/lib.rs

@@ -36,6 +36,8 @@ pub mod offline_extraction;
 mod operations;
 pub mod pack_mode;
 pub mod privacy;
+#[cfg(test)]
+mod privacy_write_tests;
 
 pub use forget_request::{
     ForgetErrorKind, ForgetInput, ForgetPolicy, ForgetRequest, ForgetRequestError, FullIdForm,
@@ -645,6 +647,9 @@ pub struct Uteke {
     /// Dream pipeline thresholds (#731). Used by contradiction scan,
     /// dedup/consolidate, and orphan detection phases.
     dream_config: DreamConfig,
+    /// Privacy filter applied to content before it is embedded or stored
+    /// (#1464). Default: mask secrets (`redact`), no allowlist.
+    privacy_config: privacy::PrivacyConfig,
     /// Memory lifecycle configuration (#928). Controls soft-delete behavior,
     /// dynamic deprecation caps, and TTL-based pruning.
     lifecycle_config: LifecycleConfig,
@@ -1050,6 +1055,7 @@ impl Uteke {
             recall_cache: recall_cache::RecallCache::new(recall_cache::RecallCacheConfig::default()),
             jaccard_weight: 0.0,
             dream_config: DreamConfig::default(),
+            privacy_config: privacy::PrivacyConfig::default(),
             lifecycle_config: LifecycleConfig::default(),
             embed_cache_path,
         })
@@ -1093,6 +1099,18 @@ impl Uteke {
         self.dream_config = config;
     }
 
+    /// Set the privacy filter (#1464): `redact` (default), `strict` or `off`,
+    /// plus the allowlist of values to keep. Called once at startup from the
+    /// CLI and server config.
+    pub fn set_privacy_config(&mut self, config: privacy::PrivacyConfig) {
+        self.privacy_config = config;
+    }
+
+    /// The privacy filter in force.
+    pub fn privacy_config(&self) -> &privacy::PrivacyConfig {
+        &self.privacy_config
+    }
+
     /// Set memory lifecycle configuration (#928).
     ///
     /// Controls soft-delete behavior, dynamic deprecation caps, and

crates/uteke-core/src/operations.rs

@@ -85,6 +85,18 @@ pub struct RememberOutcome {
     pub embedding_written: bool,
     /// Failure reason when the embedding could not be persisted.
     pub warning: Option<String>,
+    /// Secrets the privacy filter masked in the content of THIS call (#1464): a
+    /// kind and a count each, never a value. Empty when nothing was masked. On a
+    /// dedup hit nothing new is stored, so the stored memory keeps its own flags.
+    #[serde(skip_serializing_if = "Vec::is_empty")]
+    pub redactions: Vec<crate::privacy::RedactionCount>,
+}
+
+impl RememberOutcome {
+    /// Whether the privacy filter masked anything in the stored content.
+    pub fn redacted(&self) -> bool {
+        !self.redactions.is_empty()
+    }
 }
 
 impl crate::Uteke {
@@ -233,6 +245,20 @@ impl crate::Uteke {
         self.remember_embed_detailed(content, tags, metadata, namespace, &effective_type)
     }
 
+    /// Audit entry for a write whose content the privacy filter changed
+    /// (#1464): kinds and counts, never a value. Best-effort like every
+    /// timeline event.
+    pub(crate) fn record_redaction(&self, id: &str, redactions: &[crate::privacy::RedactionCount]) {
+        if redactions.is_empty() {
+            return;
+        }
+        self.try_timeline_event(
+            id,
+            crate::timeline::TimelineEventType::Redacted,
+            Some(&serde_json::json!({ "redactions": redactions })),
+        );
+    }
+
     /// Embed-then-store with honest embedding reporting (#1273).
     fn remember_embed_detailed(
         &self,
@@ -249,6 +275,11 @@ impl crate::Uteke {
                 "Unknown memory type '{memory_type}'. Valid types: fact, procedure, preference, decision, context, note, insight, reference, event"
             ))
         })?;
+        // Privacy filter (#1464): before the content is embedded, indexed or
+        // stored, so a secret never reaches the vector, FTS5 or the row.
+        let prepared = self.privacy_config.prepare(content, metadata)?;
+        let content = prepared.content.as_str();
+        let metadata = prepared.metadata.clone();
         // Detect JSON content and use flattened text for embedding
         let content_type = crate::memory::crud::detect_content_type(content);
         let embed_text = if content_type == "json" {
@@ -300,11 +331,12 @@ impl crate::Uteke {
                     id: existing_id,
                     embedding_written: true,
                     warning: None,
+                    redactions: prepared.redactions.clone(),
                 });
             }
         }
 
-        self.remember_precomputed_detailed(
+        let mut outcome = self.remember_precomputed_detailed(
             content,
             tags,
             metadata,
@@ -313,7 +345,10 @@ impl crate::Uteke {
             content_type,
             &embedding,
             generation_error,
-        )
+        )?;
+        self.record_redaction(&outcome.id, &prepared.redactions);
+        outcome.redactions = prepared.redactions;
+        Ok(outcome)
     }
 
     /// Store a new memory with a pre-computed embedding.
@@ -551,6 +586,7 @@ impl crate::Uteke {
             embedding_written,
             warning: vector_error,
             id,
+            redactions: Vec::new(),
         })
     }
 
@@ -1869,6 +1905,56 @@ impl crate::Uteke {
             }
         }
 
+        // Privacy filter (#1464) on the new content, before it is embedded or
+        // stored. A masked update also records `redacted` in the metadata:
+        // the caller's metadata when given, else the memory's current one.
+        let prepared = match content {
+            Some(c) => Some(self.privacy_config.prepare(c, None)?),
+            None => None,
+        };
+        let redactions = prepared
+            .as_ref()
+            .map(|p| p.redactions.clone())
+            .unwrap_or_default();
+        let masked_content: Option<String> = prepared.map(|p| p.content);
+        let content: Option<&str> = masked_content.as_deref();
+        let flagged_metadata: Option<serde_json::Value> = if redactions.is_empty() {
+            match (content, metadata) {
+                // Clean new content and no metadata from the caller: the flags
+                // a previous version left no longer describe the content,
+                // unless the edit kept a mask marker (a read-edit-write round
+                // trip of masked text).
+                (Some(c), None) if !c.contains("[REDACTED:") => {
+                    crate::privacy::without_redaction_flags(&existing.metadata)
+                }
+                // A metadata-only update keeps the masked content, so it keeps
+                // the flags that describe it.
+                (None, Some(m)) => crate::privacy::carry_redaction_flags(m, &existing.metadata),
+                // New content and new metadata: the flags follow the content,
+                // never what the caller's metadata says about itself.
+                (Some(c), Some(m)) if c.contains("[REDACTED:") => {
+                    let own = crate::privacy::without_redaction_flags(m);
+                    crate::privacy::carry_redaction_flags(
+                        own.as_ref().unwrap_or(m),
+                        &existing.metadata,
+                    )
+                    .or(own)
+                }
+                (Some(_), Some(m)) => crate::privacy::without_redaction_flags(m),
+                _ => None,
+            }
+        } else {
+            crate::privacy::with_redaction_flags(
+                Some(
+                    metadata
+                        .cloned()
+                        .unwrap_or_else(|| existing.metadata.clone()),
+                ),
+                &redactions,
+            )
+        };
+        let metadata: Option<&serde_json::Value> = flagged_metadata.as_ref().or(metadata);
+
         let now = chrono::Utc::now();
 
         // Re-embed if content changed
@@ -1932,6 +2018,8 @@ impl crate::Uteke {
             }
         }
 
+        self.record_redaction(id, &redactions);
+
         // Invalidate recall cache for the memory's namespace
         self.recall_cache.invalidate_namespace(&existing.namespace);