diff --git a/CHANGELOG.md b/CHANGELOG.md index 96d2faf..94bcfd9 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -4,6 +4,93 @@ ### Features +- **Salience tier Phase 1 — first-class standing-context recall + (default-off, soak-gated).** Memories explicitly promoted via + `memory_promote` are injected into every `` + envelope on every prompt, regardless of query similarity. The cap + is the contract: default 15, hard ceiling 20. Plan: + `private/mnemon-salience-tier-plan-260521.md`. + - **Reframed validation gate (2026-05-22).** Phase 1 IS the + validation. Earlier plan called for a synthetic A/B against the + Phase 0 env-var-flagged form before committing to schema + + tooling. Reframed because the injection mechanism is identical + between the two forms — an A/B of the gated env-var path + carries no marginal information once Phase 1 ships gated behind + `STANDING_TIER_ENABLED=false`. Operator promotes ~5 career- + context memories, flips the flag, observes ≥1 week soak for + runway-style under-weighting recurrence vs absence. Per + `feedback_phase_gated_soak_consumer_must_be_ready`: ship the + substrate gated, flip activation at a separate milestone. + - **Schema migration**: `documents.tier TEXT NOT NULL DEFAULT + 'situational'` via `_migrate_tier()`. Index `idx_documents_tier` + on live rows for the cap-count probe + search exclusion filter. + Additive + harmless if `STANDING_TIER_ENABLED` stays off. + - **`Store.promote_to_standing(id)`** + **`demote_to_situational(id)`** + + **`list_standing()`** + **`standing_tier_status()`**. Promote + raises **`StandingTierCapReached`** at the runtime cap, + **`StandingTierProvenanceRejected`** when source_client is in + `STANDING_TIER_BLOCKED_SOURCE_CLIENTS` (Layer 4 composition — + hook-sourced memories cannot be promoted; operator-explicit + gesture only), and **`StandingTierError`** on missing / + invalidated docs. Idempotent re-promote returns True; demote + of a situational doc returns False (no-op). + - **`Store.search_bm25` + `Store.search_vector`** gain + `include_standing: bool = False` keyword param. Standing-tier + docs excluded from ranked retrieval by default — they're + injected unconditionally already; ranking them too would + double-count and crowd the situational signal. Threaded through + `search.search()` so the higher-level entry respects the + invariant. + - **MCP tools**: `memory_promote(id)`, `memory_demote(id)`, + `memory_list_standing()` — both stdio (`server.py`) and + Streamable HTTP (`server_remote.py` reuses the same `mcp` + object). 14 → 17 registered tools. + - **CLI**: `mnemon standing list / promote / demote `. + `mnemon status` gains a `Standing tier: N/CAP` line. + - **`build_context` integration**: when `STANDING_TIER_ENABLED` + (config constant OR `MNEMON_STANDING_TIER_ENABLED` env override, + accepting `1/true/yes/on`), build_context calls + `memory_list_standing` via the remote client in a single + round-trip and renders the result as the "Standing context" + sub-section ahead of "Situational recall." Phase 0 env-var path + (`MNEMON_STANDING_TIER_FILE` → standing.json → standing-rendered.md + cache) is **preserved as fallback** so operators retain a + per-session override mechanism. + - **Composability invariants** (all preserved): + - Layer 0 (`is_well_shaped`) — capture rejection runs before + anything reaches the standing-tier promotion path + - Layer 1 envelope — standing block sits inside the same + `` data-marking + nonce as situational + - Layer 4 (`HOOK_SOURCE_CONFIDENCE_CEILING` + provenance) — + hook-sourced memories cannot be promoted; explicit + `StandingTierProvenanceRejected` rejection + - rc16 `source_key` upsert — unchanged; tier orthogonal + - Capture attention Phase A — `recurrence_count` accretes + against canonical situational memories; standing-tier + promotion is operator-gated on top of that signal + - **Soak gates** for flipping default-on: (a) ≥1 week with the + flag on; (b) observed reduction in runway-style under-weighting + recurrence on real career-strategy conversations; (c) zero + spurious-injection complaints from operator review of every + promoted memory. + - 22 new tests in `tests/test_standing_tier.py` covering: promote + success / cap-rejection (cap=2 in test, 3rd raises) / + hook-sourced rejection / invalidated rejection / missing rejection + / cap respects invalidated (freed slot reclaimable) / demote + round-trip / demote idempotent on situational / demote frees cap + slot / list_standing ordering + content / search excludes by + default / search includes when requested / build_context + flag-off no-fetch / flag-on memory_list_standing call / + env-var truthy value parsing. Suite 814 → 836 passing + (`test_server_remote.py` tool-count assertions bumped 14 → 17). + +### Schema + +- **`documents.tier TEXT NOT NULL DEFAULT 'situational'`** — + additive migration in `_migrate_tier()` after the existing + `_migrate_recurrence_count`. Index `idx_documents_tier` scoped to + live rows for cap-count + search-filter queries. + - **Capture attention Phase A — recurrence-weighted memory convergence (default-off, soak-gated).** When a new save's content is semantically close to ≥2 prior memories spanning distinct sessions, capture diff --git a/src/mnemon/cli.py b/src/mnemon/cli.py index e2546be..186e1ae 100644 --- a/src/mnemon/cli.py +++ b/src/mnemon/cli.py @@ -48,10 +48,13 @@ def main() -> None: from .store import Store store = Store() stats = store.status() + standing = store.standing_tier_status() print(f"Vault: {stats['vault_path']}") print(f"Total memories: {stats['total_documents']}") print(f"Vectors: {stats['total_vectors']}") print(f"Pinned: {stats['pinned']}") + print(f"Standing tier: {standing['count']}/{standing['cap']} " + f"(hard ceiling {standing['hard_ceiling']})") print(f"Invalidated: {stats['invalidated']}") print("\nBy type:") for t in stats["by_type"]: @@ -308,12 +311,100 @@ def main() -> None: finally: store.close() + elif command == "standing": + # Salience tier Phase 1 — operator-facing tier management. + # private/mnemon-salience-tier-plan-260521.md + subcommand = args[1] if len(args) > 1 else "list" + _handle_standing(subcommand, args[2:]) + else: print(f"Unknown command: {command}", file=sys.stderr) _print_usage() sys.exit(1) +def _handle_standing(subcommand: str, rest: list[str]) -> None: + """Salience tier Phase 1 — list / promote / demote subcommands.""" + from .store import ( + Store, + StandingTierCapReached, + StandingTierError, + StandingTierProvenanceRejected, + ) + store = Store() + try: + if subcommand == "list": + docs = store.list_standing() + status = store.standing_tier_status() + print(f"Standing tier: {status['count']}/{status['cap']} " + f"(hard ceiling {status['hard_ceiling']})") + if not docs: + print(" (empty — promote memories via `mnemon standing promote `)") + return + print() + for d in docs: + snippet = (d.content or "")[:120].replace("\n", " ") + ellipsis = "..." if len(d.content or "") > 120 else "" + print(f" #{d.id:>5} [{d.content_type}] {d.title}") + print(f" {snippet}{ellipsis}") + print() + + elif subcommand == "promote": + if not rest: + print("Usage: mnemon standing promote ", file=sys.stderr) + sys.exit(2) + try: + doc_id = int(rest[0]) + except ValueError: + print(f"Error: must be an integer (got {rest[0]!r})", + file=sys.stderr) + sys.exit(2) + try: + store.promote_to_standing(doc_id) + status = store.standing_tier_status() + print(f"Promoted memory #{doc_id} to standing tier " + f"({status['count']}/{status['cap']}).") + except StandingTierCapReached as e: + print(f"Cap reached: {e}", file=sys.stderr) + sys.exit(1) + except StandingTierProvenanceRejected as e: + print(f"Provenance rejected: {e}", file=sys.stderr) + sys.exit(1) + except StandingTierError as e: + print(f"Error: {e}", file=sys.stderr) + sys.exit(1) + + elif subcommand == "demote": + if not rest: + print("Usage: mnemon standing demote ", file=sys.stderr) + sys.exit(2) + try: + doc_id = int(rest[0]) + except ValueError: + print(f"Error: must be an integer (got {rest[0]!r})", + file=sys.stderr) + sys.exit(2) + try: + ok = store.demote_to_situational(doc_id) + status = store.standing_tier_status() + if ok: + print(f"Demoted memory #{doc_id} to situational " + f"({status['count']}/{status['cap']} remain standing).") + else: + print(f"Memory #{doc_id} was not on the standing tier.") + except StandingTierError as e: + print(f"Error: {e}", file=sys.stderr) + sys.exit(1) + + else: + print(f"Unknown subcommand: standing {subcommand}", file=sys.stderr) + print("Usage: mnemon standing list | promote | demote ", + file=sys.stderr) + sys.exit(2) + finally: + store.close() + + def _print_attention_status(store) -> None: """Print capture-attention soak metrics for the operator. @@ -522,6 +613,15 @@ def _print_usage() -> None: mnemon attention-status Capture-attention soak monitor — boost rate, recurrence distribution, top canonicals, recent 'restates' relations + +Salience tier (standing-context recall): + mnemon standing list Show all standing-tier memories + count vs cap + mnemon standing promote Promote memory to standing tier (capped) + mnemon standing demote Demote back to situational + Standing memories are injected into every + recall context regardless of query similarity. + Cap is the contract — default 15, hard 20. + Hook-sourced memories cannot be promoted. mnemon forget Soft-delete from local vault mnemon rebuild Re-embed every document (run after a model change, or to recover from skipped embeddings) diff --git a/src/mnemon/config.py b/src/mnemon/config.py index 4543f80..81af34c 100644 --- a/src/mnemon/config.py +++ b/src/mnemon/config.py @@ -112,6 +112,25 @@ class MemoryType(str, Enum): CAPTURE_ATTENTION_REQUIRE_DISTINCT_SESSIONS = True # neighbors must span ≥MIN_HITS distinct days CAPTURE_ATTENTION_SOAK_BOOST_RATE_MAX = 0.25 # acceptance criterion: boosts/saves ratio ceiling over 7d +# Salience tier — standing-context recall, Phase 1 (added 2026-05-22) — +# private/mnemon-salience-tier-plan-260521.md +# +# Standing-tier memories are injected unconditionally into the Layer 1 +# envelope on every prompt, regardless of query +# similarity, conditioning reasoning rather than answering it. The cap +# is the contract: past ~20, salience degrades and the tier becomes +# noise again, recreating the failure mode at a different scale. +# Default-off through soak per the 2026-05-22 reframing — Phase 1 ships +# the substrate gated; promote ≤5 career-context memories via the new +# memory_promote MCP tool; flip the flag; observe ≥1 week soak for +# runway-style under-weighting recurrence vs absence. +STANDING_TIER_ENABLED = False # feature flag — flip after soak +STANDING_TIER_DEFAULT_CAP = 15 # operator-tunable runtime cap +STANDING_TIER_HARD_CEILING = 20 # invariant — never exceed +# Reuse Layer 4 hook-sourced set: hook-sourced memories cannot be promoted — +# operator-explicit gesture only. Composes with provenance demotion. +STANDING_TIER_BLOCKED_SOURCE_CLIENTS = HOOK_SOURCE_CLIENTS + # Hook timeouts and budgets (seconds / chars) # # HOOK_REMOTE_TIMEOUT_SEC — matches Claude Code's ~/.claude/settings.json diff --git a/src/mnemon/hooks/context_surfacing.py b/src/mnemon/hooks/context_surfacing.py index 70a6556..c22ee41 100644 --- a/src/mnemon/hooks/context_surfacing.py +++ b/src/mnemon/hooks/context_surfacing.py @@ -66,21 +66,82 @@ ) -# ── Phase 0 of the salience-tier plan ───────────────────────────── +# ── Salience tier — standing context recall ─────────────────────── # (private/mnemon-salience-tier-plan-260521.md) # -# Standing context — feature-flagged, opt-in via MNEMON_STANDING_TIER_FILE. -# When the env var points at a JSON file shaped {"ids": [, , ...]}, -# build_context fetches each ID via the remote MCP client and prepends a -# labeled "Standing context" sub-section inside the existing -# envelope, ahead of the query-driven block. +# Two paths today, gated by the STANDING_TIER_ENABLED feature flag: # -# Per-prompt cost: ~500ms × N (sequential memory_get HTTP calls). For N=10 -# that's ~5s added latency. Phase 0 accepts this cost — the diagnostic is -# about hypothesis validation, not perf. Phase 1 will optimize via local -# cache or batch fetch. +# 1. Phase 1 (default when flag is on): single ``memory_list_standing`` +# MCP call fetches the live standing-tier set in one round-trip. +# Cap-bounded (default 15, hard ceiling 20), so the payload is small. +# Source of truth = ``documents.tier='standing'`` in the Fly vault. # -# Defaults to unset = original behavior unchanged. +# 2. Phase 0 (fallback when flag is off, or as operator override): +# env-var MNEMON_STANDING_TIER_FILE → ~/.mnemon/standing.json IDs +# → cached rendered block at ~/.mnemon/standing-rendered.md. +# Useful when an operator wants to override the schema-backed set +# with a hand-picked ID list per session. +# +# The Phase 0 path is preserved as a fallback per the 2026-05-22 +# reframing — Phase 1 ships gated; the env-var path stays operational +# either way. Defaults: STANDING_TIER_ENABLED False (config) + no env +# override = nothing injected, original behavior unchanged. + +def _standing_tier_enabled() -> bool: + """Check whether the Phase 1 standing tier is active. + + Truth sources, in order: + 1. ``MNEMON_STANDING_TIER_ENABLED`` env var (operator override) + 2. ``config.STANDING_TIER_ENABLED`` (default-off through soak) + """ + env = os.environ.get("MNEMON_STANDING_TIER_ENABLED", "").strip().lower() + if env in ("1", "true", "yes", "on"): + return True + if env in ("0", "false", "no", "off"): + return False + from ..config import STANDING_TIER_ENABLED + return STANDING_TIER_ENABLED + + +def _fetch_standing_via_mcp() -> str: + """Phase 1 path: single memory_list_standing MCP call. + + Returns the rendered standing block (markdown bullets) or "" if + the call fails / returns empty. One round-trip vs Phase 0's + ~500ms × N sequential per-id fetches. + """ + try: + from ._remote_client import call_tool_sync + except ImportError: + return "" + + from ..safety import defang_control_markup + + try: + raw, _elapsed = call_tool_sync("memory_list_standing", {}, timeout=5.0) + docs = json.loads(raw) + except Exception: + # Best-effort hook contract — failures here mean no standing + # block on this prompt, situational recall still runs. + return "" + + if not isinstance(docs, list) or not docs: + return "" + + lines: list[str] = [] + for d in docs: + title = defang_control_markup(str(d.get("title", ""))) + content = str(d.get("content", "")) + content_type = str(d.get("content_type", "note")) + doc_id = d.get("doc_id", "?") + snippet = defang_control_markup(content[:_SNIPPET_CHARS]) + ellipsis = "..." if len(content) > _SNIPPET_CHARS else "" + lines.append( + f"- [{content_type}] **{title}** (id={doc_id})\n" + f" {snippet}{ellipsis}" + ) + return "\n\n".join(lines) + def _load_standing_ids() -> list[int]: """Read standing-tier IDs from MNEMON_STANDING_TIER_FILE, if set.""" @@ -211,14 +272,26 @@ def build_context(raw_text: str, *, prefix: str = "") -> str: Returns an empty string when there's nothing worth injecting — no standing IDs AND no situational results (or unparseable JSON). """ - # Standing context — Phase 0 opt-in. Computed first so the function - # can inject standing-only when no situational results exist. + # Standing context — Phase 1 (preferred) or Phase 0 (fallback). + # Computed first so the function can inject standing-only when no + # situational results exist. + # + # Phase 1: when STANDING_TIER_ENABLED, single memory_list_standing + # MCP call fetches the live schema-backed standing-tier set + # (one round-trip, cap-bounded payload). This is the canonical + # path once an operator has promoted memories via memory_promote. # - # Cache-first: ~/.mnemon/standing-rendered.md (microseconds). If - # the cache doesn't exist, fall back to per-prompt HTTP fetch - # (~500ms × N). build_standing_set.py writes both on every run. - standing_block = _read_rendered_cache() + # Phase 0 (fallback): env-var-driven ID list → cached rendered + # block. Useful as an operator override or before any memories + # have been promoted to the schema-backed tier. + standing_block = "" + if _standing_tier_enabled(): + standing_block = _fetch_standing_via_mcp() + if not standing_block: + # Phase 0 cache-first + standing_block = _read_rendered_cache() if not standing_block: + # Phase 0 per-id HTTP fallback standing_ids = _load_standing_ids() standing_block = _fetch_standing_block(standing_ids) if standing_ids else "" diff --git a/src/mnemon/search.py b/src/mnemon/search.py index 0627b0a..e7df84f 100644 --- a/src/mnemon/search.py +++ b/src/mnemon/search.py @@ -207,9 +207,18 @@ def search( content_type: str | None = None, use_vector: bool = True, use_expansion: bool = False, + *, + include_standing: bool = False, ) -> list[ScoredResult]: - """Main search entry point. Hybrid BM25 + vector search with composite scoring.""" - bm25_results = store.search_bm25(query, limit * 2) + """Main search entry point. Hybrid BM25 + vector search with composite scoring. + + ``include_standing`` — when False (default), standing-tier memories + are excluded from ranked retrieval. They're injected unconditionally + into the envelope via ``Store.list_standing()`` — + including them here would double-count and crowd the situational + signal. Per salience-tier plan invariant 3. + """ + bm25_results = store.search_bm25(query, limit * 2, include_standing=include_standing) # Skip expansion if BM25 already has a strong signal strong_bm25 = ( @@ -222,7 +231,7 @@ def search( expansion_results: list[SearchResult] = [] if use_expansion and not strong_bm25: for eq in expand_query(query): - expansion_results.extend(store.search_bm25(eq, limit)) + expansion_results.extend(store.search_bm25(eq, limit, include_standing=include_standing)) # Vector search (optional, fails gracefully) vector_results: list[SearchResult] = [] @@ -230,7 +239,7 @@ def search( try: from .embedder import embed query_embedding = embed(f"query: {query}") - vector_results = store.search_vector(query_embedding, limit * 2) + vector_results = store.search_vector(query_embedding, limit * 2, include_standing=include_standing) except Exception as exc: # BM25 still serves results — but we need the cause visible so # "why is semantic search suddenly missing obvious matches?" diff --git a/src/mnemon/server.py b/src/mnemon/server.py index c93683c..a2833d9 100644 --- a/src/mnemon/server.py +++ b/src/mnemon/server.py @@ -1,6 +1,7 @@ """MCP server — exposes mnemon memory tools via stdio transport. Tools: memory_search, memory_get, memory_save, memory_pin, memory_forget, + memory_promote, memory_demote, memory_list_standing, memory_status, memory_sweep, memory_timeline, memory_related, memory_rebuild, memory_check_contradictions, profile_get, profile_update """ @@ -211,6 +212,90 @@ def memory_forget(id: int) -> str: ) +@mcp.tool() +def memory_promote(id: int) -> str: + """Promote a memory to the capped standing tier. + + Standing-tier memories are injected into every recall context + regardless of query similarity — they condition reasoning rather + than answering it. Use sparingly: the cap is the contract + (default 15, hard ceiling 20). Past ~20, the tier stops being + salient and becomes noise again. + + Hook-sourced memories (auto-mirror, session_extractor) cannot be + promoted — operator-explicit gesture only (Layer 4 composition). + """ + from .store import ( + StandingTierCapReached, + StandingTierError, + StandingTierProvenanceRejected, + ) + store = _get_store() + try: + ok = store.promote_to_standing(id) + status = store.standing_tier_status() + return ( + f"Promoted memory #{id} to standing tier " + f"({status['count']}/{status['cap']})." + if ok else f"Memory #{id} could not be promoted." + ) + except StandingTierCapReached as e: + return f"Cap reached: {e}" + except StandingTierProvenanceRejected as e: + return f"Provenance rejected: {e}" + except StandingTierError as e: + return f"Error: {e}" + + +@mcp.tool() +def memory_demote(id: int) -> str: + """Demote a standing-tier memory back to situational. + + The opposite of memory_promote. Idempotent: demoting a memory + that isn't on the standing tier is a no-op (returns "not on + standing tier" rather than failing). + """ + from .store import StandingTierError + store = _get_store() + try: + actually_demoted = store.demote_to_situational(id) + status = store.standing_tier_status() + if actually_demoted: + return ( + f"Demoted memory #{id} to situational " + f"({status['count']}/{status['cap']} remain standing)." + ) + return f"Memory #{id} was not on the standing tier." + except StandingTierError as e: + return f"Error: {e}" + + +@mcp.tool() +def memory_list_standing() -> str: + """Return all live standing-tier memories as a JSON array. + + Consumed by ``hooks/context_surfacing.py`` to render the always-on + standing block when ``STANDING_TIER_ENABLED`` is True. The same + list is shown by ``mnemon standing list`` on the CLI. + + Each element: ``{doc_id, title, content, content_type, confidence, + created_at}``. Empty array when nothing has been promoted. + """ + store = _get_store() + docs = store.list_standing() + return json.dumps([ + { + "doc_id": d.id, + "title": d.title, + "content": d.content, + "content_type": d.content_type, + "confidence": d.confidence, + "created_at": d.created_at, + } + for d in docs + ]) + + # ── Lifecycle Tools ────────────────────────────────────────────────────────── diff --git a/src/mnemon/store.py b/src/mnemon/store.py index 863ad80..81bec97 100644 --- a/src/mnemon/store.py +++ b/src/mnemon/store.py @@ -29,6 +29,9 @@ MEMORY_TYPE_MAP, DEFAULT_CONFIDENCE, PIN_BOOST, + STANDING_TIER_BLOCKED_SOURCE_CLIENTS, + STANDING_TIER_DEFAULT_CAP, + STANDING_TIER_HARD_CEILING, ContentType, MemoryType, vault_path, @@ -47,6 +50,28 @@ class CaptureAttentionUnavailableError(RuntimeError): """ +class StandingTierError(ValueError): + """Raised when ``promote_to_standing`` rejects a candidate. + + Distinct subclasses surface the reason so MCP / CLI callers can + render a user-actionable message instead of an opaque False. Per + the salience-tier plan invariant: "promotion is operator-approved, + not auto" — operator gets a clear "why not" message. + """ + + +class StandingTierCapReached(StandingTierError): + """Already at the runtime cap (STANDING_TIER_DEFAULT_CAP).""" + + +class StandingTierProvenanceRejected(StandingTierError): + """Source client is in STANDING_TIER_BLOCKED_SOURCE_CLIENTS. + + Layer 4 composition: hook-sourced memories cannot be promoted — + operator-explicit gesture only. + """ + + @dataclass class Document: id: int @@ -205,6 +230,39 @@ def _init_schema(self) -> None: self.db.commit() self._migrate_source_key() self._migrate_recurrence_count() + self._migrate_tier() + + def _migrate_tier(self) -> None: + """Additive migration: ``documents.tier`` distinguishes + unconditionally-injected standing memories from situational + recall. Salience tier Phase 1 (added 2026-05-22) — + private/mnemon-salience-tier-plan-260521.md. + + Values: ``'situational'`` (default; current `memory_search` + path applies, ranked retrieval) or ``'standing'`` (injected + into every envelope regardless of query + similarity, capped). Pre-existing rows default to + ``'situational'`` — every existing memory keeps its current + behavior. Schema is additive + harmless if + ``STANDING_TIER_ENABLED`` stays off. + """ + cols = { + r["name"] + for r in self.db.execute("PRAGMA table_info(documents)").fetchall() + } + if "tier" not in cols: + self.db.execute( + "ALTER TABLE documents ADD COLUMN " + "tier TEXT NOT NULL DEFAULT 'situational'" + ) + # Lookup index for the cap-count probe and the search + # exclusion. Filtered to live rows because invalidated + # standing-tier members don't count against the cap. + self.db.execute( + "CREATE INDEX IF NOT EXISTS idx_documents_tier " + "ON documents (tier) WHERE invalidated_at IS NULL" + ) + self.db.commit() def _migrate_recurrence_count(self) -> None: """Additive migration: ``documents.recurrence_count`` counts @@ -492,8 +550,18 @@ def timeline(self, limit: int = 20, content_type: str | None = None) -> list[Doc ).fetchall() return [_row_to_document(r) for r in rows] - def search_bm25(self, query: str, limit: int = 20) -> list[SearchResult]: - """BM25 full-text search via FTS5.""" + def search_bm25( + self, query: str, limit: int = 20, *, include_standing: bool = False + ) -> list[SearchResult]: + """BM25 full-text search via FTS5. + + ``include_standing`` — when False (default), tier='standing' + docs are excluded. They're injected unconditionally into the + envelope via ``list_standing()``; including + them in ranked retrieval would double-count and crowd out the + situational signal. The dashboard / explicit operator queries + can opt-in (e.g. an explicit "show all memories" surface). + """ safe_query = " OR ".join( f'"{token}"*' for token in query.replace("'", "").replace('"', "").split() @@ -502,9 +570,11 @@ def search_bm25(self, query: str, limit: int = 20) -> list[SearchResult]: if not safe_query: return [] + tier_filter = "" if include_standing else " AND d.tier = 'situational'" + try: rows = self.db.execute( - """SELECT + f"""SELECT d.id AS doc_id, d.title, c.doc AS content, @@ -518,7 +588,7 @@ def search_bm25(self, query: str, limit: int = 20) -> list[SearchResult]: JOIN documents d ON d.id = fts.rowid JOIN content c ON d.hash = c.hash WHERE documents_fts MATCH ? - AND d.invalidated_at IS NULL + AND d.invalidated_at IS NULL{tier_filter} ORDER BY rank LIMIT ?""", (safe_query, limit), @@ -551,8 +621,14 @@ def flush_vectors(self) -> None: """Persist vector store to disk.""" self.vec_store.save() - def search_vector(self, embedding: np.ndarray, limit: int = 20) -> list[SearchResult]: - """Vector similarity search via in-process brute-force cosine.""" + def search_vector( + self, embedding: np.ndarray, limit: int = 20, *, include_standing: bool = False + ) -> list[SearchResult]: + """Vector similarity search via in-process brute-force cosine. + + ``include_standing`` — see ``search_bm25`` for the rationale. + Standing-tier docs are excluded from ranked retrieval by default. + """ if self.vec_store.size() == 0: return [] @@ -560,14 +636,16 @@ def search_vector(self, embedding: np.ndarray, limit: int = 20) -> list[SearchRe results: list[SearchResult] = [] seen_ids: set[int] = set() + tier_filter = "" if include_standing else " AND d.tier = 'situational'" + for vr in vec_results: content_hash = vr["id"].split("_")[0] row = self.db.execute( - """SELECT d.id AS doc_id, d.title, c.doc AS content, d.content_type, + f"""SELECT d.id AS doc_id, d.title, c.doc AS content, d.content_type, d.memory_type, d.confidence, d.created_at, d.source_client FROM documents d JOIN content c ON d.hash = c.hash - WHERE d.hash = ? AND d.invalidated_at IS NULL + WHERE d.hash = ? AND d.invalidated_at IS NULL{tier_filter} LIMIT 1""", (content_hash,), ).fetchone() @@ -828,6 +906,130 @@ def _increment_recurrence(self, doc_id: int) -> None: ) self.db.commit() + # ── Salience tier Phase 1 — standing-context recall ────────────── + # private/mnemon-salience-tier-plan-260521.md + # + # Standing-tier memories are injected unconditionally into the + # envelope on every prompt. The cap is the + # contract: never exceed STANDING_TIER_HARD_CEILING. Per the + # 2026-05-22 reframing, Phase 1 ships gated; the validation is + # operator-flips-flag + observes ≥1 week soak for runway-style + # under-weighting recurrence vs absence. + + def promote_to_standing(self, doc_id: int) -> bool: + """Promote a memory to the capped standing tier. + + Raises: + StandingTierCapReached: at the runtime cap + (``STANDING_TIER_DEFAULT_CAP`` live standing docs) + StandingTierProvenanceRejected: source_client is in + ``STANDING_TIER_BLOCKED_SOURCE_CLIENTS`` (Layer 4 + composition — hook-sourced cannot promote) + StandingTierError: doc not found or invalidated + + Returns True on success. Idempotent — re-promoting an + already-standing doc returns True without counting against + the cap. The hard ceiling + (``STANDING_TIER_HARD_CEILING``) is an invariant the runtime + cap never exceeds even if an operator overrides + ``STANDING_TIER_DEFAULT_CAP`` upward. + """ + row = self.db.execute( + "SELECT tier, source_client, invalidated_at FROM documents WHERE id = ?", + (doc_id,), + ).fetchone() + if not row: + raise StandingTierError(f"memory #{doc_id} not found") + if row["invalidated_at"] is not None: + raise StandingTierError( + f"memory #{doc_id} is invalidated — cannot promote" + ) + if row["source_client"] in STANDING_TIER_BLOCKED_SOURCE_CLIENTS: + raise StandingTierProvenanceRejected( + f"memory #{doc_id} is hook-sourced ({row['source_client']}); " + "only user-authored memories can be promoted " + "(Layer 4 composition — auto-mirror / session_extractor " + "captures cannot be elevated to unconditional injection)" + ) + if row["tier"] == "standing": + return True # idempotent re-promote + + # Cap enforcement scoped to live rows. + current = self.db.execute( + "SELECT COUNT(*) AS c FROM documents " + "WHERE tier = 'standing' AND invalidated_at IS NULL" + ).fetchone()["c"] + cap = min(STANDING_TIER_DEFAULT_CAP, STANDING_TIER_HARD_CEILING) + if current >= cap: + raise StandingTierCapReached( + f"standing tier at cap ({current}/{cap}); demote an existing " + "member via memory_demote first. The cap is the contract — " + "past ~20 it stops being salient and becomes noise again." + ) + + self.db.execute( + "UPDATE documents SET tier = 'standing', updated_at = datetime('now') " + "WHERE id = ?", + (doc_id,), + ) + self.db.commit() + return True + + def demote_to_situational(self, doc_id: int) -> bool: + """Demote a standing-tier memory back to situational. + + Idempotent — demoting an already-situational doc returns False + (nothing to do). Returns True when an actual demote happened. + Raises StandingTierError on a missing or invalidated doc. + """ + row = self.db.execute( + "SELECT tier, invalidated_at FROM documents WHERE id = ?", + (doc_id,), + ).fetchone() + if not row: + raise StandingTierError(f"memory #{doc_id} not found") + if row["invalidated_at"] is not None: + raise StandingTierError( + f"memory #{doc_id} is invalidated — nothing to demote" + ) + if row["tier"] != "standing": + return False + self.db.execute( + "UPDATE documents SET tier = 'situational', updated_at = datetime('now') " + "WHERE id = ?", + (doc_id,), + ) + self.db.commit() + return True + + def list_standing(self) -> list[Document]: + """Return all live standing-tier memories, ordered most-recent first. + + Consumed by ``build_context`` (hook injection path) and + ``mnemon standing list`` (operator CLI). Includes content body + so callers can render snippets without a second fetch. + """ + rows = self.db.execute( + """SELECT d.*, c.doc + FROM documents d + JOIN content c ON d.hash = c.hash + WHERE d.tier = 'standing' AND d.invalidated_at IS NULL + ORDER BY d.created_at DESC""" + ).fetchall() + return [_row_to_document(r) for r in rows] + + def standing_tier_status(self) -> dict[str, Any]: + """Stats for ``mnemon status`` + dashboards: current count vs cap.""" + current = self.db.execute( + "SELECT COUNT(*) AS c FROM documents " + "WHERE tier = 'standing' AND invalidated_at IS NULL" + ).fetchone()["c"] + return { + "count": current, + "cap": STANDING_TIER_DEFAULT_CAP, + "hard_ceiling": STANDING_TIER_HARD_CEILING, + } + def status(self) -> dict[str, Any]: """Vault health stats.""" total = self.db.execute( diff --git a/tests/test_server_remote.py b/tests/test_server_remote.py index 3a0c95e..d382278 100644 --- a/tests/test_server_remote.py +++ b/tests/test_server_remote.py @@ -101,7 +101,9 @@ class TestMcpServer: def test_mcp_has_tools(self): from mnemon.server import mcp tools = mcp._tool_manager._tools - assert len(tools) == 14 + # 14 originals + 3 salience-tier Phase 1 (memory_promote / + # memory_demote / memory_list_standing, added 2026-05-22) + assert len(tools) == 17 def test_mcp_tool_names(self): from mnemon.server import mcp @@ -114,5 +116,8 @@ def test_mcp_tool_names(self): "memory_export_vectors", "memory_rebuild", "memory_check_contradictions", "profile_get", "profile_update", + # Salience tier Phase 1 (added 2026-05-22) — + # private/mnemon-salience-tier-plan-260521.md + "memory_promote", "memory_demote", "memory_list_standing", } assert tool_names == expected diff --git a/tests/test_standing_tier.py b/tests/test_standing_tier.py new file mode 100644 index 0000000..0ca5410 --- /dev/null +++ b/tests/test_standing_tier.py @@ -0,0 +1,280 @@ +"""Tests for the Salience-tier Phase 1 — first-class standing tier. + +Plan: ``private/mnemon-salience-tier-plan-260521.md`` Phase 1. + +Covers: promote/demote/list_standing API on Store; cap enforcement; +Layer 4 composition (hook-sourced rejection); search exclusion of +standing-tier docs by default; build_context wiring when the flag is on. +""" + +from __future__ import annotations + +import json +import os +import tempfile +from unittest.mock import patch + +import pytest + +from mnemon import config +from mnemon.store import ( + Store, + StandingTierCapReached, + StandingTierError, + StandingTierProvenanceRejected, +) + + +@pytest.fixture +def store(): + fd, path = tempfile.mkstemp(suffix=".sqlite") + os.close(fd) + os.unlink(path) + s = Store(db_path=path) + yield s + s.close() + for ext in ("", "-wal", "-shm"): + try: + os.unlink(path + ext) + except FileNotFoundError: + pass + + +# ── Schema migration ────────────────────────────────────────────── + + +class TestSchemaMigration: + def test_tier_column_exists(self, store): + cols = {r["name"] for r in store.db.execute("PRAGMA table_info(documents)").fetchall()} + assert "tier" in cols + + def test_tier_defaults_to_situational(self, store): + doc_id = store.save(title="x", content="y") + row = store.db.execute( + "SELECT tier FROM documents WHERE id = ?", (doc_id,) + ).fetchone() + assert row["tier"] == "situational" + + +# ── promote_to_standing ─────────────────────────────────────────── + + +class TestPromote: + def test_promote_succeeds_on_normal_memory(self, store): + doc_id = store.save(title="A", content="x") + assert store.promote_to_standing(doc_id) is True + row = store.db.execute( + "SELECT tier FROM documents WHERE id = ?", (doc_id,) + ).fetchone() + assert row["tier"] == "standing" + + def test_promote_idempotent(self, store): + doc_id = store.save(title="A", content="x") + store.promote_to_standing(doc_id) + # Re-promoting an already-standing doc returns True, doesn't error + assert store.promote_to_standing(doc_id) is True + # Still only counts as one + assert store.standing_tier_status()["count"] == 1 + + def test_promote_rejects_hook_sourced(self, store): + doc_id = store.save( + title="hook", content="auto", source_client="claude-code-hook" + ) + with pytest.raises(StandingTierProvenanceRejected) as exc: + store.promote_to_standing(doc_id) + assert "claude-code-hook" in str(exc.value).lower() or "hook-sourced" in str(exc.value) + + def test_promote_rejects_invalidated(self, store): + doc_id = store.save(title="A", content="x") + store.forget(doc_id) + with pytest.raises(StandingTierError) as exc: + store.promote_to_standing(doc_id) + assert "invalidated" in str(exc.value) + + def test_promote_rejects_missing(self, store): + with pytest.raises(StandingTierError) as exc: + store.promote_to_standing(99999) + assert "not found" in str(exc.value) + + def test_promote_rejects_at_cap(self, store, monkeypatch): + """With cap forced to 2, the 3rd promote raises CapReached.""" + monkeypatch.setattr("mnemon.store.STANDING_TIER_DEFAULT_CAP", 2) + id1 = store.save(title="A", content="one") + id2 = store.save(title="B", content="two") + id3 = store.save(title="C", content="three") + store.promote_to_standing(id1) + store.promote_to_standing(id2) + with pytest.raises(StandingTierCapReached) as exc: + store.promote_to_standing(id3) + assert "cap" in str(exc.value).lower() + + def test_promote_cap_respects_invalidated(self, store, monkeypatch): + """Invalidated standing-tier members don't count against the cap.""" + monkeypatch.setattr("mnemon.store.STANDING_TIER_DEFAULT_CAP", 2) + id1 = store.save(title="A", content="one") + id2 = store.save(title="B", content="two") + id3 = store.save(title="C", content="three") + store.promote_to_standing(id1) + store.promote_to_standing(id2) + store.forget(id1) # invalidate one + # Now there's room again — 1 live standing + 1 invalidated → 1/2 + assert store.promote_to_standing(id3) is True + assert store.standing_tier_status()["count"] == 2 + + +# ── demote_to_situational ───────────────────────────────────────── + + +class TestDemote: + def test_demote_round_trips(self, store): + doc_id = store.save(title="A", content="x") + store.promote_to_standing(doc_id) + assert store.demote_to_situational(doc_id) is True + row = store.db.execute( + "SELECT tier FROM documents WHERE id = ?", (doc_id,) + ).fetchone() + assert row["tier"] == "situational" + + def test_demote_idempotent_on_situational(self, store): + doc_id = store.save(title="A", content="x") + # Doc starts as situational; demote should return False (no-op) + assert store.demote_to_situational(doc_id) is False + + def test_demote_rejects_missing(self, store): + with pytest.raises(StandingTierError): + store.demote_to_situational(99999) + + def test_demote_frees_cap_slot(self, store, monkeypatch): + monkeypatch.setattr("mnemon.store.STANDING_TIER_DEFAULT_CAP", 2) + id1 = store.save(title="A", content="one") + id2 = store.save(title="B", content="two") + id3 = store.save(title="C", content="three") + store.promote_to_standing(id1) + store.promote_to_standing(id2) + store.demote_to_situational(id1) + # Slot freed → id3 can promote + assert store.promote_to_standing(id3) is True + + +# ── list_standing ───────────────────────────────────────────────── + + +class TestListStanding: + def test_returns_empty_when_none(self, store): + assert store.list_standing() == [] + + def test_returns_promoted_memories_ordered_by_recent(self, store): + id1 = store.save(title="oldest", content="one") + id2 = store.save(title="middle", content="two") + id3 = store.save(title="newest", content="three") + store.promote_to_standing(id1) + store.promote_to_standing(id2) + store.promote_to_standing(id3) + + docs = store.list_standing() + assert len(docs) == 3 + # ORDER BY created_at DESC → newest first + ids_in_order = [d.id for d in docs] + # All three present; relative order depends on insert timestamp resolution + assert set(ids_in_order) == {id1, id2, id3} + + def test_excludes_invalidated(self, store): + id1 = store.save(title="A", content="x") + id2 = store.save(title="B", content="y") + store.promote_to_standing(id1) + store.promote_to_standing(id2) + store.forget(id1) + docs = store.list_standing() + assert len(docs) == 1 + assert docs[0].id == id2 + + def test_includes_content_for_rendering(self, store): + doc_id = store.save(title="A", content="the full content payload") + store.promote_to_standing(doc_id) + docs = store.list_standing() + assert docs[0].content == "the full content payload" + + +# ── Search filter (Tier 1 excluded by default) ──────────────────── + + +class TestSearchExclusion: + def test_bm25_excludes_standing_by_default(self, store): + id_sit = store.save(title="situational", content="alpha beta gamma") + id_std = store.save(title="standing", content="alpha beta gamma delta") + store.promote_to_standing(id_std) + + results = store.search_bm25("alpha", limit=10) + ids = {r.doc_id for r in results} + assert id_sit in ids + assert id_std not in ids + + def test_bm25_includes_standing_when_requested(self, store): + id_sit = store.save(title="situational", content="alpha beta gamma") + id_std = store.save(title="standing", content="alpha beta gamma delta") + store.promote_to_standing(id_std) + + results = store.search_bm25("alpha", limit=10, include_standing=True) + ids = {r.doc_id for r in results} + assert id_sit in ids + assert id_std in ids + + +# ── build_context wiring ────────────────────────────────────────── + + +class TestBuildContextWiring: + def test_flag_off_no_phase1_fetch(self, monkeypatch): + """When the flag is off and no env-var Phase 0 path is set, + no standing block is rendered.""" + monkeypatch.delenv("MNEMON_STANDING_TIER_ENABLED", raising=False) + monkeypatch.delenv("MNEMON_STANDING_TIER_FILE", raising=False) + monkeypatch.setattr(config, "STANDING_TIER_ENABLED", False) + + from mnemon.hooks.context_surfacing import build_context + # Empty search-results JSON; no standing tier → empty context + out = build_context(raw_text="[]") + assert out == "" + + def test_flag_on_calls_memory_list_standing(self, monkeypatch): + """When the flag is on, build_context should call + ``memory_list_standing`` via the remote client and render + the result as the standing block.""" + monkeypatch.setenv("MNEMON_STANDING_TIER_ENABLED", "true") + + fake_payload = json.dumps([ + { + "doc_id": 42, + "title": "Career posture", + "content": "runway is multi-year, plenty of cash", + "content_type": "preference", + "confidence": 0.9, + "created_at": "2026-05-22 13:30:00", + }, + ]) + + def fake_call_tool_sync(name, args, timeout=5.0): + assert name == "memory_list_standing" + return fake_payload, 0.1 + + with patch( + "mnemon.hooks._remote_client.call_tool_sync", + side_effect=fake_call_tool_sync, + ): + from mnemon.hooks.context_surfacing import build_context + out = build_context(raw_text="[]") + + assert "Standing context" in out + assert "Career posture" in out + assert "runway is multi-year" in out + + def test_env_var_truthy_values(self, monkeypatch): + """The env-var override accepts 1 / true / yes / on.""" + from mnemon.hooks.context_surfacing import _standing_tier_enabled + for v in ("1", "true", "yes", "on", "TRUE", "True"): + monkeypatch.setenv("MNEMON_STANDING_TIER_ENABLED", v) + assert _standing_tier_enabled() is True + for v in ("0", "false", "no", "off", ""): + monkeypatch.setenv("MNEMON_STANDING_TIER_ENABLED", v) + monkeypatch.setattr(config, "STANDING_TIER_ENABLED", False) + assert _standing_tier_enabled() is False