feat(salience): Phase 1 — first-class standing tier (default-off, soak-gated)

CHANGELOG.md

@@ -4,6 +4,93 @@
 
 ### Features
 
+- **Salience tier Phase 1 — first-class standing-context recall
+  (default-off, soak-gated).** Memories explicitly promoted via
+  `memory_promote` are injected into every `<mnemon-context>`
+  envelope on every prompt, regardless of query similarity. The cap
+  is the contract: default 15, hard ceiling 20. Plan:
+  `private/mnemon-salience-tier-plan-260521.md`.
+  - **Reframed validation gate (2026-05-22).** Phase 1 IS the
+    validation. Earlier plan called for a synthetic A/B against the
+    Phase 0 env-var-flagged form before committing to schema +
+    tooling. Reframed because the injection mechanism is identical
+    between the two forms — an A/B of the gated env-var path
+    carries no marginal information once Phase 1 ships gated behind
+    `STANDING_TIER_ENABLED=false`. Operator promotes ~5 career-
+    context memories, flips the flag, observes ≥1 week soak for
+    runway-style under-weighting recurrence vs absence. Per
+    `feedback_phase_gated_soak_consumer_must_be_ready`: ship the
+    substrate gated, flip activation at a separate milestone.
+  - **Schema migration**: `documents.tier TEXT NOT NULL DEFAULT
+    'situational'` via `_migrate_tier()`. Index `idx_documents_tier`
+    on live rows for the cap-count probe + search exclusion filter.
+    Additive + harmless if `STANDING_TIER_ENABLED` stays off.
+  - **`Store.promote_to_standing(id)`** + **`demote_to_situational(id)`**
+    + **`list_standing()`** + **`standing_tier_status()`**. Promote
+    raises **`StandingTierCapReached`** at the runtime cap,
+    **`StandingTierProvenanceRejected`** when source_client is in
+    `STANDING_TIER_BLOCKED_SOURCE_CLIENTS` (Layer 4 composition —
+    hook-sourced memories cannot be promoted; operator-explicit
+    gesture only), and **`StandingTierError`** on missing /
+    invalidated docs. Idempotent re-promote returns True; demote
+    of a situational doc returns False (no-op).
+  - **`Store.search_bm25` + `Store.search_vector`** gain
+    `include_standing: bool = False` keyword param. Standing-tier
+    docs excluded from ranked retrieval by default — they're
+    injected unconditionally already; ranking them too would
+    double-count and crowd the situational signal. Threaded through
+    `search.search()` so the higher-level entry respects the
+    invariant.
+  - **MCP tools**: `memory_promote(id)`, `memory_demote(id)`,
+    `memory_list_standing()` — both stdio (`server.py`) and
+    Streamable HTTP (`server_remote.py` reuses the same `mcp`
+    object). 14 → 17 registered tools.
+  - **CLI**: `mnemon standing list / promote <id> / demote <id>`.
+    `mnemon status` gains a `Standing tier: N/CAP` line.
+  - **`build_context` integration**: when `STANDING_TIER_ENABLED`
+    (config constant OR `MNEMON_STANDING_TIER_ENABLED` env override,
+    accepting `1/true/yes/on`), build_context calls
+    `memory_list_standing` via the remote client in a single
+    round-trip and renders the result as the "Standing context"
+    sub-section ahead of "Situational recall." Phase 0 env-var path
+    (`MNEMON_STANDING_TIER_FILE` → standing.json → standing-rendered.md
+    cache) is **preserved as fallback** so operators retain a
+    per-session override mechanism.
+  - **Composability invariants** (all preserved):
+    - Layer 0 (`is_well_shaped`) — capture rejection runs before
+      anything reaches the standing-tier promotion path
+    - Layer 1 envelope — standing block sits inside the same
+      `<mnemon-context>` data-marking + nonce as situational
+    - Layer 4 (`HOOK_SOURCE_CONFIDENCE_CEILING` + provenance) —
+      hook-sourced memories cannot be promoted; explicit
+      `StandingTierProvenanceRejected` rejection
+    - rc16 `source_key` upsert — unchanged; tier orthogonal
+    - Capture attention Phase A — `recurrence_count` accretes
+      against canonical situational memories; standing-tier
+      promotion is operator-gated on top of that signal
+  - **Soak gates** for flipping default-on: (a) ≥1 week with the
+    flag on; (b) observed reduction in runway-style under-weighting
+    recurrence on real career-strategy conversations; (c) zero
+    spurious-injection complaints from operator review of every
+    promoted memory.
+  - 22 new tests in `tests/test_standing_tier.py` covering: promote
+    success / cap-rejection (cap=2 in test, 3rd raises) /
+    hook-sourced rejection / invalidated rejection / missing rejection
+    / cap respects invalidated (freed slot reclaimable) / demote
+    round-trip / demote idempotent on situational / demote frees cap
+    slot / list_standing ordering + content / search excludes by
+    default / search includes when requested / build_context
+    flag-off no-fetch / flag-on memory_list_standing call /
+    env-var truthy value parsing. Suite 814 → 836 passing
+    (`test_server_remote.py` tool-count assertions bumped 14 → 17).
+
+### Schema
+
+- **`documents.tier TEXT NOT NULL DEFAULT 'situational'`** —
+  additive migration in `_migrate_tier()` after the existing
+  `_migrate_recurrence_count`. Index `idx_documents_tier` scoped to
+  live rows for cap-count + search-filter queries.
+
 - **Capture attention Phase A — recurrence-weighted memory convergence
   (default-off, soak-gated).** When a new save's content is semantically
   close to ≥2 prior memories spanning distinct sessions, capture

src/mnemon/cli.py

@@ -48,10 +48,13 @@ def main() -> None:
         from .store import Store
         store = Store()
         stats = store.status()
+        standing = store.standing_tier_status()
         print(f"Vault: {stats['vault_path']}")
         print(f"Total memories: {stats['total_documents']}")
         print(f"Vectors: {stats['total_vectors']}")
         print(f"Pinned: {stats['pinned']}")
+        print(f"Standing tier: {standing['count']}/{standing['cap']} "
+              f"(hard ceiling {standing['hard_ceiling']})")
         print(f"Invalidated: {stats['invalidated']}")
         print("\nBy type:")
         for t in stats["by_type"]:
@@ -308,12 +311,100 @@ def main() -> None:
         finally:
             store.close()
 
+    elif command == "standing":
+        # Salience tier Phase 1 — operator-facing tier management.
+        # private/mnemon-salience-tier-plan-260521.md
+        subcommand = args[1] if len(args) > 1 else "list"
+        _handle_standing(subcommand, args[2:])
+
     else:
         print(f"Unknown command: {command}", file=sys.stderr)
         _print_usage()
         sys.exit(1)
 
 
+def _handle_standing(subcommand: str, rest: list[str]) -> None:
+    """Salience tier Phase 1 — list / promote / demote subcommands."""
+    from .store import (
+        Store,
+        StandingTierCapReached,
+        StandingTierError,
+        StandingTierProvenanceRejected,
+    )
+    store = Store()
+    try:
+        if subcommand == "list":
+            docs = store.list_standing()
+            status = store.standing_tier_status()
+            print(f"Standing tier: {status['count']}/{status['cap']} "
+                  f"(hard ceiling {status['hard_ceiling']})")
+            if not docs:
+                print("  (empty — promote memories via `mnemon standing promote <id>`)")
+                return
+            print()
+            for d in docs:
+                snippet = (d.content or "")[:120].replace("\n", " ")
+                ellipsis = "..." if len(d.content or "") > 120 else ""
+                print(f"  #{d.id:>5}  [{d.content_type}]  {d.title}")
+                print(f"         {snippet}{ellipsis}")
+                print()
+
+        elif subcommand == "promote":
+            if not rest:
+                print("Usage: mnemon standing promote <id>", file=sys.stderr)
+                sys.exit(2)
+            try:
+                doc_id = int(rest[0])
+            except ValueError:
+                print(f"Error: <id> must be an integer (got {rest[0]!r})",
+                      file=sys.stderr)
+                sys.exit(2)
+            try:
+                store.promote_to_standing(doc_id)
+                status = store.standing_tier_status()
+                print(f"Promoted memory #{doc_id} to standing tier "
+                      f"({status['count']}/{status['cap']}).")
+            except StandingTierCapReached as e:
+                print(f"Cap reached: {e}", file=sys.stderr)
+                sys.exit(1)
+            except StandingTierProvenanceRejected as e:
+                print(f"Provenance rejected: {e}", file=sys.stderr)
+                sys.exit(1)
+            except StandingTierError as e:
+                print(f"Error: {e}", file=sys.stderr)
+                sys.exit(1)
+
+        elif subcommand == "demote":
+            if not rest:
+                print("Usage: mnemon standing demote <id>", file=sys.stderr)
+                sys.exit(2)
+            try:
+                doc_id = int(rest[0])
+            except ValueError:
+                print(f"Error: <id> must be an integer (got {rest[0]!r})",
+                      file=sys.stderr)
+                sys.exit(2)
+            try:
+                ok = store.demote_to_situational(doc_id)
+                status = store.standing_tier_status()
+                if ok:
+                    print(f"Demoted memory #{doc_id} to situational "
+                          f"({status['count']}/{status['cap']} remain standing).")
+                else:
+                    print(f"Memory #{doc_id} was not on the standing tier.")
+            except StandingTierError as e:
+                print(f"Error: {e}", file=sys.stderr)
+                sys.exit(1)
+
+        else:
+            print(f"Unknown subcommand: standing {subcommand}", file=sys.stderr)
+            print("Usage: mnemon standing list | promote <id> | demote <id>",
+                  file=sys.stderr)
+            sys.exit(2)
+    finally:
+        store.close()
+
+
 def _print_attention_status(store) -> None:
     """Print capture-attention soak metrics for the operator.
 
@@ -522,6 +613,15 @@ def _print_usage() -> None:
   mnemon attention-status   Capture-attention soak monitor — boost rate,
                             recurrence distribution, top canonicals,
                             recent 'restates' relations
+
+Salience tier (standing-context recall):
+  mnemon standing list      Show all standing-tier memories + count vs cap
+  mnemon standing promote <id>   Promote memory to standing tier (capped)
+  mnemon standing demote <id>    Demote back to situational
+                            Standing memories are injected into every
+                            recall context regardless of query similarity.
+                            Cap is the contract — default 15, hard 20.
+                            Hook-sourced memories cannot be promoted.
   mnemon forget <id>        Soft-delete from local vault
   mnemon rebuild            Re-embed every document (run after a model
                             change, or to recover from skipped embeddings)

src/mnemon/config.py

@@ -112,6 +112,25 @@ class MemoryType(str, Enum):
 CAPTURE_ATTENTION_REQUIRE_DISTINCT_SESSIONS = True  # neighbors must span ≥MIN_HITS distinct days
 CAPTURE_ATTENTION_SOAK_BOOST_RATE_MAX = 0.25    # acceptance criterion: boosts/saves ratio ceiling over 7d
 
+# Salience tier — standing-context recall, Phase 1 (added 2026-05-22) —
+# private/mnemon-salience-tier-plan-260521.md
+#
+# Standing-tier memories are injected unconditionally into the Layer 1
+# <mnemon-context> envelope on every prompt, regardless of query
+# similarity, conditioning reasoning rather than answering it. The cap
+# is the contract: past ~20, salience degrades and the tier becomes
+# noise again, recreating the failure mode at a different scale.
+# Default-off through soak per the 2026-05-22 reframing — Phase 1 ships
+# the substrate gated; promote ≤5 career-context memories via the new
+# memory_promote MCP tool; flip the flag; observe ≥1 week soak for
+# runway-style under-weighting recurrence vs absence.
+STANDING_TIER_ENABLED = False                   # feature flag — flip after soak
+STANDING_TIER_DEFAULT_CAP = 15                  # operator-tunable runtime cap
+STANDING_TIER_HARD_CEILING = 20                 # invariant — never exceed
+# Reuse Layer 4 hook-sourced set: hook-sourced memories cannot be promoted —
+# operator-explicit gesture only. Composes with provenance demotion.
+STANDING_TIER_BLOCKED_SOURCE_CLIENTS = HOOK_SOURCE_CLIENTS
+
 # Hook timeouts and budgets (seconds / chars)
 #
 # HOOK_REMOTE_TIMEOUT_SEC — matches Claude Code's ~/.claude/settings.json

src/mnemon/hooks/context_surfacing.py

@@ -66,21 +66,82 @@
 )
 
 
-# ── Phase 0 of the salience-tier plan ─────────────────────────────
+# ── Salience tier — standing context recall ───────────────────────
 # (private/mnemon-salience-tier-plan-260521.md)
 #
-# Standing context — feature-flagged, opt-in via MNEMON_STANDING_TIER_FILE.
-# When the env var points at a JSON file shaped {"ids": [<id>, <id>, ...]},
-# build_context fetches each ID via the remote MCP client and prepends a
-# labeled "Standing context" sub-section inside the existing
-# <mnemon-context> envelope, ahead of the query-driven block.
+# Two paths today, gated by the STANDING_TIER_ENABLED feature flag:
 #
-# Per-prompt cost: ~500ms × N (sequential memory_get HTTP calls). For N=10
-# that's ~5s added latency. Phase 0 accepts this cost — the diagnostic is
-# about hypothesis validation, not perf. Phase 1 will optimize via local
-# cache or batch fetch.
+# 1. Phase 1 (default when flag is on): single ``memory_list_standing``
+#    MCP call fetches the live standing-tier set in one round-trip.
+#    Cap-bounded (default 15, hard ceiling 20), so the payload is small.
+#    Source of truth = ``documents.tier='standing'`` in the Fly vault.
 #
-# Defaults to unset = original behavior unchanged.
+# 2. Phase 0 (fallback when flag is off, or as operator override):
+#    env-var MNEMON_STANDING_TIER_FILE → ~/.mnemon/standing.json IDs
+#    → cached rendered block at ~/.mnemon/standing-rendered.md.
+#    Useful when an operator wants to override the schema-backed set
+#    with a hand-picked ID list per session.
+#
+# The Phase 0 path is preserved as a fallback per the 2026-05-22
+# reframing — Phase 1 ships gated; the env-var path stays operational
+# either way. Defaults: STANDING_TIER_ENABLED False (config) + no env
+# override = nothing injected, original behavior unchanged.
+
+def _standing_tier_enabled() -> bool:
+    """Check whether the Phase 1 standing tier is active.
+
+    Truth sources, in order:
+      1. ``MNEMON_STANDING_TIER_ENABLED`` env var (operator override)
+      2. ``config.STANDING_TIER_ENABLED`` (default-off through soak)
+    """
+    env = os.environ.get("MNEMON_STANDING_TIER_ENABLED", "").strip().lower()
+    if env in ("1", "true", "yes", "on"):
+        return True
+    if env in ("0", "false", "no", "off"):
+        return False
+    from ..config import STANDING_TIER_ENABLED
+    return STANDING_TIER_ENABLED
+
+
+def _fetch_standing_via_mcp() -> str:
+    """Phase 1 path: single memory_list_standing MCP call.
+
+    Returns the rendered standing block (markdown bullets) or "" if
+    the call fails / returns empty. One round-trip vs Phase 0's
+    ~500ms × N sequential per-id fetches.
+    """
+    try:
+        from ._remote_client import call_tool_sync
+    except ImportError:
+        return ""
+
+    from ..safety import defang_control_markup
+
+    try:
+        raw, _elapsed = call_tool_sync("memory_list_standing", {}, timeout=5.0)
+        docs = json.loads(raw)
+    except Exception:
+        # Best-effort hook contract — failures here mean no standing
+        # block on this prompt, situational recall still runs.
+        return ""
+
+    if not isinstance(docs, list) or not docs:
+        return ""
+
+    lines: list[str] = []
+    for d in docs:
+        title = defang_control_markup(str(d.get("title", "")))
+        content = str(d.get("content", ""))
+        content_type = str(d.get("content_type", "note"))
+        doc_id = d.get("doc_id", "?")
+        snippet = defang_control_markup(content[:_SNIPPET_CHARS])
+        ellipsis = "..." if len(content) > _SNIPPET_CHARS else ""
+        lines.append(
+            f"- [{content_type}] **{title}** (id={doc_id})\n"
+            f"  {snippet}{ellipsis}"
+        )
+    return "\n\n".join(lines)
+
 
 def _load_standing_ids() -> list[int]:
     """Read standing-tier IDs from MNEMON_STANDING_TIER_FILE, if set."""
@@ -211,14 +272,26 @@ def build_context(raw_text: str, *, prefix: str = "") -> str:
     Returns an empty string when there's nothing worth injecting — no
     standing IDs AND no situational results (or unparseable JSON).
     """
-    # Standing context — Phase 0 opt-in. Computed first so the function
-    # can inject standing-only when no situational results exist.
+    # Standing context — Phase 1 (preferred) or Phase 0 (fallback).
+    # Computed first so the function can inject standing-only when no
+    # situational results exist.
+    #
+    # Phase 1: when STANDING_TIER_ENABLED, single memory_list_standing
+    # MCP call fetches the live schema-backed standing-tier set
+    # (one round-trip, cap-bounded payload). This is the canonical
+    # path once an operator has promoted memories via memory_promote.
     #
-    # Cache-first: ~/.mnemon/standing-rendered.md (microseconds). If
-    # the cache doesn't exist, fall back to per-prompt HTTP fetch
-    # (~500ms × N). build_standing_set.py writes both on every run.
-    standing_block = _read_rendered_cache()
+    # Phase 0 (fallback): env-var-driven ID list → cached rendered
+    # block. Useful as an operator override or before any memories
+    # have been promoted to the schema-backed tier.
+    standing_block = ""
+    if _standing_tier_enabled():
+        standing_block = _fetch_standing_via_mcp()
+    if not standing_block:
+        # Phase 0 cache-first
+        standing_block = _read_rendered_cache()
     if not standing_block:
+        # Phase 0 per-id HTTP fallback
         standing_ids = _load_standing_ids()
         standing_block = _fetch_standing_block(standing_ids) if standing_ids else ""