feat: separate recall admission profiles and signals

[ezutfen]

Refs #160 (ENG-RECALL-003, parent #153). This is the first implementation slice: explicit recall-profile contracts, separated recall signals, and a bounded shadow-comparison path. No uncertified recall profile becomes authoritative in this PR.

Problem

Semantic recall previously blended semantic relevance, importance/popularity, source priors, and review status into a single trust_score multiplier (similarity * trust_score) and silently meant “active plus proposed”. That allowed highly similar proposals with unknown evidence into ordinary packets and let utility/popularity raise a value presented as trust. #157–#159 added the evidence/admission substrate, but recall did not consume or expose it cleanly.

What changed

Recall profiles and certification boundary

engram/recall_profiles.py defines the versioned recall-profiles-v1 contract:

Profile	Corpus window	Admission gate	Ranking	Budget caps	Expansion	Serving status
legacy	active + proposed	none	semantic-v3 blend	none	yes	authoritative
governed	active + governed-disputed stay kinds	yes	semantic-signals-v1	none	no	shadow-only
exploratory	active + proposed	yes	semantic-signals-v1	item 20 / byte 2048	no	shadow-only

CERTIFIED_SERVING_PROFILES is currently { "legacy" }. Ordinary POST /v1/recall therefore serves only the certified legacy semantic packet (startup remains its own profile). Explicit governed/exploratory serving requests fail closed with 422; an uncertified recall_default_profile is refused rather than honored. Configuration alone cannot promote a candidate profile into production authority. #162D/#176 remains NOT_CERTIFIED.

Authorized shadow comparison

POST /v1/recall/shadow-compare compares the authoritative legacy packet with governed/exploratory candidates without changing what is served.

The surface:

• requires REVIEW_SCOPE;
• additionally requires tenant opt-in via tenant_config.recall_profile_shadow_enabled (migration 041, default FALSE);
• writes no recall_logs row;
• increments no recall/exposure counters;
• writes no receipts;
• enqueues no jobs;
• feeds no promotion/evidence input.

The comparison runs whenever any requested profile has an eligible corpus. Legacy may correctly evaluate to an empty packet while governed evaluates a candidate-only corpus (for example, a disputed governed stay-kind).

Separated signal model

engram/recall_signals.py adds recall-signals-v1 / recall-admission-v1. Governed/exploratory candidate items expose separate:

• relevance_score;
• utility_score;
• epistemic_state (supported | contested | insufficient_evidence | unknown);
• structured warning_codes;
• admission block with profile, decision, reason codes, policy version, and durable assessment binding.

There is no blended trust_score on the signal path. Utility is 0.7 * importance + 0.3 * freshness, ranking is similarity * (0.5 + 0.5 * utility), and admission never consumes similarity, importance, or exposure counters. The legacy memory_confidence source-policy prior is not emitted as a generic epistemic low_confidence warning.

Neutral retrieval and pre-LIMIT eligibility

Signal profiles use semantic.retrieve_candidates, a neutral vector retrieval primitive that returns relevance/embedding identity without the legacy trust blend.

Mechanically expressible corpus eligibility is applied in SQL before the bounded HNSW LIMIT. For governed recall that means:

active OR (disputed AND kind is configured to stay in recall while disputed)

plus existing tenant/visibility/workspace/validity/embedding/RLS constraints. Ineligible disputed rows therefore cannot consume the bounded relevance window and starve eligible memories.

Durable admission assessment handling

Recall uses the shared digest-verified admission_assessment.resolve_bulk_admissions resolver. Persisted assessments remain readable when ENGRAM_ADMISSION_ASSESSMENT_CAPTURE_ENABLED=false; disabling capture stops new capture but does not erase enforcement of existing blocked/stale projections.

Assessment precedence is centralized:

• blocked withholds in every candidate profile/review-status branch;
• governed fails closed on stale assessments, including disputed stay-kind items;
• exploratory may mark stale state but never treats it as trusted authority.

Audit and compatibility

• Migration 040 adds recall_logs.recall_profile.
• Historical rows are reconstructed truthfully from existing mode: startup → startup, semantic → legacy; replay self-heals databases that applied the earlier migration revision.
• Migration 041 adds the fail-closed tenant shadow-inspection policy.
• SDK response/request types preserve the additive profile fields.
• MCP intentionally exposes no candidate-profile/shadow selector.
• Legacy semantic recall remains behavior-compatible: same active+proposed window, trust_score, semantic-v3 ranking, and relationship expansion.

Scope / deferred #160 work

This PR does not complete #160. The issue remains open for follow-up slices including:

• consuming ENG-PROMOTION-003C — Replace the universal 72-hour gate with versioned risk-based admission profiles #158 V2 per-surface admission decisions (semantic_exploratory, semantic_governed, startup) as the candidate policy authority;
• ENG-CLASSIFY-003 — Separate retention value from epistemic confidence and support versioned reassessment #157 bulk epistemic enrichment;
• signal-aware relationship expansion;
• evidence-root diversity/redundancy controls;
• conflict/tension-preserving budget packing;
• demonstrated-usefulness feedback policy;
• semantic Context Ledger receipt binding;
• review/historical recall surfaces;
• dogfood/exposure-concentration evaluation;
• fresh ENG-CALIBRATION-001 — Labeled admission evaluation and shadow-rollout certification #162 certification and any later default cutover.

No production-default change is authorized here.

Verification

Final reviewed head: 4fd3c56ae1b760c6126a96635ffc913295106086.

• make compose-ci: 3825 passed.
• Exact-head GitHub CI: run #522, success.
• Runtime image smoke: /health and /ready pass.
• Real PostgreSQL/RLS shards pass.
• Regression coverage includes certification-gated serving, reviewer+tenant shadow authorization, read-only shadow evaluation, assessment-capture rollback, stale/blocked precedence, pre-LIMIT eligibility, neutral retrieval, truthful migration backfill, legacy compatibility, exposure-loop separation, and the candidate-only-corpus shadow case.