feat(skill-memory): per-skill cross-session recall + historian auto-extraction

[iceteaSA]

Summary

Adds skill-memory — per-skill "motor memory" that gives a skill cross-session recall of its own hard-won lessons (gotchas, discoveries, fixes, workflow steps). When a skill's SKILL.md declares skill-memory: { enabled: true }, accumulated notes for that skill surface automatically in a <skill-memory> block appended to the skill tool's result on every load — and, with the historian extension, the historian writes those notes automatically during compaction (no agent action required).

It's fully opt-in per skill and cache-safe by construction: the block rides the tool-result tail (conversation), never the cached system/m[0] prefix, so it can't bust the prompt cache.

How it works

• Transparent recall — three-hook augmentation: tool.definition advertises an intent param; tool.execute.before stashes the intent (bounded TTL); the after-hook parses the skill's Base directory, reads its SKILL.md frontmatter, and formats the recall block. Lands in the tool RESULT (cache-safe).
• Write paths — ctx_skill_note (agent-authored) and the historian (auto-extracted). Both dedup on a normalized hash.
• Intent-scoped ranking — a recall cascade: model-matched embeddings → cosine blend over intent_embedding + delta_embedding (relevance/recency/hit weights tunable per skill via ranking_* frontmatter); FTS5 fallback over a content-linked skill_memory_fts vtable; flat recency×hit fallback.
• Historian auto-extraction — the historian sees TC: skill(<name>) markers in its chunk and emits a <skill_observations> block; both the OpenCode and Pi runners promote those post-commit as global notes (project_identity='*', source_type='historian'), recallable from any project.

Review units (4 commits)

The branch is organized into four coherent, reviewable phases:

1. P1 — transparent per-skill recall: skill_memory table migration, the three-hook augmentation, flat recall + storage, ctx_skill_note/ctx_skill_recall tools, opt-in distill-skill-memory dreamer task, TUI/ctx-status stats, docs.
2. P2 — embeddings + intent-scoped recall: delta_embedding + recall_count columns + skill_memory_fts FTS5 vtable; embed-on-write; the cosine/FTS recall cascade; a programmatic (no-LLM) reembed pre-step.
3. P3a — historian-extraction foundation: the TC: skill(<name>) marker keystone; origin_project + source_type columns; global-tier notes unified under project_identity='*' (collision-merge); partitionKey routing.
4. P3b — historian auto-extraction pipeline: prompt emits <skill_observations>, parser, validated-result threading, both runners promote via the shared promoteSkillObservations helper, plus an initializeDatabase self-heal net (re-creates skill_memory + ensureColumn so an upgraded DB recovers even if a migration row is lost).

Schema / migrations

Three migrations (numbered after the current master ceiling): skill_memory table, then delta_embedding/recall_count/FTS, then origin_project/source_type/* unification. LATEST_SUPPORTED_VERSION bumped in lockstep (the schema-version-fence test enforces it). Migration bodies are ensureColumn/IF NOT EXISTS idempotent.

Testing

Full plugin + Pi suites pass (one unrelated pre-existing full-suite ordering flake in tui-config.test.ts that passes in isolation), tsc clean both packages, lint clean, build produces all bundles. Dedicated coverage for the migrations (coexistence + fence), recall rungs, the tools, FTS triggers/backfill, the '*' collision-merge, and the historian promotion path on both runners. Verified working live: the historian auto-extraction writes genuine source_type='historian' notes, embeddings populate, and the read-side recall_count increments on surfacing.

Notes for reviewers

• Migration version numbers are placeholders relative to this fork's base — happy to renumber to whatever slots are free at merge time.
• The four commits are independently meaningful; if you'd prefer this as stacked PRs (P1 / P2 / historian), I can split it.

---

^{Need help on this PR? Tag /codesmith with what you need. Autofix is disabled.}

---

Summary by cubic

Adds per-skill cross-session recall with intent-aware embeddings and historian auto-extraction. Notes surface in a <skill-memory> block, with ctx_skill_note to write and ctx_skill_recall to read; schema ships via fork‑lane migrations, and prompt/status surfaces now include the skill tools and stats.

• New Features

  • Opt-in per-skill recall via SKILL.md (skill-memory: { enabled: true }). Transparent <skill-memory> injection on skill loads; tools ctx_skill_note and ctx_skill_recall; prompt guidance added (A1 golden + light descriptions) and the budget fixture re‑measured.
  • Ranking cascade: embed‑on‑write (intent_embedding + delta_embedding) with cosine blend, FTS5 fallback, then recency×hit; recall_count tracked. distill-skill-memory task (opt‑in) re‑embeds NULL/stale vectors first and emits a read‑only health report.
  • Historian auto-extraction: TC: skill(<name>) → <skill_observations> → global '*' notes (source_type='historian'), promoted by both OpenCode and Pi; recall unions project partition with global '*' so project skills see historian notes.
  • Migrations: fork‑lane v10000–v10002 for skill_memory (+ embeddings/FTS + origin/source/global '*'), with initializeDatabase self‑heal (table/FTS/triggers) and a dedicated runForkMigrations pass. Status/RPC/TUI surface per‑project skill‑memory stats.

• Bug Fixes

  • Recall correctness: union project and global '*'; error logging on recall; budget math counts per‑note framing and clamps pinned to min(max_pinned_tokens, max_tokens). FTS rebuilds on self‑heal when empty but rows exist.
  • Provenance/discovery: accept plain paths and file://; name‑based fallback when the provenance line is truncated; last‑line, line‑anchored match; Windows path normalization; ancestor walk stops at the worktree root; global discovery handles both skill/ and skills/.
  • Frontmatter/guards: accept inline skill-memory: { enabled: true }; reject ctx_skill_note when disabled; session‑scoped intent stash keyed per‑session with TTL; disabled‑path init guard. Pi parity: skill‑memory tools are OpenCode‑only and are subtracted in Pi parity tests; prompt guidance drops ctx_memory cross‑refs when memory is off.

^{Written for commit 88d9849. Summary will update on new commits.}

Greptile Summary

This PR introduces skill-memory — an opt-in per-skill cross-session recall system that appends a <skill-memory> block to skill tool results and auto-extracts lessons via the historian. All four critical bugs from the previous review round (intent-stash keying, missing FTS rebuild in self-heal, project-local skill recall orphaning, ctx_skill_note writing disabled-skill notes) are addressed in the current HEAD.

• Storage & recall: three fork-lane migrations (v10000–v10002) with idempotent DDL; a four-rung recall cascade (cosine blend → FTS5 → flat recency×hit) behind recallSkillMemoryBlock; recallPartitionPredicate unions the skill's own partition with the global '*' tier so historian-written notes are surfaced for project-local skills.
• Transparent injection: tool.execute.before stashes the intent param (keyed ${sessionID}:${callID}); tool.execute.after populates the SkillLoadRegistry from the provenance line in the skill output and appends the recall block; session cleanup prunes by prefix to avoid cross-session eviction.
• Historian extraction: <skill_observations> parsed by compartment-parser.ts; both OpenCode and Pi runners call promoteSkillObservations under the same promotionActive && !discardedLast gate used for facts/primers.

Confidence Score: 5/5

• The change is safe to merge. All previously identified functional defects have been fixed, and the new code is well-guarded: recall errors are logged and degrade gracefully, the intent stash is bounded (TTL + cap), and the FTS self-heal fires only on the actual gap condition.
• Every code path that could silently drop data or surface incorrect results — orphaned notes, cross-session intent eviction, project-local skill recall missing historian notes, the FTS rebuild gap — has been addressed. The remaining two findings are a double function call in searchSkillMemoryFts (same pure function, same args, idempotent result) and an API-contract mismatch in ctx-skill-note/tools.ts where an already-partitioned key is passed where a raw identity is expected (harmless due to partitionKey idempotency). Neither can produce wrong data with the current implementation.
• No files require special attention. The two minor issues in storage.ts (searchSkillMemoryFts double call) and ctx-skill-note/tools.ts (partition key passed as project identity) are self-contained and have no runtime impact.

Important Files Changed

Filename	Overview
packages/plugin/src/features/magic-context/skill-memory/storage.ts	New file: all SQL helpers for skill_memory. Correct use of parameterized queries throughout; recallPartitionPredicate correctly unions the skill's own partition with the global '*' tier. One style issue: searchSkillMemoryFts computes the same predicate object twice instead of caching the result.
packages/plugin/src/features/magic-context/skill-memory/recall.ts	New file: four-rung recall cascade (cosine/FTS/flat). Error path now logs via log() before returning empty string — the previous bare catch {} concern from the pre-fix review is addressed. rankRung1 correctly normalises recency and hit_count to [0,1]; budgetFill clamps effectiveMaxPinned to maxTokens.
packages/plugin/src/features/magic-context/fork-migrations.ts	New file: three fork-lane migrations (v10000–v10002) with per-row presence checks (not watermark-based), idempotent DDL (IF NOT EXISTS/ensureColumn), and a guarded FTS rebuild in v10001. runForkMigrations validates that no version falls below the floor and uses IMMEDIATE transactions matching upstream isolation semantics.
packages/plugin/src/features/magic-context/storage-db.ts	Self-heal net in initializeDatabase creates skill_memory table, FTS vtable, and triggers if absent; the guarded rebuild (ftsCount=0 && rowCount>0) correctly fires only on the gap rather than every boot. runForkMigrations is called in both openDatabase and openDatabaseAsync paths.
packages/plugin/src/hooks/magic-context/hook-handlers.ts	Adds intentByCallId stash (composite ${sessionID}:${callID} keys), createToolExecuteBeforeHook, and maybeInjectSkillMemory. Session cleanup via pruneIntentsForSession (prefix prune) prevents cross-session eviction — the previously-flagged bare-callID bug is addressed. Provenance hoisting (single dynamic import shared by both blocks) is also correct.
packages/plugin/src/tools/ctx-skill-note/tools.ts	New tool. Correctly guards on frontmatterConfig?.enabled before inserting (previously-flagged orphan-note bug is fixed). Semantic dedup, exact-hash dedup, and concurrent-insert race all handled. Passes part (already-partitioned key) to storage functions that expect raw projectIdentity — idempotent but violates the storage API contract.
packages/plugin/src/features/magic-context/skill-memory/promote.ts	New file: historian note promotion. Correctly writes tier='global'/project_identity='*'. Hash-dedup via findExistingNote + bumpHitCount is correct. Per-item try/catch ensures one bad observation doesn't block the rest.
packages/plugin/src/features/magic-context/skill-memory/provenance.ts	New file: skill path resolution and registry. parseSkillProvenance correctly handles both file:// URLs and plain paths; last-match wins over the entire output. resolveSkillPathByName ancestor walk stops at .git, $HOME, or filesystem root. Session-scoped registry keyed ${sessionId}:${skillId} matches the cleanup pattern in onSessionDeleted.
packages/plugin/src/features/magic-context/skill-memory/frontmatter.ts	New file: minimal YAML frontmatter parser for skill-memory:. Handles both inline {...} and block forms; FRONTMATTER_REGEX is start-anchored (no m flag). Inline YAML comment stripping and quote handling are correct. Failure modes return null (inert).
packages/plugin/src/hooks/magic-context/compartment-parser.ts	Adds ParsedSkillObservation and skillObservations parsing via SKILL_OBS_ITEM_REGEX. The regex correctly anchors to line-start (^ with gm) and captures skill-id, kind, and lesson separated by `
packages/plugin/src/tools/ctx-skill-recall/tools.ts	New tool. Registry-first resolution with disk fallback correctly mirrors discoverSkills() order (project shadows global). Correctly guards on frontmatterConfig?.enabled before recall. Test-DI seams use an internal cast that keeps the public CtxSkillRecallToolDeps contract clean.

Sequence Diagram

sequenceDiagram
    participant Agent
    participant Hook as tool.execute hooks
    participant Registry as SkillLoadRegistry
    participant IntentStash as intentByCallId
    participant DB as skill_memory (SQLite)
    participant Historian

    Agent->>Hook: tool.execute.before(skill, intent)
    Hook->>IntentStash: stash(sessionID:callID, intent)

    Agent->>Hook: tool.execute.after(skill output)
    Hook->>Hook: parseSkillProvenance(output)
    Hook->>Hook: parseFrontmatterConfig(SKILL.md)
    Hook->>Registry: set(sessionID:skillId, provenance+config)
    Hook->>IntentStash: getAndDelete(sessionID:callID)
    Hook->>DB: recallSkillMemoryBlock(skillId, intent, tier, projectIdentity)
    DB-->>Hook: "<skill-memory> block (or "")"
    Hook-->>Agent: "output.output += block"

    Agent->>DB: ctx_skill_note(skill, kind, delta)
    DB->>DB: hash-dedup + semantic-dedup
    DB->>DB: embed(intent) + embed(delta)
    DB->>DB: insertSkillMemoryNote

    Historian->>DB: promoteSkillObservations(skillObservations)
    Note over Historian,DB: tier='global', project_identity='*'
    DB->>DB: hash-dedup per observation

Loading

_{Reviews (32): Last reviewed commit: "fix(prompt-surface): admit skill-memory ..." | Re-trigger Greptile}

[cubic-dev-ai]

11 issues found across 78 files

Prompt for AI agents (unresolved issues)

Check if these issues are valid — if so, understand the root cause of each and fix them. If appropriate, use sub-agents to investigate and fix each issue separately.


<file name="packages/plugin/src/features/magic-context/dreamer/task-executor.ts">

<violation number="1" location="packages/plugin/src/features/magic-context/dreamer/task-executor.ts:419">
P2: Re-embed pre-step errors are suppressed, allowing successful task completion reporting despite a failed prerequisite data maintenance step.</violation>
</file>

<sub>Note: This PR contains a large number of files. cubic only reviews up to 40 files per PR, so some files may not have been reviewed. cubic prioritizes the most important files to review.
On a pro plan you can use ultrareview for larger PRs.

Re-trigger cubic</sub>

[cubic-dev-ai]

4 issues found across 15 files (changes from recent commits).

Prompt for AI agents (unresolved issues)

Check if these issues are valid — if so, understand the root cause of each and fix them. If appropriate, use sub-agents to investigate and fix each issue separately.


<file name="packages/plugin/src/features/magic-context/dreamer/task-executor.ts">

<violation number="1" location="packages/plugin/src/features/magic-context/dreamer/task-executor.ts:419">
P2: Re-embed pre-step errors are suppressed, allowing successful task completion reporting despite a failed prerequisite data maintenance step.</violation>
</file>

<sub>Reply with feedback, questions, or to request a fix.

Re-trigger cubic</sub>

[cubic-dev-ai]

4 issues found across 78 files

Prompt for AI agents (unresolved issues)

Check if these issues are valid — if so, understand the root cause of each and fix them. If appropriate, use sub-agents to investigate and fix each issue separately.


<file name="packages/plugin/src/features/magic-context/dreamer/task-executor.ts">

<violation number="1" location="packages/plugin/src/features/magic-context/dreamer/task-executor.ts:419">
P2: Re-embed pre-step errors are suppressed, allowing successful task completion reporting despite a failed prerequisite data maintenance step.</violation>
</file>

<sub>Note: This PR contains a large number of files. cubic only reviews up to 40 files per PR, so some files may not have been reviewed. cubic prioritizes the most important files to review.
On a pro plan you can use ultrareview for larger PRs.

Re-trigger cubic</sub>

[cubic-dev-ai]

1 issue found across 2 files (changes from recent commits).

<sub>Tip: Review your code locally with the cubic CLI to iterate faster.

Re-trigger cubic</sub>