feat(skills): add diagnostic traces and guarded coding experiments

.claude-plugin/marketplace.json

@@ -9,7 +9,7 @@
       "name": "lore",
       "source": "./",
       "description": "Shared memory for Claude Code, Codex, and DOXA: capped curated facts, evidence-backed beliefs, FTS5 session search, staged review, learned skills, and signed sync.",
-      "version": "0.62.23"
+      "version": "0.62.24"
     }
   ]
 }

.claude-plugin/plugin.json

@@ -1,7 +1,7 @@
 {
   "name": "lore",
   "description": "Memory for Claude Code that reasons about you and improves itself: Honcho-pattern belief store with deriver, dreamer and dialectic (/lore:ask), automatic skillification with a closed judge/update/retire loop, on a Hermes-pattern core \u2014 hard-capped curated memory, FTS5 session search, staged review.",
-  "version": "0.62.23",
+  "version": "0.62.24",
   "author": {
     "name": "Fabian Wilde",
     "email": "docwilde@proton.me"

CHANGELOG.md

@@ -1,5 +1,22 @@
 # Changelog
 
+## 0.62.24 — 2026-10-19
+
+**Diagnostic traces**
+
+- Add **`trace capture|show`** for linked Claude/Codex actions and successful observations, with scrubbing, source hashes and explicit omission counts; automatic native review capture is opt-in.
+- Keep traces local and separate from the reviewer digest. Provider success flags do not verify task outcomes.
+
+**Skill experiments**
+
+- Add **`skills evaluate`** for exact pending coding skills: freeze baseline, candidate, fixtures and verifier; require training improvement, validation non-regression and preservation of each replay success.
+- Consume final tests once; retain immutable candidates, task traces and reports. Human approval remains required; the included deterministic fixture does not measure real model improvements.
+
+**Quality checks**
+
+- Check a **16 KiB** skill cap, excessive growth and operator-supplied answer markers before execution; label changed proposals/baselines stale and damaged evidence unavailable in pending review.
+- Scripts require human terminal authority and run without inherited credentials. They are trusted local code, with process deadlines and output limits; no sandbox or model adapter is supplied.
+
 ## 0.62.23 — 2026-10-19
 
 **Background review history**

README.md

@@ -36,6 +36,7 @@ and relations between beliefs do not turn them into approved facts.
 - **Session search.** Local FTS5 index of Claude Code, DOXA, and Codex transcripts.
 - **File maps.** A short `path — purpose` guide to important project files.
 - **Learned skills.** Verified recipes are proposed, evaluated, and updated or retired.
+- **Local skill experiments.** [Capture diagnostic traces and compare pending coding skills](docs/skill-evaluation.md) against frozen tests before human approval.
 - **Cross-machine sync.** Signed hub, peer, or offline transfer; off by default.
 - **Agent integration.** Claude Code and Codex plugins inject startup context;
   Claude review and explicit Codex writes retain the proposal and approval gates.

docs/manual.md

@@ -235,6 +235,7 @@ Every value below is optional and lives in `~/.claude/settings.json` → `"env"`
 | `LORE_MACHINE_HOST` | `hostname` | this host's machine-memory key; override when the hostname is not the name you use |
 | `LORE_REVIEW_MODEL` | unset | umbrella override for both headless roles |
 | `LORE_DERIVER_MODEL` | `haiku` | extraction — the easy role |
+| `LORE_DIAGNOSTIC_TRACES` | unset | `1`: capture local action/observation traces for admitted native reviews; [evaluation workflow](skill-evaluation.md) |
 | `LORE_DREAMER_MODEL` | `sonnet` | belief reconciliation and promotions — the judgment-heavy role |
 | `LORE_DIALECTIC_MODEL` | session default | model for the `/lore:ask` subagent |
 | `LORE_REVIEW_MIN_MESSAGES` | 3 | skip review below this many user messages |

docs/skill-evaluation.md

@@ -0,0 +1,136 @@
+# Diagnostic traces and procedural skill evaluation
+
+This is an opt-in local experiment workflow for coding skills. Evaluation
+results are advisory evidence for a person reviewing a pending proposal.
+Nothing installs a candidate, edits a factual memory, or calls a model by
+default. The ordinary reviewer digest and its token budget stay separate.
+
+## 1. Capture diagnostic evidence
+
+```sh
+lore trace capture --transcript /absolute/session.jsonl --session-id SESSION --engine claude
+lore trace show SHA256
+```
+
+Use `--engine codex` for Codex rollout JSONL. Both formats retain textual
+user/assistant messages, full tool arguments and successful as well as failed
+tool observations, with source line numbers and call IDs. Provider error
+flags remain observations; a tool saying “tests passed” does not verify a task.
+The supplied session label is descriptive. It grants no review authority.
+
+Set `lore config set LORE_DIAGNOSTIC_TRACES 1` to capture automatically when
+a native reviewer admits a session. Its learned skill proposals include the
+trace hash. Dry runs, internal workers and skipped sessions do not capture.
+This setting increases local disk use; turn it off with `lore config unset
+LORE_DIAGNOSTIC_TRACES`.
+
+Traces are private, local, content-addressed JSON in
+`$LORE_ROOT/diagnostic-traces/`. They are not sent to the reviewer, indexed,
+injected or synced. Credentials are scrubbed before any truncation. Each
+event allows 32 KiB of scrubbed content; an artifact allows 512 KiB. Clipped
+events retain the scrubbed full-content hash and length. Omitted events and
+unsupported blocks are counted, and `complete` becomes false. Nontext and
+reasoning blocks are not a complete execution trace. Invalid or changed
+source files refuse capture, as do fields larger than the scrubber's 1 MiB
+input bound or source lines larger than 8 MiB. Existing artifacts are verified before reuse.
+Remove unneeded trace files locally to reclaim space; there is no automatic
+retention policy yet.
+
+## 2. Evaluate an exact pending skill
+
+From a human terminal, against an operator-authored suite:
+
+```sh
+lore skills evaluate PROPOSAL_ID --suite /absolute/corpus/suite.json
+lore skills evaluation REPORT_SHA256
+lore skills artifact --kind candidate --sha256 CANDIDATE_OBJECT_SHA256
+lore skills artifact --kind trace --sha256 TRACE_SHA256
+lore skills artifact --kind suite --sha256 SUITE_OBJECT_SHA256
+lore pending show PROPOSAL_ID
+```
+
+Evaluation requires the native carrier's human-review authority. Model and
+detached worker requests cannot execute a suite. The installed baseline and
+the candidate's final wrapped `SKILL.md` bytes are frozen before running.
+Add proposals use an empty baseline; retire proposals are unsupported.
+The exact proposal hash binds the results; a changed proposal or installed
+baseline makes attached evidence stale. Corrupt artifacts display unavailable.
+Human approval and the existing exact-byte approval check still apply.
+
+The suite is a directory containing one JSON manifest, two standalone scripts
+and task fixture directories. See [the deterministic contract fixture](../examples/skill-evaluation/suite.json).
+Its scores test this mechanism; they do not measure a model or prove a learned
+skill improves real coding. Copy it to a dedicated corpus directory before
+customizing it. The manifest specifies:
+
+| Field | Meaning |
+| --- | --- |
+| `schema`, `family` | `lore.skill-suite.v1`, `coding` |
+| `model`, `runtime` | Frozen operator-supplied identities; not independently attested |
+| `interpreter` | Absolute executable path; resolved and hashed before every invocation |
+| `runner`, `verifier` | Relative paths to standalone scripts in the suite |
+| `timeout_seconds` | Per-process deadline, 1–120 seconds |
+| `cases` | Unique IDs, split and relative fixture-directory path; at most 64 |
+| `answer_markers` | Optional known task-answer snippets, 24–4096 bytes after whitespace/case normalization |
+
+Each split (`train`, `replay`, `validation`, `test`) must contain a case.
+Identical fixture contents across splits refuse evaluation. A corpus allows
+512 files, 16 MiB total and directory depth 32; symlinks and special files
+are refused. Keep only corpus files in that directory. Files are copied
+from a frozen in-memory snapshot into a fresh workspace for every arm/case.
+The suite snapshot records every filename, size and hash, not retained raw
+fixture bytes. Keep the original corpus under version control for replay.
+
+The runner receives these environment variables:
+
+- `LORE_EVAL_SKILL`: absolute path to the frozen baseline or candidate text.
+- `LORE_EVAL_TASK_ID`, `LORE_EVAL_MODEL`, `LORE_EVAL_RUNTIME`: case and run identity.
+- `HOME` and `TMPDIR`: a private per-run directory; `PATH`: `/usr/bin:/bin`.
+
+Its working directory contains that case's fixture. Caller environment,
+credentials and HOME are not inherited. The interpreter executes the runner
+script, then a separately materialized verifier script in the same workspace.
+Both must exit zero for a task to pass. The verifier must independently test
+the task rather than trust the runner's own result text. Each process has a
+deadline and 1 MiB per output stream; process groups are killed on completion
+or timeout. Reports keep hashes and 1500-character scrubbed log excerpts.
+
+Scripts are trusted executable code. These directories and process groups
+are not a filesystem, network or memory sandbox. Use a pinned external
+container/runtime in the runner for experiments requiring that isolation.
+The model/runtime strings identify the chosen experiment configuration;
+LORE cannot attest a remote model's weights or the interpreter's libraries.
+LORE provides no model adapter or API spending policy here.
+
+Train, replay and validation run first. A candidate must improve training
+passes, match or beat the baseline's validation count, and preserve **each**
+baseline replay success. These are integer gates; no statistical confidence
+or generalization claim follows from a tiny suite. Only a candidate passing
+those gates reaches the final test, where every candidate case must pass.
+
+Final tests are consumed before execution, including on interruption. The
+consumption key binds test fixtures and verifier, so changing the candidate,
+runner, model or manifest cannot silently reopen them. Later candidates can
+receive `validation_passed` with `final_test: already_consumed`; that is not
+a final-test pass. Use new untouched tasks for the next final evaluation.
+
+## 3. Check generalization and growth
+
+Before execution, the candidate is checked for copied operator-supplied
+answer markers, a 16 KiB size cap, and growth exceeding both twice the
+baseline and 4096 added bytes. A blocked candidate and its quality result
+are still retained for review. Without answer markers, answer copying is
+not checked. These exact-string/size checks do not detect paraphrased leaks,
+semantic overfitting or unnecessary rules; a person must assess those.
+
+Immutable candidate, suite-manifest, verifier-trace and report objects live
+under `$LORE_ROOT/skill-evaluations/`. A per-proposal pointer attaches the
+latest report without changing the proposal bytes. Prior objects remain
+available by hash. Reports include exact baseline/candidate hashes, fixture
+hashes, executable hash, configured model/runtime, per-split counts, quality
+checks and the final-test state. They are local and excluded from sync.
+There is no automatic best-snapshot selector or performance-based promotion.
+
+This workflow covers procedural skill experiments. It does not establish
+the truth of beliefs or factual memories, generate gold reference runs, or
+infer causal errors from a trajectory's first differing action.

examples/skill-evaluation/runner.py

@@ -0,0 +1,9 @@
+"""Deterministic contract fixture, not a model or a skill-effectiveness benchmark."""
+import os
+from pathlib import Path
+
+skill = Path(os.environ["LORE_EVAL_SKILL"]).read_text()
+expression = "sorted(set(values))" if "Deduplicate before sorting" in skill else "sorted(values)"
+if "Break replay" in skill and os.environ["LORE_EVAL_TASK_ID"] == "replay":
+    expression = "[]"
+Path("solution.py").write_text(f"def solve(values):\n    return {expression}\n")

examples/skill-evaluation/suite.json

@@ -0,0 +1,17 @@
+{
+  "schema":"lore.skill-suite.v1",
+  "family":"coding",
+  "model":"deterministic-contract-fixture",
+  "runtime":"python3-stdlib",
+  "interpreter":"/usr/bin/python3",
+  "runner":"runner.py",
+  "verifier":"verifier.py",
+  "timeout_seconds":5,
+  "answer_markers":["memorized task-specific answer must never appear in a reusable skill"],
+  "cases":[
+    {"id":"train","split":"train","fixture":"tasks/train"},
+    {"id":"replay","split":"replay","fixture":"tasks/replay"},
+    {"id":"validation","split":"validation","fixture":"tasks/validation"},
+    {"id":"test","split":"test","fixture":"tasks/test"}
+  ]
+}

examples/skill-evaluation/tasks/replay/case.json

@@ -0,0 +1 @@
+{"values":[1,2,3]}

examples/skill-evaluation/tasks/test/case.json

@@ -0,0 +1 @@
+{"values":[8,2,1,2,8]}

examples/skill-evaluation/tasks/train/case.json

@@ -0,0 +1 @@
+{"values":[3,1,3]}

examples/skill-evaluation/tasks/validation/case.json

@@ -0,0 +1 @@
+{"values":[-2,7,-2,0]}

examples/skill-evaluation/verifier.py

@@ -0,0 +1,8 @@
+"""Frozen task check; the runner's own success message does not score the task."""
+import json
+from pathlib import Path
+
+namespace = {}
+exec(Path("solution.py").read_text(), namespace)
+values = json.loads(Path("case.json").read_text())["values"]
+assert namespace["solve"](values) == sorted(set(values))

plugin.json

@@ -1,7 +1,7 @@
 {
   "$schema": "https://agent-plugins.org/schemas/1.0.0/plugin.schema.json",
   "name": "lore",
-  "version": "0.62.23",
+  "version": "0.62.24",
   "description": "Unified user, repo, and session memory for Claude Code, Codex, and DOXA",
   "author": {
     "name": "Dr. Fabian Wilde",

rust/lore-core/Cargo.toml

@@ -1,6 +1,6 @@
 [package]
 name = "lore-core"
-version = "0.62.23"
+version = "0.62.24"
 edition = "2021"
 license = "AGPL-3.0-only"
 description = "Native LORE memory, beliefs, review and session index"

rust/lore-core/src/core.rs

@@ -28,6 +28,11 @@ impl Core {
             "filemap",
             "codegraph_snapshot_read_v1",
             "codegraph_snapshot_store_v1",
+            "diagnostic_trace_capture_v1",
+            "diagnostic_trace_read_v1",
+            "skill_evaluate_v1",
+            "skill_evaluation_read_v1",
+            "skill_evaluation_artifact_v1",
             "refresh_interval",
             "transcript_identity",
             "consult",
@@ -70,6 +75,11 @@ impl Core {
         let config = &self.config;
         let auth = &self.authority;
         let value = match req["op"].as_str().ok_or(Error::InvalidRequest)? {
+            "diagnostic_trace_capture_v1" => crate::diagnostics::capture(config, req)?,
+            "diagnostic_trace_read_v1" => crate::diagnostics::show(config, req)?,
+            "skill_evaluate_v1" => crate::skill_evaluation::evaluate(config, req, auth)?,
+            "skill_evaluation_read_v1" => crate::skill_evaluation::show(config, req)?,
+            "skill_evaluation_artifact_v1" => crate::skill_evaluation::artifact(config, req)?,
             "runtime_config_v1" => crate::config::runtime(config),
             "store_status_v1" => {
                 let conn = crate::store::read_only(config)?;