From 0adcf6f7015dbffa82667b7a0c08983e22665db1 Mon Sep 17 00:00:00 2001 From: lluvr Date: Fri, 6 Nov 2026 19:29:45 +0100 Subject: [PATCH 1/9] Emit PreToolUse hook captures as additionalContext JSON Claude Code only adds a PreToolUse hook's stdout to the model's context when it is a JSON object carrying hookSpecificOutput.additionalContext; plain text goes to the debug log. Emit that shape, and log a surface event only when captures were actually emitted. Signed-off-by: lluvr --- CHANGELOG.md | 4 +++ README.md | 2 +- docs/ARCHITECTURE.md | 4 +-- hooks/claude-code-pre-tool-use.sh | 45 ++++++++++++++++++++----------- scripts/test.sh | 34 +++++++++++++++++++++++ 5 files changed, 70 insertions(+), 19 deletions(-) diff --git a/CHANGELOG.md b/CHANGELOG.md index cd8d12b..42176b4 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -8,6 +8,10 @@ The format follows [Keep a Changelog](https://keepachangelog.com/en/1.1.0/) and ## [Unreleased] +### Fixed + +- The Claude Code `PreToolUse` hook now prints its captures as `{"hookSpecificOutput": {"hookEventName": "PreToolUse", "additionalContext": ...}}`. Claude Code adds plain stdout to the model's context only for events such as `SessionStart` and `UserPromptSubmit`; for `PreToolUse` it went to the debug log, so the per-action captures never reached the assistant. The hook also logs a surface event only when it actually emitted captures. + ### Changed - Repository home moved to a personal account (`lluvr/cma`). Self-references, badges, and issue/security links updated; author and publisher framing is now personal (organization publisher attribution dropped). diff --git a/README.md b/README.md index aeaa381..e358a48 100644 --- a/README.md +++ b/README.md @@ -96,7 +96,7 @@ cma surfaces relevant prior captures automatically when you (or an AI assistant) Two hooks for Claude Code: a `PreToolUse` hook for per-action surfacing and a `SessionStart` hook for session-priming context. -**Per-action surfacing** (`hooks/claude-code-pre-tool-use.sh`): when Claude is about to use a tool that touches a file or runs a command, the hook detects surface heuristically from the tool input, queries `cma surface`, and writes matched captures to stdout. Claude Code injects them as additional context. Silent for non-relevant tools (`Read`, etc.) and when no captures match. +**Per-action surfacing** (`hooks/claude-code-pre-tool-use.sh`): when Claude is about to use a tool that touches a file or runs a command, the hook detects surface heuristically from the tool input, queries `cma surface`, and prints matched captures as a JSON object with `hookSpecificOutput.additionalContext`, the form Claude Code adds to the assistant's context for `PreToolUse` hooks. Silent for non-relevant tools (`Read`, etc.) and when no captures match; a surface event is logged only when captures were actually handed to the assistant. **Session-priming context** (`hooks/claude-code-session-start.sh`): at the start of each session, surfaces recurring failure patterns and active rejections so the assistant has orientation before the first tool call. Configurable via `CMA_SESSION_START_SECTIONS` (default `recurrence,rejections`; set to `all` for `recurrence,rejections,behavior`). diff --git a/docs/ARCHITECTURE.md b/docs/ARCHITECTURE.md index 2b05aae..7449761 100644 --- a/docs/ARCHITECTURE.md +++ b/docs/ARCHITECTURE.md @@ -85,7 +85,7 @@ The fourth stage delivers surfaced captures to your context. The injection chann | Environment | Channel | |-------------|---------| -| Claude Code | `stdout` becomes additional context passed to the model | +| Claude Code | `PreToolUse`: a JSON object on `stdout` with `hookSpecificOutput.additionalContext`; `SessionStart`: plain `stdout`. Both become additional context passed to the model | | Shell | `stderr` is visible to you before the next prompt | | IDE | Inline notice in the editor (status bar, popover, comment) | | CI | Comment on the commit, PR, or pipeline run | @@ -114,7 +114,7 @@ The cma repository includes the following reference implementations of this arch `hooks/claude-code-pre-tool-use.sh` -Interception via Claude Code's `PreToolUse` hook event. Reads stdin JSON (current format) or environment variables (legacy fallback). Surfaces captures to stdout, where Claude Code injects them as additional context. Silent for non-relevant tools (`Read`, etc.) and when no captures match. +Interception via Claude Code's `PreToolUse` hook event. Reads stdin JSON (current format) or environment variables (legacy fallback). Surfaces captures as a JSON object on stdout carrying `hookSpecificOutput.additionalContext`, which Claude Code injects as additional context (plain stdout from a `PreToolUse` hook only reaches the debug log). Logs a surface event only when captures were emitted. Silent for non-relevant tools (`Read`, etc.) and when no captures match. Status: implemented, tested. diff --git a/hooks/claude-code-pre-tool-use.sh b/hooks/claude-code-pre-tool-use.sh index d9c3357..03031ad 100755 --- a/hooks/claude-code-pre-tool-use.sh +++ b/hooks/claude-code-pre-tool-use.sh @@ -3,8 +3,10 @@ # # Surfaces relevant prior captures (misses, decisions, rejections, preventions) # at the moment Claude is about to use a tool that touches a file or runs a -# command. Output goes to stdout, where Claude Code injects it as additional -# context for the assistant. +# command. Output goes to stdout as a JSON object carrying +# hookSpecificOutput.additionalContext, which is the form Claude Code adds to +# the assistant's context for PreToolUse hooks (plain stdout from a PreToolUse +# hook only reaches the debug log). # # Install: copy or symlink to a location, then add to ~/.claude/settings.json: # @@ -114,21 +116,32 @@ elif file_path: else: sys.exit(0) -try: - result = subprocess.run(args, capture_output=True, text=True, timeout=5) -except Exception: - sys.exit(0) - -if result.returncode != 0: +def run_surface(argv): + try: + result = subprocess.run(argv, capture_output=True, text=True, timeout=5) + except Exception: + return "" + if result.returncode != 0: + return "" + out = result.stdout.strip() + if out == "No captures match.": + return "" + return out + +# Query without logging first, so a surface event is only recorded when +# this hook actually hands captures to the assistant. A miss-free query +# is not a warning and should not count as one in leak detection. +if not run_surface(args + ["--no-log"]): sys.exit(0) - -output = result.stdout.strip() -if not output or output == "No captures match.": +output = run_surface(args) +if not output: sys.exit(0) -# Emit context for the assistant. Claude Code injects stdout as an -# additionalContext payload to the model. -print("Relevant prior captures from cma (action-time injection):") -print() -print(output) +context = "Relevant prior captures from cma (action-time injection):\n\n" + output +print(json.dumps({ + "hookSpecificOutput": { + "hookEventName": "PreToolUse", + "additionalContext": context, + } +})) PYEOF diff --git a/scripts/test.sh b/scripts/test.sh index 3f1b1da..15b98f7 100755 --- a/scripts/test.sh +++ b/scripts/test.sh @@ -789,6 +789,40 @@ else fail=$((fail + 1)) fi +# Output is the documented PreToolUse JSON shape, and nothing else. +hook_out=$(echo '{"tool_name":"Edit","tool_input":{"file_path":"/x/auth/y.ts"}}' | bash "$HOOK" 2>/dev/null) +hook_json_ok=$(python3 -c " +import json, sys +d = json.loads(sys.stdin.read()) +h = d['hookSpecificOutput'] +ok = (set(d) == {'hookSpecificOutput'} and h['hookEventName'] == 'PreToolUse' + and 'auth issue example' in h['additionalContext'] + and 'permissionDecision' not in h) +print('ok' if ok else 'fail') +" <<<"$hook_out" 2>/dev/null) +if [[ "$hook_json_ok" == "ok" ]]; then + printf "PASS %s\n" "hook emits hookSpecificOutput.additionalContext JSON" + pass=$((pass + 1)) +else + printf "FAIL %s (out=%q)\n" "hook emits hookSpecificOutput.additionalContext JSON" "$hook_out" + fail=$((fail + 1)) +fi + +# A silent hook run logs no surface event; an emitting run logs one. +events_count() { if [[ -f "$CMA_DIR/surface_events.jsonl" ]]; then wc -l < "$CMA_DIR/surface_events.jsonl" | tr -d ' '; else echo 0; fi; } +ev_before=$(events_count) +echo '{"tool_name":"Edit","tool_input":{"file_path":"/x/payments/y.ts"}}' | bash "$HOOK" >/dev/null 2>&1 +ev_silent=$(events_count) +echo '{"tool_name":"Edit","tool_input":{"file_path":"/x/auth/y.ts"}}' | bash "$HOOK" >/dev/null 2>&1 +ev_emit=$(events_count) +if [[ "$ev_silent" == "$ev_before" && "$ev_emit" -eq $((ev_before + 1)) ]]; then + printf "PASS %s\n" "hook logs a surface event only when it emits" + pass=$((pass + 1)) +else + printf "FAIL %s (before=%s silent=%s emit=%s)\n" "hook logs a surface event only when it emits" "$ev_before" "$ev_silent" "$ev_emit" + fail=$((fail + 1)) +fi + # Env var fallback hook_out=$(CLAUDE_TOOL_NAME=Bash CLAUDE_TOOL_INPUT='{"command":"npm test auth"}' bash "$HOOK" &1) if [[ "$hook_out" == *"auth issue example"* ]]; then From 5e7c114dc7098306c6f57a5c483bab48837727ac Mon Sep 17 00:00:00 2001 From: lluvr Date: Fri, 6 Nov 2026 19:30:50 +0100 Subject: [PATCH 2/9] Keep stats views read-only and unify the leak rule stats --rejections/--preventions now surface with --no-log, so the SessionStart hook no longer logs an event every session. stats --evidence uses the --leaks strong rule (surface and fm both present and equal), so an empty fm no longer matches an empty fm. The closure rate is printed with its counts and marked self-attested. Signed-off-by: lluvr --- CHANGELOG.md | 3 +++ cma | 20 ++++++++++++++---- scripts/test.sh | 55 +++++++++++++++++++++++++++++++++++++++++++++++++ 3 files changed, 74 insertions(+), 4 deletions(-) diff --git a/CHANGELOG.md b/CHANGELOG.md index 42176b4..7b4eb3e 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -11,6 +11,9 @@ The format follows [Keep a Changelog](https://keepachangelog.com/en/1.1.0/) and ### Fixed - The Claude Code `PreToolUse` hook now prints its captures as `{"hookSpecificOutput": {"hookEventName": "PreToolUse", "additionalContext": ...}}`. Claude Code adds plain stdout to the model's context only for events such as `SessionStart` and `UserPromptSubmit`; for `PreToolUse` it went to the debug log, so the per-action captures never reached the assistant. The hook also logs a surface event only when it actually emitted captures. +- `cma stats --rejections` and `--preventions` no longer log surface events. They are read-only views, and the `SessionStart` hook calls `--rejections` every session, which inflated the event log with entries that were never action-time warnings. +- `cma stats --evidence` now uses the same leak rule as `cma stats --leaks`: surface and fm both present and equal. Previously an absent fm on both sides matched (`"" == ""`), so the two views disagreed on the same corpus. +- `cma stats --evidence` prints the counts behind the closure rate and marks it self-attested; `--json` adds `loop_closure_evidenced`, `loop_closure_denominator`, and `loop_closure_basis`. ### Changed diff --git a/cma b/cma index 08e2f45..dd5f094 100755 --- a/cma +++ b/cma @@ -580,19 +580,23 @@ def in_window(rec): misses_w = [m for m in misses if in_window(m)] preventions_w = [p for p in preventions if in_window(p)] +# Leak rule, shared with `cma stats --leaks` (its "strong" leak): surface +# and fm both present and equal on the miss and on the surfaced capture. +# An absent fm on either side is only a weak signal there, so it is not a +# leak here either; an empty fm must never match an empty fm. leaks = [] for miss in misses_w: miss_ts = norm_ts(miss.get("timestamp", "")) miss_surface = miss.get("surface", "") miss_fm = miss.get("fm", "") - if not miss_surface and not miss_fm: + if not miss_surface or not miss_fm: continue for event in events: if norm_ts(event.get("timestamp", "")) >= miss_ts: continue for matched in event.get("matched", []): - if (matched.get("surface") == miss_surface and - matched.get("fm") == miss_fm and miss_surface): + if (matched.get("surface", "") == miss_surface and + matched.get("fm", "") == miss_fm): leaks.append(miss) break else: @@ -657,6 +661,9 @@ if output_format == "json": "preventions_evidenced": evidenced_count, "leaks": leak_count, "loop_closure_rate": closure_rate, + "loop_closure_evidenced": evidenced_count, + "loop_closure_denominator": closure_denom, + "loop_closure_basis": "self-attested", "recurring": [ {"surface": s, "fm": f, "count": n} for (s, f), n in recurring @@ -676,6 +683,8 @@ if closure_denom == 0: else: print(" Loop closure rate: {:.0%} ({}/{})".format( closure_rate, evidenced_count, closure_denom)) + print(" self-attested: {} evidenced prevention(s) you recorded, {} leak(s)".format( + evidenced_count, leak_count)) print() if recurring: @@ -1028,7 +1037,10 @@ PYEOF fi if [[ -n "$view" ]]; then - cmd_surface --type "$view" + # Stats views are read-only: listing rejections or preventions is + # not a warning surfaced at action time, so it must not log a + # surface event (the SessionStart hook calls this every session). + cmd_surface --type "$view" --no-log return 0 fi diff --git a/scripts/test.sh b/scripts/test.sh index 15b98f7..094e61a 100755 --- a/scripts/test.sh +++ b/scripts/test.sh @@ -734,6 +734,61 @@ else fail=$((fail + 1)) fi +# Evidence and leaks share one fm rule: an absent fm on both sides is a weak +# signal in --leaks, so it must not count as a leak in --evidence either. +reset +"$CMA" miss "no-fm anchor" --surface auth >/dev/null +"$CMA" surface --surface auth >/dev/null +wait_for_next_second +"$CMA" miss "no-fm recur" --surface auth >/dev/null +nofm_leaks=$(python3 -c "import json,sys; print(json.loads(sys.stdin.read())['leaks'])" <<<"$("$CMA" stats --evidence --json 2>/dev/null)") +nofm_view=$("$CMA" stats --leaks 2>&1) +if [[ "$nofm_leaks" == "0" && "$nofm_view" != *"leak(s):"* && "$nofm_view" == *"weak signal"* ]]; then + printf "PASS %s\n" "evidence and leaks agree: empty fm is not a leak" + pass=$((pass + 1)) +else + printf "FAIL %s (evidence leaks=%s)\n" "evidence and leaks agree: empty fm is not a leak" "$nofm_leaks" + fail=$((fail + 1)) +fi + +# The closure rate is shown with its counts and marked self-attested. +reset +"$CMA" miss "anchor s" --surface auth --fm fm-1 >/dev/null +anchor_id=$("$CMA" id miss) +"$CMA" surface --surface auth >/dev/null +wait_for_next_second +"$CMA" prevented "caught s" --miss-id "$anchor_id" >/dev/null +expect_contains "evidence marks closure rate self-attested" "self-attested: 1 evidenced prevention(s) you recorded, 0 leak(s)" "$CMA" stats --evidence +sa_json=$(python3 -c " +import json, sys +d = json.loads(sys.stdin.read()) +print('ok' if (d['loop_closure_basis'] == 'self-attested' and d['loop_closure_evidenced'] == 1 + and d['loop_closure_denominator'] == 1) else 'fail') +" <<<"$("$CMA" stats --evidence --json 2>/dev/null)") +if [[ "$sa_json" == "ok" ]]; then + printf "PASS %s\n" "evidence --json carries closure counts and basis" + pass=$((pass + 1)) +else + printf "FAIL %s\n" "evidence --json carries closure counts and basis" + fail=$((fail + 1)) +fi + +# Stats views never log surface events. +reset +"$CMA" reject "OPTION: r" >/dev/null +"$CMA" prevented "p" >/dev/null +"$CMA" stats --rejections >/dev/null +"$CMA" stats --preventions >/dev/null +"$CMA" stats --evidence >/dev/null +"$CMA" stats --leaks >/dev/null +if [[ ! -f "$CMA_DIR/surface_events.jsonl" ]]; then + printf "PASS %s\n" "stats views do not log surface events" + pass=$((pass + 1)) +else + printf "FAIL %s\n" "stats views do not log surface events" + fail=$((fail + 1)) +fi + # stats --behavior reset expect_contains "behavior empty data" "No misses" "$CMA" stats --behavior From b1eb12906ef1a99a92e15f848fecfdd514ff51c6 Mon Sep 17 00:00:00 2001 From: lluvr Date: Fri, 6 Nov 2026 19:36:00 +0100 Subject: [PATCH 3/9] Run without GNU timeout on stock macOS The classifier, cma-pre and the SessionStart hook called timeout directly; where it is missing they silently produced nothing. Add a with_timeout helper that uses timeout, then gtimeout, then a python3 fallback that kills the process group on expiry. Signed-off-by: lluvr --- CHANGELOG.md | 1 + cma | 30 ++++++++++++++++++++++++- hooks/claude-code-session-start.sh | 34 ++++++++++++++++++++++++++--- hooks/cma-pre | 30 ++++++++++++++++++++++++- scripts/test.sh | 35 ++++++++++++++++++++++++++++++ 5 files changed, 125 insertions(+), 5 deletions(-) diff --git a/CHANGELOG.md b/CHANGELOG.md index 7b4eb3e..b7e0cab 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -14,6 +14,7 @@ The format follows [Keep a Changelog](https://keepachangelog.com/en/1.1.0/) and - `cma stats --rejections` and `--preventions` no longer log surface events. They are read-only views, and the `SessionStart` hook calls `--rejections` every session, which inflated the event log with entries that were never action-time warnings. - `cma stats --evidence` now uses the same leak rule as `cma stats --leaks`: surface and fm both present and equal. Previously an absent fm on both sides matched (`"" == ""`), so the two views disagreed on the same corpus. - `cma stats --evidence` prints the counts behind the closure rate and marks it self-attested; `--json` adds `loop_closure_evidenced`, `loop_closure_denominator`, and `loop_closure_basis`. +- The fm classifier, `hooks/cma-pre`, and the `SessionStart` hook no longer require GNU `timeout`. On stock macOS it is absent, so the hooks printed nothing and the classifier never ran. They now fall back to `gtimeout`, then to a python3 time limit that kills the whole process group. ### Changed diff --git a/cma b/cma index dd5f094..be91b2a 100755 --- a/cma +++ b/cma @@ -16,6 +16,34 @@ mkdir -p "$CMA_DIR" # Helpers # --------------------------------------------------------------------------- +# Run a command under a time limit. GNU `timeout` is missing on stock +# macOS, so fall back to Homebrew's `gtimeout`, then to python3 (already +# a hard dependency). Exit status 124 on expiry, matching `timeout`. +with_timeout() { + local secs="$1"; shift + if command -v timeout >/dev/null 2>&1; then + timeout "$secs" "$@" + elif command -v gtimeout >/dev/null 2>&1; then + gtimeout "$secs" "$@" + else + # Like GNU timeout, signal the whole process group on expiry so a + # grandchild holding the output pipe cannot outlive the limit. + python3 -c ' +import os, signal, subprocess, sys +try: + p = subprocess.Popen(sys.argv[2:], start_new_session=True) +except OSError: + sys.exit(127) +try: + sys.exit(p.wait(timeout=float(sys.argv[1]))) +except subprocess.TimeoutExpired: + os.killpg(p.pid, signal.SIGKILL) + p.wait() + sys.exit(124) +' "$secs" "$@" + fi +} + usage_for() { case "$1" in miss) @@ -171,7 +199,7 @@ cmd_miss() { # missing commands, or timeouts do not block the capture. # See docs/ARCHITECTURE.md Section 9 (Methodology integration). if [[ -z "$fm" ]] && [[ -n "${CMA_FM_CLASSIFIER:-}" ]]; then - fm=$(printf '%s' "$description" | timeout 5 sh -c "$CMA_FM_CLASSIFIER" 2>/dev/null | head -1 || true) + fm=$(printf '%s' "$description" | with_timeout 5 sh -c "$CMA_FM_CLASSIFIER" 2>/dev/null | head -1 || true) # Trim leading/trailing whitespace fm="${fm#"${fm%%[![:space:]]*}"}" fm="${fm%"${fm##*[![:space:]]}"}" diff --git a/hooks/claude-code-session-start.sh b/hooks/claude-code-session-start.sh index 9da4ea3..fcf6efc 100755 --- a/hooks/claude-code-session-start.sh +++ b/hooks/claude-code-session-start.sh @@ -47,6 +47,34 @@ if [[ "$sections" == "all" ]]; then sections="recurrence,rejections,behavior" fi +# Run a command under a time limit. GNU `timeout` is missing on stock +# macOS, so fall back to Homebrew's `gtimeout`, then to python3 (already +# a hard dependency). Exit status 124 on expiry, matching `timeout`. +with_timeout() { + local secs="$1"; shift + if command -v timeout >/dev/null 2>&1; then + timeout "$secs" "$@" + elif command -v gtimeout >/dev/null 2>&1; then + gtimeout "$secs" "$@" + else + # Like GNU timeout, signal the whole process group on expiry so a + # grandchild holding the output pipe cannot outlive the limit. + python3 -c ' +import os, signal, subprocess, sys +try: + p = subprocess.Popen(sys.argv[2:], start_new_session=True) +except OSError: + sys.exit(127) +try: + sys.exit(p.wait(timeout=float(sys.argv[1]))) +except subprocess.TimeoutExpired: + os.killpg(p.pid, signal.SIGKILL) + p.wait() + sys.exit(124) +' "$secs" "$@" + fi +} + # Collect outputs from selected sections. # Each subsection's full output is preserved (header + data + footer) # because the framing text helps the assistant interpret the data. @@ -55,7 +83,7 @@ get_section() { local out="" case "$name" in recurrence) - out=$(timeout 5 cma stats --recurrence 2>/dev/null || true) + out=$(with_timeout 5 cma stats --recurrence 2>/dev/null || true) # Suppress empty/no-data outputs: only return content if we # actually have recurring patterns. if [[ "$out" == *"no patterns are recurring"* ]] || [[ "$out" == *"No misses recorded"* ]] || [[ -z "$out" ]]; then @@ -63,14 +91,14 @@ get_section() { fi ;; rejections) - out=$(timeout 5 cma stats --rejections 2>/dev/null || true) + out=$(with_timeout 5 cma stats --rejections 2>/dev/null || true) # Suppress when there are no active rejections if [[ "$out" == *"No captures match"* ]] || [[ -z "$out" ]]; then out="" fi ;; behavior) - out=$(timeout 5 cma stats --behavior 2>/dev/null || true) + out=$(with_timeout 5 cma stats --behavior 2>/dev/null || true) if [[ "$out" == *"No misses recorded"* ]] || [[ "$out" == *"none have intended/corrected"* ]] || [[ -z "$out" ]]; then out="" fi diff --git a/hooks/cma-pre b/hooks/cma-pre index 442a36b..bbac948 100755 --- a/hooks/cma-pre +++ b/hooks/cma-pre @@ -123,8 +123,36 @@ if ! command -v cma >/dev/null 2>&1; then exit 0 fi +# Run a command under a time limit. GNU `timeout` is missing on stock +# macOS, so fall back to Homebrew's `gtimeout`, then to python3 (already +# a hard dependency). Exit status 124 on expiry, matching `timeout`. +with_timeout() { + local secs="$1"; shift + if command -v timeout >/dev/null 2>&1; then + timeout "$secs" "$@" + elif command -v gtimeout >/dev/null 2>&1; then + gtimeout "$secs" "$@" + else + # Like GNU timeout, signal the whole process group on expiry so a + # grandchild holding the output pipe cannot outlive the limit. + python3 -c ' +import os, signal, subprocess, sys +try: + p = subprocess.Popen(sys.argv[2:], start_new_session=True) +except OSError: + sys.exit(127) +try: + sys.exit(p.wait(timeout=float(sys.argv[1]))) +except subprocess.TimeoutExpired: + os.killpg(p.pid, signal.SIGKILL) + p.wait() + sys.exit(124) +' "$secs" "$@" + fi +} + # 5-second timeout protects against pathological cases (huge data dirs, etc.) -output=$(timeout 5 cma surface --surface "$surface" --limit 3 2>/dev/null || true) +output=$(with_timeout 5 cma surface --surface "$surface" --limit 3 2>/dev/null || true) # --------------------------------------------------------------------------- # Stage 4: injection (stderr, clearly attributed) diff --git a/scripts/test.sh b/scripts/test.sh index 094e61a..679bee4 100755 --- a/scripts/test.sh +++ b/scripts/test.sh @@ -1036,6 +1036,41 @@ else fail=$((fail + 1)) fi +# --------------------------------------------------------------------------- +# Portability: GNU timeout absent (stock macOS) +# --------------------------------------------------------------------------- + +# Build a PATH with only the tools cma and its hooks use, minus timeout and +# gtimeout, so the python3 fallback is what runs. +NO_TIMEOUT_BIN=$(mktemp -d) +trap 'rm -rf "$CMA_DIR" "$HOOK_BIN_DIR" "$NO_TIMEOUT_BIN"' EXIT +for tool in bash sh python3 cat head tr sed awk wc mkdir dirname realpath date df env basename; do + tool_path=$(command -v "$tool" 2>/dev/null) && ln -sf "$tool_path" "$NO_TIMEOUT_BIN/$tool" +done +ln -sf "$CMA" "$NO_TIMEOUT_BIN/cma" +if PATH="$NO_TIMEOUT_BIN" command -v timeout >/dev/null 2>&1; then + printf "FAIL %s\n" "no-timeout PATH really lacks timeout" + fail=$((fail + 1)) +fi + +reset +"$CMA" miss "portable miss" --surface auth --fm fm-1 >/dev/null +"$CMA" miss "portable miss two" --surface auth --fm fm-1 >/dev/null +"$CMA" reject "OPTION: portable rejection" >/dev/null +output=$(PATH="$NO_TIMEOUT_BIN" bash "$PRE" --check "git commit auth-fix" 2>&1) +expect_contains "cma-pre surfaces without GNU timeout" "portable miss" printf '%s' "$output" +output=$(PATH="$NO_TIMEOUT_BIN" bash "$SS_HOOK" &1) +if [[ "$output" == *"## recurrence"* && "$output" == *"portable rejection"* ]]; then + printf "PASS %s\n" "session-start works without GNU timeout" + pass=$((pass + 1)) +else + printf "FAIL %s (out=%q)\n" "session-start works without GNU timeout" "$output" + fail=$((fail + 1)) +fi +reset +PATH="$NO_TIMEOUT_BIN" CMA_FM_CLASSIFIER='echo portable-fm' "$CMA" miss "classified" >/dev/null +expect_contains "classifier runs without GNU timeout" '"fm": "portable-fm"' cat "$CMA_DIR/misses.jsonl" + # --------------------------------------------------------------------------- # Summary # --------------------------------------------------------------------------- From bfe2b75842550e87bf5f0a8dfed6e66c61978dc1 Mon Sep 17 00:00:00 2001 From: lluvr Date: Fri, 6 Nov 2026 19:36:43 +0100 Subject: [PATCH 4/9] Make the README quickstart paste-safe Quote the --fm placeholder so the shell does not parse it as a redirect, create ~/.local/bin before linking into it, and add a step that verifies the PreToolUse hook fires. Signed-off-by: lluvr --- CONTRIBUTING.md | 5 +++-- README.md | 22 +++++++++++++++++++--- 2 files changed, 22 insertions(+), 5 deletions(-) diff --git a/CONTRIBUTING.md b/CONTRIBUTING.md index a5b37fe..fbd4f9e 100644 --- a/CONTRIBUTING.md +++ b/CONTRIBUTING.md @@ -68,8 +68,9 @@ prefixed accordingly (`cma-1.1`, `cma-mcp-0.2`). ```bash cma --help ``` - If cma is missing, build it from this repository: `ln -s - "$(pwd)/cma" ~/.local/bin/cma` (per cma's README quickstart). + If cma is missing, build it from this repository: `mkdir -p + ~/.local/bin && ln -s "$(pwd)/cma" ~/.local/bin/cma` (per cma's + README quickstart). 4. **Run the full test suite for the component you touched.** ```bash ./scripts/test.sh # bash cma tests diff --git a/README.md b/README.md index e358a48..3e7efa0 100644 --- a/README.md +++ b/README.md @@ -27,22 +27,27 @@ Clone the repository and add the script to your `PATH`: ```bash git clone https://github.com/lluvr/cma.git +mkdir -p ~/.local/bin ln -s "$(pwd)/cma/cma" ~/.local/bin/cma # or copy to anywhere on PATH cma init # create the data directory with a README ``` +If `cma --version` reports "command not found", add `~/.local/bin` to your `PATH`. + +In the examples below, `""` is a placeholder: replace it with your own tag, keeping the quotes. + Capture a failure: ```bash cma miss "fix removed the error message instead of addressing the defect" \ - --surface infra --fm + --surface infra --fm "" ``` For richer capture (texture preservation), add the situational fields: ```bash cma miss "missed validation in middleware" \ - --surface auth --fm \ + --surface auth --fm "" \ --intended "patch only the failing test" \ --corrected "trace upstream defect, fix at root" \ --excerpt-from /tmp/conversation-excerpt.txt @@ -63,7 +68,7 @@ cma's central claim is that captured failures become surfaced warnings, surfaced ```bash # 1. Capture a failure so future similar work can see it. cma miss "claimed verified without testing the cross-tenant path" \ - --surface auth --fm + --surface auth --fm "" # 2. Surface relevant prior captures before the next related action. # The PreToolUse hook does this automatically; manual invocation @@ -108,6 +113,17 @@ cma install-hook --claude-code This merges the SessionStart and PreToolUse blocks into `~/.claude/settings.json` (or `./.claude/settings.json` with `--scope project`). It is idempotent, preserves any other hooks you have configured, and writes a backup of the prior file at `settings.json.bak`. Preview without writing via `--dry-run`. +Verify the per-action hook fires by feeding it the payload Claude Code would send (run from the directory you cloned into): + +```bash +cma miss "example: skipped the token expiry check" --surface auth +echo '{"tool_name":"Edit","tool_input":{"file_path":"src/auth/login.ts"}}' \ + | bash cma/hooks/claude-code-pre-tool-use.sh +# {"hookSpecificOutput": {"hookEventName": "PreToolUse", "additionalContext": "Relevant prior captures ..."}} +``` + +No output means no capture matched or `cma` is not on the `PATH` the hook sees. Then restart Claude Code so it picks up the new settings. + If you prefer to wire it manually, the JSON blocks the command produces look like: ```json From ad3f879e6bf67608fee8c558d4f0be324dd8f0f1 Mon Sep 17 00:00:00 2001 From: lluvr Date: Fri, 6 Nov 2026 19:38:48 +0100 Subject: [PATCH 5/9] Harden CLI argument and settings handling Fail cleanly on a flag with no value, on a non-positive or non-numeric --limit, and on an empty capture description. install-hook quotes the hook paths it writes and refuses wrong-typed hooks sections instead of raising a traceback. Signed-off-by: lluvr --- CHANGELOG.md | 4 +++ cma | 85 ++++++++++++++++++++++++++++++++++--------------- scripts/test.sh | 69 +++++++++++++++++++++++++++++++++++++++ 3 files changed, 132 insertions(+), 26 deletions(-) diff --git a/CHANGELOG.md b/CHANGELOG.md index b7e0cab..4c691e5 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -15,6 +15,10 @@ The format follows [Keep a Changelog](https://keepachangelog.com/en/1.1.0/) and - `cma stats --evidence` now uses the same leak rule as `cma stats --leaks`: surface and fm both present and equal. Previously an absent fm on both sides matched (`"" == ""`), so the two views disagreed on the same corpus. - `cma stats --evidence` prints the counts behind the closure rate and marks it self-attested; `--json` adds `loop_closure_evidenced`, `loop_closure_denominator`, and `loop_closure_basis`. - The fm classifier, `hooks/cma-pre`, and the `SessionStart` hook no longer require GNU `timeout`. On stock macOS it is absent, so the hooks printed nothing and the classifier never ran. They now fall back to `gtimeout`, then to a python3 time limit that kills the whole process group. +- A flag given without its value (`cma miss "x" --surface`) now exits 1 with "`--surface requires a value`" instead of a bash `unbound variable` error. +- `cma surface --limit` now requires a positive integer. `abc` raised a Python traceback and `-1` silently dropped the oldest result. +- `cma install-hook` shell-quotes the hook paths it writes, so a checkout path containing a space works, and refuses a `settings.json` whose `hooks`, `hooks.SessionStart`, or `hooks.PreToolUse` has the wrong JSON type with a clear message instead of a traceback. +- `cma miss`, `decision`, `reject`, and `prevented` reject an empty description, matching cma-mcp. ### Changed diff --git a/cma b/cma index be91b2a..4f5d4dd 100755 --- a/cma +++ b/cma @@ -16,6 +16,16 @@ mkdir -p "$CMA_DIR" # Helpers # --------------------------------------------------------------------------- +# Fail cleanly when a flag that takes a value is the last argument. +# Call as `need_value "$@"` from a flag's case arm: $2 is the flag +# and $3 its value. +need_value() { + if [[ $# -lt 3 ]]; then + echo "cma $1: $2 requires a value" >&2 + exit 1 + fi +} + # Run a command under a time limit. GNU `timeout` is missing on stock # macOS, so fall back to Homebrew's `gtimeout`, then to python3 (already # a hard dependency). Exit status 124 on expiry, matching `timeout`. @@ -173,22 +183,24 @@ PYEOF cmd_miss() { if [[ $# -eq 0 ]]; then usage_for miss; exit 1; fi + if [[ -z "$1" ]]; then echo "cma miss: description must be non-empty" >&2; exit 1; fi local description="$1"; shift local surface="" fm="" files="" local excerpt="" intended="" corrected="" while [[ $# -gt 0 ]]; do case "$1" in - --surface) surface="$2"; shift 2 ;; - --fm) fm="$2"; shift 2 ;; - --files) files="$2"; shift 2 ;; - --excerpt) excerpt="$2"; shift 2 ;; + --surface) need_value miss "$@"; surface="$2"; shift 2 ;; + --fm) need_value miss "$@"; fm="$2"; shift 2 ;; + --files) need_value miss "$@"; files="$2"; shift 2 ;; + --excerpt) need_value miss "$@"; excerpt="$2"; shift 2 ;; --excerpt-from) + need_value miss "$@" if [[ ! -f "$2" ]]; then echo "cma miss: --excerpt-from file not found: $2" >&2; exit 1 fi excerpt=$(cat "$2"); shift 2 ;; - --intended) intended="$2"; shift 2 ;; - --corrected) corrected="$2"; shift 2 ;; + --intended) need_value miss "$@"; intended="$2"; shift 2 ;; + --corrected) need_value miss "$@"; corrected="$2"; shift 2 ;; *) echo "cma miss: unknown flag: $1" >&2; usage_for miss; exit 1 ;; esac done @@ -212,12 +224,13 @@ cmd_miss() { cmd_decision() { if [[ $# -eq 0 ]]; then usage_for decision; exit 1; fi + if [[ -z "$1" ]]; then echo "cma decision: description must be non-empty" >&2; exit 1; fi local description="$1"; shift local surface="" applies_when="" while [[ $# -gt 0 ]]; do case "$1" in - --surface) surface="$2"; shift 2 ;; - --applies-when) applies_when="$2"; shift 2 ;; + --surface) need_value decision "$@"; surface="$2"; shift 2 ;; + --applies-when) need_value decision "$@"; applies_when="$2"; shift 2 ;; *) echo "cma decision: unknown flag: $1" >&2; usage_for decision; exit 1 ;; esac done @@ -227,12 +240,13 @@ cmd_decision() { cmd_reject() { if [[ $# -eq 0 ]]; then usage_for reject; exit 1; fi + if [[ -z "$1" ]]; then echo "cma reject: description must be non-empty" >&2; exit 1; fi local description="$1"; shift local surface="" revisit_when="" while [[ $# -gt 0 ]]; do case "$1" in - --surface) surface="$2"; shift 2 ;; - --revisit-when) revisit_when="$2"; shift 2 ;; + --surface) need_value reject "$@"; surface="$2"; shift 2 ;; + --revisit-when) need_value reject "$@"; revisit_when="$2"; shift 2 ;; *) echo "cma reject: unknown flag: $1" >&2; usage_for reject; exit 1 ;; esac done @@ -242,12 +256,13 @@ cmd_reject() { cmd_prevented() { if [[ $# -eq 0 ]]; then usage_for prevented; exit 1; fi + if [[ -z "$1" ]]; then echo "cma prevented: description must be non-empty" >&2; exit 1; fi local description="$1"; shift local miss_id="" warning_id="" while [[ $# -gt 0 ]]; do case "$1" in - --miss-id) miss_id="$2"; shift 2 ;; - --warning-id) warning_id="$2"; shift 2 ;; + --miss-id) need_value prevented "$@"; miss_id="$2"; shift 2 ;; + --warning-id) need_value prevented "$@"; warning_id="$2"; shift 2 ;; *) echo "cma prevented: unknown flag: $1" >&2; usage_for prevented; exit 1 ;; esac done @@ -263,15 +278,19 @@ cmd_surface() { local surface="" file="" type="" limit="10" log_event=true while [[ $# -gt 0 ]]; do case "$1" in - --surface) surface="$2"; shift 2 ;; - --file) file="$2"; shift 2 ;; - --type) type="$2"; shift 2 ;; - --limit) limit="$2"; shift 2 ;; + --surface) need_value surface "$@"; surface="$2"; shift 2 ;; + --file) need_value surface "$@"; file="$2"; shift 2 ;; + --type) need_value surface "$@"; type="$2"; shift 2 ;; + --limit) need_value surface "$@"; limit="$2"; shift 2 ;; --no-log) log_event=false; shift ;; *) echo "cma surface: unknown flag: $1" >&2; usage_for surface; exit 1 ;; esac done + if ! [[ "$limit" =~ ^[1-9][0-9]*$ ]]; then + echo "cma surface: --limit requires a positive integer" >&2; exit 1 + fi + local log_arg="log" $log_event || log_arg="nolog" @@ -392,10 +411,10 @@ cmd_distill() { while [[ $# -gt 0 ]]; do case "$1" in - --retire) retire_pattern="$2"; shift 2 ;; + --retire) need_value distill "$@"; retire_pattern="$2"; shift 2 ;; --review) review_mode=true; shift ;; - --scope) scope="$2"; shift 2 ;; - --surface) surface="$2"; shift 2 ;; + --scope) need_value distill "$@"; scope="$2"; shift 2 ;; + --surface) need_value distill "$@"; surface="$2"; shift 2 ;; -*) echo "cma distill: unknown flag: $1" >&2; usage_for distill; exit 1 ;; *) if [[ -z "$learning" ]]; then learning="$1"; else @@ -1132,9 +1151,9 @@ cmd_id() { while [[ $# -gt 0 ]]; do case "$1" in --type) - type="$2"; shift 2 ;; + need_value id "$@"; type="$2"; shift 2 ;; --surface) - surface="$2"; shift 2 ;; + need_value id "$@"; surface="$2"; shift 2 ;; miss|decision|rejection|prevention|core) # Allow `cma id miss` shorthand as well as --type miss. type="$1"; shift ;; @@ -1232,7 +1251,7 @@ cmd_install_hook() { $dry_run && mode="dry" python3 - "$settings_path" "$session_hook" "$pretool_hook" "$mode" <<'PYEOF' -import json, os, shutil, sys +import json, os, shlex, shutil, sys settings_path, session_hook, pretool_hook, mode = sys.argv[1:] @@ -1248,26 +1267,40 @@ if os.path.exists(settings_path): print(" Fix the file (or remove it) and re-run; no changes made.", file=sys.stderr) sys.exit(1) +def refuse(what, expected): + print(f"cma install-hook: {what} in {settings_path} must be {expected}; no changes made.", + file=sys.stderr) + sys.exit(1) + +if not isinstance(existing, dict): + refuse("the top level", "a JSON object") hooks = existing.setdefault("hooks", {}) +if not isinstance(hooks, dict): + refuse('"hooks"', "a JSON object") session_list = hooks.setdefault("SessionStart", []) pretool_list = hooks.setdefault("PreToolUse", []) +for event, lst in (("SessionStart", session_list), ("PreToolUse", pretool_list)): + if not isinstance(lst, list): + refuse(f'"hooks.{event}"', "a JSON array") def has_cma_hook(hook_list, hook_path): for group in hook_list: - for h in group.get("hooks", []): - if hook_path in h.get("command", ""): + if not isinstance(group, dict): + continue + for h in group.get("hooks", []) or []: + if isinstance(h, dict) and hook_path in str(h.get("command", "")): return True return False added = [] if not has_cma_hook(session_list, session_hook): session_list.append({ - "hooks": [{"type": "command", "command": f"bash {session_hook}"}] + "hooks": [{"type": "command", "command": "bash " + shlex.quote(session_hook)}] }) added.append("SessionStart") if not has_cma_hook(pretool_list, pretool_hook): pretool_list.append({ - "hooks": [{"type": "command", "command": f"bash {pretool_hook}"}] + "hooks": [{"type": "command", "command": "bash " + shlex.quote(pretool_hook)}] }) added.append("PreToolUse") diff --git a/scripts/test.sh b/scripts/test.sh index 679bee4..e3d044a 100755 --- a/scripts/test.sh +++ b/scripts/test.sh @@ -302,6 +302,24 @@ reset expect_exit "prevented succeeds" 0 "$CMA" prevented "almost X, did Y instead" expect_exit "prevented with miss-id" 0 "$CMA" prevented "x" --miss-id abc123 +# Flag with no value, and empty descriptions, fail cleanly (no bash traceback). +expect_exit "miss flag without value exits 1" 1 "$CMA" miss "t" --surface +expect_contains "miss flag without value message" "--surface requires a value" "$CMA" miss "t" --surface +expect_contains "surface flag without value message" "--file requires a value" "$CMA" surface --file +expect_contains "id flag without value message" "--surface requires a value" "$CMA" id --surface +reset +expect_exit "miss with empty description exits 1" 1 "$CMA" miss "" +expect_exit "decision with empty description exits 1" 1 "$CMA" decision "" +expect_exit "reject with empty description exits 1" 1 "$CMA" reject "" +expect_exit "prevented with empty description exits 1" 1 "$CMA" prevented "" +if [[ ! -e "$CMA_DIR/misses.jsonl" ]]; then + printf "PASS %s\n" "empty description writes nothing" + pass=$((pass + 1)) +else + printf "FAIL %s\n" "empty description writes nothing" + fail=$((fail + 1)) +fi + # --------------------------------------------------------------------------- # cma id helper # --------------------------------------------------------------------------- @@ -411,6 +429,50 @@ fi popd >/dev/null || exit 1 rm -rf "$ih_dir" +# Hook paths containing a space are quoted in the written command. +ih_dir=$(mktemp -d) +mkdir -p "$ih_dir/with space" "$ih_dir/proj" +cp "$CMA" "$ih_dir/with space/cma" +cp -r "$(dirname "$CMA")/hooks" "$ih_dir/with space/hooks" +pushd "$ih_dir/proj" >/dev/null || exit 1 +"$ih_dir/with space/cma" install-hook --claude-code --scope project >/dev/null +quoted_ok=$(python3 -c " +import json, shlex +d = json.load(open('.claude/settings.json')) +cmds = [h['command'] for ev in ('SessionStart', 'PreToolUse') for g in d['hooks'][ev] for h in g['hooks']] +ok = all(len(shlex.split(c)) == 2 and ' ' in shlex.split(c)[1] for c in cmds) and len(cmds) == 2 +print('ok' if ok else 'fail: ' + repr(cmds)) +") +if [[ "$quoted_ok" == "ok" ]]; then + printf "PASS %s\n" "install-hook quotes hook paths with spaces" + pass=$((pass + 1)) +else + printf "FAIL %s (%s)\n" "install-hook quotes hook paths with spaces" "$quoted_ok" + fail=$((fail + 1)) +fi +out=$("$ih_dir/with space/cma" install-hook --claude-code --scope project 2>&1) +expect_contains "install-hook idempotent with quoted paths" "already present" printf '%s' "$out" +popd >/dev/null || exit 1 +rm -rf "$ih_dir" + +# Wrong-typed hooks sections are refused cleanly, file left unchanged. +ih_dir=$(mktemp -d) +pushd "$ih_dir" >/dev/null || exit 1 +mkdir -p .claude +for bad in '{"hooks": []}' '{"hooks": {"PreToolUse": {}}}' '[]'; do + printf '%s\n' "$bad" > .claude/settings.json + err=$("$CMA" install-hook --claude-code --scope project 2>&1); rc=$? + if [[ "$rc" == 1 && "$err" != *Traceback* && "$err" == *"no changes made"* && "$(cat .claude/settings.json)" == "$bad" ]]; then + printf "PASS %s\n" "install-hook refuses wrong-typed settings: $bad" + pass=$((pass + 1)) + else + printf "FAIL %s (rc=%s err=%q)\n" "install-hook refuses wrong-typed settings: $bad" "$rc" "$err" + fail=$((fail + 1)) + fi +done +popd >/dev/null || exit 1 +rm -rf "$ih_dir" + # Malformed JSON is refused without overwriting. ih_dir=$(mktemp -d) pushd "$ih_dir" >/dev/null || exit 1 @@ -463,6 +525,13 @@ expect_exit "surface --limit 1 succeeds" 0 "$CMA" surface --limit 1 expect_contains "surface --surface docs filters" "first miss" "$CMA" surface --surface docs expect_exit "surface --bogus exits 1" 1 "$CMA" surface --bogus +# --limit must be a positive integer; -1 used to drop the oldest result. +expect_exit "surface --limit abc exits 1" 1 "$CMA" surface --limit abc +expect_contains "surface --limit abc message" "positive integer" "$CMA" surface --limit abc +expect_exit "surface --limit -1 exits 1" 1 "$CMA" surface --limit -1 +expect_exit "surface --limit 0 exits 1" 1 "$CMA" surface --limit 0 +expect_matches "surface --limit 2 returns two" 'second miss.*first miss' "$CMA" surface --type miss --limit 2 --no-log + # Surface --type filter excludes other types output=$("$CMA" surface --type miss 2>&1) if [[ "$output" != *"a decision"* ]]; then From 772a4c4cc4f45d55ab0ec974b2dc3a34c063cd8d Mon Sep 17 00:00:00 2001 From: lluvr Date: Fri, 6 Nov 2026 19:41:48 +0100 Subject: [PATCH 6/9] Keep captured data private and its output inert Create a new data directory 0700 and new data files 0600, since they hold conversation excerpts. Strip escape sequences and control characters (keeping newline and tab) from captured text written to the terminal or to hook output. Signed-off-by: lluvr --- CHANGELOG.md | 2 + cma | 110 +++++++++++++++++++++++++++++- docs/DATA.md | 2 + hooks/claude-code-pre-tool-use.sh | 4 ++ scripts/test.sh | 31 +++++++++ 5 files changed, 146 insertions(+), 3 deletions(-) diff --git a/CHANGELOG.md b/CHANGELOG.md index 4c691e5..c4d6179 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -19,6 +19,8 @@ The format follows [Keep a Changelog](https://keepachangelog.com/en/1.1.0/) and - `cma surface --limit` now requires a positive integer. `abc` raised a Python traceback and `-1` silently dropped the oldest result. - `cma install-hook` shell-quotes the hook paths it writes, so a checkout path containing a space works, and refuses a `settings.json` whose `hooks`, `hooks.SessionStart`, or `hooks.PreToolUse` has the wrong JSON type with a clear message instead of a traceback. - `cma miss`, `decision`, `reject`, and `prevented` reject an empty description, matching cma-mcp. +- A new data directory is created with mode 0700 and new data files with mode 0600, since captures can hold conversation excerpts. Existing installs keep their modes (see `docs/DATA.md`). +- Captured text is stripped of ANSI/OSC escape sequences and other control characters (newlines and tabs kept) before cma prints it to the terminal, so it also reaches the `SessionStart` and `PreToolUse` hook context clean. ### Changed diff --git a/cma b/cma index 4f5d4dd..ff6b521 100755 --- a/cma +++ b/cma @@ -10,7 +10,10 @@ VERSION="1.1.0" SCHEMA_VERSION="1.0" CMA_DIR="${CMA_DIR:-$HOME/.cma}" -mkdir -p "$CMA_DIR" +# The data directory holds conversation excerpts; create it private. +if [[ ! -d "$CMA_DIR" ]]; then + (umask 077 && mkdir -p "$CMA_DIR") +fi # --------------------------------------------------------------------------- # Helpers @@ -174,7 +177,9 @@ for i in range(0, len(extras) - 1, 2): if value: rec[key] = value -with open(path, "ab") as f: +# New data files are created owner-only (0600): they hold excerpts. +fd = os.open(path, os.O_WRONLY | os.O_APPEND | os.O_CREAT, 0o600) +with os.fdopen(fd, "ab") as f: f.write((json.dumps(rec) + "\n").encode("utf-8")) print(f"Captured {type_} {rec_id}") @@ -298,6 +303,20 @@ cmd_surface() { import json, os, sys, time from glob import glob +# Captured text is untrusted: strip ANSI/OSC escapes and other control +# characters (keeping newline and tab) before it reaches a terminal or +# hook context. +import re +class _CleanOut: + _ctrl = re.compile(r"\x1b\[[0-?]*[ -/]*[@-~]|\x1b\][^\x07\x1b]*(?:\x07|\x1b\\)|[\x00-\x08\x0b-\x1f\x7f-\x9f]") + def __init__(self, stream): + self._stream = stream + def write(self, text): + return self._stream.write(self._ctrl.sub("", text)) + def flush(self): + self._stream.flush() +sys.stdout = _CleanOut(sys.stdout) + data_dir, surface_filter, file_filter, type_filter, limit_str, log_arg = sys.argv[1:] limit = int(limit_str) if limit_str else 10 should_log = (log_arg == "log") @@ -383,7 +402,8 @@ if should_log: ], } log_path = os.path.join(data_dir, "surface_events.jsonl") - with open(log_path, "a") as f: + fd = os.open(log_path, os.O_WRONLY | os.O_APPEND | os.O_CREAT, 0o600) + with os.fdopen(fd, "a") as f: f.write(json.dumps(event) + "\n") if not filtered: @@ -429,6 +449,20 @@ cmd_distill() { import json, os, sys from collections import Counter +# Captured text is untrusted: strip ANSI/OSC escapes and other control +# characters (keeping newline and tab) before it reaches a terminal or +# hook context. +import re +class _CleanOut: + _ctrl = re.compile(r"\x1b\[[0-?]*[ -/]*[@-~]|\x1b\][^\x07\x1b]*(?:\x07|\x1b\\)|[\x00-\x08\x0b-\x1f\x7f-\x9f]") + def __init__(self, stream): + self._stream = stream + def write(self, text): + return self._stream.write(self._ctrl.sub("", text)) + def flush(self): + self._stream.flush() +sys.stdout = _CleanOut(sys.stdout) + data_dir = sys.argv[1] groups = Counter() total = 0 @@ -482,6 +516,20 @@ PYEOF python3 - "$CMA_DIR" "$retire_pattern" <<'PYEOF' import json, os, sys, time +# Captured text is untrusted: strip ANSI/OSC escapes and other control +# characters (keeping newline and tab) before it reaches a terminal or +# hook context. +import re +class _CleanOut: + _ctrl = re.compile(r"\x1b\[[0-?]*[ -/]*[@-~]|\x1b\][^\x07\x1b]*(?:\x07|\x1b\\)|[\x00-\x08\x0b-\x1f\x7f-\x9f]") + def __init__(self, stream): + self._stream = stream + def write(self, text): + return self._stream.write(self._ctrl.sub("", text)) + def flush(self): + self._stream.flush() +sys.stdout = _CleanOut(sys.stdout) + data_dir, pattern = sys.argv[1:3] core_path = os.path.join(data_dir, "core.jsonl") @@ -587,6 +635,20 @@ import json, os, sys from collections import Counter from datetime import datetime, timedelta, timezone +# Captured text is untrusted: strip ANSI/OSC escapes and other control +# characters (keeping newline and tab) before it reaches a terminal or +# hook context. +import re +class _CleanOut: + _ctrl = re.compile(r"\x1b\[[0-?]*[ -/]*[@-~]|\x1b\][^\x07\x1b]*(?:\x07|\x1b\\)|[\x00-\x08\x0b-\x1f\x7f-\x9f]") + def __init__(self, stream): + self._stream = stream + def write(self, text): + return self._stream.write(self._ctrl.sub("", text)) + def flush(self): + self._stream.flush() +sys.stdout = _CleanOut(sys.stdout) + data_dir = sys.argv[1] window_days = int(sys.argv[2]) output_format = sys.argv[3] # "prose" or "json" @@ -765,6 +827,20 @@ PYEOF import json, os, sys from collections import defaultdict +# Captured text is untrusted: strip ANSI/OSC escapes and other control +# characters (keeping newline and tab) before it reaches a terminal or +# hook context. +import re +class _CleanOut: + _ctrl = re.compile(r"\x1b\[[0-?]*[ -/]*[@-~]|\x1b\][^\x07\x1b]*(?:\x07|\x1b\\)|[\x00-\x08\x0b-\x1f\x7f-\x9f]") + def __init__(self, stream): + self._stream = stream + def write(self, text): + return self._stream.write(self._ctrl.sub("", text)) + def flush(self): + self._stream.flush() +sys.stdout = _CleanOut(sys.stdout) + data_dir = sys.argv[1] misses_path = os.path.join(data_dir, "misses.jsonl") @@ -832,6 +908,20 @@ PYEOF python3 - "$CMA_DIR" <<'PYEOF' import json, os, sys +# Captured text is untrusted: strip ANSI/OSC escapes and other control +# characters (keeping newline and tab) before it reaches a terminal or +# hook context. +import re +class _CleanOut: + _ctrl = re.compile(r"\x1b\[[0-?]*[ -/]*[@-~]|\x1b\][^\x07\x1b]*(?:\x07|\x1b\\)|[\x00-\x08\x0b-\x1f\x7f-\x9f]") + def __init__(self, stream): + self._stream = stream + def write(self, text): + return self._stream.write(self._ctrl.sub("", text)) + def flush(self): + self._stream.flush() +sys.stdout = _CleanOut(sys.stdout) + data_dir = sys.argv[1] def load_jsonl(path): @@ -978,6 +1068,20 @@ import json, os, sys from collections import Counter from datetime import datetime, timedelta, timezone +# Captured text is untrusted: strip ANSI/OSC escapes and other control +# characters (keeping newline and tab) before it reaches a terminal or +# hook context. +import re +class _CleanOut: + _ctrl = re.compile(r"\x1b\[[0-?]*[ -/]*[@-~]|\x1b\][^\x07\x1b]*(?:\x07|\x1b\\)|[\x00-\x08\x0b-\x1f\x7f-\x9f]") + def __init__(self, stream): + self._stream = stream + def write(self, text): + return self._stream.write(self._ctrl.sub("", text)) + def flush(self): + self._stream.flush() +sys.stdout = _CleanOut(sys.stdout) + data_dir = sys.argv[1] window_days = int(sys.argv[2]) diff --git a/docs/DATA.md b/docs/DATA.md index 15707af..d294334 100644 --- a/docs/DATA.md +++ b/docs/DATA.md @@ -18,6 +18,8 @@ $CMA_DIR/ All files are JSON Lines (JSONL): one JSON object per line, append-only, never edited in place. The format is durable, recoverable line by line, and parseable by any JSON-aware tool. +Captures can hold conversation excerpts, so cma creates a new data directory with mode 0700 and new data files with mode 0600. Existing directories and files keep their modes; tighten them with `chmod 700 "$CMA_DIR" && chmod 600 "$CMA_DIR"/*.jsonl`. + ## 2. Schema Every record begins with `schema_version`, `type`, `id`, and `timestamp`. Type-specific fields follow. diff --git a/hooks/claude-code-pre-tool-use.sh b/hooks/claude-code-pre-tool-use.sh index 03031ad..472ca8f 100755 --- a/hooks/claude-code-pre-tool-use.sh +++ b/hooks/claude-code-pre-tool-use.sh @@ -137,6 +137,10 @@ output = run_surface(args) if not output: sys.exit(0) +# Captured text is untrusted: strip ANSI/OSC escapes and control characters +# (keeping newline and tab) before handing it to the assistant. +import re +output = re.sub(r"\x1b\[[0-?]*[ -/]*[@-~]|\x1b\][^\x07\x1b]*(?:\x07|\x1b\\)|[\x00-\x08\x0b-\x1f\x7f-\x9f]", "", output) context = "Relevant prior captures from cma (action-time injection):\n\n" + output print(json.dumps({ "hookSpecificOutput": { diff --git a/scripts/test.sh b/scripts/test.sh index e3d044a..7ada6e8 100755 --- a/scripts/test.sh +++ b/scripts/test.sh @@ -1105,6 +1105,37 @@ else fail=$((fail + 1)) fi +# --------------------------------------------------------------------------- +# Data privacy and output hygiene +# --------------------------------------------------------------------------- + +priv_root=$(mktemp -d) +priv_dir="$priv_root/fresh/data" +CMA_DIR="$priv_dir" "$CMA" miss $'esc \e[2J\e]0;pwned\a text\x07 end' --surface auth --excerpt "secret" >/dev/null +CMA_DIR="$priv_dir" "$CMA" surface --surface auth >/dev/null +perm_dir=$(python3 -c "import os,sys; print(oct(os.stat(sys.argv[1]).st_mode & 0o777))" "$priv_dir") +perm_files=$(python3 -c "import os,sys; print(sorted({oct(os.stat(os.path.join(sys.argv[1], f)).st_mode & 0o777) for f in os.listdir(sys.argv[1]) if f.endswith('.jsonl')}))" "$priv_dir") +if [[ "$perm_dir" == "0o700" && "$perm_files" == "['0o600']" ]]; then + printf "PASS %s\n" "new data dir is 0700 and data files 0600" + pass=$((pass + 1)) +else + printf "FAIL %s (dir=%s files=%s)\n" "new data dir is 0700 and data files 0600" "$perm_dir" "$perm_files" + fail=$((fail + 1)) +fi +surf_out=$(CMA_DIR="$priv_dir" "$CMA" surface --no-log) +CMA_DIR="$priv_dir" "$CMA" miss "second" --surface auth --fm f --intended $'\e[31mred\e[0m' >/dev/null +CMA_DIR="$priv_dir" "$CMA" miss "third" --surface auth --fm f >/dev/null +all_out="$surf_out$(CMA_DIR="$priv_dir" "$CMA" stats --behavior)$(CMA_DIR="$priv_dir" CMA_SESSION_START_SECTIONS=all bash "$SS_HOOK" Date: Fri, 6 Nov 2026 19:42:31 +0100 Subject: [PATCH 7/9] Update the README test count Signed-off-by: lluvr --- README.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/README.md b/README.md index 3e7efa0..807e9a9 100644 --- a/README.md +++ b/README.md @@ -15,7 +15,7 @@ cma is methodology-agnostic. It owns the data substrate (how captures are stored ## Status -cma 1.0 reference implementation. All seven primitives fully functional: `cma miss`, `cma decision`, `cma reject`, `cma prevented`, `cma surface`, `cma distill` (default + `--review` + `--retire`), and `cma stats` (default + `--rejections` + `--preventions` + `--recurrence` + `--leaks` + `--behavior`). Action-time injection (Claude Code hook + shell wrapper). Texture preservation on misses. Test suite (143 cases) covers functional paths, edge cases, JSON validity, the leak-detection join, hook integration, and shell wrapper modes. +cma 1.0 reference implementation. All seven primitives fully functional: `cma miss`, `cma decision`, `cma reject`, `cma prevented`, `cma surface`, `cma distill` (default + `--review` + `--retire`), and `cma stats` (default + `--rejections` + `--preventions` + `--recurrence` + `--leaks` + `--behavior`). Action-time injection (Claude Code hook + shell wrapper). Texture preservation on misses. Test suite (173 cases) covers functional paths, edge cases, JSON validity, the leak-detection join, hook integration, and shell wrapper modes. The full surface is specified in [DESIGN.md](docs/DESIGN.md). Additive features (action-time injection, texture preservation, counterfactual capture, recurrence detection) layer on without changing the locked surface. From f765d3640cb4c236059cbffdfe3e0bbfb9670aaf Mon Sep 17 00:00:00 2001 From: lluvr Date: Sat, 7 Nov 2026 10:36:44 +0100 Subject: [PATCH 8/9] Say what the loop numbers are and propose how to measure the rest The README called cma stats --leaks the validation evidence that the loop closes and called behaviour change the central claim, while docs/ARCHITECTURE.md lists the outcome layer as aspirational. The prevention, leak, and closure-rate numbers are counts of records the user writes, so describe them that way: a new README section says what each number is and what is not yet measured, and DESIGN, ARCHITECTURE, and the cma-mcp README drop the matching claims. docs/EVALUATION.md proposes the experiment that would measure the claim: sessions with the PreToolUse hook on or off at random, repeat misses of an already-captured failure shape as the outcome, what to log, rough sample sizes, and the threats to validity. Signed-off-by: lluvr --- CHANGELOG.md | 1 + CONTRIBUTING.md | 2 +- README.md | 25 ++++++++++++++------ cma-mcp/README.md | 6 ++--- docs/ARCHITECTURE.md | 14 +++++------ docs/DESIGN.md | 4 ++-- docs/EVALUATION.md | 56 ++++++++++++++++++++++++++++++++++++++++++++ 7 files changed, 88 insertions(+), 20 deletions(-) create mode 100644 docs/EVALUATION.md diff --git a/CHANGELOG.md b/CHANGELOG.md index c4d6179..bb9fa7e 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -24,6 +24,7 @@ The format follows [Keep a Changelog](https://keepachangelog.com/en/1.1.0/) and ### Changed +- The README and `docs/` now state that the prevention, leak, and closure-rate numbers are self-reported counts, and that whether surfaced warnings reduce repeat failures is not yet measured. `docs/EVALUATION.md` proposes how to measure it. - Repository home moved to a personal account (`lluvr/cma`). Self-references, badges, and issue/security links updated; author and publisher framing is now personal (organization publisher attribution dropped). - Zenodo DOI references removed from `CITATION.cff` pending re-issuance under the new account; `SECURITY.md`'s publisher-identity claim softened accordingly. - Repository layout: the deep-dive docs (`DESIGN`, `ARCHITECTURE`, `DATA`, `DECISIONS`, `GOVERNANCE`) moved under `docs/`, and `test.sh` / `bench.sh` moved under `scripts/`. All internal references updated. diff --git a/CONTRIBUTING.md b/CONTRIBUTING.md index fbd4f9e..74339e9 100644 --- a/CONTRIBUTING.md +++ b/CONTRIBUTING.md @@ -26,7 +26,7 @@ cma/ ├── cma bash CLI (the seven primitives) ├── hooks/ Claude Code + shell preexec integrations ├── scripts/ test.sh, bench.sh, canon_audit.sh -├── docs/ DESIGN.md ARCHITECTURE.md DATA.md DECISIONS.md GOVERNANCE.md +├── docs/ DESIGN.md ARCHITECTURE.md DATA.md DECISIONS.md GOVERNANCE.md EVALUATION.md ├── CHANGELOG.md cma's release history ├── README.md cma overview ├── cma-mcp/ Python MCP distribution wrapper diff --git a/README.md b/README.md index 807e9a9..f89c9cc 100644 --- a/README.md +++ b/README.md @@ -63,7 +63,7 @@ Run `cma --help` for the full command surface. ## The compound loop in four commands -cma's central claim is that captured failures become surfaced warnings, surfaced warnings change behavior, and the resulting evidence is computable. The full chain in one walkthrough: +cma is built on a chain: captured failures become surfaced warnings, surfaced warnings change what you do next, and the record of both can be computed over. cma does the capturing, the surfacing, and the computing. Whether the warnings change behavior is the part it does not yet measure: the numbers it reports are counts of what you recorded (see [What the numbers are](#what-the-numbers-are)). The full chain in one walkthrough: ```bash # 1. Capture a failure so future similar work can see it. @@ -75,10 +75,10 @@ cma miss "claimed verified without testing the cross-tenant path" \ # works too. The surface event is logged for leak detection. cma surface --surface auth -# 3. The warning fires, behavior changes, and the repeat is avoided. -# Link the prevention to the original miss so cma can attribute -# the closure. `cma id miss` returns the most recent miss id; -# use it to chain without scrolling back through terminal output. +# 3. The warning fires. If it changed what you did, record that +# and link it to the original miss so cma can attribute it. +# `cma id miss` returns the most recent miss id; use it to chain +# without scrolling back through terminal output. cma prevented "almost claimed verified again; ran the cross-tenant test instead" \ --miss-id "$(cma id miss --surface auth)" @@ -89,9 +89,20 @@ cma stats --evidence # of which evidenced: 1 # Leaks: 0 # Loop closure rate: 100% (1/1) +# self-attested: 1 evidenced prevention(s) you recorded, 0 leak(s) ``` -The closure rate counts only preventions evidenced by a surface event between the linked miss and the prevention. Self-attestation without that chain is captured but does not inflate the rate. `cma stats --evidence --json` emits the structured record for downstream consumers. +`cma stats --evidence --json` emits the structured record for downstream consumers. + +### What the numbers are + +Every number above is a count of records you wrote, or a rate computed from those counts. + +- A **prevention** is your own report that a warning changed what you did. cma checks one thing about it: that the linked miss was actually surfaced between the miss and the prevention. A prevention with that surface event is counted as evidenced; one without it is still captured but stays out of the rate. +- A **leak** is a miss you captured after a warning for the same surface and failure mode had been surfaced. +- The **loop closure rate** is evidenced preventions divided by evidenced preventions plus leaks. It is self-reported, and the output labels it that way: it moves with what you choose to record, and it does not compare against what would have happened without the warning. + +What works and is checked by the test suite: capture, action-time surfacing, the surface-event log, and the joins that produce these counts. What is not yet measured: whether surfaced warnings reduce repeat failures. [EVALUATION.md](docs/EVALUATION.md) proposes how to measure that. ## Action-time injection @@ -184,7 +195,7 @@ Triggers fire on commands likely to warrant surfacing: editors (`vim`, `nvim`, ` Failure isolation: if cma is missing, errors, or times out (default 5 seconds), the wrapped command still runs cleanly. The wrapper never blocks an action on its own failure. -Both integrations log surface events to `~/.cma/surface_events.jsonl`. `cma stats --leaks` later joins these events against subsequent misses to flag failures that occurred despite a relevant warning being surfaced, the validation evidence that the loop closes. +Both integrations log surface events to `~/.cma/surface_events.jsonl`. `cma stats --leaks` later joins these events against subsequent misses to flag failures that occurred despite a relevant warning being surfaced. A leak shows where a warning did not prevent a repeat; the leak and prevention counts are self-reported and do not by themselves show that warnings prevent repeats (see [What the numbers are](#what-the-numbers-are)). ## MCP distribution (cma-mcp) diff --git a/cma-mcp/README.md b/cma-mcp/README.md index 06109d4..eb5dd87 100644 --- a/cma-mcp/README.md +++ b/cma-mcp/README.md @@ -5,8 +5,8 @@ The Model Context Protocol layer for cma, an executable compound practice loop: capture failures, surface them at the moment of action, -track decisions and rejections, and check whether warnings actually -prevented repeats. +track decisions and rejections, and count the repeats you report as +prevented and the ones that recurred anyway. cma-mcp wraps the canonical `cma` command-line tool and exposes its seven primitives to MCP-compatible AI clients. It does not reimplement @@ -76,7 +76,7 @@ command; the stdio handshake runs). | `cma_prevented` | `cma prevented` | A surfaced warning actually changed behavior | | `cma_distill` | `cma distill` (modes: default / retire / review) | Promote, retire, or preview a distilled learning | | `cma_surface` | `cma surface` | Pull relevant prior captures before acting (logs surface event for leak detection) | -| `cma_stats` | `cma stats` (views: default / leaks / recurrence / preventions / rejections / behavior) | Inspect loop-closing evidence | +| `cma_stats` | `cma stats` (views: default / leaks / recurrence / preventions / rejections / behavior) | Inspect loop counts (self-reported) | **Four resources** for read-only context: diff --git a/docs/ARCHITECTURE.md b/docs/ARCHITECTURE.md index 7449761..fcda65e 100644 --- a/docs/ARCHITECTURE.md +++ b/docs/ARCHITECTURE.md @@ -234,23 +234,23 @@ The architecture is designed to produce data supporting three independent layers **Metrics**, computable from texture-preserved captures: -- Counterfactual rate: misses with both `intended` and `corrected` set, where the two differ. Direct evidence of mid-course correction. -- Prevention-to-leak ratio: `cma prevented` captures versus leak detections. Higher ratio means warnings are working. +- Counterfactual rate: misses with both `intended` and `corrected` set, where the two differ. A record of mid-course correction, as you described it. +- Prevention-to-leak ratio: `cma prevented` captures versus leak detections. A higher ratio is consistent with warnings working, but both counts depend on what you chose to record. - Decision-shift rate: decisions captured that contradict prior decisions (your reflection). -**Strength**: structural data; not self-report alone. **Weakness**: requires you to capture texture (the `--intended`, `--corrected` flags). Coverage depends on capture discipline. +**Strength**: computed from structured fields, not free text. **Weakness**: every input is self-reported. You decide what counts as a miss, a prevention, or a correction, so these metrics measure your recording as much as your behavior. They also require you to capture texture (the `--intended`, `--corrected` flags), and coverage depends on capture discipline. ### 5.3 Outcome layer **Question**: Does your work get measurably better? -**Metrics**, requiring [Touchstone](https://github.com/Clarethium/touchstone) integration: +**Metrics**, requiring an external measure of work quality or of repeat failures: -- Touchstone score trends on your actual outputs over time. +- Quality-score trends on your actual outputs over time, from an evaluation suited to the output type. - Score deltas correlated with cma usage intensity (captures per task, surfacings per task). - Failure-shape distribution shifts (fewer recurrences of named failure shapes). -**Strength**: objective, model-independent measurement. **Weakness**: requires your actual work to be amenable to Touchstone evaluation, which depends on output type. Currently aspirational; the bridge between cma, a methodology's catalog, and Touchstone measurement is the next major architectural project after this one. +**Strength**: measures the result rather than your report of it. **Weakness**: requires your actual work to be amenable to external evaluation, which depends on output type. Currently aspirational: nothing in cma measures this layer, so whether surfaced warnings change behavior is not yet established. [EVALUATION.md](EVALUATION.md) proposes a first experiment: compare repeat failures with the action-time hook on and off. The three layers are independent. A system that passes the Process layer alone is a journal. A system that passes Process + Behavior is a working compound practice loop. A system that passes all three is a methodology with empirical grounding. @@ -352,4 +352,4 @@ The classifier is external to cma: this repository ships no classifier. Methodol ## 10. References - [DESIGN.md](DESIGN.md): the seven cma 1.0 primitives. -- [Touchstone](https://github.com/Clarethium/touchstone): the measurement infrastructure for the outcome layer of validation. +- [EVALUATION.md](EVALUATION.md): a proposed experiment for the behavior and outcome layers. diff --git a/docs/DESIGN.md b/docs/DESIGN.md index bc10338..b62c9c0 100644 --- a/docs/DESIGN.md +++ b/docs/DESIGN.md @@ -84,7 +84,7 @@ cma reject ### cma prevented -Capture a catch. A moment where a surfaced warning prevented a recurrence. Without prevention captures, compound learning is a claim; with them, it is evidence. +Capture a catch. A moment where a surfaced warning prevented a recurrence. A prevention is your own report that a warning changed what you did; the closure rate in `cma stats --evidence` is computed from these reports, so it is self-reported rather than an independent measurement. ``` cma prevented @@ -170,7 +170,7 @@ cma stats - **Default mode**: summary dashboard. Total captures by type, recent activity, most-active surfaces, top failure shapes, prevention rate, recurrence trends. - **`--rejections` view**: list of active rejections with surfaces, ages, and revisit triggers. - **`--leaks` view**: failures that occurred despite an active warning. Each leak increments the warning's weight. -- **`--preventions` view**: captured preventions with linked misses. Evidence of the loop closing. +- **`--preventions` view**: captured preventions with linked misses: your own record of warnings that changed what you did. - **`--recurrence` view**: failure shapes ordered by recurrence rate. Identifies preventions that are not working. - **`--behavior` view**: behavior-layer signals from misses carrying intended/corrected texture, grouped by surface and failure mode. diff --git a/docs/EVALUATION.md b/docs/EVALUATION.md new file mode 100644 index 0000000..7ae993b --- /dev/null +++ b/docs/EVALUATION.md @@ -0,0 +1,56 @@ +# Evaluation proposal + +Status: proposal. Nothing described here is built. + +## What is and is not measured today + +cma captures failures, surfaces matching ones before an action, logs each surfacing, and joins those logs into counts (`cma stats --evidence`, `--leaks`, `--recurrence`). Those counts are self-reported: a prevention is your note that a warning helped, and a miss is a failure you chose to record. They show that the loop runs. They do not show that surfacing a warning makes the same failure less likely to happen again, because there is no comparison against sessions where nothing was surfaced. + +This document proposes the smallest experiment that would answer that question. + +## Design + +Run sessions with the Claude Code `PreToolUse` hook on or off, chosen at random when each session starts (a coin flip, recorded before any work happens). Strict alternation also works, but a coin flip avoids lining up with weekly patterns such as "Mondays are refactor days". + +Hold everything else constant across both arms. In particular, the `SessionStart` hook also surfaces recurring failures, so either keep it off in both arms or on in both arms for the whole experiment. + +## Outcome + +The outcome is a **repeat**: a new miss whose `(surface, fm)` pair matches a miss that was captured before the session started. This is the same pairing `cma stats --leaks` already uses. The comparison is the repeat rate per session (or per 100 tool calls, if session length varies a lot) in the on arm versus the off arm. + +Preventions are not the outcome. They are recorded only when a warning appeared, so the off arm cannot produce them and the comparison would be rigged. + +## What to log + +Most of the data already exists: + +- `misses.jsonl`: timestamp, `surface`, `fm` for every captured failure. +- `surface_events.jsonl`: what was surfaced and when, in the on arm. + +Three additions are needed, per session: + +- a session id, start and end time, and the arm (`on` or `off`); +- the number of tool calls, as an exposure measure; +- in the off arm, a shadow record of what the hook would have surfaced (computed but not shown). Without it you cannot tell whether an off session ever touched a captured failure shape, and sessions that never touch one carry no information in either arm. + +Claude Code passes a session id to hooks on stdin, so the hook is the natural place to write these records. + +## How much data + +Repeats are rare, so the sample has to be larger than intuition suggests. A rough guide for comparing two per-session rates at the usual thresholds (5% false-positive rate, 80% chance of detecting a real effect): + +| Baseline repeats per session | Effect to detect | Sessions per arm | +|---|---|---| +| 0.3 | halved (0.3 to 0.15) | about 160 | +| 0.3 | cut by a third (0.3 to 0.2) | about 400 | +| 0.1 | halved (0.1 to 0.05) | about 470 | + +Measure the baseline first: the off-arm repeat rate over the first 30 or so sessions tells you which row you are in. As a floor, do not read the result before both arms together contain at least 30 repeats; below that, one bad week decides the outcome. Count only sessions that touched a captured failure shape (from the surface events and shadow records), and analyze per session rather than per miss, since misses within one session are not independent. + +## Threats to validity + +- **Self-report.** Repeats are still misses you choose to record. If you know which arm a session is in, you may record more carefully in one of them. Mitigate by keeping the arm out of view during the session (the assistant sees the warnings; you need not), by deciding in advance what counts as a miss, and by reviewing captures for a session before revealing its arm. +- **Selection.** Only failures you noticed and captured can repeat. Shapes that are easy to notice will dominate the outcome, and failures you never recorded are invisible to both arms. Random assignment keeps this balanced between arms, but limits the result to the kinds of failure you capture. +- **Novelty.** Early on, the act of capturing failures makes you more careful in both arms, and repeat rates may fall for reasons unrelated to the hook. Run long enough to see the rates settle, and check whether the difference between arms holds in the second half on its own. +- **Contamination.** You remember warnings from on sessions during off sessions, and other memory (project notes, assistant memory files) can carry the same lessons. This biases the result toward no difference, so a positive finding is conservative; a null finding is weaker. +- **Scope.** Fewer repeats of captured shapes is not the same as better work. This experiment tests one link in the chain: whether surfacing a warning reduces the repeat it warns about. From d905d92c5fd69a6bea4514ab0c06ae9c3ba0d721 Mon Sep 17 00:00:00 2001 From: lluvr Date: Sat, 7 Nov 2026 10:37:39 +0100 Subject: [PATCH 9/9] Make the root docs standalone Apply the approach of 5a37d21 to the rest of the repository's docs: drop the named references to related projects from the README, NOTICE, CITATION.cff, GOVERNANCE, DECISIONS, DESIGN, ARCHITECTURE, and the pull request template. The fm tag is described as belonging to whatever methodology the user practices, with no catalog named. Remove the brand contact address. Security reports go through GitHub private vulnerability reporting (cma-mcp README now points there, as SECURITY.md already did); conduct reports go to the maintainer through GitHub. The author website pointing at the brand blog is removed from both CITATION.cff files; author and copyright names are unchanged. Signed-off-by: lluvr --- .github/PULL_REQUEST_TEMPLATE.md | 8 -------- CITATION.cff | 2 -- CODE_OF_CONDUCT.md | 7 +++++-- NOTICE | 35 ++++++-------------------------- README.md | 18 ++++------------ cma-mcp/CITATION.cff | 1 - cma-mcp/README.md | 7 ++++++- docs/ARCHITECTURE.md | 4 ++-- docs/DECISIONS.md | 9 ++++---- docs/DESIGN.md | 2 +- docs/GOVERNANCE.md | 10 +-------- 11 files changed, 29 insertions(+), 74 deletions(-) diff --git a/.github/PULL_REQUEST_TEMPLATE.md b/.github/PULL_REQUEST_TEMPLATE.md index c607430..1e99924 100644 --- a/.github/PULL_REQUEST_TEMPLATE.md +++ b/.github/PULL_REQUEST_TEMPLATE.md @@ -41,14 +41,6 @@ before opening a substantial change. Especially: - [ ] No new runtime dependency added (bash cma: bash + python3 stdlib only; cma-mcp: Python stdlib only) - [ ] `bash scripts/canon_audit.sh` passes locally -## Related-project impact - - - ## DECISIONS