diff --git a/SKILL.md b/SKILL.md index daa87f7..44436e4 100644 --- a/SKILL.md +++ b/SKILL.md @@ -130,12 +130,13 @@ Selecting a profile: the user's words above, env `FABLE_ROUTING=quality|balanced ## Enforcement layer (hooks — mechanics in `hooks/README.md`) -Four hooks turn the most-shirked rules into hard blocks. Armed **per project** by a `.fable/` directory (searched upward, bounded at the git root); without it they pass through silently. Pressure applies **per round** via `.fable/LEDGER.md`: +Five hooks turn the most-shirked rules into hard blocks. Armed **per project** by a `.fable/` directory (searched upward, bounded at the git root); without it they pass through silently. Pressure applies **per round** via `.fable/LEDGER.md`: ``` - [ ] 1. card (machine-checkable acceptance) <- open: guards enforce -- [x] 2. done -- evidence: pytest 21/21 <- [x] REQUIRES a substantive evidence note +- [x] 2. done -- evidence: `pytest -q` 21/21 <- [x] REQUIRES substantive evidence; a cited `command` is checked against the machine-written evidence log - [~] 3. not this round -- deferred: reason +REPLAY: on <- optional: re-run cited acceptances before the round may end PAUSED: reason <- a line anywhere: enforcement off ``` @@ -145,7 +146,8 @@ missing.) - **Spawn Guard** (PreToolUse Agent/Task/Workflow): blocks a detailed spawn while the ledger has no **open** cards — no ledger, and equally a ledger holding only a finished round's closed cards (design gate: new fan-out needs a live card) — and blocks any spawn requesting a **model stronger than the session's** (model ceiling — checked on the `model` param and `model:` literals in Workflow scripts; stays active even when paused, it protects quota, not workflow). - **Fail-Streak Reminder** (PostToolUse Bash, advisory): every 3rd consecutive failing command injects the attribution ladder — stops grinding on the wrong layer mechanically, not by willpower. -- **Close Guard** (Stop): blocks ending the turn while open `- [ ]` items remain, **and** while any `- [x]` lacks an `-- evidence:` note (evidence-on-close: adjectives don't close cards). +- **Evidence Logger** (PostToolUse Bash, passive): appends every command's real outcome (command, exit code, output tail) to `.fable/evidence.jsonl` — the machine-written record the Close Guard checks citations against. Records even while paused; evidence gaps are worse than pauses. +- **Close Guard** (Stop): blocks ending the turn while open `- [ ]` items remain, while any `- [x]` lacks an `-- evidence:` note (evidence-on-close: adjectives don't close cards), while any cited evidence `command` has **no successful run in the evidence log** (machine corroboration: a citation that never ran is not evidence), and — with `REPLAY: on` — while any cited acceptance fails when **re-run now** ('passed once' is not 'still passes'). - **Profile Injector** (SessionStart): injects tier + routing + habits, **sized to the ledger state** — full when a round is starting/active, minimal when idle, one line when paused. **Wrap-up lint**: `python3 /hooks/fable_lint.py ` — machine-checks the discipline itself (SPEC source tags present, open cards name acceptance, closed cards carry evidence). Run it at step 7 of the execution template; findings are open work. diff --git a/hooks/README.md b/hooks/README.md index dfcd261..8cb113d 100644 --- a/hooks/README.md +++ b/hooks/README.md @@ -4,14 +4,15 @@ The enforcement layer: turn a few of fable-mode's prose rules into Claude Code hooks that actually block — ledger-before-delegation and close-verification, built around this repo's SPEC.md/PROGRESS.md conventions. -## Four hooks + one lint CLI +## Five hooks + one lint CLI | Hook | Event | What it does | |---|---|---| | `fable_profile_inject.py` | `SessionStart` | When the project has opted in, **auto-inject the tier by model + the six levers + ledger context recovery** (no need to type "use fable mode") | | `fable_spawn_guard.py` | `PreToolUse` (Agent\|Task\|Workflow) | When opted in: **block a detailed spawn with no ledger** (forces the plan gate) and **block any spawn requesting a model stronger than the session's** (the model ceiling) | | `fable_fail_streak.py` | `PostToolUse` (Bash) | Advisory, never blocks: at every 3rd **consecutive failing command**, inject the attribution ladder (harness → deployment → product; fix the class via an invariant). Streak state: `$TMPDIR/fable-mode-sessions/.fails`, reset on success. | -| `fable_close_guard.py` | `Stop` | While the ledger still has unchecked items, **block ending the turn** (cures early stopping / spinning). When all items are checked, **block if any `- [x]` lacks an evidence marker** (`-- evidence:` / `证据:`) — evidence-on-close. | +| `fable_evidence_log.py` | `PostToolUse` (Bash) | Passive recorder: appends every command's **real outcome** (command, exit code, output tail) to `.fable/evidence.jsonl` — the machine-written record citations are checked against. Records even while PAUSED. | +| `fable_close_guard.py` | `Stop` | While the ledger still has unchecked items, **block ending the turn** (cures early stopping / spinning). When all items are checked: **block if any `- [x]` lacks an evidence marker** (`-- evidence:` / `证据:`), **block if a cited evidence `command` has no successful run in the evidence log** (machine corroboration), and with `REPLAY: on` **block if a cited acceptance fails when re-run now**. | `fable_lint.py` is **not a hook** — a one-shot CLI (`python3 fable_lint.py `) for wrap-up or CI: SPEC exists and carries source tags ([measured]/[inferred]/[not-shown] @@ -53,6 +54,7 @@ defaults to the conservative tier. This is SessionStart-only info (there is no - [x] 2. done -- evidence: pytest 21/21 - [~] 3. not this round -- deferred: reason PAUSED: reason <- optional line anywhere: suspend enforcement +REPLAY: on <- optional: re-run cited acceptances at turn-end ROUTING: frugal <- optional: model-routing profile for this round TIER: throughput <- optional: concurrency tier for this round ``` @@ -141,7 +143,10 @@ use your actual absolute clone path if it differs: "command": "python3 ${CLAUDE_CONFIG_DIR:-$HOME/.claude}/skills/fable-mode/hooks/fable_spawn_guard.py"}]}], "PostToolUse": [{"matcher": "Bash", "hooks": [{"type": "command", - "command": "python3 ${CLAUDE_CONFIG_DIR:-$HOME/.claude}/skills/fable-mode/hooks/fable_fail_streak.py"}]}], + "command": "python3 ${CLAUDE_CONFIG_DIR:-$HOME/.claude}/skills/fable-mode/hooks/fable_fail_streak.py"}]}, + {"matcher": "Bash", + "hooks": [{"type": "command", + "command": "python3 ${CLAUDE_CONFIG_DIR:-$HOME/.claude}/skills/fable-mode/hooks/fable_evidence_log.py"}]}], "Stop": [{"hooks": [{"type": "command", "command": "python3 ${CLAUDE_CONFIG_DIR:-$HOME/.claude}/skills/fable-mode/hooks/fable_close_guard.py"}]}] } @@ -160,6 +165,7 @@ To disable entirely, remove the hooks block from settings.json. No third-party deps, just run: ```bash -python3 tests/test_guards.py # 13 cases: opt-in detection, ledger presence, small-spawn/fork exemptions, git-root boundary, loop-safety, fail-open -python3 tests/test_inject.py # 9 cases: per-model tier, env override, ledger context recovery, JSON envelope, fail-open +python3 tests/test_guards.py # opt-in detection, ledger presence, exemptions, git-root boundary, loop-safety, fail-open +python3 tests/test_inject.py # per-model tier, env override, ledger context recovery, JSON envelope, fail-open +python3 tests/test_evidence.py # evidence log recording, citation corroboration, REPLAY re-runs ``` diff --git a/hooks/_fable_common.py b/hooks/_fable_common.py index 342cf27..40517d2 100755 --- a/hooks/_fable_common.py +++ b/hooks/_fable_common.py @@ -125,6 +125,128 @@ def load_session_model(session_id): EVIDENCE_RE = re.compile(r"(evidence|verified|证据|凭证|验证)\s*[:：]", re.IGNORECASE) +# --- machine-written evidence log (.fable/evidence.jsonl) --- +# +# The Evidence Logger hook appends one JSON line per Bash command: +# {"ts": , "cmd": , "exit": , "tail": } +# The Close Guard checks cited `commands` on `- [x]` cards against this log, +# so "the acceptance actually ran" is machine truth, not a self-reported note. + +EVIDENCE_LOG = "evidence.jsonl" +EVIDENCE_LOG_MAX_BYTES = 512 * 1024 # rotate: keep the newest half beyond this +EVIDENCE_TAIL_CHARS = 200 + +_BACKTICK_RE = re.compile(r"`([^`]+)`") + + +def evidence_log_path(fable_dir): + return os.path.join(fable_dir, EVIDENCE_LOG) + + +def response_exit_code(tool_response): + """Best-effort exit code from a Bash tool_response; None when unknown.""" + r = tool_response + if isinstance(r, str): + m = re.search(r"[Ee]xit code[: ]+([0-9]+)", r) + return int(m.group(1)) if m else None + if not isinstance(r, dict): + return None + for key in ("exitCode", "exit_code", "code", "returncode"): + v = r.get(key) + if isinstance(v, int): + return v + for key in ("is_error", "isError"): + if r.get(key) is True: + return 1 + text = " ".join(str(r.get(k, "")) for k in ("stdout", "stderr", "output")) + m = re.search(r"[Ee]xit code[: ]+([0-9]+)", text) + return int(m.group(1)) if m else None + + +def append_evidence(fable_dir, cmd, exit_code, tail): + """Append one run record; rotate the log when it grows too large. + Best-effort, fail-open — recording must never disturb the session.""" + try: + path = evidence_log_path(fable_dir) + try: + if os.path.getsize(path) > EVIDENCE_LOG_MAX_BYTES: + with open(path, encoding="utf-8", errors="replace") as fh: + lines = fh.readlines() + with open(path, "w", encoding="utf-8") as fh: + fh.writelines(lines[len(lines) // 2:]) + except OSError: + pass + rec = {"ts": time.time(), "cmd": str(cmd)[:2000], + "exit": exit_code, + "tail": str(tail or "")[-EVIDENCE_TAIL_CHARS:]} + with open(path, "a", encoding="utf-8") as fh: + fh.write(json.dumps(rec, ensure_ascii=False) + "\n") + except Exception: + pass + + +def _norm_cmd(s): + return re.sub(r"\s+", " ", str(s)).strip() + + +def cited_commands(card_line): + """Backtick-quoted commands in the *evidence part* of a `- [x]` line. + Returns [] when the evidence note cites no command (prose-only note).""" + m = EVIDENCE_RE.search(card_line) + if not m: + return [] + return [_norm_cmd(c) for c in _BACKTICK_RE.findall(card_line[m.end():]) + if _norm_cmd(c)] + + +def evidence_log_has_run(log_path, cited, want_success=True): + """True if the log records a run whose command matches `cited` + (normalized substring, either direction) — successful when want_success.""" + try: + with open(log_path, encoding="utf-8", errors="replace") as fh: + for line in fh: + try: + rec = json.loads(line) + except ValueError: + continue + cmd = _norm_cmd(rec.get("cmd", "")) + if not cmd: + continue + if cited in cmd or cmd in cited: + if not want_success or rec.get("exit") == 0: + return True + except Exception: + return False + return False + + +def uncorroborated_citations(ledger_p, log_path): + """`- [x]` cards whose cited evidence command never ran successfully. + + Machine check for "the acceptance actually ran": a card that cites a + `command` as evidence must have a successful run of that command in the + evidence log. Cards with prose-only evidence are not checked here (the + substantive-string rule still applies to them). Returns [] when the log + doesn't exist yet (projects predating the logger) — fail-open. + """ + if not os.path.isfile(log_path): + return [] + bad = [] + try: + with open(ledger_p, encoding="utf-8", errors="replace") as fh: + for line in fh: + s = line.strip() + if s[:5].lower() != "- [x]": + continue + cites = cited_commands(s) + if cites and not any( + evidence_log_has_run(log_path, c) for c in cites): + bad.append(s) + except Exception: + return [] + return bad + + # --- model-routing profiles (quality / balanced / frugal) --- ROUTING_PROFILES = ("quality", "balanced", "frugal") @@ -151,6 +273,22 @@ def read_tier(path): return None +_REPLAY_RE = re.compile(r"^REPLAY\s*[:：]\s*(on|off)\b", re.IGNORECASE) + + +def read_replay(path): + """True when the ledger opts into acceptance replay (`REPLAY: on`).""" + try: + with open(path, "r", encoding="utf-8", errors="replace") as fh: + for line in fh: + m = _REPLAY_RE.match(line.strip()) + if m: + return m.group(1).lower() == "on" + except Exception: + return False + return False + + def read_routing(path): """Per-round routing profile from a `ROUTING: ` ledger line. diff --git a/hooks/fable_close_guard.py b/hooks/fable_close_guard.py index e0158a4..0a8bba9 100755 --- a/hooks/fable_close_guard.py +++ b/hooks/fable_close_guard.py @@ -22,10 +22,66 @@ sys.path.insert(0, os.path.dirname(os.path.abspath(__file__))) from _fable_common import ( # noqa: E402 read_hook_input, start_dir, find_fable_dir, ledger_path, parse_ledger, - closed_without_evidence, + closed_without_evidence, evidence_log_path, uncorroborated_citations, + read_replay, cited_commands, ) MAX_LIST = 12 +REPLAY_CMD_TIMEOUT = 30 # seconds per acceptance command (FABLE_REPLAY_TIMEOUT) +REPLAY_TOTAL_BUDGET = 120 # seconds across all replays in one stop + + +def replay_failures(ledger_p, project_root): + """Re-run each `- [x]` card's cited acceptance command; list the failures. + + Only runs when the ledger has `REPLAY: on` (checked by the caller) — an + explicit opt-in, because re-running acceptances at every stop costs real + time. A command that exits non-zero or times out is a failure: 'it passed + once' is not 'it still passes'. Budgeted so a heavy suite can't hang the + stop indefinitely. Fail-open on unexpected errors. + """ + import subprocess + try: + timeout = int(os.environ.get("FABLE_REPLAY_TIMEOUT", + str(REPLAY_CMD_TIMEOUT))) + except ValueError: + timeout = REPLAY_CMD_TIMEOUT + failures = [] + seen = set() + spent = 0.0 + try: + with open(ledger_p, encoding="utf-8", errors="replace") as fh: + lines = [l.strip() for l in fh] + except Exception: + return [] + import time as _time + for s in lines: + if s[:5].lower() != "- [x]": + continue + for cmd in cited_commands(s): + if cmd in seen: + continue + seen.add(cmd) + if spent >= REPLAY_TOTAL_BUDGET: + failures.append((cmd, "not replayed: %ds replay budget spent " + "(raise FABLE_REPLAY_TIMEOUT or drop " + "REPLAY: on)" % REPLAY_TOTAL_BUDGET)) + continue + t0 = _time.time() + try: + p = subprocess.run(cmd, shell=True, cwd=project_root, + capture_output=True, text=True, + timeout=min(timeout, + REPLAY_TOTAL_BUDGET - spent)) + if p.returncode != 0: + tail = (p.stderr or p.stdout or "").strip()[-160:] + failures.append((cmd, "exit %d: %s" % (p.returncode, tail))) + except subprocess.TimeoutExpired: + failures.append((cmd, "timed out")) + except Exception as e: + failures.append((cmd, "could not run: %r" % e)) + spent += _time.time() - t0 + return failures def main(): @@ -65,6 +121,45 @@ def main(): "evidence.\n" % (len(bad), path, lines) ) return 2 + # Machine corroboration: a card that cites a `command` as evidence + # must have a successful run of that command in the evidence log + # (written by the Evidence Logger hook, not by the model). + unc = uncorroborated_citations(path, evidence_log_path(fable_dir)) + if unc: + shown = unc[:MAX_LIST] + lines = "\n".join(" " + it for it in shown) + if len(unc) > len(shown): + lines += "\n ... and %d more" % (len(unc) - len(shown)) + sys.stderr.write( + "[fable-mode] BLOCKED stop: %d checked card(s) cite an " + "evidence `command` with NO successful run recorded in the " + "evidence log (%s):\n%s\n" + "The log is written by the Evidence Logger hook from real " + "tool results — a cited command that never ran (or never " + "exited 0) is not evidence. Run the acceptance command now, " + "or fix the citation to the command that actually ran.\n" + % (len(unc), evidence_log_path(fable_dir), lines) + ) + return 2 + # Acceptance replay (opt-in via `REPLAY: on`): 'passed once' is not + # 'still passes' — re-run each card's cited acceptance before the + # round may end, so a later card can't silently break an earlier one. + if read_replay(path): + fails = replay_failures(path, os.path.dirname(fable_dir)) + if fails: + shown = fails[:MAX_LIST] + lines = "\n".join(" `%s` -> %s" % f for f in shown) + if len(fails) > len(shown): + lines += "\n ... and %d more" % (len(fails) - len(shown)) + sys.stderr.write( + "[fable-mode] BLOCKED stop: REPLAY is on and %d cited " + "acceptance command(s) do not pass when re-run now:\n%s\n" + "A card whose acceptance no longer passes is not done — " + "fix the regression (or, if the command is genuinely " + "stale, fix the citation), then stop.\n" + % (len(fails), lines) + ) + return 2 return 0 # all closed, all evidenced -> allow stop shown = open_items[:MAX_LIST] diff --git a/hooks/fable_evidence_log.py b/hooks/fable_evidence_log.py new file mode 100644 index 0000000..3e0a17a --- /dev/null +++ b/hooks/fable_evidence_log.py @@ -0,0 +1,56 @@ +#!/usr/bin/env python3 +"""fable-mode Evidence Logger (PostToolUse hook on Bash). + +Machine-written evidence: appends every Bash command's real outcome +(command, exit code, output tail) to `.fable/evidence.jsonl`. The Close +Guard then verifies that any `command` a `- [x]` card cites as evidence +actually ran — and succeeded — in this recorded history. + +This closes the honor-system gap: a model can type `-- evidence: pytest +21/21` without ever running pytest, but it cannot forge an entry in a log +only this hook writes. (It could still edit the file by hand — the log is +tamper-evident-by-convention, not cryptographic — but a fabrication now +requires a visible, auditable act instead of a plausible sentence.) + +Passive recorder: always exit 0, never blocks, never prints. Armed per +project by `.fable/`; records even while PAUSED (pausing enforcement must +not create evidence gaps). Fail-open on any error. +""" +import os +import sys + +sys.path.insert(0, os.path.dirname(os.path.abspath(__file__))) +from _fable_common import ( # noqa: E402 + read_hook_input, start_dir, find_fable_dir, append_evidence, + response_exit_code, +) + + +def main(): + data = read_hook_input() + fable_dir = find_fable_dir(start_dir(data)) + if not fable_dir: + return 0 # not opted in -> inert + + tool_input = data.get("tool_input") or {} + cmd = tool_input.get("command") if isinstance(tool_input, dict) else None + if not cmd: + return 0 + + r = data.get("tool_response") + exit_code = response_exit_code(r) + tail = "" + if isinstance(r, dict): + tail = str(r.get("stdout") or r.get("output") or r.get("stderr") or "") + elif isinstance(r, str): + tail = r + append_evidence(fable_dir, cmd, exit_code, tail) + return 0 + + +if __name__ == "__main__": + try: + sys.exit(main()) + except Exception as e: # passive recorder: never disturb the session + sys.stderr.write("[fable-mode] evidence log error (ignored): %r\n" % e) + sys.exit(0) diff --git a/hooks/fable_profile_inject.py b/hooks/fable_profile_inject.py index 7f1275a..dddfcfe 100644 --- a/hooks/fable_profile_inject.py +++ b/hooks/fable_profile_inject.py @@ -142,7 +142,11 @@ def build_context(profile, model, ledger_state, open_items, routing): "probes; tag SPEC decisions [measured]/[inferred]/[not-shown]. " "Guards block spawning without OPEN cards, stopping with open cards, " "and checking a card `- [x]` without a substantive `-- evidence:` " - "note.", + "note. Cite the acceptance `command` in the evidence note — cited " + "commands are corroborated against the machine-written evidence log " + "(.fable/evidence.jsonl); a citation that never ran (or never exited " + "0) blocks the stop. Optional ledger directive: `REPLAY: on` (cited " + "acceptances re-run before the round may end).", "Fable-5 habits: (1) audit every progress claim against a tool " "result — unverified means say 'unverified'; (2) don't end the turn " "on an actionable plan/promise — act now; (3) lead with the outcome; " diff --git a/install.sh b/install.sh index 9ce3131..bfc2efd 100755 --- a/install.sh +++ b/install.sh @@ -31,6 +31,7 @@ SPECS = [ ("SessionStart", None, "fable_profile_inject.py"), ("PreToolUse", "Agent|Task|Workflow", "fable_spawn_guard.py"), ("PostToolUse", "Bash", "fable_fail_streak.py"), + ("PostToolUse", "Bash", "fable_evidence_log.py"), ("Stop", None, "fable_close_guard.py"), ] NAMES = {fname for _, _, fname in SPECS} @@ -69,16 +70,19 @@ if mode == "uninstall": print("fable-mode hooks removed from %s" % settings_path) sys.exit(0) -# install: prune our old entries first (handles moves/upgrades), then add fresh. +# install: prune our old entries first (handles moves/upgrades), then add +# fresh. Prune ONCE per event before appending — pruning inside the append +# loop would strip the groups just added for an earlier spec on the same +# event (two hooks share PostToolUse/Bash). added = 0 +for event in {e for e, _, _ in SPECS}: + hooks[event] = prune(hooks.get(event, [])) for event, matcher, fname in SPECS: - entries = prune(hooks.get(event, [])) group = {"hooks": [{"type": "command", "command": "python3 %s" % os.path.join(hooks_dir, fname)}]} if matcher: group["matcher"] = matcher - entries.append(group) - hooks[event] = entries + hooks[event].append(group) added += 1 data["hooks"] = hooks diff --git a/templates/LEDGER.template.md b/templates/LEDGER.template.md index 02bc3e9..4a6534a 100644 --- a/templates/LEDGER.template.md +++ b/templates/LEDGER.template.md @@ -11,6 +11,12 @@ Only mark `- [x]` after the acceptance command actually ran — the Close Guard blocks turn-end for any `- [x]` without an evidence note (`-- evidence:` / `verified:` / `证据:`): +Cite the acceptance `command` inside the evidence note — the Close Guard +corroborates cited commands against `.fable/evidence.jsonl` (machine-written +by the Evidence Logger): a citation that never ran, or never exited 0, blocks +the stop. Optional: a `REPLAY: on` line re-runs cited acceptances before the +round may end ('passed once' is not 'still passes'). + - [ ] 1. — acceptance: `` - [ ] 2. — acceptance: `` - [x] 0. (example) scaffold -- evidence: `pytest -q` -> 12 passed diff --git a/tests/test_evidence.py b/tests/test_evidence.py new file mode 100644 index 0000000..a38329c --- /dev/null +++ b/tests/test_evidence.py @@ -0,0 +1,124 @@ +#!/usr/bin/env python3 +"""Tests for machine-written evidence: the Evidence Logger hook, the Close +Guard's citation corroboration, and REPLAY re-runs. Same conventions as +test_guards.py.""" +import json, os, subprocess, tempfile, shutil, sys + +HOOKS = os.path.join(os.path.dirname(os.path.abspath(__file__)), "..", "hooks") +CLOSE = os.path.join(HOOKS, "fable_close_guard.py") +EVLOG = os.path.join(HOOKS, "fable_evidence_log.py") + +passed = failed = 0 +def check(name, got, want): + global passed, failed + ok = got == want + print(("PASS" if ok else "FAIL"), name, "got=%s want=%s" % (got, want)) + if ok: passed += 1 + else: failed += 1 + +def run(script, payload): + p = subprocess.run([sys.executable, script], input=json.dumps(payload), + capture_output=True, text=True) + return p.returncode + +tmps = [] +def proj(ledger=None, evlog=None): + d = tempfile.mkdtemp(prefix="fbev_") + tmps.append(d) + os.mkdir(os.path.join(d, ".git")) + fd = os.path.join(d, ".fable"); os.mkdir(fd) + if ledger is not None: + with open(os.path.join(fd, "LEDGER.md"), "w") as f: f.write(ledger) + if evlog is not None: + with open(os.path.join(fd, "evidence.jsonl"), "w") as f: + for rec in evlog: + f.write(json.dumps(rec) + "\n") + return d + +OK = {"ts": 1.0, "exit": 0} + +# ---- Evidence Logger (recorder) ---- +# 1. appends a record with the real command and exit code +d = proj(ledger="- [ ] 1. card\n") +run(EVLOG, {"cwd": d, "tool_name": "Bash", + "tool_input": {"command": "pytest -q"}, + "tool_response": {"stdout": "21 passed", "exitCode": 0}}) +lp = os.path.join(d, ".fable", "evidence.jsonl") +rec = json.loads(open(lp).read().strip()) +check("evlog/records-cmd", rec["cmd"], "pytest -q") +check("evlog/records-exit", rec["exit"], 0) + +# 2. not opted in -> writes nothing +d2 = tempfile.mkdtemp(prefix="fbev_"); tmps.append(d2) +os.mkdir(os.path.join(d2, ".git")) +run(EVLOG, {"cwd": d2, "tool_name": "Bash", + "tool_input": {"command": "echo x"}, + "tool_response": {"exitCode": 0}}) +check("evlog/inert-without-optin", + os.path.exists(os.path.join(d2, ".fable", "evidence.jsonl")), False) + +# 3. records even while PAUSED (evidence gaps are worse than pauses) +d = proj(ledger="- [ ] 1. card\nPAUSED: side work\n") +run(EVLOG, {"cwd": d, "tool_name": "Bash", + "tool_input": {"command": "echo paused"}, + "tool_response": {"exitCode": 0}}) +check("evlog/records-while-paused", + os.path.exists(os.path.join(d, ".fable", "evidence.jsonl")), True) + +# ---- Close Guard: citation corroboration ---- +# 1. cited command with successful run in log -> allow +d = proj(ledger="- [x] 1. done -- evidence: `pytest -q` 21 passed\n", + evlog=[dict(OK, cmd="cd /x && pytest -q", tail="21 passed")]) +check("corroborate/cited-and-ran-allows", run(CLOSE, {"cwd": d}), 0) + +# 2. cited command NOT in log -> BLOCK (fabricated citation) +d = proj(ledger="- [x] 1. done -- evidence: `pytest -q` 21 passed\n", + evlog=[dict(OK, cmd="echo hello", tail="hello")]) +check("corroborate/cited-never-ran-blocks", run(CLOSE, {"cwd": d}), 2) + +# 3. cited command ran but FAILED -> BLOCK +d = proj(ledger="- [x] 1. done -- evidence: `pytest -q` all good\n", + evlog=[{"ts": 1.0, "cmd": "pytest -q", "exit": 1, "tail": "2 failed"}]) +check("corroborate/cited-but-failed-blocks", run(CLOSE, {"cwd": d}), 2) + +# 4. no log file at all (pre-logger project) -> old behavior, allow +d = proj(ledger="- [x] 1. done -- evidence: `pytest -q` 21 passed\n") +check("corroborate/no-log-failopen", run(CLOSE, {"cwd": d}), 0) + +# 5. prose-only evidence (no backtick command) -> not machine-checked, allow +d = proj(ledger="- [x] 1. done -- evidence: screenshot at docs/x.png\n", + evlog=[dict(OK, cmd="echo hi", tail="hi")]) +check("corroborate/prose-evidence-unchecked", run(CLOSE, {"cwd": d}), 0) + +# 6. acceptance backtick BEFORE the evidence marker is not a citation +d = proj(ledger="- [x] 1. thing — acceptance: `make test` -- evidence: " + "ran the suite, 40 green\n", + evlog=[dict(OK, cmd="echo unrelated", tail="")]) +check("corroborate/acceptance-part-not-cited", run(CLOSE, {"cwd": d}), 0) + +# 7. PAUSED still disables the close guard entirely (regression) +d = proj(ledger="- [x] 1. done -- evidence: `pytest -q` ok\nPAUSED: side\n", + evlog=[dict(OK, cmd="echo other", tail="")]) +check("corroborate/paused-allows", run(CLOSE, {"cwd": d}), 0) + +# ---- Close Guard: REPLAY re-runs ---- +# 1. replay armed, cited acceptance passes when re-run -> allow +d = proj(ledger="REPLAY: on\n- [x] 1. done -- evidence: `true` clean exit\n", + evlog=[dict(OK, cmd="true", tail="")]) +check("replay/pass-allows", run(CLOSE, {"cwd": d}), 0) + +# 2. replay armed, cited command now fails -> BLOCK (regression caught) +d = proj(ledger="REPLAY: on\n- [x] 1. done -- evidence: `false` ok\n", + evlog=[dict(OK, cmd="false", tail="")]) +check("replay/fail-blocks", run(CLOSE, {"cwd": d}), 2) + +# 3. no REPLAY line -> no replay (the same failing card passes the stop) +d = proj(ledger="- [x] 1. done -- evidence: `false` ok\n", + evlog=[dict(OK, cmd="false", tail="")]) +check("replay/off-by-default", run(CLOSE, {"cwd": d}), 0) + +for d in tmps: + shutil.rmtree(d, ignore_errors=True) + +print("\n%d passed, %d failed" % (passed, failed)) +sys.exit(1 if failed else 0) diff --git a/tests/test_install.py b/tests/test_install.py index 534250d..bcf7639 100644 --- a/tests/test_install.py +++ b/tests/test_install.py @@ -6,7 +6,10 @@ REPO = os.path.dirname(os.path.dirname(os.path.abspath(__file__))) INSTALL_SRC = os.path.join(REPO, "install.sh") NAMES = ["fable_profile_inject.py", "fable_spawn_guard.py", - "fable_fail_streak.py", "fable_close_guard.py"] + "fable_fail_streak.py", "fable_evidence_log.py", + "fable_close_guard.py"] +# expected hook-group count per event (PostToolUse carries two: streak + evidence log) +EXPECT = {"SessionStart": 1, "PreToolUse": 1, "PostToolUse": 2, "Stop": 1} passed = failed = 0 @@ -62,7 +65,8 @@ def cmds(d): # B. idempotent — second run must not duplicate run(skill, cfg) d = load(cfg) - check("install/idempotent", all(len(d["hooks"][e]) == 1 for e in d["hooks"])) + check("install/idempotent", + all(len(d["hooks"][e]) == EXPECT[e] for e in d["hooks"])) # C. merge — preserve unrelated config + a user's own hook with open(os.path.join(cfg, "settings.json"), "w") as f: