diff --git a/CHANGELOG.md b/CHANGELOG.md index 862fbfa..5e0f2f3 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -5,12 +5,59 @@ All notable changes to Vera are documented here. Format follows ## [Unreleased] +## [0.2.0] — 2026-04-21 + +Theme: visibility and control. 0.1.0 worked; 0.2.0 makes it durable +for week-over-week use. + +### Added +- **Cost tracking.** `vera.llm.chat()` returns `(text, usage)` with token + counts plus an estimated USD cost from a public rate table. Every LLM + call (chat / chat-retry / audit) appends a `USAGE` entry to the + provenance log. `vera status` prints `Spend (30d): $X.XX over N calls` + when any USAGE entries exist. +- **Visible regenerate.** Rule violations no longer trigger a silent + retry. A dim `⟲ regenerating: ` line prints before attempt + 2; if the rule layer disagrees twice, a `⚠ rule layer disagreed + twice` footer is appended to the response. +- **`vera prune`.** New command archives transcripts older than N days + (default 90) into `MEMORY_DIR/_archive/`, always retaining the most + recent `--keep` (default 50). Supports `--dry-run`. `vera status` + surfaces archive counts. +- **Smarter memory summary.** `load_memory_summary()` now layers durable + facts (deduped `RULE` / `OBSERVED` captures from the last 30 days) + on top of headlines from the last 5 transcripts. No mid-word + truncation. +- **Audit controls.** `vera audit --last N`, `--since YYYY-MM-DD`, + `--model MODEL`, plus `VERA_AUDIT_MODEL` env var read in + `detect_provider()`. +- **Lenient VERA-CAPTURE parsing.** Accepts markdown blockquotes, list + bullets, indentation, case-insensitive keys, and blank-line + separators. Malformed blocks produce a console warning plus a + `MALFORMED` provenance entry instead of silently dropping. + +### Changed +- `chat()` signature: now returns `(text, usage)`. All internal callers + updated. +- `ProvenanceKind` widened to include `MALFORMED`. +- `load_memory_summary()` rewritten (backwards-compatible at call-site; + output format changed). +- `run_audit()` signature: adds `n`, `since`, `model_override` kwargs. + +### Fixed +- Install instructions across `README.md`, `LAUNCH.md`, `install.sh`, + and `launch/READY-TO-PASTE.md` now consistently point at + `pipx install vera-clerk` (PyPI) rather than the stale + `git+https://github.com/iamitp/vera` form. + +## [0.1.1] — Unreleased pre-0.2.0 work + ### Added - `vera audit --share` flag: writes an anonymized, copy-pasteable `.share.md` snippet alongside the audit report, with a Vera attribution footer. - `examples/` directory with a realistic transcript, audit report, and share snippet so the loop can be evaluated without running the CLI. -- Test suite under `tests/` (21 tests covering provenance, rules, audit +- Test suite under `tests/` (24 tests covering provenance, rules, audit share-snippet, and VERA-CAPTURE parsing). - GitHub Actions CI running pytest on Python 3.10 / 3.11 / 3.12 plus a build+twine-check job. diff --git a/LAUNCH.md b/LAUNCH.md index ac1067a..df0fe81 100644 --- a/LAUNCH.md +++ b/LAUNCH.md @@ -10,7 +10,7 @@ > > Local markdown. Any model. One command: > -> `pipx install git+https://github.com/iamitp/vera && vera init && vera chat` +> `pipx install vera-clerk && vera init && vera chat` > > MIT. Free forever. github.com/iamitp/vera @@ -43,7 +43,7 @@ What makes it different from Khoj / Mem0 / Letta / MemPalace: - **Provenance tags** on every memory write (OBSERVED / INFERRED / ASSUMED / CANDIDATE — the four categories are enforced in the system prompt) - **No vendor lock-in** (markdown on your disk, BYO API key, works with Claude or GPT) -MIT. Python 3.10+. `pipx install git+https://github.com/iamitp/vera`. +MIT. Python 3.10+. `pipx install vera-clerk`. Repo: https://github.com/iamitp/vera diff --git a/README.md b/README.md index 266caa3..a1ac0b2 100644 --- a/README.md +++ b/README.md @@ -60,7 +60,7 @@ To avoid re-pasting every chat, put the prompt into Custom Instructions for a [C For users who want hard rule-blocking, a second independent auditor model, and markdown memory on disk. ```bash -pipx install git+https://github.com/iamitp/vera +pipx install vera-clerk export ANTHROPIC_API_KEY=sk-ant-... # or OPENAI_API_KEY vera init && vera chat ``` diff --git a/install.sh b/install.sh index 7c603fb..c9d7627 100644 --- a/install.sh +++ b/install.sh @@ -22,7 +22,7 @@ if ! command -v pipx >/dev/null 2>&1; then fi # 2. install vera -pipx install --force git+https://github.com/iamitp/vera >/dev/null +pipx install --force vera-clerk >/dev/null echo "✓ Vera installed." echo "" diff --git a/launch/READY-TO-PASTE.md b/launch/READY-TO-PASTE.md index 878f8c9..b63a310 100644 --- a/launch/READY-TO-PASTE.md +++ b/launch/READY-TO-PASTE.md @@ -43,7 +43,7 @@ The conversation was about rewriting a payments service in Rust. The user mentio A sycophantic AI would have accepted the number and built recommendations on top. That's the point. -For power users, a CLI (pipx install git+https://github.com/iamitp/vera@v0.1.0) adds a second independent auditor model, hard rule-blocking with regenerate, and on-disk markdown memory you can grep and sync. But the paste-in prompt covers 80% of the value in 30 seconds. +For power users, a CLI (pipx install vera-clerk) adds a second independent auditor model, hard rule-blocking with regenerate, and on-disk markdown memory you can grep and sync. But the paste-in prompt covers 80% of the value in 30 seconds. Happy to answer questions about which failures each discipline catches, why I didn't just ship an extension, or how it compares to guard-rails / constitutional-AI patterns. ``` @@ -106,7 +106,7 @@ Vera doesn't fix hallucination. It makes it visible. ``` For power users: a CLI adds a second independent model as the auditor, hard rule enforcement, and markdown memory on disk you own. -pipx install git+https://github.com/iamitp/vera@v0.1.0 +pipx install vera-clerk But the paste-in prompt covers 80% of it in 30 seconds. @@ -136,7 +136,7 @@ Also: type /audit and the model self-critiques for sycophancy, hedging, and unso Real example from a run against the Anthropic API this morning (examples/live-demo/ in the repo): the conversation was about a Rust rewrite, and the user said 400 ms of p99 was JSON serialisation of 50 kB. The /audit pass worked out the implied throughput (~0.1 MB/s, ~1000x too slow) and flagged it as likely measurement error. -For power users, the CLI (pipx install git+https://github.com/iamitp/vera@v0.1.0) adds: +For power users, the CLI (pipx install vera-clerk) adds: - A second independent model as auditor (not self-critique) - Hard rule enforcement (banned phrases fail the turn, model regenerates) - Markdown memory on disk you own diff --git a/pyproject.toml b/pyproject.toml index 4873145..b434c50 100644 --- a/pyproject.toml +++ b/pyproject.toml @@ -1,6 +1,6 @@ [project] name = "vera-clerk" -version = "0.1.0" +version = "0.2.0" description = "The AI memory that argues with itself. Local-first. Model-agnostic. Every answer with a receipt." readme = "README.md" requires-python = ">=3.10" diff --git a/tests/test_audit.py b/tests/test_audit.py index 64797c4..aeef165 100644 --- a/tests/test_audit.py +++ b/tests/test_audit.py @@ -82,7 +82,7 @@ def test_audit_cli_share_flag_writes_sibling(tmp_path, monkeypatch): "# Vera Audit 2026-04-15 14:30\n\n## Sycophancy\nClean.\n" ) - monkeypatch.setattr(cli_mod, "run_audit", lambda provider, mem: report) + monkeypatch.setattr(cli_mod, "run_audit", lambda provider, mem, **kw: report) monkeypatch.setattr(cli_mod, "detect_provider", lambda: object()) monkeypatch.setattr(cli_mod, "MEMORY_DIR", tmp_path / "memory") @@ -92,3 +92,70 @@ def test_audit_cli_share_flag_writes_sibling(tmp_path, monkeypatch): share_path = report.with_suffix(".share.md") assert share_path.exists() assert "# What Vera caught in my own chat history" in share_path.read_text() + + +def test_audit_cli_passes_flags_to_run_audit(tmp_path, monkeypatch): + from click.testing import CliRunner + from vera import cli as cli_mod + + captured = {} + + def fake_run_audit(provider, mem, **kw): + captured.update(kw) + out = tmp_path / "r.md" + out.write_text("# Vera Audit now\n\nclean\n") + return out + + monkeypatch.setattr(cli_mod, "run_audit", fake_run_audit) + monkeypatch.setattr(cli_mod, "detect_provider", lambda: object()) + monkeypatch.setattr(cli_mod, "MEMORY_DIR", tmp_path / "memory") + + runner = CliRunner() + result = runner.invoke( + cli_mod.main, + ["audit", "--last", "2", "--since", "2026-04-01", "--model", "claude-haiku-4-5-20251001"], + ) + assert result.exit_code == 0, result.output + assert captured["n"] == 2 + assert str(captured["since"]) == "2026-04-01" + assert captured["model_override"] == "claude-haiku-4-5-20251001" + + +def test_audit_cli_rejects_bad_since(tmp_path, monkeypatch): + from click.testing import CliRunner + from vera import cli as cli_mod + + monkeypatch.setattr(cli_mod, "detect_provider", lambda: object()) + runner = CliRunner() + result = runner.invoke(cli_mod.main, ["audit", "--since", "yesterday"]) + assert result.exit_code != 0 + assert "YYYY-MM-DD" in result.output + + +def test_select_transcripts_respects_last_and_since(tmp_path): + import os + from datetime import datetime, timedelta + from vera.audit import _select_transcripts + + for name, days in [("a.md", 100), ("b.md", 60), ("c.md", 30), ("d.md", 5), ("e.md", 1)]: + p = tmp_path / name + p.write_text("x") + ts = (datetime.now() - timedelta(days=days)).timestamp() + os.utime(p, (ts, ts)) + + picked = _select_transcripts(tmp_path, n=2, since=None) + assert [p.name for p in picked] == ["d.md", "e.md"] + + cutoff = (datetime.now() - timedelta(days=45)).date() + picked = _select_transcripts(tmp_path, n=10, since=cutoff) + assert [p.name for p in picked] == ["c.md", "d.md", "e.md"] + + +def test_detect_provider_reads_vera_audit_model(monkeypatch): + import importlib + monkeypatch.setenv("ANTHROPIC_API_KEY", "x") + monkeypatch.setenv("VERA_AUDIT_MODEL", "claude-haiku-4-5-20251001") + import vera.config + importlib.reload(vera.config) + provider = vera.config.detect_provider() + assert provider.audit_model == "claude-haiku-4-5-20251001" diff --git a/tests/test_captures.py b/tests/test_captures.py index a332425..9b75e64 100644 --- a/tests/test_captures.py +++ b/tests/test_captures.py @@ -2,7 +2,9 @@ def test_no_captures_returns_empty(): - assert parse_captures("Hello world.") == [] + captures, warnings = parse_captures("Hello world.") + assert captures == [] + assert warnings == [] def test_single_capture(): @@ -12,8 +14,9 @@ def test_single_capture(): kind: RULE content: no em dashes """ - out = parse_captures(resp) - assert out == [("RULE", "no em dashes")] + captures, warnings = parse_captures(resp) + assert captures == [("RULE", "no em dashes")] + assert warnings == [] def test_multiple_captures(): @@ -29,23 +32,96 @@ def test_multiple_captures(): kind: INFERRED content: user speaks German """ - out = parse_captures(resp) - assert out == [ + captures, _ = parse_captures(resp) + assert captures == [ ("OBSERVED", "user lives in Berlin"), ("INFERRED", "user speaks German"), ] -def test_malformed_capture_ignored(): +def test_malformed_capture_emits_warning(): resp = """VERA-CAPTURE: kind: content: """ - assert parse_captures(resp) == [] + captures, warnings = parse_captures(resp) + assert captures == [] + assert len(warnings) == 1 + assert "missing kind" in warnings[0] def test_capture_block_requires_marker(): resp = """kind: RULE content: nope """ - assert parse_captures(resp) == [] + captures, warnings = parse_captures(resp) + assert captures == [] + assert warnings == [] + + +def test_capture_inside_blockquote(): + resp = """Response text. + +> VERA-CAPTURE: +> kind: RULE +> content: no emojis in filenames +""" + captures, warnings = parse_captures(resp) + assert captures == [("RULE", "no emojis in filenames")] + assert warnings == [] + + +def test_capture_with_list_bullets(): + resp = """Response. + +- VERA-CAPTURE: +- kind: OBSERVED +- content: user uses postgres at work +""" + captures, warnings = parse_captures(resp) + assert captures == [("OBSERVED", "user uses postgres at work")] + assert warnings == [] + + +def test_capture_with_indentation(): + resp = """Response. + + VERA-CAPTURE: + kind: RULE + content: be terse +""" + captures, warnings = parse_captures(resp) + assert captures == [("RULE", "be terse")] + + +def test_capture_case_insensitive_keys(): + resp = """VERA-CAPTURE: +Kind: INFERRED +Content: user cares about latency +""" + captures, _ = parse_captures(resp) + assert captures == [("INFERRED", "user cares about latency")] + + +def test_capture_tolerates_blank_line_after_marker(): + resp = """VERA-CAPTURE: + +kind: RULE +content: confirm before deleting +""" + captures, warnings = parse_captures(resp) + assert captures == [("RULE", "confirm before deleting")] + assert warnings == [] + + +def test_capture_fields_five_lines_apart_still_parsed(): + resp = """VERA-CAPTURE: +kind: RULE + + +content: don't paraphrase code +""" + captures, warnings = parse_captures(resp) + # Blank line after fields started ends the block; content should not be found + assert captures == [] + assert len(warnings) == 1 diff --git a/tests/test_cost.py b/tests/test_cost.py new file mode 100644 index 0000000..de3254f --- /dev/null +++ b/tests/test_cost.py @@ -0,0 +1,85 @@ +import json +from datetime import datetime, timedelta + +from vera.llm import estimate_usd, USD_PER_MTOK +from vera.provenance import log_usage, spend_since + + +def test_estimate_usd_known_model(): + in_rate, out_rate = USD_PER_MTOK[("anthropic", "claude-sonnet-4-6")] + usd = estimate_usd("anthropic", "claude-sonnet-4-6", 1_000_000, 500_000) + assert usd == in_rate + 0.5 * out_rate + + +def test_estimate_usd_unknown_model_returns_none(): + assert estimate_usd("anthropic", "made-up-model", 100, 100) is None + + +def test_log_usage_writes_expected_shape(tmp_path): + log = tmp_path / "log.jsonl" + usage = { + "provider": "anthropic", + "model": "claude-sonnet-4-6", + "input_tokens": 1200, + "output_tokens": 340, + "usd": 0.0087, + } + log_usage(usage, source="chat", log_path=log) + entry = json.loads(log.read_text().strip()) + assert entry["kind"] == "USAGE" + assert entry["source"] == "chat" + assert entry["provider"] == "anthropic" + assert entry["model"] == "claude-sonnet-4-6" + assert entry["input_tokens"] == 1200 + assert entry["output_tokens"] == 340 + assert entry["usd"] == 0.0087 + assert "ts" in entry + + +def test_spend_since_sums_usage_within_window(tmp_path): + log = tmp_path / "log.jsonl" + # one call today + log_usage( + {"provider": "anthropic", "model": "claude-sonnet-4-6", + "input_tokens": 1000, "output_tokens": 1000, "usd": 0.018}, + source="chat", log_path=log, + ) + # one old entry written manually + old = datetime.now() - timedelta(days=60) + with log.open("a") as f: + f.write(json.dumps({ + "ts": old.isoformat(), "kind": "USAGE", "source": "chat", + "provider": "anthropic", "model": "claude-sonnet-4-6", + "input_tokens": 5000, "output_tokens": 5000, "usd": 0.09, + }) + "\n") + usd, calls = spend_since(log, days=30) + assert calls == 1 + assert abs(usd - 0.018) < 1e-9 + + +def test_spend_since_ignores_non_usage_entries(tmp_path): + log = tmp_path / "log.jsonl" + from vera.provenance import log_write + log_write("OBSERVED", "hi", source="chat", log_path=log) + usd, calls = spend_since(log, days=30) + assert usd == 0.0 + assert calls == 0 + + +def test_spend_since_handles_missing_log(tmp_path): + log = tmp_path / "missing.jsonl" + usd, calls = spend_since(log, days=30) + assert usd == 0.0 + assert calls == 0 + + +def test_spend_since_counts_unknown_usd_calls_but_adds_zero(tmp_path): + log = tmp_path / "log.jsonl" + log_usage( + {"provider": "anthropic", "model": "mystery", + "input_tokens": 100, "output_tokens": 100, "usd": None}, + source="chat", log_path=log, + ) + usd, calls = spend_since(log, days=30) + assert calls == 1 + assert usd == 0.0 diff --git a/tests/test_prune.py b/tests/test_prune.py new file mode 100644 index 0000000..5de097c --- /dev/null +++ b/tests/test_prune.py @@ -0,0 +1,126 @@ +import os +from datetime import datetime, timedelta +from pathlib import Path + +from click.testing import CliRunner + + +def _make_transcript(dir_: Path, name: str, days_old: int) -> Path: + dir_.mkdir(parents=True, exist_ok=True) + p = dir_ / name + p.write_text(f"## you\nhello\n\n## vera\nhi\n") + ts = (datetime.now() - timedelta(days=days_old)).timestamp() + os.utime(p, (ts, ts)) + return p + + +def _reload_cli_with_home(monkeypatch, home: Path): + monkeypatch.setenv("VERA_HOME", str(home)) + import importlib + import vera.config + import vera.provenance + import vera.audit + import vera.cli + # config reads env at import; reload the module graph that depends on it. + importlib.reload(vera.config) + importlib.reload(vera.provenance) + importlib.reload(vera.audit) + importlib.reload(vera.cli) + return vera.cli + + +def test_prune_skips_when_under_keep_threshold(tmp_path, monkeypatch): + cli = _reload_cli_with_home(monkeypatch, tmp_path) + mem = cli.MEMORY_DIR + for i in range(3): + _make_transcript(mem, f"2026-01-0{i+1}.md", days_old=200) + + runner = CliRunner() + result = runner.invoke(cli.main, ["prune", "--older-than", "30", "--keep", "50"]) + assert result.exit_code == 0, result.output + assert "nothing to archive" in result.output + assert len(list(mem.glob("*.md"))) == 3 + + +def test_prune_dry_run_does_not_move(tmp_path, monkeypatch): + cli = _reload_cli_with_home(monkeypatch, tmp_path) + mem = cli.MEMORY_DIR + for i in range(10): + _make_transcript(mem, f"old-{i:02d}.md", days_old=200) + for i in range(5): + _make_transcript(mem, f"new-{i:02d}.md", days_old=1) + + runner = CliRunner() + result = runner.invoke( + cli.main, ["prune", "--older-than", "30", "--keep", "3", "--dry-run"] + ) + assert result.exit_code == 0, result.output + assert "would archive" in result.output + # Nothing moved on dry-run + assert not (mem / cli.ARCHIVE_DIRNAME).exists() + + +def test_prune_moves_old_transcripts(tmp_path, monkeypatch): + cli = _reload_cli_with_home(monkeypatch, tmp_path) + mem = cli.MEMORY_DIR + for i in range(8): + _make_transcript(mem, f"old-{i:02d}.md", days_old=200) + for i in range(3): + _make_transcript(mem, f"new-{i:02d}.md", days_old=1) + + runner = CliRunner() + result = runner.invoke(cli.main, ["prune", "--older-than", "30", "--keep", "3"]) + assert result.exit_code == 0, result.output + assert "Archived" in result.output + archive = mem / cli.ARCHIVE_DIRNAME + assert archive.exists() + archived = list(archive.glob("*.md")) + # 8 old minus 0 kept from the new-only tail: all 8 olds move, 3 news stay + assert len(archived) == 8 + # Newest 3 are preserved + remaining = {p.name for p in mem.glob("*.md")} + assert remaining == {f"new-{i:02d}.md" for i in range(3)} + + +def test_prune_respects_keep_on_old_files(tmp_path, monkeypatch): + cli = _reload_cli_with_home(monkeypatch, tmp_path) + mem = cli.MEMORY_DIR + # 10 old files, no new ones. keep=5 → only 5 oldest get moved. + for i in range(10): + _make_transcript(mem, f"old-{i:02d}.md", days_old=200) + + runner = CliRunner() + result = runner.invoke(cli.main, ["prune", "--older-than", "30", "--keep", "5"]) + assert result.exit_code == 0, result.output + archive = mem / cli.ARCHIVE_DIRNAME + assert len(list(archive.glob("*.md"))) == 5 + assert len(list(mem.glob("*.md"))) == 5 + + +def test_status_shows_archive_count(tmp_path, monkeypatch): + cli = _reload_cli_with_home(monkeypatch, tmp_path) + mem = cli.MEMORY_DIR + archive = mem / cli.ARCHIVE_DIRNAME + _make_transcript(mem, "recent.md", days_old=1) + _make_transcript(archive, "archived.md", days_old=200) + + runner = CliRunner() + result = runner.invoke(cli.main, ["status"]) + assert result.exit_code == 0, result.output + assert "Memory: 1 transcripts" in result.output + assert "Archive: 1 transcripts" in result.output + + +def test_memory_summary_prefers_durable_facts(tmp_path, monkeypatch): + cli = _reload_cli_with_home(monkeypatch, tmp_path) + from vera.provenance import log_write + log = cli.PROVENANCE_LOG + log_write("RULE", "no em dashes ever", source="chat", log_path=log) + log_write("OBSERVED", "user works in payments", source="chat", log_path=log) + _make_transcript(cli.MEMORY_DIR, "2026-04-20.md", days_old=1) + + summary = cli.load_memory_summary() + assert "Durable facts" in summary + assert "[RULE] no em dashes ever" in summary + assert "[OBSERVED] user works in payments" in summary + assert "2026-04-20" in summary diff --git a/vera/__init__.py b/vera/__init__.py index 3dc1f76..d3ec452 100644 --- a/vera/__init__.py +++ b/vera/__init__.py @@ -1 +1 @@ -__version__ = "0.1.0" +__version__ = "0.2.0" diff --git a/vera/audit.py b/vera/audit.py index caae41b..e8e59c8 100644 --- a/vera/audit.py +++ b/vera/audit.py @@ -4,10 +4,11 @@ from __future__ import annotations import json import re -from datetime import datetime +from datetime import date, datetime from pathlib import Path from .config import Provider, AUDIT_DIR, PROVENANCE_LOG from .llm import chat +from .provenance import log_usage AUDITOR_SYSTEM = """ You are Vera's auditor. Your only loyalty is to calling out the primary @@ -33,20 +34,42 @@ ) -def run_audit(provider: Provider, transcripts_dir: Path) -> Path: +def _select_transcripts( + transcripts_dir: Path, + n: int, + since: date | None, +) -> list[Path]: + if not transcripts_dir.exists(): + return [] + all_ = sorted(transcripts_dir.glob("*.md"), key=lambda p: p.stat().st_mtime) + if since is not None: + cutoff = datetime.combine(since, datetime.min.time()).timestamp() + all_ = [p for p in all_ if p.stat().st_mtime >= cutoff] + return all_[-n:] if n > 0 else all_ + + +def run_audit( + provider: Provider, + transcripts_dir: Path, + n: int = 5, + since: date | None = None, + model_override: str | None = None, +) -> Path: AUDIT_DIR.mkdir(parents=True, exist_ok=True) - transcripts = sorted(transcripts_dir.glob("*.md"))[-5:] + transcripts = _select_transcripts(transcripts_dir, n, since) if not transcripts: out = AUDIT_DIR / f"{datetime.now():%Y-%m-%d}.md" out.write_text(f"# Vera Audit {datetime.now():%Y-%m-%d}\n\nNo transcripts to audit.\n") return out body = "\n\n---\n\n".join(t.read_text() for t in transcripts) - resp = chat( + model = model_override or provider.audit_model + resp, usage = chat( provider, AUDITOR_SYSTEM, [{"role": "user", "content": f"Audit these transcripts:\n\n{body}"}], - model=provider.audit_model, + model=model, ) + log_usage(usage, source="audit", log_path=PROVENANCE_LOG) out = AUDIT_DIR / f"{datetime.now():%Y-%m-%d_%H%M}.md" out.write_text(f"# Vera Audit {datetime.now():%Y-%m-%d %H:%M}\n\n{resp}\n") return out diff --git a/vera/cli.py b/vera/cli.py index d6f184c..12c9851 100644 --- a/vera/cli.py +++ b/vera/cli.py @@ -1,17 +1,21 @@ -"""Vera CLI: init / chat / audit / rules.""" +"""Vera CLI: init / chat / audit / rules / prune / status.""" from __future__ import annotations +import json +import re import sys -from datetime import datetime +from datetime import datetime, timedelta from pathlib import Path import click from rich.console import Console from rich.markdown import Markdown from .config import VERA_HOME, MEMORY_DIR, AUDIT_DIR, RULES_FILE, PROVENANCE_LOG, detect_provider -from .provenance import SYSTEM_ADDENDUM, log_write +from .provenance import SYSTEM_ADDENDUM, log_write, log_usage, spend_since from .rules import ensure_rules, check_response, DEFAULT_RULES from .llm import chat as _llm_chat from .audit import run_audit, build_share_snippet +ARCHIVE_DIRNAME = "_archive" + console = Console() def build_system(rules_text: str, memory_summary: str) -> str: @@ -40,34 +44,115 @@ def build_system(rules_text: str, memory_summary: str) -> str: The CLI parses these captures and writes them to memory. """ -def parse_captures(response: str) -> list[tuple[str, str]]: - """Extract VERA-CAPTURE blocks from the model response.""" - out = [] +_CAPTURE_LEADER = re.compile(r"^[\s>\-*]*") + + +def _strip_leader(line: str) -> str: + """Drop leading whitespace, blockquote '>', and list bullets.""" + return _CAPTURE_LEADER.sub("", line).strip() + + +def parse_captures(response: str) -> tuple[list[tuple[str, str]], list[str]]: + """Extract VERA-CAPTURE blocks from the model response. + + Returns (captures, warnings). A malformed block (marker present but + kind or content missing within the next 6 lines) adds a warning + instead of silently dropping. Accepts blockquote and list prefixes + and arbitrary indentation around the marker and its fields. + """ + out: list[tuple[str, str]] = [] + warnings: list[str] = [] lines = response.splitlines() i = 0 while i < len(lines): - if lines[i].strip() == "VERA-CAPTURE:": + stripped = _strip_leader(lines[i]) + if stripped == "VERA-CAPTURE:": kind = content = "" - for j in range(i+1, min(i+4, len(lines))): - s = lines[j].strip() - if s.startswith("kind:"): - kind = s.split(":",1)[1].strip() - elif s.startswith("content:"): - content = s.split(":",1)[1].strip() + scanned = 0 + for j in range(i + 1, min(i + 7, len(lines))): + s = _strip_leader(lines[j]) + if not s: + # Blank line between marker and first field is tolerated; + # blank line after we've started reading fields ends the block. + if kind or content: + break + continue + if s.lower().startswith("kind:"): + kind = s.split(":", 1)[1].strip() + elif s.lower().startswith("content:"): + content = s.split(":", 1)[1].strip() + else: + # Non-field line ends the block. + break + scanned = j - i if kind and content: out.append((kind, content)) - i += 3 + else: + warnings.append( + f"VERA-CAPTURE at line {i+1} missing " + f"{'kind' if not kind else 'content'}" + ) + i += max(scanned, 1) else: i += 1 - return out + return out, warnings + +def _truncate_line(text: str, limit: int = 160) -> str: + line = text.strip().splitlines()[0] if text.strip() else "" + return line if len(line) <= limit else line[:limit].rstrip() + "…" + + +def _recent_durable_facts(log_path: Path, days: int = 30, limit: int = 20) -> list[str]: + """Return deduped RULE / OBSERVED entries from the last N days.""" + if not log_path.exists(): + return [] + cutoff = datetime.now() - timedelta(days=days) + seen: set[str] = set() + out: list[tuple[datetime, str, str]] = [] + for raw in log_path.read_text().splitlines(): + try: + entry = json.loads(raw) + except json.JSONDecodeError: + continue + if entry.get("kind") not in ("RULE", "OBSERVED"): + continue + try: + ts = datetime.fromisoformat(entry["ts"]) + except (KeyError, ValueError): + continue + if ts < cutoff: + continue + content = _truncate_line(str(entry.get("content", ""))) + if not content or content in seen: + continue + seen.add(content) + out.append((ts, entry["kind"], content)) + out.sort(key=lambda x: x[0], reverse=True) + return [f"[{k}] {c}" for _, k, c in out[:limit]] + def load_memory_summary() -> str: - if not MEMORY_DIR.exists(): - return "" - items = [] - for f in sorted(MEMORY_DIR.glob("*.md"))[-10:]: - items.append(f"- {f.stem}: {f.read_text()[:200]}") - return "\n".join(items) + """Compose the memory-summary block the chat system prompt sees. + + Two layers: durable facts (RULE / OBSERVED captures in the last 30 days, + deduped) plus headlines from the last 5 transcripts. No mid-word + truncation — one line per entry, 160-char cap. + """ + parts: list[str] = [] + facts = _recent_durable_facts(PROVENANCE_LOG) + if facts: + parts.append("Durable facts (recent captures):") + parts.extend(f"- {f}" for f in facts) + + if MEMORY_DIR.exists(): + recent = sorted(MEMORY_DIR.glob("*.md"))[-5:] + if recent: + parts.append("") + parts.append("Recent transcripts:") + for f in recent: + first = _truncate_line(f.read_text()) + parts.append(f"- {f.stem}: {first}" if first else f"- {f.stem}: (empty)") + return "\n".join(parts) @click.group() def main(): @@ -141,16 +226,23 @@ def chat(): memory_summary = load_memory_summary() system = build_system(rules_text, memory_summary) - for attempt in range(2): - response = _llm_chat(provider, system, messages) - violation = check_response(response, rules_text) - if not violation: - break + response, usage = _llm_chat(provider, system, messages) + log_usage(usage, source="chat", log_path=PROVENANCE_LOG) + violation = check_response(response, rules_text) + if violation: + console.print(f"[dim]⟲ regenerating: {violation}[/dim]") messages_retry = messages + [ {"role": "assistant", "content": response}, {"role": "user", "content": violation}, ] - response = _llm_chat(provider, system, messages_retry) + response, usage = _llm_chat(provider, system, messages_retry) + log_usage(usage, source="chat-retry", log_path=PROVENANCE_LOG) + second_violation = check_response(response, rules_text) + if second_violation: + response = ( + f"{response}\n\n" + f"> ⚠ rule layer disagreed twice: {second_violation}" + ) messages.append({"role": "assistant", "content": response}) console.print() @@ -158,12 +250,16 @@ def chat(): console.print() # Capture user-stated facts / rules - for kind, content in parse_captures(response): + captures, capture_warnings = parse_captures(response) + for kind, content in captures: log_write(kind, content, source="chat", log_path=PROVENANCE_LOG) if kind == "RULE": with RULES_FILE.open("a") as f: f.write(f"\n- \"{content}\" (captured {datetime.now():%Y-%m-%d})\n") rules_text = RULES_FILE.read_text() + for warn in capture_warnings: + console.print(f"[dim]warning: malformed capture skipped — {warn}[/dim]") + log_write("MALFORMED", warn, source="chat", log_path=PROVENANCE_LOG) with transcript_path.open("a") as f: f.write(f"## you\n{user_input}\n\n## vera\n{response}\n\n") @@ -175,10 +271,25 @@ def chat(): is_flag=True, help="Also emit an anonymized, copy-pasteable snippet suitable for social media.", ) -def audit(share_flag: bool): +@click.option("--last", "last_n", type=int, default=5, + help="Audit the last N transcripts by mtime (default 5).") +@click.option("--since", "since_str", type=str, default=None, + help="Only audit transcripts touched on/after YYYY-MM-DD.") +@click.option("--model", "model_override", type=str, default=None, + help="Override the audit model. Also reads VERA_AUDIT_MODEL.") +def audit(share_flag: bool, last_n: int, since_str: str | None, model_override: str | None): """Run the adversarial auditor on recent transcripts.""" provider = detect_provider() - out = run_audit(provider, MEMORY_DIR) + since = None + if since_str: + try: + since = datetime.strptime(since_str, "%Y-%m-%d").date() + except ValueError: + raise click.BadParameter("--since must be YYYY-MM-DD") + out = run_audit( + provider, MEMORY_DIR, + n=last_n, since=since, model_override=model_override, + ) console.print(f"[green]Audit written:[/green] {out}") console.print() console.print(Markdown(out.read_text())) @@ -194,13 +305,62 @@ def rules(): """Print the active rules file.""" console.print(Markdown(ensure_rules(RULES_FILE))) +@main.command() +@click.option("--older-than", "older_than_days", type=int, default=90, + help="Archive transcripts older than N days (default 90).") +@click.option("--keep", type=int, default=50, + help="Always keep at least N most-recent transcripts (default 50).") +@click.option("--dry-run", is_flag=True, help="Show what would be moved, don't move.") +def prune(older_than_days: int, keep: int, dry_run: bool): + """Archive old transcripts into MEMORY_DIR/_archive/ to cap disk growth.""" + if not MEMORY_DIR.exists(): + console.print(f"[dim]No memory dir at {MEMORY_DIR}; nothing to prune.[/dim]") + return + archive_dir = MEMORY_DIR / ARCHIVE_DIRNAME + transcripts = sorted(MEMORY_DIR.glob("*.md"), key=lambda p: p.stat().st_mtime) + if len(transcripts) <= keep: + console.print( + f"[dim]{len(transcripts)} transcripts ≤ keep={keep}; nothing to archive.[/dim]" + ) + return + cutoff = datetime.now() - timedelta(days=older_than_days) + # Newest `keep` (by mtime) are never moved; consider only the older tail. + tail = transcripts[:-keep] if keep > 0 else transcripts + to_archive = [t for t in tail if datetime.fromtimestamp(t.stat().st_mtime) < cutoff] + if not to_archive: + console.print( + f"[dim]No transcripts older than {older_than_days} days beyond the last {keep}.[/dim]" + ) + return + if dry_run: + console.print( + f"[yellow]Dry run — would archive {len(to_archive)} transcripts " + f"to {archive_dir}:[/yellow]" + ) + for t in to_archive: + console.print(f" {t.name}") + return + archive_dir.mkdir(parents=True, exist_ok=True) + for t in to_archive: + t.rename(archive_dir / t.name) + console.print( + f"[green]Archived {len(to_archive)} transcripts to {archive_dir}[/green]" + ) + + @main.command() def status(): """Show Vera state.""" console.print(f"Home: {VERA_HOME}") console.print(f"Rules: {RULES_FILE} ({'exists' if RULES_FILE.exists() else 'missing'})") console.print(f"Memory: {len(list(MEMORY_DIR.glob('*.md'))) if MEMORY_DIR.exists() else 0} transcripts") + archive_dir = MEMORY_DIR / ARCHIVE_DIRNAME + if archive_dir.exists(): + console.print(f"Archive: {len(list(archive_dir.glob('*.md')))} transcripts") console.print(f"Audits: {len(list(AUDIT_DIR.glob('*.md'))) if AUDIT_DIR.exists() else 0} reports") + usd, calls = spend_since(PROVENANCE_LOG, days=30) + if calls: + console.print(f"Spend (30d): ${usd:.2f} over {calls} calls") if __name__ == "__main__": main() diff --git a/vera/config.py b/vera/config.py index 175c9ad..f4ab4cd 100644 --- a/vera/config.py +++ b/vera/config.py @@ -18,8 +18,13 @@ class Provider: audit_model: str def detect_provider() -> Provider: + audit_override = os.environ.get("VERA_AUDIT_MODEL") if os.environ.get("ANTHROPIC_API_KEY"): - return Provider("anthropic", "claude-sonnet-4-6", "claude-opus-4-6") + return Provider( + "anthropic", + "claude-sonnet-4-6", + audit_override or "claude-opus-4-6", + ) if os.environ.get("OPENAI_API_KEY"): - return Provider("openai", "gpt-4o", "gpt-4o") + return Provider("openai", "gpt-4o", audit_override or "gpt-4o") raise SystemExit("Set ANTHROPIC_API_KEY or OPENAI_API_KEY.") diff --git a/vera/llm.py b/vera/llm.py index 5d69d0c..99d0ef2 100644 --- a/vera/llm.py +++ b/vera/llm.py @@ -1,9 +1,39 @@ -"""Thin provider wrapper — Anthropic or OpenAI.""" +"""Thin provider wrapper — Anthropic or OpenAI. + +Returns (text, usage) where usage carries token counts, the model that +answered, and an estimated USD cost. Cost table is best-effort: if the +model is not in USD_PER_MTOK the usd field is None, so callers can log +without implying false precision. +""" from __future__ import annotations -from typing import Iterable from .config import Provider -def chat(provider: Provider, system: str, messages: list[dict], model: str | None = None) -> str: +# USD per million tokens, keyed by (provider, model). Update as vendors +# publish new pricing. Unknown models fall through to None. +USD_PER_MTOK: dict[tuple[str, str], tuple[float, float]] = { + ("anthropic", "claude-sonnet-4-6"): (3.0, 15.0), + ("anthropic", "claude-opus-4-6"): (15.0, 75.0), + ("anthropic", "claude-opus-4-7"): (15.0, 75.0), + ("anthropic", "claude-haiku-4-5-20251001"): (0.25, 1.25), + ("openai", "gpt-4o"): (2.5, 10.0), + ("openai", "gpt-4o-mini"): (0.15, 0.6), +} + + +def estimate_usd(provider_name: str, model: str, in_tokens: int, out_tokens: int) -> float | None: + rates = USD_PER_MTOK.get((provider_name, model)) + if rates is None: + return None + rate_in, rate_out = rates + return (in_tokens / 1_000_000) * rate_in + (out_tokens / 1_000_000) * rate_out + + +def chat( + provider: Provider, + system: str, + messages: list[dict], + model: str | None = None, +) -> tuple[str, dict]: model = model or provider.chat_model if provider.name == "anthropic": from anthropic import Anthropic @@ -14,11 +44,25 @@ def chat(provider: Provider, system: str, messages: list[dict], model: str | Non system=system, messages=messages, ) - return resp.content[0].text + text = resp.content[0].text + in_tok = getattr(resp.usage, "input_tokens", 0) + out_tok = getattr(resp.usage, "output_tokens", 0) elif provider.name == "openai": from openai import OpenAI client = OpenAI() oai_msgs = [{"role": "system", "content": system}] + messages resp = client.chat.completions.create(model=model, messages=oai_msgs, max_tokens=2048) - return resp.choices[0].message.content - raise ValueError(f"Unknown provider: {provider.name}") + text = resp.choices[0].message.content + in_tok = getattr(resp.usage, "prompt_tokens", 0) + out_tok = getattr(resp.usage, "completion_tokens", 0) + else: + raise ValueError(f"Unknown provider: {provider.name}") + + usage = { + "provider": provider.name, + "model": model, + "input_tokens": in_tok, + "output_tokens": out_tok, + "usd": estimate_usd(provider.name, model, in_tok, out_tok), + } + return text, usage diff --git a/vera/provenance.py b/vera/provenance.py index a9a2ea4..e3a6156 100644 --- a/vera/provenance.py +++ b/vera/provenance.py @@ -1,11 +1,13 @@ """Provenance tagging. Every autonomous memory write gets stamped.""" from __future__ import annotations import json -from datetime import datetime +from datetime import datetime, timedelta from pathlib import Path from typing import Literal -ProvenanceKind = Literal["OBSERVED", "INFERRED", "ASSUMED", "CANDIDATE", "RULE", "EXTERNAL"] +ProvenanceKind = Literal[ + "OBSERVED", "INFERRED", "ASSUMED", "CANDIDATE", "RULE", "EXTERNAL", "MALFORMED", +] SYSTEM_ADDENDUM = """ PROVENANCE RULES (non-negotiable). Every factual claim you write into memory @@ -31,3 +33,50 @@ def log_write(kind: ProvenanceKind, content: str, source: str, log_path: Path) - } with log_path.open("a") as f: f.write(json.dumps(entry) + "\n") + + +def log_usage(usage: dict, source: str, log_path: Path) -> None: + """Append a USAGE entry (one per LLM call) to the provenance log.""" + log_path.parent.mkdir(parents=True, exist_ok=True) + entry = { + "ts": datetime.now().isoformat(), + "kind": "USAGE", + "source": source, + "provider": usage.get("provider"), + "model": usage.get("model"), + "input_tokens": usage.get("input_tokens", 0), + "output_tokens": usage.get("output_tokens", 0), + "usd": usage.get("usd"), + } + with log_path.open("a") as f: + f.write(json.dumps(entry) + "\n") + + +def spend_since(log_path: Path, days: int) -> tuple[float, int]: + """Return (total_usd, call_count) for USAGE entries in the last N days. + + Unknown-cost calls (usd=None) contribute 0 USD but still count. + """ + if not log_path.exists(): + return 0.0, 0 + cutoff = datetime.now() - timedelta(days=days) + total = 0.0 + calls = 0 + for raw in log_path.read_text().splitlines(): + try: + entry = json.loads(raw) + except json.JSONDecodeError: + continue + if entry.get("kind") != "USAGE": + continue + try: + ts = datetime.fromisoformat(entry["ts"]) + except (KeyError, ValueError): + continue + if ts < cutoff: + continue + calls += 1 + usd = entry.get("usd") + if isinstance(usd, (int, float)): + total += usd + return total, calls