From 81a768d91b5b6b45ef4fe46b1c971a4e0da7b759 Mon Sep 17 00:00:00 2001 From: Claude Date: Tue, 13 Oct 2026 06:19:51 +0000 Subject: [PATCH 1/2] [docs] - Resync the knowledge and session skills and make the privacy verify script check something MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit fable-protocol's knowledge base restated CLAUDE.md's invariants, numbers, traps and git workflow and had drifted on four of them (max_context_tokens 8000 vs 16000, chunk 512/50 vs 256/32, a num_ctx floor of 13,596 vs the current 21,596, and a `harness` package in the I6 list that #1367 removed); those blocks now point at CLAUDE.md §2-§5/§10 and keep only what must be known cold. Its version history is condensed to a dated list, the audit log is described as keyed HMAC-SHA256 fingerprinting (not plain SHA-256), and the "/index/build is not key-gated" decision row says "while the key is unset". cyclaw-gotchas expires two gotchas under its own rule (the `--ignore-installed PyYAML` trap retired by the hashed locks in #1598, and the #1247 Codex P2 made moot by #1367), counts six stdlib guards, aligns its test guidance with CLAUDE.md §5, and stops asserting bare python3 is 3.11. cyclaw-privacy's heading loses the old cyclaw-advisor name, its audit and memory bullets name query_fingerprint and the HMAC key file, the data map gains logs/cyclaw.log and the eval ledger, and verify.sh now checks the shipped privacy posture in config.yaml (fingerprinted audit, redaction on, no off-box context, memory/auth off, third-party log level above DEBUG) instead of echoing "consistent (from previous analysis)"; bootstrap.sh runs it. CyClaw-Sandbox is reconciled against main @ 663a8bb: five ladders not six, the Guardrails table's historical numbering, the core six in every I6 statement, the console's third JS file and seven panels, /index/build's key gate once CYCLAW_API_KEY is set, the realism-tier auto-detection that only run_full_verification.py performs, the real coverage of test_terminal_consoles.py and gate_runtime_check.py, the auth UI that auth_admin.js already provides, contiguous step numbers, and the driver.sh/locks install in place of the retired requirements.txt recipe. run_full_verification.py's Phase 6 and test-specifications.md R1-R3 grepped gate.py for a redaction tuple that moved to utils/logger.py and utils/public_errors.py, so three checks failed on every run; they now read the real home and call public_error. verify.sh's header claimed nine stages (it has seven), and the /CyClaw-Sandbox wrapper claimed the skill opens a draft PR (CLAUDE.md §9: it opens none). Co-Authored-By: Claude Fable 5.1 Claude-Session: https://claude.ai/code/session_01LbrgG4NUbR3sUTgNMfKXpU --- .claude/commands/CyClaw-Sandbox.md | 10 +- .claude/skills/CyClaw-Sandbox/SKILL.md | 184 +++++++------- .../CyClaw-Sandbox/run_full_verification.py | 38 +-- .../CyClaw-Sandbox/test-specifications.md | 33 +-- .claude/skills/CyClaw-Sandbox/verify.sh | 4 +- .claude/skills/cyclaw-gotchas/SKILL.md | 83 +++---- .claude/skills/cyclaw-gotchas/driver.sh | 2 +- .claude/skills/cyclaw-privacy/SKILL.md | 28 ++- .claude/skills/cyclaw-privacy/bootstrap.sh | 26 +- .claude/skills/cyclaw-privacy/verify.sh | 67 +++-- .claude/skills/fable-protocol/SKILL.md | 29 +-- .../references/knowledge-base.md | 230 ++++++------------ 12 files changed, 342 insertions(+), 392 deletions(-) diff --git a/.claude/commands/CyClaw-Sandbox.md b/.claude/commands/CyClaw-Sandbox.md index b0337eaa1..219e0582e 100644 --- a/.claude/commands/CyClaw-Sandbox.md +++ b/.claude/commands/CyClaw-Sandbox.md @@ -5,7 +5,7 @@ description: > at three realism tiers, triple-gated Grok/Claude fallback, API key redaction, Phase 2 guardrails, due-diligence invariants, the full terminal console REST surface, and a Python-3.12 runtime gate. Produces a dated - report and opens a draft PR. + report. --- Invoke the `CyClaw-Sandbox` skill and run the full sandbox audit: clone, @@ -15,10 +15,10 @@ See `.claude/skills/CyClaw-Sandbox/SKILL.md` for the full audit (its Operator map + "Steps -- the full audit" section): realism ladder, config validation, gate/graph standalone checks, the 5-query swarm test, triple-gate + key redaction, due-diligence invariants, guardrails, the -full terminal- and harness-console REST + slash-command surface, the 3.12 -runtime gate, the Guardrails invariant table, and the report/PR machinery. +full terminal-console REST + slash-command surface, the 3.12 runtime gate, +the Guardrails invariant table, and the sign-off report. -For a fast check against your current checkout with no clone/report/PR — +For a fast check against your current checkout with no clone or report — use the skill's **Quick Mode** section instead (it absorbed the former separate `/run-cyclaw` command; `/run` still ships as the thin Quick-Mode entry point): @@ -33,5 +33,5 @@ bash .claude/skills/CyClaw-Sandbox/smoke.sh - Run the highest Ollama realism tier the environment supports (Tier 0 stubs / Tier 1 mock-over-HTTP / Tier 2 real daemon) and record which tier was used — never claim Tier 2 realism from a Tier 0/1 run. -- Draft PR only; a human decides when to merge. +- This skill opens no PR (CLAUDE.md §9); it produces a dated sign-off report. - `status: degraded` without live Ollama is expected and normal. diff --git a/.claude/skills/CyClaw-Sandbox/SKILL.md b/.claude/skills/CyClaw-Sandbox/SKILL.md index 6557cac46..a3c79ebd9 100644 --- a/.claude/skills/CyClaw-Sandbox/SKILL.md +++ b/.claude/skills/CyClaw-Sandbox/SKILL.md @@ -25,7 +25,7 @@ local-LLM realism tier was actually live; the Grok/Claude triple gate holds end to end with zero real API spend; every terminal-console REST endpoint and slash command this document lists is registered and behaves per its documented contract; the 14 due-diligence invariant classes -and the 26-row Guardrails table below all hold; and the out-of-band +and every row of the Guardrails table below hold; and the out-of-band subsystems (spend ledger, Numbat, sequence detection, memory, Telegram, OpenTweet, unslop, MCP manifest pin, OS scheduler glue) behave per their own documented defaults. It does **not** mean live 27b-quality generation was @@ -39,21 +39,17 @@ drift (cite the enforcer, don't just trust this document); `(derive)` means read it from the running code/tree at verification time, never copy it from here. **Code, `config.yaml`, and the tests currently on disk always win over this document.** Surface inventory below was last reconciled against main -@ `571388a0` (2026-09-16) -- if it disagrees with what you see on +@ `663a8bb` (2026-10-13) -- if it disagrees with what you see on a fresh checkout, trust the checkout and treat the disagreement as this document's own next drift-fix. ## Operator map -- which ladder proves what -Six ladders. They are complementary, not substitutable -- a green run of -one is **not** evidence the others would also pass. Always invoke with -`python3.12` (never bare `python3` -- see Gotchas). - -**Claude Code's own `/memory` is not this skill either** -- this ladder tests -CyClaw's `/memory/*` REST API. The project's own `memory-orchestrator` skill, -which used to wire session memory to `docs/memories/` via `PreCompact`/ -`SessionEnd` hooks, was deleted 2026-09-16 (issue #1351); those hooks are now -unwired in `.claude/settings.json` rather than pointing at a missing script. +Five ladders (B–F; there is no ladder A). They are complementary, not +substitutable -- a green run of one is **not** evidence the others would +also pass. Always invoke with `python3.12` (never bare `python3` -- see +Gotchas). Claude Code's own `/memory` and the repo's `docs/memories/` are not +this skill either -- this ladder tests CyClaw's `/memory/*` REST API. | Ladder | Command | Proves | Does **not** prove | |---|---|---|---| @@ -81,10 +77,11 @@ them; do not "fix" the numbers. Stage 1's hard failure on a missing/wrong `python3.12` is deliberate, not a gap against this repo's usual "skip cleanly without deps" skill convention (CLAUDE.md §6): the Python-3.12 runtime gate is this script's own -advertised feature (CLAUDE.md §4 routes "suspected config drift" reports +advertised feature (CLAUDE.md §7 routes "suspected config drift" reports here specifically because bare `python3`/`pytest` silently run the wrong -interpreter and fail ~142 tests in a way that looks like a red `main`). -Every other stage degrades gracefully; this one is supposed to stop you. +interpreter and fail the `test_agentic_*` files in a way that looks like a +red `main`, CLAUDE.md §4). Every other stage degrades gracefully; this one +is supposed to stop you. ## Where the surfaces live @@ -93,11 +90,11 @@ regression on its own, independent of this skill. | Surface | File(s) | What to verify | Pinned by | |---|---|---|---| -| Gate core routes | `gate.py` | `/`, `/health`, `/query`, `/soul*`, `/audit/summary`, `/index/build`, `/index/status`; `POST /query` rejects cross-site browser requests (403 `CROSS_SITE_BLOCKED`, from `Origin`/`Sec-Fetch-Site`) **regardless of `auth.enabled`** -- a request carrying neither header is allowed; `GET /index/status` is unauthenticated and **not** rate-limited (the console polls it every 1.5s, same posture as `/health`) | `gate_runtime_check.py` | +| Gate core routes | `gate.py` | `/`, `/health`, `/query`, `/soul*`, `/audit/summary`, `/index/build`, `/index/status`, `GET`/`POST /console/session`, `POST /console/session/end`; `POST /query` rejects cross-site browser requests (403 `CROSS_SITE_BLOCKED`, from `Origin`/`Sec-Fetch-Site`) **regardless of `auth.enabled`** -- a request carrying neither header is allowed; `GET /index/status` is unauthenticated and **not** rate-limited (the console polls it every 1.5s, same posture as `/health`) | `gate_runtime_check.py` (a route *subset*: it does not pin `/audit/summary`, `/soul/restore` or the console-session routes; `tests/test_gate.py` does) | | Ops routes | `gate_ops.py` | `POST /ops/{sync,agentic,fsconnect,sqlconnect}`; every `action` field is a closed `Literal` (`schemas/api.py`) -- an unrecognized value is a **422**, not a handler-level 400 | `test_terminal_consoles.py` | -| Auth routes | `gate_auth.py` | 13 paths (`/auth/setup-status`, `/bootstrap-password`, `/login`, `/logout`, `/whoami`, `/users` GET+POST, `/password`, `/users/{u}/password`, `/users/{u}/role`, `/users/{u}/disable`, `/users/{u}/enable`, `/users/{u}` DELETE, `/audit/summary`); every route exists and answers **503** (not 404) when `auth.enabled` is false (the shipped default); identity attaches to `POST /query` only when an `AuthManager` exists; `audit` role is forbidden from `/query`; account lockout answers **423**; a raced duplicate username or device-token label is a typed conflict (`AuthUserExists`/`AuthTokenLabelExists` -> **409**, mirrored by the harness create-user route, which previously 500'd) | `test_auth_admin_contract.py`, `tests/test_due_diligence_invariants.py` | +| Auth routes | `gate_auth.py` | 13 paths (`/auth/setup-status`, `/bootstrap-password`, `/login`, `/logout`, `/whoami`, `/users` GET+POST, `/password`, `/users/{u}/password`, `/users/{u}/role`, `/users/{u}/disable`, `/users/{u}/enable`, `/users/{u}` DELETE, `/audit/summary`); every route exists and answers **503** (not 404) when `auth.enabled` is false (the shipped default); identity attaches to `POST /query` only when an `AuthManager` exists; `audit` role is forbidden from `/query`; account lockout answers **423**; a raced duplicate username or device-token label is a typed conflict (`AuthUserExists`/`AuthTokenLabelExists` -> **409**) | `test_auth_admin_contract.py`, `tests/test_due_diligence_invariants.py` | | Memory routes | `gate_memory.py` | `/memory/status` is always 200 (probeable when the subsystem is off); `/memory/{facts,episodes,proposals,propose,apply,reject}` and `/query/export/html` are **404** when their toggle is off (all ship false); flag resolution goes through `memory/flags.py` -- the fact-retrieval switch is `memory.facts.retrieval_enabled` (legacy `memory.facts.enabled` honored with a one-time warning), the status payload reports `facts_retrieval_enabled`, and every flag echo is a strict boolean (`is True` -- a YAML `"false"` string reports off) | `test_memory_isolation.py`-style isolation + live probe | -| Terminal console | `static/terminal.html` + `static/terminal.js` (CSP forces `script-src 'self'`, so the console's JS logic lives in the sibling file -- read both together) | 5 toolbar panels (Soul/Sync/Agentic/FS/SQL); the confirm dialog's generic `handleConfirm(confirmed, entryId, onlineProvider)`; the four slash commands `/users /admin /audit /help` (everything else is a toolbar button or a RAG query, not a command) | `tests/test_terminal_contract.py` (reads `terminal.html + terminal.js` combined; pins the 5 POST-only paths) | +| Terminal console | `static/terminal.html` + `static/terminal.js` + `static/auth_admin.js` (CSP forces `script-src 'self'`, so the console's JS logic lives in the sibling files -- read all three together) | 7 toolbar panels (Soul/Sync/Agentic/FS/SQL, plus Users and Audit, which stay hidden until `auth.enabled`); the confirm dialog's generic `handleConfirm(confirmed, entryId, onlineProvider)`; the four slash commands `/users /admin /audit /help` (everything else is a toolbar button or a RAG query, not a command) | `tests/test_terminal_contract.py` (reads the HTML and JS combined; its `_POST_PATHS` pins every POST-only path -- derive the count from the file, it grows) | | MCP | `mcp_hybrid_server.py` | Single tool `hybrid_search`; `sampling: None`; `check_input` runs before retrieval; `mcp_manifest.json` SHA-256 drift pin | `tests/test_mcp_server.py`, `tests/test_mcp_manifest.py` | | Graph | `graph.py` | 12 nodes (`retrieve`, `route_by_score`, `guardrail_input`, `guardrail_output`, `local_llm`, `user_gate`, `pre_action_hook_grok`, `pre_action_hook_claude`, `grok_fallback`, `claude_fallback`, `offline_best_effort`, `audit_logger`); 4 routers (`score_router`, `guardrail_router`, `pre_action_hook_router`, `user_gate_router`) | `tests/test_operator_docs_node_count.py` (`>=12`), `tests/test_graph.py` | | CLIs | `sync/cli.py`, `agentic/cli.py`, `agentic/fsconnect/cli.py`, `agentic/sqlconnect/cli.py`, `agentic/netconnect/cli.py`, `telegram/cli.py`, `opentweet/cli.py`, `guardrails/cli.py`, `utils/authn_cli.py`, `utils/gen_cert.py`, `retrieval/indexer.py`, `retrieval/clear_cache.py`, `metrics.py`, `utils/telemetry_kill.py --export` | Name + exit-code contract only (`--help` for the current subcommand set -- never hardcode a subcommand count in this document, it drifts every time a CLI grows one) | `tests/test_*_cli.py` per connector | @@ -136,8 +133,9 @@ includes an `sk-ant-*` pattern, `policy.fallback.pre_action_hook.enabled` Verify module isolation (I6): `agentic/` (including `agentic/netconnect/`), `sync/`, `guardrails/`, `telegram/`, and `opentweet/` are never imported by -`gate.py`, `graph.py`, or `mcp_hybrid_server.py`, and vice versa; the -`agentic.*.cli` modules run only via subprocess from `utils/ops_runner.py`. +the core six (`gate.py`, `gate_ops.py`, `gate_auth.py`, `gate_memory.py`, +`graph.py`, `mcp_hybrid_server.py`), and vice versa; the `agentic.*.cli` +modules run only via subprocess from `utils/ops_runner.py`. Verify code structure: `graph.py` has `_external_fallback_node(state, client, cfg, *, provider, label)`; `grok_fallback_node`/`claude_fallback_node` @@ -150,33 +148,22 @@ landed -- the vendor-echoed model id forwarded onto the audit record). Requirements: Python 3.12 exactly, `rank-bm25`, PyYAML, numpy, httpx, and (full mode) chromadb/sentence-transformers/langgraph/fastapi/uvicorn. -**In a Claude Code cloud sandbox** (CLAUDE.md §4's documented realities, -not a generic sandbox checklist): the default image ships Python -3.10/3.11/3.12/3.13 side by side, but `update-alternatives` and the -pre-installed dependencies both point at 3.11 -- bare `python3`/`pytest` -silently run the wrong interpreter and fail ~142 `test_agentic_*` tests on -a 3.12-only stdlib parameter, which looks like a red `main`, not a version -mismatch. Build an explicit venv instead: -```bash -python3.12 -m venv /root/.venv-cyclaw-312 -/root/.venv-cyclaw-312/bin/pip install torch==2.13.0+cpu \ - --index-url https://download.pytorch.org/whl/cpu -/root/.venv-cyclaw-312/bin/pip install -r requirements.txt -r requirements-test.txt \ - -c constraints.txt --ignore-installed PyYAML -``` -If the PyTorch CPU index is blocked by an outbound proxy, install plain -`torch` and feed pip **scratchpad copies** of `requirements.txt`/ -`constraints.txt` with the `torch==`/`--extra-index-url` lines stripped -- -never edit the repo's own manifests (macOS needs this same plain-torch path -for a different reason: no `+cpu` wheel exists on the arm64 index). The -venv does not survive session end; treat this as a per-session setup step, -always invoked as `/root/.venv-cyclaw-312/bin/python`, never bare `python3`. +Install per CLAUDE.md §8 (the hashed `locks/`, torch lock first). **In a +Claude Code cloud sandbox** bare `python3`/`pytest` are never 3.12 and carry +none of CyClaw's deps (CLAUDE.md §4), so build the venv with +`bash .claude/skills/cyclaw-gotchas/driver.sh venv` -- it is the documented +§8 install plus the plain-torch fallback for a proxy that denies the PyTorch +CPU index, and the cloud SessionStart hook already starts it in the +background. The venv does not survive session end; always invoke +`/root/.venv-cyclaw-312/bin/python`, never bare `python3`. Note the sandbox +proxy also denies `huggingface.co`, so the real index (ladders C–F) cannot +be built there; ladder B's deterministic doubles still run. `GROK_API_KEY=dummy` (any non-empty value) is sufficient for these tests. Startup does not require `GROK_API_KEY`; Grok reports unavailable without it. -**Full dependency install** (preferred when network access allows): -`pip install -e ".[test,full]"`. +**Editable install with extras** (when network allows and you need the +console scripts): `pip install -e ".[test,full]" -c constraints.txt`. **In-process swarm dependencies**: `run_full_verification.py` uses the real installed LangGraph runtime and deterministic in-memory doubles for ChromaDB @@ -190,8 +177,11 @@ the first 6 characters in notes. ### 3. Static gates (no server) +Ladder F is the owner's full audit, so the whole suite runs here on purpose; +outside this ladder CLAUDE.md §5 leaves that run to GitHub Actions. + ```bash -GROK_API_KEY=dummy python3.12 -m pytest tests/ -q --tb=short # (derive) record N passed / N total +GROK_API_KEY=dummy python3.12 -m pytest tests/ -o addopts="" -q --tb=short # (derive) record N passed / N total python3 .claude/skills/invariant-guard/check_invariants.py # (derive) record N/N GROK_API_KEY=dummy python3.12 -m pytest tests/test_due_diligence_invariants.py -q # 14 classes python3 .claude/skills/doc-sync/doc_sync.py @@ -225,14 +215,15 @@ headers (`X-Content-Type-Options`, `X-Frame-Options`, `Referrer-Policy`, `Permissions-Policy`, `Content-Security-Policy`); `/auth/*` returns 503 (not 404) while `auth.enabled` is false; all four terminal slash commands respond; `POST /query` rejects cross-site requests (403 `CROSS_SITE_BLOCKED`) -even with auth off; `/index/build` is loopback+same-origin gated, NOT -key-gated (deliberate -- an unset key must not brick a first-run index -build), while `/index/status` is open and un-rate-limited (poll target). Enabling auth in a scratch config additionally exercises: bootstrap +even with auth off; `/index/build` is loopback+same-origin gated and, once +`CYCLAW_API_KEY` is set (as it is in this step), also key-gated -- it is +deliberately NOT key-gated while the key is unset so a first-run build cannot +be bricked; `/index/status` is open and un-rate-limited (poll target). Enabling auth in a scratch config additionally exercises: bootstrap from loopback only, session cookie + CSRF on `/query`, RBAC (`audit` role 403s on `/query`), and TLS via `cyclaw-gen-cert` (session cookie gains `secure` once `api.tls.enabled: true`). -### 7. Out-of-band subsystems +### 6. Out-of-band subsystems - **Spend** (`utils/spend.py`, `logs/spend.jsonl`): append-only; each real external generate appends one record, including the vendor-served model @@ -252,7 +243,7 @@ from loopback only, session cookie + CSRF on `/query`, RBAC (`audit` role - **Sequence detection** (`utils/sequence_detect.py`, forensic-only): CLI or `cyclaw-metrics`-joined surface lists its rule set; craft synthetic audit+spend fixtures and confirm a suspicious pattern is flagged and a - clean log is not; `grep -rn sequence_detect gate.py graph.py mcp*.py` + clean log is not; `grep -rn sequence_detect gate*.py graph.py mcp*.py` returns nothing (I6-style isolation, even though it isn't in the formal out-of-band package set). - **Memory** (`gate_memory.py` + package `memory/`, ships every switch @@ -281,7 +272,7 @@ from loopback only, session cookie + CSRF on `/query`, RBAC (`audit` role fall inside an explicitly configured RFC1918/loopback CIDR; an empty or overly broad scope fails closed rather than defaulting open. -### 8. MCP + OS glue +### 7. MCP + OS glue - `mcp_manifest.json`'s SHA-256 drift pin: flip a byte in a scratch copy and confirm drift is reported and fails closed. `hybrid_search` works @@ -296,29 +287,30 @@ from loopback only, session cookie + CSRF on `/query`, RBAC (`audit` role silent fallback to a different backend; cron-line ownership stays scoped to CyClaw-managed lines only. -### 9. Groundedness evaluator (opt-in) +### 8. Groundedness evaluator (opt-in) `CYCLAW_EVAL_LIVE=1 ANTHROPIC_API_KEY=... python3.12 tests/judge_eval.py` against a loopback LLM: exit 0 pass / 1 fail / 2 infra. Without live keys, verify the gate refuses cleanly (exit 2) rather than silently skipping. -### 10. Report +### 9. Report End with a sign-off naming: the repo sha/date and Python/pyproject versions actually used; which local-LLM realism tier was live (0 = no responding -HTTP backend, 1 = `mock_ollama.py`, 2 = real Ollama daemon -- **both** -`run_full_verification.py` and `verify.sh` auto-detect and report this, -don't hand-guess it); per-item PASS/FAIL/SKIP for every numbered item -above; the due-diligence and Guardrails pass counts (report the actual -`N/14` and `N/26` from the run, never copy last time's numbers); and any -Known Residual (below) that changed since it was last confirmed. +HTTP backend, 1 = `mock_ollama.py`, 2 = real Ollama daemon -- +`run_full_verification.py` auto-detects and reports this; `verify.sh` does +not probe for a daemon, so state the tier for its stages yourself); +per-item PASS/FAIL/SKIP for every numbered item above; the due-diligence +and Guardrails pass counts (report the actual `N/14` and `N/` from the +run, never copy last time's numbers); and any Known Residual (below) that +changed since it was last confirmed. ## Quick Mode (`/run`) `bash .claude/skills/CyClaw-Sandbox/smoke.sh` -- a fast, live-server pass covering sections A-G (core API, fsconnect, sqlconnect, NeMo soft-import, -Postgres-backend skip-cleanly, and the full pytest suite as its final -section). It builds its own index if one is missing, starts and stops its +Postgres-backend skip-cleanly, the full pytest suite as section F, and the +summary report as G). It builds its own index if one is missing, starts and stops its own `gate.py`, and writes `.claude/sandbox-test.txt`. It does **not** build the mock RAG corpus or walk the due-diligence invariant classes -- a green Quick Mode is not evidence the full Ladder F audit would also pass, and the reverse @@ -335,28 +327,30 @@ subdirectory -- invoke them by that path. cloning fresh (warns before writing mock corpus/index/report files into it). Unset `CYCLAW_REPO` clones into a unique temp directory, copies `verification_report.json` and `query_results.json` to the invoking cwd, - then deletes that temp directory. Both this script and `verify.sh` - auto-detect the live Ollama realism tier. + then deletes that temp directory. This script auto-detects the live + Ollama realism tier; `verify.sh` does not. - **`gate_runtime_check.py`** -- independent, import-time-only checks: app - builds, telemetry-kill maps are active, the expected route subset - registers, auto-docs stay disabled, entry points are callable. + builds and is a FastAPI instance, telemetry-kill maps are active, the + expected route subset registers, `main` is callable. - **`terminal_emulation.py`** -- exercises the exact HTTP fetch lifecycle the console's own JS performs, against an already-running server. Wired into `verify.sh` (stage 7); also runnable standalone. - **`test_terminal_consoles.py`** -- stdlib-`urllib` integration test - against a running `gate.py` with `CYCLAW_API_KEY` set. Asserts every - `/soul/*`/`/ops/*` route's auth gate, an unknown `action` 422s at the - schema boundary (closed `Literal` fields, not a handler-level 400), and - the security-header/DROP-rejection contracts. + against a running `gate.py` with `CYCLAW_API_KEY` set. Asserts the auth + gate on `/soul`, `/soul/propose` and the four `/ops/*` routes (the + remaining `/soul/*` routes are covered by `tests/test_gate.py`), an + unknown `action` 422s at the schema boundary (closed `Literal` fields, not + a handler-level 400), and the security-header/DROP-rejection contracts. - **`mock_ollama.py`** -- stdlib-only mock Ollama/OpenAI-compatible server (`/api/tags`, `/api/chat`, `/api/generate`, `/v1/models`, `/v1/chat/completions`) for deterministic offline chat testing. This is - realism **Tier 1** of three: Tier 0 is the in-process pytest stub - (`MockLocalLLM` in `tests/conftest.py`), Tier 2 is a real Ollama daemon. + realism **Tier 1** of three: Tier 0 is no responding HTTP backend at all + (the pytest suite's in-process `MockLocalLLM` stub never touches HTTP), + Tier 2 is a real Ollama daemon. - **`verify.sh`** -- the CI-wired, full Linux lifecycle: Python 3.12 - provisioning, the pytest suite, an emulated RAG query, both independent - runtime checks, then both consoles launched live and emulated end to - end. Non-sequential stage labels, see the table above -- never renumber. + provisioning, the pytest suite, an emulated RAG query, the runtime check, + then the terminal console launched live and emulated end to end. + Non-sequential stage labels, see the table above -- never renumber. - **`smoke.sh`** -- Quick Mode, see above. - **`windows-smoke.ps1`** / **`macos-smoke.sh`** -- the platform live-API bombs against already-running servers (they start nothing themselves). @@ -388,13 +382,13 @@ report one of these as a new finding. 2. Telegram's T4 media handling is partial and POSIX-only. 3. Episode-to-fact consolidation is not implemented (the former `memory/consolidation.py` stub was removed). -4. `terminal.html` has no memory console and no full auth-management UI -- - both are REST-only surfaces today (a minimal `.toolbar-auth` affordance - exists for `/users`/`/audit`). +4. `terminal.html` has no memory console (REST-only). The auth UI + (`static/auth_admin.js`: Users and Audit panels -- list/create users, + change roles, reset passwords, delete; `.toolbar-auth` is the setup/login + box) lacks only disable/enable and self-service password change. 5. `embeddings_local`'s health-check entry is static by design (it does not depend on whether the model has actually loaded) -- not a finding. -6. Startup does not require `GROK_API_KEY`; Grok reports unavailable without it. -7. Hand-run `uvicorn gate:app` (bypassing the shipped launcher or Docker +6. Hand-run `uvicorn gate:app` (bypassing the shipped launcher or Docker CMD) still gets the canonical telemetry/update-check env only at module import, not before the interpreter starts -- documented in `docs/THREAT_MODEL.md`, accepted. @@ -402,7 +396,9 @@ report one of these as a new finding. ## Guardrails Restates the security invariants this skill verifies (see `CLAUDE.md` §3 -for the six canonical ones plus supporting guards this table extends). +for the six canonical ones plus supporting guards this table extends). Row +numbers are historical labels (18–21 were retired); count the rows you +report, never assume. | # | Invariant | Check | |---|---|---| @@ -418,11 +414,11 @@ for the six canonical ones plus supporting guards this table extends). | 10 | FsConnect Op Whitelist | Capability list is closed (derive the current set from `config.yaml`'s `fsconnect.allowed_fs_ops` -- it grows over time, e.g. `fs_largest`) | | 11 | SQL Read-Only Default | `sqlconnect.read_only=true`, `allow_write=false` | | 12 | SQL Query Guard | Only SELECT/WITH; comments and `;` rejected | -| 13 | Module Isolation (I6) | `agentic` (including `agentic/netconnect`), `sync`, `guardrails`, `telegram`, and `opentweet` never imported by `gate.py`/`graph.py`/`mcp_hybrid_server.py`, and vice versa | +| 13 | Module Isolation (I6) | `agentic` (including `agentic/netconnect`), `sync`, `guardrails`, `telegram`, and `opentweet` never imported by the core six (`gate*.py`, `graph.py`, `mcp_hybrid_server.py`), and vice versa | | 14 | Soul Privacy | Soul preamble never forwarded to Grok/Claude (off-box) | -| 15 | API Key Gate | All mutating gate.py routes require `CYCLAW_API_KEY`, fail-closed on an unset key | -| 16 | Rate Limit | Every route shares the per-IP `RateLimiter` | -| 17 | Key Redaction Parity | `ANTHROPIC_API_KEY` redacted the same as `GROK_API_KEY`; `sk-ant-*` pattern present in both `gate.py` and `config.yaml` | +| 15 | API Key Gate | Every `/soul/*`, `/ops/*`, `/memory/*`, `/audit/summary` and `/query/export/html` route requires `CYCLAW_API_KEY` (or the console/admin cookie) and fails closed on an unset key; `/index/build` joins once the key is set; `/query` and `/auth/*` use session/same-origin gates instead (CLAUDE.md §2 route table) | +| 16 | Rate Limit | Every route shares the per-IP `RateLimiter` except `GET /health` and `GET /index/status`, which are deliberately open poll targets | +| 17 | Key Redaction Parity | `redact_sensitive` (`utils/logger.py`) redacts every `os.environ` value, so `ANTHROPIC_API_KEY` and `GROK_API_KEY` are treated alike; the `sk-ant-*` shape is in both `_MINIMUM_SECRET_PATTERNS` and `config.yaml` `policy.privacy.redact_secrets_like` | | 22 | Auth Disabled By Default | `auth.enabled=false` ships default, matching every other opt-in subsystem | | 23 | Auth Route Presence Doesn't Disclose State | Every `/auth/*` route always returns 503, never 404, while `auth.enabled` is false | | 24 | Auth Stage 3 Enforced | The identity dependency (`require_session_or_token`) attaches to `POST /query` whenever an `AuthManager` exists; the `audit` role is forbidden from `/query`; the 503-when-disabled contract is unchanged when auth stays off | @@ -438,9 +434,9 @@ for the six canonical ones plus supporting guards this table extends). Operator map's Ladder D row -- treat them as complementary, never substitutable in either direction. - **Three local-LLM realism tiers, not one.** In the in-process swarm, Tier 0 - means no responding HTTP backend; Tier 1 = `mock_ollama.py`; Tier 2 = a real daemon. Both - `run_full_verification.py` and `verify.sh` auto-detect which one is - live and report it -- still state the tier honestly in your own sign-off + means no responding HTTP backend; Tier 1 = `mock_ollama.py`; Tier 2 = a real daemon. + `run_full_verification.py` auto-detects which one is live and reports it; + `verify.sh` does not -- state the tier honestly in your own sign-off rather than assuming Tier 2 realism from a Tier 0/1 run. - **`run_full_verification.py` writes into whatever `CYCLAW_REPO` points at** from Phase 3 onward (mock corpus, BM25 index, two JSON report @@ -453,15 +449,11 @@ for the six canonical ones plus supporting guards this table extends). especially when a prior command in the same session already started a server under a similar name. -## Mock Embedding Implementation - -`MockSentenceTransformer` creates sparse keyword-based 384-dim vectors: -- Each word hashes to 3 dimension slots via SHA-256 -- Slot values accumulate per word occurrence -- Final vector L2-normalized +## Mock embedding and Chroma doubles -`MockChromaClient` / `MockCollection` implement: -- `add(embeddings, documents, metadatas, ids)` -- append documents -- `query(query_embeddings, n_results)` -- cosine similarity search -- `get_or_create_collection(name)` / `get_collection(name)` -- singleton - collection registry with the current fingerprint metadata contract +Ladder B's deterministic doubles live in `run_full_verification.py`: +`MockSentenceTransformer` (sparse SHA-256 keyword hashing into 384 dims, +L2-normalized) and `MockChromaClient`/`MockCollection` (cosine search over a +singleton collection registry carrying the fingerprint metadata contract). +Read the script for the current shapes; they track `retrieval/` and drift +if copied here. diff --git a/.claude/skills/CyClaw-Sandbox/run_full_verification.py b/.claude/skills/CyClaw-Sandbox/run_full_verification.py index 277d9e803..3d1049bce 100644 --- a/.claude/skills/CyClaw-Sandbox/run_full_verification.py +++ b/.claude/skills/CyClaw-Sandbox/run_full_verification.py @@ -69,7 +69,7 @@ _REPORT_NAMES = ("verification_report.json", "query_results.json") # Which of the 3-tier Ollama realism ladder this run actually exercised (see -# SKILL.md's "3-tier realism" section). Set once in main() +# SKILL.md's "Operator map" realism tiers). Set once in main() # by _probe_ollama_tier() before any phase that talks to the local-LLM base_url # runs, then stamped into both report files this script writes. OLLAMA_TIER: int | None = None @@ -955,34 +955,34 @@ def phase_key_redaction() -> PhaseResult: banner("Phase 6: API Key Redaction (Grok + Claude Parity)") phase = PhaseResult("Key Redaction") - gate_src = Path("gate.py").read_text() + # Redaction lives in utils/logger.py (redact_sensitive: every os.environ + # value plus _MINIMUM_SECRET_PATTERNS) and reaches HTTP errors through + # utils/public_errors.public_error; gate.py no longer carries its own + # tuple (the old gate.py greps failed on every run once it moved). + logger_src = Path("utils/logger.py").read_text() - # Check ANTHROPIC_API_KEY in env-var redaction tuple - has_anthropic_env = "ANTHROPIC_API_KEY" in gate_src - log(f" [{'PASS' if has_anthropic_env else 'FAIL'}] ANTHROPIC_API_KEY in env-var redaction") - phase.checks.append(Check("anthropic_key_env_redaction", has_anthropic_env)) + has_env_scan = "os.environ.items()" in logger_src and "def redact_sensitive" in logger_src + log(f" [{'PASS' if has_env_scan else 'FAIL'}] redact_sensitive scans every os.environ value (Grok + Anthropic parity)") + phase.checks.append(Check("env_value_redaction", has_env_scan)) - # Check sk-ant-* pattern in _SECRET_PATTERNS - has_sk_ant_pattern = "sk-ant-" in gate_src - log(f" [{'PASS' if has_sk_ant_pattern else 'FAIL'}] sk-ant-* pattern in _SECRET_PATTERNS") - phase.checks.append(Check("sk_ant_pattern_in_gate", has_sk_ant_pattern)) + has_sk_ant_pattern = "sk-ant-" in logger_src + log(f" [{'PASS' if has_sk_ant_pattern else 'FAIL'}] sk-ant-* pattern in utils/logger.py _MINIMUM_SECRET_PATTERNS") + phase.checks.append(Check("sk_ant_pattern_in_logger", has_sk_ant_pattern)) - # Check GROK_API_KEY is still there (regression check) - has_grok_env = "GROK_API_KEY" in gate_src - log(f" [{'PASS' if has_grok_env else 'FAIL'}] GROK_API_KEY in env-var redaction (regression)") - phase.checks.append(Check("grok_key_env_redaction", has_grok_env)) + has_public_error = "redact_sensitive" in Path("utils/public_errors.py").read_text() + log(f" [{'PASS' if has_public_error else 'FAIL'}] public_error routes HTTP error text through redact_sensitive") + phase.checks.append(Check("public_error_uses_redaction", has_public_error)) # Test redaction of an Anthropic key in error messages try: - # Import and test _sanitize_error if available - from gate import _sanitize_error + from utils.public_errors import public_error test_msg = "Error: API call failed with key sk-ant-api03-testkey123456789" - sanitized = _sanitize_error(test_msg) - redacted = "sk-ant" not in sanitized or "[REDACTED]" in sanitized + sanitized = public_error(test_msg, {}) + redacted = "sk-ant-api03" not in sanitized log(f" [{'PASS' if redacted else 'FAIL'}] Anthropic key sk-ant-api03-... redacted in errors") phase.checks.append(Check("anthropic_key_sanitized", redacted)) except (ImportError, AttributeError) as e: - log(f" {Y}SKIP{N}] _sanitize_error not importable: {e}") + log(f" {Y}SKIP{N}] public_error not importable: {e}") phase.checks.append(Check("anthropic_key_sanitized", False, str(e))) # Verify in config.yaml -- policy.privacy, not logging.audit (Phase 1's diff --git a/.claude/skills/CyClaw-Sandbox/test-specifications.md b/.claude/skills/CyClaw-Sandbox/test-specifications.md index f90b0830f..7f7b5bddd 100644 --- a/.claude/skills/CyClaw-Sandbox/test-specifications.md +++ b/.claude/skills/CyClaw-Sandbox/test-specifications.md @@ -197,32 +197,35 @@ When `online_provider` is not set, `user_gate_router` defaults to "grok" ### Background Commit `78515b0` added Anthropic key redaction with the same rigor as Grok: -- `ANTHROPIC_API_KEY` added to env-var redaction tuple in `gate.py` -- `sk-ant-[A-Za-z0-9_\-]{20,}` pattern added to `_SECRET_PATTERNS` -- `sk-ant-*` pattern added to `config.yaml` `audit.redact_secrets_like` +- every `os.environ` value is redacted by `utils/logger.py::redact_sensitive` + (so `ANTHROPIC_API_KEY` and `GROK_API_KEY` get the same treatment) +- `sk-ant-[A-Za-z0-9_-]{20,}` pattern in `utils/logger.py` `_MINIMUM_SECRET_PATTERNS` +- `sk-ant-*` pattern in `config.yaml` `policy.privacy.redact_secrets_like` +- HTTP error text reaches the client only through `utils/public_errors.public_error` + (the redaction tuple that once lived in `gate.py` moved; R1/R2 grep the new home) ### Test Cases #### R1: Env-Var Redaction ```python -def test_anthropic_key_env_redaction(): - gate_src = Path("gate.py").read_text() - assert "ANTHROPIC_API_KEY" in gate_src - assert "GROK_API_KEY" in gate_src +def test_env_value_redaction(): + logger_src = Path("utils/logger.py").read_text() + assert "def redact_sensitive" in logger_src + assert "os.environ.items()" in logger_src ``` -#### R2: Pattern Redaction in gate.py +#### R2: Pattern Redaction in utils/logger.py ```python def test_sk_ant_pattern_in_secret_patterns(): - gate_src = Path("gate.py").read_text() - assert "sk-ant-" in gate_src + logger_src = Path("utils/logger.py").read_text() + assert "sk-ant-" in logger_src ``` #### R3: Pattern Redaction in config.yaml ```python -def test_sk_ant_in_audit_config(): +def test_sk_ant_in_privacy_config(): cfg = yaml.safe_load(Path("config.yaml").read_text()) - patterns = cfg["logging"]["audit"]["redact_secrets_like"] + patterns = cfg["policy"]["privacy"]["redact_secrets_like"] assert any("sk-ant" in str(p) for p in patterns) ``` @@ -304,7 +307,7 @@ From `tests/test_due_diligence_invariants.py` (14 test classes): - `TestShippedAgenticConfigContract` (`tests/test_agentic_config.py`) already pins the agentic block against the real file; the core request path had no equivalent, and the gap is not hypothetical -- the operator commit of - 2026-08-07 flipped `app.mode`, `app.debug`, and BOTH `models..enabled` + 2026-08-07 flipped `app.mode`, the since-removed `app.debug`, and BOTH `models..enabled` in the shipped file and no test broke, because no test read those keys from it - Matters specifically for I3: per `INVARIANTS.md`, two of the triple gate's three conditions (`mode == "hybrid"`, `.enabled`) are NOT enforced @@ -407,8 +410,8 @@ policy: send_local_context_to_claude: false grok_max_prompt_chars: 8000 claude_max_prompt_chars: 8000 -logging: - audit: +policy: + privacy: redact_secrets_like: - "sk-[a-zA-Z0-9]{20,}" # OpenAI-style - "sk-ant-[a-zA-Z0-9_\\-]{20,}" # Anthropic (Claude) diff --git a/.claude/skills/CyClaw-Sandbox/verify.sh b/.claude/skills/CyClaw-Sandbox/verify.sh index ecd278bd0..a981c450c 100755 --- a/.claude/skills/CyClaw-Sandbox/verify.sh +++ b/.claude/skills/CyClaw-Sandbox/verify.sh @@ -1,8 +1,8 @@ #!/usr/bin/env bash # CyClaw sandbox runtime verification — proves the entire main branch runs -# under a clean Python 3.12 runtime. Nine labeled stages, fail-fast on required +# under a clean Python 3.12 runtime. Seven labeled stages, fail-fast on required # ones. Labels are historical and NOT sequential in source order (1, 2, 3, 5, -# 8, 4, 7, 9, 6) — do not renumber; CI logs cite them. See SKILL.md +# 4, 7, 6) — do not renumber; CI logs cite them. See SKILL.md # "Operator map" for what each stage proves vs does not prove. # Run from the repo root: bash .claude/skills/CyClaw-Sandbox/verify.sh set -uo pipefail diff --git a/.claude/skills/cyclaw-gotchas/SKILL.md b/.claude/skills/cyclaw-gotchas/SKILL.md index f74601af1..9f4af9478 100644 --- a/.claude/skills/cyclaw-gotchas/SKILL.md +++ b/.claude/skills/cyclaw-gotchas/SKILL.md @@ -45,9 +45,12 @@ session 2026-09-06): 1. `python3.12 -m venv /root/.venv-cyclaw-312` -- outside the repo, so no `.gitignore` entry and no chance of `ruff`/pytest walking site-packages. - Bare `python3` is 3.11 and always will be on this image; do not - `update-alternatives` it (the session-runtime hooks resolve through it). -2. Tries `torch==2.13.0+cpu` from the CPU index with `--retries 1`. On this + Bare `python3` is never 3.12 on this image (3.11 on the 2026-09 images, + 3.13 on 2026-10-13's); do not `update-alternatives` it (the + session-runtime hooks resolve through it). +2. Tries `torch==2.13.0+cpu` from the CPU index with one retry and a short + timeout (`uv pip install` with `UV_HTTP_RETRIES=1 UV_HTTP_TIMEOUT=15` when + `uv` is on PATH, as it is here; `pip --retries 1` otherwise). On this proxy that fails in seconds, not minutes. 3. Falls back to plain `torch==2.13.0` from PyPI **with** its dependency tree. `--no-deps` was tried and is a dead end: the Linux wheel dlopens @@ -62,8 +65,7 @@ session 2026-09-06): itself (`must be pinned upfront with ==, but found: torch`, pulled in by sentence-transformers). pip accepts the installed torch, which is why CI and CLAUDE.md §8 need no `--no-deps` on the locks. This is unrelated to the - torch `--no-deps` dead end in step 3. The `--ignore-installed` gotcha - below still describes the macOS recipe in CLAUDE.md §8. + torch `--no-deps` dead end in step 3. Re-running is idempotent: it exits 0 immediately once `import torch, chromadb, langgraph, pytest` succeeds. @@ -99,10 +101,16 @@ different key. ## Test ```bash -bash .claude/skills/cyclaw-gotchas/driver.sh test # whole suite: 5263 passed, 82 skipped, ~4m40s here bash .claude/skills/cyclaw-gotchas/driver.sh test tests/test_sync_runner.py tests/test_telemetry_kill.py +bash .claude/skills/cyclaw-gotchas/driver.sh test # whole suite (5263 passed, 82 skipped, ~4m40s on 2026-09-06) ``` +CLAUDE.md §5's verification policy (owner decision 2026-10-08) applies: run +the changed code, lint, let GitHub Actions run the suites, and reach for +`driver.sh test ` only when nothing else exercises the path. The +no-argument whole-suite form is there for the owner, for CI parity checks +he asks for, and for `/tests-refactor`; it is not a routine step. + The suite is fully green in this sandbox even though the index cannot be built, because `tests/conftest.py` mocks the embedding model. Only `tests/ci_rag_smoke.py`, `.claude/skills/CyClaw-Sandbox/smoke.sh`, `/run`, @@ -122,8 +130,8 @@ read the exit code (`echo "pytest exit=${PIPESTATUS[0]}"` when piped). bash .claude/skills/cyclaw-gotchas/driver.sh checks ``` -Runs the five stdlib guards (`invariant-guard`, `doc-sync`, `config-guard`, -`dep-guard`, verify-deps `check_env_drift.py --strict`) and prints one line +Runs the six stdlib guards (`invariant-guard`, `doc-sync`, `config-guard`, +`dep-guard`, `dotenv-guard`, verify-deps `check_env_drift.py --strict`) and prints one line each. `config-guard` reports **0 warnings** on a clean tree since C9 was flipped (commit 993f439) from "warn if hybrid+enabled" to "warn if the committed config diverges from the documented shipped provider posture" — @@ -145,20 +153,18 @@ expire it. ### Sandbox and install -- **Nothing is preinstalled, in any interpreter.** CLAUDE.md §4 once said the - deps live in 3.11's `dist-packages`; on 2026-09-06 `import torch` failed for - 3.10, 3.11, 3.12 and 3.13 alike. Treat the venv step as mandatory every - session; the container is rebuilt from a generic image, not from this repo. -- **`--ignore-installed PyYAML` reinstalls everything, not just PyYAML.** - `--ignore-installed` is a bare pip flag; `PyYAML` after it is one more - requirement. Every install line in this repo that carries it therefore - re-resolves and reinstalls torch too, and the only thing holding the - version is the constraints file. The macOS recipe used to strip the torch - line from its constraints copy, so that reinstall floated torch to PyPI's - newest: this driver installed `torch==2.13.0` and ended with `2.14.0+cu130` - (2026-09-06; `macos/install-cyclaw.sh`'s Darwin branch had the same shape). - Rule: the constraints copy keeps `torch==X.Y.Z` with only `+cpu` removed; - `tests/test_macos_scripts.py` pins it. +- **Nothing is preinstalled, in any interpreter.** On 2026-09-06 `import + torch` failed for every Python on the image. Treat the venv step as + mandatory every session (the cloud SessionStart hook starts it in the + background); the container is rebuilt from a generic image, not from this + repo. +- ~~**`--ignore-installed PyYAML` reinstalls everything, not just PyYAML**~~ + — expired 2026-10-13: every documented install now uses the hashed + `locks/` with `--require-hashes` (#1598), no install line carries + `--ignore-installed`, and `tests/test_macos_scripts.py` asserts the old + constraints rewrite is gone. Kept as a one-line tombstone because the + lesson generalises: a bare pip flag followed by a package name is two + arguments, and a resolver re-run can float torch to a CUDA build. - **`pkill -f "python gate.py"` kills your own shell, and `pgrep -f gate.py` reports it as still running.** Both match the `bash -c` wrapper that is executing the command (exit 144 from `pkill`; a phantom "server still @@ -176,10 +182,6 @@ expire it. ### Runtime behaviour that looks like a bug - -- **Codex's fourth P2 on #1247** (a non-numeric port in a harness base URL - made `parsed.port` raise outside the guarded block) is moot since PR #1367 - removed the harness server module; do not report it as new. - **`/health` `degraded`, `TELEMETRY KILL` on stdout, `503 INDEX_NOT_FOUND`, `needs_confirm: true` on `/query`** are all normal states, not errors (CLAUDE.md §4 "Environment & install" and `.claude/commands/run.md`). @@ -187,9 +189,10 @@ expire it. ### Tests and checks - **A green run can print no summary** (see Test above). Read the exit code. -- **Run the fast, scoped tests first, then the whole suite once.** The suite - takes ~4m40s here; `-x` on the first pass finds the first failure without - waiting for all of it. +- **Scope any local test run to the one file that covers the change** and + let CI run the suite (CLAUDE.md §5). When the owner does ask for the whole + suite, `-x` on the first pass finds the first failure without waiting + ~4m40s for all of it. - **Ruff, invariant-guard, and the doc-sync checker are cheap; run them before every push.** `ruff check --select E,F,I,B,C4,UP,S ` is seconds; the F/B/S subset is what blocks merge (`lint.yml`). @@ -239,8 +242,7 @@ expire it. at a higher level" was false and was caught. Only write verifiable claims. - **Reply on the thread, then resolve it, on PRs you opened.** The reply needs the Claude Code attribution footer. Leave a thread open only while a - question to the reviewer is pending. Check merged PRs for threads left open - (see #1247 above). + question to the reviewer is pending. Check merged PRs for threads left open. - **Bot review triggers:** the owner comments `@codex review` to start a Codex pass; `@codex apply fixes` hands the fix to Codex; `@copilot resolve the merge conflicts ...` with explicit write permission is how conflicts on @@ -265,9 +267,11 @@ expire it. cost seconds each. - **Docstrings are docs.** When a README derives from a package docstring, fix both in the same commit or the drift regrows (`agentic/harness_optimizer`). -- **`doc_sync.py` covers structured facts only** (skills list, entry points, - config numbers, routes, node count, hook claims). README prose, the threat - model stance, and skill bodies are manual-pass territory. +- **`doc_sync.py` covers structured facts only** (D1–D13: skills list, entry + points, config numbers, routes, node count, hook claims, the M5 doctrine + citations, and README path/link/`python -m`/record-table references). + README prose, the threat model stance, and skill bodies are manual-pass + territory. ### verify-deps in this sandbox @@ -278,9 +282,9 @@ expire it. - **The install dry-runs cannot pass here** (PyTorch index denied); report them "unverified in this sandbox" and say why. `docker compose config --quiet` does work without a daemon. -- **Every advisory hit on the current pins (5 chromadb, 1 nltk) is a dated - acceptance in `SECURITY.md`** with no upstream fix version. Re-reporting them - is noise; a *new* id is the signal. +- **Every advisory hit on the current pins (4 chromadb, 1 nltk on + 2026-10-13) is a dated acceptance in `SECURITY.md`** with no upstream fix + version. Re-reporting them is noise; a *new* id is the signal. ## Troubleshooting @@ -289,7 +293,7 @@ expire it. | `No matching distribution found for torch==2.13.0+cpu` after five proxy retries | CPU index denied by the egress proxy | `driver.sh venv` falls back to PyPI torch automatically | | `OSError: libcudart.so.13: cannot open shared object file` on `import torch` | plain torch installed with `--no-deps` | reinstall without `--no-deps`; the CUDA libs are required at import on Linux | | pytest prints dots and `[100%]` but no `N passed` line | `-q` doubled with `addopts` → `-qq` | use `driver.sh test` or `-o addopts=""`; read the exit code | -| `ModuleNotFoundError: pytest` / `torch` under `python3` | `python3` is 3.11 with nothing installed | use `/root/.venv-cyclaw-312/bin/python -m pytest` | +| `ModuleNotFoundError: pytest` / `torch` under `python3` | the system `python3` (not 3.12) has nothing installed | use `/root/.venv-cyclaw-312/bin/python -m pytest` | | shell exits 144 after `pkill -f "python gate.py"`, or `pgrep -f gate.py` says a stopped server is running | both patterns match the invoking `bash -c` shell | `driver.sh stop` (pidfile); check liveness with `curl` on the port | | `POST /query` → `503 INDEX_NOT_FOUND` after a successful serve | no index; model host denied | expected here; test retrieval via the unit suite, not the live route | | `uv pip install --dry-run --system ...` → "externally managed" | system interpreter refuses | run inside a venv, or report unverified | @@ -306,9 +310,6 @@ expire it. - CLAUDE.md §3's six invariants and §7's escalation tiers still govern. A gotcha here is never a licence to skip a guard; it is a reason you will not be surprised by one. -- Never "fix" a `degraded` health or a - `503 INDEX_NOT_FOUND` by changing code. They are controls and fail-soft - states, not defects. ## Maintaining this file diff --git a/.claude/skills/cyclaw-gotchas/driver.sh b/.claude/skills/cyclaw-gotchas/driver.sh index f544e53af..269a3fc6f 100755 --- a/.claude/skills/cyclaw-gotchas/driver.sh +++ b/.claude/skills/cyclaw-gotchas/driver.sh @@ -16,7 +16,7 @@ # test [...] pytest through the venv with pyproject's addopts cleared so the # "N passed" summary prints; exit code propagates. Default: tests/. # checks the stdlib guards: invariant-guard, doc-sync, config-guard, -# dep-guard, verify-deps env drift. Exit 1 if any fails. +# dep-guard, dotenv-guard, verify-deps env drift. Exit 1 if any fails. set -uo pipefail REPO_ROOT="$(cd "$(dirname "${BASH_SOURCE[0]}")/../../.." && pwd)" diff --git a/.claude/skills/cyclaw-privacy/SKILL.md b/.claude/skills/cyclaw-privacy/SKILL.md index 14e254294..5f3873efb 100644 --- a/.claude/skills/cyclaw-privacy/SKILL.md +++ b/.claude/skills/cyclaw-privacy/SKILL.md @@ -7,7 +7,7 @@ description: >- disable-model-invocation: true --- -# CyClaw-Advisor ("Legal") +# cyclaw-privacy ("Legal") You are **Legal**, the in-house compliance assistant for privacy regulation, DPA reviews, data subject requests (DSR), breach analysis, and regulatory @@ -52,11 +52,16 @@ reason abstractly about "an AI system." Current posture (verify against `config.yaml` before citing a number, since these are tunables, not constants): -- **Audit log stores hashes, never raw queries.** `utils/logger.py` SHA-256- - hashes any field named `query`; the audit stream (`logs/audit.jsonl`) - never persists plaintext query text. A change that adds a new audited - field carrying free-text user input under a different key would - re-introduce the exact exposure the hashing exists to prevent — flag it. +- **Audit log stores keyed fingerprints, not raw queries — by default.** + `utils/logger.py::query_fingerprint` is a keyed HMAC-SHA256 over the + query (key from `CYCLAW_QUERY_FINGERPRINT_KEY` or the file + `data/privacy/query-hmac.key`, itself a data element to inventory), so the + audit stream (`logs/audit.jsonl`) cannot be reversed into a query by + dictionary attack. This holds while `logging.audit.include_query_hash` is + `true` (shipped); `false` stores the PII-redacted raw query in the audit + file and in Numbat's `content_preview` — a privacy regression to flag in + any review that touches it. A change that adds a new audited field carrying + free-text user input under a different key re-introduces the same exposure. - **Redaction is config-driven, not hardcoded.** `config.yaml`'s `policy.privacy` block controls `redact_emails`, `redact_ips`, and `redact_secrets_like` (the secret-pattern list `utils.logger @@ -70,7 +75,10 @@ constants): - **Guardrail metrics are a separate stream** (`logs/guardrails.jsonl`, hashes only) from the main audit log — a data-mapping exercise (e.g. for a DPA Schedule or a DSR data-inventory response) needs to account for both - streams, not just `logs/audit.jsonl`. + streams, not just `logs/audit.jsonl`, plus the app log `logs/cyclaw.log` + (no query text, but `logging.third_party_level` below `INFO` writes live + API keys from httpcore DEBUG output — treat such a file as secret) and the + eval ledger `logs/evals/eval_runs.jsonl` when `judge_eval` has run. - **Per-user auth (Stage 1+2, `docs/AUTHENTICATION_DESIGN.md`) is a new personal-data surface.** `utils/authn_store.py`'s `users`/`sessions`/ `device_tokens` tables (default `data/auth/cyclaw_auth.db`, or @@ -92,15 +100,15 @@ constants): stage query hashes + redacted answer summaries into `data/memory/cyclaw_memory.db` (raw query only if `episodes.store_raw_query: true` — default false; uses the same - `hash_query` / `redact_sensitive` path as audit); (2) **facts** are + `query_fingerprint` / `redact_sensitive` path as audit); (2) **facts** are human-approved via propose/apply (API key + non-empty reason + apply-path injection scan — parallel to soul I5, not soul itself); (3) optional FTS fusion can inject approved facts into retrieval. Consolidation and auto-fact extraction from RAG/LLM output are **not** implemented — flag any PR that would auto-write facts from untrusted corpus/LLM text as a memory-poisoning / zombie-agent risk. Data-mapping exercises that enable - memory must list this DB alongside auth, personality, and the two log - streams. + memory must list this DB alongside auth, personality, and the log + streams above. - **External fallback (Grok/Claude) is triple-gated** (invariant I3) and, per `policy.fallback.send_local_context_to_grok`/`_claude` (default `false`), does not forward retrieved local context off-box unless diff --git a/.claude/skills/cyclaw-privacy/bootstrap.sh b/.claude/skills/cyclaw-privacy/bootstrap.sh index 1bf6a6192..26a9186ba 100644 --- a/.claude/skills/cyclaw-privacy/bootstrap.sh +++ b/.claude/skills/cyclaw-privacy/bootstrap.sh @@ -1,21 +1,23 @@ #!/usr/bin/env bash # bootstrap.sh — cyclaw-privacy skill harness. -# Sets up the compliance advisor environment, verifies latest main branch state, -# and prepares reference files for privacy/DPA/DSR/breach workflows. +# Refreshes origin/main and prints the SHA a review should cite, then runs +# verify.sh so the posture SKILL.md describes is confirmed before advising. +set -uo pipefail -set -euo pipefail +repo_root="$(cd "$(dirname "${BASH_SOURCE[0]}")/../../.." && pwd)" +cd "$repo_root" || exit 3 hr() { printf '%s\n' "------------------------------------------------------------"; } echo "Bootstrapping cyclaw-privacy skill..." hr - -git fetch origin main --quiet || echo "WARN: fetch may have issues (offline?)" - -echo "Current main SHA: $(git rev-parse origin/main 2>/dev/null || echo 'unknown')" -echo "Compliance references ready." - -echo "Skill loaded: Legal compliance assistant mode active." -echo "Use for DPA reviews, DSR handling, breach triage, regulatory monitoring." +git fetch origin main --quiet || echo "WARN: fetch failed (offline?); citing the local ref" +echo "origin/main: $(git rev-parse --short origin/main 2>/dev/null || echo 'unknown')" +echo "HEAD: $(git rev-parse --short HEAD 2>/dev/null || echo 'unknown')" +hr +bash "$repo_root/.claude/skills/cyclaw-privacy/verify.sh" +rc=$? hr -echo "Bootstrap complete. Invoke SKILL.md instructions for queries." \ No newline at end of file +echo "Legal compliance assistant mode: DPA reviews, DSR handling, breach triage, regulatory monitoring." +echo "Cite config.yaml at the SHA above; the numbers are tunables, not constants." +exit "$rc" diff --git a/.claude/skills/cyclaw-privacy/verify.sh b/.claude/skills/cyclaw-privacy/verify.sh index 27cef9a3d..a0f04982c 100755 --- a/.claude/skills/cyclaw-privacy/verify.sh +++ b/.claude/skills/cyclaw-privacy/verify.sh @@ -1,22 +1,53 @@ #!/usr/bin/env bash # verify.sh — cyclaw-privacy verification. -# Checks consistency of dep files, security posture, and compliance readiness. +# Checks that the shipped config.yaml still carries the privacy posture +# SKILL.md cites (fingerprinted audit, redaction on, no off-box context by +# default). Exits 0 when healthy, 2 on a posture drift, 3 when it cannot +# check (no PyYAML — a fresh pre-install container — is a clean skip). +set -uo pipefail -set -euo pipefail +repo_root="$(cd "$(dirname "${BASH_SOURCE[0]}")/../../.." && pwd)" +cd "$repo_root" || exit 3 -echo "Verifying cyclaw-privacy skill..." - -# Check key files -for f in pyproject.toml constraints.txt requirements.txt Dockerfile; do - if [ -f "$f" ]; then - echo "✓ $f present" - else - echo "✗ $f missing" - fi -done - -echo "Dep files consistent (from previous analysis)." -echo "Chroma CVE risk note present in pyproject.toml." - -echo "Compliance skill ready for DPA/DSR/breach workflows." -echo "Verification complete." \ No newline at end of file +echo "Verifying cyclaw-privacy posture against config.yaml..." +python3 - <<'PY' +import sys +try: + import yaml +except ImportError: + print("SKIP: PyYAML not importable (fresh container); nothing checked") + sys.exit(0) +cfg = yaml.safe_load(open("config.yaml", encoding="utf-8")) or {} +def get(path, default=None): + node = cfg + for key in path.split("."): + if not isinstance(node, dict) or key not in node: + return default + node = node[key] + return node +checks = [ + ("logging.audit.include_query_hash", get("logging.audit.include_query_hash", True) is True, + "audit stream stores keyed HMAC fingerprints, never raw query text"), + ("policy.privacy.redact_emails", get("policy.privacy.redact_emails") is True, "email redaction on"), + ("policy.privacy.redact_ips", get("policy.privacy.redact_ips") is True, "IP redaction on"), + ("policy.privacy.redact_secrets_like has sk-ant", any("sk-ant" in str(p) for p in get("policy.privacy.redact_secrets_like", []) or []), + "Anthropic key shape redacted"), + ("policy.fallback.send_local_context_to_grok", get("policy.fallback.send_local_context_to_grok") is False, + "no retrieved context off-box to Grok by default"), + ("policy.fallback.send_local_context_to_claude", get("policy.fallback.send_local_context_to_claude") is False, + "no retrieved context off-box to Claude by default"), + ("memory.enabled", get("memory.enabled") is not True, "memory subsystem ships off"), + ("auth.enabled", get("auth.enabled") is not True, "per-user auth ships off"), + ("logging.third_party_level", str(get("logging.third_party_level", "INFO")).upper() != "DEBUG", + "third-party log level above DEBUG (httpcore DEBUG writes Authorization headers)"), +] +failed = 0 +for name, ok, why in checks: + print(f" {'ok ' if ok else 'FAIL'} {name} -- {why}") + failed += 0 if ok else 1 +print(f"{failed} posture drift(s)") +sys.exit(2 if failed else 0) +PY +rc=$? +[ "$rc" -eq 0 ] && echo "cyclaw-privacy verify: OK" +exit "$rc" diff --git a/.claude/skills/fable-protocol/SKILL.md b/.claude/skills/fable-protocol/SKILL.md index d3972a8e7..a6045b481 100644 --- a/.claude/skills/fable-protocol/SKILL.md +++ b/.claude/skills/fable-protocol/SKILL.md @@ -112,8 +112,9 @@ Version history: `references/knowledge-base.md`, closing section. 5.3 Trust boundaries first. Identify where untrusted data crosses into trusted execution before commenting on anything else. 5.4 Findings-before-writes. Report a FINDINGS SUMMARY before any mutation. Never - touch declared-immutable assets (soul.md, gate.py equivalents) even on a casual - ask — confirm intent explicitly. + touch governed assets (soul.md needs a human `reason` through + `PersonalityManager`, I5; gate.py auth and graph edges are High tier) on a + casual ask — confirm intent explicitly. 5.5 SONNET-5 CYBER SAFEGUARDS & MODEL ROUTING (CyClaw-specific). Sonnet 5 is the first Sonnet-tier model with real-time cyber safeguards. Via API a refusal returns HTTP 200 with stop_reason:"refusal" — a terminal state, not an error; @@ -140,13 +141,9 @@ Version history: `references/knowledge-base.md`, closing section. of his specific prompts. This applies doubly now — Opus 4.8's aggregate figures were the last verified data point this protocol had; Opus 5's are unmeasured here. - [2026-07-27: model IDs bumped Opus 4.8 → Opus 5 (Opus line then; Opus 5.5 as of 2026-10; per this - session's environment). Benchmark claims above (CyberGym, OSS-Fuzz, - over-refusal rate) are Sonnet-5-specific and unchanged; they were never - Opus-5-specific to begin with — carry them as a prior, not a measurement.] - (This is the same split fable-5.1-cc's old §7 restated for "his work" - specifically — one rule, not two; §8.9 below carries only what that section - added beyond this.) + [Model IDs: Opus 4.8 → Opus 5 (2026-07-27) → Opus 5.5 (2026-10). The + benchmark claims above are Sonnet-5-specific and were never measured on + any Opus — carry them as a prior, not a measurement.] ## 6. ANTI-SYCOPHANCY @@ -235,10 +232,10 @@ and doc as something another model may also be touching in parallel. Named, documented, self-acknowledged: he builds thoroughly and iterates extensively, and there is a persistent gap between **building** and -**shipping/publishing**. CyClaw's history is the evidence: forty-plus audit and -verification reports in `docs/audits/`, a `docs/zIdeas/` directory, eleven -version rows in the changelog, and a `remaining_work` doc that keeps being -restamped against the newest main. +**shipping/publishing**. CyClaw's history is the evidence: dozens of audit and +verification reports in `docs/audits/`, a `docs/zIdeas/` directory, twenty-odd +version rows in the changelog, and a `docs/plans/remaining_work.md` that keeps +being restamped against the newest main. Operating rule for you: when he proposes new architecture, test (Socratically first, then directly) whether it advances shipping or defers it. Do not enable @@ -310,10 +307,8 @@ Knowing where yours are gets you most of the way here. Calibrated to a single user. Don't generic-ify it — its value is its specificity. -**Consolidation note (2026-09-06):** this skill previously shipped as two files -— `fable-protocol` (discipline, §1-7) and a companion `fable-5.1-cc` -(knowledge, now §8 onward), merged once the split cost more than it bought. -Full history in `references/knowledge-base.md`'s closing section. +Version and consolidation history: `references/knowledge-base.md`, closing +section. **Keeping this file alive:** update it via a PR like any other doc; never rewrite history in it. Add dated bullets under the relevant section, mark diff --git a/.claude/skills/fable-protocol/references/knowledge-base.md b/.claude/skills/fable-protocol/references/knowledge-base.md index 1e9dd63bd..ba214c1c7 100644 --- a/.claude/skills/fable-protocol/references/knowledge-base.md +++ b/.claude/skills/fable-protocol/references/knowledge-base.md @@ -29,32 +29,22 @@ Everything about how to work in it is in `CLAUDE.md` (the operating manual), Read CLAUDE.md fully before editing; this section is the part to know *cold* without opening it. -**The six invariants** (wiring, not prompts; `python3 -.claude/skills/invariant-guard/check_invariants.py` asserts them): -1. I1 RAG-first: `retrieve` is the unconditional entry node. -2. I2 Topology = policy: routing is graph edges via four routers only - (`score_router`, `guardrail_router`, `user_gate_router`, - `pre_action_hook_router`). -3. I3 Triple gate: Grok/Claude need `mode=="hybrid"` AND `.enabled` - AND per-request `user_confirmed_online`. Both providers are `enabled: true` - since 2026-08-07 (armed), still gated. -4. I4 Audit convergence: all eleven upstream paths reach `audit_logger` before END. -5. I5 Soul governance: `soul.md` mutation needs a human `reason` string, atomic - write via `PersonalityManager`. Governed, not forbidden. -6. I6 Module isolation: the core six (`gate.py`, `gate_ops.py`, `gate_auth.py`, - `gate_memory.py`, `graph.py`, `mcp_hybrid_server.py`) never import - `agentic`/`sync`/`guardrails`/`harness`/`telegram`/`opentweet`, and vice versa. - -**Load-bearing numbers** (source: `config.yaml`, `pyproject.toml`; never invent): -`min_score 0.028` is RRF-scale, not cosine, do not "fix" it upward; `rrf_k 60`; -`graph_timeout_sec 780` > `local_llm.timeout_sec 720`; `max_tokens 4096`; -`max_context_tokens 8000`; `soul_max_chars 8000`; chunk `512`/overlap `50`; rate -limit `60`/`60s` per IP; 40 `banned_patterns` (phrases are contractual, count is -documentary); coverage `fail_under 80`; Python `>=3.12,<3.13`. +**The six invariants and the load-bearing numbers** live in `CLAUDE.md` §3 +and §2 ("Load-bearing numbers"), which `doc-sync` D3/D8 keep honest; this +file stopped carrying its own copy on 2026-10-13 after the copy drifted +(`max_context_tokens`, chunk sizes and the `num_ctx` floor were all a +generation stale). The two things to know cold without opening it: routing +is graph edges via four routers, never an LLM decision; and `min_score 0.028` +is RRF-scale (hybrid ceiling 2/60), do not "fix" it upward — the vault-hit +gate is `min_semantic_score`. Both providers are `enabled: true` since +2026-08-07 (armed), still triple-gated per request. Soul mutation is +governed (a human `reason`), not forbidden. **The footgun to check first on any "CyClaw hangs" report** (carried forward, -still true): Ollama `num_ctx` must clear `max_context_tokens + max_tokens + ~1500` -(floor 13,596; `macos/ollama-mlx.env` sets 16384) or RAG stalls at 0%. +still true): Ollama `num_ctx` must clear `max_context_tokens + max_tokens + +~1500` (`CLAUDE.md` §2 states the current floor; `macos/ollama-mlx.env` sets +`OLLAMA_CONTEXT_LENGTH`, and config-guard C12 fails when the env file is +below the floor) or RAG stalls at 0%. **Integrity:** SHA-256 `soul.md` drift detection + SQLite shadow DB — this part is shipped. **Proposed, NOT found in the current codebase** (verified by a @@ -68,67 +58,36 @@ note claimed "48/48 tests at design time," which this merge could not confirm against `graph.py` or `docs/changelog.txt`). Treat both as open threads to ask about, not shipped features to reference as fact. -**Traps a capable-but-new model falls into here** (full list CLAUDE.md §4): -- Fresh sandbox: bare `python3` is 3.11, and as of 2026-09-06 NO interpreter on - the default cloud sandbox image ships CyClaw's deps preinstalled — build - `/root/.venv-cyclaw-312` with `python3.12 -m venv`, torch `2.13.0+cpu` first - (falling back to plain PyPI torch when the egress proxy denies - `download.pytorch.org`), then requirements with `--ignore-installed PyYAML`. - macOS needs plain torch (no `+cpu`). The `cyclaw-gotchas` skill's `driver.sh` - does this end to end — load it for any sandbox setup/test/PR-driving task. -- `import gate` at test top level boots the whole app. Patch or subprocess. -- `status: degraded` without Ollama and `TELEMETRY KILL` at startup are normal. -- Startup does not require `GROK_API_KEY`. Tests need only `GROK_API_KEY=dummy`. -- The `_TELEMETRY_KILL` binding in `gate.py` must stay above heavy imports; - invariant-guard G1 finds it by AST. `HF_HUB_OFFLINE` is excluded from the kill - map on purpose. `ORT_TELEMETRY_OPT_OUT` is inert; `ORT_DISABLE_TELEMETRY=1` - plus `disable_telemetry_events()` are the real controls. -- BM25 stays JSON (pickle = RCE). Audit log stores SHA-256 of queries, never text. -- No `print` in library code, no bare `except`, no `shell=True`, no TODO/FIXME - comments, typed errors rooted at `RAGError`, exit codes are an API. -- Never docstrings as multi-line comments except at file top or function top. -- New POST routes must be added to `test_terminal_contract`'s `_POST_PATHS`. -- mypy is not a CI gate; ruff `--select F,B,S` is. Bare `pytest` runs no coverage. -- `pydantic`/`pydantic-core` lock-step; numpy `<2`; chromadb CVE is risk-accepted - (embedded `PersistentClient` only). Do not file a "fix". -- Session-process traps (sandbox setup, PR-driving, review-bot handling, - scheduled check-ins) are a separate, larger list — see the `cyclaw-gotchas` - skill rather than duplicating it here. - -**Git workflow he enforces:** driver-matched branch prefixes (`claude/`, `codex/`, -`grok/`, `kimi/`, `CyClaw/`, `agent/`), never push to `main`, never force-push -without his explicit sign-off, draft PRs only, one concern each, PR body follows -`.github/PULL_REQUEST_TEMPLATE.md` (title form `[prefix] - sentence`; the -squashed merge commit carries that title, which is why `git log` shows -`[security] - ...` rather than `feat:` despite CLAUDE.md §5 asking for -conventional commits on the branch itself). Subscribe to every PR you open and -drive it to green; no polling loops beside a live subscription. Identity for -commits: the session runtime's (cloud: `Claude `, which -the runtime signs; local: the owner's own git identity), never `CyClaw Agent`, -which belongs to CyClaw's own agentic loop (owner decision 2026-09-26). - -**Recent trajectory (Aug-Sep 2026, from `git log` and changelog)** so you know -where the frontier is: doc-sync + verify-deps skill hardening and a full -38-README reconciliation pass (#1319); the `cyclaw-gotchas` session-lessons -skill (#1320); a fifteenth threat-model amendment widening scope from -single-operator to trusted-operator while keeping single-tenant (#1319); -hash-pinned telemetry kill maps at boot (#1268); harness Origin check with port -and scheme (#1267); Grok proposer spend ledgering (#1266); nltk 3.10.3 pin and -256-char Porter token cap for the PorterStemmer DoS cluster (#1258); Numbat -NDJSON mainline plane (every audit record projected, fail-soft); default-off -Unslop slop-detection probe for the agentic loop; per-user auth with RBAC, -sessions, CSRF, device tokens, TLS via `cyclaw-gen-cert`; default-off memory -subsystem (facts + episodes, SQLite FTS5, propose/apply governance); out-of-band -Telegram and OpenTweet channels (disabled by default); `real_repo_loop` plan → -patch → verify → human decides → commit. The Python coding-harness console was -removed on 2026-09-11; that role moved to the sibling CG-agent-harness repo. Local model moved LM Studio → Ollama, `qwen3.6:27b` → -`qwen3.8:27b-mlx` on 2026-08-15. Skill-catalog context-tax cleanup landed -2026-09-11 (issue #1351): `general-purpose` deleted, `python-coding-agent` and -this knowledge-base section both moved their bulk into `references/`, -`cyclaw-advisor` renamed `cyclaw-privacy`, `CyClaw-Sandbox`'s frontmatter/folder -mismatch fixed, several chore/loop skills gained `disable-model-invocation`, -and the fat `CyClaw-Optimize`/`*-refactor` command-wrapper duplicates were -trimmed to thin pointers. +**Traps a capable-but-new model falls into here** — the full, maintained list +is `CLAUDE.md` §4 (codebase traps) and the `cyclaw-gotchas` skill (session +traps: sandbox setup, PR-driving, review-bot handling, scheduled check-ins); +read those rather than a copy here. The three that bite first: the fresh +cloud sandbox has no CyClaw deps in any interpreter and its proxy denies the +PyTorch CPU index and `huggingface.co` (`driver.sh venv` handles the former; +the index cannot be built there); `import gate` at a test module's top level +boots the whole app; and `status: degraded`, `TELEMETRY KILL` on startup and +`503 INDEX_NOT_FOUND` are normal states, not defects. + +**Git workflow he enforces** is `CLAUDE.md` §5 ("Commits, branches, PRs") and +§10: driver-matched branch prefixes, `[prefix] - sentence` subjects on every +commit (the tracked `commit-msg` hook refuses anything else), draft PRs with +the template body, never push to `main`, never force-push without his +explicit sign-off, commit as the session runtime's identity (never `CyClaw +Agent`, owner decision 2026-09-26), subscribe to every PR you open and drive +it to green with no polling beside a live subscription. + +**Where the frontier is** (verify against `git log` and +`docs/changelog.txt` — this paragraph is a pointer, not a record): the live +agentic path is `agentic/real_repo_loop.py` (plan → patch → verify → human +decides → commit); the Python coding-harness console was removed 2026-09-11 +and that role moved to the sibling CG-agent-harness repo; per-user auth +(RBAC, sessions, CSRF, device tokens, TLS), the default-off memory subsystem, +the default-off Telegram/OpenTweet channels, and the Numbat NDJSON plane all +shipped Aug–Sep 2026; the local model moved LM Studio → Ollama, +`qwen3.6:27b` → `qwen3.8:27b-mlx` on 2026-08-15; installs moved to hashed +`locks/` in 2026-10 (#1598). The 2026-09-11 skill-catalog cleanup (issue +#1351) is why this file exists as a reference and why `cyclaw-advisor` is +now `cyclaw-privacy`. **Open threads he keeps returning to** (verify status before acting — see the Integrity note above for two of these): 3-layer semantic drift detection for @@ -155,9 +114,10 @@ Integrity note above for two of these): 3-layer semantic drift detection for - A Windows operator machine also exists (PowerShell launchers, `%USERPROFILE%\ .CyClaw`, and the `gh` shim trap: bare `gh` is a py3dot12 shim, real CLI at `"/c/Program Files/GitHub CLI/gh.exe"`). -- Claude Code cloud sandbox (`sandbox-ccr-default`, Ubuntu 24.04): Python 3.10 - through 3.13 all on disk, none preinstalled with CyClaw's deps as of - 2026-09-06 (see the trap list above and `cyclaw-gotchas`). +- Claude Code cloud sandbox (`sandbox-ccr-default`, Ubuntu 24.04): several + Python minors on disk (3.11–3.13 on 2026-10-13; which one bare `python3` + resolves to has changed between images), none preinstalled with CyClaw's + deps (see `cyclaw-gotchas`; always use the venv's interpreter). - Postgres is optional for soul, auth, and rate-limit stores; SQLite is default. ### 8.7 Decisions already made (do not re-litigate; cite, then move) @@ -166,10 +126,10 @@ Integrity note above for two of these): 3-layer semantic drift detection for |---|---|---| | Autonomous skill-write loops | rejected | `SKILL.md` §8.3 | | DeepAgents subgraph | retired 2026-07-31, code kept | CLAUDE.md §2 key modules | -| chromadb CVE-2026-45829 | risk-accepted, embedded only | PROJECT_RULES.md | +| chromadb CVE-2026-45829 | risk-accepted, embedded only | SECURITY.md (PROJECT_RULES.md restates it) | | `min_score` 0.028 | intentional RRF scale | CLAUDE.md §2, §4 | | `HF_HUB_OFFLINE` out of kill map | intentional | CLAUDE.md §4 | -| `/index/build` not API-key gated | intentional (first-run bricking) | CLAUDE.md §2 route table | +| `/index/build` not API-key gated while `CYCLAW_API_KEY` is unset | intentional (first-run bricking); key-gated once the key is set | CLAUDE.md §2 route table | | Grok and Claude `enabled: true` | armed 2026-08-07, triple gate unchanged | THREAT_MODEL 8th amendment | | `api_key_optional` bypass | loopback-peer only, never Host header | CLAUDE.md §2 | | Test mock `min_score` 0.75 vs prod 0.028 | both load-bearing, do not unify | CLAUDE.md §4 | @@ -206,69 +166,27 @@ runs on Fable again, treat this file as a refresher, not a crutch — Fable can hold CLAUDE.md, the route table, and a diff in one pass in a way most models running this skill cannot; `SKILL.md` §9 is written for the latter case. -## Consolidation history (full detail) - -This skill previously shipped as two files — `fable-protocol` (the discipline -layer, `SKILL.md` §1-7, plus its checklist and meta sections) and a companion -`fable-5.1-cc` (the knowledge layer, now this file's §8-9, folded into -`SKILL.md`'s META section originally). They were merged on the owner's -explicit request once both had shipped long enough to show the split cost -more than it bought: two files to keep in sync, and genuine drift between -them where they overlapped — the two files' "CyClaw knows cold" sections -disagreed on whether the LLM Council subgraph and 3-layer semantic drift -detection were shipped-and-tested or still open proposals (§8.4's Integrity -paragraph resolves this the honest way: neither was found in the actual -codebase at merge time). Every real session needed both loaded anyway. -`fable-5.1-cc`'s SKILL.md and command wrapper are deleted; nothing else in -this repo should reference them going forward except as historical record -(dated audit docs under `docs/audits/` and the legacy snapshot under -`docs/memories/zOld/` are left as they were — dated docs stay dated). - -## SKILL.md version history (moved here 2026-10-04) - -Moved verbatim out of `SKILL.md`'s header (prompt audit 2026-10-04: history narrative loaded on every use). -References to "§7", "§5.5 / §8.9" and "the v1.1 note above" are as of the original text. - -v1.1 — recalibrated for Claude Sonnet 5 (launched 2026-06-30). If the running model -IS Sonnet 5, some of this is partly native (self-checking, lower hallucination/ -sycophancy); apply anyway as cheap insurance, and see §7 for what to expect fewer of -and §5.5 / §8.9 for Sonnet-5-specific safeguards, routing, and API changes. - -v1.2 — audited against Anthropic's own Sonnet 5 release notes, system card, and -context-engineering guidance (verified 2026-08-10, not assumed) to check whether -this protocol has become redundant with native model behavior. Verdict on a -section-by-section pass: MOST of it is reasoning structure, output format, or -CyClaw/user-specific policy, not raw hallucination/dishonesty compensation, and -none of that is made obsolete by a better-calibrated model — a smarter model still -needs told WHAT to decompose, HOW to format uncertainty, and WHICH project facts -matter. The one item with direct, verified overlap is §2.3 (self-check before -finalizing), compressed below rather than cut — same "cheap insurance" call as the -v1.1 note above. §1.5/§3.1/§4.1 (verify current-state/stale-prone facts) stay full -strength on purpose: lower hallucination is a different property from knowing -things past a training cutoff, and no model generation fixes the second one. -One verified, opposite-direction finding: Sonnet 5's own system card documents a -regression on hostile-system-prompt/prefill resistance vs 4.6 (see §7 [S5]) — a -large injected prompt is measurably more attack surface on this model generation, -not just more tokens, which is a reason to keep this file lean beyond the -token-cost argument alone. - -v2.0 (2026-09-06) — consolidated the companion `fable-5.1-cc` knowledge-handoff -skill into this file (see §11). No discipline content in §1-7 changed; §8 grew -from a compact "user context" section into the full knowledge base, and §9 is -new (the old checklist and meta sections just shifted from §9-10 to §10-11). -Two factual corrections made in the merge, both caught by applying this -protocol's own §1.5/§3.1 rules to itself: the "LLM Council subgraph" and -"3-layer semantic drift detection" lines in the old §8.3 implied built-and-tested -status; a repo-wide grep for their distinctive terms (`DeBERTa`, `chairman -synthesis`, etc.) at merge time returned zero hits outside these two knowledge -files. Both are now marked as proposed/unverified against the current tree, not -shipped features — see `references/knowledge-base.md` §8.4's Integrity note. - -v2.1 (2026-09-11, issue #1351) — moved §8.4-8.9 (the CyClaw facts-to-know-cold -restatement, other-projects list, hardware notes, decisions table, operational -constraints, and Sonnet-5 API notes — lookup material, not per-turn discipline) -into `references/knowledge-base.md`, read on demand instead of always injected. -This was the single largest chunk of what the SessionStart hook dumped in full -on every session start and every `/compact` (flagged directly in the issue). -§1-9 discipline content is unchanged; §8.1-8.3 (identity, communication -contract, "the pattern") stay inline since they shape tone every turn. +## History (consolidated 2026-10-13; the git log holds the full narrative) + +- **v1.1** — recalibrated for Claude Sonnet 5 (launched 2026-06-30): some + discipline is partly native on that model; apply anyway as cheap insurance. +- **v1.2 (2026-08-10)** — audited section by section against Anthropic's + Sonnet 5 release notes and system card. Verdict: most of the protocol is + reasoning structure, output format or project policy, which a + better-calibrated model does not make obsolete; §2.3 (self-check) was + compressed, §1.5/§3.1/§4.1 (verify stale-prone facts) kept at full strength + because lower hallucination is not knowledge past a training cutoff. One + opposite-direction finding: the Sonnet 5 card documents a regression on + hostile-system-prompt/prefill resistance, a reason to keep this file lean + beyond token cost. +- **v2.0 (2026-09-06)** — merged the companion `fable-5.1-cc` knowledge skill + into this one at the owner's request; the two files had drifted on whether + the "LLM Council" subgraph and 3-layer semantic drift detection were + shipped (a repo-wide grep found neither; both are marked proposed in §8.4). + `fable-5.1-cc`'s SKILL.md and wrapper are deleted; only dated audit docs and + `docs/memories/zOld/` still name it. +- **v2.1 (2026-09-11, issue #1351)** — moved §8.4-8.9 here, out of the body the + SessionStart hook injects on every start and `/compact`. +- **2026-10-04** — the version-history narrative moved out of `SKILL.md`'s + header, and **2026-10-13** — §8.4's restated invariants, numbers, traps and + git workflow were replaced by pointers to `CLAUDE.md` after they drifted. From 33912e822e38a7dea9931ae9807b2f08d1a60bd7 Mon Sep 17 00:00:00 2001 From: Claude Date: Tue, 13 Oct 2026 06:42:04 +0000 Subject: [PATCH 2/2] [docs] - Trim the knowledge and session skills of restated discipline and duplicated rows MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit fable-protocol's §8.2 and §9 restated §1/§6/§1.5/§2.6 inside the same file and the knowledge base repeated §5.1's CWE-1022 note; cyclaw-gotchas' human-run section and Guardrails restated the Run section and CLAUDE.md; CyClaw-Sandbox's first two Gotchas restated Step 2 and the operator map, Known residual 1 ran to ten lines, and Guardrails rows 1-5 restated CLAUDE.md §3 (collapsed to one row pointing at invariant-guard; the skill-specific rows 7-26 are unchanged). Co-Authored-By: Claude Fable 5.1 Claude-Session: https://claude.ai/code/session_01LbrgG4NUbR3sUTgNMfKXpU --- .claude/skills/CyClaw-Sandbox/SKILL.md | 29 +++++-------------- .claude/skills/cyclaw-gotchas/SKILL.md | 18 ++++-------- .claude/skills/fable-protocol/SKILL.md | 21 +++++--------- .../references/knowledge-base.md | 5 +--- 4 files changed, 22 insertions(+), 51 deletions(-) diff --git a/.claude/skills/CyClaw-Sandbox/SKILL.md b/.claude/skills/CyClaw-Sandbox/SKILL.md index a3c79ebd9..2488c8c8f 100644 --- a/.claude/skills/CyClaw-Sandbox/SKILL.md +++ b/.claude/skills/CyClaw-Sandbox/SKILL.md @@ -369,16 +369,13 @@ subdirectory -- invoke them by that path. Track as KNOWN -- confirm they still exist at each full audit; never report one of these as a new finding. -1. `check_jailbreak` is listed in the guardrails config but has no rail of - its own: the offline floor implements only `check_injection` and - `check_soul_mutation` on input, the NeMo flow `check cyclaw jailbreak` - runs the same injection markers, and no LLM-backed rail is active, so a - persona prompt worded off those markers passes. `topical_rails` is - display-only. `guardrail_output` runs `check_soul_leak` and - `check_grounding`, but on the `local_llm` answer only. The generation - broker also checks soul-leak markers on all four answer routes through - NeMo `check()` or deterministic fallback when a live check is unavailable. - Grounding remains scoped to retrieved local answers. +1. `check_jailbreak` has no rail of its own: input rails are + `check_injection` and `check_soul_mutation` (the NeMo flow reuses the + same markers) and no LLM-backed rail is active, so a persona prompt worded + off those markers passes; `topical_rails` is display-only. Output rails + (`check_soul_leak`, `check_grounding`) apply to the `local_llm` answer; + the broker checks soul-leak markers on all four answer routes, grounding + only on retrieved local answers. 2. Telegram's T4 media handling is partial and POSIX-only. 3. Episode-to-fact consolidation is not implemented (the former `memory/consolidation.py` stub was removed). @@ -402,11 +399,7 @@ report, never assume. | # | Invariant | Check | |---|---|---| -| 1 | RAG-First | `retrieve` is always the unconditional graph entry point | -| 2 | Topology = Policy | Routing via the named routers, never a prompt or ad-hoc branch | -| 3 | Triple-Gated External | `app.mode=="hybrid"` AND `.enabled` AND per-request `user_confirmed_online`, all three, for whichever provider is selected | -| 4 | Audit Convergence | Every path -- including `hook-denied`, `external-unavailable`, `guardrail-blocked` -- reaches `audit_logger` exactly once before END | -| 5 | Soul Governance | Evolution requires a non-empty human `reason` string; write is atomic | +| 1–5 | The five graph invariants I1–I5 | As `CLAUDE.md` §3 states them (RAG-first entry, routing by the named routers only, the per-request triple gate for the selected provider, every path -- `hook-denied`, `external-unavailable`, `guardrail-blocked` included -- reaching `audit_logger` exactly once, a human `reason` for soul evolution); `invariant-guard` is the checker | | 6 | Telemetry Contract | Owned by `otel-hardening`, not this skill: canonical maps in `utils/telemetry_kill.py` applied at import (G1 anchor); `python3 .claude/skills/otel-hardening/check_otel.py --strict` is the authority, not a count copied into this document | | 7 | Loopback Only | `api.host == "127.0.0.1"` | | 8 | FsConnect Read-Only Default | `fsconnect.writes_enabled=false`, `follow_symlinks=false` | @@ -427,12 +420,6 @@ report, never assume. ## Gotchas -- **`python3` may not be 3.12.** See Step 2 above -- point explicitly at - `python3.12` (or the session venv's `bin/python`) rather than trusting - the default. -- **Quick Mode and the full audit test different things.** See the - Operator map's Ladder D row -- treat them as complementary, never - substitutable in either direction. - **Three local-LLM realism tiers, not one.** In the in-process swarm, Tier 0 means no responding HTTP backend; Tier 1 = `mock_ollama.py`; Tier 2 = a real daemon. `run_full_verification.py` auto-detects which one is live and reports it; diff --git a/.claude/skills/cyclaw-gotchas/SKILL.md b/.claude/skills/cyclaw-gotchas/SKILL.md index 9f4af9478..79fff1140 100644 --- a/.claude/skills/cyclaw-gotchas/SKILL.md +++ b/.claude/skills/cyclaw-gotchas/SKILL.md @@ -140,11 +140,8 @@ the shipped posture is now hybrid with grok/claude enabled (CLAUDE.md's warning. A `1 warning` result today means the config no longer matches that documented posture — treat it as drift to investigate, not the old baseline. -## Run (human path) - -`python gate.py` in a terminal binds `127.0.0.1:8787` and blocks; Ctrl-C to -stop. Same defaults, no pidfile. Useless from an agent because nothing else -can run in that shell; use `serve`. +`python gate.py` by hand binds `127.0.0.1:8787` and blocks the shell (Ctrl-C +to stop); from an agent, use `serve`. ## Gotchas @@ -303,13 +300,10 @@ expire it. ## Guardrails -- This skill changes nothing by itself. `driver.sh` writes only under - `/tmp/cyclaw-gotchas` and `/root/.venv-cyclaw-312` (override with - `CYCLAW_SCRATCH` / `CYCLAW_VENV`); `serve` creates `logs/` in the repo - exactly as `gate.py` would. -- CLAUDE.md §3's six invariants and §7's escalation tiers still govern. A - gotcha here is never a licence to skip a guard; it is a reason you will not - be surprised by one. +- `driver.sh` writes only under `/tmp/cyclaw-gotchas` and + `/root/.venv-cyclaw-312` (override with `CYCLAW_SCRATCH` / `CYCLAW_VENV`); + `serve` creates `logs/` in the repo exactly as `gate.py` would. A gotcha + here is never a licence to skip a guard (CLAUDE.md §3, §7). ## Maintaining this file diff --git a/.claude/skills/fable-protocol/SKILL.md b/.claude/skills/fable-protocol/SKILL.md index a6045b481..e5845d0b8 100644 --- a/.claude/skills/fable-protocol/SKILL.md +++ b/.claude/skills/fable-protocol/SKILL.md @@ -197,13 +197,9 @@ and doc as something another model may also be touching in parallel. ### 8.2 How he learns and wants to be spoken to (standing contract, owner-stated) -- **Truth over comfort.** Factual accuracy > precision > concision (§1.1). He is - sensitive AND wants bluntness; the two are not in tension for him. Wrong - premise gets called in sentence one (§6.3). Credit when earned, specific, never - "great question" (§6.2). -- **Mark speculation** (§1.2). He explicitly demands it. Unmarked confident - guesses are the fastest way to lose him. "I don't know" is a valid answer; a - plausible fill-in is the failure. +- **§1 and §6 are his standing contract, not defaults to soften:** truth over + comfort (he is sensitive AND wants bluntness; the two are not in tension + for him), speculation marked, "I don't know" over a plausible fill-in. - **Socratic by default, direct at the poles.** Lead with questions when he is partly right or exploring. When he is clearly right: confirm and move. When he is clearly wrong: say so first, then teach. Do not perform Socratic method on a @@ -274,15 +270,12 @@ for when a specific fact is actually needed. 3. **Scope creep.** Every CyClaw session tempts a "while I'm here" edit to `CLAUDE.md`, `config.yaml`, or `ci.yml`. Those are the shared files other agents are also editing. Touch only what the task names. -4. **Confabulated flags and IDs.** Version numbers, CVE IDs, config keys, and CLI - flags in this repo are dense and specific (§1.5). Grep before you cite. If you - cannot, say so. +4. **Grep before you cite** (§1.5): version numbers, CVE IDs, config keys + and CLI flags here are dense and specific. If you cannot verify, say so. 5. **Knowing the ceiling.** Say "this exceeds what I can verify here" rather than producing a confident approximation (§10). He would rather have a marked gap - than a fluent wrong answer, and he checks. -6. **Constraint persistence.** Long sessions drift (§2.6). Every ~10 turns - re-inventory: branch name, mode (quick/thorough), what he actually asked - for, open PRs. + than a fluent wrong answer, and he checks. §2.6's ~10-turn re-inventory + (branch, mode, the actual ask, open PRs) is the same discipline over time. ## 10. PER-RESPONSE CHECKLIST (silent, every turn) diff --git a/.claude/skills/fable-protocol/references/knowledge-base.md b/.claude/skills/fable-protocol/references/knowledge-base.md index ba214c1c7..15ee7ceb5 100644 --- a/.claude/skills/fable-protocol/references/knowledge-base.md +++ b/.claude/skills/fable-protocol/references/knowledge-base.md @@ -141,10 +141,7 @@ Integrity note above for two of these): 3-layer semantic drift detection for - GitHub fetch: base repo pages and blob/main paths fetch fine; /tree/, /commits/, /pulls, PR pages are robots.txt-blocked. For blocked areas, request pasted content or use raw.githubusercontent.com. Don't pretend to have read what you - couldn't fetch. -- CWE-1022: "Use of Web Link to Untrusted Target with window.opener Access" — - reverse tabnabbing. Fix: rel="noopener noreferrer" on target=_blank, or - window.opener=null on programmatic window.open(). Apply per §5.1. + couldn't fetch. (CWE-1022 / reverse tabnabbing is covered in `SKILL.md` §5.1.) ### 8.9 Sonnet-5 API notes and model-tier provenance