diff --git a/.claude/README.md b/.claude/README.md index 0a696a95a..77c0fb472 100644 --- a/.claude/README.md +++ b/.claude/README.md @@ -67,12 +67,9 @@ All `*-refactor` skills follow the same seven-step cycle: Markdown, no frontmatter, applies repo-wide) ``` -2026-09-16 (issue #1351): `patterns/` (9 behavioral-pattern docs), `utility-prompts/` -(4 prompt scaffolds), `tools/` (4 tool-usage docs), `memory/` (legacy, superseded -by `docs/memories/`), and `ponytail-marketplace.json` (the local plugin -marketplace it registered, along with the `ponytail` plugin, is gone from -`settings.json` too) were all deleted — see `CLAUDE.md` §9 for which skills went -with them. +Issue #1351 (2026-09-16) deleted `patterns/`, `utility-prompts/`, `tools/`, +`memory/` (superseded by `docs/memories/`), and `ponytail-marketplace.json` +(with the `ponytail` plugin entry in `settings.json`); do not re-create them. ## Skill Caching Policy @@ -83,13 +80,9 @@ version-controlled, shared with every collaborator on this repo), **user** utilities shipped by Claude Code itself or by installed marketplaces). This directory intentionally vendors only the **project** scope — every skill -in `CLAUDE.md` §9's tables already lives under `.claude/skills/` here, one -folder per skill, named for the `name:` in its `SKILL.md` frontmatter. -(2026-09-11, issue #1351: `CyClaw-Sandbox/`'s frontmatter used to declare -`name: cyclaw-swarm-verification`, a folder/frontmatter mismatch that made -repo audits relying on either name alone miss the other — corrected to -`name: CyClaw-Sandbox`.) User-scope and built-in -skills are **not** copied in, for three concrete reasons: +in `CLAUDE.md` §9's tables lives under `.claude/skills/` here, one folder per +skill, named for the `name:` in its `SKILL.md` frontmatter. User-scope and +built-in skills are **not** copied in, for three concrete reasons: 1. **YAGNI / no current caller.** No CyClaw code path or documented workflow invokes them — they serve the operator across unrelated repos, not this @@ -116,126 +109,70 @@ and skills separately. Wrappers are deliberately thin — the `SKILL.md` stays the single source of truth and the wrapper only points at it, so the two cannot drift apart in substance. Five commands are standalone by design and have no skill folder (`/audit`, `/check-soul`, `/conversation-summary`, -`/run`, `/status`) — they are short inline procedures, not skills. -2026-08-11: added the two missing wrappers (`config-guard`, `dep-guard`). -2026-09-04: added `fable-5.1-cc`, the last skill that shipped without one. -2026-09-06: `fable-5.1-cc` was consolidated into `fable-protocol` (its content -is now that skill's §8 onward) and deleted, wrapper and all — see -`fable-protocol/SKILL.md` §11. -2026-09-11 (issue #1351): `general-purpose` (skill + wrapper) deleted — it -duplicated the Agent tool's built-in `general-purpose` subagent type -verbatim, adding no project-specific content. 2026-09-11 (PR #1369): -`cyclaw-advisor` and `CyClaw-Sandbox` gained `disable-model-invocation: true` -(their wrappers are unaffected — a wrapper is already user-typed by -definition). 2026-09-11 (issue #1351 follow-up): ten more slash-only chore/ -loop skills gained `disable-model-invocation: true`; `cyclaw-advisor`'s -folder and wrapper renamed `cyclaw-privacy` (see Key Conventions below for -why); `CyClaw-Optimize.md`, `logging-refactor.md`, `tests-refactor.md`, -`speed-refactor.md`, and `architecture-refactor.md` — the five wrappers that -had drifted from thin pointers into near-complete `SKILL.md` copies, one -already stale against its source — trimmed back to the thin-wrapper pattern -this policy describes. The wrapper set -still covers every remaining skill — verify with +`/run`, `/status`) — they are short inline procedures, not skills, so a +deletion pass that treats them as orphaned wrappers removes real commands +(this happened once under issue #1351; both were restored from `main`). + +Verify the wrapper set still covers every skill with `comm -23 <(ls .claude/skills | sort) <(ls .claude/commands | sed 's/\.md$//' | sort)`, -which must print nothing (except a stray untracked `__pycache__` under a -since-deleted skill dir, which is a build artifact, not a wrapper gap — `git -clean -ndx .claude/skills` finds those; delete, don't wrap them). -2026-09-16 (issue #1351): eleven more skill+command pairs deleted — -`code-explorer`, `create-session-notes`, `memory-consolidation`, -`memory-extraction`, `memory-orchestrator`, `next-action-suggestion`, -`ponytail`, `python-coding-agent`, `session-title`, `tool-summary`, `wrap-up` -— per the issue's delete-or-command-only list. `audit.md` and `run.md` were -swept up in the same deletion pass by mistake: both are standalone commands -with no skill twin (the five-standalone-commands rule above names them -explicitly), so deleting them removed `/audit` and `/run` outright rather than -trimming a wrapper. Restored from `main` unchanged. Deleting -`memory-orchestrator` also orphaned the `PreCompact`/`SessionEnd` hooks in -`settings.json` that invoked its `orchestrate.py` — unwired them (same call -as the 2026-09-04 dangling-hook incident below: don't resurrect a deleted -skill to satisfy a hook, remove the hook) rather than leaving them to error on -every compact/session-end. `CLAUDE.md` §9's "Agent skills" section has the -per-skill rationale. -2026-09-16 (issue #1351, second follow-up): three more auto-invocable skills -gained `disable-model-invocation: true` — `fable-protocol` (the single -biggest description in the catalog, and now redundant with the -`fable-protocol-loader.sh` hook for the one model tier that needs it unasked; -see that hook's section below), `CyClaw-Optimize` (opens PRs on its own), -and `babysit-github-pr` (starts an extended watch loop against live GitHub -state). Two more had their descriptions trimmed to the issue's ~600-char -budget while staying auto-invocable: `otel-hardening` (1129 → 588 chars) and -`verify-deps` (1288 → 539 chars) — both keep an explicit "use `dep-guard`/ -`invariant-guard`/`injection-redteam` instead" when-not clause. `dep-guard` -vs `verify-deps` staying as two separate skills (the issue suggested picking -one name) was deliberately left alone — that is an information-architecture -merge, not a routing-cost fix, and deserves its own decision. - -## Environment Doctor — settings.json audit (2026-08-11) - -A doctor-style audit of `.claude/settings.json` was run against the live tree -(don't re-create what passes; fix only what's broken). Verdict and findings: - -**settings.json: the dangling hook is gone (resolved 2026-09-04).** Valid JSON -against the declared schema, and every hook path now resolves. The history is -worth keeping: from some point before 2026-08-22 until 2026-09-04 the registered -`UserPromptSubmit` hook pointed at -`.claude/skills/fable-protocol/context_gate.py`, a file deleted on `main`. Because -the command ended in `2>/dev/null || true` it silently no-opped on every prompt -instead of erroring — which is precisely why it went unnoticed for weeks. The -operator's call was to unwire it rather than write a replacement: the skill stays -reachable as `/fable-protocol` and through its `description` trigger, and injecting -a ~22KB skill into every prompt is cost the skill's own v1.2 preamble warns -about (§7's [S5] row makes the same case on attack-surface grounds). **Lesson -encoded:** the sync-check hook added the same day carries no `|| true`, because the -script already always exits 0 and that tail is the thing that hid the last failure. -Otherwise: no personal data (no usernames, absolute machine paths, or -emails — keep it that way, this file is shared with every collaborator); -hooks anchor to repo-relative paths so they survive any checkout location. -The `SessionStart` sync-check added 2026-09-04 has no `|| true` — that -is deliberate, not an oversight (see the dangling-hook note above). - -(2026-09-16 update, issue #1351: the paragraph above once described five hook -entries resolving to four scripts, including `memory-orchestrator/orchestrate.py` -referenced twice for `PreCompact`/`SessionEnd`, and a persona-loader -`SessionStart` entry for `python-coding-agent`. All three were removed along -with their target skills — `settings.json` now registers only the -`SessionStart` hooks below, and the observation that the memory hooks lacked -a `|| true` guard is moot; they no longer exist to lack one.) - -**`session-start-venv.sh` (third `SessionStart` hook, added 2026-10-05).** -Cloud-only (`CLAUDE_CODE_REMOTE=true`; a no-op on a laptop). It detaches -`cyclaw-gotchas/driver.sh venv` and returns at once, so the Python 3.12 venv at -`/root/.venv-cyclaw-312` builds while the session starts instead of when the -first `pytest` needs it. `driver.sh venv` takes a lock, so an agent that runs it -by hand meanwhile waits for the build and then gets `venv ready`. The driver -uses `uv` when it is on PATH (cold build 413 s with pip vs 122 s with uv, -measured 2026-10-05 on the default cloud image) and pip otherwise. Log: -`$TMPDIR/cyclaw-session-start-venv.log` (`/tmp` when unset). - -**`fable-protocol-loader.sh` (second `SessionStart` hook as of 2026-09-16; -was the third, added 2026-09-06, until the persona-loader hook ahead of it -was deleted).** -This is the replacement the 2026-09-04 note said was not being written, on a -different trigger and with a gate. It injects `fable-protocol/SKILL.md` as -`additionalContext` once per SessionStart (startup/resume/clear/compact), the -same moment and mechanism as the persona loader, NOT per prompt — so the -per-prompt cost that got the old `UserPromptSubmit` hook unwired does not -return. 2026-09-16 (issue #1351): re-gated from an opt-out blocklist (inject -unless the model id contains `fable`/`mythos`) to an opt-in allowlist — -injects only when the model id contains `sonnet`. Opus, Haiku, Fable, and any -absent/unrecognized string are all skipped now, each reachable only via -explicit `/fable-protocol`; the flip pairs with the skill itself gaining -`disable-model-invocation: true` the same day (see the wrapper-policy note -above), so the ~1.5KB body is no longer the catalog's biggest routing-cost -line AND is no longer auto-injected outside the one tier it's tuned for. Why -SessionStart and why a gate that can miss: verified against Claude Code -2.1.261's hook schema, SessionStart is the only event whose stdin JSON -carries `model` (optional), and no event fires on a mid-session `/model` -switch — so switching to Sonnet mid-session (or wanting the protocol on -Opus/Haiku at all) is not re-gated; `CLAUDE.md` §10 tells the operator to run -`/fable-protocol` by hand in that case. The script exits 0 unconditionally -and keeps stdout JSON-only, with diagnostics on stderr; re-tested 2026-09-16 -with sonnet/opus/haiku/fable/mythos/absent/malformed stdin against the new -allowlist logic. +which must print nothing (a stray untracked `__pycache__` under a deleted +skill dir is a build artifact — `git clean -ndx .claude/skills` finds those; +delete, don't wrap them). + +Rules that came out of issue #1351 and its follow-ups (2026-09-11 → 09-16; +the per-skill rationale is in `CLAUDE.md` §9): + +- A wrapper that grows into a near-copy of its `SKILL.md` is drift; trim it + back to a pointer. +- A skill that duplicates a built-in (the `general-purpose` subagent type) or + is a slash-only chore/loop gets deleted or `disable-model-invocation: true`. + Skills that act on live state unasked (`CyClaw-Optimize` opens PRs, + `babysit-github-pr` starts a watch loop) are user-invoked only, as is + `fable-protocol`, whose Sonnet-tier injection moved to + `fable-protocol-loader.sh`. +- Auto-invocable descriptions stay under the ~600-char routing budget + (`otel-hardening` and `verify-deps` were cut to size and keep their "use X + instead" clauses). `dep-guard` and `verify-deps` remain two skills on + purpose: merging them is an information-architecture decision, not a + routing fix. +- Deleting a skill that a `settings.json` hook invokes means removing the + hook, never resurrecting the skill (`memory-orchestrator`'s + `PreCompact`/`SessionEnd` hooks went this way). + +## settings.json — hooks and the rules behind them + +`settings.json` registers exactly three hooks, all `SessionStart`, in this +order; every path resolves and is repo-relative so it survives any checkout +location. The file holds no personal data (no usernames, machine paths, or +emails) — keep it that way, it is shared with every collaborator. + +1. **`session-start-sync-check.sh`** (2026-09-04) — git identity and + local↔remote divergence report; never resets, rebases, or pushes. +2. **`fable-protocol-loader.sh`** (2026-09-06, re-gated 2026-09-16) — injects + `fable-protocol/SKILL.md` as `additionalContext` once per SessionStart + (startup/resume/clear/compact), never per prompt, and only when the stdin + model id contains `sonnet`; Opus, Haiku, Fable, and absent or unrecognized + ids are skipped and reach the protocol only via `/fable-protocol`. + SessionStart is the only hook event whose stdin carries `model`, and no + event fires on a mid-session `/model` switch, so `CLAUDE.md` §10 tells the + operator to run it by hand then. Exits 0 unconditionally, stdout JSON-only, + diagnostics on stderr. +3. **`session-start-venv.sh`** (2026-10-05) — cloud-only + (`CLAUDE_CODE_REMOTE=true`; a no-op on a laptop). Detaches + `cyclaw-gotchas/driver.sh venv` so the Python 3.12 venv at + `/root/.venv-cyclaw-312` builds while the session starts; a manual + `driver.sh venv` waits on the same lock and then reports `venv ready`. The + driver uses `uv` when on PATH (cold build 122 s vs 413 s with pip, measured + 2026-10-05) and pip otherwise. Log: `$TMPDIR/cyclaw-session-start-venv.log`. + +**Why no hook ends in `|| true`.** From before 2026-08-22 until 2026-09-04 a +`UserPromptSubmit` hook pointed at a file deleted on `main`; its +`2>/dev/null || true` tail made it a silent no-op on every prompt for weeks. +The fix was to unwire it (a ~22KB per-prompt injection was the wrong cost +anyway — the loader above is the gated replacement), and every hook since +exits 0 on its own so nothing hides a failure. The same rule removed the +`PreCompact`/`SessionEnd` hooks orphaned by `memory-orchestrator`'s deletion +and the `python-coding-agent` persona loader (issue #1351). **Remote-environment env-var misconfiguration (root cause of the stray `C:\Users\...` directory).** The Claude Code remote execution environment for @@ -278,13 +215,10 @@ repo: ## Key Conventions -- Skill folders match the `name:` in SKILL.md frontmatter. Most are - `kebab-case`; two ship mixed-case by convention (`CyClaw-Optimize`, - `CyClaw-Sandbox`). (`OTel-Hardening` was a third until issue #1135 renamed - it `otel-hardening` to satisfy the Agent Skills spec's lowercase - name-matches-directory rule; `CyClaw-Sandbox` itself carried a similar - mismatch — frontmatter `name: cyclaw-swarm-verification` — until issue - #1351 aligned it to `name: CyClaw-Sandbox`.) +- Skill folders match the `name:` in SKILL.md frontmatter (a mismatch makes + audits keyed on either name miss the other — `CyClaw-Sandbox` had one until + issue #1351). Most are `kebab-case`; two ship mixed-case by convention + (`CyClaw-Optimize`, `CyClaw-Sandbox`). - All SKILL.md files use YAML frontmatter: `name:`, `description:` - Refactor progress is tracked in `/tmp/refactor-CyClaw.md` - Claude Code commits as the session runtime's identity (cloud: diff --git a/.claude/skills/babysit-github-pr/README.md b/.claude/skills/babysit-github-pr/README.md index 2d32fa110..8e737797d 100644 --- a/.claude/skills/babysit-github-pr/README.md +++ b/.claude/skills/babysit-github-pr/README.md @@ -7,55 +7,18 @@ comments; code fixes and review responses require the agent or operator. ## Installation -### Project-Local (`.claude/skills/`) - -Already installed in CyClaw. For other repos: - -```bash -# Copy the skill into your repo's .claude/skills/ directory -mkdir -p .claude/skills -cp -r ~/.claude/skills/babysit-github-pr .claude/skills/ -``` - -Then invoke with `/babysit-github-pr ` from within Claude Code. - -### User-Wide (`~/.claude/skills/`) - -```bash -# Copy to your home directory's skills folder (once per machine) -mkdir -p ~/.claude/skills -cp -r /path/to/babysit-github-pr ~/.claude/skills/ -``` - -Then invoke `/babysit-github-pr` from any repo. +Already installed in CyClaw (`.claude/skills/babysit-github-pr/`). For another +repo, copy the folder into its `.claude/skills/` (project scope) or into +`~/.claude/skills/` (user scope, once per machine), then invoke +`/babysit-github-pr ` from Claude Code. ## Prerequisites -Before using this skill, ensure: - -1. **`gh` CLI** (GitHub CLI) — authenticated with `repo` scope - ```bash - gh auth login - gh auth status # Verify you see "✓ Token: ghu_..." and "✓ Scopes: repo, gist" - ``` - -2. **`jq`** — JSON query tool (for parsing GitHub API responses) - ```bash - # macOS - brew install jq - - # Linux (Ubuntu/Debian) - sudo apt-get install jq - - # Windows (choco or scoop) - choco install jq # or: scoop install jq - ``` - -3. **`python3`** (3.8+) — for failure classification and command detection - -4. **`git`** — version control (already assumed) - -5. **Project's test/lint commands discoverable** from one of: +1. **`gh` CLI** authenticated with `repo` scope (`gh auth login`; `gh auth + status` should show `✓ Scopes: repo, gist`). +2. **`jq`** (`brew install jq` / `sudo apt-get install jq` / `choco install jq`). +3. **`python3`** (3.8+) for failure classification and command detection, and `git`. +4. **Project's test/lint commands discoverable** from one of: - `Makefile` (targets: `test`, `lint`) - `package.json` (scripts: `test`, `lint`) - `pyproject.toml` (pytest/tox config) @@ -199,12 +162,10 @@ Address the blocker, then re-invoke. ## For Repository Owners -This skill works on any GitHub repository with: -- Checks reported on the GitHub PR. Automated retries use GitHub Actions runs -- A detectable test/lint command (Makefile, package.json, pyproject.toml, or workflows) -- Branch protection rules (optional but recommended) - -No configuration in the repo itself is needed — the skill auto-detects everything. +Works on any GitHub repository whose checks report on the PR (automated +retries use GitHub Actions runs) and whose test/lint command is detectable +(Makefile, package.json, pyproject.toml, or workflows); branch protection is +optional but recommended. No in-repo configuration is needed. --- diff --git a/.codex/README.md b/.codex/README.md index 093f4c929..a896cda49 100644 --- a/.codex/README.md +++ b/.codex/README.md @@ -68,26 +68,20 @@ in the same change. ## Current-source discipline -Fetch and inspect current code before reusing a skill's observations. See -`$cyclaw-project-guidance` for the source map instead of copying another dated -runtime snapshot here. - -All 22 skill entry points have UI metadata. The directory `Cyclaw-Sandbox` -keeps its historical spelling; its skill/invocation name is `cyclaw-sandbox`. -The optimization skill consistently uses `cyclaw-optimize`. Preserve existing -invocation policy when updating metadata. - -The dependency workflow distinguishes `requirements.txt` (runtime) from -`requirements-test.txt` (tests), uses the actual pip install commands for -dry-runs, and preserves plain Torch constraints on macOS. - -Use the canonical `.claude/skills/doc-sync/` checker. The Codex Python/shell -entry points are wrappers, not separate parser implementations. Historical -sandbox reports describe prior runs only; `NEW_SKILL.md` redirects to the -current sandbox skill. That bundle was reconciled with `3d913754` on -2026-09-12 after the Python coding console removal; the terminal gateway, -`agentic/` CLI and retained `agentic/harness_optimizer/` components remain. Validate actual -commands and paths, not just the presence of a skill name in this table. +Fetch and inspect current code before reusing a skill's observations; see +`$cyclaw-project-guidance` for the source map instead of copying a dated +runtime snapshot here. Validate actual commands and paths, not just the +presence of a skill name in the table above. + +All 22 skill entry points carry UI metadata. The directory `Cyclaw-Sandbox` +keeps its historical spelling; its invocation name is `cyclaw-sandbox`, and +the optimization skill is `cyclaw-optimize`. Preserve existing invocation +policy when updating metadata. The dependency workflow distinguishes +`requirements.txt` (runtime) from `requirements-test.txt` (tests), dry-runs +the actual pip commands, and keeps plain Torch constraints on macOS. The +Codex doc-sync entry points wrap the canonical `.claude/skills/doc-sync/` +checker rather than reimplementing its parser. Historical sandbox reports +describe prior runs only; `NEW_SKILL.md` redirects to the current sandbox skill. ## Prompt Templates diff --git a/.githooks/README.md b/.githooks/README.md index 5011c0ede..bd950080b 100644 --- a/.githooks/README.md +++ b/.githooks/README.md @@ -43,7 +43,7 @@ sees too late or cannot see: |-------|-----------|----------|----------| | Credential-shaped strings in added lines (provider prefixes; plus `gitleaks` when installed) | staged diff | every commit being published | none — fix, or mark a fixture `gitleaks:allow` | | Credential / runtime-state filenames (`.env`, `*.key`, `*.db`, `CONSOLIDATED.md`, …) | staged | every commit being published | `SEC_BLOCKED_EXCEPT` in `security.conf` | -| Protected control files (`soul.md`, `utils/sanitizer.py`, `gate_auth.py`, scanner ignore files, `.githooks/*`) | staged | pushed range | `HOOK_OPERATOR_ACK=1` (operator only) | +| Protected control files (`soul.md`, `utils/sanitizer.py`, `gate_auth.py`, scanner ignore files, `.githooks/*`) | staged (blocks) | pushed range (reminder only) | `HOOK_OPERATOR_ACK=1` (operator only) | | Privacy: operator identifiers from the untracked private file, absolute home-directory paths, invisible bidi / Unicode-tag characters | staged diff | every commit being published | none — use a placeholder; fixture lines take `gitleaks:allow` | | Agent session residue (`.aider*`, `settings.local.json`, `*.har`, shell histories, …), new files over `SEC_MAX_NEW_FILE_KB`, media with GPS/author metadata (needs `exiftool`) | staged | filenames only | `SEC_BLOCKED_EXCEPT` / raise the cap with the operator | | `.gitignore` rule removed (also checked on pushed commits); agent tool wiring (`.mcp.json`, `.claude/settings.json`, `.codex/config.toml`) changed or deleted | staged | rule removal: blocks; wiring: reminder | `HOOK_OPERATOR_ACK=1` (operator only) | @@ -65,46 +65,22 @@ and the `main` ruleset remain the controls. ## PR body template (not a git hook) Git cannot intercept PR bodies created via the GitHub UI, `gh pr create`, or -the GitHub connector API. For that layer: - -1. **Always** fill [`.github/PULL_REQUEST_TEMPLATE.md`](../.github/PULL_REQUEST_TEMPLATE.md) fully when opening a PR (required for Grok Build + GitHub connector on cgfixit/CyClaw). -2. **Local check:** `scripts/check-pr-template.sh path/to/body.md` before create. -3. **CI:** blocking check via `.github/workflows/pr-template-check.yml` (same headers as `scripts/check-pr-template.sh`). - -Example: - -```bash -# draft body from template, edit, then gate -cp .github/PULL_REQUEST_TEMPLATE.md /tmp/pr-body.md -# ... fill sections ... -scripts/check-pr-template.sh /tmp/pr-body.md -gh pr create --title '[docs] - …' --body-file /tmp/pr-body.md -``` +the GitHub connector API, so that layer is a script plus CI: fill +[`.github/PULL_REQUEST_TEMPLATE.md`](../.github/PULL_REQUEST_TEMPLATE.md) +fully, run `scripts/check-pr-template.sh path/to/body.md` before creating, and +`.github/workflows/pr-template-check.yml` blocks on the same rules. ## Install (once per clone) ```bash -bash scripts/install-githooks.sh -# or: git config core.hooksPath .githooks && chmod +x .githooks/* -``` - -Verify: - -```bash -git config core.hooksPath # → .githooks -``` - -The pre-push gate deliberately does not rebase or force-push for you. Rebase, -inspect conflicts, rerun the relevant checks, and use force-with-lease only -with explicit approval. It also cannot infer multi-PR semantics: map shared -files and trial the chronological merge order as required by -`.codex/Codex_instructions.md`. - -## Bypass (emergency only) - -```bash -git commit --no-verify -git push --no-verify +bash scripts/ensure-githooks.sh # idempotent; what every agent setup runs +git config core.hooksPath # → .githooks ``` -Do not use bypass for routine feature work. +`scripts/install-githooks.sh` (or `git config core.hooksPath .githooks`) sets +it unconditionally. The pre-push gate never rebases or force-pushes for you: +rebase, inspect conflicts, rerun the checks, and use force-with-lease only with +explicit approval. It cannot infer multi-PR semantics either; map shared files +and trial the merge order as `.codex/Codex_instructions.md` requires. +`--no-verify` is an emergency bypass, never routine (`pre-push` still scans +commits made that way; CI is the backstop). diff --git a/README.md b/README.md index af5809473..782ecf938 100644 --- a/README.md +++ b/README.md @@ -31,13 +31,9 @@ Step-by-step installs and every REST call: [Full Setup Guide](setup-guide.md). Contributor rules for people and agents: [`CLAUDE.md`](CLAUDE.md), [`AGENTS.md`](AGENTS.md), and the tracked git-hook gate in [`docs/GITHOOKS.md`](docs/GITHOOKS.md). ---- - -Security review #1526 has an explicit [residual decision register](SECURITY.md#issue-1526-residual-decisions-2026-10-12): -TLS logout and path privacy, Git helper containment, lock/metadata drift checks, -approved-head patch publication, and encrypted scheduled sync. Native Windows -verification remains disabled; supported confinement still permits host reads. -The register distinguishes implemented controls, accepted limits, and optional research. +Security review #1526's [residual decision register](SECURITY.md#issue-1526-residual-decisions-2026-10-12) +separates implemented controls, accepted limits (native Windows verification +stays disabled; supported confinement still permits host reads), and optional research. ## Quick Start @@ -69,12 +65,10 @@ CI checks every lock group against recorded inputs and hashes, and checks wheel [Windows](setup-guide.md#windows-powershell), [Linux](setup-guide.md#linux-bash). Confirm: `curl http://127.0.0.1:8787/health`, then open -`http://127.0.0.1:8787/`. - -Updating an existing environment? Rerun the install, then the -[NeMo verification](setup-guide.md#verify-the-installed-nemo-runtime). A manual -macOS install uses plain `torch==2.13.0` (no `+cpu` wheel) then the hashed -macOS lock ([macOS (Apple Silicon)](setup-guide.md#macos-apple-silicon)). +`http://127.0.0.1:8787/`. A manual macOS install uses plain `torch==2.13.0` +(no `+cpu` wheel) then the hashed macOS lock +([macOS (Apple Silicon)](setup-guide.md#macos-apple-silicon)); updating an +existing environment ends with the [NeMo verification](setup-guide.md#verify-the-installed-nemo-runtime). --- @@ -137,10 +131,8 @@ is now the separate [CG-agent-harness](https://github.com/cgfixit/CG-agent-harne `/query` always rejects cross-site requests. - Restart the gateway after editing sanitizer config (it is cached). `CYCLAW_EMBED_CACHE_SIZE` (default 2048) is an env var, not a YAML key. - -Check policy offline with -`python .claude/skills/invariant-guard/check_invariants.py`; the unit suite is -`GROK_API_KEY=dummy python -m pytest tests/ -q --tb=short` with no live provider. +- Offline checks: `python .claude/skills/invariant-guard/check_invariants.py` + and `GROK_API_KEY=dummy python -m pytest tests/ -q --tb=short` (no live provider). **Local records.** Paths are relative to the repository. @@ -203,11 +195,9 @@ does not verify individual claims, which the [evals](docs/EVALS.md) measure. 7. **Audit convergence.** Every path converges at `audit_logger` before END; `cyclaw-metrics` reads the audit offline. -[The six invariants](CLAUDE.md#3-the-six-invariants) cover retrieval-first -entry, topology, external consent, audit convergence, soul governance, and -import isolation (core modules never import `agentic`, `sync`, `guardrails`, -`telegram`, or `opentweet`; the invariant checker enforces it). -[`INVARIANTS.md`](INVARIANTS.md) states Rules 1–9 and the test behind each. +[The six invariants](CLAUDE.md#3-the-six-invariants) pin these guarantees plus +import isolation of the core modules; [`INVARIANTS.md`](INVARIANTS.md) states +Rules 1–9 and the test behind each. **Storage and console.** `indexing.vector_backend` defaults to embedded `chroma` (optional `pgvector`); BM25 is JSON, never pickle. @@ -234,10 +224,9 @@ when auth is on. ## Architecture -`gate.py` initializes the soul at startup. Requests pass the body cap, Host -allowlist, any required authentication, the **60/min per-IP limit (before -injection filtering)**, and the config-driven filter, then enter the graph as -`GraphState`, where edges control routing. +`gate.py` initializes the soul at startup; requests pass the body cap, Host +allowlist, any required authentication, the 60/min per-IP limit, and the +injection filter (in that order), then enter the graph as `GraphState`. ```mermaid flowchart TD @@ -410,12 +399,11 @@ no judge result is published yet. ## Current development -Checked against [`origin/main` at `2d7191a`](https://github.com/cgfixit/CyClaw/commit/2d7191a5036a04cd090061bcea3e406e44744c24) -on 2026-10-08. Recent work has gone into the tracked git-hook secrets gate -([`docs/GITHOOKS.md`](docs/GITHOOKS.md)), sandboxed agentic verification, -hashed installs, credential and request-body hardening, and console first-run -fixes; see [merged PRs](https://github.com/cgfixit/CyClaw/pulls?q=is%3Apr+is%3Amerged) -for details. +Recent work: the tracked git-hook secrets gate ([`docs/GITHOOKS.md`](docs/GITHOOKS.md)), +sandboxed agentic verification, hashed installs, credential and request-body +hardening, and console first-run fixes. The +[merged PRs](https://github.com/cgfixit/CyClaw/pulls?q=is%3Apr+is%3Amerged) +and [`docs/changelog.txt`](docs/changelog.txt) are the record. --- @@ -448,11 +436,8 @@ decision before committing; push and draft PR are separate decisions. Checks run in fresh gitless copies under Linux bubblewrap or macOS Seatbelt and fail closed without them (Windows refuses); no platform uses a microVM ([threat model](docs/THREAT_MODEL.md)). `real-repo-run*` is CLI-only -(`POST /ops/agentic` returns 422). -Git origin identity is pinned across reopened runs, and timed-out Git calls -reap helper processes ([workspace controls](agentic/README.md#c-real-repo-workspace-tools-repoworkspacetools)). -Windows Git helper launch waits for Job Object containment before starting Git. -Orphan cleanup is limited to generated clone directories, preserving workspace metadata. +(`POST /ops/agentic` returns 422). Git helper containment, identity pinning, +and orphan cleanup: [workspace controls](agentic/README.md#c-real-repo-workspace-tools-repoworkspacetools). ```bash python -m agentic.cli status @@ -485,9 +470,8 @@ to `guardrails/`. Active rails are Python checks and add no model calls; NVIDIA's model-assisted `self_check_*` rails are inactive. NeMo failures audit -`guardrail_degraded`. Existing environments: rerun the install, then -`python -m pip check` and `python -m guardrails.verify_install` -([setup guide](setup-guide.md#verify-the-installed-nemo-runtime), +`guardrail_degraded`. Verify an install with `python -m pip check` and +`python -m guardrails.verify_install` ([setup guide](setup-guide.md#verify-the-installed-nemo-runtime), [package guide](guardrails/README.md), [NeMo reference](docs/NeMo/README.md), [Track B record](docs/audits/2026-10-03-nemo-track-b.md)). @@ -504,12 +488,11 @@ vendors or imports it. | Offline scoring | None | Operator CLI and `numbat-rules.yml`; fixture, stream-contract, and CEL checks block CI | Nothing scores the live file during a request. The CLI and CEL extra are not -in standard installs, so keep their switches off until installed (an enabled -hook without the binary denies every confirmed external call), trial rules -before promoting any to `enforce: true`, and never use `numbat hook` as the -command engine (it exits 0 on errors). Stream events include hostname, user, -and uid. Once enabled, `/health` reports hook and CEL readiness and -`/audit/summary` shows `pre_action_hook_last_verdict`. +in standard installs: keep their switches off until installed (an enabled hook +without the binary denies every confirmed external call), trial rules before +promoting any to `enforce: true`, and never use `numbat hook` as the command +engine (it exits 0 on errors). Once enabled, `/health` reports hook and CEL +readiness and `/audit/summary` shows `pre_action_hook_last_verdict`. [Pre-action gate](docs/security-philosophy/numbat_pre_action_gate.md), [stream](docs/security-philosophy/numbat_secondary_evaluator.md), [phase status](docs/plans/NUMBAT_AND_ALWAYS_ON_ROADMAP.md), and the diff --git a/agentic/README.md b/agentic/README.md index ae6aaf6a5..081399e39 100644 --- a/agentic/README.md +++ b/agentic/README.md @@ -50,11 +50,9 @@ when enabled it is advisory: `agentic/real_repo_loop.py` takes it as an optional probe, records an audit event if the probe itself raises, and appends a nudge to rejection feedback rather than blocking a run on its own. -The vendored scanners live at `agentic/vendor/unslop/` **inside** `agentic/` on -purpose. Placing them here rather than under `utils/` means the core six -(`gate.py`, `gate_ops.py`, `gate_auth.py`, `gate_memory.py`, `graph.py`, -`mcp_hybrid_server.py`) never acquire a path to them, so the I6 module-isolation -invariant holds without needing a special case. +The vendored scanners live at `agentic/vendor/unslop/` rather than under +`utils/` so the core six never acquire an import path to them and I6 holds +without a special case. --- @@ -278,8 +276,8 @@ fixed argv-list command; neither command sends a reachability probe. ## Security best practices -These are operator practices on top of the built-in gates. The gates fail closed; -this section is how to **use** them without accidentally widening blast radius. +Operator practices on top of the fail-closed gates: how to use them without +widening blast radius. ### Least privilege — open only what you need @@ -294,11 +292,10 @@ this section is how to **use** them without accidentally widening blast radius. | SQL | keep `read_only: true` / `allow_write: false` (required in v0.1) | | LAN inventory | `netconnect.enabled` plus the narrowest explicit CIDRs; only `self`/`arp` | -Prefer **local model** for implement iterations; use cloud (`--provider` + -`--confirm-online`) for a **one-shot plan** only when needed. Do not leave -`allow_shell_execution` or `allow_filesystem_write_tools` true — shell remains -unimplemented/hard-refused; FS write tools are for the retired harness surface, -not a host shell. +Prefer the local model for implement iterations and cloud (`--provider` + +`--confirm-online`) for a one-shot plan only. Leave `allow_shell_execution` +(hard-refused) and `allow_filesystem_write_tools` (retired harness surface +only) false. ### Gate hygiene @@ -322,18 +319,15 @@ not a host shell. ### Human review is the real control Automated checks (injection scan, code-shape heuristics, executor pytest/ruff) -are **necessary but not sufficient**. - -- Always inspect `real-repo-run-status` **diff + untracked new files** before - `approve`. An empty tracked diff can still hide brand-new files. -- Review **`--instruction`**, **`--plan-file`**, and PR/issue context as - untrusted: paste-from-ticket is a confused-deputy path; the CLI already - scans these, but humans still decide. -- Prefer **two-stage** workflows: cloud (or human) plan → local implement → - human approve → optional push → optional draft PR. Separate decisions so a - single flag cannot silently escalate. -- Prefer **draft PRs only** (enforced: `pr_create` always `--draft`). Promote - to ready-for-review yourself after reading the PR on GitHub. +are necessary but not sufficient. + +- Inspect `real-repo-run-status` **diff + untracked new files** before + `approve`; an empty tracked diff can still hide new files. +- Treat `--instruction`, `--plan-file`, and PR/issue context as untrusted + (paste-from-ticket is a confused-deputy path); the CLI scans them, humans decide. +- Keep plan → implement → approve → push → draft PR as separate decisions so + no single flag can escalate. `pr_create` is always `--draft`; promote it + yourself after reading the PR on GitHub. ### Secrets, credentials, and identity @@ -490,41 +484,40 @@ atomic write + sha256 versioning (soul propose/apply pattern). ### C. Real-repo workspace tools (`RepoWorkspaceTools`) -Jailed clone under `deepagent_github.workspace_root`. Containment uses -`fsconnect.pathsafe.ScopedRoots` for reads; git writes use argv-list `git` with -scrubbed env and `cwd` pinned to the clone. Every workspace Git call disables -hooks and fsmonitor and checks local config names and values against the -post-clone snapshot. The origin URL is also authenticated by a per-workspace -key and record in `workspace_root/origin-pins/`, outside each disposable clone. -`attach()` verifies that record before reopening a run; Git calls compare the -current origin with the pinned value held in process memory. A missing or -altered record refuses attach, and a changed origin refuses Git operations even -if the clone config and its sibling snapshot were changed together. The key -sits beside the record under `workspace_root`, so a same-user process that can -write there can re-sign it; the record guards clone-local and accidental edits. -A `.git` file, `.git/commondir` or a legacy `.git/remotes/origin` or `.git/branches/origin` -refuses Git, and every call pins `GIT_DIR`/`GIT_COMMON_DIR` to the clone. The record -is removed when the clone is discarded. Narrow display settings and -branch-tracking additions are allowed. -Runs created before this pin existed cannot be reopened for Git operations; -`real-repo-run-decide --decision reject` or, once decided, -`real-repo-run-discard` reclaims them without loading the pin, then the -operator can start a new run. -Git commands and config reads reap helper processes on timeout through a POSIX -process group or a Windows Job Object. A POSIX helper that creates a new session -can escape the process group. Windows releases Git from a fixed isolated waiter -only after Job assignment; assignment failure never starts Git. -Operator-owned global/system Git config remains trusted. Verification runs in -a fresh gitless mirror with no copyback. Linux requires bubblewrap filesystem, -PID and network isolation; macOS uses Seatbelt to confine writes to the mirror -and disposable scratch. Both deny writes to the authoritative clone and its -configuration snapshot. Windows refuses verification until a filesystem and -network boundary is available; the Git Job Object does not make Windows -verification safe. Linux native -CI uses Ubuntu 22.04 with stock host policy. On Ubuntu 24.04, AppArmor can deny -bubblewrap network-namespace setup (`RTM_NEWADDR: Operation not permitted`); -verification then refuses before launching the check. CyClaw does not disable -AppArmor or share the host network to bypass that refusal. +Jailed clone under `deepagent_github.workspace_root`. Reads go through +`fsconnect.pathsafe.ScopedRoots`; git writes use argv-list `git` with a scrubbed +env, `cwd` and `GIT_DIR`/`GIT_COMMON_DIR` pinned to the clone, hooks and +fsmonitor disabled, and local config checked against the post-clone snapshot +(narrow display settings and branch-tracking additions are allowed; a `.git` +file, `.git/commondir`, or legacy `.git/remotes|branches/origin` refuses Git). + +**Origin pin.** A per-workspace key and record under +`workspace_root/origin-pins/`, outside the disposable clone, authenticate the +origin URL. `attach()` verifies the record before reopening a run, and Git +calls compare the live origin with the pinned value held in memory, so a +missing or altered record refuses attach and a changed origin refuses Git even +when clone config and snapshot were edited together. The key sits beside the +record, so a same-user process that can write there can re-sign it: the pin +guards clone-local and accidental edits, not a same-user attacker. Discarding +the clone removes the record. Runs created before the pin existed cannot be +reopened for Git; `real-repo-run-decide --decision reject` or, once decided, +`real-repo-run-discard` reclaims them without loading the pin. Orphan cleanup +deletes only the generated `cyclaw-repo-clone-*/repo` shape, never workspace +metadata such as `origin-pins`. + +**Helpers and verification.** Git commands reap helper processes on timeout +through a POSIX process group (a helper that opens a new session can escape it) +or a Windows Job Object (Git starts only after assignment succeeds). +Operator-owned global/system Git config stays trusted. Verification runs in a +fresh gitless mirror with no copyback: Linux requires bubblewrap filesystem, +PID, and network isolation; macOS uses Seatbelt to confine writes to the mirror +and scratch; both deny writes to the authoritative clone and its snapshot. +Windows refuses verification until an equivalent boundary exists (the Job +Object alone is not one). Linux CI runs Ubuntu 22.04 with stock host policy; on +Ubuntu 24.04, AppArmor can deny bubblewrap's network namespace +(`RTM_NEWADDR: Operation not permitted`) and verification refuses before the +check launches — CyClaw never disables AppArmor or shares the host network to +get past it. | Tool / method | Default | What it does | |---|---|---| @@ -677,32 +670,18 @@ Cache-derived and scope-filtered only. No active scan command exists. ### K. Explicitly **not** available -- Host shell / PowerShell / bash agent tools (no Grok-Build / Kimi-style shell harness) -- Unrestricted filesystem tools over the live repo -- Autonomous push/PR without human gates -- Core request-path import of any of the above +The list under [What never to build or enable casually](#what-never-to-build-or-enable-casually): +no host shell tools, no unrestricted filesystem tools over the live repo, no +autonomous push/PR, and no core request-path import of any of the above. --- ## Main CLI commands -```bash -python -m agentic.cli status -python -m agentic.cli context --repo | --pr N | --issue N -python -m agentic.cli propose-skill ... -python -m agentic.cli apply-skill ... --reason ... --confirm -python -m agentic.cli real-repo-run-plan ... -python -m agentic.cli real-repo-run ... --checks-file ... --branch ... --commit-message ... --reason ... --confirm -python -m agentic.cli real-repo-run-status --run-id ... -python -m agentic.cli real-repo-run-decide --run-id ... --decision approve|reject -python -m agentic.cli real-repo-run-push --run-id ... -python -m agentic.cli real-repo-run-publish --run-id ... --reason ... --confirm -python -m agentic.cli real-repo-run-discard --run-id ... -python -m agentic.cli deepagent-plan --repo --instruction "..." # retired probe, no invoke -python -m agentic.cli test -``` - -Flags the block above elides, all real (`agentic/cli.py` argparse): +Subcommands: `status`, `context`, `propose-skill`, `apply-skill`, the +`real-repo-run*` family shown in [section D](#d-real-repo-loop-live-coding-pipeline), +`deepagent-plan` (retired probe, never invokes), and `test`. Flags those +examples elide, all real (`agentic/cli.py` argparse): ```bash python -m agentic.cli --config path/to/config.yaml # global; default config.yaml @@ -765,6 +744,3 @@ python -m agentic.netconnect.cli test | [`docs/agentic/SKILLS_REGISTRY_GOVERNANCE.md`](../docs/agentic/SKILLS_REGISTRY_GOVERNANCE.md) | Skills registry governance | | [`macos/README.md`](../macos/README.md) | launchd glue — the fsconnect trash-empty plist generator's runtime home | | [`docs/THREAT_MODEL.md`](../docs/THREAT_MODEL.md) | Threat-model amendments for this layer | - -Orphan cleanup only deletes the generated `cyclaw-repo-clone-*/repo` directory -shape; workspace metadata such as `origin-pins` cannot be a discard target. diff --git a/deploy/apparmor/README.md b/deploy/apparmor/README.md index 32e5fa055..e39194f0b 100644 --- a/deploy/apparmor/README.md +++ b/deploy/apparmor/README.md @@ -111,24 +111,16 @@ To load it at boot, install it to `/etc/apparmor.d/cyclaw-bwrap` first; the [setup guide](../../setup-guide.md) has those commands. **Why not Ubuntu's `bwrap-userns-restrict`.** AppArmor's extra-profiles ship a -stricter option. Its `bwrap` profile is confined rather than unconfined, and -it stacks every bwrap child under `unpriv_bwrap`, which denies capabilities -inside the namespace. CyClaw uses the one-rule profile by default because: - -- `bwrap-userns-restrict` is not userns-only. Its `bwrap` profile also allows - `capability`, `mount`, `ptrace`, `dbus`, `network`, and `rwlkm` on `/**`. - That is a much larger policy to review, and it must match the host's - AppArmor ABI (the upstream copy declares `abi `). -- Its main gain, children without capabilities, is what CyClaw's own sandbox - already gets from `--cap-drop ALL`. What it adds is protection against - *other* bwrap callers on the same host. -- CI proves the one-rule profile on `ubuntu-latest` (stock refuses, the - profile loads, the verifier passes). CI does not exercise the stricter - profile. +stricter option that confines `bwrap` and stacks every child under +`unpriv_bwrap` (no capabilities inside the namespace). CyClaw defaults to the +one-rule profile because the stricter one is not userns-only (it also allows +`capability`, `mount`, `ptrace`, `dbus`, `network`, and `rwlkm` on `/**`, a +much larger policy that must match the host ABI — upstream declares +`abi `), its main gain is what `--cap-drop ALL` already provides +(what it adds is protection against *other* bwrap callers on the host), and CI +proves only the one-rule profile on `ubuntu-latest`. **Residual risk.** With `cyclaw-bwrap` loaded, any local user can run -`/usr/bin/bwrap` to get a user namespace, so bwrap becomes an exemption from -the host's userns restriction. That's acceptable on a single-user CyClaw -workstation. On shared or multi-user hosts, load `bwrap-userns-restrict` -instead and rerun the verifier. Don't load both, because both attach to -`/usr/bin/bwrap`. +`/usr/bin/bwrap` to get a user namespace — acceptable on a single-user CyClaw +workstation. On shared hosts load `bwrap-userns-restrict` instead and rerun the +verifier; never load both, since both attach to `/usr/bin/bwrap`. diff --git a/deploy/falco/README.md b/deploy/falco/README.md index 332ede4d0..e45744954 100644 --- a/deploy/falco/README.md +++ b/deploy/falco/README.md @@ -22,15 +22,14 @@ sits in the overall posture and what it does **not** cover. | Unexpected outbound connection | Non-tool egress to anything other than loopback / local Ollama | WARNING | | Optional tool outbound connection | `git`/Git HTTPS helpers/`gh`/`rclone` external egress that needs audit correlation | NOTICE | -These map directly to the out-of-band agentic & sync write paths and the gate's -egress — the surfaces a 2026 review would (fairly) want eyes on. +These map to the out-of-band agentic and sync write paths and the gate's egress. ## Why it ships disabled Falco needs a **privileged** sidecar with host kernel access (`/proc`, `/sys/kernel/debug`, the modern eBPF probe). That privilege is itself attack -surface, so it is gated behind a Compose profile and off by default. A plain -`docker compose up` never starts it. +surface, so it sits behind a Compose profile; a plain `docker compose up` +never starts it. ## Enable it @@ -116,20 +115,15 @@ destination exceptions are the narrowest safe tuning unit. ## Requirements & caveats -- Linux host with a kernel new enough for Falco's modern eBPF probe - (≥ 5.8; the image pin is `falcosecurity/falco:0.39.2`). -- Will **not** run in environments without host kernel access (most CI, many - managed/rootless container hosts). That is expected — it is an operator-run - monitor, not a CI gate. -- Detection only. To *block* syscalls, tighten the seccomp profile instead — and - do that only after using these traces to build a verified allow-list (see - [`docs/SECCOMP_EBPF_HARDENING.md`](../../docs/SECCOMP_EBPF_HARDENING.md)). -- The monitoring profile is **privileged**. Leave it off unless you need the - eBPF tripwire and accept host-kernel access for the sidecar. -- The read-only Docker socket mount still exposes a powerful host API; `:ro` - protects the socket inode, not the API methods reachable through it. -- The `0.39.2` image is digest-pinned in `docker-compose.yml` (multi-arch - manifest-list digest resolved 2026-08-12), kept alongside the tag for human - readability. Re-pin the digest on any Falco version bump; Falco 0.44 removed - the gVisor engine, so host eBPF cannot provide complete in-sandbox syscall - visibility. +- Linux host with a kernel ≥ 5.8 for the modern eBPF probe; it will **not** + run without host kernel access (most CI, many managed or rootless hosts) — + it is an operator-run monitor, not a CI gate. +- Detection only; to *block*, tighten seccomp after these traces have built a + verified allow-list ([`docs/SECCOMP_EBPF_HARDENING.md`](../../docs/SECCOMP_EBPF_HARDENING.md)). +- The monitoring profile is **privileged**, and the read-only Docker socket + mount still exposes a powerful host API (`:ro` protects the inode, not the + API methods). +- `falcosecurity/falco:0.39.2` is digest-pinned in `docker-compose.yml` + (multi-arch manifest-list digest, resolved 2026-08-12) beside the tag. + Re-pin on any bump; Falco 0.44 removed the gVisor engine, so host eBPF + cannot give complete in-sandbox syscall visibility. diff --git a/docs/AUTHENTICATION_DESIGN.md b/docs/AUTHENTICATION_DESIGN.md index 1f58c6622..2c960636f 100644 --- a/docs/AUTHENTICATION_DESIGN.md +++ b/docs/AUTHENTICATION_DESIGN.md @@ -59,85 +59,64 @@ credential is not readable on the wire. --- -## 2. What exists today (verified, not assumed) - -| Fact | Evidence | -|---|---| -| `/query`, `/`, `/health`, `/static/*` require **no** authentication | route introspection of `gate.py` | -| `/soul/*`, `/audit/summary`, `/ops/*` require a **single shared** bearer secret | `require_api_key`, `gate.py:105` | -| The secret is compared in constant time and **fails closed** when unset | `hmac.compare_digest`, `gate.py:150`; unset `CYCLAW_API_KEY` → 401 | -| Failed auth is **already rate-limited** | `dependencies=[Depends(_enforce_rate_limit), Depends(require_api_key)]` — limiter runs *before* auth, pinned by `TestFailedAuthDoesNotBypassRateLimit` | -| The console **already sends** a bearer token on every call, including `/query` | `static/terminal.js` `authHeaders()` | -| Telegram **already sends** one to `/query` | `telegram/client.py:769`; config field documented *"Optional CyClaw API key for POST /query"* | -| No password, session, or TLS infrastructure exists | no argon2/bcrypt/passlib/itsdangerous/`SessionMiddleware`/`ssl_keyfile` anywhere | - -**Amendment (2026-08-09, Stages 1-2 landed; amended 2026-08-15, Stages 3-4 -and 6 landed, PR #940).** The table above reflects the state at proposal -time. Several rows are now stale. Password, session, AND TLS infrastructure -all exist: `utils/authn.py` (Stage 1, PR #829), the account/session/token -store (Stage 2, PR #830), and `gate._serve` passing -`ssl_certfile`/`ssl_keyfile` to `uvicorn.run` when `api.tls.enabled` is the -literal boolean true (Stage 4, PR #940 — missing or unreadable files fail -closed at boot). `/auth/login`, `/auth/logout`, and `/auth/whoami` also now -exist and are registered regardless of `auth.enabled` (returning 503 when -it is false, matching `gate_ops.py`'s convention for `/ops/*`) — so the -unauthenticated-route row above is stale too: `/auth/login` carries no -authentication either, by necessity (a caller has no credential yet to -present), gated only by the standard rate limit and the same-origin check. -Stage 3 attaches `require_session_or_token` to `POST /query` when -`auth.enabled` is the literal boolean `true` (session cookie or device -token; no CSRF on the query path). The shipped default still leaves -`/query` open. The console row is stale as well: `terminal.js` no longer -sends the typed API key on `/query` at all — once auth is on, `/query` -authenticates with the `cyclaw_session` cookie (or a named device token -for programmatic clients), and the key field serves `/soul/*` and `/ops/*` -only. The Telegram row stays accurate in form — it still sends a bearer -token to `/query` — but the token must now be a named device token, not -the shared `CYCLAW_API_KEY`. - -**Amendment (2026-08-15).** A second, independent escape hatch landed -alongside the session/RBAC system this document specifies: -`config.yaml`'s `security.api_key_optional` (default `false`). It bypasses -`require_api_key` — the shared-secret mechanism row 2 above describes — for -every route it gates (`/soul/*`/`/ops/*`/`/memory/*`/`/audit/summary`). It is orthogonal to everything in -this document: it does not touch `auth.enabled`, sessions, device tokens, or -`/auth/*`, and an operator can run with `auth.enabled: true` (this design) -and `api_key_optional: true` (bypassing the older mechanism) at the same -time without either one affecting the other. The bypass is granted only to -a request that is simultaneously from a **loopback socket peer**, carries **no reverse-proxy forwarding header** (`X-Forwarded-For`/`-Host`/`-Proto`, `X-Real-IP`, `Forwarded` — a proxy on this host makes every remote caller present a loopback peer), and is **not cross-site** (the operator's own browser is a loopback peer too, and a CORS-simple POST executes before CORS withholds the response). `gate.py`'s `_api_key_bypass_allowed` treats every one as necessary. It never widens what a remote caller can -reach; `config-guard`'s C13 check warns when it is combined with a -non-loopback `api.host`. (C13 deliberately ignores `security.allowed_hosts`: -that list filters `Host` headers and opens no listening socket.) - -One place the two systems **do** interact, and must not: §7's rule below -("a non-loopback bind is allowed when authentication is enabled and TLS is -enabled") is satisfied by `auth.enabled` + `api.tls.enabled` + a real -`/query` credential — all properties of the SESSION system. None of them -say anything about the API-key routes `api_key_optional` opens. Composed, -they would admit a LAN bind on the strength of a session while `/soul/*` -and the `/ops/*` subprocess shims sit unauthenticated. `_require_loopback_bind` -therefore refuses that route while `api_key_optional` is true; the -`CYCLAW_ALLOW_NON_LOOPBACK_BIND` env override (an explicit operator -statement of "my own auth is in front") still outranks it. - -That bind-time refusal is defence in depth, not the primary control. The -primary one is per-request: the bypass requires all of a loopback peer, no forwarding header, and a non-cross-site request, which holds -regardless of how the process was launched — including the container's +## 2. Starting point and current state + +At proposal time (2026-08-08) `/query`, `/`, `/health`, and `/static/*` took +no credential; `/soul/*`, `/audit/summary`, and `/ops/*` took one shared +bearer secret (`require_api_key`, constant-time compare, failing closed when +`CYCLAW_API_KEY` was unset, rate-limited *before* auth — pinned by +`TestFailedAuthDoesNotBypassRateLimit`); both real clients (the console and +Telegram) already sent `Authorization: Bearer`; and no password, session, or +TLS infrastructure existed. Two consequences drove the design: **the transport +was already built**, so requiring a credential on `/query` meant turning on a +check the clients already satisfied, and **what was missing was identity, not +authentication** — a shared secret cannot say *which* machine asked or revoke +one device without rotating every other, which is the gap that matters on a +LAN. + +**Current state (Stages 1–6 landed, PRs #829/#830/#940).** Password, session, +and TLS infrastructure exist: `utils/authn.py`, the account/session/token +store, and `gate._serve` passing `ssl_certfile`/`ssl_keyfile` to +`uvicorn.run` when `api.tls.enabled` is the literal boolean `true` (missing or +unreadable files fail closed at boot). Every `/auth/*` route is registered +regardless of `auth.enabled` and answers 503 while it is false (the `/ops/*` +convention); `/auth/login` itself carries no credential, by necessity, behind +the rate limit and the same-origin check. Stage 3 attaches +`require_session_or_token` to `POST /query` when `auth.enabled` is the literal +`true` (session cookie or named device token; no CSRF on the query path); the +shipped default leaves `/query` open. `terminal.js` no longer sends the typed +API key on `/query` — with auth on, `/query` authenticates with the +`cyclaw_session` cookie and the key field serves `/soul/*` and `/ops/*` only; +Telegram still sends a bearer token, which must now be a named device token, +not the shared `CYCLAW_API_KEY`. + +**The orthogonal escape hatch (2026-08-15).** `security.api_key_optional` +(ships `false`) bypasses `require_api_key` on every route it gates +(`/soul/*`, `/ops/*`, `/memory/*`, `/audit/summary`) and touches nothing in +this document — not `auth.enabled`, sessions, device tokens, or `/auth/*`; the +two can be on at once without affecting each other. The bypass is granted only +to a request that is simultaneously from a **loopback socket peer**, carries +**no reverse-proxy forwarding header** (`X-Forwarded-For`/`-Host`/`-Proto`, +`X-Real-IP`, `Forwarded` — a proxy on this host makes every remote caller a +loopback peer), and is **not cross-site** (the operator's own browser is a +loopback peer too, and a CORS-simple POST executes before CORS withholds the +response); `gate.py`'s `_api_key_bypass_allowed` treats each as necessary. +`config-guard`'s C13 warns when it is combined with a non-loopback `api.host` +(C13 ignores `security.allowed_hosts`, which filters `Host` headers and opens +no socket). + +The one place the two systems interact, and must not compose: §7's bind rule +is satisfied by `auth.enabled` + `api.tls.enabled` + a real `/query` +credential, all properties of the SESSION system that say nothing about the +API-key routes `api_key_optional` opens. Composed, they would admit a LAN bind +on the strength of a session while `/soul/*` and the `/ops/*` shims sat +unauthenticated, so `_require_loopback_bind` refuses that route while +`api_key_optional` is true; `CYCLAW_ALLOW_NON_LOOPBACK_BIND` ("my own auth is +in front") still outranks it. That bind-time refusal is defence in depth; the +primary control is the per-request three-condition check, which holds however +the process was launched — including the container's `uvicorn gate:app --host 0.0.0.0`, which reaches no bind guard at all. -Two consequences worth stating plainly. - -**The transport is already built.** Both real clients speak `Authorization: -Bearer`. Requiring credentials on `/query` is not a new protocol — it is turning -on a check the clients already satisfy. - -**What is missing is identity, not authentication.** A single shared secret *is* -authentication in the "something you know" sense. What it cannot do is say -*which* machine asked, or let one device be revoked without rotating the -credential for every other device. For a LAN with several machines, that is the -gap that matters, and it is why this design is per-user rather than a wider -allow-list on the existing key. - --- ## 3. What changes in the threat model @@ -346,30 +325,21 @@ implemented in `gate.py`'s `_require_loopback_bind`/`_auth_and_tls_enabled` is > enabled **and** `/query` demonstrably enforces a credential; otherwise > refuse. -The third condition is load-bearing, not optional. The two config flags are a -statement of *intent*; an operator who sets `auth.enabled: true` reasonably -believes authentication is on. `gate.py`'s -`_request_path_enforcement_active()` probes the live FastAPI app for the -`/query` credential dependency rather than trusting the flag. Stage 3 -(PR #940) attaches that dependency when `auth.enabled` is the literal -boolean `true`, so the path past loopback opens by itself once both flags -are on — no further edit to the guard is required. The shipped default -(auth off) still leaves the probe false. This is a strictly better gate -than the env-var-only guard: it permits the operator's goal (LAN + TLS + -auth) and refuses the genuinely unsafe case (LAN + no auth, LAN + auth -claimed but not enforced). With Stage 4 landed in the same PR, the guard's -TLS half is now real too: `gate._serve` passes -`ssl_certfile`/`ssl_keyfile` to `uvicorn.run` when `api.tls.enabled` is the -literal boolean true, and missing or unreadable cert files refuse to boot -rather than falling back to plaintext. The residual gap is narrower than -the one this section previously documented: the Docker `CMD` and a hand-run -`uvicorn gate:app` bypass `_serve` entirely, so the guard's proof covers -the documented entry points (`python gate.py`, `cyclaw-server`) only — see -`docs/THREAT_MODEL.md`'s ninth amendment §5 for the precise scope. -`CYCLAW_ALLOW_NON_LOOPBACK_BIND` is retained only as a -deliberate escape hatch for someone fronting CyClaw with their own reverse -proxy. See `docs/THREAT_MODEL.md`'s ninth amendment for the fully verified, -currently-accurate statement of this rule. +The third condition is load-bearing: the two flags state *intent*, and +`gate.py`'s `_request_path_enforcement_active()` probes the live FastAPI app +for the `/query` credential dependency rather than trusting them. Stage 3 +attaches that dependency when `auth.enabled` is the literal boolean `true`, so +the path past loopback opens by itself once both flags are on, and the shipped +default (auth off) leaves the probe false. The guard therefore permits the +operator's goal (LAN + TLS + auth) and refuses LAN + no auth and LAN + auth +claimed but not enforced. Its TLS half is real too: `gate._serve` passes the +cert files to `uvicorn.run`, and missing or unreadable files refuse to boot +rather than fall back to plaintext. Residual gap: the Docker `CMD` and a +hand-run `uvicorn gate:app` bypass `_serve`, so the proof covers the +documented entry points (`python gate.py`, `cyclaw-server`) only — see +`docs/THREAT_MODEL.md`'s ninth amendment §5. `CYCLAW_ALLOW_NON_LOOPBACK_BIND` +remains only as the escape hatch for an operator fronting CyClaw with their own +reverse proxy. --- @@ -420,21 +390,15 @@ Each stage is independently reviewable and leaves the tree working. 3. **Should `/health` stay open?** **Resolved: yes** (§6). It reports status, mode, and timeouts — no corpus content — and stays unauthenticated even with `auth.enabled: true`. -4. **Bootstrap.** Proposed: `cyclaw-user add` refuses to run over HTTP and must - be run locally, so there is never a window where a default credential exists. - No default account is ever created. - **Resolved (2026-08-08, amended 2026-08-09):** first enable with an empty - users table creates `admin` seeded with the hash of a random secret that is - discarded inside the same call — never returned, printed, logged, or stored - in plaintext — so the account exists but cannot be logged into until - `cyclaw-user passwd admin` sets a real password locally via `getpass`. - The first iteration of this decision printed a generated one-time password - to the server console instead; CodeQL flagged it (alert #1057), and the - objection is substantive, not pedantic: a service's stdout is persisted by - systemd's journal, Docker's log driver, and any log shipper, so - "printed once" was never really once. The discard design keeps this - decision's actual requirement — no fixed default credential, ever — while - removing every output channel a credential could reach. +4. **Bootstrap.** **Resolved (2026-08-08, amended 2026-08-09):** first enable + with an empty users table creates `admin` seeded with the hash of a random + secret discarded inside the same call — never returned, printed, logged, or + stored — so the account exists but cannot be logged into until + `cyclaw-user passwd admin` (local `getpass`) or the loopback-only console + panel in §4.6 sets a real password. The first iteration printed a one-time + password to the server console; CodeQL #1057 flagged it, and rightly: a + service's stdout persists in journald, Docker's log driver, and any log + shipper, so "printed once" never was. No default credential ever exists. --- diff --git a/docs/DOCKER.md b/docs/DOCKER.md index bb2eb4236..0a1619cfa 100644 --- a/docs/DOCKER.md +++ b/docs/DOCKER.md @@ -6,12 +6,10 @@ This document is the operator guide for running CyClaw from the published contai image. It does **not** replace [`setup-guide.md`](../setup-guide.md) (native install) or [`docs/THREAT_MODEL.md`](./THREAT_MODEL.md) (threat scope). -The base image includes `nemoguardrails==0.24.0`. Existing images must be -rebuilt from the updated dependency manifests, published through the GHCR -workflow, and pulled before recreating containers. The Dockerfile runs -`pip check` and `python -m guardrails.verify_install` as the runtime user with -build network access disabled. A failed engine or allow/refuse check stops -the build before publication. Existing runtime fallback behavior is unchanged. +The image includes `nemoguardrails==0.24.0`; the Dockerfile runs `pip check` +and `python -m guardrails.verify_install` as the runtime user with build +networking disabled, so a failed engine or allow/refuse check stops the build +before publication. Runtime fallback behavior is unchanged. ## What is published @@ -33,23 +31,14 @@ dedicated `linux/arm64` image is verified against the torch pin. ## What is *not* in the image (by design) -`.dockerignore` + the multi-stage `Dockerfile` keep private / regenerable state out -of layers: - -- `data/` corpus vectors, runtime caches -- `index/` ChromaDB + BM25 state -- `logs/`, `checkpoints/` -- `.env`, keys, `*.pem` -- tests, docs, `.git`, `.github` - -The image includes the repository default `config.yaml` through `COPY . .`. -Compose overrides it with your read-only `./config.yaml` bind mount. Build only -from a checkout whose config is safe to distribute; keep production settings in -the operator mount. - -The image is the **gate + graph + retrieval runtime** only. `data/` (corpus, -soul.md, agentic registry) is not baked into layers; compose bind-mounts `./data` -at runtime. Treat production `soul.md` as operator-owned. +`.dockerignore` + the multi-stage `Dockerfile` keep private and regenerable +state out of layers: `data/` (corpus, soul.md, agentic registry), `index/`, +`logs/`, `checkpoints/`, `.env`/keys/`*.pem`, tests, docs, `.git`, `.github`. +The image is the gate + graph + retrieval runtime only; compose bind-mounts +`./data` at runtime, and production `soul.md` is operator-owned. The +repository default `config.yaml` does ride along via `COPY . .` (compose +overrides it with your read-only `./config.yaml` mount), so build only from a +checkout whose config is safe to distribute. ## Security posture (must preserve) @@ -101,33 +90,24 @@ required for GHCR pull/run. ### Prepare host storage -Before either Compose or `docker run`, create the bind directories next to -`docker-compose.yml` and ensure container UID/GID `1000:1000` can read and write -them, including existing files the application updates. Image-layer ownership -is hidden by bind mounts. Compose refuses missing writable sources instead of -letting rootful Docker create root-owned directories that the app cannot write. +Before Compose or `docker run`, create the bind directories next to +`docker-compose.yml` and make sure container UID/GID `1000:1000` can read and +write them, existing files included (bind mounts hide image-layer ownership; +compose refuses a missing writable source rather than letting rootful Docker +create a root-owned directory the app cannot write): ```bash mkdir -p data index logs checkpoints +# new, empty deployment directory on rootful Linux only: +sudo chown 1000:1000 data index logs checkpoints && sudo chmod 0750 data index logs checkpoints ``` -For a **new, empty deployment directory on rootful Linux**, an administrator can -set ownership on just those directories before adding data: - -```bash -sudo chown 1000:1000 data index logs checkpoints -sudo chmod 0750 data index logs checkpoints -``` - -For existing state, review its ownership and permissions first. Grant UID 1000 -the required access using your host's ownership or ACL policy; do not recursively -change an existing corpus or deployment tree blindly. `config.yaml` must be a -regular file readable by the container. Docker Desktop and rootless/user-namespace -Docker translate host identities differently; verify access under that runtime -instead of assuming the Linux ownership command applies. - -After pulling or building the image, this probe exercises the configured mounts -as the service user without starting the gateway or calling any model: +For existing state, grant UID 1000 access through your host's ownership or ACL +policy instead of recursively changing a corpus blindly; `config.yaml` must be +a regular file readable by the container. Docker Desktop and rootless or +user-namespace Docker translate identities differently — verify under that +runtime. This probe then exercises the mounts as the service user without +starting the gateway or calling a model: ```bash docker compose run --rm --no-deps --entrypoint python cyclaw -c \ diff --git a/docs/GITHOOKS.md b/docs/GITHOOKS.md index 9bd9994a2..62ac0b473 100644 --- a/docs/GITHOOKS.md +++ b/docs/GITHOOKS.md @@ -17,20 +17,15 @@ bash scripts/ensure-githooks.sh # idempotent; the one setup line every a git config core.hooksPath # -> .githooks ``` -Put that one line in every agent's setup step. The Claude Code SessionStart -hook (`.claude/hooks/session-start-sync-check.sh`), `copilot-setup-steps.yml`, -and `.codex/Codex_instructions.md` already call it; add it to any container -entrypoint or other setup script. A fresh cloud clone has no hooks until it -runs. `ensure-githooks.sh` is never fatal and sets `core.hooksPath` only when -no scope has set it, so an operator's own hook manager is left alone. - -Manual alternative: `bash scripts/install-githooks.sh` (or -`git config core.hooksPath .githooks`) sets the path in this clone -unconditionally, even over a value `ensure-githooks.sh` would have kept. - -Optional tools, used when present: `gitleaks` (entropy-based detection of -unprefixed secrets) and `exiftool` (image and document metadata). Without them -the gate still runs its own checks and says what it skipped. +Put that line in every agent's setup step; the Claude Code SessionStart hook +(`.claude/hooks/session-start-sync-check.sh`), `copilot-setup-steps.yml`, and +`.codex/Codex_instructions.md` already call it. A fresh cloud clone has no +hooks until it runs. `ensure-githooks.sh` is never fatal and sets +`core.hooksPath` only when no scope has set it, so an operator's own hook +manager is left alone; `bash scripts/install-githooks.sh` sets it +unconditionally. Optional tools used when present: `gitleaks` (entropy-based +detection of unprefixed secrets) and `exiftool` (image and document metadata); +without them the gate says what it skipped. ## Files @@ -112,20 +107,18 @@ blocking checks; the reminders were already shown at commit time. | `ALLOW_MAIN_PUSH=1` | a direct push to a protected branch | | `ALLOW_FORCE_WITH_LEASE=true` | a non-fast-forward push | -An agent never sets these on its own. When the gate refuses, the agent stops, -shows the operator the message and the diff, and waits. The operator reruns -the command with the variable after reading the change, for example -`HOOK_OPERATOR_ACK=1 git commit`. The variable is visible in the agent's -transcript, so a self-granted override is detectable after the fact. - -Findings about secrets, blocked files, identifiers, home paths and invisible -characters have no override variable. Fix the content. For a confirmed -synthetic fixture, put `gitleaks:allow` on the line (also honored by CI's -gitleaks) or extend `SEC_ALLOW_REGEX` / `SEC_BLOCKED_EXCEPT` in -`security.conf`, which is itself a protected path. `SEC_BLOCKED_EXCEPT` frees a -path from every blocked glob. `SEC_BLOCKED_KEY_EXCEPT` (ships `*.pub`) frees a -path only from the SSH key-name globs (`SEC_KEY_NAME_GLOBS`), so `id_rsa.pub` -passes while `data/auth/backup.pub` is still blocked by its path-anchored rule. +An agent never sets these on its own: when the gate refuses, it stops, shows +the operator the message and the diff, and waits for the operator to rerun +with the variable (`HOOK_OPERATOR_ACK=1 git commit`). The variable is visible +in the transcript, so a self-granted override is detectable after the fact. + +Secrets, blocked files, identifiers, home paths, and invisible characters have +no override: fix the content. For a confirmed synthetic fixture, put +`gitleaks:allow` on the line (CI's gitleaks honors it too) or extend +`SEC_ALLOW_REGEX` / `SEC_BLOCKED_EXCEPT` in `security.conf`, itself a protected +path. `SEC_BLOCKED_EXCEPT` frees a path from every blocked glob; +`SEC_BLOCKED_KEY_EXCEPT` (ships `*.pub`) frees it only from the SSH key-name +globs, so `id_rsa.pub` passes while `data/auth/backup.pub` stays blocked. ## Privacy setup @@ -155,11 +148,9 @@ The gate sees text, not intent. These rules stay in `AGENTS.md` and review: ## Known limits -- `git commit --no-verify` skips pre-commit; pre-push still scans those commits. `git push --no-verify` skips both; CI is the backstop. -- A clone that never set `core.hooksPath` has no gate. -- An agent can set an override variable itself. The gate makes that a visible rule violation, not an impossible one. -- Regex detection misses secrets with no known prefix and low entropy, and flags look-alike fixtures. -- It does not scan inside archives, images or binaries; it refuses large new files and reminds on media instead. +- `git commit --no-verify` skips pre-commit (pre-push still scans those commits); `git push --no-verify` skips both, and a clone that never set `core.hooksPath` has no gate. CI is the backstop. +- An agent can set an override variable itself; the gate makes that a visible rule violation, not an impossible one. +- Regex detection misses unprefixed low-entropy secrets and flags look-alike fixtures; archives, images, and binaries are not scanned inside (large new files are refused, media gets a reminder). ## Reuse in another repository diff --git a/docs/NeMo/README.md b/docs/NeMo/README.md index 2fc4f804e..090619118 100644 --- a/docs/NeMo/README.md +++ b/docs/NeMo/README.md @@ -24,11 +24,9 @@ grant or deny access. Guardrails can deny a request or replace an answer. They cannot select an online route, expand a tool registry, or override a deterministic denial. -The request path reaches this package through `utils/guardrail_bridge.py`. -The core six modules never import `guardrails` directly. They are `gate.py`, -`gate_ops.py`, `gate_auth.py`, `gate_memory.py`, `graph.py`, and -`mcp_hybrid_server.py`. The graph remains 12 nodes, with no `safe_generate` -call on its generation path. +The request path reaches this package only through `utils/guardrail_bridge.py` +(I6: the core six never import `guardrails`). The graph remains 12 nodes, with +no `safe_generate` call on its generation path. The bridge activates only for literal boolean `True`. Startup rejects non-booleans such as `"true"`, `"false"`, and `1` through @@ -65,22 +63,15 @@ billed an answer. All graph paths still converge on `audit_logger`. ## Other entry points -`safe_generate` is a separate diagnostic path used by -`python -m guardrails.cli check`. It applies offline input checks and can -call NeMo `generate_async` when the engine is available. That call can invoke -the model for intent generation. It is not the gateway broker and must not -be wired into `/query`, where it would duplicate generation. - -`agentic/executor` has its own platform sandbox and approval contracts. It -has no NeMo call. The former harness console routes were removed in PR #1367. -MCP `tools/call` remains unwrapped. - -| Module | Role | -|---|---| -| `guardrails.broker.GuardrailBroker` | NeMo `check()` and deterministic fallback around existing generation. Never grants external access | - -`python -m guardrails.call_inventory` fails closed on unregistered -`ChatOpenAI`, `ChatXAI`, `ChatAnthropic`, or `generate_async` call sites. +`safe_generate` is the diagnostic path behind `python -m guardrails.cli check`: +offline input checks plus, when the engine is available, a NeMo +`generate_async` call that can invoke the model. It is not the gateway broker +(`guardrails.broker.GuardrailBroker`, which never grants external access) and +must not be wired into `/query`, where it would duplicate generation. +`agentic/executor` has its own sandbox and approval contracts and no NeMo call; +MCP `tools/call` remains unwrapped. `python -m guardrails.call_inventory` fails +closed on unregistered `ChatOpenAI`, `ChatXAI`, `ChatAnthropic`, or +`generate_async` call sites. ## Rail coverage @@ -101,16 +92,13 @@ enforced offline and are absent from the active NeMo flow list. ### Model-assisted rails declined Track B phase 4 declines NVIDIA's model-assisted `self_check_input`, -`self_check_output`, and `self_check_facts` rails. They are inactive and add -no calls to `/query`. The retained input and facts prompt templates in -`guardrails/config/config.yml` are inert historical material. Activating -model-assisted rails requires a separate policy, privacy, latency, and -model-call review. - -The active `check cyclaw facts` flow uses the same deterministic grounding -action as `check grounding`. Its name does not mean that NVIDIA -`self_check_facts` runs. This implementation provides neither a general -jailbreak classifier nor claim-level fact verification. +`self_check_output`, and `self_check_facts`; they are inactive, add no calls to +`/query`, and their retained prompt templates in `guardrails/config/config.yml` +are inert. Activating them needs a separate policy, privacy, latency, and +model-call review. The active `check cyclaw facts` flow runs the same +deterministic grounding action as `check grounding` — not NVIDIA's +`self_check_facts` — so this layer provides neither a jailbreak classifier nor +claim-level fact verification. ## Grounding @@ -194,13 +182,11 @@ mismatch, a missing model or an unreachable Ollama exits `2`. ## Verification commands Run `python -m pip check` and `python -m guardrails.verify_install` after the -normal platform install. The strict smoke constructs the production engine -and requires exact input and output verdicts from the shipped rules with the -offline loader flags set. Fallback checks cannot satisfy it. CI runs this on Linux, -Windows, macOS, Conda, the installed wheel outside the checkout, and Docker. -Docker runs it with networking disabled, including in the final build stage -before GHCR can publish an image. - +platform install. The strict smoke constructs the production engine and +requires exact input and output verdicts from the shipped rules with the +offline loader flags set; fallback checks cannot satisfy it. CI runs it on +Linux, Windows, macOS, Conda, the installed wheel outside the checkout, and +Docker (networking disabled, including the final build stage before GHCR publish). The real-engine lane in [`.github/workflows/nemo-guardrails.yml`](../../.github/workflows/nemo-guardrails.yml) diff --git a/docs/README.md b/docs/README.md index a50c2b9c0..86e83c4ca 100644 --- a/docs/README.md +++ b/docs/README.md @@ -24,7 +24,7 @@ grepping. | `channels/TELEGRAM_DESIGN.md` | Telegram channel architecture, T1–T4 phase gates. | | `channels/OPENTWEET_DESIGN.md` | OpenTweet X channel: weekly OOB poster, draft-default, I6 isolation. | | `agentic/AGENTIC_README.md`, `agentic/SKILLS_REGISTRY_GOVERNANCE.md`, `agentic/GITHUB_WRITE_ENABLEMENT.md` | Agentic layer governance (binding). | -| `AUTHENTICATION_DESIGN.md` | Per-user auth: all six stages landed — `utils/authn.py` (1), sessions + `/auth/*` (2), credential on `/query` when `auth.enabled` (3), TLS via `gate._serve` + `cyclaw-gen-cert` (4), the re-keyed bind guard (5), and RBAC roles `admin`/`operator`/`audit` with HTTP user admin (6). | +| `AUTHENTICATION_DESIGN.md` | Per-user auth, all six stages landed: primitives, sessions + `/auth/*`, the credential on `/query` when `auth.enabled`, TLS (`cyclaw-gen-cert`), the re-keyed bind guard, and RBAC roles `admin`/`operator`/`audit`. | | `memory/` | Optional memory subsystem plan and README. | | [`../spend/README.md`](../spend/README.md) | Online-LLM token ledger: what `logs/spend.jsonl` records, why dollars are derived at read time, and how `cyclaw-metrics` reports them. | | `EVALS.md` | Benchmarks and evals: the four planes (PR retrieval gate, operator dogfood, Anthropic judge, local judge + calibration), what has been measured with dates and commits, and what has not. Dated run records go to `audits/`. | diff --git a/docs/SECCOMP_EBPF_HARDENING.md b/docs/SECCOMP_EBPF_HARDENING.md index bf881edde..b18830ca0 100644 --- a/docs/SECCOMP_EBPF_HARDENING.md +++ b/docs/SECCOMP_EBPF_HARDENING.md @@ -32,14 +32,12 @@ Falco alert arrives after the syscall. `cap_drop: ALL` should deny privileged raw/packet sockets, but that is a negative test to retain, not a reason to claim kernel isolation. -The former repository seccomp profiles were not a working Stage 3. The profile -wired to the gate omitted normal Uvicorn/Python syscalls including `listen`, -`accept*`, `exit_group`, clocks, and thread setup; the supposed gate floor was -x86-64-only; and the legacy rclone file contained invalid syscall names. They -have been removed. Compose now forces `seccomp:builtin`, so a daemon configured -with another default or `unconfined` cannot silently weaken the service. Docker -describes its builtin as a moderately protective compatibility baseline and -recommends not replacing it casually. +The former repository seccomp profiles were not a working Stage 3 and have been +removed ([`deploy/seccomp/README.md`](../deploy/seccomp/README.md) records +why). Compose now forces `seccomp:builtin`, so a daemon configured with another +default or `unconfined` cannot silently weaken the service; Docker describes +its builtin as a moderately protective compatibility baseline not to be +replaced casually. ## 2. Stage 3 - tight gate-specific seccomp @@ -501,13 +499,13 @@ plane. The actual enforcement bridge is: executor container. 5. Keep Falco alerts as regression evidence after enforcement. -Immediate repository tuning adds `/app/index` to expected writes and `git` to -the monolithic optional-tool exec set. Optional `git`/`gh`/`rclone` network -connections are logged separately at `NOTICE`; they are not broadly allowed by -SaaS IP range. Python egress outside the configured local model endpoint stays -`WARNING` because the same process can represent either an explicit cloud-model -choice or hostile test code. Correlate it with CyClaw's audit event before -adding a narrow actor + action + destination exception. +The shipped Falco rules already reflect this: `/app/index` is an expected +write, `git` is in the optional-tool exec set, optional `git`/`gh`/`rclone` +egress logs at `NOTICE` rather than being allowed by SaaS IP range, and Python +egress outside the configured local model endpoint stays `WARNING` because the +same process can be an explicit cloud-model choice or hostile test code — +correlate with the audit event before adding a narrow actor + action + +destination exception. When roles are split, label and maintain separate Falco rules for `gate|agentic|sync|telegram|executor`: @@ -536,18 +534,12 @@ sudo ig run ghcr.io/inspektor-gadget/gadget/trace_dns:latest \ Pin gadget image digests before operational use. Custom bpftrace is YAGNI until these tools fail to answer a specific question. -## 6. Final deliverable package +## 6. Operator enablement checklist -This hardening increment contains: - -- Updated Stage 3-5 and residual-risk text in `docs/THREAT_MODEL.md`. -- This canonical trace/evaluation procedure. -- `deploy/seccomp/README.md`; unsafe, untraced JSON profiles removed. -- `deploy/apparmor/cyclaw-gate` and an opt-in Compose overlay. -- Falco expected-path/process tuning and operator guidance. -- A static regression test for the deploy contract. - -### Operator enablement checklist +The hardening increment this document belongs to shipped the Stage 3–5 text in +`docs/THREAT_MODEL.md`, this procedure, `deploy/seccomp/README.md` (untraced +JSON profiles removed), `deploy/apparmor/cyclaw-gate` with its opt-in Compose +overlay, Falco tuning, and a static deploy-contract regression test. Fail closed. Do not mark a stage complete from static parsing alone. diff --git a/docs/SYNC_README.md b/docs/SYNC_README.md index 702c0216d..b136e814d 100644 --- a/docs/SYNC_README.md +++ b/docs/SYNC_README.md @@ -1,22 +1,16 @@ # CyClaw Sync — Rclone Dropbox Corpus Integration -**Module:** `sync/` (Dropbox corpus sync, v1.4.0 cycle) -**Status:** Out-of-band, audit-logged, **zero new Python dependencies**. Not a graph node; core modules never import `sync/`. - -CyClaw's sync module mirrors a Dropbox folder into your local `data/corpus/` -without weakening any of CyClaw's security invariants. It is a thin Python -wrapper around the `rclone` binary and emits per-file audit events into the -same `logs/audit.jsonl` the gateway uses. It runs as a **separate process**, -registered via `sync/scheduler.py`'s own `cron` (Linux/macOS, the default) or -`launchd` (macOS, opt-in — see "Scheduling" below) backends, or manually via a -systemd `--user` timer if you prefer that over cron (`sync/scheduler.py` does -not generate a systemd unit itself). - -> **Sync is NOT a graph node and is never imported by `gate.py` / `graph.py` / -> `mcp_hybrid_server.py`.** Primary invocation is `python -m sync.cli`. -> Optionally, authenticated loopback `POST /ops/sync` (`gate_ops.py`) reaches -> the same CLI only through `utils/ops_runner.py` as an argv-list subprocess — -> still no in-process `import sync`, no public listener, and no new Python dependency. +**Module:** `sync/` — out-of-band, audit-logged, **zero new Python +dependencies**. Not a graph node; the core modules never import it. + +The sync module mirrors a Dropbox folder into `data/corpus/` as a thin wrapper +around the `rclone` binary, emitting per-file audit events into the same +`logs/audit.jsonl` the gateway uses. It runs as a **separate process** — +`python -m sync.cli`, scheduled through `sync/scheduler.py`'s `cron` (default) +or opt-in `launchd` backend, or a systemd `--user` timer you write yourself — +and the optional authenticated loopback `POST /ops/sync` (`gate_ops.py`) +reaches the same CLI only through `utils/ops_runner.py`'s argv-list +subprocess: no in-process `import sync`, no public listener. --- @@ -66,19 +60,12 @@ sync root is confined to `data/corpus` and the soul filter rule is unconditional ### Why these invariants hold -1. **RAG-First.** Sync only changes files on disk that are later consumed at - index time. The retrieve node remains the graph entry point. `graph.py` is - not modified. -2. **Topology = Policy.** Sync adds no StateGraph node and no edge. Corpus - mutation stays *outside* the graph, exactly as soul evolution does. -3. **Triple-Gated External.** Dropbox is contacted only by the `rclone` - subprocess you explicitly invoke — never triggered by a user query, never an - inbound surface. -4. **Audit Convergence.** Sync events terminate in the same `audit.jsonl` with - the same PII redaction and timestamp handling as the gateway. -5. **Soul Governance.** `data/personality/**` and `*.db*` are excluded by - default; a synced file can never overwrite `soul.md` / `cyclaw_soul.db` and - bypass the `apply_evolution()` injection scan behind `POST /soul/apply`. +Sync only changes files on disk that the indexer consumes later, so `retrieve` +stays the graph entry (I1) and no node or edge is added (I2). Dropbox is +contacted only by the `rclone` subprocess you invoke, never by a query (I3). +Events land in the same `audit.jsonl` with the same redaction (I4), and +`data/personality/**` plus `*.db*` are excluded so a synced file can never +reach `soul.md` or `cyclaw_soul.db` around the `POST /soul/apply` scan (I5). --- @@ -182,22 +169,14 @@ python -m sync.cli test pytest tests/test_sync_*.py ``` -With rclone absent, `python -m sync.cli setup` exits **3** with -`RCLONE_NOT_INSTALLED` — that clean failure is itself the not-installed path -working as intended. The other two subcommands differ, and measured rather than -assumed: `test` exits **2** (the self-test reports "7/8 passed" with the rclone -point failed — it ran and failed, which is exit 2's meaning), and `status` -exits **0**, printing `[ERR] rclone binary not found on PATH` and continuing, -because `status` is an informational report rather than a health gate. Do not -read `status` exiting 0 as "sync is healthy"; read the report. - -An unreadable or malformed `config.yaml` exits **3**. That is worth stating -because it used to exit **1**, which this table already spends on the -max-delete/max-transfer safety fuse — so `POST /ops/sync` labelled a missing -config file as `safety_abort` and the console announced a tripped deletion fuse. -`load_sync_config` now converts those load failures to `SYNC_CONFIG_INVALID`, -and `sync/cli.py::main` has a dispatch-point handler so nothing else can escape -the table either. +With rclone absent, `setup` exits **3** (`RCLONE_NOT_INSTALLED`), `test` exits +**2** (the self-test ran and one point failed), and `status` exits **0** while +printing `[ERR] rclone binary not found on PATH` — it is an informational +report, not a health gate, so read the report rather than the exit code. An +unreadable or malformed `config.yaml` exits **3** (`SYNC_CONFIG_INVALID`); it +once exited **1**, the safety-fuse code, so `POST /ops/sync` mislabelled a +missing config as `safety_abort`. `sync/cli.py::main` now has a dispatch-point +handler so nothing escapes the exit-code table below. --- @@ -272,43 +251,28 @@ uses `sync.schedule_weekday`, `"monthly"` uses `sync.schedule_day`). | macOS only | `launchd` (opt-in: `scheduler_backend: "launchd"`) | Generates `~/Library/LaunchAgents/com.cgfixit.cyclaw.sync.plist` via `plistlib` from real, resolved install paths — no `REPLACE_*` placeholders to hand-edit. Supports `daily`/`weekly`/`monthly` via `StartCalendarInterval`. **`install()` never calls `launchctl load`/`bootstrap`** — it prints the exact `launchctl bootstrap gui/ ` command; loading a persistent background agent is always an explicit operator step. `remove()` best-effort `launchctl bootout`s before deleting the file. No secret/token is embedded; `sync.scheduled` retrieves the config password at execution time, and rclone owns its OAuth state. | `Label` `com.cgfixit.cyclaw.sync` | | Windows | (only option) | `schtasks /Create` with `/SC DAILY`, `/SC WEEKLY /D `, or `/SC MONTHLY /D `, plus `/ST HH:MM /RL LIMITED /F`. | task name `CyClaw Dropbox Sync` | -`cron`/`schtasks` scheduled commands `cd` into the repo root (so `config.yaml` -resolves) and run `python -m sync.cli sync`, propagating `--config` when the -loaded config's identity differs from the default (e.g. `--config -/alt/cfg.yaml` at setup time); the `launchd` plist does the same via -`WorkingDirectory` + `ProgramArguments` instead. Every path is safely -escaped: the POSIX cron line is `shlex.quote`d per token, the Windows `.bat` -launcher quotes and `%`-doubles each path, and the launchd `ProgramArguments` -array is exec'd directly by the kernel with no shell involved at all (so none -of the cron/`.bat` escaping applies or is needed there) — a repo or config -path containing spaces or shell/batch metacharacters (`$()`, backticks, -`%VAR%`) is passed through literally on every backend. On Windows the task -points at the generated `cyclaw_sync.bat` launcher (written next to the rclone -logs) rather than an inline `cmd /c` string — this avoids the quote fragility of -passing a full command through `schtasks /TR`. - -> **Overlap protection:** `run_sync` holds a single-instance OS-backed lock -> on `sync.lock` under the rclone log dir, storing PID + start timestamp while -> held. A scheduled run and a manual run therefore cannot drive rclone -> concurrently — the second exits with `SYNC_RUNTIME_ERROR` rather than racing. The -> descriptor remains open through the optional post-sync check, and the OS -> releases ownership automatically on a clean exit or crash. The empty lock file -> remains for reuse; its existence does not mean a sync is active and it must not -> be manually deleted while a run is in progress. This lock is shared by every -> scheduler backend (cron, launchd, schtasks) since it lives inside `run_sync` -> itself, not in the scheduler transport. +Scheduled commands `cd` into the repo root (so `config.yaml` resolves) and run +`python -m sync.cli sync`, propagating `--config` when the loaded config is not +the default; the launchd plist does the same via `WorkingDirectory` + +`ProgramArguments`. Every path is escaped for its transport — the cron line is +`shlex.quote`d per token, the Windows `.bat` launcher (written beside the rclone +logs, instead of a fragile inline `schtasks /TR` string) quotes and +`%`-doubles each path, and launchd execs the argument array with no shell — so +spaces and `$()`, backticks, or `%VAR%` pass through literally everywhere. + +> **Overlap protection:** `run_sync` holds a single-instance OS-backed lock on +> `sync.lock` under the rclone log dir (PID + start timestamp while held), so a +> scheduled and a manual run cannot drive rclone concurrently; the second exits +> `SYNC_RUNTIME_ERROR`. The OS releases the lock on clean exit or crash; the +> empty file remains for reuse and does not mean a sync is active, so never +> delete it mid-run. The lock lives in `run_sync`, so every backend shares it. > -> **More robust Linux option:** a systemd `--user` `Type=oneshot` service driven -> by a timer unit additionally gives journald logging and `Persistent=true` -> catch-up after downtime. Cron remains the implemented portable baseline (it -> works on macOS/WSL/BSD too); `sync/scheduler.py` does not generate a systemd -> unit. -> -> See `docs/work/MACOS_LAUNCHD_INTEGRATION_PLAN.md` for the launchd design -> and historical validation limits. Current CI includes macOS, Windows, and -> Linux; that does not prove a live `launchctl` load. The fsconnect and Telegram -> CLIs also ship generate-only plist commands; see -> [`macos/README.md`](../macos/README.md). +> A systemd `--user` `Type=oneshot` service plus timer adds journald logging +> and `Persistent=true` catch-up on Linux; cron stays the implemented portable +> baseline and `sync/scheduler.py` generates no systemd unit. launchd design +> and validation limits: `docs/work/MACOS_LAUNCHD_INTEGRATION_PLAN.md` (CI does +> not prove a live `launchctl` load); the fsconnect and Telegram CLIs ship the +> same generate-only plist pattern ([`macos/README.md`](../macos/README.md)). --- @@ -388,11 +352,11 @@ No field is ever named `query` (that would be SHA-256-hashed by the logger). ## Why rclone (and not Maestral or the Dropbox SDK) -| Tool | Verdict | -|---|---| -| **rclone** ✅ | Battle-tested transport, **zero Python deps**, runs as a separate process, refresh token stays in `rclone.conf`. The CLI surface is the security boundary. Native Windows support. | -| Maestral | Always-on **daemon** — wrong shape for an offline-first agent; no native Windows. | -| `dropbox` Python SDK | Forces CyClaw to **hold the refresh token** and drags `requests`/`urllib3` into a deliberately minimal dependency tree. | +rclone is a battle-tested separate-process transport with zero Python deps, +native Windows support, and a refresh token that never leaves `rclone.conf`. +Maestral is an always-on daemon (wrong shape for an offline-first agent, no +native Windows); the `dropbox` SDK would make CyClaw hold the refresh token and +pull `requests`/`urllib3` into a deliberately minimal dependency tree. ## Encrypted rclone configuration in scheduled jobs diff --git a/docs/THREAT_MODEL.md b/docs/THREAT_MODEL.md index 0a50a6f9b..41cedd979 100644 --- a/docs/THREAT_MODEL.md +++ b/docs/THREAT_MODEL.md @@ -79,7 +79,7 @@ least-privilege database role. MSSQL's ODBC read-only hint is not enforcement. | **Index-progress probe** (`GET /index/status`, unauthenticated by design) | Returns build state only — never corpus content — from a lock-guarded snapshot. Deliberately **exempt** from the per-IP limit: the console polls it every 1.5 s for the length of a build, which would otherwise spend two-thirds of the operator's own budget. Same posture `GET /health` already has | `gate.py`, `static/terminal.js` | | **Uncontrolled external model calls** | Triple-gate: `mode=hybrid` **and** the selected provider's `grok.enabled`/`claude.enabled` **and** `user_confirmed_online` | `graph.py`, `config.yaml` | | **Uncontrolled evaluation egress** | Standalone CLI refuses unless `CYCLAW_EVAL_LIVE=1` and `ANTHROPIC_API_KEY` are both present; Anthropic origin is exact-pinned; contestant is loopback-only; embedding downloads are forced offline | `tests/judge_eval.py`, `tests/test_judge_eval.py` | -| **Telemetry / data exfil via tracing** | Telemetry-kill env vars set before any third-party import from one shared mapping (`utils/telemetry_kill.py`), applied at import time by every maintained Python chokepoint — gateway, MCP server, metrics, harness server, the retrieval indexer/vector-store/cache CLIs, the auth/cert CLIs, and the sync/agentic/guardrails/telegram/opentweet packages (invariant-guard G1 pins 15 of these orderings) — AND delivered as literal environment before the interpreter starts at every process boundary: Docker ENV, the shipped launchers, generated launchd plists / Windows tasks / cron lines, verifier and `gh` children (`build_telemetry_safe_env`). An ambient value in the operator's environment is overwritten, not inherited; the declarative-OTel config names are removed outright. These controls silence telemetry readers only — they are not a network kill switch, and intentional policy-gated egress is classified separately (SECURITY.md); HF Hub network calls are additionally cut off via `local_files_only=True` once the embedding model (and `models.reranker`'s cross-encoder, which `retrieval/rerank.py` loads the same way) is confirmed cached on disk (the `HF_HUB_OFFLINE`/`TRANSFORMERS_OFFLINE` env vars alone do not gate this in-process — huggingface_hub latches that constant at its own import time, which the eligibility probe itself triggers before the env vars are set; `local_files_only` is passed directly to `SentenceTransformer(...)` instead, which gates independently); raw query text never persisted (hashes only) | `utils/telemetry_kill.py`, `gate.py`, `mcp_hybrid_server.py`, `retrieval/vector_store.py`, `retrieval/embeddings.py`, `retrieval/rerank.py`, `utils/logger.py` | +| **Telemetry / data exfil via tracing** | Telemetry-kill env vars set before any third-party import from one shared mapping (`utils/telemetry_kill.py`), applied at import time by every maintained Python chokepoint — gateway, MCP server, metrics, the retrieval indexer/vector-store/cache CLIs, the auth/cert CLIs, and the sync/agentic/guardrails/telegram/opentweet packages (invariant-guard G1 pins 14 of these orderings) — AND delivered as literal environment before the interpreter starts at every process boundary: Docker ENV, the shipped launchers, generated launchd plists / Windows tasks / cron lines, verifier and `gh` children (`build_telemetry_safe_env`). An ambient value in the operator's environment is overwritten, not inherited; the declarative-OTel config names are removed outright. These controls silence telemetry readers only — they are not a network kill switch, and intentional policy-gated egress is classified separately (SECURITY.md); HF Hub network calls are additionally cut off via `local_files_only=True` once the embedding model (and `models.reranker`'s cross-encoder, which `retrieval/rerank.py` loads the same way) is confirmed cached on disk (the `HF_HUB_OFFLINE`/`TRANSFORMERS_OFFLINE` env vars alone do not gate this in-process — huggingface_hub latches that constant at its own import time, which the eligibility probe itself triggers before the env vars are set; `local_files_only` is passed directly to `SentenceTransformer(...)` instead, which gates independently); raw query text never persisted (hashes only) | `utils/telemetry_kill.py`, `gate.py`, `mcp_hybrid_server.py`, `retrieval/vector_store.py`, `retrieval/embeddings.py`, `retrieval/rerank.py`, `utils/logger.py` | | **DoS (request flood / runaway process)** | Per-IP rate limit (60/min) on every API route except the two unauthenticated read-only probes `GET /health` and `GET /index/status`. The console page `GET /` and the `/static` mount are static-asset surfaces and carry no rate-limit dependency either; container `mem`/`pids`/`cpus` limits | `utils/ratelimit.py`, `docker-compose.yml` | | **Compromised out-of-band subprocess** (rclone/gh) | argv-list only (no `shell=True`); absolute binary paths; Docker builtin seccomp; non-root; `no-new-privileges`; `cap_drop: ALL`; read-only rootfs | `sync/`, `agentic/`, `Dockerfile`, `docker-compose.yml`, `deploy/seccomp/` | diff --git a/docs/agentic/AGENTIC_README.md b/docs/agentic/AGENTIC_README.md index 8007f2817..53d543d26 100644 --- a/docs/agentic/AGENTIC_README.md +++ b/docs/agentic/AGENTIC_README.md @@ -195,38 +195,26 @@ which subcommand carries them** — this is easy to get backwards: plan. `real-repo-run` and `real-repo-run-plan` each read `--provider` independently; there is no cross-check between them. -To get "cloud plans, local Qwen implements": pass `--provider`/ -`--confirm-online` to `real-repo-run-plan` only, and **omit `--provider` -entirely on the follow-up `real-repo-run` call**. Passing `--provider` to -*both* is allowed and does something real (the plan text still reaches the -prompt) but silently defeats the two-stage economics above — the cloud model -is now billed on every `--max-iterations` attempt, not once, with no warning -from the CLI either way. As of this writing the whole pipeline, this two-stage recipe -included, is CLI-only (see the reachability note above), so drive this step -from a terminal. +For "cloud plans, local Qwen implements": pass `--provider`/`--confirm-online` +to `real-repo-run-plan` only and **omit `--provider` on the follow-up +`real-repo-run`**. Passing it to both is allowed and works, but bills the +cloud model on every `--max-iterations` attempt instead of once, with no CLI +warning. The whole pipeline is CLI-only, so drive this from a terminal. **The DeepAgents-graph path, retired (owner decision, 2026-07-31)** -(`agentic/deepagent_github/builder.py`'s `create_deep_agent` integration, -plus the harness optimizer's fixture-based evaluation loop): this is the -subsystem the now-superseded "fixture-only evaluation... does not write the -real repo" description below used to describe accurately. It was never -live-fired against a real model -- `agentic.cli`'s `deepagent-plan` -subcommand deliberately probes the build's gate state without ever calling -`.invoke()` on the constructed agent -- and no further development is -planned on it: the pipeline above is the one live real-repo coding path -going forward. The code, its tests, and its `deepagents-harness` CI lane -remain in the repository unmodified (this is a documentation-only decision, -not a deletion), so the rest of this description stays accurate for anyone -reading it: it remains disabled by default and out-of-band. Its canonical -plan is `docs/work/GITHUB_DEEP_AGENT_HARNESS_OPTIMIZER_PLAN.md` (see its -own retirement note); the implemented phase 6-9 controls and operator -boundaries are in `docs/work/DEEP_AGENT_HARNESS_PHASES_6_9.md`. It uses -scoped proposer-workspace tools, a virtual in-state Deep Agents backend, -local-only memory/skills, fixture-only evaluation, and human-gated local -candidate artifacts, and does not execute shell commands, write the real -repo, write GitHub, expose unrestricted filesystem tools, or import from the -core request path -- all still true of THIS subsystem specifically, not of -the agentic layer as a whole now that the pipeline above exists. +(`agentic/deepagent_github/builder.py`'s `create_deep_agent` integration plus +the harness optimizer's fixture-based evaluation loop) was never live-fired +against a real model — `deepagent-plan` probes the build's gate state without +ever calling `.invoke()` — and gets no further development; the pipeline above +is the one live coding path. Its code, tests, and `deepagents-harness` CI lane +stay in the repository unmodified, disabled by default and out-of-band. It +uses scoped proposer-workspace tools, a virtual in-state backend, local-only +memory/skills, fixture-only evaluation, and human-gated local candidate +artifacts, and does not execute shell commands, write the real repo or GitHub, +expose unrestricted filesystem tools, or import from the core request path — +true of this subsystem, not of the layer as a whole. Plan and phase 6–9 +controls: `docs/work/GITHUB_DEEP_AGENT_HARNESS_OPTIMIZER_PLAN.md` and +`docs/work/DEEP_AGENT_HARNESS_PHASES_6_9.md`. > **Egress classification** (SECURITY.md): the agentic layer's GitHub reads/writes via `gh` and the gated cloud-planner adapters are class-3 intentional policy-gated traffic. Every `gh` child is spawned with `build_telemetry_safe_env()` so `GH_TELEMETRY=false` and the update-notifier opt-outs always win over ambient values, and the verifier's minimal child env re-applies the full canonical telemetry block (`agentic/executor/runner.py`) — note `NO_PROXY="*"` there means "bypass any proxy", a best-effort control, not fail-closed networking. diff --git a/docs/agentic/GITHUB_WRITE_ENABLEMENT.md b/docs/agentic/GITHUB_WRITE_ENABLEMENT.md index 79118fae5..3a762c8a2 100644 --- a/docs/agentic/GITHUB_WRITE_ENABLEMENT.md +++ b/docs/agentic/GITHUB_WRITE_ENABLEMENT.md @@ -1,9 +1,6 @@ # GitHub Write Enablement — procedure and security review -> **Status update — 2026-09-06 (docs review, Claude Code):** COMPLETE (as a procedure/checklist doc — meaning the enablement it describes is done and signed, not that writes are unrestricted). Verified against live code: `agentic/writer.py` has `EXECUTION_ENABLED = True` and the disable-only `CYCLAW_AGENTIC_WRITE_DISABLE` env kill switch (`_WRITE_DISABLE_ENV`); `config.yaml` ships `agentic.mode: "write"` / `agentic.writes_enabled: true` but `agentic.enabled: false` (line ~799) and both `deepagent_github.allow_github_writes: false` / `allow_git_write_tools: false` (lines 844, 853) — exactly the layered posture this doc's gate-chain table describes. This doc is self-maintaining (it already tracks its own checklist re-runs and the 2026-08-07 sign-off inline), so nothing here is stale relative to code. -> -> **What's left:** -> - Nothing outstanding — this is a living operational record, current as of its last internal edit. Keep updating it in place (not this stamp) if any of the six gates change state; do not archive it while `agentic.enabled` could still flip to `true`. +> **Status update — 2026-09-06 (docs review), re-verified 2026-10-13:** COMPLETE as a procedure/checklist record — the enablement it describes is done and signed, not "writes are unrestricted." Live code and config match the gate-chain table below: `agentic/writer.py` ships `EXECUTION_ENABLED = True` with the disable-only `CYCLAW_AGENTIC_WRITE_DISABLE` kill switch; `config.yaml` ships `agentic.mode: "write"`, `agentic.writes_enabled: true`, `agentic.enabled: false`, and `deepagent_github.allow_github_writes` / `allow_git_write_tools` both `false`. This is a living operational record: update it in place if any gate changes state, and do not archive it while `agentic.enabled` could still flip to `true`. **Status: ARMED (operator-signed 2026-08-07).** `agentic/writer.py` ships `EXECUTION_ENABLED = True`, and `config.yaml` ships `mode: write` + diff --git a/docs/m5-48gb-coding-expectations.md b/docs/m5-48gb-coding-expectations.md index 65608b78a..659b39e53 100644 --- a/docs/m5-48gb-coding-expectations.md +++ b/docs/m5-48gb-coding-expectations.md @@ -20,12 +20,10 @@ mistakes durable. ## Operator SKU -Recorded facts only. The Pro bin was **verified 2026-09-06** from the -operator's System Information (Hardware overview + Graphics/Displays panes): -this is the **18-core CPU / 20-core GPU** bin, not the 15/16 one. Serial number, -Hardware UUID, and Provisioning UDID were deliberately left out — per-device -identifiers do not belong in a public repo; the Model Identifier and SKU below -are shared by every unit of this configuration and are what a reader needs. +Recorded facts only, verified 2026-09-06 from System Information: the +**18-core CPU / 20-core GPU** bin, not the 15/16 one. Per-device identifiers +(serial, Hardware UUID, Provisioning UDID) are deliberately omitted; the Model +Identifier and SKU below are shared by every unit of this configuration. | Field | This machine | |---|---| @@ -42,15 +40,12 @@ are shared by every unit of this configuration and are what a reader needs. | Not this machine | **M5 Max** (18/32 or 18/40, ~460 or ~614 GB/s, max **128 GB**) | | Local coder | `qwen3.8:27b-mlx`, 4-bit, `reasoning_effort: none`, `num_ctx` 32768 | -48 GB on an M5 Pro is a mid-high Pro config (Pro ceiling is 64 GB). It is also -a valid Max *option* on the 40-core GPU SKU — that coincidence is why Max -review units get quoted as if they were this laptop. They are not. - ## Which chip this is (M5 vs M5 Pro vs M5 Max) -"MacBook Pro M5 48 GB" in operator speech means **M5 Pro + 48 GB**. Base M5 -cannot be configured with 48 GB. M5 Max can be configured with 48 GB on the -40-core GPU SKU, but that is a different SoC. +"MacBook Pro M5 48 GB" in operator speech means **M5 Pro + 48 GB** (Pro +ceiling 64 GB). Base M5 cannot be configured with 48 GB; M5 Max can, on the +40-core GPU SKU — that coincidence is why Max review units get quoted as if +they were this laptop. They are a different SoC. | Chip | Typical CPU / GPU | Memory bandwidth | Unified memory ceiling | 48 GB available? | |---|---|---|---|---| @@ -87,22 +82,19 @@ What you ship and run is the constraint: | Plan file | `_MAX_PLAN_CHARS` **6,000**; `planner_max_tokens` **3072** | A plan the size of a diff means the planner misbehaved | | Local model | `qwen3.8:27b-mlx`, `reasoning_effort: none` | Thinking-on burns the 32k window before a patch exists | -The machine can physically hold 64k–128k context. The shipped env file sets +The machine can physically hold 64k–128k context; the shipped env file sets `OLLAMA_CONTEXT_LENGTH` to 32768 so a sourced `ollama serve` does not sit at -0% processing on the RAG floor. That number is not a parameter of the -registry tag. Raising `num_ctx` to 64k on this Mac is still feasible. It -makes TTFT worse and still leaves you with a 27B. That is "more than burst" -on **volume**, not on **judgment**. - -Project decision: local 27B is a **supervised executor**. Claude Code / Grok -stays the architect for anything that touches invariants. Burst pattern stays -one implementation file + one test, thinking off, I6 list off-limits -(`gate.py` / `graph.py` / `mcp_hybrid_server.py` / `agentic/` for the 27B -job itself / `soul.md` / `INVARIANTS.md` / `config.yaml`). - -You can give Qwen a bigger slice of one file, a test, and a short plan. You -should not give it a whole implementation plan and hope 48 GB turns it into -Claude. +0% processing on the RAG floor (the registry tag carries no such parameter). +Raising `num_ctx` to 64k is feasible, worsens TTFT, and still leaves you with a +27B: "more than burst" on **volume**, not on **judgment**. + +Project decision: the local 27B is a **supervised executor**; Claude Code / +Grok stays the architect for anything touching invariants. The burst pattern +is one implementation file + one test, thinking off, with the I6 list +off-limits (`gate.py` / `graph.py` / `mcp_hybrid_server.py` / `agentic/` for +the 27B job itself / `soul.md` / `INVARIANTS.md` / `config.yaml`). A bigger +slice of one file, a test, and a short plan is fine; a whole implementation +plan is not what 48 GB buys. ## What this cached model can do on this Mac @@ -219,14 +211,11 @@ Red flags: ## Bottom line -M5 Pro 48 GB can run more than burst. You chose 32k and one-file jobs because -a 27B at long context is slow, stall-prone, and bad at architecture. That was -the correct product call on this SKU. An M5 Max would change decode speed and -the ceiling for a second resident model. It would not change the 27B's job. - -A pre-compact scratchpad is worth doing as **paging**, identical in spirit to -`plan.md` + session notes. It does not mitigate "only a 27B." It mitigates -forgetting the test failure from two turns ago. If you want better plans, keep -planning on Claude/Grok (or a human), keep Qwen as the executor, and make the -scratchpad a short, typed, human-owned working set — not a place the 27B -journals its feelings before the window collapses. +M5 Pro 48 GB can run more than burst; 32k and one-file jobs were chosen +because a 27B at long context is slow, stall-prone, and bad at architecture. +An M5 Max would change decode speed and the ceiling for a second resident +model, not the 27B's job. A pre-compact scratchpad is worth doing as +**paging** (same spirit as `plan.md` + session notes): it mitigates forgetting +the test failure from two turns ago, not "only a 27B." Keep planning on +Claude/Grok or a human, keep Qwen as the executor, and keep the scratchpad a +short, typed, human-owned working set. diff --git a/docs/security-philosophy/numbat_pre_action_gate.md b/docs/security-philosophy/numbat_pre_action_gate.md index 926105e40..83e077c16 100644 --- a/docs/security-philosophy/numbat_pre_action_gate.md +++ b/docs/security-philosophy/numbat_pre_action_gate.md @@ -77,23 +77,19 @@ fixture jobs run. advises; a different binary (`/bin/true`, another release) denies the call itself. 3. A private temp directory receives the event, a byte-for-byte snapshot of - the rule files in each rule directory (every name ending in `.yaml` or - `.yml`, in any case, which is how Numbat finds them), and the - engine's canary rule. The read is bounded: past the caps on rule files, - rule bytes and directory entries (`_MAX_RULE_*` in - `utils/numbat_gate.py`, far above any real rule set), or past the call's - `timeout_sec`, the call is denied, so a `rules_dirs` pointed at a home - directory or a slow mount cannot stall a request. The whole decision - runs on a worker thread the call stops waiting for at `timeout_sec`, so - even a filesystem call that never returns (a stalled network mount under - `rules_dirs` or the binary) only holds the call until its deadline. A - stuck worker is left behind, and only a few can be stuck at once - (`_MAX_GATE_WORKERS`, `_MAX_RULE_READERS`). Then - `numbat rules test --fixture --no-builtin-rules --rules-dir ... --rules-dir ` - evaluates the event against the operator's rules only. The shipped catalog - is detection-only, so it is not loaded. The engine classifies the same - bytes Numbat reads, so an edit that lands mid-request cannot pair one - version's `enforce` flag with another version's match. + the rule files in each rule directory (every `.yaml`/`.yml` name in any + case, as Numbat finds them), and the engine's canary rule. The read is + bounded — past the caps on rule files, bytes, and directory entries + (`_MAX_RULE_*` in `utils/numbat_gate.py`) or past `timeout_sec` the call is + denied — and the whole decision runs on a worker thread the call stops + waiting for at `timeout_sec`, so a stalled mount under `rules_dirs` or a + hung binary holds the call only until its deadline; a stuck worker is left + behind and only a few can be (`_MAX_GATE_WORKERS`, `_MAX_RULE_READERS`). + Then `numbat rules test --fixture --no-builtin-rules --rules-dir ... --rules-dir ` + evaluates the event against the operator's rules only (the detection-only + shipped catalog is not loaded). The engine classifies the same bytes Numbat + reads, so a mid-request edit cannot pair one version's `enforce` flag with + another version's match. 4. The canary rule, `cyclaw.gate.canary`, matches every call. A run that does not report it denies. Exit 0 with no output is also what `/bin/true` or a CLI that skipped the event prints, so only the canary's match shows that diff --git a/guardrails/README.md b/guardrails/README.md index 6a6db82f5..b6c9753bd 100644 --- a/guardrails/README.md +++ b/guardrails/README.md @@ -68,7 +68,7 @@ a model when the engine is available. The gateway instead uses non-generating | `integration.py` | NeMo wrapper + `check_input` / `check_output` | | `rails.py` | Offline floor (injection, soul-mutation intent, grounding) | | `metrics.py` | Separate `logs/guardrails.jsonl` (hashes, not the core audit stream) | -| `cli.py` / `selftest.py` | Operator surface | +| `cli.py` / `selftest.py` / `verify_install.py` | Operator surface; `verify_install` is the strict install smoke (real engine, shipped rules, exact verdicts) CI runs on every platform | | `errors.py` | `GuardrailsError` hierarchy, rooted at `utils.errors.RAGError` | | `boundary.py` | Provider-independent typed decisions + provenance (#1134 Phase 1). Hashes and reason codes only. Never imported by the core six (I6). Production code does not consume these types; only tests import them. `profiles.py` mirrors `GuardrailStage` by hand. | | `broker.py` | NeMo non-generating `LLMRails.check` around the existing generation helper (#1134 Phase 3). Never grants I3, never calls `generate_async`; graph reaches it only via `utils/guardrail_bridge` | @@ -94,15 +94,10 @@ Canonical table: [`docs/NeMo/README.md`](../docs/NeMo/README.md). | `check_jailbreak` input rail | **Not enforced as a rail of its own** — configured in `input_rails`; offline floor uses `check_injection` / `check_soul_mutation`. With NeMo installed, the Colang flow `check cyclaw jailbreak` runs the same injection-marker action as `check_injection`, and no LLM-backed rail (`self_check_input`) is active. So no layer has a jailbreak classifier: a persona prompt is caught only when it matches those markers (`you are now …`, `from now on you are …`), and one worded differently passes. | | Topical rails (`stay_in_local_knowledge`, `no_unauthed_external_advice`) | **Not enforced offline** — configured but not referenced in `integration.py`. | -When `nemoguardrails` is absent, the enabled floor enforces `check_injection`, -`check_soul_mutation`, `check_grounding`, and `check_soul_leak` within their -route scopes. `check_jailbreak` and the topical rails still skip. A configured -name alone is not evidence that a rail runs. - -Issue #1486 Track B declines model-assisted `self_check_input`, -`self_check_output`, and `self_check_facts`. No such flow is active. The retained -prompt templates are inert. Active `/query` flows add no model calls. - -See the [Track B verification record](../docs/audits/2026-10-03-nemo-track-b.md) -for phase acceptance and remaining limits. The older #1134 history includes -separate follow-ups for NLI and platform sandbox acceptance. +Without `nemoguardrails` the enabled floor still enforces `check_injection`, +`check_soul_mutation`, `check_grounding`, and `check_soul_leak` in their route +scopes; `check_jailbreak` and the topical rails skip either way. A configured +name alone is not evidence that a rail runs. Model-assisted `self_check_*` +rails are declined (issue #1486 Track B) and inert; active `/query` flows add +no model calls. Phase acceptance and remaining limits: +[Track B record](../docs/audits/2026-10-03-nemo-track-b.md). diff --git a/llm/README.md b/llm/README.md index ac6bd16c7..5cb41f71c 100644 --- a/llm/README.md +++ b/llm/README.md @@ -13,30 +13,25 @@ decided entirely by `graph.py`'s edges and `gate.py`'s construction gates | `GrokClient` | x.ai | OpenAI-compatible `/chat/completions`; ignores ambient `HTTP(S)_PROXY` (`trust_env=False`) so `GROK_API_KEY` cannot transit an operator proxy | | `ClaudeClient` | Anthropic | Messages API; ignores ambient `HTTP(S)_PROXY` (`trust_env=False`) so `ANTHROPIC_API_KEY` cannot transit an operator proxy | -The two external clients are only ever **constructed** when -`mode == "hybrid"` and the provider's `enabled` flag is true, and only ever -**called** after per-request user confirmation — the triple gate (I3) is -enforced in `gate.py` + `graph.py`, not here. - -Read that conditional against the shipped config before assuming it means -"off": `app.mode` ships `"hybrid"` and both `models.grok.enabled` and -`models.claude.enabled` ship `true` (armed 2026-08-07), so on a fresh clone -**both external clients are constructed at boot** and two of the three gates -are already open. The only remaining gate is the per-request -`user_confirmed_online`, which cannot be pre-set in config. Billed calls are -recorded to `logs/spend.jsonl` — see [`spend/README.md`](../spend/README.md). +The two external clients are **constructed** only when `mode == "hybrid"` and +the provider's `enabled` flag is true, and **called** only after per-request +user confirmation — the triple gate (I3) lives in `gate.py` + `graph.py`, not +here. Read that against the shipped config: `app.mode` is `"hybrid"` and both +`models.grok.enabled` and `models.claude.enabled` are `true`, so on a fresh +clone both clients are constructed at boot and the only closed gate is the +per-request `user_confirmed_online`, which no config key can pre-set. Billed +calls land in `logs/spend.jsonl` ([`spend/README.md`](../spend/README.md)). ## Guarded generation -The gateway ships with `guardrails.enabled: true`. `graph.py` calls these -clients through the callable injected by `utils/guardrail_bridge.py`, which -checks input before generation and output afterward on all four answer -routes. Missing or failed NeMo checks use deterministic input and -soul-leak checks and audit `guardrail_degraded`. Grounding applies only to -retrieved local answers. Active rails add no model calls. An unexpected -wrapper failure returns `GUARDRAIL_ERROR` without replaying generation; -the clients' bounded HTTP retry policy below remains separate. -See the [NeMo reference](../docs/NeMo/README.md) for rail scope and limitations. +With the shipped `guardrails.enabled: true`, `graph.py` calls these clients +through the callable `utils/guardrail_bridge.py` injects: input checked before +generation and output after on all four answer routes, deterministic input and +soul-leak checks (audited `guardrail_degraded`) when NeMo is missing or fails, +grounding on retrieved local answers only, no added model calls. An unexpected +wrapper failure returns `GUARDRAIL_ERROR` without replaying generation; the +HTTP retry policy below is separate. Rail scope and limits: +[NeMo reference](../docs/NeMo/README.md). ## Shared behavior diff --git a/macos/README.md b/macos/README.md index 0cb9c9202..71a18e663 100644 --- a/macos/README.md +++ b/macos/README.md @@ -12,14 +12,15 @@ state lives under `~/.CyClaw`. | Script | What it does | |---|---| -| `setup-cyclaw.sh` | **The one-command onboarding entry point (#1053).** Wraps everything below into a single operator decision: clone (if no checkout is found), run `setup-from-clone.sh --no-start`, then optionally start the gateway, open the loopback console, and autofill the generated key into `#apiKeyInput` in browser memory only. Works standalone too — download just this file and it offers to clone. `--repo PATH`, `--clone-dir PATH`, `--start`/`--no-start`, `--browser`/`--no-browser`, `--autofill-api-key`/`--no-autofill-api-key`, `--skip-prompts`, `--dry-run`; forwards its remaining flags (`--skip-install`, `--skip-keys`, `--small-model`, `--ollama-model TAG`, etc.) straight to `setup-from-clone.sh`. It does not reimplement installation or key persistence — every write still goes through the scripts below. Run it with `--help` for the authoritative flag list. | -| `setup-from-clone.sh` | **One-shot after `git clone`** on Apple Silicon. Chains `install-cyclaw.sh` + `setup-cyclaw-keys.sh` (prompts for Telegram / Claude / Grok / GitHub), checks Ollama, builds the retrieval index, then starts the gateway. `--dry-run`, `--skip-prompts`, `--no-start`, `--small-model`, `--ollama-model TAG`. Note `--skip-prompts` implies no server start; pass `--start` to launch anyway. The script accepts a wider flag set than the common ones listed here — including `--skip-install`, `--skip-python-deps`, `--skip-keys`, `--skip-ollama`, `--skip-index`, `--skip-privacy`, `--no-browser`, `--no-fsconnect`, `--no-profile-edit`, `--no-path-edit`, `--grok-dummy`, `--rotate-key`, `--ollama-install-script`, and `--yes`; run it with `--help` for the authoritative list. Called directly, it is the multi-question path `setup-cyclaw.sh` exists to front. | -| `install-cyclaw.sh` | Home layout, venv, `cyclaw` shim, optional PATH / rc function. `--repo-path`, `--replace-repo`, `--skip-python-deps`, `--no-profile-edit`, `--no-path-edit`, `--no-fsconnect`. `--replace-repo` is intentionally destructive only for an unusable directory at the default `~/.CyClaw/repo` clone target; it does not apply with `--repo-path`. | -| `uninstall-cyclaw.sh` | Removes the rc function, PATH entry, and the `cyclaw keys` source block. Strips an allowlisted secret line from `~/.CyClaw/.env` or `~/.CyClaw/repo/.env` only when the Keychain value read back is byte-equal, and leaves ordinary settings. Does not follow `CYCLAW_REPO`. `--remove-keychain` skips that strip so the purge is not also the only copy. Keeps `~/.CyClaw` unless `--remove-home`. Optional `--remove-fsconnect`. Optional `--remove-keychain` (prompted y/N) removes the Linux API-key libsecret item and selected key file; on macOS it lists, counts, then deletes every exact `com.cgfixit.cyclaw.*` Keychain service named in `utils/secret-policy.tsv` for `id -un` — never a wildcard, never a secret value. `--yes` / `--assume-yes` confirms already-requested destructive flags only. Best-effort unschedules Dropbox sync, `launchctl bootout`s CyClaw LaunchAgent labels (telegram-poll/health, fsconnect-trash, gate, keys-rotate, opentweet, sync, plus the retired console's `harness` label), then frees a leftover loopback listener on `CYCLAW_GATE_PORT` (default 8787). | -| `invoke-cyclaw.sh` | Starts `python gate.py` from `~/.CyClaw/venv`, preserving the bind guard, `api.tls`, and `proxy_headers=False`. Loads non-secret settings from `~/.CyClaw/.env`, then secrets from the Keychain into that process only. `--gate-port` / `CYCLAW_GATE_PORT` select the port (default 8787, overriding `api.port`); the console URL follows `api.host`, `api.tls.enabled`, and that effective port via `utils/gateway_url.py`. `--no-browser` / `--repo` select browser behavior and checkout. `--print-pairing-url` explicitly prints the one-time link for headless use; redirected stdout triggers a secret-link warning. On first run with no `CYCLAW_API_KEY` in the Keychain it runs `setup-cyclaw-keys.sh --skip-prompts` (no profile edit, no dotenv write) to generate one. It opens the console at a one-time `#pair=` link (`CYCLAW_CONSOLE_PAIRING_CODE`, single-use, `security.console_pairing_ttl_sec`), so operator tools arrive unlocked without the key reaching the page. | -| `setup-cyclaw-keys.sh` | Apple Silicon key bootstrap. Autogenerates `CYCLAW_API_KEY`; prompts for Telegram / Claude (`ANTHROPIC_API_KEY`) / Grok / GitHub (skip allowed). Secrets go to the Keychain. `~/.CyClaw/.env` (chmod 600) stays the home for ordinary settings and does not receive secret-classified names unless `--write-env-file` is set. Re-running moves allowlisted plaintext lines into the Keychain and removes those lines (no backup). `--rotate`, `--no-env-file`, `--fill-browser` (loopback `#apiKeyInput` only; the console trades the key for its own signed HttpOnly cookie and clears the field, so the key never reaches localStorage), `--schedule-rotate monthly\|weekly\|never` (writes, never loads, a LaunchAgent), `--unschedule-rotate` (removes that LaunchAgent again), `--restart-servers` (best-effort free the configured gate loopback port after a write; does not start the server). Further flags exist — `--no-keychain`, `--no-repo-env`, `--print-key`/`--no-print-key`, `--copy-key`/`--no-copy-key`, `--clipboard-ttl N`, `--open-consoles`, `--gate-port`, `--repo-path`, `--skip-prompts`, `--grok-dummy`; run it with `--help` for the authoritative list. | +| `setup-cyclaw.sh` | **One-command onboarding (#1053).** Clones when no checkout is found, runs `setup-from-clone.sh --no-start`, then optionally starts the gateway, opens the loopback console, and autofills the generated key into `#apiKeyInput` (browser memory only). Works standalone: download just this file. Own flags: `--repo PATH`, `--clone-dir PATH`, `--start`/`--no-start`, `--browser`/`--no-browser`, `--autofill-api-key`/`--no-autofill-api-key`, `--skip-prompts`, `--dry-run`; everything else is forwarded to `setup-from-clone.sh`. It adds no installer or secret store of its own; `--help` is authoritative. | +| `setup-from-clone.sh` | **One-shot after `git clone`** (Apple Silicon). Chains `install-cyclaw.sh` + `setup-cyclaw-keys.sh` (prompts for Telegram / Claude / Grok / GitHub), checks Ollama, builds the index, then starts the gateway. Common flags: `--dry-run`, `--skip-prompts` (implies no start; add `--start`), `--no-start`, `--small-model`, `--ollama-model TAG`; also `--skip-install`, `--skip-python-deps`, `--skip-keys`, `--skip-ollama`, `--skip-index`, `--skip-privacy`, `--no-browser`, `--no-fsconnect`, `--no-profile-edit`, `--no-path-edit`, `--grok-dummy`, `--rotate-key`, `--ollama-install-script`, `--yes`. Use it directly for scripted runs. | +| `install-cyclaw.sh` | Home layout, venv, `cyclaw` shim, optional PATH / rc function. `--repo-path`, `--replace-repo`, `--skip-python-deps`, `--no-profile-edit`, `--no-path-edit`, `--no-fsconnect`. `--replace-repo` is destructive only for an unusable directory at the default `~/.CyClaw/repo` target; it does not apply with `--repo-path`. | +| `uninstall-cyclaw.sh` | Removes the rc function, PATH entry, and the `cyclaw keys` source block; strips a secret-classified line from `~/.CyClaw/.env` or `~/.CyClaw/repo/.env` only when the Keychain value reads back byte-equal (skipped with `--remove-keychain`, so the purge never deletes the only copy). Does not follow `CYCLAW_REPO`. Keeps `~/.CyClaw` unless `--remove-home`; `--remove-fsconnect` optional. `--remove-keychain` (prompted y/N; `--yes`/`--assume-yes` confirms already-requested destructive flags only) removes the Linux libsecret item and key file, or on macOS every exact `com.cgfixit.cyclaw.*` service named in `utils/secret-policy.tsv` for `id -un` — never a wildcard or a secret value. Best-effort unschedules Dropbox sync, `launchctl bootout`s CyClaw LaunchAgent labels (telegram-poll/health, fsconnect-trash, gate, keys-rotate, opentweet, sync, the retired `harness`), then frees a leftover listener on `CYCLAW_GATE_PORT` (default 8787). | +| `invoke-cyclaw.sh` | Starts `python gate.py` from `~/.CyClaw/venv`, preserving the bind guard, `api.tls`, and `proxy_headers=False`. Loads non-secret settings from `~/.CyClaw/.env`, then secrets from the keystore into that process only. `--gate-port` / `CYCLAW_GATE_PORT` override `api.port` (default 8787); the console URL follows `api.host`, `api.tls.enabled`, and that port via `utils/gateway_url.py`. `--no-browser` / `--repo`. `--print-pairing-url` prints the one-time link for headless use (warns on redirected stdout). With no stored `CYCLAW_API_KEY` it runs `setup-cyclaw-keys.sh --skip-prompts` (no profile edit, no dotenv write) to generate one, then opens the console at a single-use `#pair=` link (`CYCLAW_CONSOLE_PAIRING_CODE`, `security.console_pairing_ttl_sec`) so operator tools arrive unlocked without the key reaching the page. | +| `cyclaw-linux-key.sh` | Sourced by `invoke-cyclaw.sh` on Linux: libsecret (`secret-tool`, service `com.cgfixit.cyclaw.api-key`) first, then `$XDG_CONFIG_HOME/cyclaw/api-key` (0700/0600; symlinked, foreign-owned, group-readable, or malformed files are refused). The key never becomes an argv token. Details: [Linux launcher credentials](#linux-launcher-credentials-and-headless-pairing). | +| `setup-cyclaw-keys.sh` | Apple Silicon key bootstrap. Autogenerates `CYCLAW_API_KEY`; prompts for Telegram / Claude (`ANTHROPIC_API_KEY`) / Grok / GitHub (skip allowed). Secrets go to the Keychain; `~/.CyClaw/.env` (chmod 600) keeps ordinary settings and receives secret-classified names only with `--write-env-file`. Re-running moves plaintext secret lines into the Keychain and removes them (no backup). `--rotate`, `--no-env-file`, `--fill-browser` (loopback `#apiKeyInput` only; the console trades the key for its signed HttpOnly cookie and clears the field), `--schedule-rotate monthly\|weekly\|never` (writes, never loads, a LaunchAgent), `--unschedule-rotate`, `--restart-servers` (frees the gate port after a write; does not start the server). Also `--no-keychain`, `--no-repo-env`, `--print-key`/`--no-print-key`, `--copy-key`/`--no-copy-key`, `--clipboard-ttl N`, `--open-consoles`, `--gate-port`, `--repo-path`, `--skip-prompts`, `--grok-dummy`. | | `cyclaw-keychain-load.sh` | Shared sourced helper for onboarding and launchers. Filters dotenv settings, scrubs secret-classified dotenv values, and loads allowed Keychain items into the process. | -| `cyclaw-public-env.sh` | Sourced by the rc block. Exports non-secret dotenv settings. Defines the secret classification (allowlist plus `*_API_KEY` / `*_TOKEN` / `*_SECRET` / `*_PASSWORD`). | +| `cyclaw-public-env.sh` | Sourced by the rc block. Exports non-secret dotenv settings. Reads the secret classification from `utils/secret-policy.tsv` (shared with Python and PowerShell) and refuses to run without it. | | `setup-fsconnect.sh` | Creates confined `~/CyClaw-FS` (`chmod 700`). Unless `--prepare-only`, enables list/stat/read via `_enable_fsconnect_readlist.py`. | | `_enable_fsconnect_readlist.py` | Writes the confined read/list `fsconnect:` profile into `config.yaml` (writes stay off). | | `cyclaw-keychain-set.sh` | Interactive Keychain store. Bare `-w` (secret never in argv); `-T /usr/bin/security`. Requires a TTY. | @@ -33,46 +34,36 @@ no Homebrew required. ## One command (Apple Silicon) -`setup-cyclaw.sh` is (arguably) the single entry point for a fresh machine — clone, -install, keys, index, start, browser, in that order, with the choices asked -once up front instead of scattered across several scripts' worth of prompts: +`setup-cyclaw.sh` is the single entry point for a fresh machine — clone, +install, keys, index, start, browser, with the choices asked once up front: ```bash -bash macos/setup-cyclaw.sh # inside a clone +bash macos/setup-cyclaw.sh # inside a clone, or standalone (it offers to clone) ``` -It offers to clone when no checkout is found, so the same file also works -standalone: download `macos/setup-cyclaw.sh` by itself and run -`bash setup-cyclaw.sh` (interactive prompts need a real Terminal, not a -piped stdin — pass `--skip-prompts` for a non-interactive run instead). It -delegates to `setup-from-clone.sh` for the actual install/keys/index work -below — it adds no second secret store or installer of its own. - -Onboarding, the launcher, and key setup use the same console URL resolver -when the checkout and Python dependencies are available. Wildcard bind -addresses become loopback browser destinations. Standalone key setup still -works without a checkout, using its default `http://127.0.0.1:` URL. +Interactive prompts need a real Terminal, not a piped stdin; pass +`--skip-prompts` for a non-interactive run. Onboarding, the launcher, and key +setup share one console URL resolver (`utils/gateway_url.py`) when the checkout +and Python dependencies are present: wildcard bind addresses become loopback +browser destinations, and standalone key setup falls back to +`http://127.0.0.1:`. ## One-shot after clone (Apple Silicon) -`setup-from-clone.sh` is the operator-facing "I just cloned this, make it -run" path that `setup-cyclaw.sh` calls. It does **not** reimplement the -scripts above — it chains them and fills the four holes `setup-guide.md` -documents that Option A leaves open (Ollama, the retrieval index, API keys, -and starting the server). Use it directly for scripted/CI-style runs where the -extra clone-detection and browser-autofill layer isn't wanted. +`setup-from-clone.sh` is the path `setup-cyclaw.sh` calls. It chains the +scripts above to fill the four holes `setup-guide.md`'s Option A leaves open +(Ollama, the retrieval index, API keys, starting the server). Use it directly +for scripted runs that do not want clone detection or browser autofill: ```bash git clone https://github.com/CGFixIT/CyClaw.git && cd CyClaw bash macos/setup-from-clone.sh ``` -Privacy matches `cyclaw-privacy`: secrets are never logged, never written -to `config.yaml`, never placed on a child process argv. Secrets persist in -the Keychain via `setup-cyclaw-keys.sh`. Ordinary settings persist in -`~/.CyClaw/.env` (chmod 600). -fsconnect writes and indexing stay off. LaunchAgents are **not** generated -or loaded (those still need `--confirm --reason`). +Secrets are never logged, written to `config.yaml`, or placed on a child +argv; they persist in the Keychain via `setup-cyclaw-keys.sh`, ordinary +settings in `~/.CyClaw/.env` (chmod 600). fsconnect writes and indexing stay +off, and no LaunchAgent is generated or loaded (those need `--confirm --reason`). ## Key bootstrap @@ -86,21 +77,22 @@ load that file through the `# >>> cyclaw keys >>>` block, which does not export secret-classified names. If `CYCLAW_HOME` was set when the keys script ran, the block follows that directory instead of `~/.CyClaw`. -A name is secret-classified when it is on the allowlist in -`macos/cyclaw-public-env.sh` (`CYCLAW_API_KEY`, `TELEGRAM_BOT_TOKEN`, -`GROK_API_KEY`, `ANTHROPIC_API_KEY`, `GH_TOKEN`, `GITHUB_TOKEN`, -`CLAUDE_API_KEY`) or when it ends in `_API_KEY`, `_TOKEN`, `_SECRET`, or -`_PASSWORD`. Those values live in the Keychain. `cyclaw` and launchd read -them into the CyClaw process only. `--write-env-file` is the explicit -opt-in that also writes them into the dotenv; the rc block and the -launchers still do not export those lines. - -Re-running the keys script copies each allowlisted secret into the Keychain -when that item is missing, then removes the plaintext line only when the -value read back is byte-equal. A mismatch keeps the line and warns. Other -lines stay. No backup file is written. A pattern match that has no Keychain -service is left in place and is not loaded. If the Keychain store fails, -the plaintext line stays, because it is still the only copy. +`utils/secret-policy.tsv` (shared with the Python and PowerShell loaders) +defines secret classification: an `exact` name row, each mapped to its +Keychain service (`CYCLAW_API_KEY`, `TELEGRAM_BOT_TOKEN`, `GROK_API_KEY`, +`ANTHROPIC_API_KEY`, `GH_TOKEN`/`GITHUB_TOKEN`, `CYCLAW_SQL_DSN`, the +`CYCLAW_*_DB_URL` family, `RCLONE_CONFIG_PASS`, …), or a `suffix` row +(`API_KEY`, `CREDENTIALS`, `PASSWORD`, `SECRET`, `TOKEN`, `PAT`, `DSN`, +`DB_URL`, `KEY`). Those values live in the Keychain, and `cyclaw` and launchd +read them into the CyClaw process only. `--write-env-file` is the explicit +opt-in that also writes them into the dotenv; the rc block and the launchers +still never export those lines. + +Re-running the keys script copies each classified secret into the Keychain +when its item is missing, then removes the plaintext line only when the value +read back is byte-equal; a mismatch or failed store keeps the line (it is +still the only copy) and warns. Other lines stay and no backup is written. A +suffix match with no Keychain service is left in place and not loaded. ```bash bash macos/setup-cyclaw-keys.sh @@ -154,10 +146,9 @@ reads. ## 401 / key drift recovery Soul routes return `401 bad_credentials` when the Keychain item, the live gate -process environment, and the browser's console cookie disagree. This usually -follows `--rotate`, a reinstall, an expired cookie, or a leftover listener that -still holds the old key. CyClaw never stores the operator key in `localStorage` -and does not read that secret from `~/.CyClaw/.env`. +process environment, and the browser's console cookie disagree — usually after +`--rotate`, a reinstall, an expired cookie, or a leftover listener holding the +old key. 1. Stop stragglers on the configured loopback port (default 8787). `uninstall-cyclaw.sh` does this best-effort before teardown. After a @@ -201,12 +192,10 @@ redirected output does not contain a pairing link. The explicit flag warns on non-TTY stdout because the single-use, short-lived link is a credential: keep it private and delete any saved copy after pairing. It contains no API key. -`uninstall-cyclaw.sh --remove-keychain` prompts before clearing the exact -libsecret service/account and deleting the selected key file. Add `--yes` for -an intentional noninteractive purge. Ordinary uninstall and `--remove-home` -alone keep those credentials. Use the same `XDG_CONFIG_HOME` as at launch; -unrelated files are retained. If `secret-tool` is unavailable or clearing it -fails, the uninstaller reports that the keyring still needs manual verification. +`uninstall-cyclaw.sh --remove-keychain` (see [401 recovery](#401--key-drift-recovery)) +clears the exact libsecret service/account and the selected key file; run it +with the same `XDG_CONFIG_HOME` as at launch. If `secret-tool` is unavailable +or clearing fails, the uninstaller says the keyring still needs manual verification. ## `LaunchAgents/` — templates only diff --git a/powershell/README.md b/powershell/README.md index 9180780ec..d04b45597 100644 --- a/powershell/README.md +++ b/powershell/README.md @@ -26,52 +26,48 @@ Dropbox sync is already scheduled with `python -m sync.cli schedule` (live `schtasks /Create`, daily by default -- `sync.schedule_frequency` also accepts `weekly` / `monthly`; task name `CyClaw Dropbox Sync`). -`Uninstall-CyClaw.ps1` best-effort deletes a **fixed** list of CyClaw task -names (`CyClaw Dropbox Sync`, `CyClaw fsconnect-trash`, -`CyClaw telegram-poll`, `CyClaw telegram-health`, `CyClaw gate`, -`CyClaw harness` (the retired console's name, kept for older installs), `CyClaw opentweet`) so a later generator cannot outlive uninstall. It never -uses a wildcard `/TN`. A missing task is a no-op (query-then-delete; -Windows PowerShell 5.1 must not abort uninstall on `schtasks` stderr). -Credential Manager items are **not** deleted unless `-RemoveCredentials` is -passed (the macOS twin is `--remove-keychain`). That switch prompts y/N -unless `-Yes` is also passed, and it leaves plaintext lines in place. The -default uninstall never writes a credential. - -Telegram / API-key injection for those later generators goes through -`CyClaw-CredMan-Env.ps1` so tokens never appear in task XML. +`Uninstall-CyClaw.ps1` best-effort deletes a **fixed** list of task names +(`CyClaw Dropbox Sync`, `CyClaw fsconnect-trash`, `CyClaw telegram-poll`, +`CyClaw telegram-health`, `CyClaw gate`, `CyClaw opentweet`, and the retired +`CyClaw harness`) so a later generator cannot outlive uninstall; never a +wildcard `/TN`, and a missing task is a no-op (query-then-delete, so Windows +PowerShell 5.1 does not abort on `schtasks` stderr). Credential Manager items +go only with `-RemoveCredentials` (the macOS twin is `--remove-keychain`), +which prompts y/N unless `-Yes` is passed and leaves plaintext lines in place; +the default uninstall never writes a credential. Generators inject tokens +through `CyClaw-CredMan-Env.ps1`, so none appears in task XML. ## Secret classification `%USERPROFILE%\.CyClaw\.env` stays the default home for ordinary settings -(ports, paths, mode flags, model names, feature toggles). `cyclaw` still -loads those lines. A name is secret-classified when `Test-CyclawSecretName` -in `CyClaw-SecretStore.ps1` says so: the allowlist (`CYCLAW_API_KEY`, -`TELEGRAM_BOT_TOKEN`, `GROK_API_KEY`, `ANTHROPIC_API_KEY`, `GH_TOKEN`, -`GITHUB_TOKEN`, `CLAUDE_API_KEY`) or a name ending in `_API_KEY`, `_TOKEN`, -`_SECRET`, or `_PASSWORD`. +(ports, paths, mode flags, model names, feature toggles), and `cyclaw` loads +those lines. `Test-CyclawSecretName` in `CyClaw-SecretStore.ps1` classifies a +name as secret from `utils/secret-policy.tsv` (shared with the bash and Python +loaders): an `exact` row mapped to its Credential Manager target +(`CYCLAW_API_KEY`, `TELEGRAM_BOT_TOKEN`, `GROK_API_KEY`, `ANTHROPIC_API_KEY`, +`GH_TOKEN`/`GITHUB_TOKEN`, `CYCLAW_SQL_DSN`, the `CYCLAW_*_DB_URL` family, +`RCLONE_CONFIG_PASS`, …) or a `suffix` row (`API_KEY`, `CREDENTIALS`, +`PASSWORD`, `SECRET`, `TOKEN`, `PAT`, `DSN`, `DB_URL`, `KEY`). -Allowlisted secrets are stored in Credential Manager and are not written -into `.env` unless `-WriteEnvFile` is passed. Even then, `Invoke-CyClaw.ps1` -does not export secret-classified lines. Do not persist provider keys with -`setx` or `[Environment]::SetEnvironmentVariable(..., "User")`. A name that -matches the suffix pattern but has no Credential Manager target (for example -`DB_PASSWORD`) is not loaded and is not deleted. `CLAUDE_API_KEY` is not -loaded and is not copied; `llm/client.py` reads `ANTHROPIC_API_KEY`. +Classified secrets live in Credential Manager and reach `.env` only with +`-WriteEnvFile`; even then `Invoke-CyClaw.ps1` never exports those lines. Do +not persist provider keys with `setx` or +`[Environment]::SetEnvironmentVariable(..., "User")`. A suffix match with no +Credential Manager target (`DB_PASSWORD`) is neither loaded nor deleted; +`CLAUDE_API_KEY` is classified but never loaded or copied, because +`llm/client.py` reads `ANTHROPIC_API_KEY`. -Re-running install copies each allowlisted secret into Credential Manager -when that item is missing, then removes the plaintext line only when the -value read back is ordinal-equal. A mismatch keeps the line and warns. -Uninstall never writes a credential; it only removes a matching line. -Other lines stay. No backup file is written. If the store fails or the -item is unreadable or empty, the plaintext line stays, because it is -still the only copy. A missing optional credential stays unset and the -gateway still starts. A present item that cannot be read, or is empty, -aborts that launch. On Windows, `Invoke-CyClaw.ps1` generates a missing -`CYCLAW_API_KEY` from 20 random bytes and stores it in Credential Manager -before launch. If it cannot store the key, it warns and starts without one; -key-based operator access then fails closed. An enabled admin login and the -explicit constrained-loopback bypass retain their separate access rules -([operator access](../README.md#api-key-setup-soul-mutations)). +Re-running install copies each classified secret into Credential Manager when +its item is missing, then removes the plaintext line only when the value reads +back ordinal-equal; a mismatch, failed store, or unreadable/empty item keeps +the line (still the only copy) and warns. Other lines stay and no backup is +written; uninstall only ever removes a matching line. At launch a missing +optional credential stays unset and the gateway starts, but a present item +that cannot be read or is empty aborts. `Invoke-CyClaw.ps1` generates a missing +`CYCLAW_API_KEY` (20 random bytes) into Credential Manager first; if it cannot +store it, it warns and starts without one, so key-based operator access fails +closed while an enabled admin login and the constrained-loopback bypass keep +their own rules ([operator access](../README.md#api-key-setup-soul-mutations)). ## Related diff --git a/spend/README.md b/spend/README.md index f29f4896d..4c9e6ac29 100644 --- a/spend/README.md +++ b/spend/README.md @@ -20,18 +20,13 @@ OpenAI's `tiktoken` — Anthropic's billed `input_tokens` is the count to trust. ## What the ledger is not -The spend ledger is not the audit log. `logs/audit.jsonl` is CyClaw's security -evidence stream and stays authoritative for what happened on a request; -`logs/spend.jsonl` is a separate append-only file for what a request cost. -`utils/spend.py` deliberately does not route through `utils/logger.py`'s -`audit_log`, so metrics can reprice cost history without rewriting or polluting -the audit trail. - -The spend ledger is also not a policy decision point. Nothing on the `/query` -path reads it to decide anything — it is written after a billed call completes -and is consumed only by the `cyclaw-metrics` CLI. `gate.py`, `graph.py`, and -`mcp_hybrid_server.py` never import the metrics or sequence-detection modules -for a routing decision. +Not the audit log: `logs/audit.jsonl` stays authoritative for what happened on +a request; `logs/spend.jsonl` is a separate append-only file for what it cost, +written outside `utils/logger.py`'s `audit_log` so history can be repriced +without touching the audit trail. Not a policy point either: nothing on the +`/query` path reads it; it is written after a billed call completes and +consumed only by `cyclaw-metrics`, and the core modules never import the +metrics or sequence-detection code. ## What is recorded @@ -60,11 +55,10 @@ threading lock so concurrent calls cannot interleave a partial line. ### What is never recorded `utils/spend.py` never persists query text, prompt text, message content, -response bodies, API keys, or authorization headers. The optional `query_hash` -is a pseudonymous correlation key, not a request identity, and it is the only field that can -be joined back to an audit record. The `route_path` and `query_hash` fields are -both validated against strict patterns before being written, so a malformed or -oversized value is dropped rather than stored. +response bodies, API keys, or authorization headers. `query_hash` is a +pseudonymous correlation key — the only field joinable to an audit record — +and both it and `route_path` are pattern-validated before writing, so a +malformed or oversized value is dropped rather than stored. ## API keys @@ -106,40 +100,28 @@ logging: spend_file: "logs/spend.jsonl" ``` -Relative paths anchor to the repo root, not the current working directory, so -`cyclaw-metrics` reports the same file no matter where it is invoked. When -`logging.spend_file` is absent from a config, the ledger falls back to -`logs/spend.jsonl`, and `cyclaw-metrics` silently omits its Spend section rather -than inventing an empty one. - -Ledger writes are best-effort by design. A write failure logs a WARNING and -returns; it never raises into the request path, because a full disk must not -turn a successful paid answer into a failed one. +Relative paths anchor to the repo root, so `cyclaw-metrics` reports the same +file wherever it is invoked. Without `logging.spend_file` the ledger falls back +to `logs/spend.jsonl` and `cyclaw-metrics` omits its Spend section. Writes are +best-effort: a failure logs a WARNING and returns, never raising into the +request path, because a full disk must not turn a paid answer into a failed one. ## Who writes to the ledger -Two production call sites record usage, distinguished on the ledger by -the `source` field: - -- **`llm/client.py`** — the `/query` online fallback path, recorded as - `source: "query"`. This is the triple-gated Grok/Claude escalation that a - human confirmed per request. -- **`agentic/deepagent_github/chat_client.py`** — the out-of-band cloud planner - used by the agentic layer, recorded as `source: "agentic"`. +Two production call sites, distinguished by `source`: -The opt-in evaluators (`tests/judge_eval.py` and `tests/judge_calibrate.py`) -also use `llm/client.py` with `source: "eval"`, overriding the destination to -`logs/evals/spend.jsonl`. These synthetic evaluation costs are separate from -the production ledger; a local judge makes no billed external call. +- **`llm/client.py`** — the triple-gated `/query` fallback, `source: "query"`. +- **`agentic/deepagent_github/chat_client.py`** — the out-of-band cloud + planner, `source: "agentic"`. -Keeping both production paths on one file with a `source` tag means a monthly -total is a single pass over one file, while `utils/sequence_detect.py` can still restrict itself -to `source == "query"` rows so the two planes never mix in a forensic join. +The opt-in evaluators (`tests/judge_eval.py`, `tests/judge_calibrate.py`) also +use `llm/client.py` with `source: "eval"` and write to +`logs/evals/spend.jsonl` instead; a local judge makes no billed call. One file +with a `source` tag keeps a monthly total a single pass while +`utils/sequence_detect.py` restricts itself to `source == "query"` rows. -A billed non-JSON response is still recorded. If a provider returns 2xx with a -body CyClaw cannot parse, the line is written with `usage_missing: true` rather -than skipped — quota was consumed either way, and a silently missing line would -understate the bill. +A billed non-JSON response is still recorded, with `usage_missing: true`: +quota was consumed either way, and a missing line would understate the bill. ## How dollars are derived @@ -168,19 +150,15 @@ while Claude's `output_tokens` is already the inclusive billing total. ### Rate staleness -The rate table derives a `PRICED_AS_OF` date (the oldest entry in -`_RATE_VERIFIED`, one per `_RATES` row) and is considered stale after 30 -days. `rates_are_stale()` and `warn_if_priced_as_of_stale()` expose that check so -a long-running deployment surfaces "these dollar figures are from an old rate -card" instead of quietly reporting confident, wrong totals. The token counts -remain correct regardless — only the derived dollars go stale, which is the -entire reason dollars are not persisted. - -To re-price after a vendor changes rates, update the `_RATES` row and its -verification date in `_RATE_VERIFIED` in `utils/spend.py`; `PRICED_AS_OF` is -computed from those dates, never hand-edited (`tests/test_spend.py` asserts every -rate row has one). Every historical line re-prices on the next -`cyclaw-metrics` run; no migration or backfill is needed. +`PRICED_AS_OF` is the oldest date in `_RATE_VERIFIED` (one per `_RATES` row) +and the table counts as stale after `STALE_AFTER_DAYS` (30); `rates_are_stale()` +and `warn_if_priced_as_of_stale()` surface that so a long-running deployment +says "old rate card" instead of reporting confident, wrong totals. Token counts +stay correct regardless — only derived dollars go stale, which is why dollars +are not persisted. To re-price, update the `_RATES` row and its +`_RATE_VERIFIED` date in `utils/spend.py` (never hand-edit `PRICED_AS_OF`; +`tests/test_spend.py` asserts every row has a date). History re-prices on the +next `cyclaw-metrics` run with no migration. ## Reading the ledger @@ -190,20 +168,13 @@ The Spend section is part of the standard metrics output: python -m metrics ``` -The same report is available as `cyclaw-metrics` once the project itself is -installed with `pip install -e .`, since the short name is a console script that -pip only writes when the project is installed. - -The Spend section prints a `today` and a `last_7d` window. Each window reports -total input and output tokens, a derived USD figure, a per-provider row count, -and a per-source row count. When any row in the window carried vendor ticks, the -output also shows `table_usd`, `ticked_table_usd`, `vendor_usd`, and -`delta_usd` side by side, so a drift between CyClaw's rate table and the -vendor's own billing is visible rather than hidden behind a single number. - -Two counters flag data-quality problems in the window: `usage_missing` counts -billed calls whose usage could not be parsed, and `rate_unknown` counts rows -whose model has no entry in the rate table. +(`cyclaw-metrics` is the same report once the project is installed with +`pip install -e .`.) The Spend section prints `today` and `last_7d` windows: +input/output tokens, derived USD, and per-provider and per-source row counts. +When any row carried vendor ticks it also shows `table_usd`, +`ticked_table_usd`, `vendor_usd`, and `delta_usd` side by side so rate-table +drift is visible. Two data-quality counters: `usage_missing` (billed calls with +unparseable usage) and `rate_unknown` (models absent from the rate table). ### Verifying the rate table against the vendor @@ -279,21 +250,14 @@ shared `query_hash` to surface offline forensic patterns — for example, a blocked injection attempt followed by an online escalation inside a 15-minute window. Its output is printed as a Sequences section by `cyclaw-metrics`. -That detector is forensic and CLI-only. It restricts spend rows to -`source == "query"`, counting and dropping agentic rows so the two planes never -mix, and its findings carry only hashes, event names, timestamps, and -provider/model tags — never query text, IP addresses, soul content, or secrets. - -The correlation-window rule assumes CyClaw's shipped threat model: a -single-operator, loopback-bound host where a sequence within the window is the -operator's own activity. The window is a correlation aid, not an actor -identifier. +The detector is forensic and CLI-only: it keeps `source == "query"` rows +(counting and dropping agentic ones), and its findings carry only hashes, event +names, timestamps, and provider/model tags — never query text, IPs, soul +content, or secrets. The window assumes the shipped threat model (one operator +on a loopback host): it is a correlation aid, not an actor identifier. ## Related documentation -- `docs/online-llm/readme.md` — how the Grok and Claude fallbacks are gated and - configured, including the triple gate and the per-query provider selection. -- `docs/THREAT_MODEL.md` — the security scope the ledger's redaction rules are - written against. -- `CLAUDE.md` — the operating manual, including the audit-log privacy rules that - the spend ledger deliberately parallels. +- `docs/online-llm/readme.md` — Grok/Claude gating and per-query provider selection. +- `docs/THREAT_MODEL.md` — the scope the ledger's redaction rules are written against. +- `CLAUDE.md` — the audit-log privacy rules the ledger parallels. diff --git a/sync/README.md b/sync/README.md index d4fa06160..2ecf5acf8 100644 --- a/sync/README.md +++ b/sync/README.md @@ -20,6 +20,7 @@ authority; this file is the in-tree map. | `filters.py` | Generates the rclone filter file (what syncs, what never does). | | `runner.py` | Drives `rclone copy` (pull, default) or `rclone bisync` (opt-in) as an argv-list subprocess with timeouts. | | `scheduler.py` | Scheduled-job backends: cron (default), launchd (Darwin-only, opt-in via `sync.scheduler_backend`, writes a plist but never auto-loads it), Windows Task Scheduler. | +| `scheduled.py` | What the generated cron line / plist / task actually runs (`python -m sync.scheduled`): reads only `RCLONE_CONFIG_PASS` from the OS keystore at execution time through `utils.child_environment`, then invokes the normal CLI, so no scheduler artifact ever carries the password. `--check-credentials` verifies config decryption without transferring data. | | `selftest.py` | Pre-flight checks behind `sync test`. | ## Exit codes (an API — keep them) diff --git a/tests/README.md b/tests/README.md index d9d1bcfa1..70852fee8 100644 --- a/tests/README.md +++ b/tests/README.md @@ -18,13 +18,11 @@ GROK_API_KEY=dummy pytest tests/test_graph.py -q # one file GROK_API_KEY=dummy python -m tests.ci_rag_smoke # real-index RAG smoke ``` -Use `GROK_API_KEY=dummy` for routine tests; they do not require a real provider -key. The separate RAG smoke loads the embedding model and a real index; seed -the model cache first if it must run with `HF_HUB_OFFLINE=1`. - -Python 3.12 is required (`requires-python >=3.12,<3.13`). Build a 3.12 venv -and invoke pytest through it; an unsupported host interpreter can fail on -3.12-only stdlib calls (full recipe in `CLAUDE.md` §4 "Environment & install"). +Routine tests need no real provider key (`GROK_API_KEY=dummy`). The RAG smoke +loads the embedding model and a real index; seed the model cache first if it +must run with `HF_HUB_OFFLINE=1`. Run pytest through a Python 3.12 venv +(`requires-python >=3.12,<3.13`); a 3.11 host interpreter fails on 3.12-only +stdlib calls (recipe in `CLAUDE.md` §4 "Environment & install"). The optional groundedness evaluator is a separate live-spend command and is not part of required CI: @@ -33,19 +31,15 @@ part of required CI: CYCLAW_EVAL_LIVE=1 python tests/judge_eval.py ``` -It also requires `ANTHROPIC_API_KEY` and a running loopback local LLM. Missing -either gate returns exit 2. It uses only the tracked synthetic corpus under -`tests/fixtures/groundedness/`; reports and its isolated index stay ignored -under `logs/evals/`. Exit 0 means the approved rubric passed, exit 1 means a -complete run missed a threshold, and exit 2 means the run was refused or could -not complete. - -Setting `evals.local_judge.enabled: true` in `config.yaml` (with a `model` tag -that differs from `models.local_llm.model`) swaps the Anthropic judge for a -second loopback model: the same command then needs no `ANTHROPIC_API_KEY` and -nothing leaves the host. `CYCLAW_EVAL_LIVE=1` still gates the run. Before -trusting any judge, run the hand-labeled calibration set (36 rows in -`tests/fixtures/groundedness/calibration.json`, judge only, no contestant): +It needs `ANTHROPIC_API_KEY` and a running loopback local LLM, uses only the +tracked synthetic corpus under `tests/fixtures/groundedness/`, and keeps its +reports and isolated index ignored under `logs/evals/`. Exit 0: rubric passed; +1: a complete run missed a threshold; 2: refused or incomplete (including a +missing gate). `evals.local_judge.enabled: true` (with a `model` tag different +from `models.local_llm.model`) swaps the Anthropic judge for a second loopback +model — no `ANTHROPIC_API_KEY`, nothing leaves the host, `CYCLAW_EVAL_LIVE=1` +still gates. Before trusting any judge, run the hand-labeled calibration set +(36 rows in `tests/fixtures/groundedness/calibration.json`, judge only): ```bash CYCLAW_EVAL_LIVE=1 python tests/judge_calibrate.py diff --git a/tools/lora_finetune/README.md b/tools/lora_finetune/README.md index c2db0ad1e..682169e59 100644 --- a/tools/lora_finetune/README.md +++ b/tools/lora_finetune/README.md @@ -49,11 +49,10 @@ HF_HUB_OFFLINE=1 TRANSFORMERS_OFFLINE=1 python -m pytest tests/ -q --tb=short python dryrun_finetune.py ``` -The offline flags prevent an installed Transformers package from fetching -tokenizer assets during tests. The root command `python -m pytest tests/` targets the main repository's -`tests/` directory. It does not collect this kit's tests. The dedicated LoRA -workflow runs the commands above with only pytest installed. It does not install -the blocked training profile. +The offline flags stop an installed Transformers from fetching tokenizer assets +during tests. The repo-root `python -m pytest tests/` does not collect this +kit's tests; the dedicated LoRA workflow runs the commands above with only +pytest installed and never installs the blocked training profile. To regenerate the canonical dataset and JSONL preview: @@ -76,15 +75,12 @@ datasets==5.0.1 accelerate==1.15.0 ``` -The conflict is deliberate. The patched Transformers and Datasets floors must -not be lowered merely to make the resolver succeed. `requirements.txt` records -the specific upstream constraints and vulnerability rationale that established -this block. - -OSV-Scanner excludes `tools/lora_finetune` from the repository-wide dependency -walk because this optional file is neither installed nor a lockfile. After a -compatible CUDA profile is established and installed on its target host, audit -that environment from the repository root with: +The conflict is deliberate: the patched Transformers and Datasets floors must +not be lowered to make the resolver succeed, and `requirements.txt` records the +upstream constraints and vulnerability rationale. OSV-Scanner excludes +`tools/lora_finetune` from the repository-wide walk because this optional file +is neither installed nor a lockfile; once a compatible CUDA profile is +installed on its target host, audit that environment from the repository root with: ```bash pip-audit -r tools/lora_finetune/requirements.txt diff --git a/utils/README.md b/utils/README.md index 66fcccd1a..64ee91c6b 100644 --- a/utils/README.md +++ b/utils/README.md @@ -13,7 +13,11 @@ modules"; this file groups them by concern. | Module | Role | |---|---| -| `sanitizer.py` | Injection filter for `/query`; patterns come from `config.yaml` (`banned_patterns`). `lru_cache`d by config path — restart to pick up edits. | +| `sanitizer.py` | Injection filter for `/query`; patterns come from `config.yaml` (`banned_patterns`). `lru_cache`d by config path — restart to pick up edits. `_unicode_confusables.py` is its generated homoglyph table (Unicode 17.0.0; regenerate with `unicode_data/generate.py`, never hand-edit). | +| `secret_policy.py` | Reads the inert, versioned `utils/secret-policy.tsv` (exact names → keystore service, plus classified suffixes) shared with the bash and PowerShell loaders, so one file decides what counts as a secret everywhere. | +| `child_environment.py` | Explicit capability environments for trusted helper subprocesses: a child starts from an empty env and receives only the named variables, so a credential in the gateway never leaks to a helper by inheritance. Limits inheritance; not process isolation. | +| `public_errors.py` | Keeps filesystem layout and credentials out of HTTP diagnostics: maps internal errors to the generic bodies the gateway returns. | +| `external_budget.py` | Durable UTC daily/monthly attempt ceilings shared by the Grok and Claude transports (`policy.external_call_limits`, shipped 100/1,000, SQLite ledger at `ledger_path`); a reservation precedes every POST attempt, retries included, zero denies all, and unavailable storage refuses egress. | | `authn.py` | Per-user authentication primitives (scrypt hash/verify, lockout arithmetic, session/CSRF/token id generation). Pure functions — no DB, no HTTP. | | `authn_store.py` | SQLite/Postgres backend for users/sessions/device tokens (`CYCLAW_AUTH_DB_URL`). | | `authn_manager.py` | `AuthManager` gluing `authn.py` + `authn_store.py`; no HTTP awareness. | @@ -35,7 +39,7 @@ modules"; this file groups them by concern. |---|---| | `personality.py` | Soul versioning, SHA-256 drift detection, injection scan + human-`reason` gate on write (invariant I5), atomic writes. | | `personality_db.py` | Soul DB backend: SQLite default, Postgres via `CYCLAW_DB_URL`. | -| `logger.py` | Audit JSONL: keyed HMAC-SHA256 query fingerprints, recursive PII/secret redaction, and private log/key files. `CYCLAW_QUERY_FINGERPRINT_KEY` overrides the persistent 0600 key file; otherwise the key is generated atomically at `policy.privacy.query_fingerprint_key_file` on first use. `setup_logging` rejects a short environment value, a bad existing key file, or a non-string key name. A missing file is not created at boot. Query fingerprinting is the default; setting `logging.audit_fields.include_query_hash: false` retains raw query text subject to redaction. `hash_query` remains ordinary SHA-256 for content-integrity identifiers. `audit_log` also projects each record — *after* fingerprinting and redaction — into the Numbat stream via a lazy, fail-soft `numbat_emitter` call, so the derived stream inherits the same privacy contract. `setup_logging` writes the application log (`logging.log_file`) from one bounded writer thread (`_BackgroundFileHandler`), so a stalled log disk cannot hold a thread that logs; past `logging.max_queued_records` waiting lines it drops and counts, and exit waits at most `logging.drain_wait_sec`. The gateway's console (stderr) is written the same way; an agentic CLI's stays synchronous. Like `FileHandler`, a closed handler reopens on its next line, since `uvicorn.run()`'s `dictConfig` closes every handler. | +| `logger.py` | Audit JSONL: keyed HMAC-SHA256 query fingerprints (default; `logging.audit_fields.include_query_hash: false` keeps redacted raw text instead), recursive PII/secret redaction, private log/key files. `CYCLAW_QUERY_FINGERPRINT_KEY` overrides the 0600 key file otherwise generated atomically at `policy.privacy.query_fingerprint_key_file` on first use; `setup_logging` rejects a short env value, a bad key file, or a non-string key name, and never creates a missing file at boot. `hash_query` stays plain SHA-256 for content-integrity ids. `audit_log` projects each record — *after* fingerprinting and redaction — into the Numbat stream via a lazy, fail-soft `numbat_emitter` call. The application log (`logging.log_file`) and the gateway's stderr console go through one bounded writer thread (`_BackgroundFileHandler`): past `logging.max_queued_records` it drops and counts, exit waits at most `logging.drain_wait_sec`, and a handler closed by `uvicorn.run()`'s `dictConfig` reopens on its next line. An agentic CLI's console stays synchronous. | | `errors.py` | Typed exception hierarchy rooted at `RAGError` (`.code`/`.message`/`.details`). Never raise bare `Exception`. | | `spend.py` | Append-only Grok/Claude token ledger (`logs/spend.jsonl`, `logging.spend_file`). Tokens are ground truth; dollars derived at read time by `metrics.py`. Separate stream from `audit.jsonl`; never persists query/prompt content. See [`spend/README.md`](../spend/README.md). | | `sequence_detect.py` | Offline forensic sequence detection over a join of `audit.jsonl` + `spend.jsonl`, keyed on the persistent HMAC-SHA256 `query_hash`. Correlates blocked-injection events against later online escalations inside a 15-minute window. Spend rows are restricted to `source == "query"` so the agentic ledger plane never mixes in. Findings carry fingerprints, event names, timestamps, and provider/model tags — never query text, IPs, soul content, or secrets. Forensic/CLI only: `gate.py`, `graph.py`, and the MCP server must not import it (it is not a `/query` policy point). | @@ -45,6 +49,7 @@ modules"; this file groups them by concern. | Module | Role | |---|---| | `ratelimit.py` | Per-IP rate limiting; in-memory / SQLite / Postgres backends. | +| `bounded_executor.py` | Runs blocking work with a deadline that still holds after the caller gives up: the worker is bounded until it finishes, so a timed-out call cannot leave an unbounded thread behind. | | `health.py` | `check_all()` behind `/health` probes the gateway's loaded config, including the resolved local backend and model pin. Its short-lived status cache is keyed by that config. Restart after config edits. `degraded` without the local model is normal; external-provider probes require `api.health_probe_external_providers: true` (ships false). | | `config_validation.py` | Boot-time config validation; fails fast on a broken `config.yaml`. | | `gateway_url.py` | Shared launcher console URL resolver: configured host/TLS plus the effective port; wildcard binds map to loopback browser destinations. Reads YAML without importing or starting the gateway. |