Repository navigation
[docs] - Resync the checker skills with their scripts, the hashed-lock install and the retrieval gate - #1643
Conversation
…k install and the retrieval gate Reads every checker skill against the code it describes and fixes what had drifted. invariant-guard claimed 47 assertions (the checker prints 46), a target-set check for pre_action_hook_router the script does not do, and a mutation self-test that severed a grok_fallback -> audit_logger edge graph.py no longer has, so the sed changed nothing and the assertion passed on the PASS line's wording; verify.sh now severs guardrail_output -> audit_logger, asserts the sed applied, and matches a FAIL line. dep-guard's D8 list gains nemo-guardrails.yml (which hardcodes the torch pin and was not scanned) and points its doc scan at docs/work/SETUP.md; its SKILL.md stops claiming a torch-cpu extra, a requirements.txt install path, and "does not track coverage wiring" (D10 does). config-guard documents C0 and mutation E, the real C2 severity (FAIL only when graph_timeout <= llm_timeout), what boot validation now covers, and the RRF ceiling instead of a 0.05-0.08 band. dotenv-guard's suffix rule now defers to utils/secret-policy.tsv (DB_URL and the exact names were missing from three copies), notes the empty baseline, the blocking CI verify step, and the chmod-000-as-root caveat. index-doctor states what doctor.py checks (RRF floor) versus the production gate (min_semantic_score plus the rerank veto), the 2/60 score ceiling, the /index/build route, and the dry-run default of clear_cache. verify-deps drops the uv story told three times, the [tool.uv.sources] references, the pyodbc-undeclared claim, the 14-site pip count, the eighteen-item verify list and the requirements.txt-carries-torch table row, and gains the hashed locks as the documented install surface. doc-sync loses its dated baseline section and points at CLAUDE.md §4 "Boot semantics". Install hints everywhere point at CLAUDE.md §8's locks or driver.sh venv, and docs/PROPOSED_SKILLS.md is cited at its real path docs/work/. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01LbrgG4NUbR3sUTgNMfKXpU
|
PR is a complete fill of |
Condenses verify-deps' E1/E7 narratives, the huggingface_hub/starlette promotion story and two gotchas, merges the duplicated plain/--strict command blocks in dep-guard and config-guard, and shortens injection-redteam's dry-corpus paragraph. No check ID, exit code or rule changes; every verify.sh still passes. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01LbrgG4NUbR3sUTgNMfKXpU
Codex Review SummaryThis comment shows the latest Codex review activity on this pull request.
ℹ️ About Codex in GitHubYour team has set up Codex to review pull requests in this repo. Reviews are triggered when you
Codex reacts with 👀 while any review is running, comments if it has suggestions, and reacts with 👍 once all reviews finish with no findings. |
![chatgpt-codex-connector[bot]](https://avatars.githubusercontent.com/in/1144995?s=60&v=4){kind=small}
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: 4c44693a58
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
| installed. In a fresh container it is not — build it with | ||
| `bash .claude/skills/cyclaw-gotchas/driver.sh venv` (the hashed `locks/` | ||
| install from CLAUDE.md §8, torch first). The embedding model must also be |
There was a problem hiding this comment.
In the cloud-sandbox environment this step targets, driver.sh venv creates /root/.venv-cyclaw-312 in a subprocess and cannot activate it in the caller, while the following command still uses bare python3; when the system interpreter lacks the retrieval dependencies, the documented preparation succeeds but doctor.py immediately exits 3. Invoke the doctor with the created venv's interpreter or explicitly activate that venv before Step 1.
AGENTS.md reference: AGENTS.md:L35-L37
Useful? React with 👍 / 👎.
| gate), asserts each sed actually changed the file, and asserts the checker | ||
| exits 2 and reports all four as `FAIL` lines. A checker that cannot fail |
There was a problem hiding this comment.
The new claim that all four mutations are reported as FAIL is not enforced by verify.sh: only the audit assertion searches for FAIL.*, while the import, retrieve-edge, and path-map greps also match their normal PASS lines. If any of those three checker rules regresses, mutation B still supplies exit 2 and the harness falsely prints that all four violations were detected; require a FAIL marker for each assertion and verify each sed applied.
AGENTS.md reference: AGENTS.md:L24-L31
Useful? React with 👍 / 👎.
| pip install --dry-run --require-hashes --no-deps -r locks/requirements-torch-lock-linux.txt --index-url https://download.pytorch.org/whl/cpu | ||
| pip install --dry-run --require-hashes -r locks/requirements-lock-linux.txt | ||
| pip install --dry-run --require-hashes -r locks/requirements-test-lock-linux.txt |
There was a problem hiding this comment.
In the fresh venv prescribed above, the first --dry-run only plans the torch installation and does not install it, so the next invocation runs with torch absent. Because the runtime lock deliberately omits torch while packages such as sentence-transformers depend on it, pip under --require-hashes must resolve an unhashed dependency and the documented verification can fail on valid locks. Combine the lock files into one dry-run with the CPU index, or actually install torch before separately dry-running the remaining locks.
AGENTS.md reference: AGENTS.md:L35-L37
Useful? React with 👍 / 👎.
| - **WARN is intentional friction, not a bug.** A changed `rrf_k`, a sub-30 s | ||
| timeout margin, or an operator changing the shipped provider posture are legitimate — the | ||
| WARN exists so the change is conscious and documented, not silent. Use | ||
| `--strict` only where you want the shipped defaults locked. |
There was a problem hiding this comment.
When an operator sets a positive but sub-30-second timeout margin, the documented default checker emits only a warning and exits 0, and this paragraph now explicitly calls that setting legitimate. validate_boot_timeout_config, however, unconditionally raises ConfigError for the same values, so the suggested interpretation can approve a configuration that cannot boot unless the operator happened to add --strict. Treat this C2 condition as a failure or at least stop describing it as an acceptable warning.
AGENTS.md reference: AGENTS.md:L5-L7
Useful? React with 👍 / 👎.
| | C5 | FAIL | `personality.soul_max_chars` is positive and **<** `retrieval.max_context_tokens * _CHARS_PER_TOKEN` (3 — mirrors `graph.py CHARS_PER_TOKEN`, the worst-case chars/token floor used to derive the character budget) so the soul cannot crowd out retrieved context | | ||
| | C6 | FAIL | `api.rate_limit.max_requests` / `window_seconds` are positive integers | | ||
| | C7 | WARN | `retrieval.min_score` stays on the RRF scale (≤ 0.1) — **the acknowledged trap**; a stricter gate belongs in 0.05–0.08, never the cosine-scale 0.5 | | ||
| | C7 | WARN | `retrieval.min_score` stays on the RRF scale (≤ 0.1) — **the acknowledged trap**. The hybrid ceiling is 2/60 ≈ 0.033 and the keyword-only degrade tops out at 1/60 ≈ 0.017, so anything above ~0.033 can never pass; a stricter vault-hit gate is `min_semantic_score` (cosine), never a bigger `min_score` | |
There was a problem hiding this comment.
For a value such as retrieval.min_score: 0.05, C7 currently reports no warning because _RRF_SANITY_CEILING remains 0.1, even though the newly documented 2/60 ceiling means no hybrid result can clear that value and the keyword-only path tops out even lower. This leaves the checker silent for the entire 0.033–0.1 range that the same row says can never pass; derive the warning ceiling from rrf_k or lower it to the real fused-score maximum.
AGENTS.md reference: AGENTS.md:L5-L7
Useful? React with 👍 / 👎.
| # Each query is answerable by the committed corpus (data/corpus/cyclaw_overview.md), | ||
| # phrased near corpus headings so it clears the gate at stable runtime. The | ||
| # "one sentence" probe mirrors CyClaw-Sandbox P13, the canonical vault-hit test. | ||
| # "one sentence" probe mirrors the vault-hit check CyClaw-Sandbox/smoke.sh runs. |
There was a problem hiding this comment.
The sandbox smoke does not run the one-sentence query: its direct local-path request is What is RRF fusion in CyClaw?, followed by What is CyClaw? on the explicitly declined-online path. Consequently, a regression specific to Describe CyClaw in one sentence. is covered only by index-doctor despite this comment and the matching SKILL.md text claiming duplicate smoke coverage; either use the actual shared query or add this probe to smoke.sh.
AGENTS.md reference: AGENTS.md:L49-L49
Useful? React with 👍 / 👎.
| "CLAUDE.md", | ||
| "AGENTS.md", | ||
| "docs/SETUP.md", | ||
| "docs/work/SETUP.md", |
There was a problem hiding this comment.
D8 now scans docs/work/SETUP.md, but that file only points readers back to the root guide; the actual setup-guide.md contains multiple hardcoded torch==2.13.0 and torch==2.13.0+cpu installation claims and is absent from _TORCH_DOC_FILES. A Torch bump can therefore leave the canonical platform instructions stale while dep-guard reports that all documentation agrees, so add setup-guide.md to the scanned paths.
AGENTS.md reference: AGENTS.md:L35-L37
Useful? React with 👍 / 👎.
| single pip path (no `||` fallback, no stderr redirect), never installs the | ||
| test lock into the image, and that the torch the torch lock pins is the |
There was a problem hiding this comment.
The newly documented E5 guarantee is not implemented: check_docker_install_contract() only searches for requirements-test\.txt, which does not match locks/requirements-test-lock-linux.txt. If a separate COPY and pip install --require-hashes -r locks/requirements-test-lock-linux.txt are added to the Dockerfile, the checker still exits 0 and reports that the image is test-tool-free; extend the rejection and its mutation test to cover the hashed test-lock filename.
Useful? React with 👍 / 👎.
…-13) High-signal since 3fe8217 (#1619): #1637 telemetry/egress inventory completion (fail-closed kill block, full spawn-site table for PowerShell/pip/brew/git-remote cat-3, agent-tooling boundary, OLLAMA_HOST pin, secret-env wrappers); #1643/#1645 docs resyncs correcting I6/harness and checker skill drifts. Graph remains 12 nodes.
2 participants
Proposed changes
Skill-catalog resync, part 2 of 3 (part 1 is #1642, the loop/persona skills; part 3 is #1645, the knowledge and session skills). This one covers the nine checker skills:
invariant-guard,dep-guard,config-guard,dotenv-guard,doc-sync,index-doctor,injection-redteam,otel-hardening,verify-deps. Each SKILL.md was read end to end and every claim checked against its own script,config.yaml,graph.py, the workflows and CLAUDE.md §4/§8.Three of the findings are in the scripts, not the prose, and are fixed here because they are small and each has a self-test that proves the fix:
invariant-guard/verify.shmutation B tested nothing. It sed-severedgrok_fallback -> audit_logger, an edgegraph.pyhas not had since the answer nodes converged onguardrail_output, so the file was unchanged and the assertiongrep -q "reach audit_logger"matched the checker's PASS line. It now seversguardrail_output -> audit_logger, asserts the sed applied, and matches aFAILline. Verified: the mutation is detected (all four injected violations detected, exit 2).dep-guardD8 did not scannemo-guardrails.yml, which hardcodes the torch pin (torch-cpu-2.13.0,torch==2.13.0+cpu), and listeddocs/SETUP.md, which does not exist (the file isdocs/work/SETUP.md). Both corrected; D8 now reports 5 CI files agreeing with the pin.check_config.py's PyYAML hint named the retired--ignore-installed PyYAMLinstall; its docstring described a boot validator that only range-checks (it now hard-fails the timeout margin, inline credentials, the reranker block and more);check_dotenv.pyandENVLINE_SPEC.mdlisted eight secret suffixes whileenvline.pydefers toutils/secret-policy.tsv(which addsDB_URLand exact names);check_env_drift.py's header omitted E7;doctor.pycited a "CyClaw-Sandbox P13" that does not exist.Prose corrections, by skill:
<n>/<n>); the I2 row no longer claims a target-set check forpre_action_hook_routerthat the script does not perform; the stale "min_score0.028 ≈ top-3-4 rank, 0.5 routes every query to the user gate" bullet now states the real gate (min_semantic_score+ rerank veto;min_scoregates the keyword-only degrade, hybrid ceiling 2/60);docs/PROPOSED_SKILLS.md→docs/work/PROPOSED_SKILLS.md.pip install -r requirements.txt -c constraints.txt --ignore-installed PyYAML; CLAUDE.md §8 is the hashedlocks/install. The "torch-cpu extra" gotcha described apyproject.tomlextra that does not exist (torch is a base dependency,+cpulives inconstraints.txt). "dep-guard does not track coverage wiring" contradicted its own D10 row. The verify description enumerated 8 mutations; the script runs 22. "Medium–High tier" is not a tier CLAUDE.md §7 has.gate.py.verify.shruns in the blocking CI job, that the shell loaders do not implementENVLINE_SPEC.md, and thechmod 000-as-root caveat.doctor.pychecks the RRF floor while production gates on cosine + reranker (soci_rag_smokeremains the gate test); fixes "scores of 0.03–0.06 are normal" (the ceiling is 0.0333);/index/buildexists now;cyclaw-clear-cacheis a dry run without--apply;verify.shindexes all ofdata/corpus/, not one file; install pointer →driver.sh venv.CONTRACT_DIGEST/_EXPECTED_CONTRACT_DIGEST(following the old step broke gateway boot); the blocking CI jobs are invariant-guard, doc-sync and dotenv-guard, not invariant-guard alone; the Guardrails section no longer restates CLAUDE.md §3. The spawn-site paragraph (unshare→bwrap, the Keychainsecuritybinary) is deliberately not touched: [security] - Classify the Keychain CLI and uv in the telemetry egress inventory #1637 rewrites it.requirements.txtcarries torch or that the Dockerfile installsrequirements.txt; E5 describes the lock-based Docker contract; Step 4's dry-run block follows §8; the uv/first-indexstory told three times is told once;[tool.uv.sources](not inpyproject.toml) and thepyodbc-is-undeclared claim (it is themssqlextra) are gone; the eighteen-item verify list is a paragraph (the script runs 24);--strictonextract_pins.pyis documented.test_contracts.py; a misplaced D8 parenthetical is back on D8.--strictcommand blocks in dep-guard and config-guard merged; injection-redteam's dry-corpus paragraph shortened. No check ID, exit code or rule changed; everyverify.shstill passes.Net: 339 insertions, 372 deletions across 16 files.
Invariant / Governance Impact: none of the six invariants or I6 changes; no file outside
.claude/skills/is touched. The invariant-guard mutation test is strengthened (it now actually breaks I4 in the scratch tree). The agent instruction surface changed, which the pre-commit security gate flagged as a reminder.Types of changes
Scope note: Docs + audits, plus three checker-script corrections under
.claude/skills/.Benefits / why
dep-guardnow sees the one workflow that still hardcodes the torch pin outsideci.yml.driver.sh venv.min_scoreguidance in three skills (and theconfig-guardband) would have led an agent to a value that can never pass; it now states the RRF ceiling and the real gate.Risks to monitor
dep-guardD8 now readsnemo-guardrails.yml; a future edit that changes its torch cache-key form would need the regex updated (D8 fails loudly, not silently).otel-hardeningspawn-site paragraph still saysunsharewhere the code usesbwrap; left for [security] - Classify the Keychain CLI and uv in the telemetry egress inventory #1637, which rewrites that paragraph. Trial-merged with [security] - Classify the Keychain CLI and uv in the telemetry egress inventory #1637 on top of main: clean.Checklist
docs/THREAT_MODEL.md,INVARIANTS.mdandSECURITY.md(and any relevant Phase docs)verify.sh(invariant-guard, dep-guard, config-guard, doc-sync, otel-hardening, injection-redteam, verify-deps: all OK, re-run after the trim),check_deps.py,check_config.py --strict,check_dotenv.py,check_env_drift.py --strict,check_otel.py --strict,doc_sync.py(0 drift), ruff on the touched Python (the two remaining findings predate this branch).index-doctoranddotenv-guard/verify.shcould not run here (no torch; root cannot be denied achmod 000read).Further comments
Findings deliberately left out of this PR:
doctor.pystill compares the fused score withmin_scorerather than walking the production gate (documented as a gap; changing it needs the retrieval stack installed to test);check_otel.py'sKNOWN_EXTERNAL_COMPONENTSmissingsecurityis fixed by #1637; thedotenv-guard/verify.shroot caveat is documented rather than coded around.Suggested merge order of open PRs
Trial merges verified (
git merge --no-commit --no-ff+ abort) against main @ 663a8bb on 2026-10-13, including this PR stacked on top of #1637's branch (the only other open PR touching a file this one edits).#1637 · Safe ✅
Impact: otel-hardening inventory; shares
otel-hardening/SKILL.mdwith this PR in disjoint hunks.Action: merge anytime; this PR merges cleanly after it.
#1638 · Safe ✅
Impact: docs/audits report; independent.
Action: merge anytime.
#1640 · Safe ✅
Impact: docs/audits report; independent.
Action: merge anytime.
#1641 · Safe ✅
Impact: CI bwrap/AppArmor workflow; independent.
Action: merge anytime.
#1642 · Safe ✅
Impact: loop/persona skill files; disjoint from this PR.
Action: merge anytime.
#1643 (this PR) · Safe ✅
Impact: checker skill files only.
Action: merge after #1637 or before it; both orders trial-merged clean.
#1645 · Safe ✅
Impact: knowledge/session skill files; disjoint from this PR.
Action: merge anytime.
ELI5
The repo has nine small "inspector" scripts, each with an instruction sheet. Several sheets described inspections the scripts don't do, quoted old install commands, or gave a retrieval tuning number that could never work. Worse, one self-test for the most important inspector (
invariant-guard) had been quietly passing for a while because it tried to cut a wire that no longer exists, so it never actually proved the inspector could catch a broken audit path. This PR fixes the sheets to match the scripts, makes that self-test cut a real wire, teaches the dependency inspector about one workflow it had been ignoring, and then shortens the sheets' repeated stories. Nothing in the running server changes. What could still go wrong: the sheets quote how many self-test scenarios each script runs, and those counts will drift as scripts grow; the sentences say so, and the scripts print their own tallies, so trust the tally over the sheet.🤖 Generated with Claude Code
https://claude.ai/code/session_01LbrgG4NUbR3sUTgNMfKXpU
Generated by Claude Code
Last updated: 2026-10-13 02:44 ET