diff --git a/.gitattributes b/.gitattributes index 63d068761..380592018 100644 --- a/.gitattributes +++ b/.gitattributes @@ -1,6 +1,7 @@ flake.lock linguist-generated=true bun.lock linguist-generated=true machines/*/facter.json linguist-generated=true +modules/home/ai/evals/harborize/*-harbor/** linguist-generated=true packages/docs/src/grammars/*.json linguist-vendored=true *.sh linguist-detectable=false scripts/** linguist-detectable=false diff --git a/.gitignore b/.gitignore index 03645feb5..38649fe50 100644 --- a/.gitignore +++ b/.gitignore @@ -27,6 +27,10 @@ yarn-debug.log* yarn-error.log* pnpm-debug.log* +# Python bytecode +__pycache__/ +*.pyc + # wrangler (Cloudflare Workers) .wrangler/ .dev.vars* diff --git a/modules/home/ai/evals/harborize/README.md b/modules/home/ai/evals/harborize/README.md new file mode 100644 index 000000000..78a641b84 --- /dev/null +++ b/modules/home/ai/evals/harborize/README.md @@ -0,0 +1,350 @@ +# Harborize evaluation workspace + +Change-owned corpus for the `validate-harborize-instrument` OpenSpec change. +It validates the harborize instrument at version 0.2.1 by producing evaluation packages whose skill injection is proven at each level it can fail, and it stays in the repository after the change closes as the permanent home of the injection canary and the evaluation corpus. + +This file has two halves and they have different lifetimes. +Everything down to "Change-scoped rung evidence" outlives the change. +Everything below that heading is execution evidence for one change and is migrated into `verify.md` and deleted when the change archives, which task 10.6 carries. + +## Upstream pins and installed versions + +Every source anchor in the change documents is read at these pinned revisions in the local ghq clones, whose HEADs were confirmed equal to the pins: + +| repo | path | pin | +|---|---|---| +| Harbor | `~/ghq/github.com/harbor-framework/harbor` | `ac398bbda7c4c1073461797d3b95c2455cc671b5` | +| BenchFlow | `~/ghq/github.com/benchflow-ai/benchflow` | `d30527b82027a416e72014920cdf43a534967ad3` | +| SkillsBench | `~/ghq/github.com/benchflow-ai/skillsbench` | `9a1f4dd5f7659f75707435da3ce854b6e48321d1` | + +The pins are reading pins, not install pins: CLI installs track PyPI latest stable (settled 2026-08-15), so the executed CLIs may drift from the anchors. + +| CLI | installed | pin's version | +|---|---|---| +| `harbor` | 0.21.0 | 0.21.0 (equal at rung 0 time) | +| `bench` | benchflow 0.7.4 | 0.6.8.dev0 (drifted; anchors cite the pin) | + +Where an anchor was re-read in the installed tree rather than the pin, the text citing it says so. + +## Instrument freeze baseline + +Instrument version: 0.2.1, frozen for the duration of the change. + +The freeze anchor is the git tree object the instrument path resolves to: + +``` +64bc599cba1680db3678b67aea187bb3da0f6d20 +``` + +Recompute and compare it with `git rev-parse 'HEAD:modules/home/ai/plugins/testing-and-quality/.apm/skills/harborize'`. +This is the primary check because it records file modes and symlinks, is immune to untracked files and to how the path was spelled on the command line, and is stable across a rebase of the base branch — it is identical at the pre-rebase tip `5f33c36e`, at the current `harborize-instrument` tip, and at this branch's head, where a commit id is not. +A commit id is deliberately not the anchor for that last reason; the baseline this change started from has already been rebased once. + +A `jj diff -r @` cannot substitute, because it reports only what the working-copy commit changed against its parents and would show nothing for an edit squashed into the `harborize-instrument` chain. + +Secondary working-tree check, over the same path (13 files): + +``` +3fdd30d1fa2a69a5e53c8d34474c107a516c32e29dcc5087b9bd7738b22ccd4e +``` + +``` +find modules/home/ai/plugins/testing-and-quality/.apm/skills/harborize -type f \ + -not -path '*/__pycache__/*' -exec shasum -a 256 {} + | LC_ALL=C sort | shasum -a 256 +``` + +Both this recipe and the digest live here rather than only in the change's tasks.md, because they outlive the change while tasks.md archives. +Three limits are worth stating, because each was reproduced rather than predicted, and together they are why this is the secondary check. +It hashes regular files only, so a mode change, a new symlink or a new empty directory inside the instrument leaves it invariant. +It covers every file present rather than every file tracked, so any untracked file inside the directory flips it, and several such classes are gitignored and so invisible to `git status` too — `.DS_Store` is the sharpest on macOS, where one Finder visit plants one, and `*.log`, `result*`, `*~`, `*.swp` and a `.pyc` outside a `__pycache__` directory behave the same. +And it is a function of the path string `find` was given, because each `shasum` line embeds it: the value above is what this exact invocation produces from the repository root, while a `./` prefix, an absolute path, or a `cd` into the directory each produce a different digest from an unchanged tree. +A trailing slash does not, because `find` normalizes it. + +The `__pycache__` exclusion is still required: the change is obliged to run the instrument's own `scripts/*.py`, CPython writes bytecode beside them, and `.gitignore` then hides it, so without the exclusion complying with one task makes another report a freeze violation that never happened. + +## Kernel probe outcome + +Probe command, run immediately after the daemon started and before any task was authored, with the anchored `grep -qE` whose exit status is the criterion: + +``` +docker run --rm alpine sh -c ' + [ -f /proc/config.gz ] || exit 2 + zcat /proc/config.gz | grep -qE "^CONFIG_NFT_FIB_INET=[ym]"' +``` + +Exit status: 0, on 2026-08-18. + +Branch taken (task 1.3, exit-0 arm): egress control is available, so `no-network` may be declared anywhere the change declares it. +Every package keeps `network_mode = "public"` at the environment baseline and carries the `no-network` override on the `[agent]` phase only, per task 5.4. + +Attribution recorded with the result (task 1.4): `no-network` is the harborize instrument's own authoring default (`SKILL.md:124`, `references/emitters.md:74`), not Harbor's. +Harbor's default is `public` (`models/task/config.py:249-252`, `NetworkPolicy` at `:66`), BenchFlow's is the same (`task/config.py:720-723`), and 86 of the 87 SkillsBench corpus tasks declare `network_mode: public`. +What the probe gates is whether a `no-network` declaration can be enforced at all: on failure `_enable_egress_control` goes false (`docker.py:188-195`), which zeroes `capabilities.disable_internet` (`:289-293`), and `environments/base.py:773-781` raises at environment start. + +## Agent install caveat + +`install()` curls its bootstrap (`claude_code.py:425-449`) during `_prepare`/`_setup_agent` (`trial.py:408-414`), which no network policy wraps, while only `_run_agent_phase` (`trial.py:465-469`) and the verifier phases enter `_phase_network_policy`. +A cell whose environment baseline is `no-network` therefore fails during agent install indistinguishably from an injection failure at the reward level, which is why task 5.4 puts `no-network` on the `[agent]` phase rather than on the baseline. +The anchors above are claude-code's, but the mechanism is the phase boundary rather than the adapter: every installed adapter's install runs inside `_prepare`/`_setup_agent`. + +## Layout + +Job output goes to `logs/harborize/` at the repository root, which `.gitignore:57` already excludes; nothing under this workspace writes logs inside the corpus. +No file in this workspace carries the `.nix` extension, because `flake.nix:6` calls `inputs.import-tree ./modules` bare and every `*.nix` file under `modules/` is evaluated as a flake-parts module — a corpus fixture with that extension would break the flake rather than fail as a fixture. + +### Flake-evaluation guard + +Run whenever a task writes new corpus files (tasks 1.11, 5.10): + +``` +if fd -H -e nix . modules/home/ai/evals/harborize | rg -q .; then + echo "FAIL: a .nix file is inside the corpus"; exit 1 +fi +nix eval .#nixosConfigurations --apply builtins.attrNames +``` + +What the eval proves and does not prove: `builtins.attrNames` forces the attrset spine and never the values, so it proves that import-tree's enumeration over `modules/` still succeeds and that every module file parses — which is exactly the hazard the extension audit guards — and it evaluates no module body. +It is therefore not evidence that the nix skill composition or its exclusion list evaluates. +The eval that would exercise those is `.#homeConfigurations."crs58@".config.programs.claude-code.skills`, which triggers an import-from-derivation and is out of scope for a no-build guard. +The darwin arm has the identical scope. + +## Deliverable channel + +Both packages route their oracle-to-verifier deliverable through `/logs/artifacts/`, and neither uses `/logs/verifier/` for anything but the reward file the verifier itself writes. + +`/logs/verifier/` is not usable as that channel, which the rung-4 run established as a fact rather than a reading. +BenchFlow wipes the directory's contents immediately before the verifier runs: `harden_before_verify` executes `_CLEAR_VERIFIER_DIR_CMD` (`sandbox/lockdown.py:775-784`, called at `:1205-1212`) unconditionally on the agent container, so the bind mount survives and the content does not. +`_verify_test_script`'s own conditional clear (`task/verifier_core.py:360-372`) is the branch that does respect a mount, and it is not the branch that removes the file. +Under Harbor a shared verifier performs no wipe, so a package written against Harbor alone passes there and scores 0 under BenchFlow with no error and no diagnostic beyond a `grep` miss. + +`/logs/artifacts/` is bind-mounted by both runners for the whole trial — BenchFlow at `sandbox/docker.py:186-201`, Harbor at `models/trial/paths.py:38`, `:195-202` — and neither hardening step touches it. +It is also the one channel that would survive Harbor's separate fork, whose wipe is followed by an artifact re-upload (`trial.py:601-607`, `artifact_handler.py:210-254`); that is a property of the path, not a fork this corpus exercises. + +## Generated Harbor heads + +Each `-harbor/` directory is emitted by `bench tasks export` and is never hand-edited; re-export after every edit to the authored tree. +Two properties of the exporter are worth knowing before reading one of them. + +No generation marker can be written into the emitted files. +The exporter copies `environment/`, `oracle/` and `verifier/` verbatim and rebuilds `task.toml` with `tomli_w`, which emits no comments (`benchflow/task/export.py:268-273`, `:367-388`), so a header would vanish on the next export and the committed tree would stop equalling a fresh one. +The marker therefore lives outside the emitted content, as `.gitattributes` `linguist-generated=true`, alongside this repository's other machine-emitted files. + +`compatibility/export-report.json` records `source_task_dir` as the exporting checkout's absolute path and cannot be made relative: `TaskPaths.__init__` calls `Path(task_dir).resolve()` (`benchflow/task/paths.py:55-56`) and no CLI flag controls it. +That field is provenance of one export run on one machine, not a portable reference, and re-exporting from a different checkout rewrites it. +The `losses: []` beside it means lossless over the contract files the exporter enumerates, which does not include the authored package README. + +## Base image pinning + +Every Dockerfile in the corpus pins its base by index digest, `ubuntu:24.04@sha256:561618e2c15bf2397621dd04f96926663a3b5616c189cf7e38db7e82f5c538ea`, which is the image the recorded evidence below was produced against. +The pin is not decorative: at rung-0 time the floating `ubuntu:24.04` tag had already moved to a later build than the one this host held, so an unpinned instrument would compare measurements taken in different environments. +The multi-arch index digest is pinned rather than a platform manifest, so the same line resolves correctly on `linux/arm64` and `linux/amd64`; a platform digest carries one architecture only and would either fail to resolve or silently emulate. + +## Verifier forks + +Both packages run a shared verifier. +Neither declares `verifier.sandbox_mode` and neither ships a `verifier/Dockerfile`, and the choice is forced rather than preferred — see design decision D11 and each package README for the evidence. +The short form: BenchFlow refuses at launch to run a package declaring separate rather than falling back to shared (`runtime_capabilities.py:186-192`, `sandbox/setup.py:676`, `:819-842`), and Harbor's separate verifier empties `/logs/verifier` before running (`trial.py:599`), which destroys the channel both packages use. +Separate mode is exercised nowhere in this corpus. + +Record a fork by resolving it from the exported head rather than by reading the source layout. +Harbor derives the mode from `verifier.environment_mode` or `[verifier.environment]` and from nothing else (`models/task/verifier_mode.py:10-21`), so a `verifier/Dockerfile` on its own infers nothing. + +## What `bench tasks check` does and does not clear + +Both packages pass `--level schema`, `--level structural` and `--level runtime-capability --sandbox docker`. +Neither passes `--level publication-grade`, and the reason is unrelated to the co-present-head argument design decision D7 makes. +D7's checks do pass: `structural_checks.py:208-212` (no `task.toml` or `instruction.md` beside `task.md`) and `:218-222` (no `solution/`) both hold, because each Harbor head is a sibling directory rather than co-present. +The failing check is `:224-226`, which requires a native `verifier/` package carrying `verifier.md` and `rubrics/` — a verifier-strategy document neither package authors. +Recorded so publication-grade is not later cited as a standard this corpus meets. + +## Harbor's static gate and its blind spot + +`Task._validate_tests` returns early whenever a verifier environment is configured (`models/task/task.py:126-144`, early return at `:134-135`), so it structurally cannot catch a separate-mode package missing `/tests/test.sh`. +Both heads in this corpus resolve to shared and therefore do not reach that early return; the record is a source reading against harbor 0.21.0, kept because it bounds what `Task()` construction is evidence of. + +## Standing precondition + +The injection canary is re-run at the start of every later evaluation round, before any metered batch. +It is a positive control for skill delivery, and every silent-null class the instrument documents produces a clean run and a plausible negative result without it. + +--- + +# Change-scoped rung evidence + +Records for the `validate-harborize-instrument` change only. +Task 10.6 migrates this section into `verify.md` and deletes it at archive time. + +## Rung 0 — prerequisites (complete) + +- Docker daemon: answering (`docker info` exit 0) before the probe ran. +- Kernel probe: exit 0, recorded above with its branch and attribution. +- CLIs: `harbor` 0.21.0 and `bench` (benchflow 0.7.4) on `PATH` from PyPI, with + all three ghq clones verified clean at their pins after install. +- Flake-evaluation guard: run at this rung's close, after this README became the + first corpus file under `modules/`. Result: no `.nix` file found, `nix eval` + exit 0 on both `nixosConfigurations` and `darwinConfigurations`. + +## Canary skill and condition directory (tasks 2.1-2.4) + +`conditions/canary/` is `dir(C)`: exactly one skill directory, `harborize-injection-canary`, carrying the token `HARBORIZE-CANARY-9F3A21`. +The name collides with none of the deployed skill directories, and the directory holds nothing else. +A stray non-hidden child *directory* without a `SKILL.md` turns the whole condition into a hard error at resolution (`_find_skill_dirs`, `skills.py:382-416`), which is the case that motivated keeping it empty; a stray *file* is filtered out silently by the `child.is_dir()` test at `skills.py:396`, so the emptiness is maintained rather than enforced. + +Leakage-audit expectation (task 2.3): the asserted literal appears in both the verifier and the `SKILL.md` by design, and `audit_leakage.py` check 1 flags exactly that pattern (`MIN_LITERAL_LENGTH` at `:44`, `check_literals` at `:96`). +The flag is correct on its own terms — the canary's mechanism is its answer key — and the instrument is not edited to exempt it because it is frozen at 0.2.1. + +Exposure of the shared fork (task 2.4), corrected in the review round: the consequence originally recorded — that a real agent could read the token out of the verifier script — does not arise. +Both runners upload the verifier's own directory during the verification phase, after the agent phase has ended (Harbor `verifier/verifier.py:147-153` reached from `_run_shared_verifier`, phase order fixed at `trial/single_step.py:41` then `:52`; BenchFlow `task/verifier_core.py:385` in `_verify_test_script` (`:346`), reached from `verify()` (`:260`)). +Rung 6's criterion is unchanged and rests on a different reason: a model-driven trial's reward conflates delivery with the model's own behaviour, so the adapter's registration directory is the deterministic witness. + +## Rung 1 — adapter allowlist gate (tasks 3.1-3.3) + +`scripts/design_matrix.py` with `--units harborize-injection-canary --design marginals`, three runs against throwaway `/tmp` outputs: + +- `cells/cells-nonconsuming.json` (agent `aider`): exit 1 naming `aider` and + the reason, `/tmp/design-neg` never created. +- `cells/cells-acp.json` (agent `acp:claude-agent`): exit 1 on the ACP + shorthand, `/tmp/design-acp` never created. +- `cells/cells.json` (the codex cell this change will use): exit 0, + `conditions.json`, `manifest.sh`, `jobs.json` written to `/tmp/design-ok` + and then discarded — nothing the gate emitted was executed, committed or + consumed by a later rung, keeping the whole rung inside the proposal's + condition-lattice Non-goal. + +## Rung 2 — host-side resolution (tasks 4.1-4.3) + +`checks/resolve_check.py` calls `harbor.skills.resolve_skills` (`skills.py:111-123`) and `compute_skill_digest` (`skills.py:200-209`) against `conditions/canary`, run with the interpreter backing the `harbor` entrypoint: + +``` +HARBOR_PY=$(sed -n '1s|^#!||p' "$(command -v harbor)") +"$HARBOR_PY" modules/home/ai/evals/harborize/checks/resolve_check.py +``` + +- `dir(C)`: exactly one entry, `name` `harborize-injection-canary`, `digest` + `sha256:47016a2e2b3f220c90fc183411a8ae8dbd2f37c4c6becc268460a5588ba85cd9`. +- Missing path: `FileNotFoundError` on the host, exit 1. +- A file rather than a directory: `ValueError: Skill path must be a directory`, + exit 1. +- A child directory without a `SKILL.md`: `ValueError` naming `not-a-skill`, + exit 1. +- No argument: a usage line on stderr, exit 2. + +Each raise happened on the host before any container started (`_find_skill_dirs`, `skills.py:382-416`). + +What this rung proves is host-side resolution and request, never delivery: a trial's `lock.json` cannot substitute for delivery evidence, because `_write_trial_lock` runs at `trial.py:104` inside `Trial.__init__`, before `_resolve_injected_skills` at `:107` and long before `_upload_injected_skills` at `:411`, and `_build_agent_skill_locks` (`models/job/lock.py:462-475`) calls only host-side functions. + +## Task package authoring, dual head (tasks 5.1-5.10) + +Two authored BenchFlow-native trees and two derived Harbor heads: + +- `injection-canary/`: shared verifier fork, oracle output at + `/logs/verifier/canary-output.txt`, `environment/skills/` empty with a + `.gitkeep`, `sandbox.network_mode: public` baseline with the `no-network` + override on `[agent]` only (probe exit 0 permits it). +- `pipeline-event-summary/`: the mechanical package sampling the + `preferences-json-querying` skill's claimed contract, single binary reward + key, shared verifier fork, verifier expectation held as a hand-derived literal + rather than recomputed from a fixture copy; fork choice and reward channel + recorded in its README. +- Both heads exported with `bench tasks export ... --target harbor --overwrite`, + status lossless, 0 losses, reports under `compatibility/export-report.json`; + the exported `task.toml` carries `[environment] network_mode = "public"` with + the `[agent]` phase override, and the heads are never hand-edited. +- Fork verified from the heads rather than the layout: + `resolve_task_verifier_mode(Task().config)` is + `VerifierEnvironmentMode.SHARED` for both, matching what each README records. +- Leakage audit (task 5.8): mechanical exit 0 — after renaming the summary key + `pipelines` to `per_pipeline`, because the skill's prose contains the word + `pipelines` and the frozen check 1 flags any quoted verifier literal of eight + characters or more recoverable from skill content; canary exit 1 with exactly + the expected check-1 literal flag naming the token, justification in the + package README. +- Extension audit and flake-evaluation guard re-run after export (task 5.10): + no `.nix` file inside the corpus, `nixosConfigurations` and + `darwinConfigurations` eval exit 0. + +## Rung 3 — static task validation (tasks 6.1-6.3) + +- `bench tasks check --level structural`: exit 0, no issues, on both + `injection-canary` and `pipeline-event-summary`. +- `bench tasks check --level runtime-capability --sandbox docker`: exit + 0 on both, which is what establishes that neither package declares a feature + BenchFlow would refuse to launch. +- `harbor.models.task.task.Task()` constructed both exported heads + successfully (exit 0 each). The stub `harbor task(s) check` and its redirect + to the metered `harbor check` LLM-rubric were not invoked. +- Harbor gate blind spot (task 6.3): recorded above as a permanent note, with + the correction that neither head in this corpus reaches the early return. + +## Host-side container dry runs, beyond the task list + +Each oracle and verifier pair run under the pinned base image before any runner touches them. +These are not a rung's pass criterion; they are the re-verification of the artifacts the review round changed. + +- Mechanical, shared-fork layout: oracle output scores 1; a summary with one + digit of the median altered scores 0; a missing `summary.json` scores 0. +- Canary: with `dir(C)` mounted at `/harbor/skills` reward 1 and the token + written; with no skill mounted reward 0 and an empty output file. +- Canary with `HOME` unset in the container: the oracle exits 0 and writes its + output file, where it previously aborted under `set -u` before writing + anything — a failure the verifier could not distinguish from a genuine + injection failure. + +## Rung 4 — delivery proof under BenchFlow (tasks 7.1-7.5) + +Every run below is `--agent oracle --sandbox docker` with zero model calls, against benchflow 0.7.4. +Each run uses its own `--jobs-dir`: `bench eval run` resumes a completed rollout found in an existing jobs directory and reports the cached result (`1 resumed` in the progress line), so reusing one directory across runs silently re-reports the earlier score. + +- 7.1 canary, `--skill-mode with-skill --skills-dir conditions/canary`: reward 1, + 0 errored. The first attempt scored 0 and is what uncovered the + `/logs/verifier` channel defect recorded above; the run recorded here is + against the corrected package. +- 7.2 same rollout: `effective_skills_dir` and `requested_skills_dir` both equal + the host `conditions/canary` that was passed, `error` null, and no rollout + raised `experiment_fidelity/skill_deployment_missing`. +- 7.3 falsifiability control, `--skill-mode no-skill` and no `--skills-dir`: + reward 0, with `skill_source` `none` and both skills-dir fields null. The + canary can fail, so 7.1 is evidence rather than a constant. +- 7.4 fidelity control: `experiment_fidelity/skill_deployment_missing` raised, + naming `expected=extra-skill,harborize-injection-canary`. The construction the + task prescribed — a hand-authored `environment/_deps/skills/` plus a + hand-written `COPY _deps/skills /skills/` — cannot provoke the assertion at + 0.7.4, and that is the finding rather than an execution detail. BenchFlow + itself stages the host skills directory into `environment/_deps/skills` and + appends the `COPY` line, working on a temp copy of the task + (`sandbox/setup.py:529-568`, `rollout/__init__.py:979-1016`), so a + hand-authored baked set is overwritten before the build. The runtime link step + then runs `rm -rf && ln -sfn /skills ` + (`agents/install.py:61-90`), which destroys a pre-baked discovery directory + too: a second attempt that baked a decoy at `/root/.claude/skills` also scored + 1. The control that does provoke it puts a symlinked skill directory in the + condition directory: `deploy_skills` computes `expected` from the host glob, + which follows the symlink, while the staging copy drops symlinked entries by + design (`_stage_ignore`, `sandbox/setup.py:44-53`, the #411 fix), so the + in-container catalogue is genuinely short one skill and + `_link_skill_paths` (`:139-181`) raises. +- 7.5 one `dir(C)` shape exists in this corpus, `conditions/canary`, so 7.1 + covers it. + +Beyond the task list, the mechanical package was re-run the same way after the channel change (`--skill-mode no-skill`, which is its condition): reward 1, 0 errored. +The throwaway packages and condition directories built for 7.4 lived outside the repository and were deleted after the runs. + +Candidate instrument-defect entry for task 10.4, recorded here because section 10 is a later task: `references/emitters.md:330-340` presents `/logs/verifier` only as the reward-file path, which is correct, and the instrument nowhere warns that the same directory is cleared before the verifier under BenchFlow. An emitted task that used it as a deliverable channel would score 0 under one runner and 1 under the other. + +## Rung 5 — Harbor oracle inhabitation (tasks 8.1-8.3) + +`harbor run -p -k 5 -o logs/harborize/gate1 --job-name -y`, harbor 0.21.0, oracle by default, zero model calls. + +- 8.1 `pipeline-event-summary-harbor`: 5 trials, reward 1.0 on all five, + `n_errored_trials` 0, pass@2/4/5 all 1.0. +- 8.1 `injection-canary-harbor`: 5 trials, reward 1.0 on all five, + `n_errored_trials` 0. The canary head needs the condition directory on the + command line (`--skill conditions/canary`); without it the same head scores 0 + across five trials, which is the Harbor-side counterpart of the 7.3 control + and was run first. +- 8.2 no `agent/exit-code.txt` exists under any of the fifteen trial + directories. Absence is the passing reading: `OracleAgent.run` writes that file + only when `result.return_code != 0` (`agents/oracle.py:149-151`). +- 8.3 no trial errored in any of the three jobs, so the triage path was never + entered and no exception name was read. Recorded as vacuous rather than as + performed. diff --git a/modules/home/ai/evals/harborize/cells/cells-acp.json b/modules/home/ai/evals/harborize/cells/cells-acp.json new file mode 100644 index 000000000..1c477cad5 --- /dev/null +++ b/modules/home/ai/evals/harborize/cells/cells-acp.json @@ -0,0 +1 @@ +[{"name": "neg-acp", "runner": "harbor", "agent": "acp:claude-agent", "model": "anthropic/claude-opus-5"}] diff --git a/modules/home/ai/evals/harborize/cells/cells-nonconsuming.json b/modules/home/ai/evals/harborize/cells/cells-nonconsuming.json new file mode 100644 index 000000000..29d307cc8 --- /dev/null +++ b/modules/home/ai/evals/harborize/cells/cells-nonconsuming.json @@ -0,0 +1 @@ +[{"name": "neg-aider", "runner": "harbor", "agent": "aider", "model": "anthropic/claude-opus-5"}] diff --git a/modules/home/ai/evals/harborize/cells/cells.json b/modules/home/ai/evals/harborize/cells/cells.json new file mode 100644 index 000000000..596d8aa11 --- /dev/null +++ b/modules/home/ai/evals/harborize/cells/cells.json @@ -0,0 +1 @@ +[{"name": "codex-sol", "runner": "harbor", "agent": "codex", "model": "gpt-5.6-sol", "env": {"HARBOR_TELEMETRY": "0", "CODEX_FORCE_AUTH_JSON": "1"}}] diff --git a/modules/home/ai/evals/harborize/checks/resolve_check.py b/modules/home/ai/evals/harborize/checks/resolve_check.py new file mode 100644 index 000000000..d8fb49581 --- /dev/null +++ b/modules/home/ai/evals/harborize/checks/resolve_check.py @@ -0,0 +1,33 @@ +"""Resolve a condition directory through the installed Harbor CLI's own skills API. + +Run with the interpreter that backs the `harbor` entrypoint rather than a bare +`python3`, because the proposition this check witnesses is that the *installed* +CLI resolves the directory: + + "$(sed -n '1s|^#!||p' "$(command -v harbor)")" checks/resolve_check.py + +The house PEP-723 header is deliberately absent. `uv run --script` always builds +an isolated environment, so a `dependencies = ["harbor"]` declaration would +resolve a second copy of Harbor and stop measuring the CLI under test. +""" + +import json +import sys + +from harbor.skills import compute_skill_digest, resolve_skills + + +def main() -> int: + if len(sys.argv) != 2: + print(f"usage: {sys.argv[0]} ", file=sys.stderr) + return 2 + resolved = resolve_skills([sys.argv[1]]) + print(json.dumps([ + {"name": s.name, "source": str(s.source), "digest": compute_skill_digest(s.source)} + for s in resolved + ], indent=2)) + return 0 + + +if __name__ == "__main__": + sys.exit(main()) diff --git a/modules/home/ai/evals/harborize/conditions/canary/harborize-injection-canary/SKILL.md b/modules/home/ai/evals/harborize/conditions/canary/harborize-injection-canary/SKILL.md new file mode 100644 index 000000000..38db7c14b --- /dev/null +++ b/modules/home/ai/evals/harborize/conditions/canary/harborize-injection-canary/SKILL.md @@ -0,0 +1,10 @@ +--- +name: harborize-injection-canary +description: Injection canary for the harborize instrument. Carries a single token that a canary task's oracle extracts and its verifier asserts, so a failure to deliver injected skills fails a task rather than silently voiding a batch. +--- + +# Harborize injection canary + +This skill exists to be delivered, not to be used. + +The canary token is HARBORIZE-CANARY-9F3A21 and it appears nowhere else in the deployed tree. diff --git a/modules/home/ai/evals/harborize/injection-canary-harbor/compatibility/export-report.json b/modules/home/ai/evals/harborize/injection-canary-harbor/compatibility/export-report.json new file mode 100644 index 000000000..76786a1ca --- /dev/null +++ b/modules/home/ai/evals/harborize/injection-canary-harbor/compatibility/export-report.json @@ -0,0 +1,34 @@ +{ + "emitted_files": [ + "environment/Dockerfile", + "environment/skills/.gitkeep", + "instruction.md", + "solution/solve.sh", + "task.toml", + "tests/test.sh" + ], + "input_hashes": { + "environment/Dockerfile": "8692c0b78579ac8fb84ec5c5fa9c3341eecfe766da41b1433c586e4bfae05197", + "environment/skills/.gitkeep": "e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855", + "oracle/solve.sh": "b08466724dd8c2719dd74faab2a8c05a1eced3f26fa3ede48befa6f0752fa16f", + "task.md": "13d6b3e3fdebd7c71d3e4c234935575190d822ca45396af5ddd00a19829af347", + "verifier/test.sh": "452e9202c46294f89e839395a8634505d6c8b1189d461ea348c5a3de6efedf75" + }, + "losses": [], + "output_hashes": { + "environment/Dockerfile": "8692c0b78579ac8fb84ec5c5fa9c3341eecfe766da41b1433c586e4bfae05197", + "environment/skills/.gitkeep": "e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855", + "instruction.md": "7b950b03fd156f60ab8e54f47bdc2831d652ce8eecb0483348f9ac8af6fdda3a", + "solution/solve.sh": "b08466724dd8c2719dd74faab2a8c05a1eced3f26fa3ede48befa6f0752fa16f", + "task.toml": "314bbffd78b7daab9f4e1d7fee64fc45fc50906f62418551936c37da85cd0584", + "tests/test.sh": "452e9202c46294f89e839395a8634505d6c8b1189d461ea348c5a3de6efedf75" + }, + "restored_extension_paths": [], + "selected_entrypoint": "task.md", + "selected_oracle_dir": "oracle", + "selected_prompt": "instruction.md", + "selected_verifier_dir": "verifier", + "source_task_dir": "/Users/crs58/.treehouse/vanixiets-22808a/6/vanixiets/modules/home/ai/evals/harborize/injection-canary", + "status": "lossless", + "target": "harbor" +} diff --git a/modules/home/ai/evals/harborize/injection-canary-harbor/environment/Dockerfile b/modules/home/ai/evals/harborize/injection-canary-harbor/environment/Dockerfile new file mode 100644 index 000000000..73a824557 --- /dev/null +++ b/modules/home/ai/evals/harborize/injection-canary-harbor/environment/Dockerfile @@ -0,0 +1,3 @@ +FROM ubuntu:24.04@sha256:561618e2c15bf2397621dd04f96926663a3b5616c189cf7e38db7e82f5c538ea + +WORKDIR /root diff --git a/modules/home/ai/evals/harborize/injection-canary-harbor/environment/skills/.gitkeep b/modules/home/ai/evals/harborize/injection-canary-harbor/environment/skills/.gitkeep new file mode 100644 index 000000000..e69de29bb diff --git a/modules/home/ai/evals/harborize/injection-canary-harbor/instruction.md b/modules/home/ai/evals/harborize/injection-canary-harbor/instruction.md new file mode 100644 index 000000000..a07276ee4 --- /dev/null +++ b/modules/home/ai/evals/harborize/injection-canary-harbor/instruction.md @@ -0,0 +1,3 @@ +Find the canary token and make it the exact content of `/logs/artifacts/canary-output.txt`, as a single line, deduplicated and sorted, with no other output in that file. + +The token is not in this instruction. diff --git a/modules/home/ai/evals/harborize/injection-canary-harbor/solution/solve.sh b/modules/home/ai/evals/harborize/injection-canary-harbor/solution/solve.sh new file mode 100755 index 000000000..cbaf029a2 --- /dev/null +++ b/modules/home/ai/evals/harborize/injection-canary-harbor/solution/solve.sh @@ -0,0 +1,19 @@ +#!/bin/bash +set -euo pipefail +out=/logs/artifacts/canary-output.txt +mkdir -p /logs/artifacts +: > "$out" +# These roots are the oracle discovery-path union, not the adapter registration +# destinations; no adapter registration runs on the oracle path. +# HOME is defaulted because under set -u a container exporting none aborts here +# before any output is written, which reads downstream as a false canary alarm. +home=${HOME:-/root} +for root in /harbor/skills "$home/.claude/skills" "$home/.codex/skills" \ + "$home/.opencode/skills" "$home/.agents/skills" /skills; do + [ -d "$root" ] || continue + for f in "$root"/*/SKILL.md; do + [ -f "$f" ] || continue + grep -ho 'HARBORIZE-CANARY-[A-Z0-9]*' "$f" >> "$out" || true + done +done +sort -u -o "$out" "$out" diff --git a/modules/home/ai/evals/harborize/injection-canary-harbor/task.toml b/modules/home/ai/evals/harborize/injection-canary-harbor/task.toml new file mode 100644 index 000000000..19c64c902 --- /dev/null +++ b/modules/home/ai/evals/harborize/injection-canary-harbor/task.toml @@ -0,0 +1,65 @@ +schema_version = "1.3" +artifacts = [] + +[metadata] +author_name = "vanixiets harborize validation" +difficulty = "easy" +category = "software-engineering" +subcategory = "skill-injection-canary" +category_confidence = "high" +task_type = [ + "verification", +] +modality = [ + "text", +] +interface = [ + "terminal", +] +skill_type = [ + "instrument-integrity", +] +tags = [ + "canary", + "skill-injection", +] + +[verifier] +type = "test-script" +timeout_sec = 300.0 +service = "main" +pytest_plugins = [] + +[verifier.env] + +[verifier.judge] +model = "claude-sonnet-4-6" +rubric_path = "tests/rubric.toml" +input_dir = "/app" +input_type = "deliverables" +context = "" + +[verifier.hardening] +cleanup_conftests = true + +[agent] +timeout_sec = 300.0 +network_mode = "no-network" + +[oracle.env] + +[reward] + +[environment] +network_mode = "public" +build_timeout_sec = 600.0 +os = "linux" +cpus = 1 +memory_mb = 2048 +storage_mb = 5120 +gpus = 0 +mcp_servers = [] +setup_commands = [] +allow_internet = true + +[environment.env] diff --git a/modules/home/ai/evals/harborize/injection-canary-harbor/tests/test.sh b/modules/home/ai/evals/harborize/injection-canary-harbor/tests/test.sh new file mode 100755 index 000000000..33f6dc7d1 --- /dev/null +++ b/modules/home/ai/evals/harborize/injection-canary-harbor/tests/test.sh @@ -0,0 +1,8 @@ +#!/bin/bash +set -euo pipefail +mkdir -p /logs/verifier +if grep -qx 'HARBORIZE-CANARY-9F3A21' /logs/artifacts/canary-output.txt; then + echo 1 > /logs/verifier/reward.txt +else + echo 0 > /logs/verifier/reward.txt +fi diff --git a/modules/home/ai/evals/harborize/injection-canary/README.md b/modules/home/ai/evals/harborize/injection-canary/README.md new file mode 100644 index 000000000..0407e3ff9 --- /dev/null +++ b/modules/home/ai/evals/harborize/injection-canary/README.md @@ -0,0 +1,50 @@ +# injection-canary + +Instrument-integrity canary for the harborize instrument. + +Instrument version: 0.2.1 (frozen for the duration of the `validate-harborize-instrument` change). + +Upstream pins every source anchor cites: harbor `ac398bbda7c4c1073461797d3b95c2455cc671b5`, benchflow `d30527b82027a416e72014920cdf43a534967ad3`, skillsbench `9a1f4dd5f7659f75707435da3ce854b6e48321d1`. + +## What it asserts + +The oracle greps every skill discovery root (`/harbor/skills`, the per-agent `.claude`/`.codex`/`.opencode`/`.agents` paths, and `/skills`) for the canary token and writes what it finds to `/logs/artifacts/canary-output.txt`; the verifier rewards 1 exactly when that file holds `HARBORIZE-CANARY-9F3A21`. + +The token exists in exactly two places: the canary `SKILL.md` this package injects, and the asserted literal in its own `verifier/test.sh`. +It appears nowhere in the skill corpus under evaluation, which is what makes a reward of 1 evidence that delivered skill content reached a container discovery path. +The verifier's copy does not weaken that inference, because no agent phase can observe it — both runners upload the verifier's directory during the verification phase, after the agent phase has ended (see the fork section). +The claim this replaces, that the token lives only in the injected `SKILL.md`, was false in the other direction: the verifier's copy is real, and is why the leakage audit flags this package below. + +The evidence is one-directional and only the positive direction is read. +Under the oracle, which greps every discovery root unconditionally, reward 1 and reward 0 are both informative, and the falsifiability control at task 7.3 establishes that the 0 is reachable. +Under a model-driven agent only reward 1 carries delivery information, because a model that never greps scores 0 with the skill perfectly delivered. +That is why rung 6 asserts the adapter's registration directory instead. + +## Verifier fork: shared + +No `verifier/Dockerfile` and no `verifier.sandbox_mode`, so the verifier runs in the agent sandbox. +The fork is forced rather than preferred, and each runner forces it for its own reason. + +BenchFlow refuses to launch a task that declares separate rather than falling back to shared. +`runtime_capabilities.py:186-192` raises an unsupported-feature issue — its reason string reads "separate verifier sandboxes are parsed but not executed", which is the wording of a refusal and not of a fallback — and `raise_for_task_runtime_support` is a fail-closed pre-launch gate (`sandbox/setup.py:676`, `:819-842`). + +Harbor's separate verifier empties `/logs/verifier` before the verifier runs (`trial.py:599`), through the same host bind it mounts at `:686-692`. +`_run_shared_verifier` (`trial.py:536-567`) performs no wipe, and both packages are single-step, which is the precondition that claim needs: the multi-step path calls `_reset_shared_step_verifier_dirs` (`multi_step.py:202`, defined at `:338-342`) before every shared step verifier, so a package that ever grows `[[steps]]` loses that directory too. `SingleStepTrial.__init__` raises on a stepped task (`single_step.py:28-29`), so the two cannot be confused silently. + +None of that decides the deliverable channel, because `/logs/verifier` is not the channel this package uses and cannot be. +BenchFlow clears the directory's contents on the agent container immediately before the verifier runs, unconditionally (`sandbox/lockdown.py:775-784` called from `harden_before_verify` at `:1205-1212`), so a `canary-output.txt` written there is gone before `grep` looks for it under BenchFlow and present under Harbor's shared fork — the runner-dependent silent 0 the corpus exists to prevent. +The oracle therefore writes `/logs/artifacts/canary-output.txt`, which both runners bind for the whole trial and neither hardening step touches; the workspace README's channel section carries the anchors and the rung-4 evidence. + +## Known leakage flag + +`audit_leakage.py` check 1 flags the token (`MIN_LITERAL_LENGTH` at `:44`, `check_literals` at `:96`) because the asserted literal appears in both the verifier and the injected `SKILL.md`. +That is by design: the canary's mechanism is its answer key. +The instrument is frozen at 0.2.1 and is not edited to exempt it; the instrument-side question is deferred to the next revision. + +A consequence that was accepted and turns out not to arise: the fork being shared does not put the token within a running agent's reach. +Both runners upload the verifier's own directory during the verification phase, after the agent phase has ended — Harbor at `verifier/verifier.py:147-153`, reached from `_run_shared_verifier`, with the phase order fixed at `trial/single_step.py:41` then `:52`; BenchFlow at `task/verifier_core.py:385` in `_verify_test_script` (`:346`), reached from `verify()` (`:260`). +No agent phase observes `test.sh`. + +Rung 6 still scores on the adapter's registration directory rather than on the reward, for a reason that does not depend on that: a model-driven trial's reward conflates delivery with the model's own behaviour, since a model that never greps scores 0 with the skill perfectly delivered. +The registration directory is the deterministic witness; the reward is not. +The falsifiability control runs under the oracle, which greps skill directories and never reads the verifier. diff --git a/modules/home/ai/evals/harborize/injection-canary/environment/Dockerfile b/modules/home/ai/evals/harborize/injection-canary/environment/Dockerfile new file mode 100644 index 000000000..73a824557 --- /dev/null +++ b/modules/home/ai/evals/harborize/injection-canary/environment/Dockerfile @@ -0,0 +1,3 @@ +FROM ubuntu:24.04@sha256:561618e2c15bf2397621dd04f96926663a3b5616c189cf7e38db7e82f5c538ea + +WORKDIR /root diff --git a/modules/home/ai/evals/harborize/injection-canary/environment/skills/.gitkeep b/modules/home/ai/evals/harborize/injection-canary/environment/skills/.gitkeep new file mode 100644 index 000000000..e69de29bb diff --git a/modules/home/ai/evals/harborize/injection-canary/oracle/solve.sh b/modules/home/ai/evals/harborize/injection-canary/oracle/solve.sh new file mode 100755 index 000000000..cbaf029a2 --- /dev/null +++ b/modules/home/ai/evals/harborize/injection-canary/oracle/solve.sh @@ -0,0 +1,19 @@ +#!/bin/bash +set -euo pipefail +out=/logs/artifacts/canary-output.txt +mkdir -p /logs/artifacts +: > "$out" +# These roots are the oracle discovery-path union, not the adapter registration +# destinations; no adapter registration runs on the oracle path. +# HOME is defaulted because under set -u a container exporting none aborts here +# before any output is written, which reads downstream as a false canary alarm. +home=${HOME:-/root} +for root in /harbor/skills "$home/.claude/skills" "$home/.codex/skills" \ + "$home/.opencode/skills" "$home/.agents/skills" /skills; do + [ -d "$root" ] || continue + for f in "$root"/*/SKILL.md; do + [ -f "$f" ] || continue + grep -ho 'HARBORIZE-CANARY-[A-Z0-9]*' "$f" >> "$out" || true + done +done +sort -u -o "$out" "$out" diff --git a/modules/home/ai/evals/harborize/injection-canary/task.md b/modules/home/ai/evals/harborize/injection-canary/task.md new file mode 100644 index 000000000..1ec2c644d --- /dev/null +++ b/modules/home/ai/evals/harborize/injection-canary/task.md @@ -0,0 +1,38 @@ +--- +schema_version: '1.3' +metadata: + author_name: vanixiets harborize validation + difficulty: easy + category: software-engineering + subcategory: skill-injection-canary + category_confidence: high + task_type: + - verification + modality: + - text + interface: + - terminal + skill_type: + - instrument-integrity + tags: + - canary + - skill-injection +verifier: + type: test-script + timeout_sec: 300.0 +agent: + timeout_sec: 300.0 + network_mode: no-network +sandbox: + network_mode: public + build_timeout_sec: 600.0 + os: linux + cpus: 1 + memory_mb: 2048 + storage_mb: 5120 + gpus: 0 +--- + +Find the canary token and make it the exact content of `/logs/artifacts/canary-output.txt`, as a single line, deduplicated and sorted, with no other output in that file. + +The token is not in this instruction. diff --git a/modules/home/ai/evals/harborize/injection-canary/verifier/test.sh b/modules/home/ai/evals/harborize/injection-canary/verifier/test.sh new file mode 100755 index 000000000..33f6dc7d1 --- /dev/null +++ b/modules/home/ai/evals/harborize/injection-canary/verifier/test.sh @@ -0,0 +1,8 @@ +#!/bin/bash +set -euo pipefail +mkdir -p /logs/verifier +if grep -qx 'HARBORIZE-CANARY-9F3A21' /logs/artifacts/canary-output.txt; then + echo 1 > /logs/verifier/reward.txt +else + echo 0 > /logs/verifier/reward.txt +fi diff --git a/modules/home/ai/evals/harborize/pipeline-event-summary-harbor/compatibility/export-report.json b/modules/home/ai/evals/harborize/pipeline-event-summary-harbor/compatibility/export-report.json new file mode 100644 index 000000000..785df76a0 --- /dev/null +++ b/modules/home/ai/evals/harborize/pipeline-event-summary-harbor/compatibility/export-report.json @@ -0,0 +1,37 @@ +{ + "emitted_files": [ + "environment/Dockerfile", + "environment/build-events.jsonl", + "environment/skills/.gitkeep", + "instruction.md", + "solution/solve.sh", + "task.toml", + "tests/test.sh" + ], + "input_hashes": { + "environment/Dockerfile": "a01ec6eddaa1a340c47a64713645465c07aac07480ef6567716409c103dfff61", + "environment/build-events.jsonl": "1101fcf99801cfa569b02fc23a941fea9f483a00d15f5c984d1bb1ffd03727be", + "environment/skills/.gitkeep": "e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855", + "oracle/solve.sh": "f56198fbc2f37b281464c233d3a94f3895469d313753a53d61a2dea1233a4519", + "task.md": "80a9af1f2b6d417a2b194dbd2b75a40b6c49b9342669d3e66d5974dc9abc4747", + "verifier/test.sh": "d0911278b112af23a9a6ec9794f8e71dbab79f47d9e6e025b4f8ba04ea799cb2" + }, + "losses": [], + "output_hashes": { + "environment/Dockerfile": "a01ec6eddaa1a340c47a64713645465c07aac07480ef6567716409c103dfff61", + "environment/build-events.jsonl": "1101fcf99801cfa569b02fc23a941fea9f483a00d15f5c984d1bb1ffd03727be", + "environment/skills/.gitkeep": "e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855", + "instruction.md": "e5fd5660c1ff8755b1525ba89da8c0d03e152a7d8c601588db045af36242b44d", + "solution/solve.sh": "f56198fbc2f37b281464c233d3a94f3895469d313753a53d61a2dea1233a4519", + "task.toml": "d05d8c5c4286c33af00119b4841655b10846634a0378f8afddb8995d76faf104", + "tests/test.sh": "d0911278b112af23a9a6ec9794f8e71dbab79f47d9e6e025b4f8ba04ea799cb2" + }, + "restored_extension_paths": [], + "selected_entrypoint": "task.md", + "selected_oracle_dir": "oracle", + "selected_prompt": "instruction.md", + "selected_verifier_dir": "verifier", + "source_task_dir": "/Users/crs58/.treehouse/vanixiets-22808a/6/vanixiets/modules/home/ai/evals/harborize/pipeline-event-summary", + "status": "lossless", + "target": "harbor" +} diff --git a/modules/home/ai/evals/harborize/pipeline-event-summary-harbor/environment/Dockerfile b/modules/home/ai/evals/harborize/pipeline-event-summary-harbor/environment/Dockerfile new file mode 100644 index 000000000..1535b1fcd --- /dev/null +++ b/modules/home/ai/evals/harborize/pipeline-event-summary-harbor/environment/Dockerfile @@ -0,0 +1,8 @@ +FROM ubuntu:24.04@sha256:561618e2c15bf2397621dd04f96926663a3b5616c189cf7e38db7e82f5c538ea +ENV DEBIAN_FRONTEND=noninteractive + +RUN apt-get update && apt-get install -y python3 + +WORKDIR /root + +COPY build-events.jsonl /root/ diff --git a/modules/home/ai/evals/harborize/pipeline-event-summary-harbor/environment/build-events.jsonl b/modules/home/ai/evals/harborize/pipeline-event-summary-harbor/environment/build-events.jsonl new file mode 100644 index 000000000..35f849507 --- /dev/null +++ b/modules/home/ai/evals/harborize/pipeline-event-summary-harbor/environment/build-events.jsonl @@ -0,0 +1,15 @@ +{"pipeline": "alpha", "status": "success", "duration_ms": 300, "job": {"name": "build", "retries": 0}} +{"pipeline": "alpha", "status": "success", "duration_ms": 450, "job": {"name": "build", "retries": 1}} +{"pipeline": "alpha", "status": "failed", "duration_ms": 500, "job": {"name": "test", "retries": 2}} +{"pipeline": "alpha", "status": "success", "duration_ms": 380, "job": {"name": "build", "retries": 0}} +{"pipeline": "alpha", "status": "failed", "duration_ms": 520, "job": {"name": "lint", "retries": 1}} +{"pipeline": "alpha", "status": "success", "duration_ms": 410, "job": {"name": "build", "retries": 0}} +{"pipeline": "beta", "status": "failed", "duration_ms": 610, "job": {"name": "package", "retries": 3}} +{"pipeline": "beta", "status": "success", "duration_ms": 590, "job": {"name": "package", "retries": 1}} +{"pipeline": "beta", "status": "success", "duration_ms": 700, "job": {"name": "deploy", "retries": 0}} +{"pipeline": "beta", "status": "failed", "duration_ms": 650, "job": {"name": "deploy", "retries": 2}} +{"pipeline": "beta", "status": "success", "duration_ms": 620, "job": {"name": "package", "retries": 0}} +{"pipeline": "gamma", "status": "success", "duration_ms": 250, "job": {"name": "check", "retries": 0}} +{"pipeline": "gamma", "status": "success", "duration_ms": 280, "job": {"name": "check", "retries": 0}} +{"pipeline": "gamma", "status": "success", "duration_ms": 300, "job": {"name": "check", "retries": 1}} +{"pipeline": "gamma", "status": "failed", "duration_ms": 260, "job": {"name": "audit", "retries": 1}} diff --git a/modules/home/ai/evals/harborize/pipeline-event-summary-harbor/environment/skills/.gitkeep b/modules/home/ai/evals/harborize/pipeline-event-summary-harbor/environment/skills/.gitkeep new file mode 100644 index 000000000..e69de29bb diff --git a/modules/home/ai/evals/harborize/pipeline-event-summary-harbor/instruction.md b/modules/home/ai/evals/harborize/pipeline-event-summary-harbor/instruction.md new file mode 100644 index 000000000..902aac3a8 --- /dev/null +++ b/modules/home/ai/evals/harborize/pipeline-event-summary-harbor/instruction.md @@ -0,0 +1,11 @@ +Summarize the newline-delimited JSON build event log at `/root/build-events.jsonl` and write the result to `/logs/artifacts/summary.json`. + +The summary object carries exactly these keys, every value an integer: + +- `total_events`: the number of event records in the log. +- `failed_events`: the number of records whose `status` field equals `failed`. +- `per_pipeline`: an object mapping each distinct `pipeline` value to an object with `events` (that pipeline's record count) and `failed` (that pipeline's records with `status` equal to `failed`). +- `median_duration_ms`: the median of `duration_ms` across all records, as an integer. + With an odd record count this is the middle value of the sorted durations; with an even count it is the mean of the two middle values, rounded down. + +The result is judged by exact structural equality against the same computation over the log, so every count must be exact and no key may be missing, renamed, or extra. diff --git a/modules/home/ai/evals/harborize/pipeline-event-summary-harbor/solution/solve.sh b/modules/home/ai/evals/harborize/pipeline-event-summary-harbor/solution/solve.sh new file mode 100755 index 000000000..8305f2c31 --- /dev/null +++ b/modules/home/ai/evals/harborize/pipeline-event-summary-harbor/solution/solve.sh @@ -0,0 +1,38 @@ +#!/bin/bash +set -euo pipefail +mkdir -p /logs/artifacts +python3 - <<'EOF' +import json + +events = [] +with open('/root/build-events.jsonl') as fh: + for line in fh: + line = line.strip() + if line: + events.append(json.loads(line)) + +pipelines = {} +for event in events: + stats = pipelines.setdefault(event['pipeline'], {'events': 0, 'failed': 0}) + stats['events'] += 1 + if event['status'] == 'failed': + stats['failed'] += 1 + +durations = sorted(event['duration_ms'] for event in events) +mid = len(durations) // 2 +if len(durations) % 2: + median = durations[mid] +else: + median = (durations[mid - 1] + durations[mid]) // 2 + +summary = { + 'total_events': len(events), + 'failed_events': sum(1 for e in events if e['status'] == 'failed'), + 'per_pipeline': pipelines, + 'median_duration_ms': median, +} + +with open('/logs/artifacts/summary.json', 'w') as fh: + json.dump(summary, fh, indent=2, sort_keys=True) + fh.write('\n') +EOF diff --git a/modules/home/ai/evals/harborize/pipeline-event-summary-harbor/task.toml b/modules/home/ai/evals/harborize/pipeline-event-summary-harbor/task.toml new file mode 100644 index 000000000..da1a68900 --- /dev/null +++ b/modules/home/ai/evals/harborize/pipeline-event-summary-harbor/task.toml @@ -0,0 +1,67 @@ +schema_version = "1.3" +artifacts = [] + +[metadata] +author_name = "vanixiets harborize validation" +difficulty = "medium" +category = "software-engineering" +subcategory = "build-event-analytics" +category_confidence = "high" +task_type = [ + "calculation", + "extraction", +] +modality = [ + "json", +] +interface = [ + "terminal", +] +skill_type = [ + "json-querying", +] +tags = [ + "ndjson", + "aggregation", + "build-metrics", +] + +[verifier] +type = "test-script" +timeout_sec = 300.0 +service = "main" +pytest_plugins = [] + +[verifier.env] + +[verifier.judge] +model = "claude-sonnet-4-6" +rubric_path = "tests/rubric.toml" +input_dir = "/app" +input_type = "deliverables" +context = "" + +[verifier.hardening] +cleanup_conftests = true + +[agent] +timeout_sec = 600.0 +network_mode = "no-network" + +[oracle.env] + +[reward] + +[environment] +network_mode = "public" +build_timeout_sec = 600.0 +os = "linux" +cpus = 1 +memory_mb = 2048 +storage_mb = 5120 +gpus = 0 +mcp_servers = [] +setup_commands = [] +allow_internet = true + +[environment.env] diff --git a/modules/home/ai/evals/harborize/pipeline-event-summary-harbor/tests/test.sh b/modules/home/ai/evals/harborize/pipeline-event-summary-harbor/tests/test.sh new file mode 100755 index 000000000..0f3e6c953 --- /dev/null +++ b/modules/home/ai/evals/harborize/pipeline-event-summary-harbor/tests/test.sh @@ -0,0 +1,51 @@ +#!/bin/bash +set -euo pipefail +mkdir -p /logs/verifier +python3 - <<'EOF' +import json + +# Hand-derived by reading environment/build-events.jsonl record by record, and +# deliberately not recomputed here: a verifier that reruns the oracle's own +# algorithm agrees with a wrong oracle and cannot fail on one. +expected = { + 'total_events': 15, + 'failed_events': 5, + 'per_pipeline': { + 'alpha': {'events': 6, 'failed': 2}, + 'beta': {'events': 5, 'failed': 2}, + 'gamma': {'events': 4, 'failed': 1}, + }, + 'median_duration_ms': 450, +} + +# Write the failing reward before reading anything. A verifier that dies partway +# leaves no reward file, and both runners treat that as a trial-level crash +# rather than as a score, so a hostile or oversized submission would void the +# trial instead of scoring 0. +with open('/logs/verifier/reward.txt', 'w') as fh: + fh.write('0\n') + + +def integers_only(node): + if isinstance(node, bool): + return False + if isinstance(node, int): + return True + if isinstance(node, dict): + return all(integers_only(v) for v in node.values()) + return False + + +try: + with open('/logs/artifacts/summary.json') as fh: + actual = json.load(fh) + # Inside the guard on purpose: integers_only recurses, so a deeply nested + # submission raises RecursionError here rather than at the parse. + ok = integers_only(actual) and actual == expected +except Exception: + ok = False + +if ok: + with open('/logs/verifier/reward.txt', 'w') as fh: + fh.write('1\n') +EOF diff --git a/modules/home/ai/evals/harborize/pipeline-event-summary/README.md b/modules/home/ai/evals/harborize/pipeline-event-summary/README.md new file mode 100644 index 000000000..2d141a8be --- /dev/null +++ b/modules/home/ai/evals/harborize/pipeline-event-summary/README.md @@ -0,0 +1,57 @@ +# pipeline-event-summary + +The change's one mechanical evaluation package, sampling the claimed contract of the `preferences-json-querying` skill (NDJSON aggregation, grouping, and nested access with DuckDB or jaq) rather than that skill's own examples: given a build event log, produce an exact aggregate summary. + +Instrument version: 0.2.1 (frozen for the duration of the `validate-harborize-instrument` change). + +Upstream pins every source anchor cites: harbor `ac398bbda7c4c1073461797d3b95c2455cc671b5`, benchflow `d30527b82027a416e72014920cdf43a534967ad3`, skillsbench `9a1f4dd5f7659f75707435da3ce854b6e48321d1`. + +## Reward + +A single binary reward key, so the predicate is decidable on final state and multi-dimensional rubrics — which would silently disable Harbor's pass@k and BenchFlow's compare-lift — are not in play. + +The verifier holds its expectation as a literal, hand-derived by reading the fifteen fixture records, and rewards 1 only on exact structural equality with integers-only values. +It does not recompute the summary from a copy of the fixture. +Recomputation is the obvious shape and it is wrong here: the verifier would run the oracle's own algorithm, agree with a wrong oracle, and be structurally incapable of failing on one. +That is not hypothetical for this task. +The median convention is the case that bites — a verifier sharing the oracle's `durations[n // 2]` accepts the upper median and rejects the true median the task statement asks for, and the disagreement is invisible while the fixture holds an odd number of records. +Both sides now state the convention explicitly: `task.md` defines the even-count case and `oracle/solve.sh` implements it. + +The literal is what makes the oracle-inhabitation rung a real check on the oracle rather than a check that the oracle agrees with itself. +It also fixes the answer to the shipped fixture: growing `environment/build-events.jsonl` without updating the literal fails `harbor run -k 5` loudly instead of silently redefining the expected answer on both sides at once. + +## Verifier fork: shared + +No `verifier/Dockerfile` and no `verifier.sandbox_mode`, so the verifier runs in the agent sandbox. +The fork is forced rather than preferred, and both runners force it independently against harbor 0.21.0 and benchflow 0.7.4. + +BenchFlow does not fall back to shared when a package declares separate: it refuses to launch the task at all. +`runtime_capabilities.py:186-192` raises an unsupported-feature issue whose reason string is "separate verifier sandboxes are parsed but not executed", and `raise_for_task_runtime_support` is a fail-closed pre-launch gate (`sandbox/setup.py:676`, `:819-842`). +Reproduced: `bench tasks check --level runtime-capability --sandbox docker` exits 0 on this package as shipped and reports that one issue the moment `verifier.sandbox_mode: separate` is added. +The proposal requires this package to pass under both runners, so a Harbor-only fork is not available to it. + +Harbor's separate verifier would also destroy this package's agent-to-verifier channel. +`_run_separate_verifier` empties `/logs/verifier` before the verifier runs (`trial.py:599`, `environments/base.py:626-638`), and that path is bind-mounted from the host (`trial.py:686-692`), so a deliverable written there would be deleted and every agent including the oracle would score 0. +The deliverable is `/logs/artifacts/summary.json`, which the artifact re-upload restores even after that wipe (`trial.py:601-607`, `artifact_handler.py:210-254`), so the fork choice below is decided by the two reasons that remain rather than by this one. + +`_run_shared_verifier` (`trial.py:536-567`) performs no wipe, and both packages are single-step, which is the precondition that claim needs: the multi-step path calls `_reset_shared_step_verifier_dirs` (`multi_step.py:202`, defined at `:338-342`) before every shared step verifier, so a package that ever grows `[[steps]]` loses this channel. `SingleStepTrial.__init__` raises on a stepped task (`single_step.py:28-29`), so the two cannot be confused silently. +The channel is `/logs/artifacts/`, not `/logs/verifier/`, and under BenchFlow that is forced. +BenchFlow clears `/logs/verifier` on the agent container immediately before the verifier runs, unconditionally (`sandbox/lockdown.py:775-784` from `harden_before_verify` at `:1205-1212`), so a `summary.json` written there is present for Harbor's shared verifier and gone for BenchFlow's — a silent runner-dependent 0. +`/logs/artifacts/` is bind-mounted by both runners for the whole trial (`sandbox/docker.py:186-201`, `models/trial/paths.py:38`, `:195-202`) and no hardening step touches it; the workspace README's channel section carries the rung-4 evidence. +`/logs/verifier/reward.txt` remains what it is under both runners: the file the verifier itself writes, after the clear. + +## What the agent can and cannot see + +The verifier's expectation is a literal, so it is worth stating why that is not a leak. +On both runners the verifier's own directory is uploaded during the verification phase, after the agent phase has ended — Harbor at `verifier/verifier.py:147-153`, reached from `_run_shared_verifier`, with the phase order fixed at `trial/single_step.py:41` then `:52`; BenchFlow at `task/verifier_core.py:385` in `_verify_test_script` (`:346`), reached from `verify()` (`:260`). +No agent phase ever observes `test.sh`, under either fork. + +## Leakage audit + +`audit_leakage.py --task pipeline-event-summary --skills ` exits 0: the verifier carries no quoted expectation literal recoverable from the skill (the summary key `per_pipeline` was chosen over `pipelines`, which appears in the skill's prose and would trip check 1 as a false positive on a schema key). + +## Environment pinning + +The package's one Dockerfile, `environment/Dockerfile`, pins its base image by index digest, `ubuntu:24.04@sha256:561618e2c15bf2397621dd04f96926663a3b5616c189cf7e38db7e82f5c538ea`, which is the image this corpus's recorded evidence was produced against. +The floating tag had already drifted to a later build at rung-0 time, so an unpinned instrument would compare measurements taken in different environments. +The index digest is pinned rather than a platform manifest, so the same line resolves correctly on `linux/arm64` and `linux/amd64`. diff --git a/modules/home/ai/evals/harborize/pipeline-event-summary/environment/Dockerfile b/modules/home/ai/evals/harborize/pipeline-event-summary/environment/Dockerfile new file mode 100644 index 000000000..1535b1fcd --- /dev/null +++ b/modules/home/ai/evals/harborize/pipeline-event-summary/environment/Dockerfile @@ -0,0 +1,8 @@ +FROM ubuntu:24.04@sha256:561618e2c15bf2397621dd04f96926663a3b5616c189cf7e38db7e82f5c538ea +ENV DEBIAN_FRONTEND=noninteractive + +RUN apt-get update && apt-get install -y python3 + +WORKDIR /root + +COPY build-events.jsonl /root/ diff --git a/modules/home/ai/evals/harborize/pipeline-event-summary/environment/build-events.jsonl b/modules/home/ai/evals/harborize/pipeline-event-summary/environment/build-events.jsonl new file mode 100644 index 000000000..35f849507 --- /dev/null +++ b/modules/home/ai/evals/harborize/pipeline-event-summary/environment/build-events.jsonl @@ -0,0 +1,15 @@ +{"pipeline": "alpha", "status": "success", "duration_ms": 300, "job": {"name": "build", "retries": 0}} +{"pipeline": "alpha", "status": "success", "duration_ms": 450, "job": {"name": "build", "retries": 1}} +{"pipeline": "alpha", "status": "failed", "duration_ms": 500, "job": {"name": "test", "retries": 2}} +{"pipeline": "alpha", "status": "success", "duration_ms": 380, "job": {"name": "build", "retries": 0}} +{"pipeline": "alpha", "status": "failed", "duration_ms": 520, "job": {"name": "lint", "retries": 1}} +{"pipeline": "alpha", "status": "success", "duration_ms": 410, "job": {"name": "build", "retries": 0}} +{"pipeline": "beta", "status": "failed", "duration_ms": 610, "job": {"name": "package", "retries": 3}} +{"pipeline": "beta", "status": "success", "duration_ms": 590, "job": {"name": "package", "retries": 1}} +{"pipeline": "beta", "status": "success", "duration_ms": 700, "job": {"name": "deploy", "retries": 0}} +{"pipeline": "beta", "status": "failed", "duration_ms": 650, "job": {"name": "deploy", "retries": 2}} +{"pipeline": "beta", "status": "success", "duration_ms": 620, "job": {"name": "package", "retries": 0}} +{"pipeline": "gamma", "status": "success", "duration_ms": 250, "job": {"name": "check", "retries": 0}} +{"pipeline": "gamma", "status": "success", "duration_ms": 280, "job": {"name": "check", "retries": 0}} +{"pipeline": "gamma", "status": "success", "duration_ms": 300, "job": {"name": "check", "retries": 1}} +{"pipeline": "gamma", "status": "failed", "duration_ms": 260, "job": {"name": "audit", "retries": 1}} diff --git a/modules/home/ai/evals/harborize/pipeline-event-summary/environment/skills/.gitkeep b/modules/home/ai/evals/harborize/pipeline-event-summary/environment/skills/.gitkeep new file mode 100644 index 000000000..e69de29bb diff --git a/modules/home/ai/evals/harborize/pipeline-event-summary/oracle/solve.sh b/modules/home/ai/evals/harborize/pipeline-event-summary/oracle/solve.sh new file mode 100755 index 000000000..8305f2c31 --- /dev/null +++ b/modules/home/ai/evals/harborize/pipeline-event-summary/oracle/solve.sh @@ -0,0 +1,38 @@ +#!/bin/bash +set -euo pipefail +mkdir -p /logs/artifacts +python3 - <<'EOF' +import json + +events = [] +with open('/root/build-events.jsonl') as fh: + for line in fh: + line = line.strip() + if line: + events.append(json.loads(line)) + +pipelines = {} +for event in events: + stats = pipelines.setdefault(event['pipeline'], {'events': 0, 'failed': 0}) + stats['events'] += 1 + if event['status'] == 'failed': + stats['failed'] += 1 + +durations = sorted(event['duration_ms'] for event in events) +mid = len(durations) // 2 +if len(durations) % 2: + median = durations[mid] +else: + median = (durations[mid - 1] + durations[mid]) // 2 + +summary = { + 'total_events': len(events), + 'failed_events': sum(1 for e in events if e['status'] == 'failed'), + 'per_pipeline': pipelines, + 'median_duration_ms': median, +} + +with open('/logs/artifacts/summary.json', 'w') as fh: + json.dump(summary, fh, indent=2, sort_keys=True) + fh.write('\n') +EOF diff --git a/modules/home/ai/evals/harborize/pipeline-event-summary/task.md b/modules/home/ai/evals/harborize/pipeline-event-summary/task.md new file mode 100644 index 000000000..2a922bf8f --- /dev/null +++ b/modules/home/ai/evals/harborize/pipeline-event-summary/task.md @@ -0,0 +1,48 @@ +--- +schema_version: '1.3' +metadata: + author_name: vanixiets harborize validation + difficulty: medium + category: software-engineering + subcategory: build-event-analytics + category_confidence: high + task_type: + - calculation + - extraction + modality: + - json + interface: + - terminal + skill_type: + - json-querying + tags: + - ndjson + - aggregation + - build-metrics +verifier: + type: test-script + timeout_sec: 300.0 +agent: + timeout_sec: 600.0 + network_mode: no-network +sandbox: + network_mode: public + build_timeout_sec: 600.0 + os: linux + cpus: 1 + memory_mb: 2048 + storage_mb: 5120 + gpus: 0 +--- + +Summarize the newline-delimited JSON build event log at `/root/build-events.jsonl` and write the result to `/logs/artifacts/summary.json`. + +The summary object carries exactly these keys, every value an integer: + +- `total_events`: the number of event records in the log. +- `failed_events`: the number of records whose `status` field equals `failed`. +- `per_pipeline`: an object mapping each distinct `pipeline` value to an object with `events` (that pipeline's record count) and `failed` (that pipeline's records with `status` equal to `failed`). +- `median_duration_ms`: the median of `duration_ms` across all records, as an integer. + With an odd record count this is the middle value of the sorted durations; with an even count it is the mean of the two middle values, rounded down. + +The result is judged by exact structural equality against the same computation over the log, so every count must be exact and no key may be missing, renamed, or extra. diff --git a/modules/home/ai/evals/harborize/pipeline-event-summary/verifier/test.sh b/modules/home/ai/evals/harborize/pipeline-event-summary/verifier/test.sh new file mode 100755 index 000000000..0f3e6c953 --- /dev/null +++ b/modules/home/ai/evals/harborize/pipeline-event-summary/verifier/test.sh @@ -0,0 +1,51 @@ +#!/bin/bash +set -euo pipefail +mkdir -p /logs/verifier +python3 - <<'EOF' +import json + +# Hand-derived by reading environment/build-events.jsonl record by record, and +# deliberately not recomputed here: a verifier that reruns the oracle's own +# algorithm agrees with a wrong oracle and cannot fail on one. +expected = { + 'total_events': 15, + 'failed_events': 5, + 'per_pipeline': { + 'alpha': {'events': 6, 'failed': 2}, + 'beta': {'events': 5, 'failed': 2}, + 'gamma': {'events': 4, 'failed': 1}, + }, + 'median_duration_ms': 450, +} + +# Write the failing reward before reading anything. A verifier that dies partway +# leaves no reward file, and both runners treat that as a trial-level crash +# rather than as a score, so a hostile or oversized submission would void the +# trial instead of scoring 0. +with open('/logs/verifier/reward.txt', 'w') as fh: + fh.write('0\n') + + +def integers_only(node): + if isinstance(node, bool): + return False + if isinstance(node, int): + return True + if isinstance(node, dict): + return all(integers_only(v) for v in node.values()) + return False + + +try: + with open('/logs/artifacts/summary.json') as fh: + actual = json.load(fh) + # Inside the guard on purpose: integers_only recurses, so a deeply nested + # submission raises RecursionError here rather than at the parse. + ok = integers_only(actual) and actual == expected +except Exception: + ok = False + +if ok: + with open('/logs/verifier/reward.txt', 'w') as fh: + fh.write('1\n') +EOF diff --git a/modules/home/ai/plugins/testing-and-quality/.apm/skills/harborize/CHANGELOG.md b/modules/home/ai/plugins/testing-and-quality/.apm/skills/harborize/CHANGELOG.md new file mode 100644 index 000000000..028c32832 --- /dev/null +++ b/modules/home/ai/plugins/testing-and-quality/.apm/skills/harborize/CHANGELOG.md @@ -0,0 +1,186 @@ +# Changelog + +Notable changes to the harborize instrument. +The format follows Keep a Changelog, and the version numbers apply to this skill as a measurement instrument rather than to any package it emits. +Evaluation results are indexed by the instrument version that produced them, so a comparison spanning versions has to be made deliberately. +The instrument is not modified while an evaluation is being authored or run; revisions happen between rounds. + +## 0.2.1 — 2026-08-14 + +Container-boundary round. +Every entry below corrects a 0.2.0 claim that was true on the host and asserted about the container, or records a consequence of that boundary the text did not carry. +0.2.0 repaired the instructions that could not run; this version repairs the evidence they produce. +No package layout, estimator or budget option changed, so a package built under 0.2.0 remains readable, and no results exist under either version. + +Written against the same upstream revisions as 0.2.0: harbor `ac398bbda7c4c1073461797d3b95c2455cc671b5`, benchflow `d30527b82027a416e72014920cdf43a534967ad3`, skillsbench `9a1f4dd5f7659f75707435da3ce854b6e48321d1`. +Every anchor cited in this entry was read at those revisions. + +### Fixed + +- The instrument claimed delivery where only resolution is proven. + `_write_trial_lock` runs inside `Trial.__init__` at harbor `trial.py:104`, before `_resolve_injected_skills` at `:107` and long before `_upload_injected_skills` at `:411`, and `_build_agent_skill_locks` (`models/job/lock.py:462-475`) calls only host-side functions. + So `lock.json` records host-side resolution and pins digests, and proves nothing about what reached the container: it stays fully populated through an upload failure, a permissions failure, or an adapter that never reads the injected directory. + `scripts/collect_rewards.py` said the lock's `skills` list records every skill "actually resolved and uploaded"; the upload half was false. + SKILL.md said a run batch whose skills never arrived cannot be silently analyzed as if they had; that was false for the 17-adapter class below, which writes a fully populated lock and passes `harbor_injection`. + Both are narrowed to resolution and request, with what the lock cannot see stated beside them, and the check is now documented as one-sided: a disagreement proves the lattice did not vary, agreement proves only that request and resolution matched. +- The adapter enumeration in `references/emitters.md` was wrong in a way that read as correct. + It stated twenty adapters, which is exactly what `rg -l 'skills_dir' src/harbor/agents/installed/*.py` returns, but that glob is not the adapter set: it misses `installed/cline/cline.py` in a subdirectory and `terminus_2/terminus_2.py` outside `installed/`, and it counts modules where the question is registry entries. + Every adapter it named was correct. + The figure is 22 of 39 registered agents, from `rg -c 'AgentName\.[A-Z_0-9]+: ' src/harbor/agents/factory.py` and `rg -l 'self\.skills_dir' src/harbor/agents/ --type py`, the latter returning 23 files being 22 adapters plus the `agents/base.py:85` definition site. + Both full lists are now enumerated rather than sampled. +- The canonical package layout and the publication-grade instruction disagreed. + The layout placed `task.md`, `task.toml` and `instruction.md` in one directory, which `bench tasks check --level publication-grade` rejects: benchflow `_utils/task_authoring/structural_checks.py:208-212` refuses a co-present `task.toml`/`instruction.md`, and `:218-222` refuses `solution/` in favour of `oracle/`. + skillsbench `.github/scripts/validate_tasks.py:21-26` forbids exactly `instruction.md`, `task.toml`, `solution` and `tests`, which is the set Harbor requires, and all 87 tasks under `tasks/` plus the 14 under `tasks-extra/` are native, with zero `task.toml` and zero `solution/`. + The layout is now two directories: an authored BenchFlow-native tree and a generated `-harbor` sibling, with `bench tasks export` as the sanctioned route rather than an aside, and each static gate named against the directory it can run in. +- Pi's subscription-auth exposure was understated. + `pi.py:102-105` injects `ANTHROPIC_OAUTH_TOKEN` merely because the variable is present in the resolved environment, with no force flag, unlike claude-code (`claude_code.py:1587-1597`) and codex (`codex.py:1301-1329`). + Since Phase 5 bars subscription-authenticated cells from a reported batch, enforcement on a Pi cell requires an explicit scrub rather than withholding a flag. + +### Added + +- A hard-fail adapter allowlist in `scripts/design_matrix.py`. + Of the 39 factory-registered Harbor agents, 22 consume the injected skills and 17 ignore them with no error, no warning and no log line, so a cell naming one of the 17 uploads the skills, registers nothing, exits 0 and collapses every condition to the empty condition. + `HARBOR_SKILL_CONSUMING_AGENTS` carries the consuming set with the commands that regenerate it and the revision it was derived from; `check_harbor_agents` runs from `load_cells`, so no caller reaches a manifest without passing it, and it refuses with a nonzero exit naming the adapter and why. + Any `acp:` shorthand is refused for the same reason: `factory.py:167-175` routes every such name through `AgentName.ACP`, which is a non-consumer, so an ACP-shorthand cell drops skills on the Harbor arm while the BenchFlow arm works. + The gate runs before any container starts, and no artifact written after a run distinguishes this failure, which is why it is a gate rather than a runtime canary. +- The per-adapter destination table, because no single canary assertion covers a grid. + claude-code copies to `$CLAUDE_CONFIG_DIR/skills//` with `CLAUDE_CONFIG_DIR` set to `/logs/agent/sessions` (`claude_code.py:1530-1542`, `:1718`; `models/trial/paths.py:36`), codex and pi both write `$HOME/.agents/skills//` (`codex.py:1199-1207`, `pi.py:75-83`), and opencode writes `~/.config/opencode/skills//` (`opencode.py:425-433`). + claude-code's destination sits inside the `/logs/agent` bind mount (`trial.py:1284-1288`), so it is readable on the host after a run with no verifier code at all. + Harbor and BenchFlow declare the same path only for codex; `pi-acp` declares a superset (benchflow `agents/registry.py:560`), and `opencode` (`:700`) and `claude-agent-acp` (`:518`) declare paths Harbor does not write. + Those three cells' cross-runner comparability is recorded as an open question rather than resolved. +- The free delivery canary, which already exists upstream. + BenchFlow's oracle path proves delivery end to end with no model call: `rollout/__init__.py:1160` takes the `primary_agent == "oracle"` branch and still calls `deploy_skills` at `:1174`, `agents/install.py:313-314` computes the expected catalogue from `Path(skills_dir).glob("*/SKILL.md")`, `:350` falls back to the five `_ORACLE_SKILL_PATHS` at `install.py:30-36`, and `_link_skill_paths` asserts in the container that the catalogue at each discovery path equals the host's before raising `experiment_fidelity/skill_deployment_missing` at `:176-179`. + The command and pass criterion are documented, along with the fact that Harbor has no free equivalent: every adapter's registration copy is built in `run()` rather than `setup()`, so `--install-only` (`cli/jobs.py:901-910`) reaches the upload and never the registration. +- Two blocking prerequisites in Phase 3, where an author hits them. + Harbor's default `network_mode = "no-network"` is rejected at environment start when the daemon's kernel lacks `CONFIG_NFT_FIB_INET`: the probe is `environments/docker/docker.py:113-117`, failure zeroes `capabilities.disable_internet` (`:188-193`, `:289-293`), and `environments/base.py:772-781` raises. + The one-line daemon probe is given, along with the note that Harbor's own probe passes when `/proc/config.gz` is absent and the hand probe does not. + claude-code's `install()` curls its bootstrap from the network (`claude_code.py:425-449`) during `_prepare`/`_setup_agent` (`trial.py:408-414`), which is outside the agent-phase network policy that wraps only `_run_agent_phase` (`trial.py:465-469`) and the verifier phases, so a no-network cell fails during agent install indistinguishably from an injection failure at the reward level. +- `HARBOR_TELEMETRY=0` on every emitted Harbor line, overridable per cell. + `telemetry.py:239` sets `uses_skills` from the requested list without consulting the effective directory or the adapter, so it classifies a voided run as skill-bearing. + That is a third silent-null surface, recorded as one not to consume in analysis. + +## 0.2.0 — 2026-08-14 + +Contract repair round. +Every entry below corrects a documented instruction that could not run as written, or states a consequence the text left unstated. +The canonical package layout is unchanged apart from the verifier Dockerfile that separate mode requires, so packages built under 0.1.0 remain readable; none of them were validated, and results produced under 0.1.0 are not comparable to results produced under this version. + +The round was written in parallel across the files and then reconciled against an adversarial audit, so several entries below record the reconciled state rather than the first pass: the budget menu, the `audit_leakage.py` and `census.py` invocations, the verifier-environment fork, the manifest quoting, and the single-task interval all changed during reconciliation. + +Upstream revisions this version was written against: harbor `ac398bbda7c4c1073461797d3b95c2455cc671b5`, benchflow `d30527b82027a416e72014920cdf43a534967ad3`, skillsbench `9a1f4dd5f7659f75707435da3ce854b6e48321d1`. +Every anchor cited anywhere in the skill was re-read at those revisions during reconciliation. + +### Fixed + +- Skill injection under Harbor (F1). + The generated run line passed `--ak skills_dir=/tmp/cond-`, a host path handed to a container-side field that the Claude Code adapter reads with `cp -r /* $CLAUDE_CONFIG_DIR/skills/ 2>/dev/null || true`. + The copy found nothing, the redirection swallowed the failure, the run exited 0, and every condition collapsed to the empty condition without a diagnostic. + Harbor's injection flag is `--skill` (also spelled `--skills`), which resolves paths on the host, uploads each skill per trial under `/harbor/skills/`, records name, source and content digest in the job lock, and raises `FileNotFoundError` on a bad path before any container starts. + The root cause was a documentation hole: the emitter reference showed a Harbor run line carrying no injection flag at all, so the generator guessed. +- The validation gate named a removed command (F2). + `harbor tasks check` prints an error and raises `SystemExit(1)` unconditionally, and `tasks` is a hidden alias of the same typer app as `task`, so both spellings fail rather than validate. + The gate is now schema-level construction of `harbor.models.task.task.Task()` for the Harbor head and `bench tasks check --level structural` for the BenchFlow head, with the limits of each stated: schema validation checks field names, types and enum membership and nothing about behaviour, and it skips the test-script check entirely whenever a verifier environment is configured. + `harbor check`, the named replacement, is documented as a different instrument — an LLM rubric review that spawns a metered Harbor job and whose exit code ignores rubric failures — to be run once at authoring time and kept out of CI. +- The verifier environment was stated as a default rather than a fork (F3). + `environment_mode = "separate"` sets `skip_tests_upload=True`, so Harbor never uploads `tests/` and the verifier image must already own `/tests/test.sh`. + With the stock `python:3.12-slim` verifier image the skill declared, every trial fails permanently, because the absent reward file raises `RewardFileNotFoundError`, which is in Harbor's default no-retry list. + Both branches are now explicit, with their leak consequences, the observation that two of the three packages emitted in iteration 1 needed `shared`, and the machinery `separate` requires: the separate verifier environment's build context is the task's `tests/` directory, so a `tests/Dockerfile` is what places the test scripts at `/tests/`. + The task.toml template marks `environment_mode` as one branch of that fork rather than a default, and records that declaring a `[verifier.environment]` table at all implies `separate`, so a shared-mode package omits the table instead of setting `environment_mode = "shared"` beside it. + The rule extends past the test script to everything it invokes: Harbor uploads nothing into a separate verifier image, so a wrapper shelling out to `uvx` in an image carrying no uv reaches the same `RewardFileNotFoundError` dead end. + The shipped Dockerfile installs the verifier's tooling at build time and the wrappers call it directly, which also leaves the verifier environment runnable with no network. +- The BenchFlow run line died at argparse (F4). + `bench eval run --run-id` does not exist in benchflow's option set. + `--trials` exists but is implemented as the trial count for `--matrix` and is consumed only inside the `--matrix` branch, so it is inert on its own and there is no plain repetition flag. +- The host-versus-container asymmetry between the two runners was undocumented (F5). + BenchFlow's `--skills-dir` is a host path, validated on the host and mounted into the sandbox at `/skills`, so the same argument shape that failed silently for Harbor is correct for benchflow, and benchflow fails loudly where Harbor does not. +- Reward shape had an unstated consequence (F6). + Harbor computes pass@k only when every trial carries exactly one reward key valued 0 or 1, so a multi-dimensional Reward Kit rubric disables the metric with no message. + BenchFlow's `eval compare-lift` defines `passed` as `reward == 1.0`, so partial credit is invisible to its headline pass rate and appears only in mean reward. + The text now carries a decision rule rather than a warning: default to a single binary reward key, and take a graded rubric only when partial credit is the estimand. +- Dangling reference links. + Phase 4 pointed at `references/harbor-emitter.md` and `references/benchflow-emitter.md`, neither of which exists, while `references/emitters.md`, which holds that content, was referenced from nowhere. +- The documented `census.py` invocation could not run. + It was shown as a single positional call, but `--out` is required and the hardcoded sink is gone. + The two-subject design needs two invocations anyway, source first, because a flat deployed root resolves every skill to the `` sentinel without a `--membership-from` pointing at a source census, and the derived plugin units disappear with it. +- The bundled `census.json` was a 0.1.0-era snapshot: 128 skills, taken before harborize was itself in the tree, carrying no `provenance`, no `membership` map and no `overlap` tables, and therefore not the schema the 0.2.0 script emits. + It was the traceable source of the stale 128. + It is regenerated with the 0.2.0 script against a repo-relative root, so it is self-describing and usable as the `--membership-from` input, and SKILL.md now names it instead of leaving it orphaned. +- The documented `audit_leakage.py` invocation could not run. + It was shown with the skill directories positional, but the script declares no positionals: `--task` and `--skills` are both required named options. + The documented form is now `--task --skills [...]`, alongside the exit-code contract (0 clean, 1 flags raised, 2 the audit could not run) and the reason 2 is separated from 0, which is that every check searches for evidence of leakage and a mistyped path finds none. +- The package-layout comment claimed `environment/skills/` is "populated per condition". + dir(C) lives outside the package and is passed to the runner's injection flag, so the directory stays empty in the canonical package. + `materialize_conditions.py` does permit a `--dest` inside a package, which is how SkillsBench's own ablation driver uses `/environment/skills`; what it refuses is a destination that a Dockerfile in an ancestor directory would COPY or ADD into an image, checked by reading that Dockerfile's own copy sources rather than by matching the path name. +- Three interpolations of dir(C) into the generated manifest were unquoted. + The census's `` sentinel is a candidate ablation unit and contains shell redirection metacharacters, so `--dest /tmp/cond-` was reparsed by bash as a redirect pair and both the destination and the following option were lost, with `bash -n` and `shellcheck` clean on the file. + Every interpolated path and job name now passes through `shlex.quote`. +- The analyzer printed zero-width 95% intervals on a single-task design. + The bootstrap resamples tasks as clusters, so with one task every replicate is the same sample and the percentile interval collapses to the point estimate, with no marking. + Phase 1 proposes 1 to 3 tasks per unit, so this was the ordinary case. + Contrasts and condition estimates spanning fewer than two distinct tasks now report the point estimate and mark the interval unavailable. + A second difference that is numerically zero is also now labelled additive rather than picking up a sign from floating-point residue. + +### Changed + +- The unit of ablation is settled rather than asked (D1). + The unit is the skill, materialized from the deployed tree. + A plugin unit is a derived aggregate, the union of its member skills' deployed directories resolved through an explicit skill-to-plugin membership map shipped beside the census, because the deployed tree is flat and carries no plugin directories or `.apm/` paths. + The census reports both trees: the deployed tree is the evaluation subject because it is what a harness loads, and the first-party source tree is the refactor subject because it is where a fix can be written. + Selection-competition simulation runs over the deployed field, which is about a third larger at 172 skills against 129, because the source field omits the 43 deployed skills belonging to no first-party plugin. + How much interference that omission hides is what stage 2 measures; the ratio of field sizes does not establish it. +- Subjective contracts reduce to mechanical checks, and judges are gated (D2). + Exactly one judge-validation package must clear the acceptance bar before any judge-based stratum enters a budget. + The bar adopted here is a human-labeled validation set of roughly 6 to 12 submissions spanning pass, fail, partial, borderline, plausible-but-wrong and polished-but-unsupported, plus demonstrated agreement with the human labels and stability across runs; it is this instrument's bar rather than a quotation from SkillsBench, whose repository carries no such text. + The attributed fact is that all 87 tasks shipped in SkillsBench declare `verifier.type: test-script` and none uses a judge verifier, which is what warrants the gate. + If the gate fails, the subjective stratum falls back to structural proxies, which ask whether the produced artifact exhibits the convention and are checkable by grep or AST. +- Budgets are costed at metered API rates (D3). + Subscription-authenticated cells are permitted for interactive exploration and are never used for a run batch whose numbers are reported, because metered costing is robust to the unresolved credential-use-policy question in either direction and because a subscription cell confounds the measurement independently, rate-limit throttling being nondeterministic and single-account auth capping concurrency. + No per-run cost figure exists yet, so one must come from a calibration batch before any budget table is presented. +- The three iteration-1 packages are reference material to be regenerated under the repaired skill (D4). + None passed the oracle inhabitation invariant, so none is cited as validated. +- The description's exclusion clause fenced off "the skill itself", which read ambiguously between the skill under evaluation and harborize. + It now excludes only the skill under evaluation, and states that revising harborize between evaluation rounds is in scope. +- `references/emitters.md` is reflowed to one sentence per line, matching the other reference files and the repository's markdown convention. + Most of its content was written this round, so the reflow and the corrections land together. + SKILL.md's older bulleted sections keep their existing hard wrapping and are left for a separate pass. +- Counts corrected against a re-run census: the first-party source tree holds 129 skills rather than 128, and the deployed field is about a third larger rather than 34 percent. + The command-style, environment-coupling and decidability figures in `references/marketplace-program.md` were reproduced exactly and stand. +- Anchor corrections: `_verifier_env_build_context` is at `trial.py:694-702`, `harbor task check` at `cli/tasks.py:476-487`, `_iter_rollouts` at `eval_lift.py:277-291`, and twenty adapter modules reference `skills_dir` rather than twenty-one. + `bench eval run`'s option set is cited as `cli/main.py:193-592` together with `cli/_options.py:16-32`, because `--model` and `--skill-mode` are declared through shared `Annotated` aliases and do not appear literally inside the command body. + The `./` prefix for a relative `--skill` path is stated as a precaution rather than a requirement: `resolve_skill_sources` takes the local branch whenever the path exists, and only a relative path that does not exist is parsed as a git source. + +### Added + +- Instrument versioning (D5). + A `version` field in the frontmatter, this changelog, the rule that results are indexed by instrument version, and the rule that the instrument is revised between rounds and not during one. + The upstream revisions each version was written against are recorded, because the 0.1.0 text cited a harbor command that has since been removed and a benchflow flag that does not exist. +- Explicit not-yet-implemented markers for the `lint` and `selection-sim` stages. + Both are static, near-free and 100-percent coverage, both gate all container spend, and neither exists in `scripts/`, so the staged ordering reads as a plan rather than as inherited implementation. +- `scripts/collect_rewards.py`, closing the gap between running the manifest and feeding the analyzer. + It walks both runners' job trees, emits the rows `analyze_lattice.py` reads, and refuses on an unresolvable condition, an ambiguous reward dict, unscored rollouts with no error policy, or a failed injection check, so a batch whose skills never arrived cannot be analyzed as though they had. +- A dependency convention for the bundled scripts. + Each script under `scripts/` carries a `uv run --script` shebang and a PEP 723 inline metadata block declaring its Python requirement and any non-standard-library dependency, and is executable. + All six are standard-library-only at this version, so every dependency list is empty. +- `design/jobs.json`, emitted beside `conditions.json` and `manifest.sh`. + It records `{runner, cell, condition, path}` per job in the shape `collect_rewards.py --job-index` consumes, and the harbor run line now emits `-o/--jobs-dir` explicitly rather than writing to harbor's configured default, so where a job landed is recorded rather than reconstructed. + Job output is laid out per runner under one root: harbor at `/harbor/__` and bench at `/bench/__/trial-NN`. +- Stronger injection evidence in `collect_rewards.py`. + The Harbor arm reads each trial's `lock.json`, whose `skills` list carries a content digest per skill actually resolved and uploaded, and falls back to the requested `config.agent.skills` only when a trial wrote no lock. + The BenchFlow arm requires a non-null `effective_skills_dir` under `with-skill` rather than trusting `skill_mode`, which records the request. + Both fields record an outcome rather than a request, which is what detecting the F1 failure signature requires. +- Shape validation for `cells.json` in `design_matrix.py`. + `runner` is restricted to the two spellings `collect_rewards.py` accepts, required keys are checked, and duplicate cell names are refused, since two cells sharing a name would write into one job directory and silently merge across cells. + +### Removed + +- The resolution-IV fractional-factorial budget option. + The design the implementation actually offers is a foldover, which has no defining relation and therefore no resolution, and it was named in one file and rejected by `design_matrix.py --design`. + Option 3 is now Foldover, described by what it yields: a solo marginal and a leave-one-out marginal per unit, whose gap aggregates every higher-order interaction involving that unit without separating them. +- The claim that invariant 4's coupling is a joint over `(task, trial-seed)`. + Neither runner exposes a per-trial seed at these revisions, so the coupling is exact at the task level and positional within a task, with trial ordinals assigned by `collect_rewards.py`. + +## 0.1.0 + +As-authored baseline, placed verbatim into the vanixiets apm marketplace. +It established the phase structure, the algebraic invariants used as the review gate, the dual Harbor and BenchFlow emission model, the condition lattice and its estimators, and the marketplace-scale staged program. +Its Harbor injection flag, its Harbor validation command, its BenchFlow run line, and its verifier-environment default were all wrong in ways that fail silently or at argparse, and no package emitted under it was validated. diff --git a/modules/home/ai/plugins/testing-and-quality/.apm/skills/harborize/SKILL.md b/modules/home/ai/plugins/testing-and-quality/.apm/skills/harborize/SKILL.md new file mode 100644 index 000000000..fbea0d81d --- /dev/null +++ b/modules/home/ai/plugins/testing-and-quality/.apm/skills/harborize/SKILL.md @@ -0,0 +1,391 @@ +--- +name: harborize +version: 0.2.1 +description: Compile an existing agent skill, a plugin of skills, or several co-present + plugins into a complete, runnable evaluation package — Harbor task + SkillsBench/BenchFlow + task package, verifier, oracle, condition lattice, run manifest, and analysis scripts. + Use whenever the user wants to evaluate, benchmark, verify, stress-test, or measure the + quality, performance, efficacy, or interaction effects of one or more SKILL.md-based + skills or plugins across models and agent harnesses, or asks to "harborize" a skill, + turn skill-creator eval artifacts into Harbor/BenchFlow tasks, or design paired + with-skill/no-skill experiments. Do not use for authoring or improving the skill that + is the subject of the evaluation (use skill-creator) or for authoring generic Harbor + tasks unrelated to skills (use Harbor's create-task). Revising harborize itself between + evaluation rounds is in scope and is what this skill's own versioning exists for. +--- + +# Harborize + +Compile skills into measurement instruments. Input: path(s) to one or more skill +folders (each containing a SKILL.md), a plugin (a folder of skills), or several +plugins. Output: a canonical evaluation package per task, runnable under BOTH the +Harbor CLI and the BenchFlow CLI, plus a condition lattice, run manifest, and +analysis pipeline that together measure the marginal value of each unit and the +interactions between units when several share a context window. + +You are building apparatus, not solving tasks. Every decision below exists to keep +the measurement valid. The algebraic invariants in +`references/algebraic-invariants.md` are the review gate — read them before +emitting anything, and audit the finished package against them. + +## Phase 0 — Intake and inventory + +For each input path: + +1. Read the SKILL.md: frontmatter (name, description) and body. The description + defines the *claimed contract* — the task class the skill promises to improve. + Tasks you design must sample that class, not the skill's examples verbatim. +2. Read invocation-mode flags. `disable-model-invocation: true` marks a + command-style skill: it is never auto-triggered, so the measured composite + drops the trigger factor to P(correct-use ∧ success | invoked), the + with-skill condition must invoke it the way its harness would, and it is + excluded from selection-competition evals. Record the mode per unit. +3. Scan for environment couplings: absolute paths, `~/` references, host + tools the skill assumes. Do not neutralize them — the container will + expose them honestly — but record each in the package README so a failed + run is attributable to coupling rather than content. +4. Inventory bundled resources (`scripts/`, `references/`, `assets/`). Scripts the + skill ships are things the verifier must NOT simply re-run as its check — + otherwise the skill contains the proof (see non-triviality, below). +5. Look for prior skill-creator artifacts: `evals/evals.json`, + `-workspace/iteration-*/`, `benchmark.json`, `feedback.json`. These are + seeds: prompts become candidate instructions, assertions become candidate + verifier criteria, a passing with-skill output becomes the oracle's target. + Treat them as drafts to harden, not ground truth — inner-loop assertions were + LLM-graded; here they must become decidable or consciously judge-graded. +6. Determine the unit structure. + The unit of ablation is the skill, materialized from the deployed tree; this is settled, so state it rather than asking. + A plugin unit is a derived aggregate — the union of its member skills' deployed directories — resolved through an explicit plugin-membership map from skill name to plugin name, shipped beside the census. + The map is required because the deployed tree is flat: one directory per skill, no plugin directories and no `.apm/` paths, so membership cannot be recovered from directory structure. + Depart from the skill unit only to screen a marketplace at plugin granularity before paying for skill-level resolution, and record the departure and its reason in the package README. + Record the unit set 𝒫. + +If the user gave a plugin or marketplace identifier rather than a path, resolve it first; for apm marketplaces, consume via the apm CLI and compose the deployed tree. +Both trees matter and the census reports both. +The deployed tree is the evaluation subject, because it is what a harness actually loads — 172 skill directories at the current vanixiets state, against 129 first-party source skills. +The first-party source tree is the refactor subject, because that is the only place a fix can be written. +Selection-competition simulation runs over the deployed field, which is about a third larger, because the source field omits the 43 deployed skills that belong to no first-party plugin and that compete for the same trigger surface. +Run it over the source field and it is computed over a smaller set of competing descriptions than the one a harness presents; how much interference that omits is what stage 2 measures rather than something the ratio establishes. +If no SKILL.md is found at the path, stop and ask. + +## Phase 1 — Task derivation (with the user) + +Propose 1–3 tasks per unit plus, when |𝒫| > 1, at least one *composition task* +that plausibly exercises two or more units together. For each task state: the +instruction sketch, what final state the verifier will inspect, and why the task +is hard enough that a capable agent benefits from the skill (simple one-step +tasks don't trigger skills and measure nothing). Get sign-off before building. + +Rules for `instruction.md` / the task body: +- Never name the skills or say skills exist. Triggering is part of what is + measured; naming the skill collapses P(trigger ∧ use ∧ success) to P(success). +- Harness-agnostic phrasing — the same text runs under every cell in the grid. +- State the goal and expected artifacts (paths, formats) concretely. Describe + what done looks like, never how it will be checked. + +## Phase 2 — Verifier design (the hard part) + +Per task, walk the decision tree and record the choice in the package README: + +- **pytest / custom shell** when success is a decidable predicate on final + state. Exact truncation; binary `reward.txt`. Prefer this whenever honest. +- **Reward Kit** (`checks.py` + optional `judge.toml`) when there are multiple + criteria, partial credit, or an irreducibly subjective dimension. Graded + `reward.json`. Every criterion gets a descriptive name (skill-creator's + assertion `text` fields translate directly). + +Reward shape decides which downstream metrics exist, and both runners fail silently rather than warn. +Harbor computes pass@k only when every trial carries exactly one reward key whose value is 0 or 1; an extra key, or a value strictly between, returns an empty result and the metric disappears with no message (harbor `src/harbor/utils/pass_at_k.py`). +BenchFlow's `bench eval compare-lift` defines `passed` as `reward == 1.0` (benchflow `src/benchflow/eval_lift.py:31-33`), so partial credit contributes nothing to its headline pass rate and surfaces only in mean reward. +The rule: default to a single binary reward key, and pay for a graded rubric only when partial credit is the estimand rather than a convenience. +When the rubric wins, either write the per-criterion detail into `/logs/artifacts` and keep the reward channel binary, or take graded rewards knowingly, report means with bootstrap intervals as `references/lattice-design.md` describes, and state in the package README that pass@k and BenchFlow's pass rate are unavailable for that task. + +Subjective contracts reduce to a mechanical check wherever one exists; a judge is for the residue. +Exactly one judge-validation package is a hard gate before any judge-based stratum enters a budget. +The gate is warranted by how little the surrounding ecosystem leans on judges: all 87 tasks shipped in SkillsBench declare `verifier.type: test-script` and none uses a judge. +The acceptance bar adopted here for a judge criterion is a human-labeled validation set of roughly 6 to 12 submissions spanning pass, fail, partial, borderline, plausible-but-wrong and polished-but-unsupported, plus demonstrated agreement with the human labels and stability across runs. +Build that one package, clear that bar, and only then price judge-graded tasks into a design. +If the gate fails, the subjective stratum falls back to structural proxies: does the produced artifact exhibit the convention, checkable by grep or AST, in place of a judgment of its quality. + +The verifier environment is an explicit fork per task, recorded in the package README with its justification. +Harbor's own default is `shared`; declaring `[verifier.environment]` at all implies `separate`. +`separate` is the only setting that hides grading logic, expected outputs and any judge API key from the agent under test, which is the literal implementation of the non-triviality invariant. +It costs machinery: separate mode sets `skip_tests_upload=True` (harbor `src/harbor/trial/trial.py:618`), Harbor therefore never uploads `tests/`, and the verifier image must already own `/tests/test.sh` (harbor `src/harbor/verifier/verifier.py:96-103`). +A verifier Dockerfile is therefore mandatory; its build context is the task's `tests/` directory (harbor `src/harbor/trial/trial.py:694-702`), so `tests/Dockerfile` can COPY the test scripts to `/tests/`. +Setting `docker_image` in `[verifier.environment]` is what defeats it: a prebuilt image is used and the Dockerfile is skipped, so a stock image such as `python:3.12-slim` runs without ever owning `/tests/test.sh`, and every trial then fails permanently, because the absent reward file raises `RewardFileNotFoundError`, which sits in Harbor's default no-retry list (harbor `src/harbor/models/job/config.py:289-300`). +The same base image is fine when it arrives through `tests/Dockerfile`, which is what builds it. +The same applies past the script to everything it calls: Harbor uploads nothing into that image, so a wrapper invoking `uvx` in an image with no uv reaches the identical dead end. +Install the verifier's tooling at image build time and call it directly, which also keeps the verifier environment runnable with no network; `references/emitters.md` carries the Dockerfile. +`shared` runs the verifier in the agent's own environment and is required when the check must inspect mutable in-place state rather than a produced artifact. +It leaks, because everything in `tests/` is then visible to the agent, so a shared-mode task depends entirely on the non-triviality audit below to stay honest. +Two of the three packages this skill emitted in iteration 1 required shared mode. + +Defaults, always: +- Agent env `network_mode = "no-network"` unless the task genuinely needs the + network; verifier env `"public"` only if a judge or fetch is required. Both + prerequisites in Phase 3 apply to that default before it can be relied on. +- Prefer exact-set assertions (the full expected commit-message list, the + complete file inventory) over negative greps for forbidden substrings — + negative patterns are brittle against paraphrase and reward wording + avoidance rather than the property. +- Verifier reads only final state and declared artifacts — path-free. If you + need process evidence (e.g., selection evals), fingerprint it into artifacts + (a skill's bundled script leaving a characteristic output) rather than + parsing transcripts. + +Non-triviality audit, mandatory: + +``` +scripts/audit_leakage.py --task --skills [ ...] +``` + +Both are required named options; the script declares no positionals, so a mis-ordered path is rejected rather than absorbed into the skill list. +Three heuristics run: quoted literals of eight characters or more taken from `verifier/` or `tests/` and found anywhere in skill content, token Jaccard above 0.6 between the oracle and any skill-bundled script, and files byte-identical across the two trees. +Exit 0 means every check that could run raised no flag, exit 1 means at least one flag was raised and the review gate fails, and exit 2 means the audit could not run — a bad invocation, a missing task package, a skill directory with no SKILL.md, or no verifier sources. +Exit 2 is separated from exit 0 deliberately: every check searches for evidence of leakage, so a mistyped path finds nothing and would otherwise read as a clean package that was never examined. +A check whose inputs are absent, such as check 2 before the oracle exists, is reported as skipped rather than passed. +It does not check that the verifier runs in a separate environment; that is the fork above and is audited by reading `task.toml`. +If it fires, redesign the task by parameterizing inputs, or accept that the skill is answer-keying and the measurement is void. + +## Phase 3 — Environment and oracle + +Two prerequisites hold before authoring, and both surface as failures that read as authoring errors. + +Harbor's `network_mode = "no-network"` is rejected at environment start when the Docker daemon's kernel lacks `CONFIG_NFT_FIB_INET`. +The kernel probe is `environments/docker/docker.py:113-117`; on failure `_enable_egress_control` goes false (`:188-193`), which zeroes `capabilities.disable_internet` (`:289-293`), and `environments/base.py:772-781` then raises "network_mode='no-network' is not supported by ... environment". +Probe a new daemon before authoring anything: + +``` +docker run --rm alpine sh -c 'zcat /proc/config.gz | grep NFT_FIB_INET' +``` + +Expect `CONFIG_NFT_FIB_INET=y` or `=m`. +Harbor's own probe passes when `/proc/config.gz` is absent, so the hand probe is stricter than Harbor is: an absent config file means unknown here and supported there. + +claude-code's install fetches from the network during agent setup, which the agent-phase network policy does not cover. +`install()` at `claude_code.py:425-449` curls `https://downloads.claude.ai/claude-code-releases/bootstrap.sh` (or npm-installs on Alpine) unless the image already carries the requested version. +`_prepare` runs `_setup_agent` at `trial.py:408-414` with no policy wrapper, while only `_run_agent_phase` (`trial.py:465-469`) and the verifier phases enter `_phase_network_policy`. +So a no-network claude-code cell fails during agent install, and at the reward level that failure is indistinguishable from an injection failure. +Bake the agent into the environment image at a pinned version, which makes `_installed_claude_satisfies_version` return early and never reach the fetch. + +- `environment/Dockerfile`: task dependencies only — never the solution, never + the skills. Skills are injected at runtime per condition; baking them in + destroys the on/off toggle. +- Oracle (`oracle/solve.sh`; the export lands it at `solution/solve.sh`, so + write it once): a script that actually solves the task. It must pass the + verifier 100% — run it 5× before anything else counts. The oracle may + consult the skill's content while you write it; the point is inhabitation, + not independence. +- Generate fixtures (repos, datasets, broken states) with a dedicated + python script COPY'd and RUN in the Dockerfile — never nested shell + heredocs, whose escaping is fragile and whose failures are silent. Freeze + content invariants (tree hashes, checksums) at build time, before any + agent touches the environment. +- Test the environment interactively (`harbor task start-env ... -i`) before + writing the oracle; missing deps surface here. + +## Phase 4 — Dual emission + +Author BenchFlow-native, then derive the Harbor head into a separate sibling +directory with `bench tasks export`. The two heads cannot share a directory: + +``` +/ # authored; the source of truth +├── task.md # YAML frontmatter (schema 1.3 vocab) + body +├── environment/ # Dockerfile only; skills/ stays EMPTY +│ └── skills/ # (dir(C) lives outside the package) +├── oracle/solve.sh +└── verifier/ # test.sh + checks.py/judge.toml + # + Dockerfile when the verifier runs separate + +-harbor/ # generated; never hand-edited +├── task.toml # [task], [verifier], [environment] +├── instruction.md # from task.md's body +├── environment/ # copied verbatim +├── solution/solve.sh # from oracle/ +├── tests/ # from verifier/ +└── compatibility/export-report.json +``` + +``` +bench tasks export -harbor --target harbor +``` + +Three independent validators force the split. +BenchFlow's `--level publication-grade` rejects a `task.toml` or `instruction.md` beside `task.md` (benchflow `_utils/task_authoring/structural_checks.py:208-212`) and rejects `solution/` in favour of `oracle/` (`:218-222`). +SkillsBench's corpus gate forbids exactly `instruction.md`, `task.toml`, `solution` and `tests` (skillsbench `.github/scripts/validate_tasks.py:21-26`), which is the set Harbor requires. +All 87 tasks under skillsbench `tasks/` are native, with zero `task.toml` and zero `solution/`, as are the 14 under `tasks-extra/`. +The export also refuses a destination overlapping its source in either direction (benchflow `task/export.py:242-251`), so the Harbor head is a sibling and never a subdirectory. + +Re-export after every edit to the native tree. +`references/emitters.md` holds the exact field mappings, frontmatter vocabulary, injection flags and wrapper `test.sh` for each runner. + +Validate both heads before any oracle run: one static gate per head, plus one authoring-time check that spawns a metered job. + +`harbor tasks check` no longer exists. +At the harbor revision recorded under instrument versioning below, `tasks` is a hidden backwards-compatible alias of the same typer app as `task`, so both spellings reach one command that prints an error and raises `SystemExit(1)` unconditionally (harbor `src/harbor/cli/tasks.py:476-487`); a pipeline that calls it fails rather than validates. +The Harbor head's static gate is schema-level and runs against the exported `-harbor`: construct `harbor.models.task.task.Task(-harbor)`, which parses `task.toml` through `TaskConfig` and asserts `instruction.md` is present. +It verifies field names, types and enum membership, and nothing about behaviour. +It does not catch the separate-mode trap either, because `Task._validate_tests` returns early whenever a verifier environment is configured, so a separate-mode task whose verifier image has no `/tests/test.sh` passes schema validation and fails at run time. + +The BenchFlow head's static gate is `bench tasks check --level structural`, raised to `--level publication-grade` before publishing. +It validates directory layout and frontmatter against the SkillsBench schema, exits 1 with one line per issue, and executes nothing. +It runs against the authored native tree only. +Pointing publication-grade at the exported Harbor head fails by construction, because that head is the split layout the level exists to reject. + +`harbor check` is the named replacement for the removed command and is a different instrument, not a drop-in substitute. +It spawns a full Harbor job that runs an LLM agent against a task-quality rubric — default `claude-code` with `claude-sonnet-4-6` — so it is metered per invocation, and its exit code reflects only whether a task errored: rubric failures are printed and ignored (harbor `src/harbor/cli/analyze.py:206-207`). +Run it once at authoring time, read the report, and keep it out of CI and out of any per-condition loop. + +With both static gates clean, run the oracle under both runners. + +## Phase 5 — Condition lattice and budget (ask, every time) + +Conditions are subsets C ⊆ 𝒫, realized purely by which skill folders +`scripts/materialize_conditions.py` copies into the injected skills dir. +Nothing else may vary between conditions. + +Injection is per runner and the two mechanisms are not interchangeable. +Harbor takes `--skill ` (`--skills` is the same option, repeatable), which accepts either one skill directory or a parent of skill directories, resolves it on the host, uploads each skill into the trial environment under `/harbor/skills/`, and records each skill's name, source and content digest in the trial's `lock.json`. +A bad path raises `FileNotFoundError` before any container starts. +The agent kwarg `--ak skills_dir=` is a different field: a container-side path that the Claude Code adapter copies from with `cp -r /* $CLAUDE_CONFIG_DIR/skills/ 2>/dev/null || true`, so a host path passed there copies nothing, the run still exits 0, and every condition silently collapses to C = ∅. +BenchFlow takes `--skills-dir `, a host path validated on the host and mounted into the sandbox at `/skills`, with `--skill-mode no-skill` for C = ∅. +Field-level detail is in `references/emitters.md`. + +Uploading and registering are separate steps, and the gap between them holds a whole class of silent nulls. +Harbor uploads the skills for every agent, and of the 39 agents its factory registers, 22 read the injected directory and 17 ignore it with no error, no warning and no log line. +`scripts/design_matrix.py` carries the consuming set as `HARBOR_SKILL_CONSUMING_AGENTS` and refuses to emit a manifest for a `cells.json` naming anything else, including any `acp:` shorthand, since `factory.py:167-175` routes those through the non-consuming `acp` adapter. +That refusal is the only cheap gate for this class: it runs before any container starts, and nothing written after a run distinguishes a non-consuming adapter from a working one. +Per-adapter destinations differ as well, so no single canary assertion covers a grid; `references/emitters.md` carries the destination table and the cells where Harbor and BenchFlow do not agree on where skills land. + +One free delivery gate exists, on the BenchFlow arm. +`bench eval run --agent oracle --skill-mode with-skill --skills-dir ` runs the whole deployment path with no model call and asserts in the container that the skill catalogue at each discovery path matches the catalogue the host counted, raising `experiment_fidelity/skill_deployment_missing` on a mismatch. +Run it once per distinct dir(C) shape before any metered batch. +Harbor has no free equivalent, because every adapter's registration copy is built in `run()` rather than `setup()` and `--install-only` skips the agent run. + +Present the menu with arithmetic filled in — runs = |C| × k × cells — and let the user choose. +Cost every figure at metered API rates. +Subscription-authenticated cells are permitted for interactive exploration and are never used for a run batch whose numbers are reported: metered costing is robust to the unresolved credential-use-policy question in either direction, and a subscription cell confounds the measurement independently, because rate-limit throttling is nondeterministic and single-account auth caps concurrency. +Enforcing that rule takes an explicit environment scrub rather than merely withholding a flag. +Claude Code and Codex gate subscription auth behind `CLAUDE_FORCE_OAUTH` and `CODEX_FORCE_AUTH_JSON`, but Pi injects `ANTHROPIC_OAUTH_TOKEN` whenever the variable is present (`pi.py:102-105`), so a Pi cell in a reported batch must declare `ANTHROPIC_OAUTH_TOKEN=""` in its `env` block or run from a shell where it is unset. +No per-run cost figure exists yet, so derive one from a calibration batch before presenting any budget table. +Do not silently default: + +1. **Paired marginals** — C = {∅} ∪ {{u} : u ∈ 𝒫}; |C| = n+1. Measures each + unit's solo efficacy E({u}) − E(∅). No interactions. +2. **Marginals + targeted pairs** — add {u,v} for pairs with overlapping + trigger surface (description keyword/intent overlap — the near-miss + structure predicts interference). Yields second differences Δ_uv for the + pairs most likely to be nonzero. +3. **Foldover** — ∅, the n singletons, the n complements of singletons, and 𝒫; + |C| = 2n+2 for n ≥ 3. It yields a solo marginal and a leave-one-out + marginal per unit, whose gap aggregates every higher-order interaction + involving that unit without separating them. It has no defining relation + and carries no resolution claim, and from n ≥ 4 it contains no two-element + subsets, so no Δ_uv comes out of it. +4. **Full factorial** — 2^n. Only for n ≤ 3 or when someone else pays. + +Add "everything on" (C = 𝒫) to options 1 and 2 when the deployment reality is +all-units-loaded — that is the condition users actually live in, and +non-monotonicity means it cannot be inferred from marginals. Options 3 and 4 +already contain it. + +`scripts/design_matrix.py` emits `conditions.json`, a run manifest of shell +commands per (condition × cell × trial), and `jobs.json` recording where each +job will land. It carries whatever concurrency and environment variables each +cell declares in `cells.json` and knows nothing about authentication pools; +which cells may appear in a batch whose numbers are reported is the metered-rate +rule above. k ≥ 3. +It refuses outright on one thing only, the non-consuming Harbor adapter above, +and it sets `HARBOR_TELEMETRY=0` on every Harbor line unless a cell overrides +it, for the reason given in Phase 6. + +Selection-competition evals (does the RIGHT unit win when many are loaded) are +**off by default**. Offer them once when |𝒫| > 1; if accepted, see the +selection-evals section of `references/lattice-design.md` for the artifact- +fingerprint pattern. + +## Phase 6 — Run and analyze + +Run oracle everywhere first; then the manifest. + +`scripts/collect_rewards.py --conditions design/conditions.json --harbor-jobs runs/harbor --benchflow-jobs runs/bench --out results.json` walks both runners' job trees and emits the `{condition, cell, task, trial, reward}` rows the analyzer reads. +Those two roots are what `design_matrix.py --jobs-root runs` lays down, and `design/jobs.json` records the same mapping in the `{runner, cell, condition, path}` shape `--job-index` consumes when a job name cannot carry its condition. +It refuses rather than guesses: an unresolvable condition, an ambiguous reward dict, unscored rollouts with no `--errors-as` choice, or a failed injection check all exit 2. +The Harbor injection check reads each trial's `lock.json`, whose `skills` list carries a content digest per skill, and falls back to the requested `config.agent.skills` only when a trial wrote no lock; the BenchFlow check requires a non-null `effective_skills_dir` under `with-skill` rather than trusting `skill_mode` alone. + +Read that check for what it proves, which is resolution and request, not delivery. +`_write_trial_lock` runs inside `Trial.__init__` at harbor `trial.py:104`, before `_resolve_injected_skills` at `:107` and long before `_upload_injected_skills` at `:411`, and `_build_agent_skill_locks` (`models/job/lock.py:462-475`) calls only host-side functions. +So a populated `lock.json` says the paths resolved on the host and pins their digests, and says nothing about what reached the container. +It stays fully populated through an upload failure, a permissions failure, or an adapter that never reads the injected directory, and a 17-adapter batch that delivered nothing writes a clean lock and passes this check. +The check is one-sided: a disagreement proves the lattice did not vary as designed, and agreement proves only that the request and the host-side resolution matched. +Delivery is covered before the run instead, by the adapter allowlist and the BenchFlow oracle canary in Phase 5. + +Harbor's own telemetry is a third surface not to consume in analysis. +`telemetry.py:239` sets `uses_skills=any(agent.skills for agent in config.agents)` from the requested list, never consulting the effective directory or the adapter, so it classifies a voided run as skill-bearing. +The emitted manifest sets `HARBOR_TELEMETRY=0` (`telemetry.py:40`, `:45`) rather than publishing that classification. + +Feed `results.json` to `scripts/analyze_lattice.py`: + +- Ê(C) per cell. The reward type is read off each task's own data: a task + whose rewards all lie in {0,1} is binary and gets a Wilson interval on its + pass rate, any other value in [0,1] makes it graded and it gets a + task-clustered bootstrap percentile interval on the mean, and a cell mixing + the two is refused as an input error rather than pooled. A cell carrying + one task gets point estimates with the bootstrap intervals suppressed. +- Paired first differences per unit and, where the design supports them, second + differences Δ_uv with sign and interval — synergy (>0), interference (<0). +- Cross-cell table: per-unit efficacy by (model × harness). Flag units whose + benefit changes sign across cells — those failed empirical naturality and + their portability claim is model-idiosyncratic. +- Never average unpaired conditions; every contrast shares tasks and trials. + +Report deltas with intervals, absolute and normalized gain, and the run count +behind each number. Small n: say so plainly. + +## Marketplace-scale work + +When the subject is an entire marketplace rather than a skill or plugin, +read `references/marketplace-program.md` first — it carries the staged +program (census -> lint -> selection-sim -> stratified dynamic eval -> +refactor loop -> CI), the current state of the vanixiets engagement, and the +next-actions checklist. Census both trees before proposing any dynamic +evaluation, source first, because the deployed run needs the source run to +resolve plugin membership over a flat tree: + +``` +scripts/census.py --root --out src-census.json +scripts/census.py --root ~/.claude/skills --out dep-census.json \ + --membership-from src-census.json +``` + +Without `--membership-from`, every skill in the flat deployed tree resolves to +the `` sentinel and the derived plugin units disappear. The skill +bundles a source census at `census.json` that can stand in for the first +invocation; check its `provenance` block against the tree in front of you +before relying on it. + +Two stages of that program are named and not yet built. +Both are static, near-free and 100-percent coverage, which is why they gate all container spend. +`lint` is a per-skill static pass: YAML-parse the frontmatter, score description quality against the trigger methodology, list environment couplings for remediation, and check progressive-disclosure structure. +`selection-sim` is the same shape one level up: simulate everything-on triggering across the full deployed description set and adjudicate the flagged competition pairs, with no container and no task execution. +Neither exists in `scripts/` at instrument version 0.2.1, so read the stage ordering as a plan rather than as inherited implementation, and do not go looking for a script that is not there. + +## Instrument versioning + +This skill is the instrument, and results are comparable only within one version of it. +`0.1.0` is the as-authored baseline, `0.2.0` was the contract-repair round, and `0.2.1` corrects the claims in it that landed on the wrong side of the container boundary; the frontmatter `version` field is the authority and `CHANGELOG.md` records what moved between them. +Stamp the instrument version into every package README and every results file, and index evaluation results by it so a cross-version comparison has to be made deliberately. +Do not modify the instrument while an evaluation is being authored or run. +Revising harborize itself is in scope and is the intended path for defects found mid-round: record the defect, finish the round, then apply the change and bump the version. + +Every upstream claim here is anchored to a revision, because the 0.1.0 text cited a harbor command that has since been removed and a benchflow flag that does not exist. +Versions 0.2.0 and 0.2.1 were both written against harbor at `ac398bbda7c4c1073461797d3b95c2455cc671b5`, benchflow at `d30527b82027a416e72014920cdf43a534967ad3`, and skillsbench at `9a1f4dd5f7659f75707435da3ce854b6e48321d1`, read from the local clones under `~/ghq/github.com/`. +Those clones sit at their own HEAD rather than at a project-wide pin, so re-read any cited file and line before relying on it. + +## Review gate + +Before handing anything over, audit the package against +`references/algebraic-invariants.md` — one line per invariant, pass/fail, in +the package README. A package failing truncation, inhabitation, non-triviality, +or coupling is not done, whatever else works. diff --git a/modules/home/ai/plugins/testing-and-quality/.apm/skills/harborize/census.json b/modules/home/ai/plugins/testing-and-quality/.apm/skills/harborize/census.json new file mode 100644 index 000000000..5291235df --- /dev/null +++ b/modules/home/ai/plugins/testing-and-quality/.apm/skills/harborize/census.json @@ -0,0 +1,5704 @@ +{ + "provenance": { + "script_version": "0.2.0", + "root": "modules/home/ai/plugins", + "layout": "nested", + "revision": "b26f5a179edbffed47253d453e3a8eb7ed43372f", + "revision_date": "2026-08-14T20:36:17-04:00", + "timestamp": "2026-08-14T20:36:17-04:00", + "membership_source": null, + "n_skills": 129, + "n_plugins": 18, + "n_external": 0, + "n_empty_desc": 0, + "top_intra": 8, + "top_inter": 8 + }, + "plugins": { + "agent-orchestration-and-meta-tooling": [ + "herdr", + "meta-agent-teams", + "meta-create-workspace-agents-md", + "meta-generate-context-test", + "meta-generate-handoff-prompt", + "meta-list-all-agents", + "meta-list-all-tools", + "meta-load-cc-docs", + "meta-load-prompting-docs", + "meta-optimize-prompt", + "meta-orchestrate-dispatch", + "meta-orchestrator-checkpoint", + "meta-orchestrator-initiate", + "meta-search-sessions", + "meta-session-resume", + "meta-skill-creator" + ], + "beads-issue-tracking-and-session-workflow": [ + "issues-beads", + "issues-beads-audit", + "issues-beads-checkpoint", + "issues-beads-evolve", + "issues-beads-init", + "issues-beads-orient", + "issues-beads-prime", + "issues-beads-seed", + "session-advisor", + "session-checkpoint", + "session-orient", + "session-plan", + "session-review", + "stigmergic-convention" + ], + "document-authoring-and-visualization": [ + "doc-to-md", + "doc-to-md-cmd", + "knowledge-graph", + "scientific-visualization", + "text-to-visual-iteration", + "web-to-markdown" + ], + "event-modeling-workflow": [ + "event-modeling-brownfield", + "event-modeling-greenfield", + "event-modeling-qlerify-session", + "event-modeling-to-eventcatalog" + ], + "formal-specification-and-refinement": [ + "nucleus-platform", + "refinement-driven-development" + ], + "nix-build-operations": [ + "nix-flake-pr-cycle", + "nixpkgs-broken-package", + "process-compose-init", + "worktree-sparsity-eval" + ], + "planning-and-development": [ + "agentic-planning-development-workflow", + "openspec-apply-change", + "openspec-archive-change", + "openspec-bdd-bridge", + "openspec-bulk-archive-change", + "openspec-continue-change", + "openspec-explore", + "openspec-ff-change", + "openspec-linear-sync", + "openspec-new-change", + "openspec-onboard", + "openspec-propose", + "openspec-sync-specs", + "openspec-update-change", + "openspec-verify-change", + "project-management" + ], + "preferences-code-and-collaboration-conventions": [ + "preferences-change-management", + "preferences-comment-cleanup", + "preferences-documentation", + "preferences-git-history-cleanup", + "preferences-git-version-control", + "preferences-style-and-conventions" + ], + "preferences-data-and-scientific-computing": [ + "preferences-data-modeling", + "preferences-json-querying", + "preferences-scalable-probabilistic-modeling-workflow", + "preferences-scientific-inquiry-methodology", + "preferences-workflow-orchestration-algebra" + ], + "preferences-domain-driven-architecture": [ + "preferences-architectural-patterns", + "preferences-architecture-diagramming", + "preferences-bounded-context-design", + "preferences-collaborative-modeling", + "preferences-discovery-process", + "preferences-domain-modeling", + "preferences-strategic-domain-analysis" + ], + "preferences-event-driven-systems": [ + "preferences-event-catalog-qlerify", + "preferences-event-catalog-tooling", + "preferences-event-modeling", + "preferences-event-sourcing", + "preferences-schema-versioning" + ], + "preferences-functional-programming-theory": [ + "preferences-algebraic-data-types", + "preferences-algebraic-laws", + "preferences-computational-system-taxonomy", + "preferences-functional-reactive-programming", + "preferences-railway-oriented-programming", + "preferences-theoretical-foundations" + ], + "preferences-nix-and-secrets": [ + "preferences-nix-checks-architecture", + "preferences-nix-ci-cd-integration", + "preferences-nix-development", + "preferences-secrets" + ], + "preferences-operations-and-reliability": [ + "preferences-adaptive-planning", + "preferences-compositional-continuous-verification", + "preferences-distributed-systems", + "preferences-observability-engineering", + "preferences-production-readiness", + "preferences-validation-assurance" + ], + "preferences-programming-languages": [ + "preferences-haskell-development", + "preferences-python-development", + "preferences-rust-development", + "preferences-typescript-nodejs-development" + ], + "preferences-web-platform-and-deployment": [ + "preferences-cloudflare-wrangler-reference", + "preferences-hypermedia-development", + "preferences-hypermedia-documents", + "preferences-react-tanstack-ui-development", + "preferences-web-application-deployment", + "preferences-web-platform-foundations" + ], + "testing-and-quality": [ + "atdd-outer-loop", + "bdd-discovery", + "bdd-formulation", + "bdd-gherkin-formulation", + "bdd-step-definitions", + "executable-specification-testing", + "harborize", + "ubiquitous-language" + ], + "version-control-and-forge": [ + "dependency-source-acquisition", + "git-commit-prompt", + "gitbutler-but-cli", + "github-browse", + "hunk-review", + "jj-git-interactive-rebase-to-jj", + "jj-history-cleanup", + "jj-summary", + "jj-version-control", + "jj-workflow" + ] + }, + "membership": { + "agentic-planning-development-workflow": "planning-and-development", + "atdd-outer-loop": "testing-and-quality", + "bdd-discovery": "testing-and-quality", + "bdd-formulation": "testing-and-quality", + "bdd-gherkin-formulation": "testing-and-quality", + "bdd-step-definitions": "testing-and-quality", + "dependency-source-acquisition": "version-control-and-forge", + "doc-to-md": "document-authoring-and-visualization", + "doc-to-md-cmd": "document-authoring-and-visualization", + "event-modeling-brownfield": "event-modeling-workflow", + "event-modeling-greenfield": "event-modeling-workflow", + "event-modeling-qlerify-session": "event-modeling-workflow", + "event-modeling-to-eventcatalog": "event-modeling-workflow", + "executable-specification-testing": "testing-and-quality", + "git-commit-prompt": "version-control-and-forge", + "gitbutler-but-cli": "version-control-and-forge", + "github-browse": "version-control-and-forge", + "harborize": "testing-and-quality", + "herdr": "agent-orchestration-and-meta-tooling", + "hunk-review": "version-control-and-forge", + "issues-beads": "beads-issue-tracking-and-session-workflow", + "issues-beads-audit": "beads-issue-tracking-and-session-workflow", + "issues-beads-checkpoint": "beads-issue-tracking-and-session-workflow", + "issues-beads-evolve": "beads-issue-tracking-and-session-workflow", + "issues-beads-init": "beads-issue-tracking-and-session-workflow", + "issues-beads-orient": "beads-issue-tracking-and-session-workflow", + "issues-beads-prime": "beads-issue-tracking-and-session-workflow", + "issues-beads-seed": "beads-issue-tracking-and-session-workflow", + "jj-git-interactive-rebase-to-jj": "version-control-and-forge", + "jj-history-cleanup": "version-control-and-forge", + "jj-summary": "version-control-and-forge", + "jj-version-control": "version-control-and-forge", + "jj-workflow": "version-control-and-forge", + "knowledge-graph": "document-authoring-and-visualization", + "meta-agent-teams": "agent-orchestration-and-meta-tooling", + "meta-create-workspace-agents-md": "agent-orchestration-and-meta-tooling", + "meta-generate-context-test": "agent-orchestration-and-meta-tooling", + "meta-generate-handoff-prompt": "agent-orchestration-and-meta-tooling", + "meta-list-all-agents": "agent-orchestration-and-meta-tooling", + "meta-list-all-tools": "agent-orchestration-and-meta-tooling", + "meta-load-cc-docs": "agent-orchestration-and-meta-tooling", + "meta-load-prompting-docs": "agent-orchestration-and-meta-tooling", + "meta-optimize-prompt": "agent-orchestration-and-meta-tooling", + "meta-orchestrate-dispatch": "agent-orchestration-and-meta-tooling", + "meta-orchestrator-checkpoint": "agent-orchestration-and-meta-tooling", + "meta-orchestrator-initiate": "agent-orchestration-and-meta-tooling", + "meta-search-sessions": "agent-orchestration-and-meta-tooling", + "meta-session-resume": "agent-orchestration-and-meta-tooling", + "meta-skill-creator": "agent-orchestration-and-meta-tooling", + "nix-flake-pr-cycle": "nix-build-operations", + "nixpkgs-broken-package": "nix-build-operations", + "nucleus-platform": "formal-specification-and-refinement", + "openspec-apply-change": "planning-and-development", + "openspec-archive-change": "planning-and-development", + "openspec-bdd-bridge": "planning-and-development", + "openspec-bulk-archive-change": "planning-and-development", + "openspec-continue-change": "planning-and-development", + "openspec-explore": "planning-and-development", + "openspec-ff-change": "planning-and-development", + "openspec-linear-sync": "planning-and-development", + "openspec-new-change": "planning-and-development", + "openspec-onboard": "planning-and-development", + "openspec-propose": "planning-and-development", + "openspec-sync-specs": "planning-and-development", + "openspec-update-change": "planning-and-development", + "openspec-verify-change": "planning-and-development", + "preferences-adaptive-planning": "preferences-operations-and-reliability", + "preferences-algebraic-data-types": "preferences-functional-programming-theory", + "preferences-algebraic-laws": "preferences-functional-programming-theory", + "preferences-architectural-patterns": "preferences-domain-driven-architecture", + "preferences-architecture-diagramming": "preferences-domain-driven-architecture", + "preferences-bounded-context-design": "preferences-domain-driven-architecture", + "preferences-change-management": "preferences-code-and-collaboration-conventions", + "preferences-cloudflare-wrangler-reference": "preferences-web-platform-and-deployment", + "preferences-collaborative-modeling": "preferences-domain-driven-architecture", + "preferences-comment-cleanup": "preferences-code-and-collaboration-conventions", + "preferences-compositional-continuous-verification": "preferences-operations-and-reliability", + "preferences-computational-system-taxonomy": "preferences-functional-programming-theory", + "preferences-data-modeling": "preferences-data-and-scientific-computing", + "preferences-discovery-process": "preferences-domain-driven-architecture", + "preferences-distributed-systems": "preferences-operations-and-reliability", + "preferences-documentation": "preferences-code-and-collaboration-conventions", + "preferences-domain-modeling": "preferences-domain-driven-architecture", + "preferences-event-catalog-qlerify": "preferences-event-driven-systems", + "preferences-event-catalog-tooling": "preferences-event-driven-systems", + "preferences-event-modeling": "preferences-event-driven-systems", + "preferences-event-sourcing": "preferences-event-driven-systems", + "preferences-functional-reactive-programming": "preferences-functional-programming-theory", + "preferences-git-history-cleanup": "preferences-code-and-collaboration-conventions", + "preferences-git-version-control": "preferences-code-and-collaboration-conventions", + "preferences-haskell-development": "preferences-programming-languages", + "preferences-hypermedia-development": "preferences-web-platform-and-deployment", + "preferences-hypermedia-documents": "preferences-web-platform-and-deployment", + "preferences-json-querying": "preferences-data-and-scientific-computing", + "preferences-nix-checks-architecture": "preferences-nix-and-secrets", + "preferences-nix-ci-cd-integration": "preferences-nix-and-secrets", + "preferences-nix-development": "preferences-nix-and-secrets", + "preferences-observability-engineering": "preferences-operations-and-reliability", + "preferences-production-readiness": "preferences-operations-and-reliability", + "preferences-python-development": "preferences-programming-languages", + "preferences-railway-oriented-programming": "preferences-functional-programming-theory", + "preferences-react-tanstack-ui-development": "preferences-web-platform-and-deployment", + "preferences-rust-development": "preferences-programming-languages", + "preferences-scalable-probabilistic-modeling-workflow": "preferences-data-and-scientific-computing", + "preferences-schema-versioning": "preferences-event-driven-systems", + "preferences-scientific-inquiry-methodology": "preferences-data-and-scientific-computing", + "preferences-secrets": "preferences-nix-and-secrets", + "preferences-strategic-domain-analysis": "preferences-domain-driven-architecture", + "preferences-style-and-conventions": "preferences-code-and-collaboration-conventions", + "preferences-theoretical-foundations": "preferences-functional-programming-theory", + "preferences-typescript-nodejs-development": "preferences-programming-languages", + "preferences-validation-assurance": "preferences-operations-and-reliability", + "preferences-web-application-deployment": "preferences-web-platform-and-deployment", + "preferences-web-platform-foundations": "preferences-web-platform-and-deployment", + "preferences-workflow-orchestration-algebra": "preferences-data-and-scientific-computing", + "process-compose-init": "nix-build-operations", + "project-management": "planning-and-development", + "refinement-driven-development": "formal-specification-and-refinement", + "scientific-visualization": "document-authoring-and-visualization", + "session-advisor": "beads-issue-tracking-and-session-workflow", + "session-checkpoint": "beads-issue-tracking-and-session-workflow", + "session-orient": "beads-issue-tracking-and-session-workflow", + "session-plan": "beads-issue-tracking-and-session-workflow", + "session-review": "beads-issue-tracking-and-session-workflow", + "stigmergic-convention": "beads-issue-tracking-and-session-workflow", + "text-to-visual-iteration": "document-authoring-and-visualization", + "ubiquitous-language": "testing-and-quality", + "web-to-markdown": "document-authoring-and-visualization", + "worktree-sparsity-eval": "nix-build-operations" + }, + "skills": { + "herdr": { + "plugin": "agent-orchestration-and-meta-tooling", + "path": "agent-orchestration-and-meta-tooling/.apm/skills/herdr", + "symlink": null, + "lines": 302, + "cmd_style": false, + "env_coupling": [], + "has_scripts": false, + "has_refs": false, + "description": "Control herdr from inside it. Manage workspaces and tabs, split panes, spawn agents, read output, and wait for state changes \u2014 all via CLI commands that talk to the running herdr instance over a local unix socket. Use when running inside herdr (HERDR_ENV=1).", + "desc_toks": [ + "agents", + "all", + "and", + "changes", + "cli", + "commands", + "control", + "env", + "for", + "from", + "herdr", + "inside", + "instance", + "local", + "manage", + "output", + "over", + "panes", + "read", + "running", + "socket", + "spawn", + "split", + "state", + "tabs", + "talk", + "that", + "the", + "unix", + "use", + "via", + "wait", + "when", + "workspaces" + ], + "decidability_guess": "subjective" + }, + "meta-agent-teams": { + "plugin": "agent-orchestration-and-meta-tooling", + "path": "agent-orchestration-and-meta-tooling/.apm/skills/meta-agent-teams", + "symlink": null, + "lines": 57, + "cmd_style": false, + "env_coupling": [ + "~/.claude/CLAUDE.md", + "~/.claude/skills/jj-version-control/SKILL.md" + ], + "has_scripts": false, + "has_refs": false, + "description": "Agent team orchestration conventions for persistent multi-agent coordination via shared task lists and messaging.", + "desc_toks": [ + "agent", + "and", + "conventions", + "coordination", + "for", + "lists", + "messaging", + "multi", + "orchestration", + "persistent", + "shared", + "task", + "team", + "via" + ], + "decidability_guess": "subjective" + }, + "meta-create-workspace-agents-md": { + "plugin": "agent-orchestration-and-meta-tooling", + "path": "agent-orchestration-and-meta-tooling/.apm/skills/meta-create-workspace-agents-md", + "symlink": null, + "lines": 727, + "cmd_style": true, + "env_coupling": [ + "~/.claude/skills/jj-version-control/SKILL.md", + "~/.claude/skills/jj-version-control/tiered-ceremony.md" + ], + "has_scripts": false, + "has_refs": false, + "description": "Generate or update a comprehensive workspace-level CLAUDE.md file from project analysis.", + "desc_toks": [ + "analysis", + "claude", + "comprehensive", + "file", + "from", + "generate", + "level", + "project", + "update", + "workspace" + ], + "decidability_guess": "decidable" + }, + "meta-generate-context-test": { + "plugin": "agent-orchestration-and-meta-tooling", + "path": "agent-orchestration-and-meta-tooling/.apm/skills/meta-generate-context-test", + "symlink": null, + "lines": 69, + "cmd_style": true, + "env_coupling": [], + "has_scripts": false, + "has_refs": false, + "description": "Generate \"brown M&M\" test questions to validate LLM context ingestion and comprehension.", + "desc_toks": [ + "and", + "brown", + "comprehension", + "context", + "generate", + "ingestion", + "llm", + "questions", + "test", + "validate" + ], + "decidability_guess": "decidable" + }, + "meta-generate-handoff-prompt": { + "plugin": "agent-orchestration-and-meta-tooling", + "path": "agent-orchestration-and-meta-tooling/.apm/skills/meta-generate-handoff-prompt", + "symlink": null, + "lines": 24, + "cmd_style": true, + "env_coupling": [], + "has_scripts": false, + "has_refs": false, + "description": "Generate an optimal handoff prompt for Claude Code agent transitions preserving critical context.", + "desc_toks": [ + "agent", + "claude", + "code", + "context", + "critical", + "for", + "generate", + "handoff", + "optimal", + "preserving", + "prompt", + "transitions" + ], + "decidability_guess": "subjective" + }, + "meta-list-all-agents": { + "plugin": "agent-orchestration-and-meta-tooling", + "path": "agent-orchestration-and-meta-tooling/.apm/skills/meta-list-all-agents", + "symlink": null, + "lines": 8, + "cmd_style": true, + "env_coupling": [], + "has_scripts": false, + "has_refs": false, + "description": "List all available subagent types with their descriptions and tool access for Task tool dispatch.", + "desc_toks": [ + "access", + "all", + "and", + "available", + "descriptions", + "dispatch", + "for", + "list", + "subagent", + "task", + "their", + "tool", + "types", + "with" + ], + "decidability_guess": "subjective" + }, + "meta-list-all-tools": { + "plugin": "agent-orchestration-and-meta-tooling", + "path": "agent-orchestration-and-meta-tooling/.apm/skills/meta-list-all-tools", + "symlink": null, + "lines": 8, + "cmd_style": true, + "env_coupling": [], + "has_scripts": false, + "has_refs": false, + "description": "List all available tools with parameter schemas and usage guidance.", + "desc_toks": [ + "all", + "and", + "available", + "guidance", + "list", + "parameter", + "schemas", + "tools", + "usage", + "with" + ], + "decidability_guess": "subjective" + }, + "meta-load-cc-docs": { + "plugin": "agent-orchestration-and-meta-tooling", + "path": "agent-orchestration-and-meta-tooling/.apm/skills/meta-load-cc-docs", + "symlink": null, + "lines": 41, + "cmd_style": true, + "env_coupling": [], + "has_scripts": false, + "has_refs": false, + "description": "Fetch or scrape the latest version of Claude Code documentation using firecrawl if available.", + "desc_toks": [ + "available", + "claude", + "code", + "documentation", + "fetch", + "firecrawl", + "latest", + "scrape", + "the", + "using", + "version" + ], + "decidability_guess": "subjective" + }, + "meta-load-prompting-docs": { + "plugin": "agent-orchestration-and-meta-tooling", + "path": "agent-orchestration-and-meta-tooling/.apm/skills/meta-load-prompting-docs", + "symlink": null, + "lines": 34, + "cmd_style": true, + "env_coupling": [], + "has_scripts": false, + "has_refs": false, + "description": "Use firecrawl to scrape the latest version of elite prompt engineering documentation.", + "desc_toks": [ + "documentation", + "elite", + "engineering", + "firecrawl", + "latest", + "prompt", + "scrape", + "the", + "use", + "version" + ], + "decidability_guess": "subjective" + }, + "meta-optimize-prompt": { + "plugin": "agent-orchestration-and-meta-tooling", + "path": "agent-orchestration-and-meta-tooling/.apm/skills/meta-optimize-prompt", + "symlink": null, + "lines": 192, + "cmd_style": false, + "env_coupling": [ + "~/projects/nix-workspace/repo-name/", + "~/projects/workspace/repo-name/" + ], + "has_scripts": false, + "has_refs": false, + "description": "Optimize a prompt draft for Claude using Anthropic's best practices and prompt engineering patterns.", + "desc_toks": [ + "and", + "anthropic", + "best", + "claude", + "draft", + "engineering", + "for", + "optimize", + "patterns", + "practices", + "prompt", + "using" + ], + "decidability_guess": "subjective" + }, + "meta-orchestrate-dispatch": { + "plugin": "agent-orchestration-and-meta-tooling", + "path": "agent-orchestration-and-meta-tooling/.apm/skills/meta-orchestrate-dispatch", + "symlink": null, + "lines": 46, + "cmd_style": true, + "env_coupling": [], + "has_scripts": false, + "has_refs": false, + "description": "Design workflow DAG for subagent dispatch to preserve orchestrator context as a scarce coordination resource.", + "desc_toks": [ + "context", + "coordination", + "dag", + "design", + "dispatch", + "for", + "orchestrator", + "preserve", + "resource", + "scarce", + "subagent", + "workflow" + ], + "decidability_guess": "subjective" + }, + "meta-orchestrator-checkpoint": { + "plugin": "agent-orchestration-and-meta-tooling", + "path": "agent-orchestration-and-meta-tooling/.apm/skills/meta-orchestrator-checkpoint", + "symlink": null, + "lines": 186, + "cmd_style": false, + "env_coupling": [ + "~/.claude/jobs/manual-handoff/meta-orchestrator-payload.md", + "~/.claude/skills/meta-orchestrator-checkpoint/SKILL.md" + ], + "has_scripts": false, + "has_refs": false, + "description": "Team-level checkpoint and handoff for missions coordinated through the master orchestrator pattern. Captures cross-cycle accumulated state across the master plus retired ACs and their WO cycles, producing a handoff payload that the next master session can resume via /meta-orchestrator-initiate. Two variants: handoff (mid-mission) and closure (mission completion).", + "desc_toks": [ + "accumulated", + "across", + "acs", + "and", + "can", + "captures", + "checkpoint", + "closure", + "completion", + "coordinated", + "cross", + "cycle", + "cycles", + "for", + "handoff", + "initiate", + "level", + "master", + "meta", + "mid", + "mission", + "missions", + "next", + "orchestrator", + "pattern", + "payload", + "plus", + "producing", + "resume", + "retired", + "session", + "state", + "team", + "that", + "the", + "their", + "through", + "two", + "variants", + "via" + ], + "decidability_guess": "subjective" + }, + "meta-orchestrator-initiate": { + "plugin": "agent-orchestration-and-meta-tooling", + "path": "agent-orchestration-and-meta-tooling/.apm/skills/meta-orchestrator-initiate", + "symlink": null, + "lines": 216, + "cmd_style": false, + "env_coupling": [ + "~/.claude/jobs/manual-handoff/observations/master-orchestrator.md", + "~/.claude/skills/meta-orchestrator-initiate/SKILL.md" + ], + "has_scripts": false, + "has_refs": false, + "description": "Team-level orchestrator initialization for missions requiring multi-pair coordination. Establishes the strategic frame, authority hierarchy, wrapper canon, and decomposition discipline for a planning master orchestrator spawning repo-coupled AC\u2194WO pairs across one or more target repositories.", + "desc_toks": [ + "across", + "and", + "authority", + "canon", + "coordination", + "coupled", + "decomposition", + "discipline", + "establishes", + "for", + "frame", + "hierarchy", + "initialization", + "level", + "master", + "missions", + "more", + "multi", + "one", + "orchestrator", + "pair", + "pairs", + "planning", + "repo", + "repositories", + "requiring", + "spawning", + "strategic", + "target", + "team", + "the", + "wrapper" + ], + "decidability_guess": "subjective" + }, + "meta-search-sessions": { + "plugin": "agent-orchestration-and-meta-tooling", + "path": "agent-orchestration-and-meta-tooling/.apm/skills/meta-search-sessions", + "symlink": null, + "lines": 67, + "cmd_style": false, + "env_coupling": [], + "has_scripts": true, + "has_refs": false, + "description": "Use when the user asks to find, recall, or locate a past Claude Code, Codex, or Pi session, conversation, or discussion by topic, keyword, project, path, or content.", + "desc_toks": [ + "asks", + "claude", + "code", + "codex", + "content", + "conversation", + "discussion", + "find", + "keyword", + "locate", + "past", + "path", + "project", + "recall", + "session", + "the", + "topic", + "use", + "user", + "when" + ], + "decidability_guess": "subjective" + }, + "meta-session-resume": { + "plugin": "agent-orchestration-and-meta-tooling", + "path": "agent-orchestration-and-meta-tooling/.apm/skills/meta-session-resume", + "symlink": null, + "lines": 85, + "cmd_style": false, + "env_coupling": [ + "~/.claude/sessions/" + ], + "has_scripts": false, + "has_refs": false, + "description": "Generate annotated resume command and add to atuin history for session continuation. Triggered by requests to prepare, save, or checkpoint a session for resumption, or \"prepare this session for resumption\".", + "desc_toks": [ + "add", + "and", + "annotated", + "atuin", + "checkpoint", + "command", + "continuation", + "for", + "generate", + "history", + "prepare", + "requests", + "resume", + "resumption", + "save", + "session", + "this", + "triggered" + ], + "decidability_guess": "subjective" + }, + "meta-skill-creator": { + "plugin": "agent-orchestration-and-meta-tooling", + "path": "agent-orchestration-and-meta-tooling/.apm/skills/meta-skill-creator", + "symlink": null, + "lines": 365, + "cmd_style": false, + "env_coupling": [], + "has_scripts": true, + "has_refs": true, + "description": "Guide for creating effective skills. This skill should be used when users want to create a new skill (or update an existing skill) that extends Claude's capabilities with specialized knowledge, workflows, or tool integrations.", + "desc_toks": [ + "capabilities", + "claude", + "create", + "creating", + "effective", + "existing", + "extends", + "for", + "guide", + "integrations", + "knowledge", + "new", + "should", + "skill", + "skills", + "specialized", + "that", + "this", + "tool", + "update", + "used", + "users", + "want", + "when", + "with", + "workflows" + ], + "decidability_guess": "subjective" + }, + "issues-beads": { + "plugin": "beads-issue-tracking-and-session-workflow", + "path": "beads-issue-tracking-and-session-workflow/.apm/skills/issues-beads", + "symlink": null, + "lines": 807, + "cmd_style": false, + "env_coupling": [ + "~/.claude/skills/issues-beads-audit/SKILL.md", + "~/.claude/skills/issues-beads-checkpoint/SKILL.md", + "~/.claude/skills/issues-beads-evolve/SKILL.md", + "~/.claude/skills/issues-beads-orient/SKILL.md", + "~/.claude/skills/issues-beads-prime/SKILL.md" + ], + "has_scripts": false, + "has_refs": false, + "description": "Comprehensive reference for beads issue tracking with bd CLI.", + "desc_toks": [ + "beads", + "cli", + "comprehensive", + "for", + "issue", + "reference", + "tracking", + "with" + ], + "decidability_guess": "subjective" + }, + "issues-beads-audit": { + "plugin": "beads-issue-tracking-and-session-workflow", + "path": "beads-issue-tracking-and-session-workflow/.apm/skills/issues-beads-audit", + "symlink": null, + "lines": 511, + "cmd_style": true, + "env_coupling": [], + "has_scripts": false, + "has_refs": false, + "description": "Periodic beads graph health checks and maintenance procedures.", + "desc_toks": [ + "and", + "beads", + "checks", + "graph", + "health", + "maintenance", + "periodic", + "procedures" + ], + "decidability_guess": "subjective" + }, + "issues-beads-checkpoint": { + "plugin": "beads-issue-tracking-and-session-workflow", + "path": "beads-issue-tracking-and-session-workflow/.apm/skills/issues-beads-checkpoint", + "symlink": null, + "lines": 621, + "cmd_style": true, + "env_coupling": [ + "~/.claude/skills/issues-beads-checkpoint/SKILL.md", + "~/projects/sciexp/planning/schemas/stigmergic-workflow/schema.cue" + ], + "has_scripts": false, + "has_refs": false, + "description": "Session wind-down action to capture learnings into issue graph and prepare for handoff via stigmergic signal tables.", + "desc_toks": [ + "action", + "and", + "capture", + "down", + "for", + "graph", + "handoff", + "into", + "issue", + "learnings", + "prepare", + "session", + "signal", + "stigmergic", + "tables", + "via", + "wind" + ], + "decidability_guess": "subjective" + }, + "issues-beads-evolve": { + "plugin": "beads-issue-tracking-and-session-workflow", + "path": "beads-issue-tracking-and-session-workflow/.apm/skills/issues-beads-evolve", + "symlink": null, + "lines": 518, + "cmd_style": true, + "env_coupling": [ + "~/.claude/skills/issues-beads-checkpoint/SKILL.md", + "~/.claude/skills/issues-beads-evolve/SKILL.md", + "~/.claude/skills/issues-beads-orient/SKILL.md", + "~/.claude/skills/issues-beads-prime/SKILL.md", + "~/.claude/skills/issues-beads/SKILL.md" + ], + "has_scripts": false, + "has_refs": false, + "description": "Adaptive refinement patterns for evolving the issue graph during development.", + "desc_toks": [ + "adaptive", + "development", + "during", + "evolving", + "for", + "graph", + "issue", + "patterns", + "refinement", + "the" + ], + "decidability_guess": "subjective" + }, + "issues-beads-init": { + "plugin": "beads-issue-tracking-and-session-workflow", + "path": "beads-issue-tracking-and-session-workflow/.apm/skills/issues-beads-init", + "symlink": null, + "lines": 103, + "cmd_style": true, + "env_coupling": [], + "has_scripts": false, + "has_refs": false, + "description": "Initialize beads issue tracking with dolt persistence for new repositories.", + "desc_toks": [ + "beads", + "dolt", + "for", + "initialize", + "issue", + "new", + "persistence", + "repositories", + "tracking", + "with" + ], + "decidability_guess": "subjective" + }, + "issues-beads-orient": { + "plugin": "beads-issue-tracking-and-session-workflow", + "path": "beads-issue-tracking-and-session-workflow/.apm/skills/issues-beads-orient", + "symlink": null, + "lines": 475, + "cmd_style": true, + "env_coupling": [ + "~/.claude/skills/issues-beads-orient/SKILL.md" + ], + "has_scripts": false, + "has_refs": false, + "description": "Session start action to run diagnostics, synthesize project status, and identify next actions.", + "desc_toks": [ + "action", + "actions", + "and", + "diagnostics", + "identify", + "next", + "project", + "run", + "session", + "start", + "status", + "synthesize" + ], + "decidability_guess": "subjective" + }, + "issues-beads-prime": { + "plugin": "beads-issue-tracking-and-session-workflow", + "path": "beads-issue-tracking-and-session-workflow/.apm/skills/issues-beads-prime", + "symlink": null, + "lines": 214, + "cmd_style": false, + "env_coupling": [ + "~/.claude/skills/issues-beads-prime/SKILL.md", + "~/.claude/skills/jj-version-control/SKILL.md", + "~/.claude/skills/preferences-git-version-control/03-jj-mode.md", + "~/.claude/skills/preferences-git-version-control/SKILL.md" + ], + "has_scripts": false, + "has_refs": false, + "description": "Minimal quick reference for beads commands when context is constrained.", + "desc_toks": [ + "beads", + "commands", + "constrained", + "context", + "for", + "minimal", + "quick", + "reference", + "when" + ], + "decidability_guess": "subjective" + }, + "issues-beads-seed": { + "plugin": "beads-issue-tracking-and-session-workflow", + "path": "beads-issue-tracking-and-session-workflow/.apm/skills/issues-beads-seed", + "symlink": null, + "lines": 366, + "cmd_style": true, + "env_coupling": [], + "has_scripts": false, + "has_refs": false, + "description": "Translate architectural planning documents into actionable beads issue graph structure.", + "desc_toks": [ + "actionable", + "architectural", + "beads", + "documents", + "graph", + "into", + "issue", + "planning", + "structure", + "translate" + ], + "decidability_guess": "subjective" + }, + "session-advisor": { + "plugin": "beads-issue-tracking-and-session-workflow", + "path": "beads-issue-tracking-and-session-workflow/.apm/skills/session-advisor", + "symlink": null, + "lines": 116, + "cmd_style": false, + "env_coupling": [ + "~/.claude/skills/jj-version-control/diamond-workflow.md", + "~/.claude/skills/session-advisor/SKILL.md" + ], + "has_scripts": false, + "has_refs": false, + "description": "Lightweight routing advisor that reads beads graph metrics and recommends which workflow skill to invoke.", + "desc_toks": [ + "advisor", + "and", + "beads", + "graph", + "invoke", + "lightweight", + "metrics", + "reads", + "recommends", + "routing", + "skill", + "that", + "which", + "workflow" + ], + "decidability_guess": "subjective" + }, + "session-checkpoint": { + "plugin": "beads-issue-tracking-and-session-workflow", + "path": "beads-issue-tracking-and-session-workflow/.apm/skills/session-checkpoint", + "symlink": null, + "lines": 466, + "cmd_style": false, + "env_coupling": [ + "~/.claude/skills/session-checkpoint/SKILL.md", + "~/projects", + "~/projects/" + ], + "has_scripts": false, + "has_refs": false, + "description": "All-horizons session skill that captures session state, evaluates surprise, propagates context downstream, assesses documentation impact, and produces a handoff narrative sufficient for the next session's /session-orient.", + "desc_toks": [ + "all", + "and", + "assesses", + "captures", + "context", + "documentation", + "downstream", + "evaluates", + "for", + "handoff", + "horizons", + "impact", + "narrative", + "next", + "orient", + "produces", + "propagates", + "session", + "skill", + "state", + "sufficient", + "surprise", + "that", + "the" + ], + "decidability_guess": "subjective" + }, + "session-orient": { + "plugin": "beads-issue-tracking-and-session-workflow", + "path": "beads-issue-tracking-and-session-workflow/.apm/skills/session-orient", + "symlink": null, + "lines": 358, + "cmd_style": false, + "env_coupling": [ + "~/.claude/skills/jj-version-control/diamond-workflow.md", + "~/.claude/skills/jj-version-control/tiered-ceremony.md", + "~/.claude/skills/session-orient/SKILL.md" + ], + "has_scripts": false, + "has_refs": false, + "description": "Strategic horizon session skill that assembles complete session context, calibrates by Cynefin domain and planning-depth signal, and produces a self-directing briefing.", + "desc_toks": [ + "and", + "assembles", + "briefing", + "calibrates", + "complete", + "context", + "cynefin", + "depth", + "directing", + "domain", + "horizon", + "planning", + "produces", + "self", + "session", + "signal", + "skill", + "strategic", + "that" + ], + "decidability_guess": "subjective" + }, + "session-plan": { + "plugin": "beads-issue-tracking-and-session-workflow", + "path": "beads-issue-tracking-and-session-workflow/.apm/skills/session-plan", + "symlink": null, + "lines": 497, + "cmd_style": false, + "env_coupling": [ + "~/.claude/skills/jj-version-control/diamond-workflow.md", + "~/.claude/skills/session-plan/SKILL.md" + ], + "has_scripts": false, + "has_refs": false, + "description": "Tactical-to-operational transition skill that transforms scope understanding into an operational execution buffer of atomic issues with dependencies, acceptance criteria, and Cynefin classification.", + "desc_toks": [ + "acceptance", + "and", + "atomic", + "buffer", + "classification", + "criteria", + "cynefin", + "dependencies", + "execution", + "into", + "issues", + "operational", + "scope", + "skill", + "tactical", + "that", + "transforms", + "transition", + "understanding", + "with" + ], + "decidability_guess": "subjective" + }, + "session-review": { + "plugin": "beads-issue-tracking-and-session-workflow", + "path": "beads-issue-tracking-and-session-workflow/.apm/skills/session-review", + "symlink": null, + "lines": 275, + "cmd_style": false, + "env_coupling": [ + "~/.claude/skills/jj-version-control/diamond-workflow.md", + "~/.claude/skills/session-review/SKILL.md" + ], + "has_scripts": false, + "has_refs": false, + "description": "Operational-to-tactical feedback skill that verifies assembled subsystems at topological convergence points in the DAG, functioning as the System 3* audit from the Viable System Model.", + "desc_toks": [ + "assembled", + "audit", + "convergence", + "dag", + "feedback", + "from", + "functioning", + "model", + "operational", + "points", + "skill", + "subsystems", + "system", + "tactical", + "that", + "the", + "topological", + "verifies", + "viable" + ], + "decidability_guess": "subjective" + }, + "stigmergic-convention": { + "plugin": "beads-issue-tracking-and-session-workflow", + "path": "beads-issue-tracking-and-session-workflow/.apm/skills/stigmergic-convention", + "symlink": null, + "lines": 232, + "cmd_style": false, + "env_coupling": [ + "~/.claude/skills/stigmergic-convention/SKILL.md" + ], + "has_scripts": false, + "has_refs": false, + "description": "Signal table schema, field definitions, update protocol, and CUE validation for stigmergic workflow coordination.", + "desc_toks": [ + "and", + "coordination", + "cue", + "definitions", + "field", + "for", + "protocol", + "schema", + "signal", + "stigmergic", + "table", + "update", + "validation", + "workflow" + ], + "decidability_guess": "subjective" + }, + "doc-to-md": { + "plugin": "document-authoring-and-visualization", + "path": "document-authoring-and-visualization/.apm/skills/doc-to-md", + "symlink": null, + "lines": 431, + "cmd_style": true, + "env_coupling": [ + "~/projects/", + "~/projects/modeling-workspace/modeling-references" + ], + "has_scripts": false, + "has_refs": false, + "description": "Convert a scholarly reference (paper or book) from PDF or arXiv LaTeX source into a structured, modular markdown repository with indexed sections, README, and token counts.", + "desc_toks": [ + "and", + "arxiv", + "book", + "convert", + "counts", + "from", + "indexed", + "into", + "latex", + "markdown", + "modular", + "paper", + "pdf", + "readme", + "reference", + "repository", + "scholarly", + "sections", + "source", + "structured", + "token", + "with" + ], + "decidability_guess": "subjective" + }, + "doc-to-md-cmd": { + "plugin": "document-authoring-and-visualization", + "path": "document-authoring-and-visualization/.apm/skills/doc-to-md-cmd", + "symlink": null, + "lines": 68, + "cmd_style": false, + "env_coupling": [ + "~/projects/modeling-workspace/modeling-references" + ], + "has_scripts": false, + "has_refs": false, + "description": "Session-priming skill for batch construction of /doc-to-md invocations. Invoke once at session start, then provide citation and PDF path pairs repeatedly to get formatted commands copied to clipboard.", + "desc_toks": [ + "and", + "batch", + "citation", + "clipboard", + "commands", + "construction", + "copied", + "doc", + "for", + "formatted", + "get", + "invocations", + "invoke", + "once", + "pairs", + "path", + "pdf", + "priming", + "provide", + "repeatedly", + "session", + "skill", + "start", + "then" + ], + "decidability_guess": "subjective" + }, + "knowledge-graph": { + "plugin": "document-authoring-and-visualization", + "path": "document-authoring-and-visualization/.apm/skills/knowledge-graph", + "symlink": null, + "lines": 52, + "cmd_style": false, + "env_coupling": [], + "has_scripts": false, + "has_refs": true, + "description": "Index curated reference corpora into a searchable knowledge graph via the cognee engine, then query it to ground technical writing, review, and analysis. Use when ingesting reference documents into named datasets or retrieving grounding context for tasks like drafting or reviewing a manuscript. A reference-knowledge index, explicitly not agent session memory.", + "desc_toks": [ + "agent", + "analysis", + "and", + "cognee", + "context", + "corpora", + "curated", + "datasets", + "documents", + "drafting", + "engine", + "explicitly", + "for", + "graph", + "ground", + "grounding", + "index", + "ingesting", + "into", + "knowledge", + "like", + "manuscript", + "memory", + "named", + "not", + "query", + "reference", + "retrieving", + "review", + "reviewing", + "searchable", + "session", + "tasks", + "technical", + "the", + "then", + "use", + "via", + "when", + "writing" + ], + "decidability_guess": "subjective" + }, + "scientific-visualization": { + "plugin": "document-authoring-and-visualization", + "path": "document-authoring-and-visualization/.apm/skills/scientific-visualization", + "symlink": null, + "lines": 82, + "cmd_style": false, + "env_coupling": [], + "has_scripts": false, + "has_refs": false, + "description": "Perceptually and cognitively grounded guidance for designing and reviewing scientific figures, tables, diagrams, and other visual displays of data. Synthesized from Ware (2020) Information Visualization 4th ed. and Schloss (2025) Annual Review of Vision Science. Use when creating new visualizations, choosing chart types, selecting color palettes, reviewing figure quality, auditing accessibility, or evaluating any scientific display for adherence to perception research best practices. Supports two modes: design (proactive guidance for new visualizations) and review (systematic audit of existing ones).", + "desc_toks": [ + "accessibility", + "adherence", + "and", + "annual", + "any", + "audit", + "auditing", + "best", + "chart", + "choosing", + "cognitively", + "color", + "creating", + "data", + "design", + "designing", + "diagrams", + "display", + "displays", + "evaluating", + "existing", + "figure", + "figures", + "for", + "from", + "grounded", + "guidance", + "information", + "modes", + "new", + "ones", + "other", + "palettes", + "perception", + "perceptually", + "practices", + "proactive", + "quality", + "research", + "review", + "reviewing", + "schloss", + "science", + "scientific", + "selecting", + "supports", + "synthesized", + "systematic", + "tables", + "two", + "types", + "use", + "vision", + "visual", + "visualization", + "visualizations", + "ware", + "when" + ], + "decidability_guess": "subjective" + }, + "text-to-visual-iteration": { + "plugin": "document-authoring-and-visualization", + "path": "document-authoring-and-visualization/.apm/skills/text-to-visual-iteration", + "symlink": null, + "lines": 116, + "cmd_style": false, + "env_coupling": [], + "has_scripts": false, + "has_refs": true, + "description": "Mechanical workflow for iterative compilation and refinement of text-based visual specifications (tikz/pgf, typst/cetz, d2, mermaid, plantuml, graphviz, and similar text-to-visual languages) into rendered output (SVG, PNG, PDF). Covers the compile-inspect-refine iteration loop using both SVG text inspection and multimodal bitmap vision for dual-channel feedback. Use when authoring, compiling, or iteratively refining any text-to-visual pipeline, choosing output formats, setting up build artifact directories, post-processing SVG output (svgo, resvg, svg2pdf), or optimizing layout and graphical properties through successive compilation cycles.", + "desc_toks": [ + "and", + "any", + "artifact", + "authoring", + "based", + "bitmap", + "both", + "build", + "cetz", + "channel", + "choosing", + "compilation", + "compile", + "compiling", + "covers", + "cycles", + "directories", + "dual", + "feedback", + "for", + "formats", + "graphical", + "graphviz", + "inspect", + "inspection", + "into", + "iteration", + "iterative", + "iteratively", + "languages", + "layout", + "loop", + "mechanical", + "mermaid", + "multimodal", + "optimizing", + "output", + "pdf", + "pgf", + "pipeline", + "plantuml", + "png", + "post", + "processing", + "properties", + "refine", + "refinement", + "refining", + "rendered", + "resvg", + "setting", + "similar", + "specifications", + "successive", + "svg", + "svgo", + "text", + "the", + "through", + "tikz", + "typst", + "use", + "using", + "vision", + "visual", + "when", + "workflow" + ], + "decidability_guess": "decidable" + }, + "web-to-markdown": { + "plugin": "document-authoring-and-visualization", + "path": "document-authoring-and-visualization/.apm/skills/web-to-markdown", + "symlink": null, + "lines": 67, + "cmd_style": true, + "env_coupling": [], + "has_scripts": false, + "has_refs": false, + "description": "Convert a URL to clean markdown by downloading the rendered HTML and extracting main content. Use when the user provides a URL and wants it saved as markdown, or asks to download/capture/convert a webpage.", + "desc_toks": [ + "and", + "asks", + "capture", + "clean", + "content", + "convert", + "download", + "downloading", + "extracting", + "html", + "main", + "markdown", + "provides", + "rendered", + "saved", + "the", + "url", + "use", + "user", + "wants", + "webpage", + "when" + ], + "decidability_guess": "subjective" + }, + "event-modeling-brownfield": { + "plugin": "event-modeling-workflow", + "path": "event-modeling-workflow/.apm/skills/event-modeling-brownfield", + "symlink": null, + "lines": 84, + "cmd_style": true, + "env_coupling": [], + "has_scripts": false, + "has_refs": false, + "description": "Reverse engineer Event Model from existing codebase to document implicit event flows.", + "desc_toks": [ + "codebase", + "document", + "engineer", + "event", + "existing", + "flows", + "from", + "implicit", + "model", + "reverse" + ], + "decidability_guess": "subjective" + }, + "event-modeling-greenfield": { + "plugin": "event-modeling-workflow", + "path": "event-modeling-workflow/.apm/skills/event-modeling-greenfield", + "symlink": null, + "lines": 57, + "cmd_style": true, + "env_coupling": [], + "has_scripts": false, + "has_refs": false, + "description": "Start 7-step Event Modeling from scratch with Qlerify for new domain discovery.", + "desc_toks": [ + "discovery", + "domain", + "event", + "for", + "from", + "modeling", + "new", + "qlerify", + "scratch", + "start", + "step", + "with" + ], + "decidability_guess": "subjective" + }, + "event-modeling-qlerify-session": { + "plugin": "event-modeling-workflow", + "path": "event-modeling-workflow/.apm/skills/event-modeling-qlerify-session", + "symlink": null, + "lines": 75, + "cmd_style": true, + "env_coupling": [], + "has_scripts": false, + "has_refs": false, + "description": "Guide interactive Qlerify session with browser integration for collaborative event modeling.", + "desc_toks": [ + "browser", + "collaborative", + "event", + "for", + "guide", + "integration", + "interactive", + "modeling", + "qlerify", + "session", + "with" + ], + "decidability_guess": "subjective" + }, + "event-modeling-to-eventcatalog": { + "plugin": "event-modeling-workflow", + "path": "event-modeling-workflow/.apm/skills/event-modeling-to-eventcatalog", + "symlink": null, + "lines": 111, + "cmd_style": true, + "env_coupling": [], + "has_scripts": false, + "has_refs": false, + "description": "Transform Qlerify JSON export to EventCatalog MDX artifacts for documentation.", + "desc_toks": [ + "artifacts", + "documentation", + "eventcatalog", + "export", + "for", + "json", + "mdx", + "qlerify", + "transform" + ], + "decidability_guess": "decidable" + }, + "nucleus-platform": { + "plugin": "formal-specification-and-refinement", + "path": "formal-specification-and-refinement/.apm/skills/nucleus-platform", + "symlink": null, + "lines": 175, + "cmd_style": false, + "env_coupling": [], + "has_scripts": false, + "has_refs": true, + "description": "Router for the Nucleus pattern: a spec-anchored, approximately verifiable data-modeling and computational-science monorepo where Lean 4 is the structural source of truth and instantiation followed by reconstruction drives structural drift toward zero. Use when standing up or operating a Nucleus instance, applying its Lean/LinkML/Arrow boundaries, wiring its module and check layout, placing components in maturity tiers, or choosing its deliberate frozen-stack exclusions. Delegate generic techniques to the sibling skills named in this router. Do not use for generic data-platform or agent-orchestration work.", + "desc_toks": [ + "agent", + "anchored", + "and", + "applying", + "approximately", + "arrow", + "boundaries", + "check", + "choosing", + "components", + "computational", + "data", + "delegate", + "deliberate", + "drift", + "drives", + "exclusions", + "followed", + "for", + "frozen", + "generic", + "instance", + "instantiation", + "its", + "layout", + "lean", + "linkml", + "maturity", + "modeling", + "module", + "monorepo", + "named", + "not", + "nucleus", + "operating", + "orchestration", + "pattern", + "placing", + "platform", + "reconstruction", + "router", + "science", + "sibling", + "skills", + "source", + "spec", + "stack", + "standing", + "structural", + "techniques", + "the", + "this", + "tiers", + "toward", + "truth", + "use", + "verifiable", + "when", + "where", + "wiring", + "work", + "zero" + ], + "decidability_guess": "decidable" + }, + "refinement-driven-development": { + "plugin": "formal-specification-and-refinement", + "path": "formal-specification-and-refinement/.apm/skills/refinement-driven-development", + "symlink": null, + "lines": 82, + "cmd_style": false, + "env_coupling": [], + "has_scripts": false, + "has_refs": true, + "description": "Approximately-verifiable, refinement-driven development for type-driven domain-driven design. Use when modeling a domain as a dependently-typed Lean 4 specification, refining/lowering it to a Rust implementation, lifting the implementation back via Charon and Aeneas to check spec<->implementation correspondence (translation validation) \u2014 mechanically when tractable, otherwise via differential testing or LLM comparison \u2014 or when generating and diffing type-system diagrams of the model and implementation to track their evolution. Mechanical on-the-nose proof is the precise ideal, not a requirement; its absence is not a failure of the method.", + "desc_toks": [ + "absence", + "aeneas", + "and", + "approximately", + "back", + "charon", + "check", + "comparison", + "correspondence", + "dependently", + "design", + "development", + "diagrams", + "differential", + "diffing", + "domain", + "driven", + "evolution", + "failure", + "for", + "generating", + "ideal", + "implementation", + "its", + "lean", + "lifting", + "llm", + "lowering", + "mechanical", + "mechanically", + "method", + "model", + "modeling", + "nose", + "not", + "otherwise", + "precise", + "proof", + "refinement", + "refining", + "requirement", + "rust", + "spec", + "specification", + "system", + "testing", + "the", + "their", + "track", + "tractable", + "translation", + "type", + "typed", + "use", + "validation", + "verifiable", + "via", + "when" + ], + "decidability_guess": "decidable" + }, + "nix-flake-pr-cycle": { + "plugin": "nix-build-operations", + "path": "nix-build-operations/.apm/skills/nix-flake-pr-cycle", + "symlink": null, + "lines": 235, + "cmd_style": false, + "env_coupling": [ + "/home/tools/commands/dev/buildbot-logs.nix", + "/home/tools/commands/system/ntfy-send.sh", + "~/.claude/skills/jj-version-control/diamond-workflow.md" + ], + "has_scripts": false, + "has_refs": false, + "description": "Operational workflow for flake-bearing repositories \u2014 enumerate flake checks for the current platform, probe targeted slices via `nix eval` and `nix build .#checks.`, run `just check-fast` with logged output, create a draft pull request in canonical form, monitor buildbot, mark ready, and optionally trigger Mergify auto-merge. Phase 1 (enumerate and probe) is a legitimate stand-alone invocation; the later phases compose into the full validate-to-merge cycle. Exercises the closure operator described in `preferences-compositional-continuous-verification`. Load when validating a flake before push, opening a PR, monitoring buildbot, debugging a failed CI check, or auditing flake-check coverage.", + "desc_toks": [ + "alone", + "and", + "auditing", + "auto", + "bearing", + "before", + "build", + "buildbot", + "canonical", + "check", + "checks", + "closure", + "compose", + "compositional", + "continuous", + "coverage", + "create", + "current", + "cycle", + "debugging", + "described", + "draft", + "enumerate", + "eval", + "exercises", + "failed", + "fast", + "flake", + "for", + "form", + "full", + "into", + "invocation", + "just", + "later", + "legitimate", + "load", + "logged", + "mark", + "merge", + "mergify", + "monitor", + "monitoring", + "name", + "nix", + "opening", + "operational", + "operator", + "optionally", + "output", + "phase", + "phases", + "platform", + "preferences", + "probe", + "pull", + "push", + "ready", + "repositories", + "request", + "run", + "slices", + "stand", + "targeted", + "the", + "trigger", + "validate", + "validating", + "verification", + "via", + "when", + "with", + "workflow" + ], + "decidability_guess": "decidable" + }, + "nixpkgs-broken-package": { + "plugin": "nix-build-operations", + "path": "nix-build-operations/.apm/skills/nixpkgs-broken-package", + "symlink": null, + "lines": 58, + "cmd_style": true, + "env_coupling": [], + "has_scripts": false, + "has_refs": false, + "description": "Triage a broken nixpkg after flake update with systematic debugging steps.", + "desc_toks": [ + "after", + "broken", + "debugging", + "flake", + "nixpkg", + "steps", + "systematic", + "triage", + "update", + "with" + ], + "decidability_guess": "decidable" + }, + "process-compose-init": { + "plugin": "nix-build-operations", + "path": "nix-build-operations/.apm/skills/process-compose-init", + "symlink": null, + "lines": 40, + "cmd_style": true, + "env_coupling": [ + "~/projects/nix-workspace/process-compose", + "~/projects/nix-workspace/process-compose-flake", + "~/projects/nix-workspace/process-compose-flake-shell", + "~/projects/nix-workspace/services-flake" + ], + "has_scripts": false, + "has_refs": false, + "description": "Add process-compose integration with dual Nix/non-Nix support for local service orchestration.", + "desc_toks": [ + "add", + "compose", + "dual", + "for", + "integration", + "local", + "nix", + "non", + "orchestration", + "process", + "service", + "support", + "with" + ], + "decidability_guess": "subjective" + }, + "worktree-sparsity-eval": { + "plugin": "nix-build-operations", + "path": "nix-build-operations/.apm/skills/worktree-sparsity-eval", + "symlink": null, + "lines": 89, + "cmd_style": false, + "env_coupling": [ + "~/.claude/skills/jj-version-control/SKILL.md", + "~/.claude/skills/jj-version-control/tiered-ceremony.md" + ], + "has_scripts": true, + "has_refs": true, + "description": "Evaluate repository size metrics to determine whether a separate working copy should use sparse checkout. **Invoke when a separate working copy is already warranted \u2014 a `git worktree` in any mode, or a `jj workspace add` in a non-flake jj repository \u2014 and only to size that invocation.** It does not decide whether to create one. In jj mode the diamond workflow's development join remains the default for parallel chains and needs no sparsity evaluation; see `~/.claude/skills/jj-version-control/tiered-ceremony.md` for the triggers and `~/.claude/skills/jj-version-control/SKILL.md` \u00a7\"Worktree interop\" for the discipline. Also invoke for periodic re-evaluation when a repo has grown significantly.", + "desc_toks": [ + "add", + "already", + "also", + "and", + "any", + "ceremony", + "chains", + "checkout", + "claude", + "control", + "copy", + "create", + "decide", + "default", + "determine", + "development", + "diamond", + "discipline", + "does", + "evaluate", + "evaluation", + "flake", + "for", + "git", + "grown", + "has", + "interop", + "invocation", + "invoke", + "join", + "metrics", + "mode", + "needs", + "non", + "not", + "one", + "only", + "parallel", + "periodic", + "remains", + "repo", + "repository", + "see", + "separate", + "should", + "significantly", + "size", + "skill", + "skills", + "sparse", + "sparsity", + "that", + "the", + "tiered", + "triggers", + "use", + "version", + "warranted", + "when", + "whether", + "workflow", + "working", + "workspace", + "worktree" + ], + "decidability_guess": "decidable" + }, + "agentic-planning-development-workflow": { + "plugin": "planning-and-development", + "path": "planning-and-development/.apm/skills/agentic-planning-development-workflow", + "symlink": null, + "lines": 135, + "cmd_style": false, + "env_coupling": [], + "has_scripts": false, + "has_refs": true, + "description": "State-machine router across a Linear-canonical board with an AFK/HIL/Manual execution-mode fork. Load when selecting an execution mode for a change, driving the agentic planning-and-development board, or routing a change between board states.", + "desc_toks": [ + "across", + "afk", + "agentic", + "and", + "between", + "board", + "canonical", + "change", + "development", + "driving", + "execution", + "for", + "fork", + "hil", + "linear", + "load", + "machine", + "manual", + "mode", + "planning", + "router", + "routing", + "selecting", + "state", + "states", + "the", + "when", + "with" + ], + "decidability_guess": "subjective" + }, + "openspec-apply-change": { + "plugin": "planning-and-development", + "path": "planning-and-development/.apm/skills/openspec-apply-change", + "symlink": null, + "lines": 185, + "cmd_style": false, + "env_coupling": [], + "has_scripts": false, + "has_refs": false, + "description": "Implement tasks from an OpenSpec change. Use when the user wants to start implementing, continue implementation, or work through tasks.", + "desc_toks": [ + "change", + "continue", + "from", + "implement", + "implementation", + "implementing", + "openspec", + "start", + "tasks", + "the", + "through", + "use", + "user", + "wants", + "when", + "work" + ], + "decidability_guess": "subjective" + }, + "openspec-archive-change": { + "plugin": "planning-and-development", + "path": "planning-and-development/.apm/skills/openspec-archive-change", + "symlink": null, + "lines": 182, + "cmd_style": false, + "env_coupling": [], + "has_scripts": false, + "has_refs": false, + "description": "Archive a completed change in the experimental workflow. Use when the user wants to finalize and archive a change after implementation is complete.", + "desc_toks": [ + "after", + "and", + "archive", + "change", + "complete", + "completed", + "experimental", + "finalize", + "implementation", + "the", + "use", + "user", + "wants", + "when", + "workflow" + ], + "decidability_guess": "subjective" + }, + "openspec-bdd-bridge": { + "plugin": "planning-and-development", + "path": "planning-and-development/.apm/skills/openspec-bdd-bridge", + "symlink": null, + "lines": 79, + "cmd_style": false, + "env_coupling": [], + "has_scripts": false, + "has_refs": false, + "description": "Keep the OpenSpec spec encoding and the Gherkin feature encoding aligned as one acceptance layer for a change whose delta specs carry behavioral requirements. Load under an OpenSpec change to lay out and align delta specs with their .feature files before apply, to route each requirement through the atdd-outer-loop Gate 1 so only behavioral requirements become committed scenarios, and to reconcile the correspondence before archive.", + "desc_toks": [ + "acceptance", + "align", + "aligned", + "and", + "apply", + "archive", + "atdd", + "become", + "before", + "behavioral", + "carry", + "change", + "committed", + "correspondence", + "delta", + "each", + "encoding", + "feature", + "files", + "for", + "gate", + "gherkin", + "keep", + "lay", + "layer", + "load", + "loop", + "one", + "only", + "openspec", + "out", + "outer", + "reconcile", + "requirement", + "requirements", + "route", + "scenarios", + "spec", + "specs", + "the", + "their", + "through", + "under", + "whose", + "with" + ], + "decidability_guess": "subjective" + }, + "openspec-bulk-archive-change": { + "plugin": "planning-and-development", + "path": "planning-and-development/.apm/skills/openspec-bulk-archive-change", + "symlink": null, + "lines": 339, + "cmd_style": false, + "env_coupling": [], + "has_scripts": false, + "has_refs": false, + "description": "Archive multiple completed changes at once. Use when archiving several parallel changes.", + "desc_toks": [ + "archive", + "archiving", + "changes", + "completed", + "multiple", + "once", + "parallel", + "several", + "use", + "when" + ], + "decidability_guess": "subjective" + }, + "openspec-continue-change": { + "plugin": "planning-and-development", + "path": "planning-and-development/.apm/skills/openspec-continue-change", + "symlink": null, + "lines": 118, + "cmd_style": false, + "env_coupling": [], + "has_scripts": false, + "has_refs": false, + "description": "Continue working on an OpenSpec change by creating the next artifact. Use when the user wants to progress their change, create the next artifact, or continue their workflow.", + "desc_toks": [ + "artifact", + "change", + "continue", + "create", + "creating", + "next", + "openspec", + "progress", + "the", + "their", + "use", + "user", + "wants", + "when", + "workflow", + "working" + ], + "decidability_guess": "subjective" + }, + "openspec-explore": { + "plugin": "planning-and-development", + "path": "planning-and-development/.apm/skills/openspec-explore", + "symlink": null, + "lines": 308, + "cmd_style": false, + "env_coupling": [], + "has_scripts": false, + "has_refs": false, + "description": "Enter explore mode - a thinking partner for exploring ideas, investigating problems, and clarifying requirements. Use when the user wants to think through something before or during a change.", + "desc_toks": [ + "and", + "before", + "change", + "clarifying", + "during", + "enter", + "explore", + "exploring", + "for", + "ideas", + "investigating", + "mode", + "partner", + "problems", + "requirements", + "something", + "the", + "think", + "thinking", + "through", + "use", + "user", + "wants", + "when" + ], + "decidability_guess": "subjective" + }, + "openspec-ff-change": { + "plugin": "planning-and-development", + "path": "planning-and-development/.apm/skills/openspec-ff-change", + "symlink": null, + "lines": 113, + "cmd_style": false, + "env_coupling": [], + "has_scripts": false, + "has_refs": false, + "description": "Fast-forward through OpenSpec artifact creation. Use when the user wants to quickly create all artifacts needed for implementation without stepping through each one individually.", + "desc_toks": [ + "all", + "artifact", + "artifacts", + "create", + "creation", + "each", + "fast", + "for", + "forward", + "implementation", + "individually", + "needed", + "one", + "openspec", + "quickly", + "stepping", + "the", + "through", + "use", + "user", + "wants", + "when", + "without" + ], + "decidability_guess": "subjective" + }, + "openspec-linear-sync": { + "plugin": "planning-and-development", + "path": "planning-and-development/.apm/skills/openspec-linear-sync", + "symlink": null, + "lines": 48, + "cmd_style": false, + "env_coupling": [], + "has_scripts": false, + "has_refs": true, + "description": "linear-cli-driven Linear-to-OpenSpec lifecycle sync overlay that binds one Linear story per OpenSpec change, supports concurrent changes across multiple Linear teams and projects in one monorepo via the openspec/linear.yaml registry, mirrors lifecycle phase to Linear state, and runs the archive-time document upsert. Load when binding a Linear story to an OpenSpec change, mirroring lifecycle phase to Linear state, or running the archive-time document upsert.", + "desc_toks": [ + "across", + "and", + "archive", + "binding", + "binds", + "change", + "changes", + "cli", + "concurrent", + "document", + "driven", + "lifecycle", + "linear", + "load", + "mirroring", + "mirrors", + "monorepo", + "multiple", + "one", + "openspec", + "overlay", + "per", + "phase", + "projects", + "registry", + "running", + "runs", + "state", + "story", + "supports", + "sync", + "teams", + "that", + "the", + "time", + "upsert", + "via", + "when", + "yaml" + ], + "decidability_guess": "decidable" + }, + "openspec-new-change": { + "plugin": "planning-and-development", + "path": "planning-and-development/.apm/skills/openspec-new-change", + "symlink": null, + "lines": 77, + "cmd_style": false, + "env_coupling": [], + "has_scripts": false, + "has_refs": false, + "description": "Start a new OpenSpec change using the experimental artifact workflow. Use when the user wants to create a new feature, fix, or modification with a structured step-by-step approach.", + "desc_toks": [ + "approach", + "artifact", + "change", + "create", + "experimental", + "feature", + "fix", + "modification", + "new", + "openspec", + "start", + "step", + "structured", + "the", + "use", + "user", + "using", + "wants", + "when", + "with", + "workflow" + ], + "decidability_guess": "subjective" + }, + "openspec-onboard": { + "plugin": "planning-and-development", + "path": "planning-and-development/.apm/skills/openspec-onboard", + "symlink": null, + "lines": 561, + "cmd_style": false, + "env_coupling": [], + "has_scripts": false, + "has_refs": false, + "description": "Guided onboarding for OpenSpec - walk through a complete workflow cycle with narration and real codebase work.", + "desc_toks": [ + "and", + "codebase", + "complete", + "cycle", + "for", + "guided", + "narration", + "onboarding", + "openspec", + "real", + "through", + "walk", + "with", + "work", + "workflow" + ], + "decidability_guess": "subjective" + }, + "openspec-propose": { + "plugin": "planning-and-development", + "path": "planning-and-development/.apm/skills/openspec-propose", + "symlink": null, + "lines": 149, + "cmd_style": false, + "env_coupling": [], + "has_scripts": false, + "has_refs": false, + "description": "Propose a new change with all artifacts generated in one step. Use when the user wants to quickly describe what they want to build and get a complete proposal with design, specs, and tasks ready for implementation.", + "desc_toks": [ + "all", + "and", + "artifacts", + "build", + "change", + "complete", + "describe", + "design", + "for", + "generated", + "get", + "implementation", + "new", + "one", + "proposal", + "propose", + "quickly", + "ready", + "specs", + "step", + "tasks", + "the", + "they", + "use", + "user", + "want", + "wants", + "what", + "when", + "with" + ], + "decidability_guess": "decidable" + }, + "openspec-sync-specs": { + "plugin": "planning-and-development", + "path": "planning-and-development/.apm/skills/openspec-sync-specs", + "symlink": null, + "lines": 262, + "cmd_style": false, + "env_coupling": [], + "has_scripts": false, + "has_refs": false, + "description": "Sync delta specs from a change to main specs. Use when the user wants to update main specs with changes from a delta spec, without archiving the change.", + "desc_toks": [ + "archiving", + "change", + "changes", + "delta", + "from", + "main", + "spec", + "specs", + "sync", + "the", + "update", + "use", + "user", + "wants", + "when", + "with", + "without" + ], + "decidability_guess": "subjective" + }, + "openspec-update-change": { + "plugin": "planning-and-development", + "path": "planning-and-development/.apm/skills/openspec-update-change", + "symlink": null, + "lines": 91, + "cmd_style": false, + "env_coupling": [], + "has_scripts": false, + "has_refs": false, + "description": "Update an OpenSpec change by revising its existing planning artifacts and keeping them coherent with one another. Use when the user wants to revise a change's plan, fold new decisions into it, or reconcile its artifacts after an edit. Never edits code.", + "desc_toks": [ + "after", + "and", + "another", + "artifacts", + "change", + "code", + "coherent", + "decisions", + "edit", + "edits", + "existing", + "fold", + "into", + "its", + "keeping", + "never", + "new", + "one", + "openspec", + "plan", + "planning", + "reconcile", + "revise", + "revising", + "the", + "them", + "update", + "use", + "user", + "wants", + "when", + "with" + ], + "decidability_guess": "subjective" + }, + "openspec-verify-change": { + "plugin": "planning-and-development", + "path": "planning-and-development/.apm/skills/openspec-verify-change", + "symlink": null, + "lines": 175, + "cmd_style": false, + "env_coupling": [], + "has_scripts": false, + "has_refs": false, + "description": "Verify implementation matches change artifacts. Use when the user wants to validate that implementation is complete, correct, and coherent before archiving.", + "desc_toks": [ + "and", + "archiving", + "artifacts", + "before", + "change", + "coherent", + "complete", + "correct", + "implementation", + "matches", + "that", + "the", + "use", + "user", + "validate", + "verify", + "wants", + "when" + ], + "decidability_guess": "subjective" + }, + "project-management": { + "plugin": "planning-and-development", + "path": "planning-and-development/.apm/skills/project-management", + "symlink": null, + "lines": 67, + "cmd_style": false, + "env_coupling": [], + "has_scripts": false, + "has_refs": true, + "description": "Human-facing project-management hub for the Linear Method ontology, conventions, and the workspace safety gate. Load when reasoning about project/issue structure, Linear conventions, or how OpenSpec, beads, and GitHub relate as PM layers.", + "desc_toks": [ + "about", + "and", + "beads", + "conventions", + "facing", + "for", + "gate", + "github", + "how", + "hub", + "human", + "issue", + "layers", + "linear", + "load", + "management", + "method", + "ontology", + "openspec", + "project", + "reasoning", + "relate", + "safety", + "structure", + "the", + "when", + "workspace" + ], + "decidability_guess": "subjective" + }, + "preferences-change-management": { + "plugin": "preferences-code-and-collaboration-conventions", + "path": "preferences-code-and-collaboration-conventions/.apm/skills/preferences-change-management", + "symlink": null, + "lines": 8, + "cmd_style": false, + "env_coupling": [], + "has_scripts": false, + "has_refs": false, + "description": "Change management practices for controlled evolution of systems and codebases. Load when planning migrations, deprecations, or breaking changes.", + "desc_toks": [ + "and", + "breaking", + "change", + "changes", + "codebases", + "controlled", + "deprecations", + "evolution", + "for", + "load", + "management", + "migrations", + "planning", + "practices", + "systems", + "when" + ], + "decidability_guess": "subjective" + }, + "preferences-comment-cleanup": { + "plugin": "preferences-code-and-collaboration-conventions", + "path": "preferences-code-and-collaboration-conventions/.apm/skills/preferences-comment-cleanup", + "symlink": null, + "lines": 58, + "cmd_style": false, + "env_coupling": [ + "~/.claude/skills/preferences-comment-cleanup/uncommentrc.toml", + "~/.claude/skills/preferences-style-and-conventions/SKILL.md" + ], + "has_scripts": false, + "has_refs": false, + "description": "Uncomment-driven workflow for auditing and removing noise comments while preserving load-bearing markers, as the operational arm of the code-comments policy. Load when you remove comments, strip comments, uncomment or declutter code, run a comment cleanup, or systematically prune noise comments across a subtree.", + "desc_toks": [ + "across", + "and", + "arm", + "auditing", + "bearing", + "cleanup", + "code", + "comment", + "comments", + "declutter", + "driven", + "for", + "load", + "markers", + "noise", + "operational", + "policy", + "preserving", + "prune", + "remove", + "removing", + "run", + "strip", + "subtree", + "systematically", + "the", + "uncomment", + "when", + "while", + "workflow", + "you" + ], + "decidability_guess": "subjective" + }, + "preferences-documentation": { + "plugin": "preferences-code-and-collaboration-conventions", + "path": "preferences-code-and-collaboration-conventions/.apm/skills/preferences-documentation", + "symlink": null, + "lines": 273, + "cmd_style": false, + "env_coupling": [ + "~/.claude/skills/preferences-git-version-control/SKILL.md" + ], + "has_scripts": false, + "has_refs": true, + "description": "Documentation conventions including structure, formatting, and maintenance practices. Load when writing or reviewing documentation.", + "desc_toks": [ + "and", + "conventions", + "documentation", + "formatting", + "including", + "load", + "maintenance", + "practices", + "reviewing", + "structure", + "when", + "writing" + ], + "decidability_guess": "subjective" + }, + "preferences-git-history-cleanup": { + "plugin": "preferences-code-and-collaboration-conventions", + "path": "preferences-code-and-collaboration-conventions/.apm/skills/preferences-git-history-cleanup", + "symlink": null, + "lines": 316, + "cmd_style": false, + "env_coupling": [ + "~/.claude/skills/jj-history-cleanup/SKILL.md", + "~/.claude/skills/jj-version-control/diamond-workflow.md", + "~/.claude/skills/preferences-git-version-control/SKILL.md" + ], + "has_scripts": false, + "has_refs": false, + "description": "Git history cleanup patterns for rewriting, squashing, and reorganizing commit history. Load when cleaning up branch history before merge.", + "desc_toks": [ + "and", + "before", + "branch", + "cleaning", + "cleanup", + "commit", + "for", + "git", + "history", + "load", + "merge", + "patterns", + "reorganizing", + "rewriting", + "squashing", + "when" + ], + "decidability_guess": "decidable" + }, + "preferences-git-version-control": { + "plugin": "preferences-code-and-collaboration-conventions", + "path": "preferences-code-and-collaboration-conventions/.apm/skills/preferences-git-version-control", + "symlink": null, + "lines": 219, + "cmd_style": false, + "env_coupling": [ + "~/.claude/skills/gitbutler-but-cli/SKILL.md", + "~/.claude/skills/issues-beads-prime/SKILL.md", + "~/.claude/skills/issues-beads/SKILL.md", + "~/.claude/skills/jj-summary/SKILL.md", + "~/.claude/skills/jj-version-control/SKILL.md" + ], + "has_scripts": false, + "has_refs": false, + "description": "Git version control conventions including atomic commits, branch workflow, and commit formatting. Load when making commits or managing branches.", + "desc_toks": [ + "and", + "atomic", + "branch", + "branches", + "commit", + "commits", + "control", + "conventions", + "formatting", + "git", + "including", + "load", + "making", + "managing", + "version", + "when", + "workflow" + ], + "decidability_guess": "decidable" + }, + "preferences-style-and-conventions": { + "plugin": "preferences-code-and-collaboration-conventions", + "path": "preferences-code-and-collaboration-conventions/.apm/skills/preferences-style-and-conventions", + "symlink": null, + "lines": 187, + "cmd_style": false, + "env_coupling": [ + "~/.claude/skills/preferences-documentation/SKILL.md", + "~/ghq/", + "~/ghq/...", + "~/projects", + "~/projects/" + ], + "has_scripts": false, + "has_refs": false, + "description": "Style and formatting conventions for code, documentation, naming, and file organization. Load when reviewing style consistency or setting up new files.", + "desc_toks": [ + "and", + "code", + "consistency", + "conventions", + "documentation", + "file", + "files", + "for", + "formatting", + "load", + "naming", + "new", + "organization", + "reviewing", + "setting", + "style", + "when" + ], + "decidability_guess": "decidable" + }, + "preferences-data-modeling": { + "plugin": "preferences-data-and-scientific-computing", + "path": "preferences-data-and-scientific-computing/.apm/skills/preferences-data-modeling", + "symlink": null, + "lines": 367, + "cmd_style": false, + "env_coupling": [ + "~/.claude/skills/preferences-algebraic-data-types/SKILL.md", + "~/.claude/skills/preferences-railway-oriented-programming/SKILL.md", + "~/projects/duckdb/tools/juliapkg", + "~/projects/omicslake-workspace/duckdb-rs", + "~/projects/omicslake-workspace/slaf" + ], + "has_scripts": false, + "has_refs": false, + "description": "Data modeling conventions including Arrow-based columnar interchange, DuckDB/DuckLake lakehouse patterns, SQLMesh transformations, and scientific data contracts. Load when designing database structures, data pipelines, cross-language analytics, or data relationships.", + "desc_toks": [ + "analytics", + "and", + "arrow", + "based", + "columnar", + "contracts", + "conventions", + "cross", + "data", + "database", + "designing", + "duckdb", + "ducklake", + "including", + "interchange", + "lakehouse", + "language", + "load", + "modeling", + "patterns", + "pipelines", + "relationships", + "scientific", + "sqlmesh", + "structures", + "transformations", + "when" + ], + "decidability_guess": "subjective" + }, + "preferences-json-querying": { + "plugin": "preferences-data-and-scientific-computing", + "path": "preferences-data-and-scientific-computing/.apm/skills/preferences-json-querying", + "symlink": null, + "lines": 326, + "cmd_style": false, + "env_coupling": [], + "has_scripts": false, + "has_refs": false, + "description": "JSON querying tools and patterns using DuckDB and jaq for data extraction and transformation. Load when querying JSON data or writing data pipelines.", + "desc_toks": [ + "and", + "data", + "duckdb", + "extraction", + "for", + "jaq", + "json", + "load", + "patterns", + "pipelines", + "querying", + "tools", + "transformation", + "using", + "when", + "writing" + ], + "decidability_guess": "decidable" + }, + "preferences-scalable-probabilistic-modeling-workflow": { + "plugin": "preferences-data-and-scientific-computing", + "path": "preferences-data-and-scientific-computing/.apm/skills/preferences-scalable-probabilistic-modeling-workflow", + "symlink": null, + "lines": 60, + "cmd_style": false, + "env_coupling": [], + "has_scripts": false, + "has_refs": false, + "description": "Principled Bayesian workflow for simulation-based inference on stochastic dynamical systems, extending Betancourt's iterative methodology with adaptations for implicit likelihoods and amortized neural posterior approximation. Load when doing simulation-based inference, stochastic dynamical systems modeling, amortized inference architecture design, or principled Bayesian workflow planning.", + "desc_toks": [ + "adaptations", + "amortized", + "and", + "approximation", + "architecture", + "based", + "bayesian", + "betancourt", + "design", + "doing", + "dynamical", + "extending", + "for", + "implicit", + "inference", + "iterative", + "likelihoods", + "load", + "methodology", + "modeling", + "neural", + "planning", + "posterior", + "principled", + "simulation", + "stochastic", + "systems", + "when", + "with", + "workflow" + ], + "decidability_guess": "subjective" + }, + "preferences-scientific-inquiry-methodology": { + "plugin": "preferences-data-and-scientific-computing", + "path": "preferences-data-and-scientific-computing/.apm/skills/preferences-scientific-inquiry-methodology", + "symlink": null, + "lines": 46, + "cmd_style": false, + "env_coupling": [], + "has_scripts": false, + "has_refs": false, + "description": "Philosophical and methodological framework for scientific inquiry grounded in Peircean pragmatism, effective theory construction, and principled iterative model building. Load when reasoning about scientific methodology, model construction and criticism, evidential standards, or the relationship between mechanistic understanding and statistical modeling.", + "desc_toks": [ + "about", + "and", + "between", + "building", + "construction", + "criticism", + "effective", + "evidential", + "for", + "framework", + "grounded", + "inquiry", + "iterative", + "load", + "mechanistic", + "methodological", + "methodology", + "model", + "modeling", + "peircean", + "philosophical", + "pragmatism", + "principled", + "reasoning", + "relationship", + "scientific", + "standards", + "statistical", + "the", + "theory", + "understanding", + "when" + ], + "decidability_guess": "subjective" + }, + "preferences-workflow-orchestration-algebra": { + "plugin": "preferences-data-and-scientific-computing", + "path": "preferences-data-and-scientific-computing/.apm/skills/preferences-workflow-orchestration-algebra", + "symlink": null, + "lines": 123, + "cmd_style": false, + "env_coupling": [], + "has_scripts": false, + "has_refs": false, + "description": "Algebraic and categorical reading of data/pipeline workflow orchestrators (Dagster, Flyte) through the Build Systems \u00e0 la Carte lens, plus the functional-programming and CCV discipline for the gaps the orchestrator does not enforce itself. Scoped to data-pipeline orchestration, not agent/subagent workflow DAGs. Load when mapping a Dagster asset graph to a free-term / store-interpreter structure, choosing between asset-based and task-based orchestrators, writing a lawful IO manager, reasoning about static-vs-dynamic dependencies in a pipeline, or enforcing type-safe FP discipline on Dagster or Flyte pipelines in Python.", + "desc_toks": [ + "about", + "agent", + "algebraic", + "and", + "asset", + "based", + "between", + "build", + "carte", + "categorical", + "ccv", + "choosing", + "dags", + "dagster", + "data", + "dependencies", + "discipline", + "does", + "dynamic", + "enforce", + "enforcing", + "flyte", + "for", + "free", + "functional", + "gaps", + "graph", + "interpreter", + "itself", + "lawful", + "lens", + "load", + "manager", + "mapping", + "not", + "orchestration", + "orchestrator", + "orchestrators", + "pipeline", + "pipelines", + "plus", + "programming", + "python", + "reading", + "reasoning", + "safe", + "scoped", + "static", + "store", + "structure", + "subagent", + "systems", + "task", + "term", + "the", + "through", + "type", + "when", + "workflow", + "writing" + ], + "decidability_guess": "decidable" + }, + "preferences-architectural-patterns": { + "plugin": "preferences-domain-driven-architecture", + "path": "preferences-domain-driven-architecture/.apm/skills/preferences-architectural-patterns", + "symlink": null, + "lines": 890, + "cmd_style": false, + "env_coupling": [], + "has_scripts": false, + "has_refs": false, + "description": "Architectural patterns including hexagonal architecture, CQRS, and event-driven design. Load when making architectural decisions or reviewing system design.", + "desc_toks": [ + "and", + "architectural", + "architecture", + "cqrs", + "decisions", + "design", + "driven", + "event", + "hexagonal", + "including", + "load", + "making", + "patterns", + "reviewing", + "system", + "when" + ], + "decidability_guess": "subjective" + }, + "preferences-architecture-diagramming": { + "plugin": "preferences-domain-driven-architecture", + "path": "preferences-domain-driven-architecture/.apm/skills/preferences-architecture-diagramming", + "symlink": null, + "lines": 79, + "cmd_style": false, + "env_coupling": [], + "has_scripts": false, + "has_refs": false, + "description": "Architecture diagramming framework based on C4 zoom hierarchy with Richards/Ford visual conventions, Wlaschin bounded context mapping, and Dilger event modeling patterns. Covers format selection (D2, TikZ, Mermaid, PlantUML), diagram compendium specification, and integration with text-to-visual-iteration and scientific-visualization skills. Load when creating architecture diagrams, selecting diagramming tools, applying C4 levels to system documentation, or deciding which diagrams a project needs.", + "desc_toks": [ + "and", + "applying", + "architecture", + "based", + "bounded", + "compendium", + "context", + "conventions", + "covers", + "creating", + "deciding", + "diagram", + "diagramming", + "diagrams", + "dilger", + "documentation", + "event", + "ford", + "format", + "framework", + "hierarchy", + "integration", + "iteration", + "levels", + "load", + "mapping", + "mermaid", + "modeling", + "needs", + "patterns", + "plantuml", + "project", + "richards", + "scientific", + "selecting", + "selection", + "skills", + "specification", + "system", + "text", + "tikz", + "tools", + "visual", + "visualization", + "when", + "which", + "with", + "wlaschin", + "zoom" + ], + "decidability_guess": "subjective" + }, + "preferences-bounded-context-design": { + "plugin": "preferences-domain-driven-architecture", + "path": "preferences-domain-driven-architecture/.apm/skills/preferences-bounded-context-design", + "symlink": null, + "lines": 757, + "cmd_style": false, + "env_coupling": [], + "has_scripts": false, + "has_refs": false, + "description": "Bounded context design including context mapping, integration patterns, and anti-corruption layers. Load when designing service boundaries or system integration.", + "desc_toks": [ + "and", + "anti", + "boundaries", + "bounded", + "context", + "corruption", + "design", + "designing", + "including", + "integration", + "layers", + "load", + "mapping", + "patterns", + "service", + "system", + "when" + ], + "decidability_guess": "subjective" + }, + "preferences-collaborative-modeling": { + "plugin": "preferences-domain-driven-architecture", + "path": "preferences-domain-driven-architecture/.apm/skills/preferences-collaborative-modeling", + "symlink": null, + "lines": 897, + "cmd_style": false, + "env_coupling": [], + "has_scripts": false, + "has_refs": false, + "description": "Collaborative modeling techniques including EventStorming and Domain Storytelling. Load when facilitating discovery workshops or modeling domain processes.", + "desc_toks": [ + "and", + "collaborative", + "discovery", + "domain", + "eventstorming", + "facilitating", + "including", + "load", + "modeling", + "processes", + "storytelling", + "techniques", + "when", + "workshops" + ], + "decidability_guess": "subjective" + }, + "preferences-discovery-process": { + "plugin": "preferences-domain-driven-architecture", + "path": "preferences-domain-driven-architecture/.apm/skills/preferences-discovery-process", + "symlink": null, + "lines": 618, + "cmd_style": false, + "env_coupling": [], + "has_scripts": false, + "has_refs": false, + "description": "Discovery process methodology for exploring problem domains before implementation. Load when starting new projects or investigating unfamiliar domains.", + "desc_toks": [ + "before", + "discovery", + "domains", + "exploring", + "for", + "implementation", + "investigating", + "load", + "methodology", + "new", + "problem", + "process", + "projects", + "starting", + "unfamiliar", + "when" + ], + "decidability_guess": "subjective" + }, + "preferences-domain-modeling": { + "plugin": "preferences-domain-driven-architecture", + "path": "preferences-domain-driven-architecture/.apm/skills/preferences-domain-modeling", + "symlink": null, + "lines": 1558, + "cmd_style": false, + "env_coupling": [], + "has_scripts": false, + "has_refs": false, + "description": "Functional domain modeling with DDD principles, aggregate design, smart constructors, and validation patterns. Load when designing domain types or business logic.", + "desc_toks": [ + "aggregate", + "and", + "business", + "constructors", + "ddd", + "design", + "designing", + "domain", + "functional", + "load", + "logic", + "modeling", + "patterns", + "principles", + "smart", + "types", + "validation", + "when", + "with" + ], + "decidability_guess": "subjective" + }, + "preferences-strategic-domain-analysis": { + "plugin": "preferences-domain-driven-architecture", + "path": "preferences-domain-driven-architecture/.apm/skills/preferences-strategic-domain-analysis", + "symlink": null, + "lines": 459, + "cmd_style": false, + "env_coupling": [], + "has_scripts": false, + "has_refs": false, + "description": "Strategic domain analysis for Core/Supporting/Generic subdomain classification. Load when assessing domain boundaries or prioritizing development investment.", + "desc_toks": [ + "analysis", + "assessing", + "boundaries", + "classification", + "core", + "development", + "domain", + "for", + "generic", + "investment", + "load", + "prioritizing", + "strategic", + "subdomain", + "supporting", + "when" + ], + "decidability_guess": "subjective" + }, + "preferences-event-catalog-qlerify": { + "plugin": "preferences-event-driven-systems", + "path": "preferences-event-driven-systems/.apm/skills/preferences-event-catalog-qlerify", + "symlink": null, + "lines": 1165, + "cmd_style": false, + "env_coupling": [], + "has_scripts": false, + "has_refs": false, + "description": "Qlerify to EventCatalog transformation workflow for converting event models to catalog artifacts. Load when transforming Qlerify exports to EventCatalog MDX.", + "desc_toks": [ + "artifacts", + "catalog", + "converting", + "event", + "eventcatalog", + "exports", + "for", + "load", + "mdx", + "models", + "qlerify", + "transformation", + "transforming", + "when", + "workflow" + ], + "decidability_guess": "subjective" + }, + "preferences-event-catalog-tooling": { + "plugin": "preferences-event-driven-systems", + "path": "preferences-event-driven-systems/.apm/skills/preferences-event-catalog-tooling", + "symlink": null, + "lines": 334, + "cmd_style": false, + "env_coupling": [ + "~/projects/lakescope-workspace/create-eventcatalog", + "~/projects/lakescope-workspace/eventcatalog", + "~/projects/lakescope-workspace/eventcatalog-mcp-server", + "~/projects/lakescope-workspace/eventstorm-to-catalog" + ], + "has_scripts": false, + "has_refs": false, + "description": "EventCatalog tooling for schema documentation and event catalog management. Load when documenting events, commands, or domain schemas.", + "desc_toks": [ + "and", + "catalog", + "commands", + "documentation", + "documenting", + "domain", + "event", + "eventcatalog", + "events", + "for", + "load", + "management", + "schema", + "schemas", + "tooling", + "when" + ], + "decidability_guess": "subjective" + }, + "preferences-event-modeling": { + "plugin": "preferences-event-driven-systems", + "path": "preferences-event-driven-systems/.apm/skills/preferences-event-modeling", + "symlink": null, + "lines": 853, + "cmd_style": false, + "env_coupling": [], + "has_scripts": false, + "has_refs": false, + "description": "Event Modeling methodology with 7-step process, Qlerify integration, and D2 diagrams. Load when designing event-driven systems or creating event models.", + "desc_toks": [ + "and", + "creating", + "designing", + "diagrams", + "driven", + "event", + "integration", + "load", + "methodology", + "modeling", + "models", + "process", + "qlerify", + "step", + "systems", + "when", + "with" + ], + "decidability_guess": "subjective" + }, + "preferences-event-sourcing": { + "plugin": "preferences-event-driven-systems", + "path": "preferences-event-driven-systems/.apm/skills/preferences-event-sourcing", + "symlink": null, + "lines": 1273, + "cmd_style": false, + "env_coupling": [], + "has_scripts": false, + "has_refs": false, + "description": "Event sourcing patterns including event replay, state reconstruction, and CQRS. Load when implementing event-sourced aggregates or projections.", + "desc_toks": [ + "aggregates", + "and", + "cqrs", + "event", + "implementing", + "including", + "load", + "patterns", + "projections", + "reconstruction", + "replay", + "sourced", + "sourcing", + "state", + "when" + ], + "decidability_guess": "subjective" + }, + "preferences-schema-versioning": { + "plugin": "preferences-event-driven-systems", + "path": "preferences-event-driven-systems/.apm/skills/preferences-schema-versioning", + "symlink": null, + "lines": 610, + "cmd_style": false, + "env_coupling": [ + "~/.claude/skills/preferences-algebraic-data-types/SKILL.md", + "~/.claude/skills/preferences-railway-oriented-programming/SKILL.md", + "~/.claude/skills/preferences-railway-oriented-programming/SKILL.md." + ], + "has_scripts": false, + "has_refs": false, + "description": "Schema versioning strategies for evolving data schemas without breaking consumers. Load when planning schema migrations or versioning event schemas.", + "desc_toks": [ + "breaking", + "consumers", + "data", + "event", + "evolving", + "for", + "load", + "migrations", + "planning", + "schema", + "schemas", + "strategies", + "versioning", + "when", + "without" + ], + "decidability_guess": "subjective" + }, + "preferences-algebraic-data-types": { + "plugin": "preferences-functional-programming-theory", + "path": "preferences-functional-programming-theory/.apm/skills/preferences-algebraic-data-types", + "symlink": null, + "lines": 1152, + "cmd_style": false, + "env_coupling": [ + "~/.claude/skills/preferences-railway-oriented-programming/SKILL.md", + "~/.claude/skills/preferences-schema-versioning/SKILL.md" + ], + "has_scripts": false, + "has_refs": false, + "description": "Algebraic data type patterns including sum types, product types, and pattern matching across languages. Load when designing type hierarchies or working with discriminated unions.", + "desc_toks": [ + "across", + "algebraic", + "and", + "data", + "designing", + "discriminated", + "hierarchies", + "including", + "languages", + "load", + "matching", + "pattern", + "patterns", + "product", + "sum", + "type", + "types", + "unions", + "when", + "with", + "working" + ], + "decidability_guess": "subjective" + }, + "preferences-algebraic-laws": { + "plugin": "preferences-functional-programming-theory", + "path": "preferences-functional-programming-theory/.apm/skills/preferences-algebraic-laws", + "symlink": null, + "lines": 943, + "cmd_style": false, + "env_coupling": [], + "has_scripts": false, + "has_refs": false, + "description": "Algebraic laws including functor/monad laws and property-based testing strategies. Load when verifying algebraic properties or writing property tests.", + "desc_toks": [ + "algebraic", + "and", + "based", + "functor", + "including", + "laws", + "load", + "monad", + "properties", + "property", + "strategies", + "testing", + "tests", + "verifying", + "when", + "writing" + ], + "decidability_guess": "subjective" + }, + "preferences-computational-system-taxonomy": { + "plugin": "preferences-functional-programming-theory", + "path": "preferences-functional-programming-theory/.apm/skills/preferences-computational-system-taxonomy", + "symlink": null, + "lines": 226, + "cmd_style": false, + "env_coupling": [], + "has_scripts": false, + "has_refs": false, + "description": "Computational system taxonomy for classifying systems by their computational properties. Load when analyzing or categorizing system architectures.", + "desc_toks": [ + "analyzing", + "architectures", + "categorizing", + "classifying", + "computational", + "for", + "load", + "properties", + "system", + "systems", + "taxonomy", + "their", + "when" + ], + "decidability_guess": "subjective" + }, + "preferences-functional-reactive-programming": { + "plugin": "preferences-functional-programming-theory", + "path": "preferences-functional-programming-theory/.apm/skills/preferences-functional-reactive-programming", + "symlink": null, + "lines": 729, + "cmd_style": false, + "env_coupling": [], + "has_scripts": false, + "has_refs": false, + "description": "Functional reactive programming foundations including FRP semantics, arrows, and presheaf models. Load when working with reactive streams or FRP abstractions.", + "desc_toks": [ + "abstractions", + "and", + "arrows", + "foundations", + "frp", + "functional", + "including", + "load", + "models", + "presheaf", + "programming", + "reactive", + "semantics", + "streams", + "when", + "with", + "working" + ], + "decidability_guess": "subjective" + }, + "preferences-railway-oriented-programming": { + "plugin": "preferences-functional-programming-theory", + "path": "preferences-functional-programming-theory/.apm/skills/preferences-railway-oriented-programming", + "symlink": null, + "lines": 1406, + "cmd_style": false, + "env_coupling": [ + "~/.claude/skills/preferences-algebraic-data-types/SKILL.md", + "~/.claude/skills/preferences-data-modeling/SKILL.md", + "~/.claude/skills/preferences-schema-versioning/SKILL.md" + ], + "has_scripts": false, + "has_refs": false, + "description": "Railway-oriented programming with Result types and workflow composition for error handling. Load when designing error handling pipelines or composing fallible operations.", + "desc_toks": [ + "and", + "composing", + "composition", + "designing", + "error", + "fallible", + "for", + "handling", + "load", + "operations", + "oriented", + "pipelines", + "programming", + "railway", + "result", + "types", + "when", + "with", + "workflow" + ], + "decidability_guess": "subjective" + }, + "preferences-theoretical-foundations": { + "plugin": "preferences-functional-programming-theory", + "path": "preferences-functional-programming-theory/.apm/skills/preferences-theoretical-foundations", + "symlink": null, + "lines": 112, + "cmd_style": false, + "env_coupling": [], + "has_scripts": false, + "has_refs": true, + "description": "Category-theory and type-theory foundations for compositional, correct-by-construction software architecture. Load when designing, refactoring, reviewing, or maintaining an architecture; when reasoning about abstractions, functors, optics, or type-level constructs; when choosing or justifying an effect system, or weighing a monad-transformer stack against a capability interface discharged by handlers; when drawing event-sourcing or CQRS boundaries, or making a read model a verifiable projection; when modeling a domain's effects and coeffects; when making a component's decide/evolve (Decider) structure explicit; and when retaining a type-checkable Lean architecture spec beside a non-Rust (for example Python) implementation. Pairs with refinement-driven-development, which owns the verified Lean-to-Rust round trip.", + "desc_toks": [ + "about", + "abstractions", + "against", + "and", + "architecture", + "beside", + "boundaries", + "capability", + "category", + "checkable", + "choosing", + "coeffects", + "component", + "compositional", + "construction", + "constructs", + "correct", + "cqrs", + "decide", + "decider", + "designing", + "development", + "discharged", + "domain", + "drawing", + "driven", + "effect", + "effects", + "event", + "evolve", + "example", + "explicit", + "for", + "foundations", + "functors", + "handlers", + "implementation", + "interface", + "justifying", + "lean", + "level", + "load", + "maintaining", + "making", + "model", + "modeling", + "monad", + "non", + "optics", + "owns", + "pairs", + "projection", + "python", + "read", + "reasoning", + "refactoring", + "refinement", + "retaining", + "reviewing", + "round", + "rust", + "software", + "sourcing", + "spec", + "stack", + "structure", + "system", + "the", + "theory", + "transformer", + "trip", + "type", + "verifiable", + "verified", + "weighing", + "when", + "which", + "with" + ], + "decidability_guess": "subjective" + }, + "preferences-nix-checks-architecture": { + "plugin": "preferences-nix-and-secrets", + "path": "preferences-nix-and-secrets/.apm/skills/preferences-nix-checks-architecture", + "symlink": null, + "lines": 350, + "cmd_style": false, + "env_coupling": [ + "/home/alice/project", + "/home/bob/project" + ], + "has_scripts": false, + "has_refs": true, + "description": "Design hermetic, cacheable nix flake checks for comprehensive repository validation. Use when structuring checks in flake-parts modules, wrapping language tools as pure derivations, designing nix-unit structural invariants, or planning NixOS VM integration tests. Also load when the 8-category check taxonomy (format, secrets-scan, lint, type-check, unit-test, integration/e2e, nix-infrastructure, build/eval) is relevant.", + "desc_toks": [ + "also", + "build", + "cacheable", + "category", + "check", + "checks", + "comprehensive", + "derivations", + "design", + "designing", + "eval", + "flake", + "for", + "format", + "hermetic", + "infrastructure", + "integration", + "invariants", + "language", + "lint", + "load", + "modules", + "nix", + "nixos", + "parts", + "planning", + "pure", + "relevant", + "repository", + "scan", + "secrets", + "structural", + "structuring", + "taxonomy", + "test", + "tests", + "the", + "tools", + "type", + "unit", + "use", + "validation", + "when", + "wrapping" + ], + "decidability_guess": "decidable" + }, + "preferences-nix-ci-cd-integration": { + "plugin": "preferences-nix-and-secrets", + "path": "preferences-nix-and-secrets/.apm/skills/preferences-nix-ci-cd-integration", + "symlink": null, + "lines": 410, + "cmd_style": false, + "env_coupling": [ + "/home/tools/commands/_dev-tools.nix" + ], + "has_scripts": false, + "has_refs": true, + "description": "Integrate nix flake checks into CI/CD pipelines via nix-eval-jobs, nix-fast-build, and buildbot-nix. Use when designing CI pipelines for nix-based repositories, migrating from hand-crafted CI matrices to nix-native check fanout, configuring buildbot-nix project registration, or planning binary cache strategy. Also load when the two-phase migration pattern (GHA+nix-fast-build to buildbot-nix delegation) or the effect execution strategy decision (platform-native vs nix-native effects) is relevant.", + "desc_toks": [ + "also", + "and", + "based", + "binary", + "build", + "buildbot", + "cache", + "check", + "checks", + "configuring", + "crafted", + "decision", + "delegation", + "designing", + "effect", + "effects", + "eval", + "execution", + "fanout", + "fast", + "flake", + "for", + "from", + "gha", + "hand", + "integrate", + "into", + "jobs", + "load", + "matrices", + "migrating", + "migration", + "native", + "nix", + "pattern", + "phase", + "pipelines", + "planning", + "platform", + "project", + "registration", + "relevant", + "repositories", + "strategy", + "the", + "two", + "use", + "via", + "when" + ], + "decidability_guess": "decidable" + }, + "preferences-nix-development": { + "plugin": "preferences-nix-and-secrets", + "path": "preferences-nix-and-secrets/.apm/skills/preferences-nix-development", + "symlink": null, + "lines": 143, + "cmd_style": false, + "env_coupling": [ + "/home/all/", + "~/projects/nix-workspace/crane-maturin", + "~/projects/nix-workspace/pyproject.nix", + "~/projects/nix-workspace/uv2nix" + ], + "has_scripts": false, + "has_refs": true, + "description": "Nix development conventions for flakes, derivations, modules, and code style. Use when authoring flake.nix files, writing derivations or builders, designing NixOS/nix-darwin/home-manager modules, or following nix formatting and naming conventions. For check architecture and CI integration, see preferences-nix-checks-architecture and preferences-nix-ci-cd-integration.", + "desc_toks": [ + "and", + "architecture", + "authoring", + "builders", + "check", + "checks", + "code", + "conventions", + "darwin", + "derivations", + "designing", + "development", + "files", + "flake", + "flakes", + "following", + "for", + "formatting", + "home", + "integration", + "manager", + "modules", + "naming", + "nix", + "nixos", + "preferences", + "see", + "style", + "use", + "when", + "writing" + ], + "decidability_guess": "decidable" + }, + "preferences-secrets": { + "plugin": "preferences-nix-and-secrets", + "path": "preferences-nix-and-secrets/.apm/skills/preferences-secrets", + "symlink": null, + "lines": 924, + "cmd_style": false, + "env_coupling": [ + "~/.claude/skills/preferences-git-version-control/SKILL.md", + "~/.claude/skills/preferences-nix-development/SKILL.md", + "~/.claude/skills/preferences-typescript-nodejs-development/SKILL.md", + "~/.claude/skills/preferences-web-application-deployment/SKILL.md", + "~/.config/sops/age" + ], + "has_scripts": false, + "has_refs": false, + "description": "Secrets management practices for handling API keys, credentials, and sensitive configuration. Load when working with secrets, sops-nix, or credential storage.", + "desc_toks": [ + "and", + "api", + "configuration", + "credential", + "credentials", + "for", + "handling", + "keys", + "load", + "management", + "nix", + "practices", + "secrets", + "sensitive", + "sops", + "storage", + "when", + "with", + "working" + ], + "decidability_guess": "subjective" + }, + "preferences-adaptive-planning": { + "plugin": "preferences-operations-and-reliability", + "path": "preferences-operations-and-reliability/.apm/skills/preferences-adaptive-planning", + "symlink": null, + "lines": 324, + "cmd_style": false, + "env_coupling": [ + "~/.claude/skills/jj-version-control/SKILL.md", + "~/.claude/skills/jj-version-control/diamond-workflow.md" + ], + "has_scripts": false, + "has_refs": false, + "description": "Adaptive planning and control theory foundations for AI agent engineering, including MPC receding-horizon planning, Viable System Model mapping, queue economics, Cynefin domain classification, and stigmergic coordination. Load when reasoning about planning depth optimization, buffer sizing, validation gate placement, replanning triggers, or understanding why session workflow skills work the way they do.", + "desc_toks": [ + "about", + "adaptive", + "agent", + "and", + "buffer", + "classification", + "control", + "coordination", + "cynefin", + "depth", + "domain", + "economics", + "engineering", + "for", + "foundations", + "gate", + "horizon", + "including", + "load", + "mapping", + "model", + "mpc", + "optimization", + "placement", + "planning", + "queue", + "reasoning", + "receding", + "replanning", + "session", + "sizing", + "skills", + "stigmergic", + "system", + "the", + "theory", + "they", + "triggers", + "understanding", + "validation", + "viable", + "way", + "when", + "why", + "work", + "workflow" + ], + "decidability_guess": "subjective" + }, + "preferences-compositional-continuous-verification": { + "plugin": "preferences-operations-and-reliability", + "path": "preferences-operations-and-reliability/.apm/skills/preferences-compositional-continuous-verification", + "symlink": null, + "lines": 152, + "cmd_style": false, + "env_coupling": [ + "~/projects/sciexp/planning/docs/notes/development/continuous-verification/compositional-continuous-verification.md" + ], + "has_scripts": false, + "has_refs": false, + "description": "Compositional Continuous Verification (CCV) \u2014 theoretical anchor for system-level approximate correctness via operating-envelope-plus-regulator pairs that compose into a single closure operator. Distinct from narrow Continuous Validation (Rosenthal\u2013Jones runtime sense). Load when designing flake checks, reasoning about coverage and traceability, proposing new regulators, auditing whether a system's verification apparatus closes, or framing the agent-side enumerate-and-audit habit on `.#checks`.", + "desc_toks": [ + "about", + "agent", + "anchor", + "and", + "apparatus", + "approximate", + "audit", + "auditing", + "ccv", + "checks", + "closes", + "closure", + "compose", + "compositional", + "continuous", + "correctness", + "coverage", + "designing", + "distinct", + "enumerate", + "envelope", + "flake", + "for", + "framing", + "from", + "habit", + "into", + "jones", + "level", + "load", + "narrow", + "new", + "operating", + "operator", + "pairs", + "plus", + "proposing", + "reasoning", + "regulator", + "regulators", + "rosenthal", + "runtime", + "sense", + "side", + "single", + "system", + "that", + "the", + "theoretical", + "traceability", + "validation", + "verification", + "via", + "when", + "whether" + ], + "decidability_guess": "decidable" + }, + "preferences-distributed-systems": { + "plugin": "preferences-operations-and-reliability", + "path": "preferences-operations-and-reliability/.apm/skills/preferences-distributed-systems", + "symlink": null, + "lines": 1322, + "cmd_style": false, + "env_coupling": [], + "has_scripts": false, + "has_refs": false, + "description": "Distributed systems patterns including consistency models, consensus, and fault tolerance. Load when designing or debugging distributed architectures.", + "desc_toks": [ + "and", + "architectures", + "consensus", + "consistency", + "debugging", + "designing", + "distributed", + "fault", + "including", + "load", + "models", + "patterns", + "systems", + "tolerance", + "when" + ], + "decidability_guess": "subjective" + }, + "preferences-observability-engineering": { + "plugin": "preferences-operations-and-reliability", + "path": "preferences-operations-and-reliability/.apm/skills/preferences-observability-engineering", + "symlink": null, + "lines": 406, + "cmd_style": false, + "env_coupling": [], + "has_scripts": false, + "has_refs": false, + "description": "Observability engineering foundations for understanding system behavior through structured events, distributed traces, and service-level objectives. Covers the observability paradigm (structured events, high cardinality, high dimensionality, explorability), SLO-driven reliability, instrumentation principles, telemetry architecture, tool category taxonomy, anti-patterns, and maturity model. Load when designing instrumentation, adding telemetry, working with OpenTelemetry, structured events, tracing, metrics, analyzing production behavior, SLO definition, or alerting strategy.", + "desc_toks": [ + "adding", + "alerting", + "analyzing", + "and", + "anti", + "architecture", + "behavior", + "cardinality", + "category", + "covers", + "definition", + "designing", + "dimensionality", + "distributed", + "driven", + "engineering", + "events", + "explorability", + "for", + "foundations", + "high", + "instrumentation", + "level", + "load", + "maturity", + "metrics", + "model", + "objectives", + "observability", + "opentelemetry", + "paradigm", + "patterns", + "principles", + "production", + "reliability", + "service", + "slo", + "strategy", + "structured", + "system", + "taxonomy", + "telemetry", + "the", + "through", + "tool", + "traces", + "tracing", + "understanding", + "when", + "with", + "working" + ], + "decidability_guess": "subjective" + }, + "preferences-production-readiness": { + "plugin": "preferences-operations-and-reliability", + "path": "preferences-operations-and-reliability/.apm/skills/preferences-production-readiness", + "symlink": null, + "lines": 416, + "cmd_style": false, + "env_coupling": [], + "has_scripts": false, + "has_refs": false, + "description": "Production readiness and observability-driven development practices for deploying, operating, and learning from production systems. Covers production readiness checklists, observability-driven development (ODD), progressive delivery, CI/CD pipeline observability, incident learning closure, and the feedback topology from alerting through SLO revision. Load when deploying to production, defining health checks, designing progressive delivery, conducting post-incident review, instrumenting CI/CD pipelines, or defining production readiness criteria.", + "desc_toks": [ + "alerting", + "and", + "checklists", + "checks", + "closure", + "conducting", + "covers", + "criteria", + "defining", + "delivery", + "deploying", + "designing", + "development", + "driven", + "feedback", + "for", + "from", + "health", + "incident", + "instrumenting", + "learning", + "load", + "observability", + "odd", + "operating", + "pipeline", + "pipelines", + "post", + "practices", + "production", + "progressive", + "readiness", + "review", + "revision", + "slo", + "systems", + "the", + "through", + "topology", + "when" + ], + "decidability_guess": "subjective" + }, + "preferences-validation-assurance": { + "plugin": "preferences-operations-and-reliability", + "path": "preferences-operations-and-reliability/.apm/skills/preferences-validation-assurance", + "symlink": null, + "lines": 292, + "cmd_style": false, + "env_coupling": [], + "has_scripts": false, + "has_refs": false, + "description": "Validation assurance foundations for evidence-based confidence in software systems, including Mayo's severity criterion for engineering, refinement as spec-implementation freedom preservation, mutation analysis, observability, regression harness design, reflexive severity, and double-loop learning triggers. Load when reasoning about test adequacy, confidence levels, evidence quality, regression protection, or whether the process itself should change.", + "desc_toks": [ + "about", + "adequacy", + "analysis", + "and", + "assurance", + "based", + "change", + "confidence", + "criterion", + "design", + "double", + "engineering", + "evidence", + "for", + "foundations", + "freedom", + "harness", + "implementation", + "including", + "itself", + "learning", + "levels", + "load", + "loop", + "mayo", + "mutation", + "observability", + "preservation", + "process", + "protection", + "quality", + "reasoning", + "refinement", + "reflexive", + "regression", + "severity", + "should", + "software", + "spec", + "systems", + "test", + "the", + "triggers", + "validation", + "when", + "whether" + ], + "decidability_guess": "decidable" + }, + "preferences-haskell-development": { + "plugin": "preferences-programming-languages", + "path": "preferences-programming-languages/.apm/skills/preferences-haskell-development", + "symlink": null, + "lines": 70, + "cmd_style": false, + "env_coupling": [], + "has_scripts": false, + "has_refs": false, + "description": "Haskell development conventions covering type classes, monadic composition, and GHC extensions. Load when working with .hs files or Haskell projects.", + "desc_toks": [ + "and", + "classes", + "composition", + "conventions", + "covering", + "development", + "extensions", + "files", + "ghc", + "haskell", + "load", + "monadic", + "projects", + "type", + "when", + "with", + "working" + ], + "decidability_guess": "subjective" + }, + "preferences-python-development": { + "plugin": "preferences-programming-languages", + "path": "preferences-programming-languages/.apm/skills/preferences-python-development", + "symlink": null, + "lines": 1067, + "cmd_style": false, + "env_coupling": [ + "~/.claude/skills/preferences-algebraic-data-types/SKILL.md", + "~/.claude/skills/preferences-architectural-patterns/SKILL.md", + "~/.claude/skills/preferences-railway-oriented-programming/SKILL.md" + ], + "has_scripts": false, + "has_refs": false, + "description": "Python development conventions including type safety with basedpyright, beartype, and Expression library patterns. Load when working with .py files or Python projects.", + "desc_toks": [ + "and", + "basedpyright", + "beartype", + "conventions", + "development", + "expression", + "files", + "including", + "library", + "load", + "patterns", + "projects", + "python", + "safety", + "type", + "when", + "with", + "working" + ], + "decidability_guess": "subjective" + }, + "preferences-rust-development": { + "plugin": "preferences-programming-languages", + "path": "preferences-programming-languages/.apm/skills/preferences-rust-development", + "symlink": null, + "lines": 61, + "cmd_style": false, + "env_coupling": [ + "~/.claude/skills/preferences-algebraic-data-types/SKILL.md", + "~/.claude/skills/preferences-architectural-patterns/SKILL.md", + "~/.claude/skills/preferences-distributed-systems/SKILL.md", + "~/.claude/skills/preferences-domain-modeling/SKILL.md", + "~/.claude/skills/preferences-railway-oriented-programming/SKILL.md" + ], + "has_scripts": false, + "has_refs": false, + "description": "Rust development conventions covering domain modeling, error handling, API design, testing, performance, and type-level programming. Load when working with .rs files or Rust projects.", + "desc_toks": [ + "and", + "api", + "conventions", + "covering", + "design", + "development", + "domain", + "error", + "files", + "handling", + "level", + "load", + "modeling", + "performance", + "programming", + "projects", + "rust", + "testing", + "type", + "when", + "with", + "working" + ], + "decidability_guess": "subjective" + }, + "preferences-typescript-nodejs-development": { + "plugin": "preferences-programming-languages", + "path": "preferences-programming-languages/.apm/skills/preferences-typescript-nodejs-development", + "symlink": null, + "lines": 888, + "cmd_style": false, + "env_coupling": [], + "has_scripts": false, + "has_refs": false, + "description": "TypeScript and Node.js development conventions including strict typing, Effect-TS patterns, and build configuration. Load when working with .ts files or Node.js projects.", + "desc_toks": [ + "and", + "build", + "configuration", + "conventions", + "development", + "effect", + "files", + "including", + "load", + "node", + "patterns", + "projects", + "strict", + "typescript", + "typing", + "when", + "with", + "working" + ], + "decidability_guess": "decidable" + }, + "preferences-cloudflare-wrangler-reference": { + "plugin": "preferences-web-platform-and-deployment", + "path": "preferences-web-platform-and-deployment/.apm/skills/preferences-cloudflare-wrangler-reference", + "symlink": null, + "lines": 951, + "cmd_style": false, + "env_coupling": [ + "~/.claude/skills/preferences-web-application-deployment/SKILL.md" + ], + "has_scripts": false, + "has_refs": false, + "description": "Cloudflare wrangler comprehensive reference for Workers, D1, R2, and KV configuration. Load when working with Cloudflare deployment or wrangler.toml.", + "desc_toks": [ + "and", + "cloudflare", + "comprehensive", + "configuration", + "deployment", + "for", + "load", + "reference", + "toml", + "when", + "with", + "workers", + "working", + "wrangler" + ], + "decidability_guess": "subjective" + }, + "preferences-hypermedia-development": { + "plugin": "preferences-web-platform-and-deployment", + "path": "preferences-web-platform-and-deployment/.apm/skills/preferences-hypermedia-development", + "symlink": null, + "lines": 92, + "cmd_style": false, + "env_coupling": [ + "~/.claude/skills/preferences-architectural-patterns/SKILL.md", + "~/.claude/skills/preferences-data-modeling/SKILL.md", + "~/.claude/skills/preferences-distributed-systems/SKILL.md", + "~/.claude/skills/preferences-domain-modeling/SKILL.md", + "~/.claude/skills/preferences-hypermedia-documents/SKILL.md" + ], + "has_scripts": false, + "has_refs": false, + "description": "Hypermedia-driven UI development patterns including architecture, SSE streaming, Datastar framework, CSS architecture, web components, templating, and event architecture. Load when building server-driven UIs or hypermedia applications.", + "desc_toks": [ + "and", + "applications", + "architecture", + "building", + "components", + "css", + "datastar", + "development", + "driven", + "event", + "framework", + "hypermedia", + "including", + "load", + "patterns", + "server", + "sse", + "streaming", + "templating", + "uis", + "web", + "when" + ], + "decidability_guess": "subjective" + }, + "preferences-hypermedia-documents": { + "plugin": "preferences-web-platform-and-deployment", + "path": "preferences-web-platform-and-deployment/.apm/skills/preferences-hypermedia-documents", + "symlink": null, + "lines": 183, + "cmd_style": false, + "env_coupling": [ + "~/.claude/skills/preferences-architectural-patterns/SKILL.md", + "~/.claude/skills/preferences-hypermedia-development/04-css-architecture.md", + "~/.claude/skills/preferences-hypermedia-development/05-web-components.md", + "~/.claude/skills/preferences-hypermedia-development/SKILL.md", + "~/.claude/skills/preferences-web-platform-foundations/SKILL.md" + ], + "has_scripts": false, + "has_refs": false, + "description": "Standalone hypermedia document authoring with Open Props, CUBE CSS, progressive enhancement, inline SVG, MathML, scroll-snap presentations, and Lit web component islands for imperative contexts. Load when creating HTML documents, slide presentations, interactive experiments, data visualizations, or SVG diagrams without a server backend.", + "desc_toks": [ + "and", + "authoring", + "backend", + "component", + "contexts", + "creating", + "css", + "cube", + "data", + "diagrams", + "document", + "documents", + "enhancement", + "experiments", + "for", + "html", + "hypermedia", + "imperative", + "inline", + "interactive", + "islands", + "lit", + "load", + "mathml", + "open", + "presentations", + "progressive", + "props", + "scroll", + "server", + "slide", + "snap", + "standalone", + "svg", + "visualizations", + "web", + "when", + "with", + "without" + ], + "decidability_guess": "subjective" + }, + "preferences-react-tanstack-ui-development": { + "plugin": "preferences-web-platform-and-deployment", + "path": "preferences-web-platform-and-deployment/.apm/skills/preferences-react-tanstack-ui-development", + "symlink": null, + "lines": 1295, + "cmd_style": false, + "env_coupling": [ + "~/.claude/skills/preferences-algebraic-data-types/SKILL.md", + "~/.claude/skills/preferences-architectural-patterns/SKILL.md", + "~/.claude/skills/preferences-railway-oriented-programming/SKILL.md", + "~/.claude/skills/preferences-typescript-nodejs-development/SKILL.md", + "~/.claude/skills/preferences-web-application-deployment/SKILL.md" + ], + "has_scripts": false, + "has_refs": false, + "description": "React and TanStack UI development patterns including component design, routing, and state management. Load when working with React components or TanStack libraries.", + "desc_toks": [ + "and", + "component", + "components", + "design", + "development", + "including", + "libraries", + "load", + "management", + "patterns", + "react", + "routing", + "state", + "tanstack", + "when", + "with", + "working" + ], + "decidability_guess": "subjective" + }, + "preferences-web-application-deployment": { + "plugin": "preferences-web-platform-and-deployment", + "path": "preferences-web-platform-and-deployment/.apm/skills/preferences-web-application-deployment", + "symlink": null, + "lines": 1160, + "cmd_style": false, + "env_coupling": [ + "~/.claude/skills/preferences-cloudflare-wrangler-reference/SKILL.md", + "~/.claude/skills/preferences-railway-oriented-programming/SKILL.md", + "~/.claude/skills/preferences-react-tanstack-ui-development/SKILL.md", + "~/.claude/skills/preferences-schema-versioning/SKILL.md", + "~/.claude/skills/preferences-typescript-nodejs-development/SKILL.md" + ], + "has_scripts": false, + "has_refs": false, + "description": "Web application deployment patterns for production hosting and CI/CD. Load when deploying web applications or configuring deployment pipelines.", + "desc_toks": [ + "and", + "application", + "applications", + "configuring", + "deploying", + "deployment", + "for", + "hosting", + "load", + "patterns", + "pipelines", + "production", + "web", + "when" + ], + "decidability_guess": "subjective" + }, + "preferences-web-platform-foundations": { + "plugin": "preferences-web-platform-and-deployment", + "path": "preferences-web-platform-and-deployment/.apm/skills/preferences-web-platform-foundations", + "symlink": null, + "lines": 258, + "cmd_style": false, + "env_coupling": [ + "~/.claude/skills/preferences-architectural-patterns/SKILL.md", + "~/.claude/skills/preferences-functional-reactive-programming/SKILL.md", + "~/.claude/skills/preferences-hypermedia-development/SKILL.md", + "~/.claude/skills/preferences-hypermedia-documents/SKILL.md", + "~/.claude/skills/preferences-react-tanstack-ui-development/SKILL.md" + ], + "has_scripts": false, + "has_refs": false, + "description": "Web platform properties, capability ladder from HTML through CSS to JavaScript, and paradigm routing between hypermedia and SPA approaches. Load when making web architecture decisions, evaluating framework choices, or reasoning about progressive enhancement.", + "desc_toks": [ + "about", + "and", + "approaches", + "architecture", + "between", + "capability", + "choices", + "css", + "decisions", + "enhancement", + "evaluating", + "framework", + "from", + "html", + "hypermedia", + "javascript", + "ladder", + "load", + "making", + "paradigm", + "platform", + "progressive", + "properties", + "reasoning", + "routing", + "spa", + "through", + "web", + "when" + ], + "decidability_guess": "subjective" + }, + "atdd-outer-loop": { + "plugin": "testing-and-quality", + "path": "testing-and-quality/.apm/skills/atdd-outer-loop", + "symlink": null, + "lines": 142, + "cmd_style": false, + "env_coupling": [], + "has_scripts": false, + "has_refs": true, + "description": "Orchestrate the ATDD outer loop that wraps inner TDD. Load when starting acceptance-driven work, choosing ATDD versus plain TDD, routing a proposition to BDD versus a property/law test versus a regression/smoke test versus a formal proof, or enforcing RED-before-GREEN through a real runner. Defers formulation, step binding, discovery, inner TDD, and severity to their owners.", + "desc_toks": [ + "acceptance", + "and", + "atdd", + "bdd", + "before", + "binding", + "choosing", + "defers", + "discovery", + "driven", + "enforcing", + "formal", + "formulation", + "green", + "inner", + "law", + "load", + "loop", + "orchestrate", + "outer", + "owners", + "plain", + "proof", + "property", + "proposition", + "real", + "red", + "regression", + "routing", + "runner", + "severity", + "smoke", + "starting", + "step", + "tdd", + "test", + "that", + "the", + "their", + "through", + "versus", + "when", + "work", + "wraps" + ], + "decidability_guess": "decidable" + }, + "bdd-discovery": { + "plugin": "testing-and-quality", + "path": "testing-and-quality/.apm/skills/bdd-discovery", + "symlink": null, + "lines": 66, + "cmd_style": false, + "env_coupling": [], + "has_scripts": false, + "has_refs": false, + "description": "Use before writing Gherkin when exploring a story's behaviour, rules, examples, questions, scope, or readiness with stakeholders", + "desc_toks": [ + "before", + "behaviour", + "examples", + "exploring", + "gherkin", + "questions", + "readiness", + "rules", + "scope", + "stakeholders", + "story", + "use", + "when", + "with", + "writing" + ], + "decidability_guess": "subjective" + }, + "bdd-formulation": { + "plugin": "testing-and-quality", + "path": "testing-and-quality/.apm/skills/bdd-formulation", + "symlink": null, + "lines": 53, + "cmd_style": false, + "env_coupling": [], + "has_scripts": false, + "has_refs": false, + "description": "Use when writing or reviewing Gherkin scenarios, especially after discovering examples or edge cases that reveal a business rule", + "desc_toks": [ + "after", + "business", + "cases", + "discovering", + "edge", + "especially", + "examples", + "gherkin", + "reveal", + "reviewing", + "rule", + "scenarios", + "that", + "use", + "when", + "writing" + ], + "decidability_guess": "subjective" + }, + "bdd-gherkin-formulation": { + "plugin": "testing-and-quality", + "path": "testing-and-quality/.apm/skills/bdd-gherkin-formulation", + "symlink": null, + "lines": 154, + "cmd_style": false, + "env_coupling": [], + "has_scripts": false, + "has_refs": true, + "description": "Use when writing, refining, or reviewing Gherkin as living documentation. Applies BRIEF, groups scenarios under Rules-vs-Examples, keeps steps declarative over imperative via the implementation-change litmus, and enforces the observable-outcome discipline \u2014 assert observable behavior against an independent literal oracle, never re-derive the expected value through the production path, never inspect private state behind a protocol. This is the formulation craft only: not discovery (see preferences-collaborative-modeling), not binding steps to a runner (see bdd-step-definitions), not orchestration or the when-BDD gate (see atdd-outer-loop).", + "desc_toks": [ + "against", + "and", + "applies", + "assert", + "atdd", + "bdd", + "behavior", + "behind", + "binding", + "brief", + "change", + "collaborative", + "craft", + "declarative", + "definitions", + "derive", + "discipline", + "discovery", + "documentation", + "enforces", + "examples", + "expected", + "formulation", + "gate", + "gherkin", + "groups", + "imperative", + "implementation", + "independent", + "inspect", + "keeps", + "literal", + "litmus", + "living", + "loop", + "modeling", + "never", + "not", + "observable", + "only", + "oracle", + "orchestration", + "outcome", + "outer", + "over", + "path", + "preferences", + "private", + "production", + "protocol", + "refining", + "reviewing", + "rules", + "runner", + "scenarios", + "see", + "state", + "step", + "steps", + "the", + "this", + "through", + "under", + "use", + "value", + "via", + "when", + "writing" + ], + "decidability_guess": "subjective" + }, + "bdd-step-definitions": { + "plugin": "testing-and-quality", + "path": "testing-and-quality/.apm/skills/bdd-step-definitions", + "symlink": null, + "lines": 92, + "cmd_style": false, + "env_coupling": [], + "has_scripts": false, + "has_refs": true, + "description": "Bind formulated Gherkin to native step definitions across pytest-bdd, cucumber-rs, and cucumber-js. Load when writing or reviewing step glue, choosing a matcher (Cucumber Expression vs regex vs typed parser), structuring a per-scenario World or context, wiring feature discovery and the run command, or diagnosing a tautologically-green or private-state-piercing Then. Runner layer of the BDD/ATDD group; points up to the concept layer, never down.", + "desc_toks": [ + "across", + "and", + "atdd", + "bdd", + "bind", + "choosing", + "command", + "concept", + "context", + "cucumber", + "definitions", + "diagnosing", + "discovery", + "down", + "expression", + "feature", + "formulated", + "gherkin", + "glue", + "green", + "group", + "layer", + "load", + "matcher", + "native", + "never", + "parser", + "per", + "piercing", + "points", + "private", + "pytest", + "regex", + "reviewing", + "run", + "runner", + "scenario", + "state", + "step", + "structuring", + "tautologically", + "the", + "then", + "typed", + "when", + "wiring", + "world", + "writing" + ], + "decidability_guess": "subjective" + }, + "executable-specification-testing": { + "plugin": "testing-and-quality", + "path": "testing-and-quality/.apm/skills/executable-specification-testing", + "symlink": null, + "lines": 151, + "cmd_style": false, + "env_coupling": [], + "has_scripts": false, + "has_refs": true, + "description": "Author one executable predicate and consume it three ways without redeclaration \u2014 runtime contract enforcement, property-test strategy inference, and SMT/concolic checking \u2014 and own the property-based-testing craft (generator and strategy design, shrinking mechanics, stateful and metamorphic testing) alongside design-by-contract as a named modality and the SMT/concolic rung. Load when wiring icontract, icontract-hypothesis, and CrossHair against a single contract; designing a generator, strategy, or shrinker; writing a stateful/model-based or metamorphic property test; placing a technique on the example-to-proof rigor ladder; or choosing between samplers and solvers. Routes the algebraic law catalogue and parametricity to preferences-algebraic-laws, the Lean-to-Rust round trip and Charon/Aeneas to refinement-driven-development, and severity, confidence, and the validation ladder to preferences-validation-assurance. Points up to atdd-outer-loop, whose Gate 1 loads this skill.", + "desc_toks": [ + "aeneas", + "against", + "algebraic", + "alongside", + "and", + "assurance", + "atdd", + "author", + "based", + "between", + "catalogue", + "charon", + "checking", + "choosing", + "concolic", + "confidence", + "consume", + "contract", + "craft", + "crosshair", + "design", + "designing", + "development", + "driven", + "enforcement", + "example", + "executable", + "gate", + "generator", + "hypothesis", + "icontract", + "inference", + "ladder", + "law", + "laws", + "lean", + "load", + "loads", + "loop", + "mechanics", + "metamorphic", + "modality", + "model", + "named", + "one", + "outer", + "own", + "parametricity", + "placing", + "points", + "predicate", + "preferences", + "proof", + "property", + "redeclaration", + "refinement", + "rigor", + "round", + "routes", + "rung", + "runtime", + "rust", + "samplers", + "severity", + "shrinker", + "shrinking", + "single", + "skill", + "smt", + "solvers", + "stateful", + "strategy", + "technique", + "test", + "testing", + "the", + "this", + "three", + "trip", + "validation", + "ways", + "when", + "whose", + "wiring", + "without", + "writing" + ], + "decidability_guess": "decidable" + }, + "harborize": { + "plugin": "testing-and-quality", + "path": "testing-and-quality/.apm/skills/harborize", + "symlink": null, + "lines": 319, + "cmd_style": false, + "env_coupling": [ + "~/.claude/skills", + "~/ghq/github.com/" + ], + "has_scripts": true, + "has_refs": true, + "description": "Compile an existing agent skill, a plugin of skills, or several co-present plugins into a complete, runnable evaluation package \u2014 Harbor task + SkillsBench/BenchFlow task package, verifier, oracle, condition lattice, run manifest, and analysis scripts. Use whenever the user wants to evaluate, benchmark, verify, stress-test, or measure the quality, performance, efficacy, or interaction effects of one or more SKILL.md-based skills or plugins across models and agent harnesses, or asks to \"harborize\" a skill, turn skill-creator eval artifacts into Harbor/BenchFlow tasks, or design paired with-skill/no-skill experiments. Do not use for authoring or improving the skill that is the subject of the evaluation (use skill-creator) or for authoring generic Harbor tasks unrelated to skills (use Harbor's create-task). Revising harborize itself between evaluation rounds is in scope and is what this skill's own versioning exists for.", + "desc_toks": [ + "across", + "agent", + "analysis", + "and", + "artifacts", + "asks", + "authoring", + "based", + "benchflow", + "benchmark", + "between", + "compile", + "complete", + "condition", + "create", + "creator", + "design", + "effects", + "efficacy", + "eval", + "evaluate", + "evaluation", + "existing", + "exists", + "experiments", + "for", + "generic", + "harbor", + "harborize", + "harnesses", + "improving", + "interaction", + "into", + "itself", + "lattice", + "manifest", + "measure", + "models", + "more", + "not", + "one", + "oracle", + "own", + "package", + "paired", + "performance", + "plugin", + "plugins", + "present", + "quality", + "revising", + "rounds", + "run", + "runnable", + "scope", + "scripts", + "several", + "skill", + "skills", + "skillsbench", + "stress", + "subject", + "task", + "tasks", + "test", + "that", + "the", + "this", + "turn", + "unrelated", + "use", + "user", + "verifier", + "verify", + "versioning", + "wants", + "what", + "whenever", + "with" + ], + "decidability_guess": "decidable" + }, + "ubiquitous-language": { + "plugin": "testing-and-quality", + "path": "testing-and-quality/.apm/skills/ubiquitous-language", + "symlink": null, + "lines": 305, + "cmd_style": false, + "env_coupling": [], + "has_scripts": false, + "has_refs": false, + "description": "Use when evaluating the ubiquitous language in a codebase - produces a glossary of domain terms with references and commentary on inconsistencies, awkward names, or overlapping concepts", + "desc_toks": [ + "and", + "awkward", + "codebase", + "commentary", + "concepts", + "domain", + "evaluating", + "glossary", + "inconsistencies", + "language", + "names", + "overlapping", + "produces", + "references", + "terms", + "the", + "ubiquitous", + "use", + "when", + "with" + ], + "decidability_guess": "subjective" + }, + "dependency-source-acquisition": { + "plugin": "version-control-and-forge", + "path": "version-control-and-forge/.apm/skills/dependency-source-acquisition", + "symlink": null, + "lines": 135, + "cmd_style": false, + "env_coupling": [ + "~/ghq", + "~/projects/" + ], + "has_scripts": false, + "has_refs": false, + "description": "Acquire a correct local copy of a dependency's or research-reference repo's upstream source and review it whole, instead of reading it one file at a time through the GitHub API or web UI. Use when acquiring or reviewing the source of a Category-2 dependency or reference repo, downloading upstream source to study locally, resolving a package's source-repository URL from cargo/uv/bun/nix metadata, or working with ghq to clone and catalog upstream sources.", + "desc_toks": [ + "acquire", + "acquiring", + "and", + "api", + "bun", + "cargo", + "catalog", + "category", + "clone", + "copy", + "correct", + "dependency", + "downloading", + "file", + "from", + "ghq", + "github", + "instead", + "local", + "locally", + "metadata", + "nix", + "one", + "package", + "reading", + "reference", + "repo", + "repository", + "research", + "resolving", + "review", + "reviewing", + "source", + "sources", + "study", + "the", + "through", + "time", + "upstream", + "url", + "use", + "web", + "when", + "whole", + "with", + "working" + ], + "decidability_guess": "decidable" + }, + "git-commit-prompt": { + "plugin": "version-control-and-forge", + "path": "version-control-and-forge/.apm/skills/git-commit-prompt", + "symlink": null, + "lines": 12, + "cmd_style": true, + "env_coupling": [], + "has_scripts": false, + "has_refs": false, + "description": "Generate optimal prompt for atomic git commits with conventional format. Use when preparing commits or cleaning up git history.", + "desc_toks": [ + "atomic", + "cleaning", + "commits", + "conventional", + "for", + "format", + "generate", + "git", + "history", + "optimal", + "preparing", + "prompt", + "use", + "when", + "with" + ], + "decidability_guess": "subjective" + }, + "gitbutler-but-cli": { + "plugin": "version-control-and-forge", + "path": "version-control-and-forge/.apm/skills/gitbutler-but-cli", + "symlink": null, + "lines": 293, + "cmd_style": false, + "env_coupling": [ + "~/.claude/skills/jj-workflow/SKILL.md", + "~/.claude/skills/preferences-git-version-control/SKILL.md" + ], + "has_scripts": false, + "has_refs": true, + "description": "Commit, push, branch, and manage version control with GitButler. Use for: commit my changes, check what changed, create a PR, push my branch, view diff, create branches, stage files, edit commit history, squash commits, amend commits, undo commits, pull requests, merge, stash work. Replaces git - use 'but' instead of git commit, git status, git push, git checkout, git add, git diff, git branch, git rebase, git stash, git merge. Covers all git, version control, and source control operations.", + "desc_toks": [ + "add", + "all", + "amend", + "and", + "branch", + "branches", + "but", + "changed", + "changes", + "check", + "checkout", + "commit", + "commits", + "control", + "covers", + "create", + "diff", + "edit", + "files", + "for", + "git", + "gitbutler", + "history", + "instead", + "manage", + "merge", + "operations", + "pull", + "push", + "rebase", + "replaces", + "requests", + "source", + "squash", + "stage", + "stash", + "status", + "undo", + "use", + "version", + "view", + "what", + "with", + "work" + ], + "decidability_guess": "decidable" + }, + "github-browse": { + "plugin": "version-control-and-forge", + "path": "version-control-and-forge/.apm/skills/github-browse", + "symlink": null, + "lines": 46, + "cmd_style": true, + "env_coupling": [], + "has_scripts": false, + "has_refs": false, + "description": "Open a GitHub issue or PR in browser and optionally manage subscriptions. Use when navigating to GitHub resources from the terminal.", + "desc_toks": [ + "and", + "browser", + "from", + "github", + "issue", + "manage", + "navigating", + "open", + "optionally", + "resources", + "subscriptions", + "terminal", + "the", + "use", + "when" + ], + "decidability_guess": "subjective" + }, + "hunk-review": { + "plugin": "version-control-and-forge", + "path": "version-control-and-forge/.apm/skills/hunk-review", + "symlink": null, + "lines": 157, + "cmd_style": false, + "env_coupling": [], + "has_scripts": false, + "has_refs": false, + "description": "Interacts with live Hunk diff review sessions via CLI. Inspects review focus, navigates files and hunks, reloads session contents, and adds inline review comments. Use when the user has a Hunk session running or wants to review diffs interactively.", + "desc_toks": [ + "adds", + "and", + "cli", + "comments", + "contents", + "diff", + "diffs", + "files", + "focus", + "has", + "hunk", + "hunks", + "inline", + "inspects", + "interactively", + "interacts", + "live", + "navigates", + "reloads", + "review", + "running", + "session", + "sessions", + "the", + "use", + "user", + "via", + "wants", + "when", + "with" + ], + "decidability_guess": "subjective" + }, + "jj-git-interactive-rebase-to-jj": { + "plugin": "version-control-and-forge", + "path": "version-control-and-forge/.apm/skills/jj-git-interactive-rebase-to-jj", + "symlink": null, + "lines": 693, + "cmd_style": true, + "env_coupling": [ + "~/.claude/skills/jj-version-control/SKILL.md", + "~/.claude/skills/jj-version-control/diamond-workflow.md", + "~/.claude/skills/jj-workflow/SKILL.md" + ], + "has_scripts": false, + "has_refs": false, + "description": "Complete reference for translating git interactive rebase operations to Jujutsu equivalents.", + "desc_toks": [ + "complete", + "equivalents", + "for", + "git", + "interactive", + "jujutsu", + "operations", + "rebase", + "reference", + "translating" + ], + "decidability_guess": "subjective" + }, + "jj-history-cleanup": { + "plugin": "version-control-and-forge", + "path": "version-control-and-forge/.apm/skills/jj-history-cleanup", + "symlink": null, + "lines": 635, + "cmd_style": true, + "env_coupling": [ + "~/.claude/skills/jj-git-interactive-rebase-to-jj/SKILL.md", + "~/.claude/skills/jj-version-control/SKILL.md", + "~/.claude/skills/jj-workflow/SKILL.md" + ], + "has_scripts": false, + "has_refs": false, + "description": "Jujutsu history cleanup patterns for rewriting and reorganizing change history.", + "desc_toks": [ + "and", + "change", + "cleanup", + "for", + "history", + "jujutsu", + "patterns", + "reorganizing", + "rewriting" + ], + "decidability_guess": "subjective" + }, + "jj-summary": { + "plugin": "version-control-and-forge", + "path": "version-control-and-forge/.apm/skills/jj-summary", + "symlink": null, + "lines": 416, + "cmd_style": false, + "env_coupling": [ + "~/.claude/skills/jj-version-control/SKILL.md", + "~/.claude/skills/jj-version-control/diamond-workflow.md", + "~/.claude/skills/jj-workflow/SKILL.md", + "~/.claude/skills/nix-flake-pr-cycle/SKILL.md", + "~/.claude/skills/preferences-git-version-control/SKILL.md" + ], + "has_scripts": false, + "has_refs": false, + "description": "Jujutsu workflow summary with essential paradigm shifts and decision guide. Reference for quick jj orientation.", + "desc_toks": [ + "and", + "decision", + "essential", + "for", + "guide", + "jujutsu", + "orientation", + "paradigm", + "quick", + "reference", + "shifts", + "summary", + "with", + "workflow" + ], + "decidability_guess": "subjective" + }, + "jj-version-control": { + "plugin": "version-control-and-forge", + "path": "version-control-and-forge/.apm/skills/jj-version-control", + "symlink": null, + "lines": 1623, + "cmd_style": false, + "env_coupling": [ + "/home/tools/hooks/verify-diamond-before-edit.sh", + "~/.claude/skills/issues-beads-prime/SKILL.md", + "~/.claude/skills/jj-history-cleanup/SKILL.md", + "~/.claude/skills/jj-summary/SKILL.md", + "~/.claude/skills/jj-workflow/SKILL.md" + ], + "has_scripts": false, + "has_refs": false, + "description": "Jujutsu version control conventions and workflow patterns.", + "desc_toks": [ + "and", + "control", + "conventions", + "jujutsu", + "patterns", + "version", + "workflow" + ], + "decidability_guess": "subjective" + }, + "jj-workflow": { + "plugin": "version-control-and-forge", + "path": "version-control-and-forge/.apm/skills/jj-workflow", + "symlink": null, + "lines": 1415, + "cmd_style": false, + "env_coupling": [ + "~/.claude/skills/jj-version-control/SKILL.md", + "~/.claude/skills/jj-version-control/diamond-workflow.md", + "~/.claude/skills/jj-version-control/tiered-ceremony.md", + "~/.claude/skills/preferences-git-version-control/03-jj-mode.md", + "~/.jjconfig.toml" + ], + "has_scripts": false, + "has_refs": false, + "description": "Jujutsu atomic workflow with full operational reference for jj-based repositories.", + "desc_toks": [ + "atomic", + "based", + "for", + "full", + "jujutsu", + "operational", + "reference", + "repositories", + "with", + "workflow" + ], + "decidability_guess": "subjective" + } + }, + "overlap": { + "intra": [ + { + "jaccard": 0.44, + "plugin": "preferences-programming-languages", + "a": "preferences-python-development", + "b": "preferences-typescript-nodejs-development" + }, + { + "jaccard": 0.4, + "plugin": "agent-orchestration-and-meta-tooling", + "a": "meta-load-cc-docs", + "b": "meta-load-prompting-docs" + }, + { + "jaccard": 0.4, + "plugin": "preferences-programming-languages", + "a": "preferences-haskell-development", + "b": "preferences-python-development" + }, + { + "jaccard": 0.3929, + "plugin": "preferences-programming-languages", + "a": "preferences-haskell-development", + "b": "preferences-rust-development" + }, + { + "jaccard": 0.3846, + "plugin": "beads-issue-tracking-and-session-workflow", + "a": "issues-beads", + "b": "issues-beads-init" + }, + { + "jaccard": 0.375, + "plugin": "planning-and-development", + "a": "openspec-archive-change", + "b": "openspec-verify-change" + }, + { + "jaccard": 0.3704, + "plugin": "planning-and-development", + "a": "openspec-continue-change", + "b": "openspec-new-change" + }, + { + "jaccard": 0.3462, + "plugin": "preferences-programming-languages", + "a": "preferences-haskell-development", + "b": "preferences-typescript-nodejs-development" + } + ], + "inter": [ + { + "jaccard": 0.3889, + "plugin_a": "version-control-and-forge", + "a": "jj-history-cleanup", + "plugin_b": "preferences-code-and-collaboration-conventions", + "b": "preferences-git-history-cleanup" + }, + { + "jaccard": 0.2963, + "plugin_a": "preferences-programming-languages", + "a": "preferences-python-development", + "plugin_b": "preferences-web-platform-and-deployment", + "b": "preferences-react-tanstack-ui-development" + }, + { + "jaccard": 0.2963, + "plugin_a": "preferences-web-platform-and-deployment", + "a": "preferences-react-tanstack-ui-development", + "plugin_b": "preferences-programming-languages", + "b": "preferences-typescript-nodejs-development" + }, + { + "jaccard": 0.2917, + "plugin_a": "preferences-domain-driven-architecture", + "a": "preferences-architectural-patterns", + "plugin_b": "preferences-event-driven-systems", + "b": "preferences-event-sourcing" + }, + { + "jaccard": 0.2692, + "plugin_a": "preferences-web-platform-and-deployment", + "a": "preferences-cloudflare-wrangler-reference", + "plugin_b": "preferences-nix-and-secrets", + "b": "preferences-secrets" + }, + { + "jaccard": 0.2667, + "plugin_a": "preferences-domain-driven-architecture", + "a": "preferences-architectural-patterns", + "plugin_b": "preferences-web-platform-and-deployment", + "b": "preferences-hypermedia-development" + }, + { + "jaccard": 0.2632, + "plugin_a": "event-modeling-workflow", + "a": "event-modeling-to-eventcatalog", + "plugin_b": "preferences-event-driven-systems", + "b": "preferences-event-catalog-qlerify" + }, + { + "jaccard": 0.2632, + "plugin_a": "version-control-and-forge", + "a": "jj-version-control", + "plugin_b": "preferences-code-and-collaboration-conventions", + "b": "preferences-git-version-control" + } + ] + } +} diff --git a/modules/home/ai/plugins/testing-and-quality/.apm/skills/harborize/references/algebraic-invariants.md b/modules/home/ai/plugins/testing-and-quality/.apm/skills/harborize/references/algebraic-invariants.md new file mode 100644 index 000000000..a2e30e6bc --- /dev/null +++ b/modules/home/ai/plugins/testing-and-quality/.apm/skills/harborize/references/algebraic-invariants.md @@ -0,0 +1,107 @@ +# Algebraic invariants (review gate) + +Each invariant states the mathematical form, the operational check, and what +breaks if it fails. Audit every package against all seven; record pass/fail +with one line of evidence in the package README. + +## 1. Truncation — verifier as decidable proposition on final state + +Form: φ = (−1)-truncated predicate on States, composed with the reflection +Traj → State. Path data is discarded by construction. + +Check: the verifier reads only final filesystem state and declared artifacts; +no transcript parsing; grep verifier/ for reads outside /app, /logs/artifacts. +If graded (Reward Kit), each criterion is still a function of final state; +judge criteria are flagged as the approximate nucleus and quarantined in the +separate verifier container. + +Failure mode: a path-dependent verifier measures the harness's style, not the +skill's efficacy, and breaks cross-harness comparability. + +## 2. Inhabitation — oracle as witness + +Form: the task type is inhabited; oracle : ⊤ → φ. + +Check: oracle passes 5/5 under both runners before any agent run counts. +Flaky oracle ⇒ the reward channel has noise floor ≥ flake rate; fix first. + +## 3. Non-triviality — □_S ⊬ φ + +Form: the skill-graded context lowers the derivation cost of φ without +containing a derivation. + +Check: `scripts/audit_leakage.py` — no verifier expectation string-recoverable +from skill content; no complete solution embedded in the skill; verifier in +separate environment so grading logic is invisible to the agent. If the skill +ships a script that IS the solution, parameterize task inputs so running the +script still requires correct invocation, adaptation, or composition. + +Failure mode: measurement is vacuous — you measured copying. + +## 4. Coupling — paired evaluation shares the task, not the randomness + +Form: contrasts are estimated under a coupling in the Markov category: joint +over tasks, marginalized to conditions. + +The coupling is exact at the task level and positional within a task. Neither +Harbor nor BenchFlow exposes a per-trial seed at the revisions this skill pins, +so replicate k of one condition and replicate k of another share the task and +the ordinal, not a randomness object. Trial ordinals are assigned by +`collect_rewards.py`. Do not claim a shared seed; state the pairing that +actually holds. + +Check: every reported delta is computed per-(task, trial) then aggregated; +identical instructions, images, and verifier across conditions; conditions +differ ONLY in the materialized skills directory. Never compare condition means +computed over different task subsets. `analyze_lattice.py` intersects the +(task, trial) keys of every arm and prints what each arm contributed and what +was dropped; that line is the evidence this invariant cites. + +## 5. Grade discipline — conditions are points of the lattice, nothing more + +Form: E : 2^𝒫 → [0,1]; the interpretation varies only in the coeffect grade. + +Check: `materialize_conditions.py` is the only mechanism that varies between +conditions. No condition-specific Dockerfiles, instructions, or verifiers. +Non-monotonicity is expected, not an anomaly: never infer E(A∪B) from E(A), +E(B). Second difference Δ_uv = E({u,v}) − E({u}) − E({v}) + E(∅) is the +interaction; Möbius inversion over observed subsets gives attribution when the +design covers them. + +Two structural facts about the units and about dir(C) hold this invariant up. + +A plugin unit is derived through the skill-to-plugin membership map the census +emits, never by directory structure: the deployed tree a harness reads is flat, +one directory per skill, with no plugin directories and no `.apm/` paths, so a +plugin unit has no directory to point at. A design that resolves plugin units +by walking the filesystem silently gets a different unit set on the deployed +tree than on the source tree. + +dir(C) must contain skill directories and nothing else. Harbor's +`_find_skill_dirs` raises on a root holding a non-hidden child directory +without a `SKILL.md` (harbor `src/harbor/skills.py:382-416`), so a stray +directory turns the whole condition into a hard error. `materialize_conditions.py` +refuses to replace a destination holding anything but skill folders for this +reason, and refuses a destination a Dockerfile above it would COPY into an +image, since baking dir(C) into a build context makes the image vary with the +condition. + +## 6. Empirical naturality — portability as uniformity over cells + +Form: the skill's benefit Δ should be (lax-)natural in the (model, harness) +index; the grid samples dinaturality at finitely many objects. + +Check: report per-cell Δ side by side. Same sign and comparable magnitude ⇒ +the portability claim is supported at the sampled objects. Sign flips ⇒ the +skill's value is model- or harness-idiosyncratic; say so, do not average it +away. + +## 7. Nucleus hygiene — know which closure operator you are using + +Form: exact nucleus where φ is decidable; approximate nucleus (LLM judge, +ε-idempotent under stochastic verification) only where it is not. + +Check: the package README lists, per criterion, exact vs judge. Judge criteria +carry k ≥ 3 trials and report agreement; a judge criterion that disagrees with +itself across trials at rate ≥ ε is noise, not signal — tighten the rubric or +demote the criterion to qualitative. diff --git a/modules/home/ai/plugins/testing-and-quality/.apm/skills/harborize/references/emitters.md b/modules/home/ai/plugins/testing-and-quality/.apm/skills/harborize/references/emitters.md new file mode 100644 index 000000000..002c169e8 --- /dev/null +++ b/modules/home/ai/plugins/testing-and-quality/.apm/skills/harborize/references/emitters.md @@ -0,0 +1,352 @@ +# Dual emission: one authored tree, one exported head + +Canonical content: the instruction body, `environment/Dockerfile`, one solver script, one verifier script and its checks. +The Harbor head is metadata plus naming shims over that content. +Author the BenchFlow-native tree, derive the Harbor head with `bench tasks export` into a sibling directory, and validate each head with the gate that can run against its layout. +Deriving rather than maintaining two trees is what keeps the heads from drifting; the two cannot share a directory, for the reasons under "Two heads, two directories" below. + +## Verified against + +Every claim below was read out of these clones at these revisions. + +| repo | path | revision | date | +|---|---|---|---| +| Harbor | `~/ghq/github.com/harbor-framework/harbor` | `ac398bbda7c4c1073461797d3b95c2455cc671b5` | 2026-08-12 | +| BenchFlow | `~/ghq/github.com/benchflow-ai/benchflow` | `d30527b82027a416e72014920cdf43a534967ad3` | 2026-08-12 | +| SkillsBench | `~/ghq/github.com/benchflow-ai/skillsbench` | `9a1f4dd5f7659f75707435da3ce854b6e48321d1` | 2026-07-23 | + +These are shallow clones sitting at HEAD rather than at released tags, so a line anchor is only valid at the revision above. +Re-read before trusting an anchor. + +The task schema is moving under both heads and the two heads are on different numbers. +Harbor's `TaskConfig.schema_version` defaults to `"1.4"` (`src/harbor/models/task/config.py:796`); the field is a plain `str` with no enum, no range check and no compatibility gate, and the only migration it performs is renaming a legacy `version` key to `schema_version` (`config.py:824-828`). +Harbor's own example packages still declare `schema_version = "1.3"` (`examples/tasks/separate-verifier-environment/task.toml:1`), and all 87 shipped SkillsBench tasks declare `schema_version: '1.3'` in their `task.md` frontmatter. +Emitting 1.3 against a 1.4 Harbor is therefore valid and matches the SkillsBench corpus. + +## Two heads, two directories + +Author the package BenchFlow-native and derive the Harbor head into a separate sibling directory. +This is the sanctioned route rather than one option among several, because the two heads cannot share a directory. + +Three independent validators forbid the co-present form. +BenchFlow's `--level publication-grade` rejects a `task.toml` or `instruction.md` beside `task.md` (`src/benchflow/_utils/task_authoring/structural_checks.py:208-212`) and rejects `solution/` in favour of `oracle/` (`:218-222`). +SkillsBench's corpus gate forbids exactly `instruction.md`, `task.toml`, `solution` and `tests` (`.github/scripts/validate_tasks.py:21-26`), which is precisely the set Harbor requires. +And the corpus itself is uniformly native: all 87 tasks under `tasks/` carry `task.md` and `oracle/solve.sh` with zero `task.toml` and zero `solution/`, as do the 14 under `tasks-extra/`. + +`bench tasks export --target harbor` performs the derivation (benchflow `src/benchflow/cli/tasks.py:281`, `src/benchflow/task/export.py:266-286`). +It writes `task.toml` and `instruction.md` from `task.md`, copies `oracle/` to `solution/`, copies `verifier/` to `tests/`, copies `environment/` verbatim, and emits `compatibility/export-report.json` naming everything that did not survive the conversion. +`--report-only` prints that report without writing files, and `--overwrite` replaces an existing export directory. +The destination may not overlap the source in either direction (`export.py:242-251`); the export refuses rather than deleting source trees it has not yet copied, which is also why the Harbor head is a sibling and never a subdirectory. + +So the native tree is the single source of truth, the export is a build product, and the loss report is machine-readable. +Re-export after every edit to the native tree and never hand-edit the Harbor head, whose contents are overwritten. +Hardlinks or a sync step remain the fallback only for a package inherited Harbor-first. + +The export copies `environment/` verbatim, `environment/skills/` included. +Keep that directory empty in the canonical package: conditions are injected at run time, and a populated `environment/skills/` bakes one condition into the image. + +## Shared to Harbor head + +``` +task.toml # metadata below +solution/solve.sh # from oracle/solve.sh +tests/test.sh # from verifier/test.sh (+ checks.py, judge.toml) +tests/Dockerfile # required by separate mode; see below +``` + +task.toml essentials (full detail: harbor repo `skills/create-task/SKILL.md`): + +```toml +schema_version = "1.3" +[task] +name = "/"; version = "1.0.0" +description = "..."; keywords = ["skills-eval", "", "rewardkit|pytest"] +[metadata] +difficulty = "easy|medium|hard"; category = "..."; tags = ["..."] +[agent] +timeout_sec = 900.0 +[verifier] +timeout_sec = 900.0 +environment_mode = "separate" # one branch of a per-task fork +[verifier.environment] # declaring this table at all implies separate +network_mode = "public" # only if judge/API needed, else no-network +[environment] +network_mode = "no-network" # agent baseline; override per task need +cpus = 1; memory_mb = 4096; storage_mb = 10240 +``` + +The verifier environment is a per-task fork recorded in the package README, not a default. +`_resolve_mode` returns the explicit `environment_mode` when one is set, otherwise infers `separate` from the presence of a `[verifier.environment]` table, otherwise defers; `resolve_task_verifier_mode` falls back to `shared` (harbor `src/harbor/models/task/verifier_mode.py`). +A shared-mode package therefore omits the `[verifier.environment]` table entirely rather than declaring it beside `environment_mode = "shared"`, because the table's presence is itself the separate-mode trigger whenever the explicit key is absent. +SKILL.md Phase 2 carries the fork and its leak consequences. + +Network layering: `[environment].network_mode` is the agent baseline, `[agent]` and `[verifier]` are phase overrides, and `allowlist` mode takes `allowed_hosts` as hostnames or CIDRs rather than URLs. +Keep the agent offline unless the skill's contract requires network. + +## Separate mode requires a verifier image that owns /tests/test.sh + +`environment_mode = "separate"` constructs the verifier environment with `skip_tests_upload=True` (harbor `src/harbor/trial/trial.py:609-619`), and the verifier's `_resolve_tests` then returns an empty upload list on the stated grounds that "the verifier image already owns `/tests/test.{sh,bat}`" (`src/harbor/verifier/verifier.py:96-103`). +Harbor never uploads `tests/` in this mode, so the image must already contain the script. + +The build context for that environment is the task's `tests/` directory (`trial.py:694-702`, `_verifier_env_build_context`), so `tests/Dockerfile` is what gets built. +Declaring `docker_image` in `[verifier.environment]` defeats this: `should_use_prebuilt_docker_image` returns True whenever `docker_image` is set and `force_build` is false (`src/harbor/environments/definition.py:26-36`), the Dockerfile is skipped, and the stock image runs with no `/tests/test.sh` and no way to acquire one. + +The same reasoning extends past the test script to everything it invokes. +Harbor uploads nothing into a separate verifier environment, so the image must itself carry every interpreter and every tool the wrapper calls. +A wrapper that shells out to `uvx` in an image with no uv writes no reward file, the absent reward file raises `RewardFileNotFoundError`, and that exception sits in Harbor's default no-retry list (`src/harbor/models/job/config.py:289-300`), so every trial fails permanently. +Install the verifier's tooling at image build time and invoke it directly rather than resolving it at run time, which also keeps the verifier environment runnable with `network_mode = "no-network"`. + +So a separate-mode package omits `docker_image` from `[verifier.environment]` and ships a Dockerfile that both installs the tooling and places the scripts. +For a Reward Kit verifier: + +```dockerfile +FROM python:3.12-slim + +RUN pip install --no-cache-dir 'harbor-rewardkit==0.1.*' + +COPY --chmod=755 test.sh /tests/test.sh +COPY checks.py judge.toml /tests/ +``` + +`harbor-rewardkit` installs a `rewardkit` console script (harbor `packages/rewardkit/pyproject.toml:37-38`), so `test.sh` calls `rewardkit /tests` with no `uvx` involved. +For a pytest verifier, swap the install line for `RUN pip install --no-cache-dir pytest==8.4.1`, drop the second `COPY`, and add `test_outputs.py` to the first. +Harbor's own separate-verifier examples take the same shape from the other direction: `examples/tasks/separate-verifier-environment/tests/Dockerfile` pairs `FROM ubuntu:24.04` with a pure-bash `test.sh` that invokes no interpreter at all. + +Author this file at `verifier/Dockerfile` in the native tree; `bench tasks export` lands it at `tests/Dockerfile`. + +## Skill injection under Harbor + +`--skill ` (alias `--skills`) is the injection mechanism (harbor `src/harbor/cli/jobs.py:642-653`, on the `start` command that `harbor run` aliases at `src/harbor/cli/main.py:164`). +It is repeatable, and each value is a host path or a git source, meaning `org/name`, `org/name@ref`, or a full URL. +`resolve_skill_sources` takes the local-path branch when the path exists or the value starts with `.`, `/` or `~`, and only otherwise tries git resolution (`src/harbor/skills.py:126-176`), so an existing relative path passes through unchanged and a relative path that does not exist relative to the working directory is parsed as a git source. +Prefix relative paths with `./` to stay on the local branch either way. + +A path value may be either a single skill directory containing `SKILL.md`, or a root whose immediate child directories each contain one; a non-hidden child directory without `SKILL.md` is a hard error, while files and dot-directories at the root are ignored (`skills.py:382-416`). +Skill names come from directory basenames and duplicates resolve last-wins. + +The flag populates `AgentConfig.skills`, which the trial resolves (`trial.py:1142-1145`) and uploads per trial into `//` (`trial.py:1179-1204`). +The destination defaults to `/harbor/skills` (`src/harbor/models/trial/paths.py:41`) unless the task sets `[environment].skills_dir`, which must then be absolute or the trial raises (`trial.py:1162-1177`). +Injection also SHA-pins each skill into the trial's lock, recording name, source, content digest, and the git URL and commit id when the source sits in a repository (`src/harbor/models/job/lock.py:141-146`, `:462-475`), which is provenance worth having on its own. +The lock records the host, not the container. +`_write_trial_lock` runs in `Trial.__init__` at `trial.py:104`, ahead of `_resolve_injected_skills` at `:107` and far ahead of `_upload_injected_skills` at `:411`, and `_build_agent_skill_locks` calls only host-side functions. +So the lock is written before anything is uploaded and stays fully populated through an upload failure, a permissions failure, or an adapter that never reads the directory. +What it proves is that the paths resolved on the host and what their contents digested to. + +`--ak skills_dir=` is not an injection mechanism and must never be used as one. +`skills_dir` is an agent constructor kwarg documented as "Skills directory path in the environment" (`src/harbor/agents/base.py:75`), meaning container-side. +A skills-aware adapter uses it as the source of an in-container copy, for instance Claude Code's `cp -r /* $CLAUDE_CONFIG_DIR/skills/ 2>/dev/null || true` (`src/harbor/agents/installed/claude_code.py:1530-1542`). +Passing a host path there names a directory that does not exist inside the container, the redirect and `|| true` swallow the failure, and the run exits 0. + +The failure signature is that every condition becomes identical to the empty condition. +All first differences collapse toward zero, second differences follow, nothing in the pipeline raises, and the numbers look like a real negative result. +The trial lock catches this particular defect, because `--ak skills_dir=` never populates `AgentConfig.skills` and the lock's `skills` list therefore comes out empty while the condition asked for units. +`scripts/collect_rewards.py` reads that list per trial and refuses the batch when it disagrees with the condition. +The check is one-sided, for the reason given above: a populated lock is host-side resolution, so it rules out this defect and not the delivery failures that follow it. + +The two settings can also collide. +The trial passes its own resolved `skills_dir` into the agent constructor (`trial.py:838-839`) and the factory merges with trial kwargs last (`src/harbor/agents/factory.py:183`), so whenever real injection or a task-level `environment.skills_dir` is in play, an `--ak skills_dir=` value is silently overridden. + +## The silent-adapter class + +Injection uploads regardless of adapter, but registration is adapter-specific. +Of the 39 agents the factory registers, 22 read the injected directory and 17 ignore it with no error, no warning and no log line. +The two commands that establish those figures: + +``` +rg -c 'AgentName\.[A-Z_0-9]+: ' src/harbor/agents/factory.py # 39 +rg -l 'self\.skills_dir' src/harbor/agents/ --type py # 23 +``` + +The second returns 23 files, being the 22 adapters plus `agents/base.py:85`, which is the definition site. +Map each module back to its registry name through `factory.py`, because the two differ: `qwen_code.py` is registered as `qwen-coder` and `installed/cline/cline.py` as `cline-cli`. + +Consuming: antigravity-cli, antigravity-sdk, claude-code, cline-cli, codex, copilot-cli, cursor-cli, eve, fx, gemini-cli, goose, grok-build, hermes, kimi-cli, kimi-code, mimo, openclaw, opencode, pi, qwen-coder, terminus-2, vibe. + +Non-consuming: acp, aider, computer-1, cortex-code, deerflow, devin, dspy-rlm, langgraph, mini-swe-agent, nemo-agent, nop, openhands, openhands-sdk, oracle, rovodev-cli, swe-agent, trae-agent. + +The narrower glob `rg -l 'skills_dir' src/harbor/agents/installed/*.py` returns twenty and reads as if it were the answer. +It is not the adapter set: it misses `installed/cline/cline.py`, which sits in a subdirectory, and `terminus_2/terminus_2.py`, which sits outside `installed/`, and it counts modules where the question is registry entries. + +`acp` is a non-consumer and also a router. +`factory.py:167-175` sends any name matching `is_acp_registry_shorthand`, meaning any name prefixed `acp:`, through `AgentName.ACP`, so an ACP-shorthand cell drops skills on the Harbor arm while the BenchFlow arm for the same agent works. + +`scripts/design_matrix.py` refuses to emit a manifest for a cells.json naming a non-consuming Harbor adapter or an `acp:` shorthand, and carries the allowlist as `HARBOR_SKILL_CONSUMING_AGENTS` with the regenerating commands beside it. +That gate is where this class has to be caught: it costs nothing, it runs before any container starts, and no artifact written after the run distinguishes a non-consuming adapter from a working one. + +## Where each adapter puts the skills + +The destination differs per adapter, so no single canary assertion covers a grid. + +| adapter | destination in the container | anchor | +|---|---|---| +| claude-code | `$CLAUDE_CONFIG_DIR/skills//`, and `CLAUDE_CONFIG_DIR` is `/logs/agent/sessions` | `claude_code.py:1530-1542`, `:1718`; `models/trial/paths.py:36` | +| codex | `$HOME/.agents/skills//` | `codex.py:1199-1207` | +| pi | `$HOME/.agents/skills//` | `pi.py:75-83` | +| opencode | `~/.config/opencode/skills//` | `opencode.py:425-433` | + +claude-code's destination is the one with a free consequence: `/logs/agent` is bind-mounted from the trial directory (`trial.py:1284-1288`), so whatever the adapter registered is readable on the host after the run with no verifier code at all. + +The two runners do not agree on these paths, so a cell's cross-runner comparability is a per-adapter question. +BenchFlow declares `skill_paths` per agent in `src/benchflow/agents/registry.py`: `codex-acp` at `:604` declares `["$HOME/.agents/skills"]`, which is exactly Harbor's codex destination, so codex is the one cell where both agree. +`pi-acp` at `:560` declares `["$HOME/.pi/agent/skills", "$HOME/.agents/skills"]`, a superset of Harbor's single path. +`opencode` at `:700` declares `["$HOME/.opencode/skills"]`, which does not overlap the `~/.config/opencode/skills` Harbor writes. +`claude-agent-acp` at `:518` declares `["$HOME/.claude/skills"]`, where Harbor redirects `CLAUDE_CONFIG_DIR` away from `~/.claude` entirely. +Whether a skill that loads under one runner loads under the other for pi, opencode and claude is an open question this instrument has not settled; do not report a cross-runner contrast on those cells as though it were. + +## Shared to BenchFlow head (SkillsBench layout) + +``` +task.md # YAML frontmatter (schema_version '1.3') + instruction body +environment/skills/ # EMPTY in the canonical package; dir(C) is injected + # at run time from outside it +oracle/solve.sh +verifier/test.sh +verifier/Dockerfile # exports to tests/Dockerfile for separate mode +``` + +Frontmatter sections: `metadata` (author_name, author_email, difficulty, category, subcategory, plus controlled-vocabulary lists for task_type, modality, interface, skill_type, and free tags), `verifier` (type: test-script, timeout_sec, service, optional hardening like cleanup_conftests), `agent.timeout_sec`, and the sandbox spec. + +The sandbox spec is spelled `sandbox` natively, taking network_mode, os, cpus, memory_mb, storage_mb, gpus and build_timeout_sec. +A top-level `environment` table is accepted as a legacy alias and converted (benchflow `src/benchflow/task/config.py:115-122`), but declaring both spellings in one file is a hard error rather than a silent merge. +The inverse conversion is what `bench tasks export` applies so the Harbor head spells it `[environment]` (`config.py:124-141`). + +Vocabulary lives in the skillsbench repo's `taxonomy.yaml` and `taxonomy.md`. +Validate with `bench tasks check --level ` (benchflow `src/benchflow/cli/tasks.py:93-162`), whose levels are schema, structural (default), runtime-capability, publication-grade, acceptance, and acceptance-live. +It exits 1 with an enumerated issue list and spends no model calls below the acceptance-live level. + +Prompt rules enforced by review there and adopted here: imperative prose, end state rather than steps, absolute paths, never mention skills, and anchored dates when answers are time-sensitive. + +## Skill injection under BenchFlow, and the asymmetry + +`--skills-dir ` takes a host path, validated on the host: a path that is not a directory raises `FileNotFoundError: skills_dir not found: ` (benchflow `src/benchflow/skill_policy.py:135`). +The directory is uploaded to `/skills` in the sandbox and symlinked into each agent's discovery paths (`src/benchflow/agents/install.py:303-360`); the expected skill set is computed as `/*/SKILL.md`, so the host directory is a root of skill directories, the same shape Harbor's `--skill` accepts as a root. + +`--skill-mode` takes `no-skill` (default), `with-skill`, or `self-gen` (`skill_policy.py:19-27`). +Two combinations are rejected outright: `no-skill` with `--skills-dir` raises "no-skill mode cannot be combined with skills_dir" (`skill_policy.py:128`), and `self-gen` with `--skills-dir` likewise. +So the empty condition is `--skill-mode no-skill` alone, and every non-empty condition is `--skill-mode with-skill --skills-dir `. +Passing `--skills-dir` under `with-skill` takes precedence over the task's bundled `environment/skills/` and mounts at `/skills`. + +The resolved host directory is recorded per rollout as `effective_skills_dir` (`skill_policy.py:60-68`), which is what `collect_rewards.py` requires to be non-null under `with-skill`; `skill_mode` on its own records the request rather than the outcome. + +The asymmetry is the reason the Harbor defect survived. +Both runners want the same host directory, but they name the argument differently, and the one whose name reads like a host path is the one that is not. +`--skills-dir` is a host path and fails loudly on a bad one; `skills_dir` under `--ak` is a container path and fails silently on a host one. +A generator that carries the argument across from the bench arm to the Harbor arm produces a run that exits 0 and measures nothing. + +## The free delivery canary + +BenchFlow's oracle path proves skill delivery end to end and makes no model call. +`rollout/__init__.py:1160` takes the `primary_agent == "oracle"` branch and still calls `deploy_skills` at `:1174`, so the whole deployment path runs while the agent is a shell script. + +The assertion is in the container. +`deploy_skills` computes the expected catalogue on the host as `Path(skills_dir).glob("*/SKILL.md")` (`agents/install.py:313-314`) and, with no `agent_cfg` on the oracle path, distributes to `_ORACLE_SKILL_PATHS` — the five discovery paths at `install.py:30-36` covering the claude, codex, opencode, agents and workspace conventions (`:350`). +`_link_skill_paths` then runs one command in the sandbox that links the tree into each path, enumerates `SKILL.md` at depth 2 under both the source and each destination, asserts the two catalogues are equal, and asserts the source catalogue equals the host-computed expected names (`install.py:146-161`). +A mismatch raises `experiment_fidelity/skill_deployment_missing`, naming the expected set (`:176-179`). + +``` +bench eval run --tasks-dir --agent oracle \ + --skill-mode with-skill --skills-dir /tmp/cond- \ + --sandbox docker --jobs-dir runs/canary/ +``` + +Pass criterion: the run exits 0, no rollout carries `experiment_fidelity/skill_deployment_missing`, and each rollout's `effective_skills_dir` is the host `dir(C)` that was passed. +Run it once per distinct dir(C) shape before any metered batch. + +Harbor has no free equivalent. +`--install-only` (`src/harbor/cli/jobs.py:901-910`) runs agent setup and exits, and setup does reach `_upload_injected_skills`, which sits in `_prepare` at `trial.py:411`. +But every adapter's registration copy is built inside `run()`, not `setup()` — `claude_code.py:1733`, `codex.py:1413`, `opencode.py:499`, `pi.py:116` — and `--install-only` skips the agent run. +So the cheapest Harbor evidence still costs a real agent invocation, which is why the adapter allowlist in `design_matrix.py` is a gate rather than a convenience. + +## Run lines + +Job output is laid out per runner under one root, which is what `scripts/design_matrix.py` emits and what `scripts/collect_rewards.py` reads back as `--harbor-jobs runs/harbor` and `--benchflow-jobs runs/bench`. + +Harbor, one line per condition per cell: + +``` +harbor run -p -a claude-code -m anthropic/ \ + -k 3 --n-concurrent 2 \ + --skill /tmp/cond- \ + -o runs/harbor --job-name __ +``` + +Omit `--skill` entirely for the empty condition. +`-k` is `--n-attempts` (`jobs.py:387-394`), `--n-concurrent` caps concurrent trials (`jobs.py:467-474`), and `--job-name` defaults to a timestamp (`jobs.py:366-371`). +`-o` is `--jobs-dir` (`jobs.py:372-383`) and the job lands at `jobs_dir / job_name` (`src/harbor/job.py:628`); emit it explicitly rather than letting harbor fall back to its configured default, so the manifest records where output went. + +`-k 3` interacts with pass@k reporting. +Eligible k values are powers of two and multiples of five up to the minimum attempts per task (`src/harbor/utils/pass_at_k.py:71-84`), so three attempts report pass@2 only. +Choose `-k 4` or `-k 5` if a specific pass@k is the headline. + +BenchFlow, one line per condition per cell per repetition: + +``` +bench eval run --tasks-dir --agent --model \ + --skill-mode with-skill --skills-dir /tmp/cond- \ + --sandbox docker --concurrency 2 \ + --jobs-dir runs/bench/__/trial-01 +# C = empty -> --skill-mode no-skill, and drop --skills-dir +``` + +There is no `--run-id`; passing one kills the arm at argument parsing, and `--jobs-dir` is the run-labelling option it was standing in for. +`bench eval run`'s options are declared inside its signature at benchflow `src/benchflow/cli/main.py:193-592`, except those that share an `Annotated` alias defined in `src/benchflow/cli/_options.py:16-32` and therefore do not appear literally inside the command body: `--model` at `main.py:270` and `--skill-mode` at `main.py:438` are the two the emitted run line depends on. +Search both files before concluding a flag does not exist. +`--concurrency` is "max concurrent tasks" (`main.py:339`), with `--build-concurrency` and `--worker-concurrency` as separate knobs. + +There is no plain repetition flag. +`--trials` exists but is documented as "Number of trials for --matrix" and is consumed only inside the matrix branch (`main.py:585-590`, `:714`), which is itself reachable only under `--tasks-dir`, since `--dataset` and `--source-repo` ignore `--matrix` entirely. +Passing `--trials 3` without `--matrix` runs once. + +Two mechanisms give k repetitions, both verified. + +An outer shell loop with a distinct `--jobs-dir` per repetition is the one that keeps `--model` and `--agent` on the command line, and is what `scripts/design_matrix.py` emits: + +```bash +for t in $(seq -w 1 3); do + bench eval run --tasks-dir --agent --model \ + --skill-mode with-skill --skills-dir /tmp/cond- \ + --sandbox docker --concurrency 2 \ + --jobs-dir runs/bench/__/trial-"$t" +done +``` + +A single-entry matrix file plus `--trials k` is the in-tool equivalent. +The matrix YAML is a `models:` mapping of alias to either a model string or a mapping with `model`, optional `agent`, and optional `agent_env` (`src/benchflow/cli/eval_artifacts.py:159-183`). +The runner iterates aliases then trials, writing each to `//trial-NN` and a `matrix-summary.json` at the root (`eval_artifacts.py:244-305`). +Note that the matrix entry overwrites `eval_config.model`, and `eval_config.agent` when the entry names one, so a command-line `--model` is ignored on this path. + +Condition ids reach these lines verbatim from `--units`, and the census's `` sentinel contains shell redirection metacharacters, so `design_matrix.py` quotes every interpolated path and job name. +A hand-written manifest must do the same. + +Auth per adapter. +Claude Code subscription auth is `CLAUDE_FORCE_OAUTH=1` plus `CLAUDE_CODE_OAUTH_TOKEN`, and Harbor raises if the first is set without the second (`claude_code.py:1587-1633`). +Codex takes `CODEX_FORCE_AUTH_JSON=1` or `CODEX_AUTH_JSON_PATH=` (`codex.py:1305-1325`). +Pi's OAuth escape hatch reads `ANTHROPIC_OAUTH_TOKEN` and is gated only on the resolved provider being anthropic (`pi.py:102-105`), so any other provider falls through to the providers table and Pi on an OpenAI model bills `OPENAI_API_KEY`. + +Pi has no force flag. +Claude Code requires `CLAUDE_FORCE_OAUTH` before it will drop the API key (`claude_code.py:1587-1597`) and Codex requires `CODEX_FORCE_AUTH_JSON` or `CODEX_AUTH_JSON_PATH` (`codex.py:1301-1329`), but Pi injects `ANTHROPIC_OAUTH_TOKEN` whenever the variable is present in the resolved environment. +SKILL.md Phase 5 bars subscription-authenticated cells from any reported run batch, and on a Pi cell that rule is enforced by scrubbing the variable, not by withholding a flag: declare `ANTHROPIC_OAUTH_TOKEN=""` in the cell's `env` block or unset it in the shell the manifest runs in. + +## Verifier scripts + +pytest wrapper, binary reward, for a separate-mode image that already carries pytest: + +```bash +#!/bin/bash +mkdir -p /logs/verifier +pytest /tests/test_outputs.py \ + && echo 1 > /logs/verifier/reward.txt || echo 0 > /logs/verifier/reward.txt +``` + +Reward Kit, graded: `rewardkit /tests` writes reward.json, criteria live in checks.py, the judge in judge.toml, and judge API keys arrive through `[verifier.env]` in task.toml, kept out of the agent environment by separate mode. +Always absolute paths, pin every version, and remember that reward files are the only output channel either runner reads. + +Under shared mode the verifier runs in the agent's own environment, so it is the agent image that must carry the tooling. +That is the branch harbor's `examples/tasks/reward-kit-example` takes: its `environment/Dockerfile` copies uv in with `COPY --from=ghcr.io/astral-sh/uv:latest /uv /uvx /bin/` and its `tests/test.sh` then calls `uvx --from harbor-rewardkit==0.1 rewardkit /tests`. +That package declares no `[verifier.environment]`, so it runs shared, which is the leaking branch and is why the separate-mode recipe above bakes the tool into the verifier image instead. + +Reward shape decides which headline statistics exist, so choose it before the budget rather than after the run. +Harbor computes pass@k only when every trial carries exactly one reward key whose value is an int or float equal to 0 or 1; a second key, a non-numeric value, or a value strictly between 0 and 1 makes `_compute_pass_at_k_for_trials` return an empty mapping and pass@k disappears from the job stats with no warning (`src/harbor/utils/pass_at_k.py:32-53`). +A multi-dimensional Reward Kit rubric is exactly that case. +BenchFlow's `eval compare-lift` defines a pass strictly as `reward == 1.0` (`src/benchflow/eval_lift.py:31-33`, `:514`), so partial credit counts as a failure in its headline pass rate and shows only in mean_reward. +Graded rubrics remain the right call when the contract is graded; the cost is that pass@k and lift pass rates stop being available and the analysis runs on means. diff --git a/modules/home/ai/plugins/testing-and-quality/.apm/skills/harborize/references/lattice-design.md b/modules/home/ai/plugins/testing-and-quality/.apm/skills/harborize/references/lattice-design.md new file mode 100644 index 000000000..36201c57b --- /dev/null +++ b/modules/home/ai/plugins/testing-and-quality/.apm/skills/harborize/references/lattice-design.md @@ -0,0 +1,124 @@ +# Condition lattice, budgets, estimators, selection evals + +## Units and conditions + +𝒫 is the set of ablation units. +A condition C ⊆ 𝒫 is realized by materializing exactly the skill folders that C denotes into dir(C) with `scripts/materialize_conditions.py`, and by nothing else. + +The unit of ablation is the skill, materialized from the deployed tree — what a harness actually reads. +A plugin unit is a derived aggregate: the union of its member skills' folders, resolved through an explicit membership map from skill name to plugin name that `scripts/census.py` emits alongside the census. +The derivation is forced by the shape of the deployed tree. +It is flat, one directory per skill, with no plugin directories and no `.apm/` paths, so a plugin unit has no directory to point at and cannot be resolved by walking the filesystem. +Source trees that do carry plugin directories still resolve structurally, and `materialize_conditions.py` keeps that path for them. + +The census runs over both trees and reports both. +The deployed tree is the evaluation subject; the first-party source tree is the refactor subject. +A stage-2 selection-competition simulation must run over the deployed field, which is about a third larger than the first-party source field, or it understates interference by that margin. + +Realization differs per runner and the difference is load-bearing. +BenchFlow takes `--skills-dir ` as a host path, validates it on the host, and uploads it into the container, so a wrong path fails loudly. +Harbor takes `--skill ` (also spelled `--skills`) as a host path, accepting either one skill directory or a root whose immediate children are all skill directories, so a single flag pointing at dir(C) covers the whole condition; it resolves each skill per trial and SHA-pins it into the trial's `lock.json`. +Harbor rejects a root holding any child directory without a SKILL.md, which is why `materialize_conditions.py` refuses to reuse a destination containing anything else. +Harbor's `skills_dir` agent kwarg is a container-side path consumed inside the container, and passing a host path through `--ak skills_dir=` fails silently and collapses every condition to the empty condition. + +## Budget menu (present with numbers filled in) + +runs(design) = |C| × k × cells. +State this with the user's actual n, k and cells. + +Every budget figure is costed at metered API rates. +Subscription-authenticated cells are permitted for interactive exploration and are never used for a run batch whose numbers are reported. +There are two reasons. +Costing as metered is robust to an unresolved question about credential-use policy in either direction, so the budget does not have to be redone if the answer changes. +Subscription cells also confound the measurement independently, because rate-limit throttling is nondeterministic and single-account authentication caps concurrency, which makes wall-clock and retry behavior part of the condition. + +No per-run cost figure exists yet, anywhere in this skill or its history. +One must come from a calibration batch — a handful of runs on the intended cell, with the token and dollar totals read off the runner — before any budget table is presented. +Do not present a budget table with an invented number in it. + +| design | \|C\| | estimands | +|---|---|---| +| paired marginals | n+1 | E({u})−E(∅) ∀u | +| + targeted pairs | n+1+p | above + Δ_uv for selected pairs | +| + everything-on | +1 | E(𝒫) — deployment condition | +| foldover | 2n+2 for n ≥ 3, min(2n+2, 2ⁿ) below | solo and leave-one-out marginals ∀u | +| full factorial | 2^n | complete E; Möbius attribution | + +The foldover is ∅, the n singletons, the n complements of singletons, and 𝒫. +It gives two contrasts per unit: the solo marginal E({u})−E(∅) and the leave-one-out marginal E(𝒫)−E(𝒫∖{u}). +The gap between those two contrasts aggregates every higher-order interaction involving u, without separating them. +The design has no defining relation, so it carries no resolution and supports no claim that main effects are clear of two-way aliasing; a resolution claim requires a generator table the implementation does not have. +It collapses to the full factorial for n ≤ 3, where the complements of the singletons are the two-element subsets and every condition is already covered, and it saves runs only from n ≥ 4. +Deduplication makes |C| smaller than 2n+2 below n = 3: two units give four conditions rather than six, and one unit gives two rather than four. +At n ≥ 4 it contains no two-element subsets, so no second difference can be computed from it — Δ_uv needs {u}, {v} and {u,v} all present — which makes it the wrong choice when interactions are the question at exactly the sizes where it saves anything. + +## Targeted-pair selection + +Rank pairs by trigger-surface overlap between unit descriptions and pick p pairs from the top of that ranking plus any the user names. +Interference concentrates where descriptions compete for the same intents, which is the near-miss structure; synergy concentrates where one unit's outputs are another's inputs. + +The ranking is computed from the census rather than assembled by hand. +`design_matrix.py --from-census --top-pairs

` scores every pair of the design's own units by description-token Jaccard, unioning member tokens for a derived plugin unit, adds the top p as conditions, and writes the full ranking with the census provenance to `design/selection.json`. +Pass the deployed tree's census, for the reason given under units above. +Pairs the user names through `--pairs` are added on top and are not subject to the cutoff. + +The census's own printed overlap tables are a display slice, not a significance threshold. +The figure of 16 flagged pairs is the top 8 intra-plugin rows plus the top 8 inter-plugin rows, both cutoffs defaulting to 8 through `census.py --top-intra` and `--top-inter` and recorded in the census's `provenance.top_intra` and `provenance.top_inter`, and it ranks over all skills rather than over the units of a particular design. +It says nothing about where overlap stops being meaningful. +Choose p from the budget and the shape of the ranking, and state the Jaccard at the cutoff next to the number of pairs bought. + +## Estimators + +The reward type selects the interval, and `analyze_lattice.py` reads it off the data per task, because the type is a property of that task's verifier rather than something the design declares. +A task whose observed rewards all lie in {0, 1} is binary and gets a Wilson 95% interval on its pass rate. +A task with any other value in [0, 1] is graded and gets a bootstrap percentile interval on the mean, resampling tasks. +A Wilson interval is never reported for a graded mean, because Wilson assumes a Bernoulli count and a mean of partial credit is not one. +A cell that mixes binary and graded tasks is refused rather than pooled, and a reward outside [0, 1] is refused, both as input errors rather than as wider intervals. +Report the binarized-at-threshold view alongside the graded view when the user wants comparability with pass rates, and state the threshold next to every number it produced. + +Contrasts are computed the same way for both types. +Take per-(task, trial) paired differences, then bootstrap over tasks, because tasks are the exchangeable unit and trials within a task are correlated. + +The cluster bootstrap needs at least two distinct tasks to have anything to resample. +Below that every replicate is the same sample and the percentile interval collapses to a point, so `analyze_lattice.py` reports the point estimate and marks the interval unavailable rather than printing a zero-width interval that reads as certainty. +Phase 1 proposes 1 to 3 tasks per unit, so a single-task cell is ordinary and a package built that way carries no task-level interval on any contrast; its README must not claim one. +A binary cell still gets its Wilson interval on E, because Wilson is computed over trials rather than tasks, and that interval treats trials as independent Bernoulli draws, which is a separate approximation from the clustering the bootstrap respects. + +Keys missing from either arm of a contrast are dropped from both, and the analyzer reports how many keys each arm contributed and how many were dropped, so the coupling invariant can be evidenced from its output. +Neither Harbor nor BenchFlow exposes a per-trial seed at the revisions this skill pins, so the coupling is exact at the task level and positional within a task: replicate k aligns with replicate k across conditions. + +Report normalized gain (Hake), (E_C − E_∅)/(1 − E_∅), alongside absolute deltas so cells with different baselines are comparable. +Report per-cell first, and pool across cells only with an explicit note that cells are not exchangeable. + +The runners' own headline numbers respond to reward shape and will disagree with the estimators above if that goes unnoticed. +Harbor computes pass@k only when every trial carries exactly one reward key valued 0 or 1, so a multi-dimensional Reward Kit rubric silently disables it. +BenchFlow's `eval compare-lift` defines a pass as `reward == 1.0`, so partial credit is invisible in its pass-rate row and appears only in its mean-reward row. + +## Interaction reading + +Δ_uv < 0 (interference): likely trigger competition, context pollution, or convention conflict. +Diagnose by inspecting which unit's fingerprints appear in artifacts of failing {u,v} runs. +Δ_uv > 0 (synergy): composition tasks confirm; check the pipeline direction. +Either way the marginal story is incomplete — surface Δ next to marginals, never hide it in an appendix. + +## Selection-competition evals (opt-in) + +Purpose: with many units loaded, does the right one act? +This is a property of the description set and the harness triggering policy jointly, a different object than task efficacy, so keep it in separate tasks. +The loaded set is the deployed set, for the reason given under units above. + +Pattern: +1. Task instruction targets exactly one unit's contract, without naming it. +2. All units in C are loaded (typically C = 𝒫). +3. Verifier checks task success and provenance: the expected unit's bundled script leaves a characteristic, hard-to-fake artifact (a stamped metadata field, a deterministic intermediate file, a format only that script emits). + Provenance stays a final-state check, never transcript parsing. +4. Report P(correct-unit-selected) and P(success | selected) separately; the product recovers the composite, the factors localize the failure. + +Fingerprints must be side effects the skill's script already produces, or a one-line addition to the bundled script made in the evaluated skill copy used for all conditions equally. +A verifier-only expectation the agent could not produce violates inhabitation, and a per-condition edit violates grade discipline. + +## Trigger evals at the lattice level + +skill-creator's trigger methodology (should/should-not, near-misses) lifts to the lattice: for each unit, should-trigger prompts where that unit should win over the others present, and should-not prompts targeting a sibling unit's contract. +These are cheap, since they observe which skill loads without executing a task, and are worth running before spending task-execution budget. +A unit that never wins selection will show zero marginal efficacy for reasons that have nothing to do with its content. diff --git a/modules/home/ai/plugins/testing-and-quality/.apm/skills/harborize/references/marketplace-program.md b/modules/home/ai/plugins/testing-and-quality/.apm/skills/harborize/references/marketplace-program.md new file mode 100644 index 000000000..197de6328 --- /dev/null +++ b/modules/home/ai/plugins/testing-and-quality/.apm/skills/harborize/references/marketplace-program.md @@ -0,0 +1,102 @@ +# Marketplace validation program (vanixiets) — state and plan + +Context for any session picking this up fresh. +The instrument, meaning this skill, and the program below were developed against the apm marketplace at github.com/cameronraysmith/vanixiets (ghq canonical path). +Consume plugins through the apm CLI (>= 0.28.0; portable form `cameronraysmith/vanixiets/modules/home/ai/plugins/#main`). + +Instrument versions are tracked from now on. +`0.1.0` is the as-authored baseline already committed; `0.2.0` is the repair round that produced the current text. +Evaluation results are indexed by the instrument version that produced them, so cross-version comparisons stay honest, and the instrument is not modified while an evaluation is being authored — revisions happen between rounds. + +## Two subjects, one census + +The deployed tree is the evaluation subject. +It is what a harness actually reads: 172 skill directories under `~/.claude/skills`, flat, one directory per skill, with no plugin directories and no `.apm/` paths, produced by composing the apm packages plus a few unmanaged skills. +A hermetic run can census the compose output instead, which differs from the delivered tree by any skill installed outside the composition. + +The first-party source tree is the refactor subject. +It is `modules/home/ai/plugins//.apm/skills/`, 18 plugins and 129 skills. +Refactor edits land there; measurements are taken on the deployed field. + +The deployed field is about a third larger than the first-party source field, 172 skills against 129, because 43 of the deployed skills belong to no first-party plugin and the census records them under the `` sentinel. +Those skills compete for the same trigger surface as the first-party ones. +A stage-2 selection-competition simulation run over the first-party descriptions alone understates interference by that margin, so it must run over the deployed field. + +Units follow from this. +The unit of ablation is the skill; a plugin unit is derived as the union of its member skills' folders through the skill-to-plugin membership map the census emits, because a flat tree has no plugin directory to point at. +See `references/lattice-design.md` for the lattice and `scripts/materialize_conditions.py --membership` for the mechanism. + +Counts and rankings in this file are a snapshot, and every one of them was reproduced against the current tree when this text was last revised. +The skill bundles a source census at `census.json`, whose `provenance` block records the root, layout, revision and script version it was taken at, and which doubles as the `--membership-from` input the deployed census needs. +Re-run `scripts/census.py` against both roots and read the numbers from its output rather than quoting them from here. + +The source tree gives 26 command-style skills (`disable-model-invocation`), 54 env-coupled skills (`~/...` references), and a decidable-versus-subjective split of 27 to 102. +The deployed tree gives 39, 57, and 36 to 136. +An earlier run reported the source split as 15 to 113; that figure is superseded and must not be carried forward. +It came from a description parser that silently emptied 24 of the 128 descriptions it read, and an empty description falls to the subjective side of the keyword heuristic by default. +Both current runs report zero empty descriptions. +The split remains a heuristic floor either way, not a verdict on any individual skill. + +Top seams by lexical Jaccard over description tokens, identical across both subjects at the head of the ranking: `preferences-python-development`:`preferences-typescript-nodejs-development` at 0.44 intra-plugin, `meta-load-cc-docs`:`meta-load-prompting-docs` at 0.40 intra-plugin, `jj-history-cleanup`:`preferences-git-history-cleanup` at 0.39 cross-plugin. +The deployed run adds `grill-me`:`grill-with-docs` at 0.42 among the external skills, which the source run cannot see. +The census is pure static analysis and is safe to run in CI. + +## Cells (the model x harness grid) + +claude-code + claude-opus-5 (`CLAUDE_FORCE_OAUTH=1` plus `CLAUDE_CODE_OAUTH_TOKEN`); codex + gpt-5.6-sol and codex + gpt-5.6-luna (`CODEX_FORCE_AUTH_JSON=1`, shared ChatGPT quota, serialize them); pi + gpt-5.6-luna (`OPENAI_API_KEY`). + +The pi cell is the only one with no subscription path. +Pi's OAuth escape hatch is gated on the model's provider being anthropic, so an OpenAI model falls through to the providers table and bills `OPENAI_API_KEY` per call. +Every figure quoted for that cell is metered spend. + +All budget figures across all cells are costed at metered API rates regardless. +Subscription-authenticated cells are for interactive exploration and are never used for a run batch whose numbers are reported. +See the budget section of `references/lattice-design.md` for the reasoning and for the requirement that a per-run cost come from a calibration batch before any budget table is presented. + +## Stages + +1. Static hardening (100 percent coverage, near-free): grow `census.py` into a linter — frontmatter validity, description quality per trigger methodology, env-coupling remediation list, progressive-disclosure structure. Keep it dependency-free so it can run inside a hermetic derivation. Fix before measuring. A `lint` skill mode does not exist yet. +2. Selection layer (100 percent coverage, description-only): everything-on triggering simulation over the deployed descriptions, adjudicating the competition pairs the census ranks. Embedding overlap beats lexical Jaccard here. A `selection-sim` skill mode does not exist yet. +3. Stratified dynamic eval: the decidable stratum becomes full packages across four cells; the subjective stratum is first refactored toward decidability with companion checkers, since conventions are lintable, and only the irreducible remainder is judge-graded, subject to the judge gate below. Plugin-level lattice over the 18 derived first-party units: marginals plus everything-on is 20 conditions. Second differences on census-ranked pairs only. Whether the 43 external deployed skills form a 19th unit or a fixed background present in every condition is undecided, and the two choices give different lattices and different everything-on conditions. +4. Refactor loop: merge near-duplicates (the jj cluster; git-versus-jj becomes one skill parameterized by VCS), split the 16-skill plugins, delete negative-marginal skills, description-optimize the survivors. Rerun only affected packages, paired against the prior version. The design goal is that E is approximately modular over the plugin partition — orthogonal trigger surfaces, near-zero interactions except designed synergies — because the compendium property and the cheap-validation property are the same property. +5. Institutionalize: packages become marketplace CI (extend apm-marketplace-validate; buildbot-nix runs the decidable stratum per PR; the full grid per model release; publish versioned benchmark results). + +The screening cost for stage 3 has to be re-derived rather than carried forward. +The figure of roughly 500 to 700 runs was scoped to the pi cell, which is entirely metered, and its task count was never stated: 20 conditions at k=3 on one cell is 60 runs per task, so that figure implies 8 to 12 tasks. +Choose the task count deliberately, multiply by a measured per-run cost, and present the result. + +## Judge gate + +Reduce a contract to a mechanical check wherever a mechanical check exists. +Exactly one judge-validation package is a hard gate before any judge-based stratum enters a budget. + +The gate is warranted by how little the surrounding ecosystem leans on judges. +All 87 tasks shipped in SkillsBench declare `verifier.type: test-script`; none uses a judge verifier. +The acceptance bar adopted here for a judge criterion is a human-labeled validation set of roughly 6 to 12 submissions spanning pass, fail, partial, borderline, plausible-but-wrong and polished-but-unsupported, plus demonstrated agreement with the human labels and stability across runs. + +If the gate fails, the subjective stratum falls back to structural proxies. +A structural proxy asks whether the produced artifact exhibits the convention, checkable by grep or AST, rather than asking for a judgment of quality. + +## Iteration-1 packages (reference only) + +Three packages exist under `~/Downloads/harbor-skill/harborize-workspace/iteration-1/`: eval-1 process-compose-init, eval-2 jj-cluster (9-condition design, 108-run manifest), eval-3 cross-plugin nix x jj (full factorial n=2, 48 runs). +They are reference artifacts and are to be regenerated under the repaired instrument. +None met the oracle inhabitation invariant, so none is validated. +None carries the README the review gate routes its evidence through. +Two ship an `environment/mkfixture.sh`, which the fixture-generation rule forbids. +Eval-1's separate-mode verifier cannot run as configured, because separate mode never uploads `tests/` and the declared stock verifier image does not already own `/tests/test.sh`. +Cite them for shape, not for results. + +## Open decisions + +Whether the `lint` and `selection-sim` skill modes come before or after a companion-checker pattern drafted against one live preferences skill is open. +It was put to the user as a fork and never answered, so it is not an inherited sequence. + +Whether the external deployed skills are an ablation unit or a fixed background is open, as noted under stage 3. + +## First actions in a fresh session + +1. Gate each regenerated package on the oracle passing its verifier 5 out of 5 in containers under both runners. `harbor tasks check` and `harbor task check` are removed and exit 1 unconditionally; `harbor check` is a different instrument, an LLM rubric review that spawns a full metered job and whose exit code ignores rubric failures, so it is not a drop-in substitute in cost or semantics. `bench tasks check` remains and validates the BenchFlow head. +2. Draft the companion-checker pattern against one live preferences skill (for example `preferences-python-development`) before committing to it across the subjective stratum. +3. Description optimization for harborize itself through skill-creator's run loop. +4. Use parallel subagents for test-case runs and blind comparison, per skill-creator's full workflow. diff --git a/modules/home/ai/plugins/testing-and-quality/.apm/skills/harborize/scripts/analyze_lattice.py b/modules/home/ai/plugins/testing-and-quality/.apm/skills/harborize/scripts/analyze_lattice.py new file mode 100755 index 000000000..fdf7084ad --- /dev/null +++ b/modules/home/ai/plugins/testing-and-quality/.apm/skills/harborize/scripts/analyze_lattice.py @@ -0,0 +1,341 @@ +#!/usr/bin/env -S uv run --script +# /// script +# requires-python = ">=3.12" +# dependencies = [] +# /// +"""Aggregate rewards into E(C) per cell with paired first and second differences. + +Input: results.json = [{"condition": "a+b", "cell": "...", "task": "...", + "trial": 0, "reward": 1.0}, ...] +Produce it with scripts/collect_rewards.py; extra keys are carried through the +collector and ignored here. + +Interval branch (invariant 5 reads E as a map into [0,1], so the estimator has +to match the reward's own type): the reward type is read off the data, per +task, because it is a property of that task's verifier. A task whose observed +rewards all lie in {0, 1} is binary and gets a Wilson interval; any other value +in [0, 1] makes the task graded and it gets a bootstrap percentile interval on +the mean, resampling tasks. A cell mixing binary and graded tasks is refused: +pooling a pass rate with a partial-credit mean is a category error, not a +wider interval. Values outside [0, 1] are refused for the same reason. + +The bootstrap resamples tasks as clusters, so it needs at least two distinct +tasks to have anything to resample. Below that every replicate is the same +sample and the percentile interval collapses to a point. A design carrying one +task per cell is refused an interval and reports its point estimate alone, +rather than printing a zero-width interval that reads as certainty. Phase 1 +proposes 1-3 tasks per unit, so a single-task cell is an ordinary case and this +is a degradation rather than an error. + +Coupling (invariant 4) is enforced here rather than assumed. Every contrast +intersects the (task, trial) keys of its arms and reports how many keys each +arm contributed and how many were dropped, so the review gate can be evidenced +from this output. Neither Harbor nor BenchFlow exposes a per-trial seed at the +revisions this skill pins, so the coupling is exact at the task level and +positional within a task: trial ordinals are assigned by the collector and +align replicate k with replicate k across conditions. + +Exit codes: + 0 analysis ran + 1 --strict-pairing was requested and a contrast dropped keys + 2 input refused: malformed rows, mixed reward types, out-of-range rewards +""" + +import argparse +import collections +import itertools +import json +import math +import pathlib +import random +import statistics as st +import sys + +ROW_KEYS = ("condition", "cell", "task", "trial", "reward") +EMPTY_CONDITION = "none" +MIN_BOOTSTRAP_TASKS = 2 +ADDITIVE_TOLERANCE = 1e-9 +NO_INTERVAL = f"[no interval: needs >= {MIN_BOOTSTRAP_TASKS} tasks]" + + +class InputError(Exception): + """The results file cannot be analyzed as given.""" + + +def load_rows(path): + try: + rows = json.loads(pathlib.Path(path).read_text()) + except (OSError, json.JSONDecodeError) as exc: + raise InputError(f"cannot read {path}: {exc}") from exc + if not isinstance(rows, list) or not rows: + raise InputError(f"{path} is not a non-empty JSON list of result rows") + for index, row in enumerate(rows): + if not isinstance(row, dict): + raise InputError(f"row {index} is not an object") + missing = [key for key in ROW_KEYS if key not in row] + if missing: + raise InputError(f"row {index} is missing {', '.join(missing)}") + if not isinstance(row["reward"], (int, float)) or isinstance( + row["reward"], bool + ): + raise InputError(f"row {index} has a non-numeric reward") + return rows + + +def classify_rewards(values): + for value in values: + if not 0.0 <= value <= 1.0: + raise InputError( + f"reward {value} lies outside [0, 1]; E is a map into [0, 1] " + "and neither estimator is defined off it" + ) + return "binary" if all(value in (0.0, 1.0) for value in values) else "graded" + + +def cell_reward_type(cell_rows): + """Reward type for a cell, refusing a cell whose tasks disagree.""" + by_task = collections.defaultdict(list) + for row in cell_rows: + by_task[row["task"]].append(float(row["reward"])) + per_task = {task: classify_rewards(values) for task, values in by_task.items()} + types = set(per_task.values()) + if len(types) > 1: + binary = sorted(t for t, kind in per_task.items() if kind == "binary") + graded = sorted(t for t, kind in per_task.items() if kind == "graded") + raise InputError( + "cell mixes reward types across tasks; analyze them separately. " + f"binary: {', '.join(binary)}. graded: {', '.join(graded)}" + ) + return types.pop() + + +def wilson(k, n, z=1.96): + if n == 0: + return 0.0, 0.0, 1.0 + p = k / n + d = 1 + z * z / n + center = (p + z * z / (2 * n)) / d + half = z * math.sqrt(p * (1 - p) / n + z * z / (4 * n * n)) / d + return p, max(0.0, center - half), min(1.0, center + half) + + +def bootstrap_percentile_ci(values, task_keys, iters, seed): + """Percentile interval on the mean, resampling tasks with replacement. + + Tasks are the exchangeable unit; trials within a task are correlated, so + the cluster is the task and every one of its values travels together. + + Returns (nan, nan) below MIN_BOOTSTRAP_TASKS distinct tasks, where every + replicate would be the same sample and the interval would be a point. + """ + by_task = collections.defaultdict(list) + for value, task in zip(values, task_keys): + by_task[task].append(value) + tasks = sorted(by_task) + if len(tasks) < MIN_BOOTSTRAP_TASKS: + return float("nan"), float("nan") + rng = random.Random(seed) + means = [] + for _ in range(iters): + sample = [ + value + for task in rng.choices(tasks, k=len(tasks)) + for value in by_task[task] + ] + means.append(st.mean(sample)) + means.sort() + lo = means[min(int(0.025 * iters), iters - 1)] + hi = means[min(int(0.975 * iters), iters - 1)] + return lo, hi + + +def format_interval(lo, hi, sign=""): + if math.isnan(lo) or math.isnan(hi): + return NO_INTERVAL + return f"[{lo:{sign}.3f},{hi:{sign}.3f}]" + + +def condition_estimate(rows, reward_type, iters, seed): + values = [float(row["reward"]) for row in rows] + tasks = [row["task"] for row in rows] + if reward_type == "binary": + successes = sum(1 for value in values if value == 1.0) + point, lo, hi = wilson(successes, len(values)) + return point, lo, hi, "wilson" + point = st.mean(values) + lo, hi = bootstrap_percentile_ci(values, tasks, iters, seed) + return point, lo, hi, "bootstrap" + + +def keyed(rows, condition): + return { + (row["task"], row["trial"]): float(row["reward"]) + for row in rows + if row["condition"] == condition + } + + +def couple(rows, conditions): + """Intersect the (task, trial) keys of several conditions. + + This is the operational form of invariant 4: a contrast is only ever taken + over keys every arm carries. The returned record is the evidence the review + gate cites, so it is emitted whether or not anything was dropped. + """ + arms = {condition: keyed(rows, condition) for condition in conditions} + shared = set.intersection(*(set(arm) for arm in arms.values())) if arms else set() + keys = sorted(shared) + record = { + "conditions": list(conditions), + "n_shared": len(keys), + "n_by_arm": {condition: len(arm) for condition, arm in arms.items()}, + "n_dropped_by_arm": { + condition: len(arm) - len(keys) for condition, arm in arms.items() + }, + } + return arms, keys, record + + +def format_coupling(record): + parts = [ + f"{condition or EMPTY_CONDITION}:{record['n_by_arm'][condition]}" + f"-{record['n_dropped_by_arm'][condition]}" + for condition in record["conditions"] + ] + return f"paired n={record['n_shared']} (arm:n-dropped {' '.join(parts)})" + + +def analyze_cell(cell, cell_rows, units, args, dropped_flag): + reward_type = cell_reward_type(cell_rows) + n_tasks = len({row["task"] for row in cell_rows}) + print(f"\n== {cell} == reward-type={reward_type} tasks={n_tasks}") + if n_tasks < MIN_BOOTSTRAP_TASKS: + print( + f" note: {n_tasks} task in this cell, so every bootstrap " + "interval is suppressed; point estimates only" + ) + + for condition in sorted({row["condition"] for row in cell_rows}): + rows = [row for row in cell_rows if row["condition"] == condition] + point, lo, hi, estimator = condition_estimate( + rows, reward_type, args.bootstrap_iters, args.bootstrap_seed + ) + label = condition or EMPTY_CONDITION + print( + f" E({label:<20}) = {point:.3f} {format_interval(lo, hi)} " + f"n={len(rows)} ({estimator})" + ) + + for unit in units: + arms, keys, record = couple(cell_rows, [unit, EMPTY_CONDITION]) + if not keys: + print(f" D({unit}) unavailable: {format_coupling(record)}") + continue + if any(record["n_dropped_by_arm"].values()): + dropped_flag.append((cell, record)) + diffs = [arms[unit][key] - arms[EMPTY_CONDITION][key] for key in keys] + tasks = [key[0] for key in keys] + lo, hi = bootstrap_percentile_ci( + diffs, tasks, args.bootstrap_iters, args.bootstrap_seed + ) + base = st.mean([arms[EMPTY_CONDITION][key] for key in keys]) + gain = ( + f" norm-gain={st.mean(diffs) / (1 - base):.2f}" if base < 1.0 else "" + ) + print( + f" D({unit}) = {st.mean(diffs):+.3f} " + f"{format_interval(lo, hi, '+')}{gain} " + f"{format_coupling(record)}" + ) + + present = {row["condition"] for row in cell_rows} + for left, right in itertools.combinations(units, 2): + pair = "+".join(sorted([left, right])) + quad = [pair, left, right, EMPTY_CONDITION] + if not set(quad) <= present: + continue + arms, keys, record = couple(cell_rows, quad) + if not keys: + continue + if any(record["n_dropped_by_arm"].values()): + dropped_flag.append((cell, record)) + seconds = [ + arms[pair][key] + - arms[left][key] + - arms[right][key] + + arms[EMPTY_CONDITION][key] + for key in keys + ] + tasks = [key[0] for key in keys] + lo, hi = bootstrap_percentile_ci( + seconds, tasks, args.bootstrap_iters, args.bootstrap_seed + ) + point = st.mean(seconds) + if abs(point) <= ADDITIVE_TOLERANCE: + tag = "additive" + else: + tag = "synergy" if point > 0 else "interference" + print( + f" D2({left},{right}) = {point:+.3f} " + f"{format_interval(lo, hi, '+')} ({tag}) " + f"{format_coupling(record)}" + ) + + +def build_parser(): + parser = argparse.ArgumentParser( + description="Aggregate condition-lattice rewards into E(C) and contrasts.", + epilog="Exit codes: 0 analysis ran, 1 strict pairing violated, " + "2 input refused.", + ) + parser.add_argument("results", help="results.json from collect_rewards.py") + parser.add_argument( + "--units", nargs="+", required=True, help="ablation unit names" + ) + parser.add_argument("--bootstrap-iters", type=int, default=2000) + parser.add_argument("--bootstrap-seed", type=int, default=0) + parser.add_argument( + "--strict-pairing", + action="store_true", + help="exit 1 if any contrast dropped an unpaired (task, trial) key", + ) + return parser + + +def main(argv=None): + args = build_parser().parse_args(argv) + try: + rows = load_rows(args.results) + except InputError as exc: + print(f"analyze_lattice: {exc}", file=sys.stderr) + return 2 + + conditions = {row["condition"] for row in rows} + for unit in args.units: + if unit not in conditions: + print( + f"analyze_lattice: unit {unit!r} appears in no condition; " + f"observed conditions: {', '.join(sorted(conditions))}", + file=sys.stderr, + ) + + dropped_flag = [] + for cell in sorted({row["cell"] for row in rows}): + cell_rows = [row for row in rows if row["cell"] == cell] + try: + analyze_cell(cell, cell_rows, args.units, args, dropped_flag) + except InputError as exc: + print(f"analyze_lattice: cell {cell}: {exc}", file=sys.stderr) + return 2 + + if dropped_flag: + print("\nunpaired keys dropped (invariant 4 evidence):") + for cell, record in dropped_flag: + print(f" {cell}: {format_coupling(record)}") + if args.strict_pairing: + return 1 + return 0 + + +if __name__ == "__main__": + sys.exit(main()) diff --git a/modules/home/ai/plugins/testing-and-quality/.apm/skills/harborize/scripts/audit_leakage.py b/modules/home/ai/plugins/testing-and-quality/.apm/skills/harborize/scripts/audit_leakage.py new file mode 100755 index 000000000..1898a4536 --- /dev/null +++ b/modules/home/ai/plugins/testing-and-quality/.apm/skills/harborize/scripts/audit_leakage.py @@ -0,0 +1,267 @@ +#!/usr/bin/env -S uv run --script +# /// script +# requires-python = ">=3.12" +# dependencies = [] +# /// +"""Non-triviality audit (algebraic invariant 3, square_S does not derive phi). + +Checks that the skills under audit cannot satisfy the task's verifier by +copying: + + 1. literals quoted expectation strings in the verifier's own sources that + appear verbatim somewhere in skill content; + 2. oracle token-Jaccard similarity above 0.6 between the oracle's + solve.sh and any script the skill bundles; + 3. identity a skill file byte-identical to a verifier file. + +Each check reports its own status, so a package whose oracle does not exist yet +records check 2 as skipped rather than passing it silently. + +Usage: + audit_leakage.py --task --skills

[ ...] + +Exit codes: + 0 every check that could run raised no flag + 1 at least one flag was raised; the review gate fails + 2 the audit could not run: bad invocation, missing task package, a + missing or SKILL.md-free skill directory, or no verifier sources + +Exit 2 is distinct from exit 0 deliberately. Every check searches for evidence +of leakage, so a mistyped path finds nothing and would otherwise be reported as +a clean package that was never examined. +""" + +import argparse +import hashlib +import json +import pathlib +import re +import sys + +TEXT_SUFFIXES = (".py", ".sh", ".md", ".toml", ".txt", ".json") +VERIFIER_DIRS = ("verifier", "tests") +ORACLE_DIRS = ("oracle", "solution") +MIN_LITERAL_LENGTH = 8 +JACCARD_THRESHOLD = 0.6 +CONTAINER_PATH_PREFIXES = ("/logs", "/tests", "/app", "/solution", "/harbor") + + +def tokens(text): + return set(re.findall(r"[A-Za-z0-9_./-]{4,}", text)) + + +def read_text_files(root, suffixes=TEXT_SUFFIXES): + for path in sorted(pathlib.Path(root).rglob("*")): + if path.is_file() and path.suffix in suffixes: + try: + yield path, path.read_text(errors="ignore") + except OSError: + continue + + +def sha256(path): + return hashlib.sha256(path.read_bytes()).hexdigest() + + +def verifier_dirs(task): + return [task / name for name in VERIFIER_DIRS if (task / name).is_dir()] + + +def oracle_scripts(task): + return [ + task / name / "solve.sh" + for name in ORACLE_DIRS + if (task / name / "solve.sh").is_file() + ] + + +def validate_inputs(task, skills): + """Return a list of fatal input problems (empty when the audit can run).""" + problems = [] + if not task.is_dir(): + problems.append(f"task package {task} is not a directory") + elif not verifier_dirs(task): + problems.append( + f"task package {task} has no verifier/ or tests/ directory; " + "there are no expectations to audit against" + ) + for skill in skills: + if not skill.is_dir(): + problems.append(f"skill path {skill} is not a directory") + elif not any(skill.rglob("SKILL.md")): + problems.append(f"skill path {skill} contains no SKILL.md at any depth") + return problems + + +def check_literals(task, skill_texts): + """Verifier expectation strings recoverable from skill content.""" + literals = set() + for directory in verifier_dirs(task): + for _, text in read_text_files(directory): + literals |= { + match + for match in re.findall( + rf"[\"']([^\"']{{{MIN_LITERAL_LENGTH},}})[\"']", text + ) + if not match.startswith(CONTAINER_PATH_PREFIXES) + } + flags = [ + f"verifier literal {literal!r} appears in {path}" + for literal in sorted(literals) + for path, text in skill_texts + if literal in text + ] + return { + "id": "literals", + "status": "ran", + "n_literals": len(literals), + "flags": flags, + } + + +def check_oracle_similarity(task, skill_texts): + """Oracle solve.sh token-similar to a script the skill ships.""" + scripts = oracle_scripts(task) + if not scripts: + return { + "id": "oracle", + "status": "skipped", + "reason": "no oracle/solve.sh or solution/solve.sh in the task package", + "flags": [], + } + flags = [] + for script in scripts: + oracle_tokens = tokens(script.read_text(errors="ignore")) + for path, text in skill_texts: + if path.suffix not in (".sh", ".py"): + continue + skill_tokens = tokens(text) + union = oracle_tokens | skill_tokens + jaccard = len(oracle_tokens & skill_tokens) / max(1, len(union)) + if jaccard > JACCARD_THRESHOLD: + flags.append( + f"{script} ~ {path} (Jaccard {jaccard:.2f}) - parameterize " + "task inputs or the skill IS the answer key" + ) + return { + "id": "oracle", + "status": "ran", + "n_oracles": len(scripts), + "flags": flags, + } + + +def check_byte_identity(task, skills): + """Skill files byte-identical to verifier files.""" + verifier_digests = {} + for directory in verifier_dirs(task): + for path in sorted(directory.rglob("*")): + if path.is_file(): + verifier_digests.setdefault(sha256(path), path) + flags = [] + for skill in skills: + for path in sorted(skill.rglob("*")): + if not path.is_file(): + continue + match = verifier_digests.get(sha256(path)) + if match is not None: + flags.append(f"{path} byte-identical to verifier file {match}") + return { + "id": "identity", + "status": "ran", + "n_verifier_files": len(verifier_digests), + "flags": flags, + } + + +def audit(task, skills): + skill_texts = [ + (path, text) for skill in skills for path, text in read_text_files(skill) + ] + return [ + check_literals(task, skill_texts), + check_oracle_similarity(task, skill_texts), + check_byte_identity(task, skills), + ] + + +def gate_line(checks): + flagged = sum(len(check["flags"]) for check in checks) + skipped = [check["id"] for check in checks if check["status"] == "skipped"] + verdict = "FAIL" if flagged else "pass" + detail = f"{flagged} flag(s)" if flagged else "no flags" + if skipped: + detail += f"; skipped: {','.join(skipped)}" + return f"invariant 3 non-triviality: {verdict} ({detail})" + + +def build_parser(): + parser = argparse.ArgumentParser( + description="Non-triviality audit for a harborize evaluation package.", + epilog="Exit codes: 0 clean, 1 flags raised, 2 audit could not run.", + ) + parser.add_argument( + "--task", + required=True, + type=pathlib.Path, + help="task package directory (holds verifier/ or tests/)", + ) + parser.add_argument( + "--skills", + required=True, + nargs="+", + type=pathlib.Path, + help="one directory per ablation unit; each must contain a SKILL.md", + ) + parser.add_argument( + "--json", + type=pathlib.Path, + help="write the machine-readable audit record here", + ) + return parser + + +def main(argv=None): + args = build_parser().parse_args(argv) + + problems = validate_inputs(args.task, args.skills) + if problems: + for problem in problems: + print(f"audit could not run: {problem}", file=sys.stderr) + return 2 + + checks = audit(args.task, args.skills) + flags = [flag for check in checks for flag in check["flags"]] + exit_code = 1 if flags else 0 + + if args.json: + args.json.parent.mkdir(parents=True, exist_ok=True) + args.json.write_text( + json.dumps( + { + "task": str(args.task), + "skills": [str(skill) for skill in args.skills], + "min_literal_length": MIN_LITERAL_LENGTH, + "jaccard_threshold": JACCARD_THRESHOLD, + "checks": checks, + "exit_code": exit_code, + "gate_line": gate_line(checks), + }, + indent=2, + ) + + "\n" + ) + + for check in checks: + if check["status"] == "skipped": + print(f"check {check['id']}: skipped - {check['reason']}") + if flags: + print("LEAKAGE FLAGS:") + for flag in flags: + print(" -", flag) + print(gate_line(checks)) + return exit_code + + +if __name__ == "__main__": + sys.exit(main()) diff --git a/modules/home/ai/plugins/testing-and-quality/.apm/skills/harborize/scripts/census.py b/modules/home/ai/plugins/testing-and-quality/.apm/skills/harborize/scripts/census.py new file mode 100755 index 000000000..cf059652e --- /dev/null +++ b/modules/home/ai/plugins/testing-and-quality/.apm/skills/harborize/scripts/census.py @@ -0,0 +1,344 @@ +#!/usr/bin/env -S uv run --script +# /// script +# requires-python = ">=3.12" +# dependencies = [] +# /// +"""Marketplace census: a full static pass over every skill in one tree. + +Emits a per-skill record set, a skill-to-plugin membership map, and +lexical-overlap rankings used to nominate candidate pairs for interference +conditions. + +The census runs over either of two subjects and reports which one it read. +The source tree, laid out as //.apm/skills//SKILL.md, is +the refactor subject. The deployed tree, a flat //SKILL.md, is +the evaluation subject; selection-competition figures must be computed from +it, since the deployed tree carries skills the source tree does not. + +A plugin is a derived aggregate unit: the union of its member skills' flat +directories, resolved through the membership map rather than through +directory structure, because the deployed tree has no plugin directories. +For a flat root, pass --membership-from pointing at a source census; skills +absent from it are recorded under the EXTERNAL sentinel rather than dropped +or guessed. + +Frontmatter is parsed without a YAML dependency so this script can run inside +a hermetic derivation. The parser covers plain, single- and double-quoted, +folded (>, >-, >+) and literal (|, |-, |+) scalars, and plain scalars +continued across indented lines. + +Usage: + census.py --root --out [--membership-from ] + [--layout auto|nested|flat] [--top-intra N] [--top-inter N] + [--timestamp ISO8601] +""" + +import argparse +import collections +import itertools +import json +import os +import pathlib +import re +import subprocess +import sys + +__version__ = "0.2.0" + +EXTERNAL = "" +"""Membership value for a skill in the censused tree that no plugin in the +reference census claims: a remote apm dependency, or a locally added skill.""" + +_BLOCK = re.compile(r"^([|>])[+\-\d]*\s*(?:#.*)?$") +_WORD = re.compile(r"[a-z]{3,}") +_DECIDABLE = re.compile( + r"\b(file|commit|build|test|yaml|json|config|flake|lint|check)\b", re.I) + + +def frontmatter(text): + """Return the raw YAML frontmatter block, or "" if there is none.""" + if not text.startswith("---") or text[3:4] not in ("\n", "\r"): + return "" + rest = text[3:] + end = re.search(r"^(---|\.\.\.)\s*$", rest, re.M) + return rest[:end.start()] if end else rest + + +def _unquote(s): + quote = s[0] + body = s[1:-1] if len(s) > 1 and s[-1] == quote else s[1:] + if quote == "'": + return body.replace("''", "'") + escapes = {"n": "\n", "t": "\t", "r": "\r", '"': '"', "\\": "\\"} + return re.sub(r"\\(.)", lambda m: escapes.get(m.group(1), m.group(1)), body) + + +def scalar(fm, key): + """Return the value of a top-level frontmatter key as flat text.""" + lines = fm.splitlines() + pattern = re.compile(rf"^{re.escape(key)}:(.*)$") + for i, line in enumerate(lines): + match = pattern.match(line) + if not match: + continue + head = match.group(1).strip() + block = _BLOCK.match(head) if head else None + if block: + return _block_scalar(lines[i + 1:], block.group(1)) + continuation = [] + for nxt in lines[i + 1:]: + if not nxt.strip() or not nxt[:1].isspace(): + break + continuation.append(nxt.strip()) + if head[:1] in ("'", '"'): + return _unquote(" ".join([head] + continuation)) + if not continuation: + head = re.sub(r"\s+#.*$", "", head) + return " ".join([head] + continuation).strip() + return "" + + +def _block_scalar(rest, style): + body = [] + for line in rest: + if line.strip() and not line[:1].isspace(): + break + body.append(line) + while body and not body[-1].strip(): + body.pop() + if not body: + return "" + indent = min(len(c) - len(c.lstrip()) for c in body if c.strip()) + body = [c[indent:] if len(c) >= indent else "" for c in body] + if style == "|": + return "\n".join(body) + folded, para = [], [] + for chunk in body: + if chunk.strip(): + para.append(chunk if chunk[:1].isspace() else chunk.strip()) + else: + folded.extend([" ".join(para), ""]) + para = [] + folded.append(" ".join(para)) + return "\n".join(folded).strip("\n") + + +def tokens(s): + return set(_WORD.findall(s.lower())) + + +def _skill_dirs(base): + if not base.is_dir(): + return [] + return [d for d in sorted(base.iterdir()) if (d / "SKILL.md").is_file()] + + +def discover(root, layout): + """Return (resolved_layout, [(skill_dir, plugin_name_or_None)]). + + Symlinked skill directories are followed and included; the link target is + recorded per skill so a self-referencing deployment is visible in the + output rather than silently counted as an ordinary directory. + """ + if layout == "auto": + layout = "flat" if _skill_dirs(root) else "nested" + if layout == "flat": + return layout, [(d, None) for d in _skill_dirs(root)] + found = [] + for plugin in sorted(d for d in root.iterdir() if d.is_dir()): + for base in (plugin / ".apm" / "skills", plugin / "skills", plugin): + dirs = _skill_dirs(base) + if dirs: + found.extend((d, plugin.name) for d in dirs) + break + return layout, found + + +def record(skill_dir, root): + text = (skill_dir / "SKILL.md").read_text(errors="ignore") + fm = frontmatter(text) + desc = scalar(fm, "description") + return { + "plugin": None, + "path": os.path.relpath(skill_dir, root), + "symlink": (os.readlink(skill_dir) if skill_dir.is_symlink() else None), + "lines": len(text.splitlines()), + "cmd_style": "disable-model-invocation: true" in fm, + "env_coupling": sorted(set(re.findall(r"~/[\w./-]+|/home/[\w./-]+", + text)))[:5], + "has_scripts": (skill_dir / "scripts").is_dir(), + "has_refs": (skill_dir / "references").is_dir(), + "description": desc, + "desc_toks": sorted(tokens(desc)), + "decidability_guess": ("decidable" if _DECIDABLE.search(desc) + else "subjective"), + } + + +def load_membership(path): + data = json.loads(pathlib.Path(path).read_text()) + known = {name: rec.get("plugin") for name, rec in data["skills"].items()} + return {k: v for k, v in known.items() if v and v != EXTERNAL} + + +def jaccard(a, b): + A, B = set(a), set(b) + return len(A & B) / max(1, len(A | B)) + + +def overlaps(skills, plugins, top_intra, top_inter): + """Rank skill pairs by description-token Jaccard and cut the top N. + + The cut is a nomination device for choosing which pairs to inspect and + which interference conditions to spend budget on. It is a ranking of + lexical similarity between two description fields. It is not a + significance threshold, carries no test, and a pair's presence or absence + is not evidence that the two skills do or do not interfere at run time. + """ + def toks(name): + return skills[name]["desc_toks"] + + intra = sorted( + ((jaccard(toks(a), toks(b)), p, a, b) + for p, members in plugins.items() + for a, b in itertools.combinations(members, 2)), + key=lambda r: (-r[0], r[1], r[2], r[3]))[:top_intra] + cross = (tuple(sorted((a, b))) for a, b in itertools.combinations(skills, 2) + if skills[a]["plugin"] != skills[b]["plugin"]) + inter = sorted( + ((jaccard(toks(a), toks(b)), skills[a]["plugin"], a, + skills[b]["plugin"], b) for a, b in cross), + key=lambda r: (-r[0], r[2], r[4]))[:top_inter] + return ( + [{"jaccard": round(j, 4), "plugin": p, "a": a, "b": b} + for j, p, a, b in intra], + [{"jaccard": round(j, 4), "plugin_a": pa, "a": a, + "plugin_b": pb, "b": b} for j, pa, a, pb, b in inter], + ) + + +def git_info(path): + def run(*args): + try: + proc = subprocess.run(["git", "-C", str(path), *args], + capture_output=True, text=True, check=True) + except (OSError, subprocess.CalledProcessError): + return None + return proc.stdout.strip() or None + return run("rev-parse", "HEAD"), run("log", "-1", "--format=%cI") + + +def build(root, layout, membership_from, top_intra, top_inter, timestamp): + layout, found = discover(root, layout) + if not found: + raise SystemExit(f"no SKILL.md found under {root} (layout={layout})") + counts = collections.Counter(d.name for d, _ in found) + dupes = sorted(n for n, c in counts.items() if c > 1) + if dupes: + raise SystemExit(f"duplicate skill names under {root}: {dupes}") + skills = {d.name: record(d, root) for d, _ in found} + membership = {d.name: p for d, p in found if p} + if layout == "flat": + if not membership_from: + print("warning: no --membership-from for a flat root; plugin " + f"membership resolves to {EXTERNAL} for every skill", + file=sys.stderr) + known = load_membership(membership_from) if membership_from else {} + membership = {name: known.get(name, EXTERNAL) for name in skills} + for name, plugin in membership.items(): + skills[name]["plugin"] = plugin + + plugins = {} + for name in sorted(skills): + plugins.setdefault(skills[name]["plugin"], []).append(name) + plugins = {p: plugins[p] for p in sorted(plugins)} + + intra, inter = overlaps(skills, plugins, top_intra, top_inter) + revision, revision_date = git_info(root) + return { + "provenance": { + "script_version": __version__, + "root": str(root), + "layout": layout, + "revision": revision, + "revision_date": revision_date, + "timestamp": timestamp or revision_date, + "membership_source": (str(pathlib.Path(membership_from).resolve()) + if membership_from else None), + "n_skills": len(skills), + "n_plugins": sum(1 for p in plugins if p != EXTERNAL), + "n_external": sum(1 for v in membership.values() if v == EXTERNAL), + "n_empty_desc": sum(1 for r in skills.values() + if not r["desc_toks"]), + "top_intra": top_intra, + "top_inter": top_inter, + }, + "plugins": plugins, + "membership": dict(sorted(membership.items())), + "skills": skills, + "overlap": {"intra": intra, "inter": inter}, + } + + +def report(census): + prov = census["provenance"] + skills, plugins = census["skills"], census["plugins"] + decidable = sum(1 for r in skills.values() + if r["decidability_guess"] == "decidable") + print(f"root={prov['root']} layout={prov['layout']} rev={prov['revision']}") + print(f"plugins={prov['n_plugins']} skills={prov['n_skills']} " + f"external={prov['n_external']} empty_desc={prov['n_empty_desc']} " + f"cmd_style={sum(1 for r in skills.values() if r['cmd_style'])} " + f"env_coupled={sum(1 for r in skills.values() if r['env_coupling'])} " + f"decidable~={decidable} subjective~={len(skills) - decidable}") + print("plugin sizes:", {p: len(s) for p, s in plugins.items()}) + print(f"\ntop {prov['top_intra']} intra-plugin overlap " + "(interference candidates within plugins):") + for r in census["overlap"]["intra"]: + print(f" {r['jaccard']:.2f} [{r['plugin']}] {r['a']} : {r['b']}") + print(f"\ntop {prov['top_inter']} inter-plugin overlap " + "(cross-plugin trigger competition):") + for r in census["overlap"]["inter"]: + print(f" {r['jaccard']:.2f} {r['plugin_a']}/{r['a']} : " + f"{r['plugin_b']}/{r['b']}") + + +def main(argv=None): + ap = argparse.ArgumentParser( + description="Static census of a skill tree.", + formatter_class=argparse.RawDescriptionHelpFormatter) + ap.add_argument("root_positional", nargs="?", metavar="ROOT", + help="deprecated positional form of --root") + ap.add_argument("--root", help="tree to census (source plugins dir or " + "deployed flat skills dir)") + ap.add_argument("--out", required=True, help="path to write census JSON") + ap.add_argument("--layout", default="auto", + choices=["auto", "nested", "flat"]) + ap.add_argument("--membership-from", metavar="CENSUS.JSON", + help="source census used to resolve skill-to-plugin " + "membership for a flat root") + ap.add_argument("--top-intra", type=int, default=8) + ap.add_argument("--top-inter", type=int, default=8) + ap.add_argument("--timestamp", help="ISO 8601 stamp for provenance; " + "defaults to the root repository's " + "HEAD committer date") + a = ap.parse_args(argv) + + root = a.root or a.root_positional + if not root: + ap.error("--root is required") + root = pathlib.Path(root).expanduser() + if not root.is_dir(): + ap.error(f"root is not a directory: {root}") + + census = build(root, a.layout, a.membership_from, a.top_intra, + a.top_inter, a.timestamp) + out = pathlib.Path(a.out).expanduser() + out.parent.mkdir(parents=True, exist_ok=True) + out.write_text(json.dumps(census, indent=1) + "\n") + report(census) + print(f"\nwrote {out}") + + +if __name__ == "__main__": + main() diff --git a/modules/home/ai/plugins/testing-and-quality/.apm/skills/harborize/scripts/collect_rewards.py b/modules/home/ai/plugins/testing-and-quality/.apm/skills/harborize/scripts/collect_rewards.py new file mode 100755 index 000000000..353df6d43 --- /dev/null +++ b/modules/home/ai/plugins/testing-and-quality/.apm/skills/harborize/scripts/collect_rewards.py @@ -0,0 +1,470 @@ +#!/usr/bin/env -S uv run --script +# /// script +# requires-python = ">=3.12" +# dependencies = [] +# /// +"""Collect Harbor and BenchFlow run output into the results.json analyze_lattice reads. + +Bridges the gap between running design/manifest.sh and feeding +scripts/analyze_lattice.py. Emits a JSON list of +{condition, cell, task, trial, reward} rows plus provenance fields the analyzer +ignores. + +Files and fields parsed, at the pinned revisions recorded below: + +Harbor (ac398bbda7c4c1073461797d3b95c2455cc671b5) + /__//result.json + is a serialized TrialResult (src/harbor/models/trial/result.py). Read + task_name, trial_name, started_at, verifier_result.rewards + (dict[str, float | int] | None, src/harbor/models/verifier/result.py) and + exception_info. Trial directories are the job directory's immediate + subdirectories (src/harbor/job.py:252 iterates them the same way); the + job's own result.json is a JobResult and is not read. + + The injection check reads the sibling lock.json + (src/harbor/models/trial/paths.py:182-183), whose top-level `skills` list + records the name, source and content digest of every skill the HOST + resolved (src/harbor/models/job/lock.py:141-146, :462-475). + + That is resolution, not delivery. The lock is written in Trial.__init__ + (src/harbor/trial/trial.py:104), before _resolve_injected_skills at :107 + and long before _upload_injected_skills at :411, and + _build_agent_skill_locks calls only host-side functions. So a populated + lock proves the paths resolved and pins their digests, and proves nothing + about what reached the container: it stays fully populated through an + upload failure, a permissions failure, or an adapter that reads the + injected directory not at all. That last class is refused before the run + by design_matrix.check_harbor_agents, because no artifact written after + the run distinguishes it. config.agent.skills in result.json is the + request as written and is used only when a trial wrote no lock. + +BenchFlow (d30527b82027a416e72014920cdf43a534967ad3) + /__/...///result.json + is the dict written at src/benchflow/rollout/_results.py:387-431. The + elided levels are whatever the manifest puts under the per-condition + directory, such as design_matrix.py's trial-NN outer loop. Read + task_name, rollout_name, rewards, error, verifier_error, + partial_trajectory and started_at. The injection check requires both + skill_mode == "with-skill" and a non-null effective_skills_dir, which + skill_policy.config_metadata fills with the resolved host directory + (src/benchflow/skill_policy.py:60-68); skill_mode alone records the + request. Rollout directories are located the way + src/benchflow/eval_lift.py:277-291 does it: the given directory if it + holds a result.json, otherwise every directory that holds one anywhere + beneath it. + +BenchFlow has no run-id flag, so condition identity has to travel through +--jobs-dir and each (cell, condition) needs its own jobs directory named for +it. `bench eval run`'s options are declared at src/benchflow/cli/main.py:193-592, +except those sharing an Annotated alias in src/benchflow/cli/_options.py:16-32 +(--model at main.py:270 and --skill-mode at main.py:438 among them), which do +not appear literally inside the command body. + +Condition identity is recovered by splitting a job directory name on its last +"__" and requiring the suffix to be a key of conditions.json. An unrecognized +suffix is an error rather than a guess. --job-index bypasses the encoding +entirely for names the split cannot resolve. + +Trial ordinals are assigned here, by sorting each (cell, condition, task) +group on (started_at, name). Neither runner exposes or accepts a per-trial seed +at these revisions, so ordinals align replicate k with replicate k; they do not +recover a shared randomness object. analyze_lattice.py states what that means +for invariant 4. + +Reward reduction: a single-key reward dict is used as-is. A multi-key dict +requires --reward-key, because Harbor computes pass@k only when every trial +carries exactly one reward key valued 0 or 1 +(src/harbor/utils/pass_at_k.py:32-53), so which key was chosen changes what the +run supports. + +Usage: + collect_rewards.py --conditions design/conditions.json \\ + [--harbor-jobs DIR] [--benchflow-jobs DIR] [--job-index FILE] \\ + [--reward-key KEY] [--errors-as drop|zero] --out results.json + +Exit codes: + 0 results written + 1 no rollouts were found under any of the given roots + 2 input refused: unresolvable condition, ambiguous reward dict, + unscored rollouts with no --errors-as choice, or a failed + injection check +""" + +import argparse +import collections +import json +import pathlib +import sys + +EMPTY_CONDITION = "none" + + +class CollectError(Exception): + """Collection cannot proceed as specified.""" + + +def read_json(path): + try: + return json.loads(path.read_text()) + except (OSError, json.JSONDecodeError) as exc: + raise CollectError(f"cannot read {path}: {exc}") from exc + + +def load_conditions(path): + data = read_json(pathlib.Path(path)) + if not isinstance(data, dict): + raise CollectError(f"{path} is not a condition-id to unit-list object") + return data + + +def split_job_name(name, condition_ids): + head, sep, tail = name.rpartition("__") + if not sep or tail not in condition_ids: + raise CollectError( + f"job directory {name!r} does not end in a known condition id. " + f"Known ids: {', '.join(sorted(condition_ids))}. " + "Rename the job, or map it explicitly with --job-index." + ) + return head, tail + + +def discover_jobs(root, runner, condition_ids): + root = pathlib.Path(root) + if not root.is_dir(): + raise CollectError(f"{runner} jobs root {root} is not a directory") + jobs = [] + for path in sorted(root.iterdir()): + if not path.is_dir() or path.name.startswith("."): + continue + cell, condition = split_job_name(path.name, condition_ids) + jobs.append( + {"runner": runner, "cell": cell, "condition": condition, "path": path} + ) + return jobs + + +def load_job_index(path, condition_ids): + data = read_json(pathlib.Path(path)) + entries = data.get("jobs") if isinstance(data, dict) else data + if not isinstance(entries, list): + raise CollectError(f"{path} has no 'jobs' list") + jobs = [] + for entry in entries: + missing = [ + key + for key in ("runner", "cell", "condition", "path") + if key not in entry + ] + if missing: + raise CollectError(f"job-index entry {entry} is missing {missing}") + if entry["runner"] not in ("harbor", "benchflow"): + raise CollectError(f"unknown runner {entry['runner']!r} in {path}") + if entry["condition"] not in condition_ids: + raise CollectError( + f"job-index condition {entry['condition']!r} is not in conditions.json" + ) + jobs.append({**entry, "path": pathlib.Path(entry["path"])}) + return jobs + + +def harbor_rollout_dirs(job_dir): + return [ + path + for path in sorted(job_dir.iterdir()) + if path.is_dir() + and not path.name.startswith(".") + and (path / "result.json").is_file() + ] + + +def benchflow_rollout_dirs(job_dir): + if (job_dir / "result.json").is_file(): + return [job_dir] + return sorted({path.parent for path in job_dir.rglob("result.json")}) + + +def harbor_lock_skills(trial_dir): + """Host-resolved skill records from a trial's lock, or None if unlocked.""" + lock_path = trial_dir / "lock.json" + if not lock_path.is_file(): + return None + skills = read_json(lock_path).get("skills") + return skills if isinstance(skills, list) else [] + + +def harbor_injection(result, trial_dir): + locked = harbor_lock_skills(trial_dir) + if locked is None: + requested = (result.get("config") or {}).get("agent", {}).get("skills") or [] + return bool(requested), f"no lock.json; config.agent.skills={requested}" + pinned = [ + f"{skill.get('name')}@{str(skill.get('digest'))[:12]}" + for skill in locked + if isinstance(skill, dict) + ] + return bool(pinned), f"lock.skills={pinned}" + + +def parse_harbor_result(result, path): + if "trial_name" not in result: + raise CollectError( + f"{path} has no trial_name; it does not look like a Harbor TrialResult" + ) + if result.get("verifier_result") is None and result.get("step_results"): + raise CollectError( + f"{path} is a multi-step trial; per-step rewards are out of scope" + ) + verifier = result.get("verifier_result") or {} + injected, evidence = harbor_injection(result, path.parent) + return { + "name": result["trial_name"], + "task": result.get("task_name"), + "rewards": verifier.get("rewards"), + "started_at": result.get("started_at") or "", + "error": (result.get("exception_info") or {}).get("exception_type"), + "injected": injected, + "injection_evidence": evidence, + } + + +def parse_benchflow_result(result, path): + if "rollout_name" not in result: + raise CollectError( + f"{path} has no rollout_name; it does not look like a BenchFlow rollout" + ) + error = ( + result.get("error") + or result.get("verifier_error") + or result.get("export_error") + ) + if result.get("partial_trajectory") is True and not error: + error = "partial_trajectory" + mode = result.get("skill_mode") + effective = result.get("effective_skills_dir") + return { + "name": result["rollout_name"], + "task": result.get("task_name"), + "rewards": result.get("rewards"), + "started_at": result.get("started_at") or "", + "error": error, + "injected": mode == "with-skill" and effective is not None, + "injection_evidence": ( + f"skill_mode={mode!r} effective_skills_dir={effective!r}" + ), + } + + +PARSERS = {"harbor": parse_harbor_result, "benchflow": parse_benchflow_result} +ROLLOUT_DIRS = {"harbor": harbor_rollout_dirs, "benchflow": benchflow_rollout_dirs} + + +def reduce_reward(rewards, reward_key, path): + if not rewards: + return None + if reward_key is not None: + if reward_key not in rewards: + raise CollectError( + f"{path} has no reward key {reward_key!r}; " + f"present: {', '.join(sorted(rewards))}" + ) + return float(rewards[reward_key]) + if len(rewards) == 1: + return float(next(iter(rewards.values()))) + raise CollectError( + f"{path} carries {len(rewards)} reward keys " + f"({', '.join(sorted(rewards))}); pass --reward-key to choose one. " + "Harbor computes pass@k only for a single 0/1 reward key, so the " + "choice changes what the run supports." + ) + + +def check_injection(job, rollout, path): + """Flag a rollout whose recorded skill request disagrees with its condition. + + One-sided. A disagreement proves the lattice did not vary as designed; + agreement proves only that the request and the host-side resolution + matched, which is all either runner records post hoc on the Harbor arm. + """ + expected = job["condition"] != EMPTY_CONDITION + if rollout["injected"] == expected: + return None + verb = "resolved no skills" if expected else "resolved skills" + return ( + f"{path}: condition {job['condition']!r} {verb} " + f"({rollout['injection_evidence']})" + ) + + +def collect(jobs, reward_key, verify_injection): + scored, unscored, injection_failures = [], [], [] + for job in jobs: + if not job["path"].is_dir(): + raise CollectError(f"job path {job['path']} is not a directory") + for rollout_dir in ROLLOUT_DIRS[job["runner"]](job["path"]): + path = rollout_dir / "result.json" + rollout = PARSERS[job["runner"]](read_json(path), path) + if rollout["task"] is None: + raise CollectError(f"{path} has no task name") + if verify_injection: + failure = check_injection(job, rollout, path) + if failure: + injection_failures.append(failure) + record = { + "condition": job["condition"], + "cell": job["cell"], + "task": rollout["task"], + "runner": job["runner"], + "job": job["path"].name, + "rollout": rollout["name"], + "started_at": rollout["started_at"], + "source": str(path), + } + reward = reduce_reward(rollout["rewards"], reward_key, path) + if reward is None or rollout["error"]: + unscored.append({**record, "reason": rollout["error"] or "no rewards"}) + else: + scored.append({**record, "reward": reward}) + return scored, unscored, injection_failures + + +def assign_trial_ordinals(rows): + groups = collections.defaultdict(list) + for row in rows: + groups[(row["cell"], row["condition"], row["task"])].append(row) + for group in groups.values(): + group.sort(key=lambda row: (row["started_at"], row["rollout"], row["source"])) + for ordinal, row in enumerate(group): + row["trial"] = ordinal + return sorted( + rows, key=lambda row: (row["cell"], row["condition"], row["task"], row["trial"]) + ) + + +def build_parser(): + parser = argparse.ArgumentParser( + description="Collect Harbor and BenchFlow run output into results.json.", + epilog="Exit codes: 0 results written, 1 nothing found, 2 input refused.", + ) + parser.add_argument( + "--conditions", + required=True, + help="design/conditions.json emitted by design_matrix.py", + ) + parser.add_argument("--harbor-jobs", help="Harbor --jobs-dir root to scan") + parser.add_argument( + "--benchflow-jobs", help="root holding one BenchFlow --jobs-dir per condition" + ) + parser.add_argument( + "--job-index", + help="JSON with a 'jobs' list of {runner, cell, condition, path}, " + "bypassing the job-name encoding", + ) + parser.add_argument( + "--reward-key", help="reward dict key to use when a rollout carries several" + ) + parser.add_argument( + "--errors-as", + choices=["drop", "zero"], + help="how to treat unscored rollouts; required when any are present", + ) + parser.add_argument( + "--no-verify-injection", + action="store_true", + help="skip the check that each rollout's recorded skill injection " + "matches its condition", + ) + parser.add_argument("--out", required=True, help="results.json to write") + parser.add_argument( + "--errors-out", + help="unscored-rollout record (default: .errors.json)", + ) + return parser + + +def main(argv=None): + args = build_parser().parse_args(argv) + if not (args.harbor_jobs or args.benchflow_jobs or args.job_index): + print( + "collect_rewards: give at least one of --harbor-jobs, " + "--benchflow-jobs, --job-index", + file=sys.stderr, + ) + return 2 + + try: + conditions = load_conditions(args.conditions) + jobs = [] + if args.harbor_jobs: + jobs += discover_jobs(args.harbor_jobs, "harbor", set(conditions)) + if args.benchflow_jobs: + jobs += discover_jobs(args.benchflow_jobs, "benchflow", set(conditions)) + if args.job_index: + jobs += load_job_index(args.job_index, set(conditions)) + scored, unscored, injection_failures = collect( + jobs, args.reward_key, not args.no_verify_injection + ) + except CollectError as exc: + print(f"collect_rewards: {exc}", file=sys.stderr) + return 2 + + if injection_failures: + print( + "collect_rewards: skill injection does not match the condition " + "for these rollouts, so the lattice did not vary as designed:", + file=sys.stderr, + ) + for failure in injection_failures: + print(f" {failure}", file=sys.stderr) + return 2 + + if unscored and args.errors_as is None: + counts = collections.Counter( + (row["cell"], row["condition"], row["reason"]) for row in unscored + ) + print( + f"collect_rewards: {len(unscored)} rollouts carry no usable reward. " + "Choose --errors-as drop (exclude them, as BenchFlow's own lift " + "report does) or --errors-as zero (score them 0, as Harbor's " + "pass@k does). Counts by (cell, condition, reason):", + file=sys.stderr, + ) + for (cell, condition, reason), count in sorted(counts.items()): + print(f" {cell} {condition} {reason}: {count}", file=sys.stderr) + return 2 + + rows = list(scored) + if args.errors_as == "zero": + rows += [ + {key: value for key, value in row.items() if key != "reason"} + | {"reward": 0.0} + for row in unscored + ] + if not rows: + print("collect_rewards: no scored rollouts found", file=sys.stderr) + return 1 + + rows = assign_trial_ordinals(rows) + out = pathlib.Path(args.out) + out.parent.mkdir(parents=True, exist_ok=True) + out.write_text(json.dumps(rows, indent=2) + "\n") + + errors_out = pathlib.Path( + args.errors_out or out.parent / f"{out.stem}.errors.json" + ) + errors_out.write_text( + json.dumps( + {"errors_as": args.errors_as, "unscored": unscored}, indent=2 + ) + + "\n" + ) + + cells = sorted({row["cell"] for row in rows}) + print( + f"rows={len(rows)} jobs={len(jobs)} cells={len(cells)} " + f"unscored={len(unscored)} -> {out} (unscored record: {errors_out})" + ) + return 0 + + +if __name__ == "__main__": + sys.exit(main()) diff --git a/modules/home/ai/plugins/testing-and-quality/.apm/skills/harborize/scripts/design_matrix.py b/modules/home/ai/plugins/testing-and-quality/.apm/skills/harborize/scripts/design_matrix.py new file mode 100755 index 000000000..50c5d7439 --- /dev/null +++ b/modules/home/ai/plugins/testing-and-quality/.apm/skills/harborize/scripts/design_matrix.py @@ -0,0 +1,474 @@ +#!/usr/bin/env -S uv run --script +# /// script +# requires-python = ">=3.12" +# dependencies = [] +# /// +"""Generate the condition lattice design and run manifest. + +Usage: + design_matrix.py --units skillA skillB skillC \ + --design marginals|pairs|foldover|full \ + [--pairs skillA:skillB ...] [--from-census census.json --top-pairs 3] \ + [--everything-on] [-k 3] --cells cells.json --out design/ + +cells.json: [{"name": "claude-code+opus-5", "runner": "harbor", + "agent": "claude-code", "model": "anthropic/claude-opus-5", + "concurrency": 2, "env": {"CLAUDE_FORCE_OAUTH": "1"}}, + {"name": "codex+luna", "runner": "benchflow", + "agent": "codex", "model": "gpt-5.6-luna"}, ...] + +`runner` is "harbor" or "benchflow". `concurrency` and `env` are copied into +the manifest verbatim; this script knows nothing about authentication pools, +and SKILL.md Phase 5 governs which cells may appear in a reported run batch. + +Emits: design/conditions.json (condition id -> sorted unit list) + design/manifest.sh (one runnable block per condition x cell) + design/jobs.json ({runner, cell, condition, path} per job, the + shape collect_rewards.py --job-index consumes) + design/selection.json (pair ranking provenance, with --from-census) + +Job directories are laid out per runner under --jobs-root: harbor at +/harbor/__, bench at /bench/__/trial-NN. +Harbor's -o/--jobs-dir is emitted explicitly rather than left to its configured +default, so the manifest and design/jobs.json agree on where output landed. +Paths in design/jobs.json are written exactly as the manifest spells them, so +they resolve against the directory the manifest and the collector are run from. + +Injection is the single mechanism that varies between conditions. Harbor takes +`--skill `; `--ak skills_dir=` is a container-side path that fails +silently against a host one. BenchFlow takes `--skills-dir ` under +`--skill-mode with-skill`, and `--skill-mode no-skill` alone for C = empty. +See references/emitters.md for the contract and the upstream anchors. + +A cells.json naming a Harbor adapter that does not read the injected directory +is refused here rather than run: that failure is invisible at every later +stage, so the allowlist below is the only gate that catches it, and it costs +nothing because it runs before any container starts. + +Invariant (grade discipline): conditions differ only in dir(C); the manifest +varies only the injected directory and the ablation id. +""" + +from __future__ import annotations + +import argparse +import itertools +import json +import pathlib +import shlex + +MATERIALIZE = pathlib.Path(__file__).resolve().parent / "materialize_conditions.py" + + +def conditions(units, design, pairs, everything_on): + """Condition sets for a design. + + `foldover` is the empty set, every singleton, every singleton's complement, + and the full set, giving |C| = 2n + 2. It is not a fractional factorial: it + carries no defining relation and no resolution claim. At n = 3 it coincides + with the full factorial. At n >= 4 it contains no two-element subsets, so + analyze_lattice.py can compute no second difference from it — {u}, {v} and + {u,v} must all be present for that. Use `pairs` or `full` when interactions + are the question. + """ + base = [frozenset()] + [frozenset([u]) for u in units] + if design == "marginals": + out = base + elif design == "pairs": + out = base + [frozenset(p) for p in pairs] + elif design == "full": + out = [frozenset(c) for r in range(len(units) + 1) + for c in itertools.combinations(units, r)] + elif design == "foldover": + allu = frozenset(units) + out = base + [allu - frozenset([u]) for u in units] + [allu] + else: + raise SystemExit(f"unknown design {design}") + if everything_on: + out.append(frozenset(units)) + seen, uniq = set(), [] + for c in out: + if c not in seen: + seen.add(c) + uniq.append(c) + return uniq + + +def cid(c): + return "none" if not c else "+".join(sorted(c)) + + +def cond_dir(ident): + """dir(C) for a condition id. + + Unit names reach here verbatim from --units, and the census's own + `` sentinel contains shell redirection metacharacters, so every + interpolation of this path into the manifest is quoted. + """ + return f"/tmp/cond-{ident}" + + +RUNNERS = ("harbor", "benchflow") + +# Registered Harbor agents whose adapter reads the injected skills directory +# and registers it into a discovery path. Regenerate at a new harbor revision +# by intersecting the factory registry with the adapters that read the dir: +# rg -n 'AgentName\.[A-Z_0-9]+: ' src/harbor/agents/factory.py +# rg -l 'self\.skills_dir' src/harbor/agents/ --type py +# The second command yields module paths; map each back to its registry name +# through factory.py and drop agents/base.py, which is the definition site. +# Derived at harbor ac398bbda7c4c1073461797d3b95c2455cc671b5: 39 registered, +# 22 consuming, 17 that accept the upload and never register it. +HARBOR_SKILL_CONSUMING_AGENTS = frozenset({ + "antigravity-cli", "antigravity-sdk", "claude-code", "cline-cli", "codex", + "copilot-cli", "cursor-cli", "eve", "fx", "gemini-cli", "goose", + "grok-build", "hermes", "kimi-cli", "kimi-code", "mimo", "openclaw", + "opencode", "pi", "qwen-coder", "terminus-2", "vibe", +}) + +# Harbor routes every agent name carrying this prefix through AgentName.ACP +# (src/harbor/agents/factory.py:167-175), and acp is one of the 17. +ACP_SHORTHAND_PREFIX = "acp:" + + +def check_harbor_agents(cells, path): + """Refuse a cell whose Harbor adapter would drop the injected skills. + + Harbor uploads the skills per trial regardless of adapter, and a + non-consuming adapter never registers them into a discovery path. Nothing + raises, the run exits 0, and every condition in that cell collapses to the + empty condition. + + lock.json cannot detect this. The lock is written in Trial.__init__ (harbor + src/harbor/trial/trial.py:104), before the skills are resolved at :107 and + long before they are uploaded at :411, and _build_agent_skill_locks + (src/harbor/models/job/lock.py:462-475) calls only host-side functions, so + a cell that delivers nothing still writes a fully populated lock. Nothing + written after the run separates this failure from a real negative result + either, so refusing here, before any container starts, is the only cheap + place it can be caught. + """ + problems = [] + for i, cell in enumerate(cells): + if cell["runner"] != "harbor": + continue + agent = str(cell["agent"]) + if agent.startswith(ACP_SHORTHAND_PREFIX): + problems.append( + f"entry {i} ({cell['name']!r}) names harbor agent {agent!r}: " + "harbor routes every acp: shorthand through AgentName.ACP, " + "which does not read the injected skills directory, so this " + "cell drops skills on the harbor arm while a benchflow arm " + "for the same agent works" + ) + elif agent not in HARBOR_SKILL_CONSUMING_AGENTS: + problems.append( + f"entry {i} ({cell['name']!r}) names harbor agent {agent!r}, " + "whose adapter does not read the injected skills directory: " + "harbor uploads the skills and nothing registers them, with " + "no error and no log line, so every condition in this cell " + "collapses to the empty condition" + ) + if problems: + raise SystemExit( + f"cells {path}: refusing to emit a manifest.\n" + + "\n".join(f" {problem}" for problem in problems) + + "\nAdapters that consume injected skills at harbor " + "ac398bbda7c4c1073461797d3b95c2455cc671b5: " + + ", ".join(sorted(HARBOR_SKILL_CONSUMING_AGENTS)) + ) + + +def load_cells(path): + """Read cells.json, refusing a shape design/jobs.json could not describe. + + `runner` is restricted to the two spellings collect_rewards.py accepts, so + design/jobs.json can be handed to it as --job-index without translation. + The adapter gate in check_harbor_agents runs from here rather than from + main, so no caller can reach a manifest without passing it. + """ + cells = json.loads(pathlib.Path(path).read_text()) + if not isinstance(cells, list) or not cells: + raise SystemExit(f"cells {path}: expected a non-empty JSON list") + names = set() + for i, cell in enumerate(cells): + if not isinstance(cell, dict): + raise SystemExit(f"cells {path}: entry {i} is not an object") + missing = [k for k in ("name", "runner", "agent", "model") if k not in cell] + if missing: + raise SystemExit( + f"cells {path}: entry {i} is missing {', '.join(missing)}" + ) + if cell["runner"] not in RUNNERS: + raise SystemExit( + f"cells {path}: entry {i} has runner {cell['runner']!r}; " + f"expected one of {', '.join(RUNNERS)}" + ) + if cell["name"] in names: + raise SystemExit(f"cells {path}: duplicate cell name {cell['name']!r}") + names.add(cell["name"]) + check_harbor_agents(cells, path) + return cells + + +def load_census(path): + """Read one census document: census.py writes one tree per --root/--out. + + Pass the DEPLOYED tree's census. It is the evaluation subject, and ranking + over the source tree understates selection competition because the deployed + tree carries skills the source tree does not. + """ + data = json.loads(pathlib.Path(path).read_text()) + if not isinstance(data, dict) or not isinstance(data.get("skills"), dict): + seen = (", ".join(sorted(map(str, data)))[:120] + if isinstance(data, dict) else type(data).__name__) + raise SystemExit( + f"census {path}: expected a census document with a 'skills' " + f"mapping (top-level keys: {seen})" + ) + return data + + +def unit_tokens(unit, census): + """Description-token set for a unit, unioning members for a plugin unit. + + Mirrors materialize_conditions.resolve_unit, including its refusal of a + bare name that is both a skill and a plugin, so pair selection and + materialization cannot disagree about what a unit denotes. + """ + skills = census["skills"] + kind, sep, bare = unit.partition(":") + if not sep or kind not in ("skill", "plugin"): + kind, bare = "", unit + + def skill_toks(name): + return set(skills.get(name, {}).get("desc_toks") or []) + + as_skill = isinstance(skills.get(bare), dict) + members = sorted( + name for name, rec in skills.items() + if isinstance(rec, dict) and rec.get("plugin") == bare + ) + if not members: + listed = (census.get("plugins") or {}).get(bare) + if isinstance(listed, list): + members = sorted(m for m in listed if isinstance(m, str)) + + if kind == "skill": + if not as_skill: + raise SystemExit(f"unit {unit}: no such skill in the census") + return skill_toks(bare) + if kind == "plugin": + if not members: + raise SystemExit(f"unit {unit}: no such plugin in the census") + return set().union(*(skill_toks(m) for m in members)) + if as_skill and members: + raise SystemExit( + f"unit {unit} names both a skill and a plugin in the census; " + f"disambiguate as skill:{bare} or plugin:{bare}" + ) + if as_skill: + return skill_toks(bare) + if members: + return set().union(*(skill_toks(m) for m in members)) + raise SystemExit( + f"unit {unit}: absent from the census as a skill and as a plugin" + ) + + +def parse_pair(spec, units): + """Split a --pairs spec into two declared units. + + A unit name may itself contain a colon (`plugin:`), so the separator + is the colon whose two sides are both declared in --units. + """ + known = set(units) + hits = [ + (spec[:i], spec[i + 1:]) + for i, ch in enumerate(spec) + if ch == ":" and spec[:i] in known and spec[i + 1:] in known + ] + if len(hits) == 1: + return hits[0] + if not hits: + raise SystemExit( + f"--pairs {spec!r}: expected : with both sides declared " + f"in --units" + ) + raise SystemExit(f"--pairs {spec!r}: ambiguous split, matches {hits}") + + +def rank_pairs(units, census): + """Unit pairs by descending description-token Jaccard, name-tie-broken.""" + toks = {u: unit_tokens(u, census) for u in units} + scored = [] + for a, b in itertools.combinations(sorted(units), 2): + union = toks[a] | toks[b] + scored.append((len(toks[a] & toks[b]) / len(union) if union else 0.0, a, b)) + return sorted(scored, key=lambda s: (-s[0], s[1], s[2])) + + +def harbor_line(env, cell, task_dir, k, ident, has_skills, jobs_dir): + skill = ( + f"--skill {shlex.quote(cond_dir(ident))} " if has_skills else "" + ) + job_name = f"{cell['name']}__{ident}" + return ( + f"{env}harbor run -p {shlex.quote(task_dir)} -a {cell['agent']} " + f"-m {cell['model']} -k {k} " + f"--n-concurrent {cell.get('concurrency', 2)} {skill}" + f"-o {shlex.quote(jobs_dir)} --job-name {shlex.quote(job_name)}" + ) + + +def bench_block(env, cell, task_dir, k, ident, has_skills, jobs_dir): + mode = ( + f"--skill-mode with-skill --skills-dir {shlex.quote(cond_dir(ident))}" + if has_skills else "--skill-mode no-skill" + ) + jobs = shlex.quote(f"{jobs_dir}/{cell['name']}__{ident}") + return [ + f"for t in $(seq -w 1 {k}); do", + f" {env}bench eval run --tasks-dir {shlex.quote(task_dir)} " + f"--agent {cell['agent']} --model {cell['model']} " + f"{mode} --sandbox docker " + f"--concurrency {cell.get('concurrency', 2)} " + f'--jobs-dir {jobs}/trial-"$t"', + "done", + ] + + +def runner_jobs_dir(jobs_root, runner): + return f"{jobs_root}/{'harbor' if runner == 'harbor' else 'bench'}" + + +def build_manifest(conds, cells, a, membership): + """Manifest lines plus the job index describing where each job will land. + + Harbor lines carry HARBOR_TELEMETRY=0 unless the cell sets it. Harbor's + telemetry derives uses_skills from the requested skill list + (src/harbor/telemetry.py:239) without consulting the effective directory or + the adapter, so it reports a run that delivered nothing as skill-bearing; + the manifest declines to publish that classification rather than correct it. + """ + mem = f" --membership {shlex.quote(membership)}" if membership else "" + lines = [ + "#!/bin/bash", + "set -euo pipefail", + f"# {len(conds)} conditions x {len(cells)} cells x k={a.k} = " + f"{len(conds) * len(cells) * a.k} runs", + "# bench k is an outer loop: --trials is inert without --matrix.", + ] + jobs = [] + for cell in cells: + cell_env = cell.get("env", {}) + if cell["runner"] == "harbor": + cell_env = {"HARBOR_TELEMETRY": "0", **cell_env} + env = "".join(f"{k}={shlex.quote(v)} " for k, v in cell_env.items()) + jobs_dir = runner_jobs_dir(a.jobs_root, cell["runner"]) + for c in conds: + ident = cid(c) + lines.append( + f"uv run --script {shlex.quote(str(MATERIALIZE))} " + f"--skills-root {shlex.quote(a.skills_root)} " + f"--units {' '.join(shlex.quote(u) for u in sorted(c))} " + f"--dest {shlex.quote(cond_dir(ident))}{mem}" + if c else "true # empty condition: nothing to materialize" + ) + if cell["runner"] == "harbor": + lines.append( + harbor_line(env, cell, a.task_dir, a.k, ident, bool(c), + jobs_dir) + ) + else: + lines.extend( + bench_block(env, cell, a.task_dir, a.k, ident, bool(c), + jobs_dir) + ) + jobs.append({ + "runner": cell["runner"], + "cell": cell["name"], + "condition": ident, + "path": f"{jobs_dir}/{cell['name']}__{ident}", + }) + return lines, jobs + + +def main(): + ap = argparse.ArgumentParser() + ap.add_argument("--units", nargs="+", required=True) + ap.add_argument("--design", required=True, + choices=["marginals", "pairs", "foldover", "full"]) + ap.add_argument("--pairs", nargs="*", default=[], + help="colon-separated unit pairs for --design pairs") + ap.add_argument("--from-census", + help="census.json for the DEPLOYED tree: ranks pairs for " + "--top-pairs and is passed to " + "materialize_conditions.py as --membership, which " + "plugin units over a flat tree require") + ap.add_argument("--top-pairs", type=int, default=0, + help="add the N highest-overlap unit pairs, on top of any " + "--pairs the user named") + ap.add_argument("--everything-on", action="store_true") + ap.add_argument("-k", type=int, default=3) + ap.add_argument("--task-dir", default="") + ap.add_argument("--skills-root", default="", + help="dir containing one folder per unit") + ap.add_argument("--jobs-root", default="runs", + help="job-output prefix; harbor lands under " + "/harbor and bench under /bench, which " + "collect_rewards.py reads as --harbor-jobs and " + "--benchflow-jobs") + ap.add_argument("--cells", required=True) + ap.add_argument("--out", default="design") + a = ap.parse_args() + + if a.top_pairs and not a.from_census: + raise SystemExit("--top-pairs requires --from-census") + + pairs = [parse_pair(p, a.units) for p in a.pairs] + + ranking = census = None + if a.from_census: + census = load_census(a.from_census) + ranking = rank_pairs(a.units, census) + for _, x, y in ranking[:a.top_pairs]: + if (x, y) not in pairs and (y, x) not in pairs: + pairs.append((x, y)) + + conds = conditions(a.units, a.design, pairs, a.everything_on) + cells = load_cells(a.cells) + + out = pathlib.Path(a.out) + out.mkdir(parents=True, exist_ok=True) + (out / "conditions.json").write_text(json.dumps( + {cid(c): sorted(c) for c in conds}, indent=2) + "\n") + if ranking is not None: + (out / "selection.json").write_text(json.dumps({ + "census": str(pathlib.Path(a.from_census).resolve()), + "census_provenance": census.get("provenance"), + "top_pairs": a.top_pairs, + "ranking": [ + {"jaccard": round(j, 4), "pair": [x, y]} for j, x, y in ranking + ], + "selected": [list(p) for p in pairs], + }, indent=2) + "\n") + + membership = ( + str(pathlib.Path(a.from_census).resolve()) if a.from_census else None + ) + lines, jobs = build_manifest(conds, cells, a, membership) + manifest = out / "manifest.sh" + manifest.write_text("\n".join(lines) + "\n") + manifest.chmod(0o755) + (out / "jobs.json").write_text( + json.dumps({"jobs_root": a.jobs_root, "jobs": jobs}, indent=2) + "\n" + ) + print( + f"{len(conds)} conditions, {len(jobs)} jobs -> {out}/conditions.json, " + f"{manifest}, {out}/jobs.json" + ) + + +if __name__ == "__main__": + main() diff --git a/modules/home/ai/plugins/testing-and-quality/.apm/skills/harborize/scripts/materialize_conditions.py b/modules/home/ai/plugins/testing-and-quality/.apm/skills/harborize/scripts/materialize_conditions.py new file mode 100755 index 000000000..55661f6e0 --- /dev/null +++ b/modules/home/ai/plugins/testing-and-quality/.apm/skills/harborize/scripts/materialize_conditions.py @@ -0,0 +1,261 @@ +#!/usr/bin/env -S uv run --script +# /// script +# requires-python = ">=3.12" +# dependencies = [] +# /// +"""Materialize dir(C): copy exactly the unit folders in C into --dest. + +The only sanctioned mechanism for varying conditions (grade discipline). + +A unit is a skill or a plugin. A skill unit is one folder holding a SKILL.md. +A plugin unit is derived: the union of its member skills' folders, resolved +through the skill-name to plugin-name membership map that census.py emits. The +derivation is required because the deployed tree a harness actually reads is +flat -- one directory per skill, no plugin directories and no .apm/ paths -- +so a plugin unit has no directory to point at. Source trees that do carry +plugin directories still resolve structurally through .apm/skills/ or skills/. + +dir(C) belongs outside the task package, which is what the emitters document +and what the generated manifest does. --dest inside a package is permitted, so +that /environment/skills works the way SkillsBench's own ablation driver +uses it, and is refused when a Dockerfile above it would copy it into an image. +Baking dir(C) into the build context makes the image vary with the condition, +which breaks the runtime on/off toggle and the grade-discipline invariant. + +Materializing into /environment/skills commits you to passing that same +path as bench's --skills-dir: benchflow's resolve_task_skill_policy treats the +bundled directory as stale and deletes it whenever a different --skills-dir is +given (benchflow src/benchflow/skill_policy.py). +""" + +from __future__ import annotations + +import argparse +import fnmatch +import json +import pathlib +import re +import shlex +import shutil +import sys + +COPY_RE = re.compile(r"^\s*(?:COPY|ADD)\s+(?P\S.*)$", re.IGNORECASE) + + +def load_membership(path: pathlib.Path) -> dict[str, str]: + """Read a skill-name to plugin-name map from census output or a flat map.""" + data = json.loads(path.read_text()) + if not isinstance(data, dict): + raise SystemExit(f"membership {path}: expected a JSON object") + mapping: dict[str, str] = {} + for key in ("membership", "plugin_membership"): + explicit = data.get(key) + if isinstance(explicit, dict): + mapping.update( + {k: v for k, v in explicit.items() if isinstance(v, str)} + ) + skills = data.get("skills") + if isinstance(skills, dict): + for name, rec in skills.items(): + if isinstance(rec, dict) and isinstance(rec.get("plugin"), str): + mapping.setdefault(name, rec["plugin"]) + plugins = data.get("plugins") + if isinstance(plugins, dict): + for plugin, members in plugins.items(): + if isinstance(members, list): + for name in members: + if isinstance(name, str): + mapping.setdefault(name, plugin) + if not mapping and all(isinstance(v, str) for v in data.values()): + mapping = dict(data) + if not mapping: + raise SystemExit(f"membership {path}: no skill -> plugin mapping found") + return mapping + + +def structural_skill_dirs(src: pathlib.Path) -> list[pathlib.Path]: + """Skill folders nested under a plugin directory in a source-layout tree.""" + for base in (src / ".apm" / "skills", src / "skills", src): + if base.is_dir(): + found = [d for d in sorted(base.iterdir()) if (d / "SKILL.md").is_file()] + if found: + return found + return [] + + +def derived_plugin_dirs( + plugin: str, members: list[str], root: pathlib.Path +) -> list[pathlib.Path]: + dirs, missing = [], [] + for name in members: + d = root / name + if (d / "SKILL.md").is_file(): + dirs.append(d) + else: + missing.append(name) + if missing: + raise SystemExit( + f"plugin {plugin}: member skills absent from {root}: " + f"{', '.join(missing)} -- the membership map and --skills-root must " + f"describe the same tree" + ) + return dirs + + +def resolve_unit( + unit: str, root: pathlib.Path, membership: dict[str, str] +) -> list[pathlib.Path]: + kind, sep, bare = unit.partition(":") + if not sep or kind not in ("skill", "plugin"): + kind, bare = "", unit + src = root / bare + as_skill = [src] if (src / "SKILL.md").is_file() else [] + members = sorted(s for s, p in membership.items() if p == bare) + + if kind == "skill": + if not as_skill: + raise SystemExit(f"unit {unit}: no SKILL.md at {src}") + return as_skill + if kind == "plugin": + if members: + return derived_plugin_dirs(bare, members, root) + found = structural_skill_dirs(src) + if found: + return found + raise SystemExit( + f"unit {unit}: not a plugin in the membership map and no nested " + f"SKILL.md under {src}" + ) + if as_skill and members: + raise SystemExit( + f"unit {unit} names both a skill folder ({src}) and a plugin in the " + f"membership map; disambiguate as skill:{bare} or plugin:{bare}" + ) + if as_skill: + return as_skill + if members: + return derived_plugin_dirs(bare, members, root) + found = structural_skill_dirs(src) + if found: + return found + known = f"{len(membership)} skills mapped" if membership else "no map given" + raise SystemExit( + f"unit {unit}: no SKILL.md at {src}, not a plugin in the membership map " + f"({known}), and no nested SKILL.md under {src}" + ) + + +def _norm(source: str) -> str: + p = source.strip().strip("\"'") + while p.startswith("./"): + p = p[2:] + return p.rstrip("/") + + +def _copy_sources(args_text: str) -> list[str]: + try: + parts = shlex.split(args_text) + except ValueError: + return [] + parts = [p for p in parts if not p.startswith("--")] + return parts[:-1] if len(parts) >= 2 else [] + + +def _ingests(source: str, rel: str) -> bool: + s = _norm(source) + if s in ("", ".", "*"): + return True + if any(ch in s for ch in "*?["): + return fnmatch.fnmatch(rel.split("/")[0], s.split("/")[0]) + return rel == s or rel.startswith(s + "/") + + +def build_context_conflict(dest: pathlib.Path) -> tuple[pathlib.Path, str] | None: + """The nearest Dockerfile above dest that would copy dest into an image.""" + target = dest.resolve() + for parent in target.parents: + dockerfile = parent / "Dockerfile" + if not dockerfile.is_file(): + continue + rel = target.relative_to(parent).as_posix() + text = re.sub(r"\\\s*\n", " ", dockerfile.read_text(errors="ignore")) + for line in text.splitlines(): + m = COPY_RE.match(line) + if m and any(_ingests(s, rel) for s in _copy_sources(m.group("args"))): + return dockerfile, line.strip() + return None + + +def prepare_dest(dest: pathlib.Path) -> None: + conflict = build_context_conflict(dest) + if conflict: + dockerfile, line = conflict + raise SystemExit( + f"refusing: {dockerfile} would copy {dest} into the image " + f"({line!r}); skills must be injected at runtime, never baked" + ) + if dest.exists(): + stray = [ + e.name + for e in sorted(dest.iterdir()) + if not (e.is_dir() and (e / "SKILL.md").is_file()) + ] + if stray: + raise SystemExit( + f"refusing to replace {dest}: it holds entries that are not skill " + f"folders ({', '.join(stray[:5])})" + ) + shutil.rmtree(dest) + dest.mkdir(parents=True) + + +def materialize( + root: pathlib.Path, + units: list[str], + dest: pathlib.Path, + membership: dict[str, str], +) -> list[pathlib.Path]: + prepare_dest(dest) + placed: dict[str, str] = {} + out: list[pathlib.Path] = [] + for u in units: + for d in resolve_unit(u, root, membership): + if d.name in placed: + raise SystemExit( + f"name collision materializing {u}: {d.name} already placed " + f"by {placed[d.name]} -- resolve before running" + ) + placed[d.name] = u + shutil.copytree(d, dest / d.name) + out.append(dest / d.name) + return out + + +def main() -> None: + ap = argparse.ArgumentParser(description=__doc__.splitlines()[0]) + ap.add_argument("--skills-root", required=True) + ap.add_argument("--units", nargs="*", default=[]) + ap.add_argument("--dest", required=True) + ap.add_argument( + "--membership", + help="census.json (or a flat {skill: plugin} map) enabling plugin units " + "over a flat deployed tree", + ) + a = ap.parse_args() + + membership = ( + load_membership(pathlib.Path(a.membership)) if a.membership else {} + ) + dirs = materialize( + pathlib.Path(a.skills_root), a.units, pathlib.Path(a.dest), membership + ) + for d in dirs: + print(d) + print( + f"dir(C) = {a.dest} <- {sorted(a.units) or '{}'} [{len(dirs)} skills]", + file=sys.stderr, + ) + + +if __name__ == "__main__": + main() diff --git a/modules/home/ai/skills/default.nix b/modules/home/ai/skills/default.nix index f1cd2db53..3c573709b 100644 --- a/modules/home/ai/skills/default.nix +++ b/modules/home/ai/skills/default.nix @@ -53,9 +53,30 @@ # atdd-outer-loop -> test-driven-development ecosystem at model-selection time and # would bypass ATDD routing, so it is withheld from delivery here. The compose # still builds it; only the harness symlink is dropped. Remove the name to re-include. - excludedUpstreamSkills = [ "tdd" ]; + # + # harborize is iterated in place through a hand-made ~/.claude/skills/harborize + # symlink into this repository. Delivering it while that symlink exists makes + # home-manager's backup pass move the repository's own sources to + # *.before-home-manager and replace them with store symlinks, because the symlink + # is an intermediate path component and the pre-link move resolves through it. + # Remove the symlink before removing this exclusion. + excludedSkills = [ + "tdd" + "harborize" + # Beads retired as a work-owning layer (Linear/OpenSpec own the work); + # issues-beads* withheld from delivery pending post-migration source + # cleanup of remaining bd references. + "issues-beads" + "issues-beads-seed" + "issues-beads-init" + "issues-beads-orient" + "issues-beads-prime" + "issues-beads-checkpoint" + "issues-beads-evolve" + "issues-beads-audit" + ]; - allSkills = removeAttrs (readSkillsFrom "${config.aiSkills.composed}/.claude/skills") excludedUpstreamSkills; + allSkills = removeAttrs (readSkillsFrom "${config.aiSkills.composed}/.claude/skills") excludedSkills; # Third-party skill directories supplied via aiSkills.extraSkillDirs. # Each entry is a directory (often a nix store path string) holding diff --git a/openspec/changes/validate-harborize-instrument/.openspec.yaml b/openspec/changes/validate-harborize-instrument/.openspec.yaml new file mode 100644 index 000000000..c89fb1574 --- /dev/null +++ b/openspec/changes/validate-harborize-instrument/.openspec.yaml @@ -0,0 +1,2 @@ +schema: superpowers-bridge +created: 2026-08-14 diff --git a/openspec/changes/validate-harborize-instrument/brainstorm.md b/openspec/changes/validate-harborize-instrument/brainstorm.md new file mode 100644 index 000000000..462deecb1 --- /dev/null +++ b/openspec/changes/validate-harborize-instrument/brainstorm.md @@ -0,0 +1,108 @@ + + +# Background + +The harborize skill is a measurement instrument that compiles agent skills into runnable evaluation packages. +It has been through two repair rounds in one day: 0.2.0 fixed documented instructions that could not run at all, and 0.2.1 fixed claims that were true on the host and asserted about the container. +`CHANGELOG.md` records what moved in each. + +Three things are true at 0.2.1 and together they define the problem this change addresses. +No package emitted by the instrument has ever been validated: the three iteration-1 packages under `~/Downloads/harbor-skill/harborize-workspace/iteration-1/` failed the oracle inhabitation invariant, two ship a fixture script the instrument forbids, and one declares a separate-mode verifier that cannot run. +No per-run cost figure exists, so every budget the instrument's Phase 5 asks the user to choose between is presented without arithmetic behind it. +And the repairs themselves are unwitnessed: the 0.2.1 round asserts a set of container-boundary facts about skill injection that nothing has yet exercised end to end. + +A marketplace-wide evaluation program is drafted in `references/marketplace-program.md` with five stages, and every one of its figures multiplies a per-run cost that has not been measured. +Running that program before the instrument produces a package that demonstrably injects is how the earlier iteration produced three unusable artifacts. + +# Decision chain + +## Q1 — what is the unit of ablation? + +Settled: the skill, materialized from the deployed tree. +A plugin unit is a derived aggregate, the union of its member skills' deployed directories, resolved through an explicit skill-to-plugin membership map shipped beside the census. +The map is required because the deployed tree is flat: one directory per skill, no plugin directories and no `.apm/` paths, so membership cannot be recovered from directory structure. + +The census covers both subjects and reports both. +The deployed tree is the evaluation subject because it is what a harness loads, currently 172 skill directories under `~/.claude/skills`. +The first-party source tree is the refactor subject because it is the only place a fix can be written, currently 129 skills across 18 plugin groups per the bundled `census.json` provenance block. + +## Q2 — mechanical contracts or judge-graded ones? + +Settled: mechanical by default, and exactly one judge-validation package is a hard gate before any judge-based stratum enters a budget. +The gate is warranted by the surrounding ecosystem rather than by taste: all 87 tasks shipped in SkillsBench declare `verifier.type: test-script` and none uses a judge verifier. +That gate is out of scope here and belongs to the dependent change, because building it costs a human-labeled validation set and this change has to establish that a mechanical package works at all first. + +## Q3 — how is a budget costed? + +Settled: every figure is costed at metered API rates. +Subscription-authenticated cells are permitted for interactive exploration and never appear in a batch whose numbers are reported, because metered costing is robust to the unresolved credential-use-policy question in either direction and because a subscription cell confounds the measurement independently through nondeterministic rate-limit throttling and single-account concurrency caps. +Enforcement is an explicit environment scrub rather than the absence of a flag: Pi injects `ANTHROPIC_OAUTH_TOKEN` whenever the variable is present and non-empty in the resolved environment, with no force flag of the kind claude-code and codex require. + +## Q4 — what happens to the three iteration-1 packages? + +Settled: reference material for shape, regenerated under the repaired instrument, never cited for results. +None of them passed oracle inhabitation, so promoting any of them would carry an unvalidated package into a measured program. + +## Q5 — how does instrument versioning interact with a round? + +Settled: results are indexed by instrument version, and the instrument is not modified while an evaluation is being authored or run. +Revisions happen between rounds. +This change runs against 0.2.1 throughout. +A defect found mid-change is recorded and deferred to the next revision rather than fixed in place, because a fix mid-round makes the round's results unattributable to any version. + +## Q6 — given Q1 through Q5, what does this change deliver? + +Settled: the smallest thing that turns the instrument's claims into evidence and produces the one number every later budget multiplies. +Three deliverables. +An injection canary package that stays in the corpus permanently as a regression test, so the container-boundary facts 0.2.1 asserts are exercised on every later round rather than assumed. +One real mechanical evaluation package, so the dual-head authoring path, the static gates and the oracle inhabitation invariant are demonstrated on a task that measures something. +And the measured per-run cost constant. + +## Q7 — in what order, given that most of the risk is free to retire? + +Settled: a seven-rung ladder numbered 0 through 6, free rungs before metered ones, each rung with an executable pass criterion. +The ordering is not a preference about thoroughness; it follows from where the failures actually sit. +Every silent-null class the instrument documents is detectable at zero marginal cost except the adapter registration copy, which is built inside each adapter's `run()` and is therefore unreachable without a real agent invocation. +So the metered rung is last and it is one short trial per cell, and everything upstream of it — prerequisites, the adapter allowlist, host-side resolution, static task validation, and the two oracle rungs that cost Docker time and zero model calls — retires the rest. + +The prerequisites rung is a real rung rather than an assumption. +OrbStack is stopped, the Docker socket is absent, and neither `harbor` nor `bench` is installed, so the ladder currently cannot start. +One prerequisite is blocking in a way that shapes authoring rather than merely delaying it: Harbor rejects any `no-network` policy at environment start when the daemon's kernel lacks `CONFIG_NFT_FIB_INET`, and the resulting failure surfaces at the reward level indistinguishably from an injection failure. +`no-network` is the harborize instrument's own authoring default rather than Harbor's, whose default is `public`, so the probe decides what a package may declare rather than whether the change can proceed. +Probing it before any task is authored is what keeps a daemon capability from being read as an authoring error. + +## Q8 — one package directory or two? + +Settled: two, with the BenchFlow-native tree authored and the Harbor head derived by `bench tasks export` into a sibling directory. +Three independent validators force the split rather than merely preferring it, and the SkillsBench corpus is uniformly native. +Deriving rather than maintaining two trees is what keeps the heads from drifting. + +## Q9 — what about the canary's answer key? + +Open, and deliberately left open. +Any delivery canary that asserts a token has to place the same literal in the verifier and in the SKILL.md, which the instrument's own leakage audit flags by construction as a quoted expectation string recoverable from skill content. +The flag is correct on its own terms and the package is still sound, because the canary is an instrument-integrity test rather than a capability measurement: the answer key living in the skill is the delivery mechanism being tested. +Adding an exemption to the audit is the obvious resolution and D5 forbids it, since that is a modification to the instrument during a round. +So the change records how it handles the flag and defers the instrument-side question. + +# Design trade-offs + +Scope against confidence. +A wider first change — several packages, a condition lattice, a plugin-level screen — would produce more measurement per unit of setup, and it would repeat the iteration-1 failure of building on an instrument whose delivery path has never been demonstrated. +The narrow change buys a permanent regression asset and a cost constant, and it defers everything that multiplies them. + +Docker time against model spend. +Rungs 4 and 5 are the strongest evidence available for free, because both runners' oracle paths execute the full deployment machinery with no model materialized: BenchFlow asserts the in-container skill catalogue against the host-computed one, and Harbor's oracle uploads a solution and execs it. +They cost image builds and container time, which is why they sit ahead of the metered rung rather than being skipped as merely cheap. + +Canary permanence against corpus noise. +Keeping the canary in the corpus forever adds a task that measures no capability, and that is the point: it is the only artifact that fails when the injection path regresses, and a regression there voids every measurement taken after it. diff --git a/openspec/changes/validate-harborize-instrument/design.md b/openspec/changes/validate-harborize-instrument/design.md new file mode 100644 index 000000000..bdd20d543 --- /dev/null +++ b/openspec/changes/validate-harborize-instrument/design.md @@ -0,0 +1,227 @@ +## Context + +The harborize instrument at `modules/home/ai/plugins/testing-and-quality/.apm/skills/harborize/` compiles agent skills into evaluation packages runnable under both the Harbor CLI and the BenchFlow CLI. +It is at version 0.2.1, reached through two repair rounds recorded in its `CHANGELOG.md`: 0.2.0 fixed documented instructions that could not run, and 0.2.1 fixed claims that were true on the host and asserted about the container. + +Nothing the instrument has emitted has been validated. +The three iteration-1 packages under `~/Downloads/harbor-skill/harborize-workspace/iteration-1/` failed the oracle inhabitation invariant, and the instrument's own `references/marketplace-program.md` records that none is cited as validated. +No per-run cost figure exists, so the staged marketplace program's screening estimate of roughly 500 to 700 runs multiplies an unmeasured constant. + +The failure classes that make this expensive to get wrong are all silent. +Harbor uploads injected skills for every agent, and of the 39 agents its factory registers, 22 read the injected directory and 17 ignore it with no error, no warning and no log line; the counts come from `rg -c 'AgentName\.[A-Z_0-9]+: ' src/harbor/agents/factory.py` returning 39 and `rg -l 'self\.skills_dir' src/harbor/agents/ --type py` returning 23 files, the latter being 22 adapters plus the definition site at `agents/base.py`, both reproduced here at the pinned revision. +A cell naming one of the 17 exits 0 and collapses every condition to the empty condition. +The trial lock cannot catch it: `_write_trial_lock` runs inside `Trial.__init__` at harbor `src/harbor/trial/trial.py:104`, before `_resolve_injected_skills` at `:107` and long before `_upload_injected_skills` at `:411`, and `_build_agent_skill_locks` (`src/harbor/models/job/lock.py:462-475`) calls only the host-side `resolve_skills`, `get_git_skill_metadata` and `compute_skill_digest`. +Harbor's telemetry cannot catch it either, since `src/harbor/telemetry.py:239` sets `uses_skills` from the requested agent list without consulting the effective directory or the adapter. + +Three upstream repositories are pinned, and every anchor in this document was read at these revisions in the local ghq clones, whose HEADs were confirmed to equal the pins. + +| repo | path | revision | +|---|---|---| +| Harbor | `~/ghq/github.com/harbor-framework/harbor` | `ac398bbda7c4c1073461797d3b95c2455cc671b5` | +| BenchFlow | `~/ghq/github.com/benchflow-ai/benchflow` | `d30527b82027a416e72014920cdf43a534967ad3` | +| SkillsBench | `~/ghq/github.com/benchflow-ai/skillsbench` | `9a1f4dd5f7659f75707435da3ce854b6e48321d1` | + +The pins are recorded explicitly because all three clones are shallow and sit at their own HEAD rather than at a released tag, so an anchor is valid only at the revision beside it and a later `ghq` sync moves every line number in this document. + +## Goals / Non-Goals + +**Goals:** +- Prove that the instrument at 0.2.1 produces a package whose skill injection is delivered, in the container, at every level where it can silently fail. +- Leave behind a permanent injection canary in the corpus, so a later regression in the injection path fails a task rather than voiding a batch. +- Produce one mechanical evaluation package that passes both static gates and the oracle inhabitation invariant under both runners. +- Measure the per-run cost constant and record the conditions it was measured under. +- Retire as much risk as possible before any model call, and reduce the metered exposure to one short trial per cell. + +**Non-Goals:** +- Marketplace-wide evaluation; the staged program stays a plan. +- Condition-lattice runs or any budget menu. Rung 1 invokes `design_matrix.py` three times against throwaway `/tmp` output directories solely to exercise its adapter gate, and nothing it emits there is executed, committed or consumed by a later rung. +- The judge-validation package, which is a hard gate for any judge-based stratum and belongs to the dependent change. +- Cost projections beyond the measured figure. +- Promotion of the three iteration-1 packages. +- Any modification to the harborize instrument, which is frozen at 0.2.1 for the duration of this change. + +## Decisions + +The first five decisions were settled before this change was opened and are encoded here rather than re-derived; they are inherited from the session that produced the 0.2.0 and 0.2.1 rounds and from the dispatch brief. +D6 onward are this change's own. + +### D1: the ablation unit is the skill, materialized from the deployed tree + +- **Choice**: the unit is the skill directory as deployed; a plugin unit is a derived aggregate, the union of its member skills' deployed directories, resolved through an explicit skill-to-plugin membership map shipped beside the census. The census covers both subjects: the deployed tree is the evaluation subject and the first-party source tree is the refactor subject. +- **Rationale**: the deployed tree is what a harness loads and it is flat, one directory per skill, with no plugin directories and no `.apm/` paths, so plugin membership cannot be recovered from directory structure. The deployed count is 172 (reproduced here by listing `~/.claude/skills`) against 129 first-party source skills across 18 plugin groups (from the `provenance` block of the instrument's bundled `census.json`, script version 0.2.0, revision `b26f5a179edbffed47253d453e3a8eb7ed43372f`). +- **Alternatives considered**: resolving plugin units by walking the filesystem, which silently yields a different unit set on the deployed tree than on the source tree. + +### D2: mechanical contracts by default, with the judge gate held out of scope + +- **Choice**: verifiers reduce to decidable predicates on final state wherever one exists; exactly one judge-validation package is a hard gate before any judge-based stratum enters a budget, and that package is not built here. +- **Rationale**: all 87 tasks shipped in SkillsBench declare `verifier.type: test-script` and none uses a judge verifier, reproduced at the pinned revision by counting `tasks/*/task.md`. The gate costs a human-labeled validation set, and this change has to demonstrate that a mechanical package works at all first. +- **Alternatives considered**: building the judge package alongside, which doubles the change and puts the harder measurement ahead of the easier one. + +### D3: every budget figure is costed at metered API rates + +- **Choice**: metered rates for anything reported; subscription-authenticated cells are for interactive exploration only. +- **Rationale**: metered costing is robust to the unresolved credential-use-policy question in either direction, and a subscription cell confounds the measurement independently through nondeterministic rate-limit throttling and single-account concurrency caps. +- **Narrowed for this change's one metered measurement, by the settled decision of 2026-08-15**: that decision names codex on the ChatGPT-subscription path as the metered adapter, which is the class this decision's second clause excludes. The narrowing is deliberate, and the rate clause survives it intact because the two clauses are separable and only the second is set aside. Harbor's codex adapter has no billed-cost field to read and derives `cost_usd` from token counts against LiteLLM's pricing table (`codex.py:724-780`, resolved at `:1120-1124`, set at `:1194`), unlike claude-code which parses an authoritative `total_cost_usd` from its own stream (`claude_code.py:858-879`, `:1463`, `:1525`). The reported figure is therefore a list-rate computation over observed token counts, which is what "costed at metered API rates" asks for, and it is a rate that does not vary with how the trial authenticated. +- **Residual confound, contained by the rung's shape rather than by argument**: the throttling and concurrency-cap confounds named above act on throughput and on batch scheduling, not on token counts, and rung 6 is one short trial on one cell (task 9.4) with no concurrency and no batch. What is not claimed is a billed charge: the record carries `"auth_mode": "chatgpt-subscription"` and the pricing basis alongside the number, and a billed-rate figure for any budget in the dependent change is a separate measurement that change must make. +- **Implementation note**: enforcement takes an explicit environment scrub rather than the absence of a flag. Claude Code gates subscription auth behind `CLAUDE_FORCE_OAUTH` and Codex behind `CODEX_FORCE_AUTH_JSON`, but Pi injects `ANTHROPIC_OAUTH_TOKEN` whenever the variable is present and non-empty in the resolved environment and the resolved provider is anthropic (harbor `src/harbor/agents/installed/pi.py:102-105`), with no force flag. The check is a walrus on the value rather than a membership test, which is what makes the empty-string scrub a sound mitigation: a Pi cell in a reported batch declares `ANTHROPIC_OAUTH_TOKEN=""` or runs from a shell where it is unset, and either form is falsy. + +### D4: the iteration-1 packages are reference material and are regenerated + +- **Choice**: read them for shape, regenerate under the repaired instrument, never cite them for results. +- **Rationale**: none passed oracle inhabitation, so promoting one carries an unvalidated package into a measured program. + +### D5: the instrument is frozen at 0.2.1 for the duration of this change + +- **Choice**: results are indexed by instrument version; the instrument is not modified while an evaluation is being authored or run, and a defect found mid-change is recorded and deferred to the next revision. +- **Rationale**: a fix applied mid-round makes the round's results unattributable to any version, which is the property the versioning exists to protect. +- **Consequence**: the three instrument-side items this change surfaces are recorded as deferred rather than fixed, and nothing under the harborize skill directory is edited. They are the canary leakage flag; the `--membership-from` census workflow; and `CHANGELOG.md:55`, which attributes `network_mode = "no-network"` to Harbor when Harbor's own default is `public` (`models/task/config.py:249-252`) and `no-network` is the instrument's own authoring prescription (`SKILL.md:124`, `references/emitters.md:74`). The third is a documentation defect in the CHANGELOG alone; `SKILL.md` and `emitters.md` state the default correctly as the instrument's own, so no emitted package is wrong because of it. +- **Enforcement**: the freeze is checked by a content digest of the harborize directory recorded at the start of the change and recomputed at the end, not by `jj diff -r @`. That diff reports only what the working-copy commit changed against its parents, so an edit squashed into the `harborize-instrument` chain this change routes onto would leave it empty while the instrument had in fact changed. + +### D6: a seven-rung cost ladder, free rungs before metered ones + +- **Choice**: run the rungs in order, each with an exact command and an executable pass criterion, with model spend confined to the last rung and reduced there to one short trial per cell. +- **Rationale**: rungs 0 through 5 spend no model calls and rung 6 is the only metered one. Only the adapter registration copy needs a real agent invocation to witness. Every adapter's registration command is built inside `run()` rather than `setup()` — for Claude Code, `_build_register_skills_command` at `claude_code.py:1530-1542` is appended to `setup_command` at `:1733-1735`, both inside `async def run` which begins at `:1601` — and `--install-only` skips the agent run: `Trial.run` guards `self._run()` on `not self.config.install_only` (`trial.py:375-378`), and `TrialConfig._install_only_disables_verification` (`models/trial/config.py:484-494`) also disables the verifier. So `--install-only` (`cli/jobs.py:901-910`) provably cannot substitute for the metered rung, and everything else is reachable for free. +- **Alternatives considered**: proving delivery from the trial lock, which is refuted by the `trial.py:104` / `:107` / `:411` ordering above; and running the metered rung first as a smoke test, which is how a daemon capability or an authoring error becomes an expensive discovery. + +### D7: two package directories, BenchFlow-native authored and the Harbor head derived + +- **Choice**: author `task.md` + `oracle/` + `verifier/` natively, then derive `-harbor` with `bench tasks export` into a sibling directory, and never hand-edit the derived head. +- **Rationale**: three independent validators forbid the co-present form. BenchFlow's `--level publication-grade` rejects a co-present `task.toml` or `instruction.md` beside `task.md` (benchflow `src/benchflow/_utils/task_authoring/structural_checks.py:208-212`) and rejects `solution/` in favour of `oracle/` (`:218-222`). SkillsBench's corpus gate forbids exactly `instruction.md`, `task.toml`, `solution` and `tests` (skillsbench `.github/scripts/validate_tasks.py:21-26`), which is the set Harbor requires. And the corpus is uniformly native: all 87 tasks under `tasks/` carry `task.md` with zero `task.toml` and zero `solution/`, reproduced here at the pinned revision. +- **Alternatives considered**: one directory carrying both heads, which fails publication-grade by construction; and maintaining two hand-written trees, which drifts. + +### D8: the canary asserts delivery where delivery is observable, per adapter + +- **Choice**: the canary package asserts, on the BenchFlow arm, that the in-container skill catalogue equals the host-computed one, and on the Harbor arm that the adapter's own destination directory carries the skill after a metered trial. No single assertion is written for the whole grid. +- **Rationale**: the destinations differ per adapter — claude-code copies into `$CLAUDE_CONFIG_DIR/skills//` with `CLAUDE_CONFIG_DIR` set to `EnvironmentPaths.agent_dir / "sessions"` at `claude_code.py:1718`, which resolves to `/logs/agent/sessions` since `agent_dir` is `logs_dir / "agent"` at `models/trial/paths.py:36`; codex writes `$HOME/.agents/skills//` (`codex.py:1199-1207`); pi writes the same path (`pi.py:75-83`); and opencode writes `~/.config/opencode/skills//` (`opencode.py:425-433`). +- **Free consequence exploited, superseded for the metered cell**: claude-code's destination sits inside the `/logs/agent` bind mount (`trial.py:1284-1288`), so what the adapter registered is readable on the host after the run with no verifier code at all. That was the cheapest per-adapter assertion available and the original reason claude-code was named the first metered cell; the settled decision names codex on the ChatGPT-subscription path as the metered adapter instead, and codex's `$HOME/.agents/skills//` destination sits in no bind mount, so its assertion is made in-container (D8's per-adapter scope is unchanged). +- **Scope of the Harbor-arm assertion**: it is the destination directory alone and deliberately not the reward. The original reason — that a real agent could reach the token through the shared verifier — does not hold and is corrected in D11; the surviving reason is stronger and adapter-independent. A model-driven trial's reward conflates delivery with the model's own behaviour, because a model that never greps the discovery roots scores 0 with the skill perfectly delivered, so only the registration directory is a deterministic witness. + +### D9: the cost constant is measured, not estimated, and recorded with its conditions + +- **Choice**: the per-run cost is taken from the metered rung's own job accounting and recorded together with the cell, the model, the task, the trial length, the auth mode, and the instrument version. +- **Rationale**: a bare number without its conditions cannot be multiplied into a budget honestly, since a cost per run is a function of the cell and the task, not of the runner. +- **Scope boundary**: recording the constant is in scope, multiplying it into a program budget is not. + +### D10: verification runs through the runners, and no Gherkin layer is introduced + +- **Choice**: each ladder rung is witnessed by the runner's own artifact — a docker exit status, a python-level refusal, `bench tasks check`'s issue list, a reward file, or a directory listing taken inside the trial container — recorded in tasks.md and verify.md. No `.feature` files and no step-definition runner are added. +- **Rationale**: this is a Gate 0 verdict rather than a per-proposition Gate 1 one, and the Gate 1 section below records it as such. Gate 0 asks whether the change has an external observable a non-implementer stakeholder would recognize, and every observable here is a CLI flag, a container path or a reward file, so a scenario's steps could name nothing except the machine. That is Gate 0's own tell for a forced outer loop, and its own prescribed response is to drop out of the outer loop rather than launder structure through Gherkin. Every ladder proposition then routes, at Gate 1, to the one row that leaves BDD by construction: an assertion that an upstream Harbor or BenchFlow API still fits is dependency compatibility, witnessed by the runner that already exercises it. +- **Alternatives considered**: binding the rungs through pytest-bdd with steps that shell out to `docker`, `bench` and `harbor`, which adds a runner and a traceability guard to a change whose deliverables are evaluation packages, and whose scenarios would restate their own commands. + +### D11: both packages run a shared verifier, and the agent-to-verifier channel is `/logs/artifacts` + +- **Choice**: both packages declare no `verifier.sandbox_mode` and ship no `verifier/Dockerfile`, so both run shared, and each writes its deliverable under `/logs/artifacts` where its verifier reads it; `/logs/verifier` carries the reward file the verifier itself writes and nothing else. Separate mode is exercised nowhere in this change; the earlier plan to exercise it on the mechanical package is withdrawn, and the reason it was withdrawn is recorded here rather than dropped. +- **Rationale**: the fork is forced rather than preferred, and each runner forces it independently. The original reading of BenchFlow was wrong in a way worth recording, because the source sentence invites it: "separate verifier sandboxes are parsed but not executed" is the reason string of a refusal, not a description of a fallback. `runtime_capabilities.py:186-192` raises an unsupported-feature issue and `raise_for_task_runtime_support` is a fail-closed pre-launch gate (`sandbox/setup.py:676`, `:819-842`), so a package declaring separate does not run shared under BenchFlow — it does not launch at all. Reproduced with `bench tasks check --level runtime-capability --sandbox docker`, which exits 0 on both packages as shipped and reports that issue the moment `verifier.sandbox_mode: separate` is added. Since D7 authors one `task.md` per package and the proposal requires the mechanical package to pass under both runners, separate mode and a runnable BenchFlow arm are mutually exclusive at benchflow 0.7.4. +- **Second and independent reason, which also refutes the channel-survives argument**: Harbor's separate verifier empties `/logs/verifier` before running, `await target_env.empty_dirs([env_paths.verifier_dir], chmod=True)` at `trial.py:599`, through the same host bind it mounts at `:686-692`. The bind is real and the channel still does not survive, because the content is deleted rather than the mount dropped, so every agent including the oracle would score 0. The only separate-mode-safe channel is `/logs/artifacts/`, which the artifact re-upload restores after the wipe (`trial.py:601-607`, `artifact_handler.py:210-254`); moving a deliverable there is a task-contract change rather than a fork choice. +- **Why `/logs/artifacts` rather than `/logs/verifier` or the workspace**: the original choice of `/logs/verifier` was read off Harbor alone and is refuted by the rung-4 run. BenchFlow clears that directory's contents on the agent container immediately before the verifier runs, unconditionally: `harden_before_verify` executes `_CLEAR_VERIFIER_DIR_CMD` (`sandbox/lockdown.py:775-784`, call at `:1205-1212`), and the mount-aware conditional clear in `_verify_test_script` (`task/verifier_core.py:360-372`) is a different branch that never reached the file. The bind survives and the content does not, so the canary written that way scored 1 under Harbor's shared verifier and 0 under BenchFlow with no error and no diagnostic beyond a `grep` miss — the exact runner-dependent confound this change exists to remove, arrived at from the opposite direction to the one predicted. `/logs/artifacts` is bind-mounted for the whole trial by both runners (`sandbox/docker.py:186-201`; `models/trial/paths.py:38`, `:195-202`), no hardening step touches it, and it is additionally the one path that would survive Harbor's separate fork per the reason above. A workspace path such as `/app` or `/root` also survives both, verified in the same round, and is rejected for the original reason: it is neither runner's contract path. +- **Cost that was accepted and does not arise**: the shared fork was taken to expose the verifier to the agent under test. It does not. Both runners upload the verifier's own directory during the verification phase, after the agent phase has ended — Harbor at `verifier/verifier.py:147-153`, reached from `_run_shared_verifier`, with the phase order fixed at `trial/single_step.py:41` then `:52`; BenchFlow at `task/verifier_core.py:385` in `_verify_test_script` (`:346`), reached from `verify()` (`:260`). The mechanical package's verifier therefore holds its expectation as a literal without leaking it, and rung 6 still declines to read the canary's reward — because a model-driven trial's reward conflates delivery with the model's own behaviour, not for the reason originally given. + +### D12: the environment baseline is public and `no-network` is an agent-phase override + +- **Choice**: every package declares `network_mode = "public"` at `[environment]` and puts `no-network` on the `[agent]` phase, dropping the phase override entirely when the kernel probe reports the option unset. +- **Rationale**: the baseline is what the container is created with (`trial.py:896`), while the phase policy is applied around `_run_agent_phase` alone (`trial.py:465-469`) and around the verifier phases. claude-code's `install()` curls its bootstrap during `_prepare`/`_setup_agent` (`trial.py:408-414`), which no policy wraps, so a `no-network` baseline breaks the one metered rung during agent install while a `no-network` agent phase leaves it intact. Harbor reads the override through `task_cfg.agent.explicit_phase_policy()` (`trial/network_policy.py:45-59`) and validates the switch at `trial.py:203-217`; BenchFlow carries the same field at `task/config.py:524-530`. +- **Correction encoded**: `no-network` is the harborize instrument's authoring default (`SKILL.md:124`, `references/emitters.md:74`), not Harbor's. Harbor's default is `public` (`models/task/config.py:249-252`), BenchFlow's is the same (`task/config.py:720-723`), and 86 of the 87 SkillsBench corpus tasks declare `network_mode: public`. The kernel probe therefore gates whether a `no-network` declaration can be *enforced*, not whether a package can be authored at all. +- **Alternatives considered**: baking the agent into the environment image at a pinned version so `_installed_claude_satisfies_version` returns early and the fetch is never reached, which the instrument itself prescribes at `SKILL.md:169`. It works and costs an image build per agent version; the phase override achieves the same result with no build and is reversible per probe outcome. + +### D13: the corpus lives at `modules/home/ai/evals/harborize/` and carries no `.nix` file + +- **Choice**: the package corpus is co-located with the skills it evaluates, at `modules/home/ai/evals/harborize/`, tracked in version control, with job output at `logs/harborize/` which `.gitignore:57` already excludes. No file inside the corpus carries the `.nix` extension. +- **Rationale**: co-location puts the measurement beside its subject, since the source of truth for every first-party skill under evaluation is `modules/home/ai/plugins/`. The location was verified against each automated surface that walks `modules/` rather than assumed safe. import-tree is called bare at `flake.nix:6` with no custom filter, so its default `nixFilter = andNot (hasInfix "/_") (hasSuffix ".nix")` (import-tree `default.nix:50`, rev `4ebb10ae17d5f1ad366e7aef5b92cb8eecf24f69`) enumerates non-nix files and drops them before anything reads them, which is why `modules/` already carries hundreds of non-nix files — 433 at `origin/main` `25d10e7c` and 480 at this change's tip, of which 34 are this corpus — and the flake still evaluates. `naming-conventions` (`modules/checks/validation.nix:186-205`) validates machine names read from `self.nixosConfigurations` and never filenames. treefmt enables only `programs.nixfmt` (`modules/formatting.nix:13`), scoped to `*.nix`, so byte-sensitive fixtures are untouched. The apm composition scans `modules/home/ai/plugins` one level deep gated on a `.apm/skills` child (`pkgs/by-name/apm-skills-compose/package.nix:11-19`), so an `evals/` sibling is outside it. +- **Consequence**: the same filter that makes every other extension safe makes `.nix` unsafe, because a `*.nix` file anywhere under `modules/` is imported and evaluated as a flake-parts module. A fixture named `expected.nix` is the realistic trap, and it would break the flake rather than fail as a fixture. Two escapes exist and either one is sufficient: no `.nix` extension on a fixture, or a `_`-prefixed directory, which is this repository's documented exclusion convention (ADR-0018, `packages/docs/src/content/docs/development/architecture/adrs/0018-deferred-module-composition-architecture.md:311`). +- **Enforcement**: an extension audit plus `nix eval .#nixosConfigurations --apply builtins.attrNames` runs when the corpus root is first written and again once the package directories and the generated Harbor heads exist, because a constraint with no check is a comment. The guard covers the generated heads too, since two of the four package directories are written by `bench tasks export` rather than authored by hand. +- **Alternatives considered**: a repository-root `evals/` directory, which is outside every `modules/` walker and so carries no `.nix` hazard, and which separates the corpus from the skills it measures; and a `_`-prefixed directory under `modules/`, which removes the hazard outright at the cost of marking the corpus as excluded machinery rather than as tracked content. + +## Gate 1 modality verdicts + +Gate 0 excludes this change from the outer loop as a whole. +The gate asks whether the change has an external observable a non-implementer stakeholder would recognize and care about, and this change's every observable is a container path, a CLI exit status or a reward file. +Its own tell for a forced outer loop — a scenario whose steps name functions, tables or endpoints — is unavoidable here, so no acceptance stream and no `.feature` file is laid out. +Gate 1 is still recorded per requirement below, because the modality column is what the pre-apply and before-archive machine checks read. + +Every ladder requirement routes to the same Gate 1 row, `Dependency compatibility / import smoke`, and therefore to `smoke`. +That is the accurate class rather than a convenient one. +Each of these propositions asserts that an upstream API in Harbor or BenchFlow still fits — which adapters consume an injected directory, that host-side resolution still returns digests, that the oracle path still deploys skills, that the adapter still copies into its destination. +The change exists precisely because those are facts about two pinned upstreams that the instrument read from source and never exercised. + +| Requirement | Proposition class | modality | Runner and witness | +|---|---|---|---| +| Environment prerequisites are established before authoring | Dependency compatibility / import smoke | `smoke` | `docker run` exit status and the kernel-probe exit status; `harbor --version` / `bench --version` | +| The evaluation corpus does not break flake evaluation | Dependency compatibility / import smoke | `smoke` | no `.nix` file inside the corpus, and `nix eval .#nixosConfigurations --apply builtins.attrNames` exits 0 | +| Harbor cells name only skill-consuming adapters | Dependency compatibility / import smoke | `smoke` | `scripts/design_matrix.py` exits nonzero naming the adapter | +| Host-side skill resolution is proven before any container starts | Dependency compatibility / import smoke | `smoke` | `harbor.skills.resolve_skills` output: one entry per expected skill, each with a sha256 | +| Both task heads pass their static gate | Dependency compatibility / import smoke | `smoke` | `bench tasks check --level structural` exit 0; `harbor.models.task.task.Task()` constructs | +| Skill delivery is proven in the container with no model call | Dependency compatibility / import smoke | `smoke` | `bench eval run --agent oracle` reaches the agent phase and reward == 1 | +| The oracle inhabits the task under Harbor | Dependency compatibility / import smoke | `smoke` | `harbor run` reward 1.0 across five trials, zero errored trials, `agent/exit-code.txt` absent or 0 | +| Adapter registration is asserted per adapter | Dependency compatibility / import smoke | `smoke` | the adapter's destination directory carries the skill, asserted in-container for codex at `$HOME/.agents/skills//`, which sits in no bind mount | +| The injection canary is retained in the corpus permanently | none — Gate 0 excluded | `none` | the canary package is present and its run is listed in the round's manifest | +| The per-run cost constant is recorded with its conditions | none — Gate 0 excluded | `none` | the results file carries the constant, the cell, the auth mode and the instrument version | +| Packages and results are stamped with the instrument version, and the instrument is unmodified | none — Gate 0 excluded | `none` | each package README stamps 0.2.1; the harborize directory digest matches its recorded baseline | + +The last three rows carry no Gate 1 modality at all, and `none` says so rather than inventing a token. +They are a corpus property over time, a recorded derived number, and a provenance property; none is an operation with an observable, so Gate 1 has nothing to route and they are witnessed by reading the artifact at the review gate. + +No row carries `bdd-scenario`, so the pre-apply feature-layout arm of the spec-and-feature alignment sub-gate no-ops for this change. +No row carries `est-property`, `est-contract` or `est-symbolic`, so the executable-specification arm no-ops as well. +`smoke` is the Gate 1 vocabulary's own name for the regression-or-smoke destination and carries no expectation of a `.feature` file or an EST artifact. + +## Risks / Trade-offs + +- [Risk] The Docker daemon's kernel lacks `CONFIG_NFT_FIB_INET`, so Harbor rejects any `no-network` policy at environment start. On failure `_enable_egress_control` goes false (`src/harbor/environments/docker/docker.py:188-195`), which zeroes `capabilities.disable_internet` (`:289-293`), and `src/harbor/environments/base.py:773-781` raises. A task carrying `no-network` at its baseline then fails at environment start and reads as an authoring error. → Mitigation: the probe is a blocking task with its own checkbox, run immediately after the daemon starts and before any task is authored, and it branches three ways rather than halting. It is anchored (`grep -qE '^CONFIG_NFT_FIB_INET=[ym]'`) so its exit status is its criterion, because an unanchored search matches `# CONFIG_NFT_FIB_INET is not set` and exits 0. Exit 1 drops the `[agent]` phase override and leaves every package fully public, which is Harbor's own default and the corpus norm. Exit 2, an absent `/proc/config.gz`, is indeterminate rather than negative: Harbor's own probe short-circuits to exit 0 in that case (`docker.py:113-117`), so the change proceeds with Harbor and treats a later rejection as the deciding evidence. +- [Risk] A claude-code cell whose environment *baseline* is `no-network` fails during agent install rather than during the agent phase, indistinguishably from an injection failure at the reward level. `_prepare` calls `_setup_agent` at `trial.py:408-414` with no policy wrapper, while only `_run_agent_phase` (`trial.py:465-469`) and the verifier phases enter `_phase_network_policy`. → Mitigation: D12, carried as its own authoring task rather than as advice. Every package declares `network_mode = "public"` at `[environment]` and confines `no-network` to the `[agent]` phase, and the metered rung's own checklist re-reads the exported head's `[environment]` table before spending. Baking the agent into the image at a pinned version (`SKILL.md:169`) remains available and is not taken, because it costs an image build per agent version and the phase override is reversible per probe outcome. +- [Risk] A broken oracle scores 0 without raising, because `OracleAgent.run` writes `exit-code.txt` on a nonzero return and proceeds to the verifier (`agents/oracle.py:149-151`). Reading rewards alone cannot separate it from a genuine zero. → Mitigation: the Harbor rung's pass criterion checks both the reward and the presence of `exit-code.txt`. +- [Risk] A trial dies on the first exception, because `RetryConfig.max_retries` defaults to 0 (`models/job/config.py:282-284`). Four further exceptions kill a trial without appearing in the nine-name no-retry list at `:288-300`: `AddTestsDirError` (`verifier/verifier.py:19`), `DownloadVerifierDirError` (`:27`), and a bare `FileNotFoundError` from `_resolve_tests` or from a missing solve.sh (`agents/oracle.py:94-95`). → Mitigation: triage reads the exception name first. Four of the nine listed names are defined in `agents/installed/base.py`, which `OracleAgent` does not subclass, so any of them appearing in an oracle rung's log means the job was not running the oracle. +- [Risk] The canary's answer key is flagged by the instrument's own leakage audit, since the same literal must appear in the verifier and in the SKILL.md and `scripts/audit_leakage.py` check 1 searches for quoted expectation strings of eight characters or more (`MIN_LITERAL_LENGTH` at `:44`, `check_literals` at `:96`). → Mitigation: record the flag with its reasoning in the package README rather than suppressing it; the audit is correct on its terms and the canary is an instrument-integrity test whose mechanism is the answer key. The instrument-side question is deferred under D5. +- [Risk] The fidelity assertion the delivery rung relies on cannot be exercised by editing `dir(C)`. `_skill_link_cmd` replaces each discovery path with a symlink to the uploaded source (`agents/install.py:90`), which makes the `actual == source_catalog` test at `:157` tautological, and a `--skill-mode no-skill` run cannot raise the error either, because `expected_skill_names` is empty when `skills_dir` is falsy (`:313-317`) and the raise is guarded on `if expected:` (`:176-180`). Without a working control, a green delivery rung proves only that nothing crashed. → Mitigation: two distinct controls rather than one. A falsifiability control (`--skill-mode no-skill`, reward 0) proves the canary is not a constant, and a fidelity control against a throwaway sibling whose `environment/Dockerfile` carries a hand-written `COPY _deps/skills /skills/` line over a different `_deps/skills` tree makes `deploy_skills` take the `already_injected` branch (`:328-332`, `:342-344`), so the container's catalogue and the host-computed `expected` diverge and `:159-161` fails. +- [Trade-off] A multi-dimensional rubric would carry more information per run and would disable both headline statistics: Harbor computes pass@k only when every trial carries exactly one reward key valued 0 or 1, and BenchFlow's compare-lift counts only `reward == 1.0` as passed (`src/benchflow/eval_lift.py:32-33`). → Accepted: a single binary reward key throughout this change. +- [Trade-off] Keeping the canary in the corpus permanently adds a task that measures no capability. → Accepted: it is the only artifact that fails when the injection path regresses. +- [Trade-off] The metered rung buys one number and one per-adapter assertion rather than a measurement. → Accepted: this change's output is an instrument and a constant, and the measurement is the dependent change. + +## Migration Plan + +There is no deployment surface: no machine configuration, endpoint, schema or nix output changes. +The work is additive and reversible in the ordinary sense, in that the packages are new directories and the instrument is untouched. + +Sequencing follows the ladder, and the ordering is load-bearing rather than a preference. +The prerequisites rung comes first because the ladder cannot start without a daemon and the two CLIs, and because the kernel probe decides whether the `[agent]` phase override in D12 can be declared at all by every task authored after it. +The five free rungs then run before any container carrying a model, and the metered rung runs last on one short trial per cell. +Rollback for the metered rung is to stop after it and report the constant; rollback for the change as a whole is to leave the packages unrun, since nothing outside the change directory is modified. + +Integration is jj-native onto the existing `harborize-instrument` bookmark, whose four commits are the 0.2.1 instrument this change validates and which is already a parent of the development join. +Isolation is the development join rather than a worktree. + +## Open Questions + +These three are open and are recorded rather than resolved. +Each carries its options and what each option costs, so a later session can decide rather than infer. + +### Where pi and opencode actually discover skills + +Harbor and BenchFlow declare different destinations for the same agent. +Harbor's pi adapter writes `$HOME/.agents/skills//` (`pi.py:75-83`) while BenchFlow's registry declares `["$HOME/.pi/agent/skills", "$HOME/.agents/skills"]` for `pi-acp` (benchflow `src/benchflow/agents/registry.py:560`), a superset. +Harbor's opencode adapter writes `~/.config/opencode/skills//` (`opencode.py:425-433`) while BenchFlow declares `["$HOME/.opencode/skills"]` (`registry.py:700`), which does not overlap. +Codex is the one cell where both agree, at `$HOME/.agents/skills`. + +Three resolutions, in increasing cost. +Declare those two cells' cross-runner numbers non-comparable, which costs nothing and forfeits two cells of the grid. +Clone the three CLIs and read which path each discovers, which costs reading time and settles the question statically for the pinned CLI versions. +Run a behavioral canary per cell, which costs one metered trial per adapter and settles it for the versions actually installed in the image. +This change takes the first option for its own reporting and leaves the other two to the dependent change. + +### Whether the oracle rung adopts BenchFlow's acceptance level + +BenchFlow ships an `acceptance` validation level whose evidence block requires `oracle_runs.required_reward` to be numeric and at least 0.99 (benchflow `src/benchflow/_utils/task_authoring/acceptance_evidence.py:104-107`), `verifier.reruns` to be an integer of at least 3, and `verifier.flake_rate` to be numeric and at most 0.05 (`:172-178`). +It overlaps the bespoke five-times oracle loop the instrument's inhabitation invariant already requires, and it is a declared evidence block rather than an executed check. + +The cost of adopting it is that it has zero adoption across the corpus: no task under `tasks/` declares a `benchflow.evidence` block, so the block would have to be written from the validator source rather than copied from a worked example, and its field names and shapes would be inferred from `acceptance_evidence.py` alone. +The cost of not adopting it is that the package cannot claim the level, and a later publication-grade or acceptance run has to add the block anyway. +Left open; the ladder's own five-times criterion is what this change enforces either way. + +### Whether the leakage audit should exempt an integrity canary + +Any delivery canary asserting a token places the same literal in the verifier and in the SKILL.md, which `audit_leakage.py` check 1 flags by construction. +The flag is correct: the check searches for a verifier expectation recoverable from skill content, and here it is recoverable by design. +Adding an exemption is an instrument modification and D5 forbids it during this round. + +The options for the next revision are to add an explicit canary exemption keyed on a package-README declaration, to leave the flag and require the README to carry the justification, or to redesign the canary so the asserted literal is generated at fixture-build time and never appears in the skill. +The third is the only one that removes the flag without weakening the audit, and it costs a fixture-generation step in the canary's environment Dockerfile. +This change takes the second option for now and records the choice in the package README. diff --git a/openspec/changes/validate-harborize-instrument/plan.md b/openspec/changes/validate-harborize-instrument/plan.md new file mode 100644 index 000000000..bdabc7a81 --- /dev/null +++ b/openspec/changes/validate-harborize-instrument/plan.md @@ -0,0 +1,766 @@ +# validate-harborize-instrument Implementation Plan + +> **For agentic workers:** REQUIRED SUB-SKILL: Use superpowers:subagent-driven-development (recommended) or superpowers:executing-plans to implement this plan task-by-task. Steps use checkbox (`- [ ]`) syntax for tracking. + +**Goal:** Prove that harborize 0.2.1 produces evaluation packages whose skill injection is delivered end to end, leave a permanent injection canary behind, and measure the per-run cost constant. + +**Architecture:** Seven rungs run in order, six of them free of model spend. +The free rungs retire every silent-null class except adapter registration, which is unreachable without a real agent invocation and is therefore the single metered rung, reduced to one short trial per cell. +Each package is authored BenchFlow-native and exported to a sibling Harbor head, because three independent validators forbid the co-present layout. + +**Tech Stack:** Docker via OrbStack; the Harbor CLI and the BenchFlow CLI installed with `uv tool install`; python 3.12 for host-side checks; the frozen harborize instrument's bundled scripts (`design_matrix.py`, `audit_leakage.py`, `materialize_conditions.py`). + +## Global Constraints + +- Instrument version is `0.2.1` and the harborize directory at `modules/home/ai/plugins/testing-and-quality/.apm/skills/harborize/` MUST NOT be modified for the duration of this change. + A defect found mid-change is recorded and deferred. +- Upstream pins, and every anchor is valid only at these revisions: harbor `ac398bbda7c4c1073461797d3b95c2455cc671b5`, benchflow `d30527b82027a416e72014920cdf43a534967ad3`, skillsbench `9a1f4dd5f7659f75707435da3ce854b6e48321d1`. +- The three ghq clones at `~/ghq/github.com/harbor-framework/harbor`, `~/ghq/github.com/benchflow-ai/benchflow` and `~/ghq/github.com/benchflow-ai/skillsbench` are read-only reference trees. + Never run `uv sync` inside them; install the CLIs from PyPI latest stable with `uv tool install` (settled 2026-08-15 — installs track PyPI releases, not sha-pinned source checkouts), so the clones are never install sources, only reading trees for the pinned anchors. +- Every package carries a single binary reward key. + A multi-dimensional rubric silently disables Harbor's pass@k and makes BenchFlow's compare-lift count only `reward == 1.0` as passed. +- Every Harbor run line sets `HARBOR_TELEMETRY=0`. + Every metered run scrubs `ANTHROPIC_OAUTH_TOKEN`. +- Every package declares `network_mode = "public"` at its environment baseline and confines `no-network` to the `[agent]` phase, because the baseline governs container creation while the phase policy wraps only the agent run. + `no-network` is the harborize instrument's authoring default, not Harbor's; Harbor's default is `public`. +- Package corpus path is `modules/home/ai/evals/harborize/` in this repository, tracked in version control. + Job output goes to `logs/harborize/`, which is already gitignored (`.gitignore:57`). + The path is settled and needs no further confirmation: the corpus is co-located with the skills it evaluates, which live under `modules/home/ai/plugins/`, and the location was verified safe against every automated surface that walks `modules/` — import-tree, treefmt, the `naming-conventions` check, and the apm skill composition. +- No file inside the corpus may carry the `.nix` extension. + `flake.nix:6` calls `inputs.import-tree ./modules` bare with no custom filter, so import-tree's default `nixFilter = andNot (hasInfix "/_") (hasSuffix ".nix")` (import-tree `default.nix:50`, rev `4ebb10ae17d5f1ad366e7aef5b92cb8eecf24f69`) imports every `*.nix` file anywhere under `modules/` and evaluates it as a flake-parts module. + A fixture named `expected.nix` is the realistic trap, and it would break the flake rather than fail as a fixture. + Two escapes exist and either one is sufficient: **never give a fixture the `.nix` extension**, or place it under a `_`-prefixed directory, which is this repository's documented exclusion convention (ADR-0018, `packages/docs/src/content/docs/development/architecture/adrs/0018-deferred-module-composition-architecture.md:311`). + Everything else is safe by construction, because non-nix files are enumerated and then dropped before anything reads them with no warning and no error, which is why `modules/` already carries hundreds of non-nix files — 433 at `origin/main` `25d10e7c` and 480 at this change's tip, of which 34 are this corpus — and the flake still evaluates. + This constraint is checked rather than trusted: Task 1 Step 7 runs the flake-evaluation guard as soon as the corpus root exists, and Task 5 Step 8 re-runs it with an extension audit once the package directories and the generated Harbor heads exist. +- Integration is jj-native onto the existing `harborize-instrument` bookmark, which is already a parent of the development join. + Routing a change onto that chain is orchestrator-owned; an implementing subagent leaves its work in the working copy and names what it touched. + +## File Structure + +``` +modules/home/ai/evals/harborize/ +├── README.md # pins, instrument version, probe result, caveats +├── cells/cells.json # cell definitions consumed by design_matrix.py +├── cells/cells-nonconsuming.json # rung 1 negative arm: a non-consuming adapter +├── cells/cells-acp.json # rung 1 negative arm: an acp: registry shorthand +├── checks/resolve_check.py # rung 2 probe; run with the harbor entrypoint's python +├── conditions/canary/ # dir(C): exactly one skill directory +│ └── harborize-injection-canary/SKILL.md +├── injection-canary/ # authored BenchFlow-native, shared verifier +│ ├── task.md +│ ├── environment/Dockerfile +│ ├── environment/skills/.gitkeep # directory stays EMPTY; .gitkeep keeps it tracked +│ ├── oracle/solve.sh +│ └── verifier/test.sh # no verifier/Dockerfile: shared mode ships none +├── injection-canary-harbor/ # generated by bench tasks export; never hand-edited +├── / , -harbor/ +└── results/cost-constant.json # written by Task 10 Step 2; absent until then +logs/harborize/ # gitignored job trees +├── canary-bench/ +└── gate1/ +``` + +Each package directory is self-contained: the authored native tree is the source of truth and its Harbor sibling is a build product re-derived after every edit. +The condition directory lives outside every package, because a populated `environment/skills/` bakes one condition into the image. +`environment/skills/` carries a `.gitkeep` because neither git nor jj tracks an empty directory, and the emptiness is the load-bearing property: it is what proves no condition is baked into the image. +BenchFlow reads that directory as a bundled skills source through `task_bundled_skills_dir` (`skill_policy.py:71`), so an accidentally populated one silently becomes a condition. + +--- + +## Task 1: Prerequisites and the blocking kernel probe + +**Files:** +- Create: `modules/home/ai/evals/harborize/README.md` + +**Interfaces:** +- Consumes: nothing. +- Produces: a running Docker daemon, `harbor` and `bench` on `PATH`, and a README recording the probe result that every later task's network mode depends on. + +- [ ] **Step 1: Start the Docker daemon** + +```bash +open -a OrbStack +docker info +``` + +Expected: `docker info` exits 0 and prints a server block. +It currently fails, because OrbStack is stopped and the socket is absent. + +- [ ] **Step 2: Run the blocking kernel probe before authoring anything** + +```bash +docker run --rm alpine sh -c ' + [ -f /proc/config.gz ] || exit 2 + zcat /proc/config.gz | grep -qE "^CONFIG_NFT_FIB_INET=[ym]"' +echo "probe exit: $?" +``` + +Read the exit status, not the output. +The anchor and `-q` are load-bearing: an unanchored `grep NFT_FIB_INET` matches the line `# CONFIG_NFT_FIB_INET is not set` and exits 0, so the negative reading would pass a scripted check. + +Branch on three outcomes rather than halting on any non-zero. +Exit 0 means egress control is available and `no-network` may be declared wherever this plan declares it. +Exit 1 means the option is explicitly unset, so drop the `[agent]` phase override from Task 5 Step 4, leave every package fully `public`, and record the loss of egress control in the workspace README. +Exit 2 means `/proc/config.gz` is absent, which is indeterminate rather than negative: Harbor's own probe short-circuits to exit 0 in exactly that case (`environments/docker/docker.py:113-117`), so Harbor proceeds and this plan proceeds with it, treating a later `no-network` rejection as the deciding evidence. + +What the probe gates is enforcement, not the default. +`no-network` is the harborize instrument's authoring default (`SKILL.md:124`, `references/emitters.md:74`), while Harbor's own default is `public` (`models/task/config.py:249-252`, `NetworkPolicy` at `:66`), BenchFlow's is the same (`task/config.py:720-723`), and 86 of the 87 SkillsBench corpus tasks declare `network_mode: public`. +When the option is missing, `_enable_egress_control` goes false (`docker.py:188-195`), which zeroes `capabilities.disable_internet` (`:289-293`), and `environments/base.py:773-781` raises at environment start for any `no-network` policy — baseline or phase. + +- [ ] **Step 3: Install the two CLIs from PyPI latest stable** + +```bash +uv tool install harbor +uv tool install benchflow +harbor --version +bench --version +``` + +Expected: both commands print a version, and neither ghq clone is read or written during the install. +Settled 2026-08-15: the installs track PyPI, and the Global Constraints' pins remain reading pins for source anchors, so record both installed versions in the workspace README — every executed-command anchor in Tasks 6 through 9 is valid at the installed version, not at the pin. + +- [ ] **Step 4: Verify the reference clones are unmodified** + +```bash +git -C ~/ghq/github.com/harbor-framework/harbor status --porcelain +git -C ~/ghq/github.com/benchflow-ai/benchflow status --porcelain +``` + +Expected: both print nothing. + +- [ ] **Step 5: Record the instrument-freeze baseline** + +```bash +jj log --ignore-working-copy -r @ -T 'change_id.short()' +git rev-parse 'HEAD:modules/home/ai/plugins/testing-and-quality/.apm/skills/harborize' +find modules/home/ai/plugins/testing-and-quality/.apm/skills/harborize -type f \ + -not -path '*/__pycache__/*' -exec shasum -a 256 {} + | LC_ALL=C sort | shasum -a 256 +``` + +Write both values into the workspace README before any other task runs. +Task 10 Step 4 recomputes the digest and compares it, because a `jj diff -r @` reports only what the working-copy commit changed against its parents and would show nothing for an edit squashed into the `harborize-instrument` chain this plan routes onto. + +- [ ] **Step 6: Write the workspace README** + +Record, one line each: the three upstream pins; instrument version 0.2.1; the probe exit status from Step 2 with its branch and its date; the freeze baseline from Step 5; the attribution that `no-network` is the instrument's authoring default rather than Harbor's; and the claude-code install caveat. +The caveat is that `install()` curls its bootstrap (`claude_code.py:425-449`) during `_prepare`/`_setup_agent` (`trial.py:408-414`), which no network policy wraps, while only `_run_agent_phase` (`trial.py:465-469`) and the verifier phases enter `_phase_network_policy`. +A cell whose environment baseline is `no-network` therefore fails during agent install indistinguishably from an injection failure at the reward level, which is why Task 5 Step 4 puts `no-network` on the agent phase and never on the baseline. + +- [ ] **Step 7: Run the flake-evaluation guard now that the corpus root exists under `modules/`** + +```bash +if fd -H -e nix . modules/home/ai/evals/harborize | rg -q .; then + echo "FAIL: a .nix file is inside the corpus"; exit 1 +fi +nix eval .#nixosConfigurations --apply builtins.attrNames +``` + +Expected: the audit finds nothing and the eval exits 0 printing the attribute list. +Step 6 writes the first corpus file under `modules/`, so this is the earliest point at which the Global Constraints' `.nix` prohibition can be violated, and the guard is what turns that constraint from a comment into a check. +The eval is the cheapest expression that forces import-tree to walk `modules/` and construct the module set. +`nix flake check` is the stronger form and is equivalent for this purpose, so prefer it when the wait is acceptable; do not add `--no-build`, which exits 1 on an unrelated import-from-derivation in this repository and would read as a guard failure. +The audit is written as an explicit `if` rather than a negated pipeline, because bash exempts a `!`-inverted command from `set -e` and an absence assertion written that way is a no-op. + +- [ ] **Step 8: Check off tasks.md §1 and hand the working copy back for routing onto `harborize-instrument`** + +--- + +## Task 2: Canary skill and condition directory + +**Files:** +- Create: `modules/home/ai/evals/harborize/conditions/canary/harborize-injection-canary/SKILL.md` + +**Interfaces:** +- Consumes: Task 1's workspace. +- Produces: `dir(C)` at `modules/home/ai/evals/harborize/conditions/canary/`, containing exactly one skill directory. The literal token `HARBORIZE-CANARY-9F3A21` is the value Task 5's oracle extracts and Task 5's verifier asserts. + +- [ ] **Step 1: Write the canary skill** + +```markdown +--- +name: harborize-injection-canary +description: Injection canary for the harborize instrument. Carries a single token that a canary task's oracle extracts and its verifier asserts, so a failure to deliver injected skills fails a task rather than silently voiding a batch. +--- + +# Harborize injection canary + +This skill exists to be delivered, not to be used. + +The canary token is HARBORIZE-CANARY-9F3A21 and it appears nowhere else in the deployed tree. +``` + +The name must not collide with any of the 172 deployed skill directories; confirm with `ls ~/.claude/skills | grep -c harborize-injection-canary` returning 0. + +- [ ] **Step 2: Verify dir(C) contains skill directories and nothing else** + +```bash +ls -A modules/home/ai/evals/harborize/conditions/canary/ +``` + +Expected: exactly `harborize-injection-canary`. +Harbor's `_find_skill_dirs` raises on a root holding a non-hidden child directory without a `SKILL.md` (`skills.py:382-416`), so a stray directory turns the whole condition into a hard error. + +- [ ] **Step 3: Record the leakage-audit expectation in the canary README section** + +Write into `modules/home/ai/evals/harborize/README.md` that the token appears in both the canary `SKILL.md` and the canary verifier by design, that `audit_leakage.py` check 1 flags exactly that pattern (`MIN_LITERAL_LENGTH = 8` at `:44`, `check_literals` at `:96`), that the flag is correct on its own terms, and that the instrument is not edited to exempt it because D5 freezes it at 0.2.1. +Record in the same place that the consequence originally anticipated does not arise: both runners upload the verifier's own directory during the verification phase, after the agent phase has ended (Harbor `verifier/verifier.py:147-153`, phase order `trial/single_step.py:41` then `:52`; BenchFlow `task/verifier_core.py:385` in `_verify_test_script` (`:346`), reached from `verify()` (`:260`)), so no agent phase observes `test.sh` under either fork. +That costs nothing, because the metered rung's pass criterion is the adapter's registration directory rather than the reward, and the falsifiability control in Task 7 Step 3 runs under the oracle, which greps skill directories and never reads the verifier. + +- [ ] **Step 4: Check off tasks.md §2 and hand the working copy back for routing** + +--- + +## Task 3: Adapter allowlist gate + +**Files:** +- Create: `modules/home/ai/evals/harborize/cells/cells.json` +- Create: `modules/home/ai/evals/harborize/cells/cells-nonconsuming.json` +- Create: `modules/home/ai/evals/harborize/cells/cells-acp.json` + +**Interfaces:** +- Consumes: Task 1's CLIs are not needed here; this rung is pure python over the frozen instrument's script. +- Produces: a validated `cells.json` that later rungs pass to `design_matrix.py`. + +- [ ] **Step 1: Write the negative-control cells file naming a non-consuming adapter** + +```json +[{"name": "neg-aider", "runner": "harbor", "agent": "aider", "model": "anthropic/claude-opus-5"}] +``` + +`aider` is in the non-consuming set: of the 39 agents Harbor's factory registers, 22 read the injected skills directory and 17 do not. + +- [ ] **Step 2: Run the gate against it and confirm it refuses** + +```bash +SKILL=modules/home/ai/plugins/testing-and-quality/.apm/skills/harborize +python3 "$SKILL/scripts/design_matrix.py" \ + --units harborize-injection-canary --design marginals \ + --cells modules/home/ai/evals/harborize/cells/cells-nonconsuming.json --out /tmp/design-neg +``` + +`--units`, `--design` and `--cells` are all required, and without `--from-census` the unit names are free strings, so this is the smallest invocation that reaches the gate. +`load_cells` calls `check_harbor_agents`, so no caller reaches a manifest without passing it. +Expected: a nonzero exit naming `aider` and the reason, with `/tmp/design-neg` never written. + +- [ ] **Step 3: Write the second negative control naming an ACP shorthand** + +```json +[{"name": "neg-acp", "runner": "harbor", "agent": "acp:claude-agent", "model": "anthropic/claude-opus-5"}] +``` + +Run the same command with `--cells modules/home/ai/evals/harborize/cells/cells-acp.json`. +Expected: a nonzero exit, because `factory.py:167-175` routes every `acp:`-prefixed name through the non-consuming ACP adapter, so the Harbor arm drops skills while a BenchFlow arm for the same agent works. + +- [ ] **Step 4: Write the real cells file and confirm it passes** + +```json +[{"name": "codex-sol", "runner": "harbor", "agent": "codex", "model": "gpt-5.6-sol", "env": {"HARBOR_TELEMETRY": "0", "CODEX_FORCE_AUTH_JSON": "1"}}] +``` + +The cell is codex on the ChatGPT-subscription path (settled decision; the metered adapter, its auth variables, and the model naming are recorded on-branch at the instrument's `references/marketplace-program.md:46`), which is why the env block carries `CODEX_FORCE_AUTH_JSON=1` and no Anthropic variables; the exact Harbor model string is confirmed against the installed codex adapter when rung 6 runs. + +Run the same command with `--cells modules/home/ai/evals/harborize/cells/cells.json --out /tmp/design-ok`. +Expected: exit 0 and a printed conditions-and-jobs summary. +The `--out` path is deliberately throwaway. +`design_matrix.py:442-466` writes `conditions.json`, `manifest.sh` and `jobs.json` there, and nothing this rung emits is executed, committed or consumed by a later rung, which is what keeps the whole rung inside the proposal's condition-lattice Non-goal. +`load_cells` runs at `:440`, before `out.mkdir` at `:442`, so a refused cells file leaves the directory uncreated and the negative controls above assert exactly that. + +- [ ] **Step 5: Check off tasks.md §3 and hand the working copy back for routing** + +--- + +## Task 4: Host-side resolution + +**Files:** +- Create: `modules/home/ai/evals/harborize/checks/resolve_check.py` + +**Interfaces:** +- Consumes: `dir(C)` from Task 2. +- Produces: recorded evidence that the condition directory resolves on the host, explicitly labelled resolution and request rather than delivery. + +- [ ] **Step 1: Write the resolution check** + +```python +import json, sys +from harbor.skills import compute_skill_digest, resolve_skills + +root = sys.argv[1] +resolved = resolve_skills([root]) +print(json.dumps([ + {"name": s.name, "source": str(s.source), "digest": compute_skill_digest(s.source)} + for s in resolved +], indent=2)) +``` + +- [ ] **Step 2: Run it against dir(C)** + +```bash +HARBOR_PY=$(sed -n '1s|^#!||p' "$(command -v harbor)") +"$HARBOR_PY" modules/home/ai/evals/harborize/checks/resolve_check.py modules/home/ai/evals/harborize/conditions/canary +``` + +Expected: exactly one entry, `name` equal to `harborize-injection-canary`, and a `digest` beginning `sha256:`. +`resolve_skills` is `skills.py:111-123` and `compute_skill_digest` is `skills.py:200-209`, which is pure over the directory. + +- [ ] **Step 3: Run the three malformed-root controls** + +```bash +HARBOR_PY=$(sed -n '1s|^#!||p' "$(command -v harbor)") +"$HARBOR_PY" modules/home/ai/evals/harborize/checks/resolve_check.py /nonexistent/path +"$HARBOR_PY" modules/home/ai/evals/harborize/checks/resolve_check.py modules/home/ai/evals/harborize/README.md +mkdir -p /tmp/badcond/not-a-skill && "$HARBOR_PY" modules/home/ai/evals/harborize/checks/resolve_check.py /tmp/badcond +``` + +Expected: `FileNotFoundError`, then `ValueError: Skill path must be a directory`, then `ValueError` naming the child directory without a `SKILL.md`, each raised on the host before any container starts (`_find_skill_dirs`, `skills.py:382-416`). + +- [ ] **Step 4: Record what this rung does and does not prove** + +Write into the workspace README that this is host-side resolution only, and that a trial's `lock.json` cannot substitute for delivery evidence: `_write_trial_lock` runs at `trial.py:104` inside `Trial.__init__`, before `_resolve_injected_skills` at `:107` and long before `_upload_injected_skills` at `:411`, and `_build_agent_skill_locks` (`models/job/lock.py:462-475`) calls only host-side functions. + +- [ ] **Step 5: Check off tasks.md §4 and hand the working copy back for routing** + +--- + +## Task 5: Package authoring, dual head + +**Files:** +- Create: `modules/home/ai/evals/harborize/injection-canary/task.md` +- Create: `modules/home/ai/evals/harborize/injection-canary/environment/Dockerfile` +- Create: `modules/home/ai/evals/harborize/injection-canary/environment/skills/.gitkeep` +- Create: `modules/home/ai/evals/harborize/injection-canary/oracle/solve.sh` +- Create: `modules/home/ai/evals/harborize/injection-canary/verifier/test.sh` +- Create: `modules/home/ai/evals/harborize//` with the same shape +- Generate: `modules/home/ai/evals/harborize/injection-canary-harbor/`, `modules/home/ai/evals/harborize/-harbor/` + +**Interfaces:** +- Consumes: the token `HARBORIZE-CANARY-9F3A21` from Task 2. +- Produces: two authored native trees and two exported Harbor heads for Tasks 6, 7 and 8. +- Acceptance condition: no file in either authored tree or either exported head carries the `.nix` extension, per the Global Constraints, and the flake still evaluates afterwards. + This is the first task that creates package directories, and two of the four are written by `bench tasks export` rather than authored here, so the audit runs against the generated heads as well and is repeated after every re-export. + +- [ ] **Step 1: Write the canary oracle** + +```bash +#!/bin/bash +set -euo pipefail +out=/logs/artifacts/canary-output.txt +mkdir -p /logs/artifacts +: > "$out" +for root in /harbor/skills "$HOME/.claude/skills" "$HOME/.codex/skills" \ + "$HOME/.opencode/skills" "$HOME/.agents/skills" /skills; do + [ -d "$root" ] || continue + for f in "$root"/*/SKILL.md; do + [ -f "$f" ] || continue + grep -ho 'HARBORIZE-CANARY-[A-Z0-9]*' "$f" >> "$out" || true + done +done +sort -u -o "$out" "$out" +``` + +The search covers Harbor's upload destination, which defaults to `/harbor/skills` (`models/trial/paths.py:41`) and is populated for every agent by `_upload_injected_skills` in `_prepare` (`trial.py:411`), BenchFlow's five oracle discovery paths (`agents/install.py:30-36`), and BenchFlow's sandbox mount at `/skills`. + +The output path is `/logs/artifacts/`, not `/logs/verifier/` and not the agent workspace, and the choice is load-bearing. +The original choice of `/logs/verifier/` was read off Harbor alone and the rung-4 run refuted it: BenchFlow clears that directory's contents on the agent container immediately before the verifier runs, unconditionally (`sandbox/lockdown.py:775-784`, called from `harden_before_verify` at `:1205-1212`), so the canary written that way scored 1 under Harbor and 0 under BenchFlow with no diagnostic beyond a `grep` miss. +`/logs/artifacts/` is bind-mounted for the whole trial by both runners (`sandbox/docker.py:186-201`; `models/trial/paths.py:38`, `:195-202`), no hardening step touches it, and it is the one path that would also survive Harbor's separate fork, whose wipe at `trial.py:599` is followed by an artifact re-upload (`trial.py:601-607`). +`/logs/verifier/` keeps its one role under both runners: the reward file the verifier writes after that clear. + +- [ ] **Step 2: Fix the canary's verifier fork as shared, and write the verifier** + +The canary ships no `verifier/Dockerfile` and declares no `verifier.sandbox_mode`, so it runs shared. +The fork is forced rather than preferred, and both runners force it. BenchFlow refuses to launch a package declaring separate rather than falling back to shared: `runtime_capabilities.py:186-192` raises an unsupported-feature issue whose reason string reads "separate verifier sandboxes are parsed but not executed", and `raise_for_task_runtime_support` is a fail-closed pre-launch gate (`sandbox/setup.py:676`, `:819-842`). +Harbor's separate verifier empties `/logs/verifier` before running (`trial.py:599`), through the same host bind it mounts at `:686-692`, so no reward-adjacent state written before verification survives it. +Both packages therefore run shared, and separate mode is exercised nowhere in this change; D11 records why. + +```bash +#!/bin/bash +mkdir -p /logs/verifier +if grep -qx 'HARBORIZE-CANARY-9F3A21' /logs/artifacts/canary-output.txt; then + echo 1 > /logs/verifier/reward.txt +else + echo 0 > /logs/verifier/reward.txt +fi +``` + +A single binary reward key, written to the only output channel either runner reads. +Harbor parses `reward.txt` into a one-key `rewards` dict whose key is the literal `reward` (`verifier/verifier.py:73`), which is what keeps pass@k enabled (`utils/pass_at_k.py:36-52`). + +- [ ] **Step 3: Write the canary `task.md` frontmatter and body** + +Frontmatter carries `schema_version: '1.3'`, a `metadata` block, `verifier.type: test-script`, `agent.timeout_sec`, and the `sandbox` spec per Step 4. +The body states the end state — that `/logs/artifacts/canary-output.txt` holds the canary token — in imperative prose with absolute paths, and never mentions skills. +The body does not name the token: only the injected `SKILL.md` carries it, which is what makes the oracle's pass evidence of delivery. + +- [ ] **Step 4: Declare the network policy on every package** + +In the native `task.md` frontmatter, using BenchFlow's own `sandbox` spelling: + +```yaml +sandbox: + network_mode: public # baseline: what the container is created with +agent: + network_mode: no-network # phase override; drop this key on probe exit 1 +``` + +`bench tasks export` renames `sandbox` to Harbor's `[environment]` on the way out (`task/export.py:383-387`) and leaves `[agent]` as it is, so the exported head reads `[environment] network_mode = "public"` with `[agent] network_mode = "no-network"`. + +The baseline is what the container is created with (`trial.py:896`) and the phase policy is applied around `_run_agent_phase` alone (`trial.py:465-469`), so a `no-network` baseline breaks claude-code's install fetch during `_prepare` while a `no-network` agent phase does not. +Harbor reads the override through `task_cfg.agent.explicit_phase_policy()` (`trial/network_policy.py:45-59`) and validates that the environment can switch policy after start (`trial.py:203-217`); BenchFlow carries the same field at `task/config.py:524-530`. +Drop the `[agent]` table entirely when Task 1 Step 2 returned exit 1, because the switch would then be rejected at environment start. + +- [ ] **Step 5: Write the mechanical package the same way** + +Pick a task whose success is a decidable predicate on final state, sampling the claimed contract of the skill under test rather than that skill's own examples. +Keep the reward binary. +Record the verifier fork in the package README with its justification, and record it as what the tree declares rather than as what its layout suggests. +The declaration is `verifier.sandbox_mode` in the native frontmatter, which the exporter renames to Harbor's `verifier.environment_mode` (`benchflow/task/config.py:83-94`, applied at `task/export.py:383-387`); a `verifier.sandbox` block exports to `[verifier.environment]` and also implies separate. +Harbor resolves the mode from those two keys and from nothing else (`models/task/verifier_mode.py:10-21`), so a `verifier/Dockerfile` alone infers nothing and the package resolves to `SHARED`. +This package's fork is shared, for the two reasons in Step 2: BenchFlow refuses to launch a package declaring separate, and Harbor's separate verifier empties `/logs/verifier`, which is why neither package puts a deliverable there. +Verify the recorded fork against the exported head — `resolve_task_verifier_mode(Task().config)` — rather than against the source directory listing. + +- [ ] **Step 6: Export both Harbor heads** + +```bash +bench tasks export modules/home/ai/evals/harborize/injection-canary modules/home/ai/evals/harborize/injection-canary-harbor \ + --target harbor --overwrite +``` + +`--overwrite` is required from the second export onwards: `export.py:255-256` raises `FileExistsError` on an existing destination, and the CLI exposes the flag at `cli/tasks.py:294-297`. + +Expected: the sibling directory is written with `task.toml`, `instruction.md`, `solution/solve.sh`, `tests/`, `environment/` and `compatibility/export-report.json`. +Read the export report and record anything that did not survive the conversion. +Never hand-edit the derived head, and re-export after every edit to the native tree. + +- [ ] **Step 7: Run the leakage audit on both packages** + +```bash +SKILL=modules/home/ai/plugins/testing-and-quality/.apm/skills/harborize +python3 "$SKILL/scripts/audit_leakage.py" --task modules/home/ai/evals/harborize/ --skills +python3 "$SKILL/scripts/audit_leakage.py" --task modules/home/ai/evals/harborize/injection-canary --skills modules/home/ai/evals/harborize/conditions/canary/harborize-injection-canary +``` + +Expected: exit 0 on the mechanical package; exit 1 on the canary with a check-1 literal flag naming the token. +Exit 2 means the audit could not run, which is a mis-invocation rather than a clean package. +Record the canary's flag with the justification from Task 2 Step 3. + +- [ ] **Step 8: Audit the four package directories for the `.nix` extension and re-run the flake-evaluation guard** + +```bash +if fd -H -e nix . modules/home/ai/evals/harborize | rg -q .; then + echo "FAIL: a .nix file is inside the corpus"; exit 1 +fi +nix eval .#nixosConfigurations --apply builtins.attrNames +``` + +Expected: the audit finds nothing and the eval exits 0. +Run it after Step 6's export rather than only after Step 5's authoring, because the two Harbor heads are generated and an extension the export introduces would otherwise reach the flake unaudited. +Re-run it after any later re-export. + +- [ ] **Step 9: Stamp both package READMEs with instrument version 0.2.1 and the three upstream pins, then check off tasks.md §5 and hand the working copy back for routing** + +--- + +## Task 6: Static task validation + +**Files:** +- Create: `modules/home/ai/evals/harborize/checks/harbor_schema_check.py` + +**Interfaces:** +- Consumes: the four package directories from Task 5. +- Produces: a clean static gate per head, and a recorded statement of what the Harbor gate cannot catch. + +- [ ] **Step 1: Run the BenchFlow gate against each authored native tree** + +```bash +bench tasks check modules/home/ai/evals/harborize/injection-canary --level structural +bench tasks check modules/home/ai/evals/harborize/ --level structural +``` + +Expected: exit 0 with no issues. +The levels are schema, structural (default), runtime-capability, publication-grade, acceptance and acceptance-live (benchflow `cli/tasks.py:93-115`), and nothing below acceptance-live spends a model call. + +- [ ] **Step 2: Write the Harbor schema check** + +```python +import sys +from harbor.models.task.task import Task + +task = Task(sys.argv[1]) +print("constructed:", task.paths.task_dir) +``` + +- [ ] **Step 3: Run it against each exported head** + +```bash +python3 modules/home/ai/evals/harborize/checks/harbor_schema_check.py modules/home/ai/evals/harborize/injection-canary-harbor +python3 modules/home/ai/evals/harborize/checks/harbor_schema_check.py modules/home/ai/evals/harborize/-harbor +``` + +Expected: both print `constructed:` and exit 0. +Do not call `harbor task check` or `harbor tasks check`: both spellings reach one command that prints an error and raises `SystemExit(1)` unconditionally (`cli/tasks.py:476-487`). +Do not follow that stub's own redirect either. +It prints "Use 'harbor check ' instead" (`cli/tasks.py:483-486`), and `harbor check` is a metered LLM-rubric run defaulting to `claude-code` and `claude-sonnet-4-6` (`cli/main.py:160`, `cli/analyze.py:100-103`), so obeying the CLI would spend money inside a rung this plan calls free. + +- [ ] **Step 4: Record the gate's blind spot** + +Write into the workspace README that `Task._validate_tests` returns early whenever a verifier environment is configured (`models/task/task.py:126-144`, early return at `:134-135`), so a separate-mode package whose verifier image does not own `/tests/test.sh` passes schema validation and fails at run time. + +- [ ] **Step 5: Check off tasks.md §6 and hand the working copy back for routing** + +--- + +## Task 7: Delivery proof under BenchFlow + +**Files:** +- Modify: `modules/home/ai/evals/harborize/README.md` (record the rung's evidence) + +**Interfaces:** +- Consumes: the canary package from Task 5 and `dir(C)` from Task 2. +- Produces: end-to-end delivery evidence with zero model calls, which is the precondition for any metered run using that `dir(C)` shape. + +- [ ] **Step 1: Run the canary under the oracle agent** + +```bash +bench eval run --tasks-dir modules/home/ai/evals/harborize/injection-canary --agent oracle \ + --skill-mode with-skill --skills-dir modules/home/ai/evals/harborize/conditions/canary \ + --sandbox docker --jobs-dir logs/harborize/canary-bench +``` + +Expected: the rollout reaches the agent phase and reward equals 1. +No model is materialized: `evaluation.py:461-462` returns None for the oracle agent. +The deployment path still runs in full: `rollout/__init__.py:1160` takes the `primary_agent == "oracle"` branch and calls `deploy_skills` at `:1174`. + +- [ ] **Step 2: Confirm the fidelity assertion passed and the effective directory is the host path** + +```bash +rg -c 'skill_deployment_missing' logs/harborize/canary-bench || echo "clean" +rg -o '"effective_skills_dir":[^,]*' logs/harborize/canary-bench -r '$0' | sort -u +``` + +Expected: `clean`, and every `effective_skills_dir` equal to the host `dir(C)` that was passed. +The assertion runs in the container: `deploy_skills` computes the expected catalogue on the host from `Path(skills_dir).glob("*/SKILL.md")` (`agents/install.py:303`, `:313-314`), falls back to the five `_ORACLE_SKILL_PATHS` (`:30-36`) at `:349-350`, and `_link_skill_paths` compares the in-container catalogue to the host's (`:146-161`) before raising at `:176-180`. + +- [ ] **Step 3: Run the falsifiability control** + +```bash +bench eval run --tasks-dir modules/home/ai/evals/harborize/injection-canary --agent oracle \ + --skill-mode no-skill \ + --sandbox docker --jobs-dir logs/harborize/canary-bench-negative +``` + +Expected: reward 0, because the token never reaches the container. +`no-skill` cannot be combined with `--skills-dir`, so the empty condition is the mode alone. +This is the canary's proof that it can fail, which is what makes its passing run evidence rather than a constant. +It is explicitly not the fidelity control: with no `skills_dir`, `expected_skill_names` is the empty tuple (`agents/install.py:313-317`) and the `experiment_fidelity/skill_deployment_missing` raise is guarded on `if expected:` (`:176-180`), so this run cannot raise it. + +- [ ] **Step 4: Run the fidelity control against a throwaway sibling** + +The fidelity assertion compares the in-container skill catalogue against the host-computed one, so no edit to `dir(C)` alone separates them: `_skill_link_cmd` replaces each discovery path with a symlink to the uploaded source (`agents/install.py:90`), which makes the `actual == source_catalog` test at `:157` tautological, leaving only the `source_catalog == expected` test at `:159-161`. +The one author-constructible divergence is an image that already carries a different condition. + +```bash +cp -R modules/home/ai/evals/harborize/injection-canary /tmp/canary-fidelity-control +mkdir -p /tmp/canary-fidelity-control/environment/_deps/skills/decoy-skill +printf -- '---\nname: decoy-skill\ndescription: decoy\n---\n' \ + > /tmp/canary-fidelity-control/environment/_deps/skills/decoy-skill/SKILL.md +printf 'COPY _deps/skills /skills/\n' \ + >> /tmp/canary-fidelity-control/environment/Dockerfile +bench eval run --tasks-dir /tmp/canary-fidelity-control --agent oracle \ + --skill-mode with-skill --skills-dir modules/home/ai/evals/harborize/conditions/canary \ + --sandbox docker --jobs-dir logs/harborize/canary-bench-fidelity +``` + +Expected: a raised `experiment_fidelity/skill_deployment_missing` naming `harborize-injection-canary` as the expected set. +`deploy_skills` finds the `COPY _deps/skills /skills/` line and takes the `already_injected` branch (`agents/install.py:328-332`, `:342-344`), performs no runtime upload, so `/skills` carries `decoy-skill` while `expected` still carries the host `dir(C)`. +Delete `/tmp/canary-fidelity-control` afterwards; it is a control fixture and not part of the corpus. + +- [ ] **Step 5: Repeat Step 1 for every distinct dir(C) shape this change uses, then check off tasks.md §7 and hand the working copy back for routing** + +--- + +## Task 8: Harbor oracle inhabitation + +**Files:** +- Modify: `modules/home/ai/evals/harborize/README.md` (record five-of-five evidence per package) + +**Interfaces:** +- Consumes: the exported Harbor heads from Task 5. +- Produces: the inhabitation witness the review gate requires before any agent run counts. + +- [ ] **Step 1: Run the oracle five times per package** + +```bash +HARBOR_TELEMETRY=0 harbor run -p modules/home/ai/evals/harborize/injection-canary-harbor \ + -k 5 -o logs/harborize/gate1 --job-name canary-gate1 -y \ + --skill ./modules/home/ai/evals/harborize/conditions/canary +HARBOR_TELEMETRY=0 harbor run -p modules/home/ai/evals/harborize/-harbor \ + -k 5 -o logs/harborize/gate1 --job-name mechanical-gate1 -y +``` + +Oracle is Harbor's default agent (`models/trial/config.py:164-168`), so `-a` and `-m` are unnecessary. +`OracleAgent.run` (`agents/oracle.py:81-136`) uploads `solution/` and execs `solve.sh`; no LLM client appears in the file. +The `./` prefix on the `--skill` path is a precaution: `resolve_skill_sources` takes the local branch whenever the path exists or begins with `.`, `/` or `~`, and parses a non-existent relative path as a git source. + +- [ ] **Step 2: Check both halves of the pass criterion** + +```bash +n=$(rg --no-filename -o '"reward":\s*[0-9.]+' logs/harborize/gate1 | sort | uniq -c | tee /dev/stderr | wc -l) +[ "$n" -gt 0 ] || { echo "FAIL: no reward lines found — the check did not run"; exit 1; } +fd -H 'exit-code.txt' logs/harborize/gate1 -x cat {} +``` + +Expected: `"reward": 1.0` on all five trials of each job, zero errored trials, and no `exit-code.txt` at all, or one containing `0`. +The non-empty assertion is load-bearing, because an empty grep prints nothing and nothing reads as "no failures" rather than as "the check did not run". +The singular `reward` is the key Harbor actually emits inside the plural `rewards` dict: `VerifierResult.rewards` is a `dict[str, float | int]` (`models/verifier/result.py:5`) and `_parse_reward_text` fills it with `{"reward": float(...)}` (`verifier/verifier.py:73`), so a search for `"rewards"` alone would find the container and not the value. +Reading rewards alone is insufficient: a broken oracle exiting nonzero does not raise, because `oracle.py:149-151` writes `exit-code.txt` and proceeds to the verifier, scoring 0. + +- [ ] **Step 3: Triage any errored trial by exception name before rerunning** + +`RetryConfig.max_retries` defaults to 0 (`models/job/config.py:282-284`), so every exception is terminal unless `-r N` is passed. +Four of the nine names in the no-retry list at `:288-300` are defined in `agents/installed/base.py`, which `OracleAgent` does not subclass, so any of them in a Gate-1 log means the job was not running the oracle. +Four further exceptions kill a trial without appearing in that list: `AddTestsDirError` (`verifier/verifier.py:19`), `DownloadVerifierDirError` (`:27`), and a bare `FileNotFoundError` from `_resolve_tests` or from a missing solve.sh (`oracle.py:94-95`). + +- [ ] **Step 4: Record five-of-five per package in the workspace README, then check off tasks.md §8 and hand the working copy back for routing** + +--- + +## Task 9: Metered registration assertion + +**Files:** +- Modify: `modules/home/ai/evals/harborize/README.md` (record the per-adapter assertion and the cost figures) + +**Interfaces:** +- Consumes: the validated cells file from Task 3 and the canary Harbor head from Task 5. +- Produces: the only evidence that Harbor's adapter registration copy ran, plus the raw job accounting Task 10 turns into the cost constant. + +- [ ] **Step 1: Scrub subscription auth and confirm the scrub** + +```bash +env | rg 'ANTHROPIC_OAUTH_TOKEN|CLAUDE_CODE_OAUTH_TOKEN|CODEX_AUTH_JSON_PATH' || echo "clean" +export ANTHROPIC_OAUTH_TOKEN="" +``` + +Expected: `clean`, or an explicit empty assignment. +Pi injects `ANTHROPIC_OAUTH_TOKEN` whenever the variable is present and non-empty in the resolved environment and the provider is anthropic, with no force flag of the kind claude-code and codex require, so withholding a flag is not enforcement. +The check is a walrus on the value rather than a membership test (`pi.py:102-105`), which is exactly why assigning the empty string suppresses the injection as reliably as unsetting the variable. + +- [ ] **Step 2: Confirm the canary head's network baseline before spending** + +```bash +rg -n 'network_mode' modules/home/ai/evals/harborize/injection-canary-harbor/task.toml +``` + +Expected: `network_mode = "public"` under `[environment]`, and `no-network` only under `[agent]` if Task 1 Step 2 permitted it. +A `no-network` baseline makes the next step fail during the agent's install fetch rather than at the registration copy, which is the failure design.md's risk register predicts and the one this ordering exists to avoid. +The mechanism is not adapter-specific: every installed adapter's install runs during `_prepare`/`_setup_agent` (`trial.py:408-414`), which no phase network policy wraps. + +- [ ] **Step 3: Run one short metered trial on the codex cell** + +```bash +HARBOR_TELEMETRY=0 CODEX_FORCE_AUTH_JSON=1 harbor run \ + -p modules/home/ai/evals/harborize/injection-canary-harbor \ + -a codex -m gpt-5.6-sol -k 1 \ + --skill ./modules/home/ai/evals/harborize/conditions/canary \ + -o logs/harborize/gate1 --job-name codex-registration -y +``` + +The metered adapter is codex via the ChatGPT-subscription path (settled decision of 2026-08-15, recorded in this change), superseding the earlier claude-code first-cell choice; the auth forms are `CODEX_FORCE_AUTH_JSON=1` or `CODEX_AUTH_JSON_PATH=` at `codex.py:1305-1325`, and each raises rather than falling back when its file is absent (`:1313`, `:1325`). +The instrument's `references/marketplace-program.md:46` is a cell roster that supplies the auth forms and the model strings; it records no supersession and names claude-code first, and its `:52-53` still carries the pre-decision costing policy, so it is not the authority for this decision. +This is the only rung that reaches the registration copy, and `--install-only` provably cannot substitute for the reasons carried in tasks.md §9's preamble, which anchors the property on codex as well as on the superseded claude-code cell: codex builds `_build_register_skills_command` (`codex.py:1199-1207`), calls it at `:1413` and appends it to `setup_command` at `:1415`, all inside `async def run` beginning at `:1333`. + +- [ ] **Step 4: Assert the registration destination in-container** + +codex registers at `$HOME/.agents/skills//` (`codex.py:1199-1207`), and that destination sits in no bind mount (`trial.py:1284-1288` binds only `/logs/agent`), so the assertion is made from inside the trial container while it still exists rather than as a host read after the run. +The criterion is the directory listing alone and deliberately not the reward. +The reward is not read as the criterion, for the reason recorded at Task 2 Step 4 rather than the one originally given: a model-driven trial's reward conflates delivery with the model's own behaviour, since a model that never greps the discovery roots scores 0 with the skill perfectly delivered. + +- [ ] **Step 5: For any additional cell, assert that adapter's own destination** + +codex and pi register at `$HOME/.agents/skills//` (`codex.py:1199-1207`, `pi.py:75-83`) and opencode at `~/.config/opencode/skills//` (`opencode.py:425-433`), none of which sits in a bind mount, so those assertions need an in-container check rather than a host read. + +- [ ] **Step 6: Record the cross-runner limitation** + +Write into the workspace README that pi, opencode and claude are not cross-runner comparable at these revisions, because BenchFlow's registry declares `["$HOME/.pi/agent/skills", "$HOME/.agents/skills"]` for `pi-acp` (`registry.py:560`) and `["$HOME/.opencode/skills"]` for `opencode` (`:700`), and codex is the one cell where both runners agree. + +- [ ] **Step 7: Check off tasks.md §9 and hand the working copy back for routing** + +--- + +## Task 10: Record, stamp and close + +**Files:** +- Modify: `modules/home/ai/evals/harborize/README.md` +- Create: `modules/home/ai/evals/harborize/results/cost-constant.json` + +**Interfaces:** +- Consumes: the job accounting from Task 9. +- Produces: the per-run cost constant the dependent change consumes, and the close-out evidence for verify.md. + +- [ ] **Step 1: Extract the per-run cost from the metered job** + +```bash +cost=$(rg --no-filename -o '"cost_usd":\s*[0-9.]+' \ + logs/harborize/gate1/codex-registration | rg -o '[0-9.]+$' | sort -rn | head -1) +[ -n "$cost" ] || { echo "FAIL: no cost_usd found — do not record a zero"; exit 1; } +echo "per-run cost: $cost" +``` + +Read the job's own accounting field rather than computing one here. +The field is `cost_usd`, not `cost` or `usage`, and it sits on `JobStats` (`models/job/result.py:28`, field at `:41`) rather than on `JobResult` (`:236`), which reaches it through `stats` at `:242` — so the accessor is `result.stats.cost_usd`. It accumulates at `:169` from the per-trial `AgentContext.cost_usd` that the adapter sets from `metrics.total_cost_usd`. +What fills that field differs by adapter, and Step 3 runs codex. +Claude Code parses an authoritative `total_cost_usd` from its own stdout stream (`claude_code.py:858-879`, `:1463`, set at `:1525`). +Codex has no such field — its CLI does not report cost in `token_count` events — and derives the figure from token counts against LiteLLM's pricing table (`codex.py:724-780`, resolved at `:1120-1124`, set at `:1194`), returning `None` when LiteLLM cannot price the resolved model. +So the recorded number is a list-rate computation rather than a billed charge, which is what keeps it a metered-rate figure under D3, and it is null rather than zero when the pricing entry is missing. +Confirm the entry exists before spending: `"$(sed -n '1s|^#!||p' "$(command -v harbor)")" -c 'import litellm; print(bool(litellm.model_cost.get("gpt-5.6-sol")))'` printed `True` at rung-0 time. +A search for `"cost"` or `"total_cost"` matches nothing, because neither spelling exists at this revision. +The non-empty assertion is required: an empty extraction must fail this task rather than leave the placeholder in Step 2 reading as a measured zero. + +- [ ] **Step 2: Write the cost record** + +```json +{ + "per_run_cost_usd": null, + "cell": "codex + gpt-5.6-sol", + "task": "injection-canary", + "trials": 1, + "auth_mode": "chatgpt-subscription", + "pricing_basis": "litellm model_cost list rates applied to observed token counts; not a billed charge", + "instrument_version": "0.2.1", + "harbor_rev": "ac398bbda7c4c1073461797d3b95c2455cc671b5", + "note": "a cost per run is a function of the cell and the task, not of the runner" +} +``` + +The placeholder is `null` rather than `0.0` so an unreplaced slot is unambiguous; a committed `0.0` would read as a measured zero, and this number is the change's headline deliverable. +Replace `per_run_cost_usd` with the figure Step 1 extracted, and assert it is non-null before the change closes. +Do not multiply it by any condition count, cell count or run total inside this change. + +- [ ] **Step 3: Confirm the canary is retained and named as a per-round precondition** + +```bash +ls modules/home/ai/evals/harborize/injection-canary modules/home/ai/evals/harborize/conditions/canary +``` + +Write into the workspace README that Task 7 Step 1 is re-run at the start of every later evaluation round, before any metered batch. + +- [ ] **Step 4: Confirm the instrument is unmodified** + +```bash +git rev-parse 'HEAD:modules/home/ai/plugins/testing-and-quality/.apm/skills/harborize' +find modules/home/ai/plugins/testing-and-quality/.apm/skills/harborize -type f \ + -not -path '*/__pycache__/*' -exec shasum -a 256 {} + | LC_ALL=C sort | shasum -a 256 +``` + +Expected: the digest equals the baseline Task 1 Step 5 recorded in the workspace README. +The digest is the check rather than `jj diff --stat -r @ -- `, because that diff reports only what the working-copy commit changed against its parents and would print nothing for an edit squashed into the `harborize-instrument` chain — the chain this plan explicitly routes onto, and therefore the most likely way the freeze would be violated. + +- [ ] **Step 5: List the deferred instrument defects** + +Write into the workspace README every instrument defect found during the change, deferred to the next revision, with its evidence. +The list opens with three already known: the canary leakage flag from Task 2 Step 3; the `--membership-from` census workflow; and `CHANGELOG.md:55`, which attributes `network_mode = "no-network"` to Harbor when Harbor's default is `public` (`models/task/config.py:249-252`) and the `no-network` default is the instrument's own authoring prescription at `SKILL.md:124`. + +- [ ] **Step 6: Write the review-gate audit into each package README** + +One line per algebraic invariant — truncation, inhabitation, non-triviality, coupling, grade discipline, empirical naturality, nucleus hygiene — with pass or fail and its evidence. +A package failing truncation, inhabitation, non-triviality or coupling is not done. + +- [ ] **Step 7: Check off tasks.md §10 and hand the working copy back for routing** diff --git a/openspec/changes/validate-harborize-instrument/proposal.md b/openspec/changes/validate-harborize-instrument/proposal.md new file mode 100644 index 000000000..5eb419142 --- /dev/null +++ b/openspec/changes/validate-harborize-instrument/proposal.md @@ -0,0 +1,83 @@ +--- +linear_story_id: fe51818e-d7f5-44a7-80c1-fa5ccc75d763 +linear_story_identifier: CAM-34 +linear_story_title: "Validate the harborize instrument before running any skill evaluations" +linear_story_url: https://linear.app/cameronraysmith/issue/CAM-34/validate-the-harborize-instrument-before-running-any-skill-evaluations +linear_story_state: Todo +linear_team: CAM +linear_project: skill-evaluation +last_synced_state: Todo +last_synced_at: "2026-08-15T05:17:31Z" +review_round: 0 +max_review_rounds: 3 +attempt_log: + - { at: "2026-08-15T05:17:31Z", transition: "Backlog->Todo", outcome: "posted", note: "T1 bind: CAM-34 created on team CAM in project Skill evaluation at state Todo; description seeded from proposal.md Why/What Changes/Capabilities; canonical crossing comment posted" } +--- + +## Why + +The harborize instrument emits evaluation packages and has never produced one that was shown to work. +All three iteration-1 packages failed the oracle inhabitation invariant, and the 0.2.1 repair round asserts a set of container-boundary facts about skill injection that nothing has exercised end to end. +No per-run cost figure exists either, so every budget in the staged marketplace program multiplies an unmeasured constant. +Running that program before the instrument is validated is how iteration 1 produced three unusable artifacts. +Validating it now yields a permanent injection regression test, one working mechanical package, and the cost constant every later budget depends on. + +## What Changes + +**Evidence for skill injection** +- From: the instrument documents where each adapter registers an injected skill and which surfaces cannot witness delivery, all read from upstream source and none exercised. +- To: an injection canary package that asserts delivery in the container, kept in the corpus permanently so a later regression in the injection path fails a task rather than silently voiding a batch. +- Reason: every silent-null class the instrument documents produces a clean run and a plausible negative result, so only a positive control distinguishes them. +- Impact: additive; a new package directory pair and a materialized condition directory. + +**A validated mechanical package** +- From: three iteration-1 packages that are reference material for shape and are not cited for results. +- To: one regenerated mechanical evaluation package authored BenchFlow-native, exported to a Harbor head, passing both static gates and the oracle inhabitation invariant under both runners. +- Reason: the dual-head authoring path and the oracle rung have never been demonstrated together on a task that measures something. Separate-mode verifier machinery was in this list and has been withdrawn: at benchflow 0.7.4 a package declaring it does not launch, and Harbor's separate verifier empties the verifier log directory the package's channel depends on, so exercising it and keeping a runnable BenchFlow arm are mutually exclusive. Design decision D11 carries the evidence. +- Impact: additive; the iteration-1 packages stay where they are and are not promoted. + +**A measured per-run cost** +- From: budget menus presented with `runs = |C| x k x cells` and no cost per run. +- To: one measured per-run cost constant, taken from the metered rung, recorded with the cell it was measured on and the instrument version that produced it. +- Reason: the staged program's screening figure of roughly 500 to 700 runs was scoped to one cell and never multiplied by a cost, so it is arithmetic waiting on this number. +- Impact: additive; the constant is an input to the dependent change, not a commitment to any budget. + +**A cost ladder as the working order** +- From: an implicit assumption that a package is validated by running it. +- To: seven rungs run in order, six of them free of model spend, each with an exact command and an executable pass criterion, with the metered rung last and reduced to one short trial per cell. +- Reason: only the adapter registration copy requires a real agent invocation to witness; everything upstream of it is detectable at zero marginal cost. +- Impact: the prerequisites rung was unmet when this change was written, so the ladder starts with real setup tasks rather than assumptions; section 1 has since run and the corpus README records the installed versions. + +## Non-goals + +These are out of scope and are named so that the apply gate's scope trigger has something to check against. +A change that grows to include any of them is not this change. + +- No marketplace-wide evaluation. The staged program in `references/marketplace-program.md` stays a plan. +- No condition-lattice runs. No emitted design leaves the throwaway gate-check path: rung 1 runs `design_matrix.py` three times against `/tmp` output directories purely to exercise its adapter gate, and nothing it writes there is executed, committed or consumed by a later rung. The change's own condition set is the single canary condition directory the delivery rungs require. +- No judge-validation package. The judge gate is a hard prerequisite for any judge-based stratum and it belongs to the dependent change. +- No cost projections beyond the measured figure. The constant is reported with its measurement conditions; multiplying it into a program budget is the dependent change's work. +- No promotion of the three iteration-1 packages. They are read for shape and regenerated, never cited for results. +- No modification to the harborize instrument. Instrument versioning freezes it at 0.2.1 for the duration; a defect found mid-change is recorded and deferred to the next revision. + +A larger dependent change, provisionally `marketplace-skill-evaluations`, consumes these deliverables afterward and is out of scope here. + +## Capabilities + +### New Capabilities + +- `evaluation-package-validation`: the instrument's demonstrated ability to produce an evaluation package whose skill injection is proven end to end at each level it can fail, and whose per-run cost is measured rather than assumed. Covers the ordered cost ladder from environment prerequisites through host-side resolution and static validation to in-container delivery and adapter registration, the permanent injection canary, and the recorded cost constant. + +### Modified Capabilities + + + +## Impact + +- New package pairs under the evaluation corpus at `modules/home/ai/evals/harborize/`, co-located with the first-party skill sources under `modules/home/ai/plugins/`: an authored BenchFlow-native tree per task plus a generated `-harbor` sibling, per the dual-head layout three independent validators force. Job output goes to `logs/harborize/`, which `.gitignore:57` already excludes. Two small repository-config edits support the corpus and are named here rather than left unexplained: `.gitignore` gains `__pycache__/` and `*.pyc`, because tasks 2.3, 5.8 and section 3 run the instrument's own `scripts/*.py` and CPython writes bytecode beside them inside the frozen instrument directory — which is also why task 1.8's freeze recipe excludes that path rather than relying on the ignore; and `.gitattributes` marks `modules/home/ai/evals/harborize/*-harbor/**` `linguist-generated`, because those heads are emitted by `bench tasks export` and a marker cannot be written into them without breaking the property that the committed tree equals a fresh export. No file in the corpus carries the `.nix` extension, because `flake.nix:6` calls import-tree bare over `modules/` and every `*.nix` file there is evaluated as a flake-parts module; the constraint is enforced by an extension audit and a flake evaluation rather than by convention. +- New host prerequisites, unmet when this change was written and satisfied at rung 0: a running Docker daemon (OrbStack was stopped and the socket absent), plus `harbor` and `bench` installed from PyPI latest stable (`uv tool install harbor`, `uv tool install benchflow`; settled 2026-08-15 — installs track PyPI releases, not sha-pinned source checkouts), never by `uv sync` inside the read-only ghq reference clones, which stay reading trees for the pinned anchors. +- Upstream pins recorded and cited: harbor `ac398bbda7c4c1073461797d3b95c2455cc671b5`, benchflow `d30527b82027a416e72014920cdf43a534967ad3`, skillsbench `9a1f4dd5f7659f75707435da3ce854b6e48321d1`. All three ghq clones are shallow and sit at those HEADs rather than at released tags, so the pins are recorded in the change and every anchor is re-read at them. +- Metered spend on the final rung only, on the codex cell via the ChatGPT-subscription path (`CODEX_FORCE_AUTH_JSON=1` or `CODEX_AUTH_JSON_PATH=`, settled decision of 2026-08-15, recorded in this change; supersedes the earlier claude-code first-cell choice), with `ANTHROPIC_OAUTH_TOKEN` still scrubbed on any Pi cell and `HARBOR_TELEMETRY=0` on every Harbor line. The instrument's `references/marketplace-program.md:46` supplies the auth forms and model strings and is cited only for those; it records no supersession, and design decision D3 carries the narrowing the settled decision implies. +- Unchanged: the harborize instrument at `modules/home/ai/plugins/testing-and-quality/.apm/skills/harborize/`, frozen at 0.2.1 for the duration, and the three iteration-1 packages under `~/Downloads/`. +- Touched, and previously mis-declared as unchanged: `modules/home/ai/skills/default.nix`. Its `excludedSkills` list gains eight retired `issues-beads*` names in a commit that lands inside this change's window and belongs to the beads retirement rather than to any rung here. The edit is recorded rather than reverted, and the declaration is corrected rather than left contradicting the branch. +- A limit of that exclusion, recorded because it reads wider than it is: `excludedSkills` acts on the composed `.claude/skills` tree home-manager consumes and not on marketplace publication, which is package-scoped through `apm.yml` and `.github/plugin/marketplace.json` with no skill-level exclusion at the compose or validate layer. The instrument therefore remains published inside the `testing-and-quality` plugin while being withheld from nix delivery, as do the eight `issues-beads*` skills. It is also narrower than global: `default.nix:94` and `:121` compute `allSkills // extraSkills` and `removeAttrs` never touches `extraSkills`, so an `aiSkills.extraSkillDirs` entry with a colliding leaf name would silently re-add an excluded skill. No current contributor collides. diff --git a/openspec/changes/validate-harborize-instrument/specs/evaluation-package-validation/spec.md b/openspec/changes/validate-harborize-instrument/specs/evaluation-package-validation/spec.md new file mode 100644 index 000000000..cd59d571b --- /dev/null +++ b/openspec/changes/validate-harborize-instrument/specs/evaluation-package-validation/spec.md @@ -0,0 +1,242 @@ +## ADDED Requirements + +### Requirement: Environment prerequisites are established before authoring + +The Docker daemon and both runner CLIs SHALL be available, and the daemon's kernel support for enforcing `no-network` MUST be probed before any task package is authored, with the probe's exit status deciding what the packages may declare. +Installing either CLI MUST NOT write into the read-only ghq reference clones. + +#### Scenario: kernel supports enforced no-network + +- **WHEN** the daemon is running and an anchored kernel probe for `^CONFIG_NFT_FIB_INET=[ym]` exits 0 +- **THEN** authoring proceeds with `network_mode = "public"` at each package's environment baseline and `no-network` on its `[agent]` phase + +#### Scenario: kernel explicitly lacks the option + +- **WHEN** the anchored probe exits 1, meaning `/proc/config.gz` records the option as unset +- **THEN** the `[agent]` phase override is dropped and every package is authored fully `public`, which is Harbor's own default and the declared mode of 86 of the 87 corpus tasks, and the loss of egress control is recorded rather than treated as a reason to halt + +#### Scenario: kernel configuration is unreadable + +- **WHEN** `/proc/config.gz` is absent, so the probe cannot decide +- **THEN** the outcome is recorded as indeterminate and authoring proceeds, because Harbor's own probe short-circuits to success in exactly that case, and a later rejection at environment start is taken as the deciding evidence + +#### Scenario: the probe's exit status is its criterion + +- **WHEN** a kernel probe is written for this requirement +- **THEN** its pattern is anchored and quiet, because an unanchored search matches the line recording the option as not set and would report success for the negative reading + +#### Scenario: runner CLIs installed outside the reference clones + +- **WHEN** `harbor` and `bench` are installed +- **THEN** both resolve on `PATH` and report a version, and no file under `~/ghq/github.com/harbor-framework/harbor` or `~/ghq/github.com/benchflow-ai/benchflow` is created or modified + +--- + +### Requirement: The evaluation corpus does not break flake evaluation + +The corpus SHALL live at `modules/home/ai/evals/harborize/` under version control, co-located with the first-party skill sources it evaluates, and no file inside it MUST carry the `.nix` extension. + +#### Scenario: job output stays outside the tracked corpus + +- **WHEN** a runner writes a job tree +- **THEN** it writes into the already-ignored `logs/harborize/` rather than into the corpus, so the tracked corpus holds authored packages, generated task heads, condition directories and recorded results and nothing else + +#### Scenario: a fixture never carries the nix extension + +- **WHEN** a fixture, an expectation file or a generated task head is written into the corpus +- **THEN** it carries an extension other than `.nix`, or it sits under an underscore-prefixed directory, because the flake's module discovery imports every `*.nix` file anywhere under the modules tree with no custom filter and would evaluate that file as a flake-parts module rather than read it as a fixture + +#### Scenario: the constraint is checked rather than trusted + +- **WHEN** the corpus root is first written, and again once the package directories and the generated task heads exist +- **THEN** an extension audit reports no `.nix` file inside the corpus and the flake's machine configurations still evaluate to an attribute list, and the audit covers the generated heads because they are produced by an exporter rather than authored by hand + +--- + +### Requirement: Harbor cells name only skill-consuming adapters + +Any Harbor cell definition used by this change SHALL name an adapter that reads the injected skills directory, and a cell naming a non-consuming adapter or an ACP registry shorthand MUST be refused before any container starts. + +#### Scenario: non-consuming adapter is refused + +- **WHEN** a cells definition names a Harbor adapter outside the consuming set and the adapter check runs +- **THEN** the check exits nonzero, names the adapter and the reason, and emits no run manifest + +#### Scenario: ACP registry shorthand is refused + +- **WHEN** a cells definition names a Harbor agent whose name begins with `acp:` +- **THEN** the check exits nonzero, because Harbor routes every such name through the non-consuming ACP adapter, which drops the injected skills on the Harbor arm while the BenchFlow arm for the same agent works + +#### Scenario: consuming adapter is accepted + +- **WHEN** a cells definition names a Harbor adapter inside the consuming set +- **THEN** the check exits zero and the manifest is emitted + +--- + +### Requirement: Host-side skill resolution is proven before any container starts + +The condition directory passed to either runner SHALL resolve on the host to exactly the expected skill set, each entry carrying a content digest, and this evidence MUST be recorded as host-side resolution rather than as delivery. + +#### Scenario: condition directory resolves to the expected skill set + +- **WHEN** Harbor's host-side skill resolution is run over the materialized condition directory +- **THEN** it returns one entry per expected skill name, each with a sha256 content digest, and the entry count equals the number of skill directories the condition declares + +#### Scenario: a malformed condition directory fails on the host + +- **WHEN** the condition directory contains a non-hidden child directory without a `SKILL.md`, or the path does not exist, or the path is not a directory +- **THEN** resolution raises on the host before any container starts + +#### Scenario: the trial lock is not accepted as delivery evidence + +- **WHEN** a trial's lock file is inspected and found to carry the full skill list with digests +- **THEN** that evidence is cited as host-side resolution and request only, because the lock is written in the trial constructor before the skills are resolved and long before they are uploaded + +--- + +### Requirement: Both task heads pass their static gate + +Each task package SHALL be authored BenchFlow-native and exported to a sibling Harbor head, and each head MUST pass the static gate that can run against its own layout before any oracle run. + +#### Scenario: BenchFlow head passes structural validation + +- **WHEN** `bench tasks check --level structural` is run against the authored native tree +- **THEN** it exits zero and reports no issues + +#### Scenario: Harbor head passes schema construction + +- **WHEN** the exported `-harbor` directory is constructed as a Harbor task model in Python +- **THEN** construction succeeds, and the gate's limits are recorded: it validates field names, types and enum membership only, and it returns early without checking for a test script whenever a verifier environment is configured + +#### Scenario: no Harbor CLI validation command is used + +- **WHEN** a static gate for the Harbor head is selected +- **THEN** `harbor task check` and `harbor tasks check` are not used, because both reach one command that prints an error and exits 1 unconditionally, and the redirect that stub prints is not followed either, because the command it names is a metered rubric run defaulting to a real agent and model + +--- + +### Requirement: Skill delivery is proven in the container with no model call + +Every distinct condition-directory shape SHALL be proven to deliver its skills inside the container through an oracle run that materializes no model, before any metered batch uses that shape. +Every package this change authors, including the canary that carries this proof, MUST route its agent-to-verifier channel through a surface both runners execute identically. + +#### Scenario: the verifier fork is the one both runners execute + +- **WHEN** a task package's verifier environment fork is chosen +- **THEN** it is shared and the package writes its deliverable into the verifier log directory rather than the agent workspace, because one runner refuses at launch to run a package declaring a separate verifier sandbox rather than falling back to shared, and the other empties the verifier log directory before a separate verifier runs, so a package declaring separate either does not launch or scores every agent zero, and a workspace path would in addition pass under one runner and fail under the other + +#### Scenario: the recorded fork is verified against the exported head + +- **WHEN** a package README records its verifier fork +- **THEN** the fork is confirmed by resolving the mode from the exported Harbor head's declared keys rather than inferred from the source directory layout, because the resolver reads only the verifier environment mode and the verifier environment table, so shipping a verifier Dockerfile alone leaves the package resolving to shared + +#### Scenario: oracle rollout delivers the declared skills + +- **WHEN** `bench eval run --tasks-dir --agent oracle --skill-mode with-skill --skills-dir --sandbox docker --jobs-dir ` is run +- **THEN** the rollout reaches the agent phase, reward equals 1, no rollout raises a skill-deployment-missing fidelity error, and each rollout's effective skills directory is the host directory that was passed + +#### Scenario: the canary is shown to be falsifiable + +- **WHEN** the same command is run in the no-skill mode, with no condition directory passed +- **THEN** reward is 0, which establishes that the passing run is evidence rather than a constant, and no fidelity error is raised, because with no condition directory the expected skill set is empty and the fidelity assertion is skipped + +#### Scenario: a diverging in-container catalogue raises the fidelity error + +- **WHEN** the same command is run against a throwaway package whose environment image already bakes a different skill set, so the in-container catalogue cannot match the host-computed one +- **THEN** the run raises the skill-deployment-missing fidelity error naming the expected skill set, and the batch does not proceed + +--- + +### Requirement: The oracle inhabits the task under Harbor + +Each task package's oracle SHALL pass the verifier under Harbor across five trials with no errored trial, and the pass criterion MUST distinguish a broken oracle from a genuine zero. + +#### Scenario: oracle passes five of five + +- **WHEN** `harbor run -p -k 5 -o --job-name -y` is run with the default agent +- **THEN** reward is 1.0 on all five trials, zero trials errored, and each trial's agent exit-code file is absent or contains 0 + +#### Scenario: a broken oracle is not read as a negative result + +- **WHEN** a trial scores 0 and the trial's agent exit-code file exists with a nonzero value +- **THEN** the result is classified as a broken oracle rather than a genuine zero, and the task is repaired before any further rung runs + +--- + +### Requirement: Adapter registration is asserted per adapter + +For each metered cell, the change SHALL assert that the adapter registered the injected skill at that adapter's own destination, and it MUST NOT substitute an install-only run for that assertion. + +#### Scenario: codex registration is asserted in-container + +- **WHEN** one short metered trial completes on the codex cell via the ChatGPT-subscription path +- **THEN** the skill directory is present under the adapter's configured skills destination (`$HOME/.agents/skills//`), asserted from inside the container because that destination sits in no host bind mount + +#### Scenario: install-only is rejected as a substitute + +- **WHEN** an install-only run is proposed as evidence of registration +- **THEN** it is rejected, because every adapter's registration command is built inside the agent's run path and install-only skips the agent run and disables the verifier + +#### Scenario: the metered cell's auth mode follows the settled decision + +- **WHEN** a metered trial is run on a cell whose adapter would accept a subscription token +- **THEN** the trial authenticates through the settled codex ChatGPT-subscription path (`CODEX_FORCE_AUTH_JSON` or `CODEX_AUTH_JSON_PATH`), any Pi cell in the same batch still scrubs `ANTHROPIC_OAUTH_TOKEN` to the empty string or unset rather than merely leaving it unflagged, and Harbor telemetry is disabled for the run + +#### Scenario: the metered cell's environment baseline permits the agent install + +- **WHEN** a metered trial is about to run against a task head +- **THEN** that head's environment baseline is confirmed to be `public` before any spend, because the agent's install fetch runs during trial preparation outside every phase network policy, and a no-network baseline would fail the trial during install indistinguishably from an injection failure + +--- + +### Requirement: The injection canary is retained in the corpus permanently + +The injection canary package SHALL remain in the corpus after this change completes and MUST be re-run at the start of every later evaluation round. + +#### Scenario: canary survives the change + +- **WHEN** the change is archived +- **THEN** the canary package is present in the corpus, its README records the leakage-audit flag and the reasoning for it, and it is named as a per-round precondition rather than a one-time check + +--- + +### Requirement: The per-run cost constant is recorded with its conditions + +The change SHALL record one measured per-run cost constant together with the conditions it was measured under, and it MUST NOT project that constant into any program budget. + +#### Scenario: cost constant carries its conditions + +- **WHEN** the metered rung completes +- **THEN** the recorded figure names the cell, the model, the task, the trial length, the auth mode, the pricing basis and the instrument version, and the record states that a cost per run is a function of the cell and the task + +#### Scenario: the figure is a rate computation, not a billed charge + +- **WHEN** the metered cell's adapter has no billed-cost field of its own and derives the figure from token counts against a pricing table +- **THEN** the record says so, the figure is still reported as a metered-rate figure because the pricing table carries metered list rates and does not vary with how the trial authenticated, and an absent pricing entry yields no number rather than a zero + +#### Scenario: no budget is derived here + +- **WHEN** the constant is reported +- **THEN** no condition count, no cell count and no run total are multiplied by it inside this change + +--- + +### Requirement: Packages and results are stamped with the instrument version, and the instrument is unmodified + +Every package README and every results file SHALL stamp the instrument version that produced it, and the harborize skill directory MUST NOT be modified for the duration of this change. + +#### Scenario: instrument version stamped + +- **WHEN** a package or a results file is written +- **THEN** it carries the instrument version 0.2.1 and the three upstream revisions the claims were verified at + +#### Scenario: a defect found mid-change is deferred + +- **WHEN** a defect in the instrument is discovered while a package is being authored or run +- **THEN** it is recorded for the next revision and the instrument is left unchanged, so the round's results stay attributable to one version + +#### Scenario: instrument directory unchanged at completion + +- **WHEN** the change completes +- **THEN** a content digest of the harborize skill directory equals the digest recorded before any other task ran, and that digest rather than a working-copy diff is the check, because a diff of the working-copy commit against its parents would show nothing for an edit squashed into the chain this change routes onto diff --git a/openspec/changes/validate-harborize-instrument/tasks.md b/openspec/changes/validate-harborize-instrument/tasks.md new file mode 100644 index 000000000..608811532 --- /dev/null +++ b/openspec/changes/validate-harborize-instrument/tasks.md @@ -0,0 +1,152 @@ +The rungs run in the order below. +Rungs 0 through 5 spend no model calls; rung 6 is the only metered one and it is last. +Each rung carries its exact command and its pass criterion, and a rung is not complete until its criterion is observed rather than assumed. + +## 1. Rung 0 — prerequisites (free, complete) + +Every prerequisite below was unmet when this section was written: OrbStack was stopped, the Docker socket was absent, neither `harbor` nor `bench` was installed, and neither ghq clone carried a virtualenv. +Section 1 has since run, and the corpus README records the installed versions. +Installing with `uv sync` inside a ghq clone writes into a read-only reference tree, so both CLIs install with `uv tool install` from PyPI latest stable (settled 2026-08-15) and never from a source checkout. + +``` +docker run --rm alpine sh -c ' + [ -f /proc/config.gz ] || exit 2 + zcat /proc/config.gz | grep -qE "^CONFIG_NFT_FIB_INET=[ym]"' +``` + +- [x] 1.1 Start OrbStack and confirm the Docker daemon answers: `docker info` exits 0 +- [x] 1.2 BLOCKING — run the kernel probe above immediately after the daemon starts and BEFORE authoring any task, and read its exit status rather than its output. The anchored `grep -qE` is load-bearing: an unanchored `grep NFT_FIB_INET` matches the line `# CONFIG_NFT_FIB_INET is not set` and exits 0, so the negative reading would pass any scripted check +- [x] 1.3 Branch on the probe's three outcomes rather than halting on any non-zero. Exit 0 means egress control is available, so `no-network` may be declared anywhere this change declares it. Exit 1 means the option is explicitly unset, so every package this change authors declares `network_mode = "public"` at the environment baseline and drops the `[agent]` phase override from task 5.4, and the loss of egress control is recorded in the workspace README. Exit 2 means `/proc/config.gz` is absent, which is indeterminate: Harbor's own probe short-circuits to exit 0 in exactly that case (`environments/docker/docker.py:113-117`), so Harbor proceeds and this change proceeds with it, recording the indeterminacy and treating a later `no-network` rejection as the deciding evidence +- [x] 1.4 Record the correct attribution alongside the probe result: `no-network` is the harborize instrument's own authoring default (`SKILL.md:124`, `references/emitters.md:74`), not Harbor's. Harbor's default is `public` (`models/task/config.py:249-252`, `NetworkPolicy` at `:66`), BenchFlow's is the same (`task/config.py:720-723`), and 86 of the 87 SkillsBench corpus tasks declare `network_mode: public`. What the probe gates is whether a `no-network` declaration can be enforced at all: on failure `_enable_egress_control` goes false (`docker.py:188-195`), which zeroes `capabilities.disable_internet` (`:289-293`), and `environments/base.py:773-781` raises at environment start +- [x] 1.5 Install the Harbor CLI from PyPI latest stable with `uv tool install harbor` (settled 2026-08-15: CLI installs track PyPI releases, not sha-pinned source checkouts; the upstream pins remain reading pins for every source anchor in this change); pass is `harbor --version` resolving on `PATH` with no file created or modified under `~/ghq/github.com/harbor-framework/harbor`, and the installed version recorded in the workspace README because it may differ from the pin +- [x] 1.6 Install the BenchFlow CLI the same way from PyPI (`uv tool install benchflow`); pass is `bench --version` resolving on `PATH` with `~/ghq/github.com/benchflow-ai/benchflow` unmodified and the installed version recorded beside 1.5's +- [x] 1.7 Create the change-owned evaluation workspace at `modules/home/ai/evals/harborize/`, co-located with the skills under evaluation in `modules/home/ai/plugins/`, and record in its README the three upstream pins, the instrument version 0.2.1, the probe outcome from 1.2 with its branch from 1.3, and the attribution from 1.4. Job output goes to `logs/harborize/`, which is already gitignored (`.gitignore:57`) +- [x] 1.8 Record the instrument-freeze baseline in the workspace README before any other task runs: the revision the change starts from and the digest below. Task 10.3 recomputes that digest, because a `jj diff -r @` cannot see an edit squashed into the `harborize-instrument` chain this change routes onto. Two properties of the recipe are load-bearing and both are failures this change reproduced rather than predicted. `-not -path '*/__pycache__/*'` is required because tasks 2.3, 5.8 and section 3 all run the instrument's own `scripts/*.py`, CPython writes `__pycache__/*.pyc` beside them, and `.gitignore` hides the result, so complying with those tasks would make 10.3 report a freeze violation that never happened. And the digest is a function of the path string `find` was given, because each `shasum` line embeds it: the recorded value is the one this exact invocation produces from the repository root, and a trailing slash, a `./` prefix, an absolute path or a `cd` into the directory each produce a different digest from an unchanged tree + +``` +find modules/home/ai/plugins/testing-and-quality/.apm/skills/harborize -type f \ + -not -path '*/__pycache__/*' -exec shasum -a 256 {} + | LC_ALL=C sort | shasum -a 256 +``` + +- [x] 1.9 Record the claude-code install caveat in the workspace README: `install()` curls its bootstrap (`claude_code.py:425-449`) during `_prepare`/`_setup_agent` (`trial.py:408-414`), which no network policy wraps, while only `_run_agent_phase` (`trial.py:465-469`) and the verifier phases enter `_phase_network_policy`. A cell whose *environment baseline* is `no-network` therefore fails during agent install indistinguishably from an injection failure at the reward level, which is why task 5.4 puts `no-network` on the `[agent]` phase rather than on the baseline +- [x] 1.10 Hard constraint binding every later task: no file inside `modules/home/ai/evals/harborize/` may carry the `.nix` extension. `flake.nix:6` calls `inputs.import-tree ./modules` bare with no custom filter, so import-tree's default `nixFilter = andNot (hasInfix "/_") (hasSuffix ".nix")` (import-tree `default.nix:50`, rev `4ebb10ae17d5f1ad366e7aef5b92cb8eecf24f69`) imports every `*.nix` file anywhere under `modules/` and evaluates it as a flake-parts module, so a fixture named `expected.nix` breaks the flake rather than failing as a fixture. Two escapes, either one sufficient: never give a fixture the `.nix` extension, or place it under a `_`-prefixed directory, which is this repository's documented exclusion convention (ADR-0018, `packages/docs/src/content/docs/development/architecture/adrs/0018-deferred-module-composition-architecture.md:311`). Every other extension is safe, because non-nix files are enumerated and then dropped before anything reads them with no warning and no error, which is why `modules/` already carries hundreds of non-nix files — 433 at `origin/main` `25d10e7c` and 480 at this change's tip, of which 34 are this corpus — and the flake still evaluates +- [x] 1.11 Run the flake-evaluation guard as soon as 1.7 writes the first corpus file under `modules/`, which is the earliest point at which 1.10 can be violated; pass is that `fd -H -e nix . modules/home/ai/evals/harborize` matches nothing and `nix eval .#nixosConfigurations --apply builtins.attrNames` exits 0 printing the attribute list. Record what that eval does and does not prove: `builtins.attrNames` forces the attrset spine and never the values, so it proves that import-tree's enumeration over `modules/` still succeeds and that every module file parses — which is exactly the hazard 1.10 names — and it does not evaluate any module body, so it is not evidence that the nix skill composition or its exclusion list evaluates. The one eval that would exercise those is `.#homeConfigurations."crs58@".config.programs.claude-code.skills`, which triggers an import-from-derivation and is out of scope for a no-build guard. Write the absence assertion as an explicit `if`, because bash exempts a `!`-inverted command from `set -e` and an absence assertion written that way is a no-op. `nix flake check` is the stronger equivalent and is preferable when the wait is acceptable; do not add `--no-build`, which exits 1 on an unrelated import-from-derivation in this repository and would read as a guard failure + +## 2. Canary skill and condition directory (free) + +- [x] 2.1 Author the canary skill directory: one `SKILL.md` carrying the asserted token, named so it cannot collide with a deployed skill name +- [x] 2.2 Materialize the condition directory `dir(C)` containing that skill directory and nothing else; a stray non-hidden child *directory* without a `SKILL.md` turns the whole condition into a hard error at resolution, while a stray file is filtered out silently by the `child.is_dir()` test at `skills.py:396` +- [x] 2.3 Record in the canary README that the asserted literal appears in both the verifier and the `SKILL.md` by design, that `audit_leakage.py` check 1 flags it by construction (`MIN_LITERAL_LENGTH` at `:44`, `check_literals` at `:96`), and that the instrument is not edited to exempt it because it is frozen at 0.2.1 +- [x] 2.4 Record what the shared verifier fork chosen in task 5.2 does and does not expose. The consequence originally anticipated — that a real agent in the metered rung could read the token out of the verifier script without the skill ever being delivered — does not arise: both runners upload the verifier's own directory during the verification phase, after the agent phase has ended (Harbor `verifier/verifier.py:147-153` reached from `_run_shared_verifier`, phase order fixed at `trial/single_step.py:41` then `:52`; BenchFlow `task/verifier_core.py:385` in `_verify_test_script` (`:346`), reached from `verify()` (`:260`)), so no agent phase observes `test.sh` under either fork. Record the correction with its anchors rather than the original claim, and record that rung 6's criterion is unchanged for a reason that never depended on it: a model-driven trial's reward conflates delivery with the model's own behaviour, since a model that never greps scores 0 with the skill perfectly delivered, so the adapter's registration directory is the deterministic witness (task 9.4). The falsifiability control in task 7.3 runs under the oracle, which greps skill directories unconditionally + +## 3. Rung 1 — adapter allowlist (free, already implemented at 0.2.1) + +The gate is `check_harbor_agents` in the instrument's `scripts/design_matrix.py`, called from `load_cells`, carrying `HARBOR_SKILL_CONSUMING_AGENTS`. +22 of the 39 factory-registered Harbor adapters consume an injected skill and 17 ignore it with no error, warning or log line. + +- [x] 3.1 Run the gate against a cells file naming a non-consuming adapter; pass is a nonzero exit naming the adapter and the reason, with no manifest emitted +- [x] 3.2 Run the gate against a cells file naming an `acp:` registry shorthand; pass is a nonzero exit, since `factory.py:167-175` routes every such name through the non-consuming ACP adapter +- [x] 3.3 Run the gate against the cells file this change will actually use, writing to a throwaway `--out` under `/tmp`; pass is exit 0 and a manifest emitted there. The emitted design is discarded rather than executed or committed, which is what keeps this rung inside the proposal's condition-lattice Non-goal; `design_matrix.py:444-464` writes `conditions.json`, `manifest.sh` and `jobs.json` into `--out`, and `:440` runs `load_cells` before `out.mkdir` at `:443`, so a refused cells file leaves the directory uncreated + +## 4. Rung 2 — host-side resolution (free, no daemon required) + +This rung replaces a refuted idea: `lock.json` cannot prove delivery, because `_write_trial_lock` runs at `trial.py:104` inside `__init__`, before `_resolve_injected_skills` at `:107` and long before `_upload_injected_skills` at `:411`, and `_build_agent_skill_locks` (`models/job/lock.py:462-475`) calls only host-side functions. + +- [x] 4.1 Call `harbor.skills.resolve_skills` (`skills.py:111-123`) over `dir(C)`; pass is one entry per expected skill name, each carrying a sha256 from `compute_skill_digest` (`skills.py:200-209`) +- [x] 4.2 Call it over a deliberately malformed root — a missing path, a file rather than a directory, and a child directory without a `SKILL.md`; pass is that each raises on the host before any container starts (`_find_skill_dirs`, `skills.py:382-416`) +- [x] 4.3 Record this rung's evidence as host-side resolution and request, never as delivery + +## 5. Task package authoring, dual head (free) + +Author BenchFlow-native and derive the Harbor head into a separate sibling directory; the two heads cannot share a directory, and the export refuses a destination overlapping its source in either direction. + +``` +bench tasks export -harbor --target harbor --overwrite +``` + +- [x] 5.1 Author the injection canary package BenchFlow-native (`task.md` + `oracle/solve.sh` + `verifier/test.sh`), with `environment/skills/` left empty and a `.gitkeep` in it so the directory survives a fresh checkout +- [x] 5.2 Fix the canary's verifier fork as shared and ship no `verifier/Dockerfile` and no `verifier.sandbox_mode`. The fork is forced rather than chosen, and both runners force it independently. BenchFlow does not fall back to shared when a package declares separate: `runtime_capabilities.py:186-192` raises an unsupported-feature issue whose reason string is "separate verifier sandboxes are parsed but not executed" — the wording of a refusal, not of a fallback — and `raise_for_task_runtime_support` is a fail-closed pre-launch gate (`sandbox/setup.py:676`, `:819-842`), so the task does not launch at all. And Harbor's separate verifier empties `/logs/verifier` before running (`trial.py:599`, through the same host bind it mounts at `:686-692`), so nothing written there before verification survives; task 5.3 records the corrected channel and why `/logs/verifier` cannot carry a deliverable under either runner. `_run_shared_verifier` (`trial.py:536-567`) performs no wipe. Shared mode is the one fork both runners execute at all +- [x] 5.3 Route the canary's oracle output to `/logs/artifacts/canary-output.txt` rather than to `/logs/verifier/` or to a workspace path, and route the mechanical package's `summary.json` the same way. This corrects the original instruction, which named `/logs/verifier/` and was read off Harbor alone; rung 4 refuted it on the first run. BenchFlow clears that directory's contents on the agent container immediately before the verifier runs, unconditionally — `harden_before_verify` executes `_CLEAR_VERIFIER_DIR_CMD` (`sandbox/lockdown.py:775-784`, call at `:1205-1212`) — while the mount-aware conditional clear in `_verify_test_script` (`task/verifier_core.py:360-372`) is a different branch that never reached the file. The bind survives and the content does not, so a canary written that way scores 1 under Harbor's shared verifier and 0 under BenchFlow with no error and no diagnostic beyond a `grep` miss. `/logs/artifacts/` is bind-mounted for the whole trial by both runners (`sandbox/docker.py:186-201`; `models/trial/paths.py:38`, `:195-202`), no hardening step touches it, and it is additionally the one path that would survive Harbor's separate fork, whose wipe at `trial.py:599` is followed by an artifact re-upload (`trial.py:601-607`, `artifact_handler.py:210-254`). `/logs/verifier/` keeps its one role under both runners: the reward file the verifier itself writes, after the clear +- [x] 5.4 Declare the network policy on every package: `network_mode = "public"` at the environment baseline, and `no-network` on the `[agent]` phase only when probe branch 1.3 permits it. The baseline is what the container is created with (`trial.py:896`) and the phase override is applied around `_run_agent_phase` alone (`trial.py:465-469`), so a `no-network` baseline breaks the claude-code install per task 1.9 while a `no-network` agent phase does not. Harbor reads the override through `task_cfg.agent.explicit_phase_policy()` (`trial/network_policy.py:45-59`) and validates it against the environment at `trial.py:203-217`; BenchFlow carries the same field at `task/config.py:524-530` +- [x] 5.5 Author the one real mechanical evaluation package the same way, with a single binary reward key; a multi-dimensional rubric silently disables Harbor's pass@k and makes BenchFlow's compare-lift count only `reward == 1.0` as passed (`eval_lift.py:32-33`) +- [x] 5.6 Choose and record the verifier fork for the mechanical package in its README, and record the choice as what the tree declares rather than as what a directory layout suggests: Harbor resolves the mode from `verifier.environment_mode` or `[verifier.environment]` in `task.toml` and from nothing else (`models/task/verifier_mode.py:10-21`), so shipping a `verifier/Dockerfile` infers nothing and a package that ships one while declaring neither key resolves to `SHARED`. The choice is shared, for the two reasons in 5.2, which apply to this package as well: BenchFlow refuses to launch a package declaring separate, and the proposal requires this package to pass under both runners; and Harbor's separate verifier would empty `/logs/verifier`, which is why neither package puts a deliverable there (task 5.3). Separate mode is therefore exercised nowhere in this change and D11 is corrected to say so. Verify the recorded choice against the exported head rather than the source layout: `resolve_task_verifier_mode(Task().config)` must equal the fork the README names +- [x] 5.7 Export both Harbor heads with the command above; never hand-edit a derived head, and re-export after every edit to the native tree. `--overwrite` is required from the second export onwards, because `export.py:255-256` raises `FileExistsError` on an existing destination +- [x] 5.8 Run `scripts/audit_leakage.py --task --skills ` on both packages; pass is exit 0 on the mechanical package, and on the canary the expected check-1 flag recorded with its justification per task 2.3 +- [x] 5.9 Stamp instrument version 0.2.1 and the three upstream pins into each package README +- [x] 5.10 Acceptance condition on this task, the first that creates package directories: no file in either authored tree or either exported head carries the `.nix` extension, and the flake still evaluates. Re-run 1.11's guard after 5.7's export rather than only after authoring, because `bench tasks export` writes two trees this change does not author by hand, and re-run it after every later re-export + +## 6. Rung 3 — static task validation (free) + +Harbor has no free CLI equivalent: `harbor task check` and `harbor tasks check` both reach a stub that prints an error and raises `SystemExit(1)` (`cli/tasks.py:476-487`). +Do not follow that stub's own redirect. +It prints "Use 'harbor check ' instead" (`cli/tasks.py:483-486`), and `harbor check` is a metered LLM-rubric run defaulting to `claude-code` and `claude-sonnet-4-6` (`cli/main.py:160`, `cli/analyze.py:100-103`), so obeying the CLI would spend money inside a rung this change calls free. + +``` +bench tasks check --level structural +``` + +- [x] 6.1 Run the command above against each authored native tree; pass is exit 0 with no issues reported (`benchflow cli/tasks.py:93-115`, default level `structural`) +- [x] 6.2 Construct `harbor.models.task.task.Task(-harbor)` in Python for each exported head; pass is that construction succeeds +- [x] 6.3 Record the Harbor gate's blind spot in the workspace README: `Task._validate_tests` returns early whenever a verifier environment is configured (`models/task/task.py:126-144`, early return at `:134-135`), so it structurally cannot catch a separate-mode package missing `/tests/test.sh` + +## 7. Rung 4 — delivery proof (Docker time, zero model calls) + +The strongest free rung. +BenchFlow's oracle path still calls `deploy_skills`: `rollout/__init__.py:1160` takes the `primary_agent == "oracle"` branch and calls it at `:1174`; `agents/install.py:303` computes the expected catalogue from `Path(skills_dir).glob("*/SKILL.md")` at `:313-314`, falls back to the five `_ORACLE_SKILL_PATHS` at `:30-36` via `:349-350`, and `_link_skill_paths` asserts the in-container catalogue equals the host's (`:146-161`) before raising `experiment_fidelity/skill_deployment_missing` at `:176-180`. +No model is materialized: `evaluation.py:461-462` returns None for the oracle agent. + +``` +bench eval run --tasks-dir --agent oracle --skill-mode with-skill \ + --skills-dir --sandbox docker --jobs-dir logs/harborize/canary-bench +``` + +- [x] 7.1 Run the command above for the canary package; pass is that the rollout reaches the agent phase AND reward equals 1 +- [x] 7.2 Confirm no rollout raises `experiment_fidelity/skill_deployment_missing` and each rollout's `effective_skills_dir` is the host `dir(C)` that was passed +- [x] 7.3 Run the falsifiability control: the same command with `--skill-mode no-skill` and no `--skills-dir`; pass is reward 0. This proves the canary can fail, which is what makes 7.1 evidence rather than a constant. It does NOT exercise the fidelity assertion, because `expected_skill_names` is empty when `skills_dir` is falsy (`agents/install.py:313-317`) and the `experiment_fidelity/skill_deployment_missing` raise is guarded on `if expected:` (`:176-180`) +- [x] 7.4 Run the fidelity control, which needs its own throwaway package or condition directory because the assertion compares the in-container catalogue against the host-computed one. The construction this task originally prescribed — copy the canary to a sibling, write `COPY _deps/skills /skills/` into its `environment/Dockerfile` by hand with a hand-authored `environment/_deps/skills/` holding a different skill set — cannot provoke the assertion at benchflow 0.7.4, and that is a finding rather than an execution detail: BenchFlow stages the host `--skills-dir` into `environment/_deps/skills` and appends that same `COPY` line itself, working on a temp copy of the task (`sandbox/setup.py:529-568`, `rollout/__init__.py:979-1016`), so a hand-authored baked set is overwritten before the build and the run scored 1. A second attempt that baked a decoy skill at `/root/.claude/skills` also scored 1, because the runtime link step runs `rm -rf && ln -sfn /skills ` (`agents/install.py:61-90`) and destroys a pre-baked discovery directory. The control that does provoke it separates the two catalogues where BenchFlow itself separates them: put a symlinked skill directory in the condition directory, since `deploy_skills` computes `expected` from `Path(skills_dir).glob("*/SKILL.md")` (`agents/install.py:312-317`), which follows the symlink, while the staging copy drops symlinked entries by design (`_stage_ignore`, `sandbox/setup.py:44-53`, the #411 fix). Pass is that `experiment_fidelity/skill_deployment_missing` is raised naming the expected skill set (`agents/install.py:139-181`). Delete the throwaway afterwards; it is not part of the corpus +- [x] 7.5 Run 7.1 once per distinct `dir(C)` shape this change uses, before any metered rung + +## 8. Rung 5 — Harbor oracle inhabitation (Docker time, zero model calls) + +Oracle is Harbor's default agent (`models/trial/config.py:164-168`), so `-a` and `-m` are unnecessary. +The canary head additionally needs its condition directory on the command line, `--skill `, because its reward is a delivery assertion; without it the head scores 0 across every trial, which is the Harbor-side counterpart of the 7.3 control. +`OracleAgent.run` (`agents/oracle.py:81-136`) uploads `solution/` and execs `solve.sh`; no LLM client appears in the file. + +``` +harbor run -p -k 5 -o logs/harborize/gate1 --job-name gate1 -y +``` + +- [x] 8.1 Run the command above for each package; pass is reward 1.0 across five trials, zero errored trials, AND `/agent/exit-code.txt` absent or containing 0 +- [x] 8.2 Check the exit-code file explicitly rather than reading rewards alone: a broken oracle exiting nonzero does not raise, since `oracle.py:149-151` writes the file and proceeds to the verifier, scoring 0 +- [x] 8.3 On any errored trial, triage by exception name before rerunning: `RetryConfig.max_retries` defaults to 0 (`models/job/config.py:282-284`) so every exception is terminal unless `-r N` is passed; four of the nine names in the no-retry list at `:288-300` are defined in `agents/installed/base.py`, which `OracleAgent` does not subclass, so any of them in a log means the job was not running the oracle; and `AddTestsDirError` (`verifier/verifier.py:19`), `DownloadVerifierDirError` (`:27`) and a bare `FileNotFoundError` from `_resolve_tests` or a missing solve.sh (`oracle.py:94-95`) kill a trial without appearing in that list. No trial errored in any of the three rung-5 jobs, so this triage was never entered + +## 9. Rung 6 — registration assertion (METERED, last, one short trial per cell) + +The only rung reaching Harbor's adapter registration copy. +`--install-only` provably cannot substitute, and the property is per adapter rather than claude-code's alone, so it is anchored on both the superseded cell and the one 9.4 actually runs: claude-code builds `_build_register_skills_command` (`claude_code.py:1530-1542`) and appends it to `setup_command` at `:1735`, both inside `async def run` beginning at `:1601`; codex builds its own (`codex.py:1199-1207`), calls it at `:1413` and appends it at `:1415`, all inside `async def run` beginning at `:1333`. `Trial.run` then guards `_run()` on `not install_only` (`trial.py:375-378`), and `TrialConfig._install_only_disables_verification` (`models/trial/config.py:484-494`) disables the verifier too, so an install-only run reaches neither adapter's registration copy. +Per-adapter destinations differ, so no single assertion covers the grid. + +- [ ] 9.1 Scrub subscription auth from the run environment before any metered trial: declare `ANTHROPIC_OAUTH_TOKEN=""` or unset it, because Pi injects it whenever the variable is present and non-empty (`pi.py:102-105` reads the value through a walrus, so an empty string is falsy and suppresses the injection), unlike claude-code and codex which require a force flag +- [ ] 9.2 Set `HARBOR_TELEMETRY=0` on every Harbor line, because `telemetry.py:239` sets `uses_skills` from the requested list and classifies a voided run as skill-bearing +- [ ] 9.3 Confirm before spending that the canary Harbor head declares `network_mode = "public"` at `[environment]`, per task 5.4; a `no-network` baseline makes this trial fail during the agent's install fetch rather than at the registration copy, which is the failure design.md's risk register predicts. The mechanism is the one recorded at task 1.9 for claude-code and is not adapter-specific: every installed adapter's install runs during `_prepare`/`_setup_agent` (`trial.py:408-414`), which no phase network policy wraps +- [ ] 9.4 Run one short metered trial on the codex cell (settled decision of 2026-08-15, recorded in this change and nowhere upstream of it: the metered adapter is codex via the ChatGPT-subscription path, `CODEX_FORCE_AUTH_JSON=1` or `CODEX_AUTH_JSON_PATH=` at `codex.py:1305-1325`; the instrument's `references/marketplace-program.md:46` is a cell roster that supplies the auth forms and the model strings and records no supersession, and its `:52-53` still carries the pre-decision policy — a frozen-instrument item for 10.4, not an authority for this decision). Both auth forms raise rather than fall back if their file is absent: `CODEX_AUTH_JSON_PATH` at `:1313` and `CODEX_FORCE_AUTH_JSON` at `:1325`, so confirm the file exists before spending. Pass is the skill directory present under `$HOME/.agents/skills//` (`codex.py:1199-1207`), asserted from inside the container because that destination sits in no host bind mount. The criterion is the directory alone and deliberately not the reward, for the reason corrected at 2.4: a model-driven trial's reward conflates delivery with the model's own behaviour rather than leaking through the verifier, so it is not read as evidence in either direction +- [ ] 9.5 For any additional cell, assert that adapter's own destination: codex and pi at `$HOME/.agents/skills//` (`codex.py:1199-1207`, `pi.py:75-83`), opencode at `~/.config/opencode/skills//` (`opencode.py:425-433`) +- [ ] 9.6 Do not report a cross-runner contrast for pi, opencode or claude: BenchFlow's registry declares different paths (`registry.py:560`, `:700`), and codex is the one cell where both runners agree. Record those cells' cross-runner numbers as non-comparable per the design's open question + +## 10. Record, stamp and close + +- [ ] 10.1 Record the measured per-run cost constant together with the cell, the model, the task, the trial length, the auth mode, the pricing basis and the instrument version; do not multiply it into any budget inside this change. Read it from Harbor's own accounting field `cost_usd`, which sits on `JobStats` (`models/job/result.py:28`, field at `:41`) and is reached as `result.stats.cost_usd` through `JobResult.stats` (`:236`, `:242`), accumulated at `:169` from the per-trial `AgentContext.cost_usd`, and treat an extraction that yields no numeric value as a failed task rather than as a measured zero. The accumulator is shared but the two adapters fill it differently, and 9.4 runs codex: claude-code parses an authoritative `total_cost_usd` from its own stdout stream (`claude_code.py:858-879`, `:1463`, set at `:1525`), while codex has no such field and derives the figure from token counts against LiteLLM's pricing table (`codex.py:724-780`, resolved at `:1120-1124`, set at `:1194`). Two consequences to record with the number. The figure is a list-rate computation rather than a billed charge, which is what keeps it a metered-rate figure under D3 even though the trial authenticates through a subscription. And it is null whenever LiteLLM cannot price the model, so confirm the entry exists before spending — verified present for `gpt-5.6-sol` and `gpt-5.6-luna` in the installed harbor tool environment at rung-0 time, and re-checkable with `litellm.model_cost.get("gpt-5.6-sol")` +- [ ] 10.2 Confirm the canary package is retained in the corpus and name it as a precondition to be re-run at the start of every later round +- [ ] 10.3 Confirm the harborize skill directory is unchanged by recomputing the content digest recorded in task 1.8 and comparing it to the recorded value. A `jj diff -r @` is not sufficient, because it reports only what the working-copy commit changed against its parents and would show nothing for an edit squashed into the `harborize-instrument` chain this change routes onto +- [ ] 10.4 List every instrument defect found mid-change as deferred to the next revision, with its evidence. The list opens with the three known at authoring time: the canary leakage flag from task 2.3; the `--membership-from` census workflow; and `CHANGELOG.md:55`, which attributes `network_mode = "no-network"` to Harbor when Harbor's default is `public` (`models/task/config.py:249-252`) and the `no-network` default is the instrument's own authoring prescription at `SKILL.md:124`. The change's own review round and the verification round after it added the nine below, each deferred under D5 rather than fixed: + - `design_matrix.py:317-319` and `:367` interpolate `cell['agent']`, `cell['model']`, `cell['concurrency']` and environment-variable keys into the emitted `manifest.sh` unquoted, while quoting only `task_dir`, `cond_dir`, `jobs_dir`, `job_name` and environment values; the emitted manifest carries `set -euo pipefail` at `:357`, so a model id containing a space, `&`, `#` or `;` produces a different command than intended. The adapter allowlist at `:153` is scoped to `runner == "harbor"`, so it does not contain the `benchflow` arm at all. The `cond_dir` docstring at `:101-107` is scoped to its own return value and is satisfied — all three call sites quote it (`:313`, `:326`, `:375`) — so the defect is the unquoted cell fields above, not that docstring + - `audit_leakage.py:104`'s literal extractor treats `"` and `'` as interchangeable delimiters and lets `[^"']` cross newlines, so on a shell file wrapping a python heredoc — the shape of every verifier in this corpus — it pairs quotes across the whole file and swallows the real expectation strings into multi-line blobs. Run at `b08c4168` against the then-current mechanical verifier, only 2 of 14 extracted literals were real; the ratio moves with the verifier and the mechanism is the finding. The matcher at `:112` is an unanchored substring test rather than a token match, which is what made the `pipelines` collision recorded at 5.8 a check bug rather than over-sensitivity + - `audit_leakage.py:41`'s `TEXT_SUFFIXES` omits `.yaml`, `.yml`, `.jsonl` and `.csv`, so a skill shipping expected output as a fixture is invisible to the audit; this corpus itself ships `build-events.jsonl` + - `audit_leakage.py:136-137` compares the oracle only against skill files with suffix `.sh` or `.py`, so an answer key in a `SKILL.md` — the most likely place — is never compared; and `:57-59` swallows `OSError`, so an unreadable skill file is silently un-audited and still counts toward a pass + - `check_oracle_similarity` (`audit_leakage.py:122-151`) compares oracle against skill and never oracle against verifier, which is the comparison that would have caught this change's own shared-derivation verifier defect + - `analyze_lattice.py:274-277` assigns the additive/synergy/interference tag from the point estimate against a `1e-9` tolerance while ignoring its own bootstrap interval, and `:314-320` warns on an unmatched `--units` without returning or setting a failure flag, so a typo exits 0 and reads as green to a CI gate + - `census.py:178-181` reads `data["skills"]` unguarded, so `--membership-from` pointed at a non-census JSON object raises a bare `KeyError`, and at a JSON array a `TypeError`, where its sibling `design_matrix.load_census:214-229` validates the same input and names the observed keys; `collect_rewards.py:442-444` returns before the `errors_out` write at `:451-459`, so `results.errors.json` is not written when every rollout errors and `--errors-as drop` is passed, losing the diagnostic exactly when it is needed; and `materialize_conditions.py:197-209` `rmtree`s an empty non-skill `--dest` that its stray guard was meant to protect, and raises `OSError` on a symlinked one + - `references/marketplace-program.md:52-53` carries the pre-decision costing policy that the settled metered-cell decision narrows, and `SKILL.md`, `references/algebraic-invariants.md`, `scripts/collect_rewards.py` and `scripts/design_matrix.py` carry six uppercase-for-emphasis sites against the repository's naming convention + - `SKILL.md` carries 27 multi-sentence lines against the one-sentence-per-line rule, where the peer `SKILL.md` corpus sits at 0 to 1, and the file is internally inconsistent about it. The comparison is deliberately like-for-like: this change's own OpenSpec artifacts carry multi-sentence lines too, but every one of them is a checkbox or a decision bullet, which is one item per line by construction, whereas the instrument's are free prose in a document whose peers do not do this; and type annotations are present in only one of the six instrument scripts, with `from __future__ import annotations` in two of six, which is an inconsistency within one instrument rather than an absence of tooling, since the repository arbitrates neither +- [ ] 10.5 Write the review-gate audit into each package README: one line per algebraic invariant, pass or fail, with its evidence +- [ ] 10.6 Migrate the corpus README's change-scoped rung evidence into `verify.md` and delete that section, leaving the README holding only what outlives the change: the pins, the instrument version and freeze baseline, the probe outcome with its branch and attribution, the install caveat, the layout constraint, the Harbor gate blind spot, and the canary's standing per-round precondition. The section is marked in place with the heading that names this obligation, so the split is checkable rather than remembered diff --git a/openspec/linear.yaml b/openspec/linear.yaml index 8b8258fca..d74d7eef7 100644 --- a/openspec/linear.yaml +++ b/openspec/linear.yaml @@ -42,3 +42,13 @@ projects: id: "ba0ffa20-a82b-40db-b292-d04f6a83eed2" "graphical-desktop-session": id: "8cf07dd5-2a48-4f45-bcd4-1add748aee46" + "skill-evaluation": + # The `id` below is the project slugId, and it must stay a slugId. + # The archive-time UPSERT resolves it via `linear document list --project`. + # It returns zero nodes with exit 0 for a project UUID, without erroring. + # The UPSERT's title scan then misses and duplicates every document. + # Nothing errors, so the duplication is only visible later in Linear. + id: "88879c3180bd" + name: "Skill evaluation" + teams: ["CAM"] + archive_documents: {}