feat(run): preserve sourced harness config

docs/claude-notes.md

@@ -128,8 +128,9 @@ Each `claude -p` dispatch loads the user/global plugins and skills from its Clau
 staging slug prevents an on-disk collision but not runtime discovery — an installed plugin exposing
 a same-named skill is discoverable in *both* arms, so the control arm is not truly skill-absent.
 `plugin_shadow.rs` detects this in every comparison environment. The shared shadow policy records
-one finding per logical skill in schema-v2 `plugin-shadow.json`, including every affected cell,
-canonical/discovery paths, source-specific remediation, and the runtime identifier the agent sees.
+one finding per logical skill and source class in schema-v3 `plugin-shadow.json`, including every
+affected cell, canonical/discovery paths, source-specific remediation, and the runtime identifier
+the agent sees.
 Claude plugin skills use their namespaced `<plugin>:<skill>` runtime ID, direct live skills retain
 the logical name, and staged subjects use their staging-directory slug. Direct live duplicates
 record user-before-project precedence; a staged subject with its distinct slug remains selected.

docs/guides/byoh.md

@@ -93,6 +93,21 @@ The scaffold and resolved descriptor output are the installed references. Reposi
 can trace the underlying schema and adapter contracts from `docs/developer_overview.md` in a source
 checkout.
 
+When a harness discovers project skills from more than its native staging directory, declare the
+extra roots beside `skills_dir`:
+
+```toml
+skills_dir = ".cool/skills"
+additional_project_skill_dirs = [".claude/skills", ".agents/skills"]
+config_dirs = [".cool", ".claude", ".agents"]
+```
+
+`skills_dir` is the only staging destination. The additional roots participate in sourced-codebase
+shadow detection and `codebase.exclude_skill_sources`; eval-magic never stages into them. Every
+path must be normalized, `/`-separated, and relative to the task repository. Its first segment must
+also appear in `config_dirs`, keeping discovery, sibling filtering, and task-repository baselining
+on one descriptor surface.
+
 ## Layer descriptors by field
 
 Descriptors load in this order:

docs/guides/codebase.md

@@ -54,6 +54,45 @@ all.
 Resolution happens before any environment is created. An unreachable repository or a ref that does
 not exist fails the run while it has still built nothing.
 
+## Project config and skill sources
+
+The sourced tree is preserved by default, including harness instructions, settings, plugins, and
+project-local skills. For example, `CLAUDE.md`, `AGENTS.md`, `.claude/settings.json`, and
+`.opencode/settings.json` remain visible in every comparison arm.
+
+Preserving project skills can contaminate a comparison when the codebase provides the
+subject or one of its staged siblings. `run` records those matches in `plugin-shadow.json` with
+`class: "codebase-sourced"`, separately from `class: "operator-environment"` findings caused by
+global skills or installed plugins. Subject collisions are comparison-invalid; sibling collisions
+follow the symmetric/asymmetric rules in `eval-magic docs isolation`.
+
+Opt an eval out of only the harness-discoverable project skill roots when the codebase's skills are
+not part of the task being measured:
+
+```json
+{
+  "codebase": {
+    "url": "https://github.com/slowdini/example-project",
+    "ref": "v1.4.0",
+    "exclude_skill_sources": true
+  }
+}
+```
+
+The default is `false`. When set to `true`, eval-magic moves every project skill root declared by
+the selected harness out of each task environment before staging. It applies equally to both arms,
+every repetition, revision mode, and `--no-stage`. Root instruction files and other harness config
+remain in place. OpenCode, for example, excludes `.opencode/skills`, `.claude/skills`, and
+`.agents/skills` because its descriptor declares all three discovery roots. For a BYOH descriptor
+with no project skill roots, the setting is recorded and makes no filesystem change.
+
+Generated staging slugs are collision-safe: if the codebase owns that exact directory, the
+runner backs it up, stages the evaluated copy for that arm, and restores the original during
+cleanup. An explicit `--stage-name` remains stricter and refuses to clobber an occupied directory.
+
+The effective `exclude_skill_sources` value is recorded with each codebase in `conditions.json`,
+every task in `dispatch.json`, every `run.json`, `benchmark.json`, and promoted `BASELINE.md`.
+
 ## What the environment contains
 
 Each dispatch gets its own private environment holding:
@@ -172,9 +211,9 @@ head -50 diff.patch
 The same difference, spelled by Git itself, is `git diff refs/eval-magic/baseline` inside the
 environment.
 
-The resolved commit appears in `conditions.json`, each `run.json`, `benchmark.json`, and the
-`BASELINE.md` written by `promote-baseline` — alongside the skill the run measured, which
-is recorded the same way:
+The resolved commit and effective skill-source policy appear in `conditions.json`, each `run.json`,
+`benchmark.json`, and the `BASELINE.md` written by `promote-baseline` — alongside the skill the run
+measured, which is recorded the same way:
 
 ```sh
 jq '.codebases, .skill_source' conditions.json

docs/guides/isolation.md

@@ -20,9 +20,14 @@ Sibling collisions have two outcomes:
 - A sibling visible in only one arm is comparison-invalid because its effect cannot be separated
   from the skill under test.
 
-The preflight reports what the environment makes discoverable. Transcript evidence can later show
-what a dispatch loaded. Eval-magic does not parse shell templates to infer that a flag or environment
-variable isolates the process.
+The preflight reports two source classes in schema-v3 `plugin-shadow.json`:
+
+- `operator-environment` — global skills, enabled plugins, and other sources inherited from the
+  machine running eval-magic.
+- `codebase-sourced` — matching project-local skills preserved from the task codebase.
+
+Transcript evidence can later show what a dispatch loaded. Eval-magic does not parse shell
+templates to infer that a flag or environment variable isolates the process.
 
 Apply the remedy to **every eval-agent command**, including every resumed turn of a scripted eval.
 Isolating only the first round allows the live copy to return on the next round. Judge commands do
@@ -80,9 +85,14 @@ label = "claude-code"
 isolates_live_sources = true
 ```
 
+The declaration covers only `operator-environment` findings. It does not claim that skills sourced
+from the task codebase are isolated. Use `codebase.exclude_skill_sources: true` for that separate
+policy when project skills should not participate; see `eval-magic docs codebase`.
+
 The declaration does not disable detection. `plugin-shadow.json` retains every source and its
-intrinsic severity as provenance. `run` presents the finding as informational, and `aggregate`
-omits the warning only while no transcript evidence contradicts the declaration.
+intrinsic severity as provenance. `run` presents operator-environment findings as informational,
+and `aggregate` omits those warnings only while no transcript evidence contradicts the declaration.
+Codebase-sourced findings remain warnings regardless of this descriptor setting.
 
 Do not set it when:
 

docs/progressive-enhancements.md

@@ -347,16 +347,19 @@ global `.opencode`, `.claude`, and `.agents` skill dirs — including skills ins
 harnesses. A logical eval skill present in any such source can contaminate the with/without
 comparison when dispatches load that source, even when the staged copy uses a unique slug.
 
-*What it unlocks:* a build-time contamination warning (shared banner + schema-v2
+*What it unlocks:* a build-time contamination warning (shared banner + schema-v3
 `plugin-shadow.json` in the iteration dir), which `aggregate` folds into `benchmark.json`
 validity warnings. The runner scans every matrix environment and the shared policy groups scanner
-facts by logical skill, records live/staged sources and affected cells, and assigns role-aware
-severity. Subject and asymmetric sibling collisions invalidate the comparison; symmetric sibling
-collisions warn. Because the scan runs before dispatch it reports *risk*, so the banner states the
-consequence conditionally; the verdict is settled afterwards by the session-surface sub-capability
-below. When the resolved descriptor declares `isolates_live_sources = true`, the scan, intrinsic
-severity, and artifact are retained, but the banner becomes an informational notice and `aggregate`
-omits the findings from validity warnings. Historical unversioned artifacts remain readable.
+facts by logical skill and source class, records live/staged sources and affected cells, and assigns
+role-aware severity. `operator-environment` findings come from inherited global/plugin sources;
+`codebase-sourced` findings come from project roots the harness descriptor declares. Subject and
+asymmetric sibling collisions invalidate the comparison; symmetric sibling collisions warn.
+Because the scan runs before dispatch it reports *risk*, so the banner states the consequence
+conditionally; the verdict is settled afterwards by the session-surface sub-capability below. When
+the resolved descriptor declares `isolates_live_sources = true`, operator-source scan facts,
+intrinsic severity, and artifact are retained, but the banner becomes informational and `aggregate`
+omits those findings. Codebase findings use the eval's separate `exclude_skill_sources` policy and
+remain warnings when preserved. Schema-v2 and historical unversioned artifacts remain readable.
 
 ### Session surface (sub-capability of transcript ingest)
 

harnesses/opencode.toml

@@ -9,7 +9,8 @@
 
 label = "opencode"
 skills_dir = ".opencode/skills"
-config_dirs = [".opencode"]
+additional_project_skill_dirs = [".claude/skills", ".agents/skills"]
+config_dirs = [".opencode", ".claude", ".agents"]
 
 [run]
 supports_guard = true

harnesses/template.toml

@@ -23,10 +23,15 @@ label = "{label}"
 ## Where the harness discovers project-local skills. Declaring skills_dir unlocks native
 ## staging; without it every run is forced to --no-stage. The first path segment of skills_dir
 ## must appear in config_dirs (it feeds the staging sibling filter and task-repository baseline).
+## If the harness also discovers compatibility roots, list them in
+## additional_project_skill_dirs. They participate in sourced-codebase shadow detection and
+## codebase.exclude_skill_sources but never receive staged skills. Each first path segment must
+## also appear in config_dirs.
 ## VERIFY: which directory does the harness actually scan for skills? Quote the doc or the
 ## observed behavior in the notes file.
 # skills_dir = ".{label}/skills"
-# config_dirs = [".{label}"]
+# additional_project_skill_dirs = [".claude/skills", ".agents/skills"]
+# config_dirs = [".{label}", ".claude", ".agents"]
 
 ## -------------------------------------------------------------------------------------------
 ## [dispatch] — the highest-leverage first enhancement: with exec_template declared, the

schema/benchmark.schema.json

@@ -107,6 +107,10 @@
           "type": "boolean",
           "description": "True when the source cannot be resolved off the host that ran it, so a published claim citing it is not reproducible from the eval config alone."
         },
+        "exclude_skill_sources": {
+          "type": "boolean",
+          "description": "Whether project-local skill roots discoverable by the selected harness were removed from the comparison environment before staging."
+        },
         "evals": {
           "type": "array",
           "items": { "type": "string" },

schema/evals.schema.json

@@ -41,6 +41,11 @@
           "type": "string",
           "minLength": 1,
           "description": "Directory on this host to build the task environment from, resolved relative to this evals.json when relative. Unlike files_root it may be absolute or escape the skill tree, because it deliberately points outside it. A path source is host-local: another machine has the directory elsewhere or not at all, so a run recorded against one is not reproducible from this config alone. When the directory is a Git repository the runner also records its origin URL and resolved SHA, which are."
+        },
+        "exclude_skill_sources": {
+          "type": "boolean",
+          "default": false,
+          "description": "Move project-local skill roots discoverable by the selected harness out of every comparison environment before staging. Root instruction files and other harness configuration remain visible."
         }
       }
     },
@@ -58,6 +63,11 @@
           "type": "string",
           "minLength": 1,
           "description": "Branch, tag, or full commit SHA to check out. Required: the runner records the resolved SHA, so an eval tracking a moving branch could not be re-run against what it measured."
+        },
+        "exclude_skill_sources": {
+          "type": "boolean",
+          "default": false,
+          "description": "Move project-local skill roots discoverable by the selected harness out of every comparison environment before staging. Root instruction files and other harness configuration remain visible."
         }
       }
     },

schema/harness-descriptor.schema.json

@@ -17,6 +17,12 @@
       "minLength": 1,
       "description": "Project-local staged-skills directory, `/`-separated relative to the repo root (e.g. \".claude/skills\"). Optional: a harness without one cannot stage skills natively, so runs fall back to --no-stage (each SKILL.md inlined into its dispatch prompt) with a preflight warning."
     },
+    "additional_project_skill_dirs": {
+      "type": "array",
+      "uniqueItems": true,
+      "items": { "type": "string", "minLength": 1 },
+      "description": "Additional project-local skill roots the harness discovers for compatibility with other harness conventions. These roots participate in codebase shadow detection and opt-in exclusion but never receive staged skills."
+    },
     "config_dirs": {
       "type": "array",
       "items": { "type": "string", "minLength": 1 },

schema/plugin-shadow.schema.json

@@ -12,7 +12,7 @@
   ],
   "properties": {
     "schema_version": {
-      "const": 2
+      "const": 3
     },
     "config_dir": {
       "type": "string"
@@ -35,12 +35,20 @@
       "type": "object",
       "additionalProperties": false,
       "required": [
+        "class",
         "skill_name",
         "role",
         "severity",
         "sources"
       ],
       "properties": {
+        "class": {
+          "enum": [
+            "operator-environment",
+            "codebase-sourced"
+          ],
+          "description": "Whether the non-staged source comes from the operator environment or from the sourced task codebase."
+        },
         "skill_name": {
           "type": "string",
           "minLength": 1

schema/run-record.schema.json

@@ -274,6 +274,10 @@
           "type": "boolean",
           "description": "True when the source cannot be resolved off the host that ran it, so a published claim citing it is not reproducible from the eval config alone."
         },
+        "exclude_skill_sources": {
+          "type": "boolean",
+          "description": "Whether project-local skill roots discoverable by the selected harness were removed from the task environment before staging."
+        },
         "dirty": {
           "type": "boolean",
           "description": "True when the copy this record describes carries uncommitted work from its source, so revision alone does not name what ran. A codebase is checked out at a commit and is never dirty; a skill is copied as it sits on disk and can be."

src/adapters/descriptor.rs

@@ -59,6 +59,8 @@ pub struct HarnessDescriptor {
     #[serde(skip_serializing_if = "Option::is_none")]
     pub skills_dir: Option<String>,
     #[serde(default, skip_serializing_if = "Vec::is_empty")]
+    pub additional_project_skill_dirs: Vec<String>,
+    #[serde(default, skip_serializing_if = "Vec::is_empty")]
     pub config_dirs: Vec<String>,
     #[serde(default, skip_serializing_if = "RunSection::is_default")]
     pub run: RunSection,
@@ -641,6 +643,72 @@ timestamp_spread = "timestamp"
         assert!(d.transcript.is_none());
     }
 
+    #[test]
+    fn additional_project_skill_dirs_load_and_reserialize() {
+        let d = load(
+            "label = \"demo\"\nskills_dir = \".demo/skills\"\n\
+             additional_project_skill_dirs = [\".claude/skills\", \".agents/skills\"]\n\
+             config_dirs = [\".demo\", \".claude\", \".agents\"]\n",
+        )
+        .unwrap();
+
+        assert_eq!(
+            d.additional_project_skill_dirs,
+            vec![".claude/skills", ".agents/skills"]
+        );
+        let shown = toml::to_string(&d).unwrap();
+        assert!(shown.contains("additional_project_skill_dirs"), "{shown}");
+    }
+
+    #[test]
+    fn rejects_config_dirs_missing_an_additional_project_skill_parent() {
+        let error = err_of(
+            "label = \"demo\"\nskills_dir = \".demo/skills\"\n\
+             additional_project_skill_dirs = [\".claude/skills\"]\n\
+             config_dirs = [\".demo\"]\n",
+        );
+
+        assert!(error.contains(".claude"), "{error}");
+        assert!(error.contains("additional project skill"), "{error}");
+    }
+
+    #[test]
+    fn rejects_additional_project_skill_dirs_without_a_native_skills_dir() {
+        let error = err_of(
+            "label = \"demo\"\nadditional_project_skill_dirs = [\".claude/skills\"]\n\
+             config_dirs = [\".claude\"]\n",
+        );
+
+        assert!(error.contains("additional_project_skill_dirs"), "{error}");
+        assert!(error.contains("skills_dir"), "{error}");
+    }
+
+    #[test]
+    fn rejects_project_skill_dirs_that_escape_or_duplicate_the_native_root() {
+        for additional in [
+            "../skills",
+            "/tmp/skills",
+            ".claude/../skills",
+            ".demo/skills",
+        ] {
+            let error = err_of(&format!(
+                "{MINIMAL}\nadditional_project_skill_dirs = [\"{additional}\"]\n"
+            ));
+            assert!(error.contains("project skill"), "{additional}: {error}");
+        }
+    }
+
+    #[test]
+    fn rejects_backslash_separated_project_skill_dirs() {
+        let error = err_of(
+            "label = \"demo\"\nskills_dir = \".demo/skills\"\n\
+             additional_project_skill_dirs = [\".claude\\\\skills\"]\n\
+             config_dirs = [\".demo\", \".claude\\\\skills\"]\n",
+        );
+
+        assert!(error.contains("`/`-separated"), "{error}");
+    }
+
     #[test]
     fn dispatch_environment_loads_and_reserializes() {
         let d = load(&format!(