fix(eval): tuning-scope fingerprint ignores docstrings

CHANGELOG.md

@@ -10,6 +10,23 @@ fixes. Security entries reference the advisory (GHSA) per the portfolio release
 
 ## [Unreleased]
 
+- **A docstring-only change to a tunable module can land without a `Tunes-Against:` trailer
+  (#174).** `tuning_scope._python_fingerprint` described itself as comment-insensitive, and it
+  was for `#` comments, which `ast` discards. A docstring is an `Expr(Constant(str))` statement,
+  though, so it was in the `ast.dump` and a one-word prose correction in
+  `src/sprout/providers/__init__.py` failed `sprout check-tuning-scope` exactly as a ranking
+  change would. There is no honest trailer for prose. Every Python comparison the gate makes
+  (the default one and the `answer.py`, `config.py` and provider-factory ones) now drops the
+  leading string of each module, class and function body first. A bare string anywhere else,
+  an f-string, and a bytes literal are still code. The exemption has the same standard as the
+  existing ones: nothing in the package reads `__doc__`, and
+  `test_package_never_reads_a_docstring` fails the day something does. Controls in both
+  directions run on every fingerprint path: a docstring-only edit passes, and a same-size
+  edit to the statement beside it still fails, with or without a docstring edit alongside.
+  The motivating docstring, which told readers to `pip install sprout` (an unrelated
+  library), is corrected. Its `KNOWN_GAP` entry and `test_the_known_gap_is_still_a_gap` in
+  `tests/test_install_instructions.py` are removed with it, as that entry said they should be.
+
 - **The published site now loads Google Analytics 4, and a `/privacy/` page says what it
   records ([ADR 0023](docs/adr/0023-google-analytics-4-on-the-published-site.md)).** Per the
   owner's 2026-09-17 decision to run GA4 on every public site in the portfolio.

CONTRIBUTING.md

@@ -144,7 +144,11 @@ failures** — never against results only visible from a private/local run. Mech
   executes — today just `Assistant.trace`, the `--debug` retrieval dump — because `sprout eval`
   replays `Assistant.answer` and never calls them, so nothing inside them can move an eval
   outcome; `tests/test_tuning_scope.py::test_eval_visible_modules_never_call_debug_only_methods`
-  fails the moment an eval-visible module calls one. Model literals, prompts, decoding parameters,
+  fails the moment an eval-visible module calls one. Python files are compared as syntax trees, so
+  `#` comments and docstrings (the leading string of a module, class or function body) are not
+  tuning either: nothing in the package reads `__doc__`, and
+  `tests/test_tuning_scope.py::test_package_never_reads_a_docstring` fails the day something does.
+  A bare string anywhere else is still code. Model literals, prompts, decoding parameters,
   real config values, retrieval/guard edits, module-level code in `answer.py`, every later
   lifecycle edit, and every unknown provider hunk remain fail-closed and require a real
   pre-existing case.

docs/ROADMAP.md

@@ -318,8 +318,9 @@ a real URL.*
 - Done: tuning only against committed eval failures is mechanically enforced by `sprout
   check-tuning-scope`, a required CI job. Changes to retrieval, generation, guards,
   calibration, lexical logic, or config must cite a case already recorded in the committed
-  eval baseline via a `Tunes-Against:` commit trailer. Comment-only YAML and the exact named
-  operational lifecycle wrapper are excluded by semantic/AST comparison. The initial lifecycle
+  eval baseline via a `Tunes-Against:` commit trailer. Comment-only YAML, comment- and
+  docstring-only Python, and the exact named operational lifecycle wrapper are excluded by
+  semantic/AST comparison. The initial lifecycle
   module is pinned to one reviewed bootstrap digest; all later lifecycle hunks are gated.
   Authorization comes from the merge-base baseline, and adversarial tests keep model, prompt,
   decoding, real-config, retrieval/guard, lifecycle-output, and unknown provider edits fail-closed.

src/sprout/cli.py

@@ -629,12 +629,13 @@ def check_tuning_scope_cmd(
     """Fail if this change tunes retrieval/prompts/guards without citing an already-committed
     eval failure (ROADMAP Phase 3: tune only against committed failures, never the held-out set).
 
-    No-op when the diff does not semantically change the tunable surface. Comment-only YAML and
-    the exact named lifecycle wrapper around an otherwise-identical provider constructor are
-    compared mechanically and excluded. The initial lifecycle module is admitted once by its
-    reviewed digest; later lifecycle and unknown provider hunks fail closed. Otherwise the commit
-    range must carry a ``Tunes-Against: <case-id>[, <case-id>...]`` trailer whose ids already
-    appear in the merge-base commit's ``<baseline>`` ``failing_examples``.
+    No-op when the diff does not semantically change the tunable surface. Comment-only YAML,
+    comment- and docstring-only Python, and the exact named lifecycle wrapper around an
+    otherwise-identical provider constructor are compared mechanically and excluded. The initial
+    lifecycle module is admitted once by its reviewed digest; later lifecycle and unknown provider
+    hunks fail closed. Otherwise the commit range must carry a
+    ``Tunes-Against: <case-id>[, <case-id>...]`` trailer whose ids already appear in the
+    merge-base commit's ``<baseline>`` ``failing_examples``.
     """
     from .eval.tuning_scope import TuningScopeError, check_tuning_scope
 

src/sprout/eval/tuning_scope.py

@@ -24,7 +24,8 @@
 exist at the branch's merge base; every later lifecycle or unknown provider hunk fails closed.
 So are the ``Assistant`` methods an eval run never executes (``Assistant.trace``, the ``--debug``
 dump): a gate on *eval outcomes* has nothing to say about code the eval never runs, and demanding
-a ``Tunes-Against`` citation for it would only be satisfiable by writing a false one.
+a ``Tunes-Against`` citation for it would only be satisfiable by writing a false one. Docstrings
+are dropped from every Python comparison for the same reason: nothing in the package reads one.
 """
 
 from __future__ import annotations
@@ -117,6 +118,53 @@ def tunable_paths(changed: list[str]) -> list[str]:
     return sorted(p for p in changed if is_tunable_path(p))
 
 
+class _DropDocstrings(ast.NodeTransformer):
+    """Erase only docstrings: the leading ``str`` constant of a module, class or function body.
+
+    ``ast`` already discards ``#`` comments, but a docstring is an ``Expr(Constant(str))``
+    statement, so without this a one-word prose correction moves the fingerprint exactly as a
+    ranking constant does. No module under ``src/sprout/`` reads ``__doc__`` (pinned by
+    ``tests/test_tuning_scope.py::test_package_never_reads_a_docstring``), so a docstring
+    cannot reach a prompt, a ranking, a guard or a threshold. A bare string anywhere other than
+    the first statement of a body, an f-string, and a bytes literal are not docstrings and stay
+    in the tree.
+    """
+
+    @staticmethod
+    def _strip(body: list[ast.stmt]) -> list[ast.stmt]:
+        if (
+            body
+            and isinstance(body[0], ast.Expr)
+            and isinstance(body[0].value, ast.Constant)
+            and isinstance(body[0].value.value, str)
+        ):
+            return body[1:] or [ast.Pass()]
+        return body
+
+    def visit_Module(self, node: ast.Module) -> ast.AST:
+        node.body = self._strip(node.body)
+        return self.generic_visit(node)
+
+    def visit_ClassDef(self, node: ast.ClassDef) -> ast.AST:
+        node.body = self._strip(node.body)
+        return self.generic_visit(node)
+
+    def visit_FunctionDef(self, node: ast.FunctionDef) -> ast.AST:
+        node.body = self._strip(node.body)
+        return self.generic_visit(node)
+
+    def visit_AsyncFunctionDef(self, node: ast.AsyncFunctionDef) -> ast.AST:
+        node.body = self._strip(node.body)
+        return self.generic_visit(node)
+
+
+def _parse_without_docstrings(source: str, *, type_comments: bool = False) -> ast.Module:
+    """Parse ``source`` and drop its docstrings; every fingerprint below starts here."""
+    tree = ast.parse(source, type_comments=type_comments)
+    _DropDocstrings().visit(tree)  # in place: visit_Module returns the same node
+    return tree
+
+
 class _LifecycleWrapperNormalizer(ast.NodeTransformer):
     """Erase only the mechanically constrained operational wrapper seam.
 
@@ -164,7 +212,7 @@ def visit_Call(self, node: ast.Call) -> ast.AST:
 
 
 def _provider_factory_fingerprint(source: str) -> str:
-    tree = ast.parse(source)
+    tree = _parse_without_docstrings(source)
     normalized = _LifecycleWrapperNormalizer().visit(tree)
     ast.fix_missing_locations(normalized)
     return ast.dump(normalized, include_attributes=False)
@@ -221,21 +269,23 @@ def visit_ClassDef(self, node: ast.ClassDef) -> ast.AST:
 
 
 def _answer_without_eval_invisible_methods(source: str) -> str:
-    tree = _DropEvalInvisibleAssistantMethods().visit(ast.parse(source))
+    tree = _DropEvalInvisibleAssistantMethods().visit(_parse_without_docstrings(source))
     ast.fix_missing_locations(tree)
     return ast.dump(tree, include_attributes=False)
 
 
 def _config_without_eval_invisible_classes(source: str) -> str:
-    tree = _DropEvalInvisibleConfigClasses().visit(ast.parse(source))
+    tree = _DropEvalInvisibleConfigClasses().visit(_parse_without_docstrings(source))
     ast.fix_missing_locations(tree)
     return ast.dump(tree, include_attributes=False)
 
 
 def _python_fingerprint(source: str) -> str:
-    """Return a formatting- and comment-insensitive Python syntax fingerprint."""
+    """Return a formatting-, comment- and docstring-insensitive Python syntax fingerprint."""
 
-    return ast.dump(ast.parse(source, type_comments=True), include_attributes=False)
+    tree = _parse_without_docstrings(source, type_comments=True)
+    ast.fix_missing_locations(tree)
+    return ast.dump(tree, include_attributes=False)
 
 
 def _source_at(ref: str, path: str, *, cwd: str | Path) -> str | None:

src/sprout/providers/__init__.py

@@ -1,7 +1,7 @@
 """Provider factory: config strings -> concrete embedder/generator, lazily imported.
 
 Lazy imports keep the offline default free of heavy dependencies — ``boto3`` is only
-imported when ``provider: bedrock`` is actually selected, so ``pip install sprout`` with
+imported when ``provider: bedrock`` is actually selected, so ``pip install sprout-plantcare`` with
 no extras still runs end to end. ``build_entailment_verifier`` follows the identical
 pattern for the ``sprout[nli]`` extra (EXP-04): ``onnxruntime``/``tokenizers`` are only
 imported when ``generation.support_verifier: nli`` is actually configured.

tests/test_install_instructions.py

@@ -67,21 +67,6 @@
 #: read all of it, so the population is asserted rather than assumed.
 MIN_CODE_FILES = 150
 
-#: The one occurrence this change could not correct, named rather than quietly skipped.
-#:
-#: `src/sprout/providers/` is `TUNABLE_SURFACE`, and `tuning_scope._python_fingerprint`
-#: builds its comparison from `ast.dump`, where a **docstring is a statement**. So a
-#: correction confined to a module docstring — text nothing in `src/` reads, measured:
-#: `__doc__` appears nowhere in the package — is indistinguishable to that gate from a
-#: change to retrieval ranking, and it demands a `Tunes-Against:` trailer citing a
-#: committed eval failure. There is no honest trailer for a comment, and writing a false
-#: one to get past a gate is worse than the wrong sentence it would fix.
-#:
-#: This is an exemption, so it is self-limiting: `test_the_known_gap_is_still_a_gap`
-#: fails the day the file is corrected or the gate learns to ignore docstrings, and the
-#: entry has to be deleted then. It cannot outlive its reason quietly.
-KNOWN_GAP = frozenset({"src/sprout/providers/__init__.py"})
-
 
 def _git(*args: str) -> str | None:
     executable = shutil.which("git")
@@ -144,8 +129,8 @@ def test_no_install_command_in_code_names_a_package_this_project_does_not_publis
     Ten occurrences in seven files, two of them exception messages raised at runtime
     telling a user the exact command to run to get an optional extra — the loudest
     possible place to name the wrong package, and the only place in the repository
-    nobody re-reads on the way past. Nine are corrected; the tenth is in `KNOWN_GAP`
-    above with its reason and its expiry.
+    nobody re-reads on the way past. All ten are corrected; the last waited on #174, a
+    tuning-scope gate that counted a docstring as code.
     """
     distribution = declared_distribution()
     files = _tracked_code_files()
@@ -156,8 +141,6 @@ def test_no_install_command_in_code_names_a_package_this_project_does_not_publis
     )
     problems: list[str] = []
     for path in files:
-        if str(path.relative_to(ROOT)) in KNOWN_GAP:
-            continue
         text = path.read_text(encoding="utf-8")
         for line_number, line in enumerate(text.splitlines(), 1):
             for target in wrongly_named_installs(line, distribution):
@@ -170,33 +153,6 @@ def test_no_install_command_in_code_names_a_package_this_project_does_not_publis
     )
 
 
-def test_the_known_gap_is_still_a_gap() -> None:
-    """An exemption that has stopped exempting anything reads as coverage.
-
-    Every entry above has to still contain the thing it is excused for. The day the
-    docstring is corrected — or the day `tuning_scope` stops treating one as a behavior
-    change — this fails and the entry goes, rather than sitting in the tuple looking like
-    a considered decision about a file nobody has looked at in a year.
-
-    It also pins the count, because an exemption list is the one list where growth is the
-    signal: a second entry means somebody widened an excuse instead of fixing a file.
-    """
-    distribution = declared_distribution()
-    assert len(KNOWN_GAP) == 1, (
-        f"the exemption list has grown to {sorted(KNOWN_GAP)}. Adding an entry here excuses a "
-        "file from naming the right package; it is not a place to put a file that is merely "
-        "inconvenient to fix."
-    )
-    for name in sorted(KNOWN_GAP):
-        path = ROOT / name
-        assert path.is_file(), f"{name} is exempted here and is not in the tree"
-        assert wrongly_named_installs(path.read_text(encoding="utf-8"), distribution), (
-            f"{name} is exempted from the scan above and no longer contains an install "
-            "command naming the wrong package. Delete the entry — an exemption that exempts "
-            "nothing is an exemption nobody can see the cost of."
-        )
-
-
 def test_the_package_derives_its_version_from_the_distribution_the_manifest_declares() -> None:
     """The rename's silent failure mode, made mechanical instead of remembered.