Skip to content

fix(eval): tuning-scope fingerprint ignores docstrings - #210

Merged
ChelseaKR merged 2 commits into
mainfrom
fix/tuning-fingerprint-ignores-docstrings
Oct 2, 2026
Merged

ChelseaKR merged 2 commits into
mainfrom
fix/tuning-fingerprint-ignores-docstrings

Conversation

@ChelseaKR

Copy link
Copy Markdown
Owner

What & why

A change confined to a docstring in a tunable module could not land. sprout check-tuning-scope treated it as a behavior change and asked for a Tunes-Against: trailer, and no honest trailer exists for prose. The cause was _python_fingerprint in src/sprout/eval/tuning_scope.py. It says it is comment-insensitive, and that holds for # comments, which ast drops. A docstring is an Expr(Constant(str)) statement, though, so it stayed in the ast.dump.

This PR adds _DropDocstrings, which removes only the leading string constant of a module, class or function body. All four Python comparisons the gate makes now use it: the default fingerprint plus the answer.py, config.py and provider-factory fingerprints. A bare string anywhere else, an f-string and a bytes literal all stay in the tree.

The exemption meets the same standard as the existing three. Nothing in the package reads a docstring, and the new test_package_never_reads_a_docstring pins that. It scans src/sprout/ for __doc__, getdoc, get_docstring, model_json_schema and use_attribute_docstrings, so it fails the day one of those appears.

A second commit fixes the docstring from the issue. It is in src/sprout/providers/__init__.py and told readers to pip install sprout, which installs an unrelated library. That commit also removes the KNOWN_GAP entry and test_the_known_gap_is_still_a_gap from tests/test_install_instructions.py, as the entry's comment said should happen.

Closes #174

How it was verified

  • Reproduced on current main (1c9ced6): with only that docstring corrected, sprout check-tuning-scope --base origin/main failed and asked for a Tunes-Against: trailer.
  • With this branch it passes. Changing one statement in the same file (the "deterministic" provider literal) in a throwaway commit still fails the check, so the real-repo check works in both directions.
  • The new tests in tests/test_tuning_scope.py run once for each fingerprint path. A docstring-only edit to a module, class, method, async method or docstring-only function passes. A same-size edit to the statement next to it fails, with or without a docstring edit in the same commit. Editing a bare string that is not in docstring position also fails. Unit cases cover the edge cases in both directions: moving a docstring below a statement, f-string and bytes literals, and cutting a function body down to its docstring.
  • Fails before, passes after: with tuning_scope.py restored from main, 9 of the 20 new tests fail (the four docstring-only gate tests and the five fingerprint-ignores-docstring cases). The other 11 are controls that pass either way, as they should. With the fix, all 20 pass.
  • make verify is green: lint, mypy strict, 1333 tests at 94.84% coverage, pip-audit, semgrep, gitleaks, eval, a11y, the site, claims and other checks, and 247 web-static tests. The local pre-push hook script also passed.

Definition of done

  • make verify is green locally (lint · type · test ≥90% · security · eval · a11y)
  • Tests added/updated; acceptance criteria linked to an issue
  • If a guardrail changed (guards.py, confidence.py, eval/), an ADR is linked and a CODEOWNER reviewed. This touches src/sprout/eval/tuning_scope.py. The issue calls widening what the gate accepts a governance decision for the owner. No ADR is included, and this needs owner review.
  • Docs / CHANGELOG updated; ISO 25010 quality characteristic named. CONTRIBUTING.md tuning discipline, the docs/ROADMAP.md Phase 3 line, the check-tuning-scope help text and an Unreleased CHANGELOG entry all now describe the docstring exemption.
  • Eval report regenerated if behavior changed; no baseline regression. Runtime behavior did not change; only the CI gate and one docstring did.
  • Rollback plan noted
  • New runtime dependency? n/a

ISO 25010 characteristic(s) this PR touches

Maintainability (modifiability): prose in tunable modules can be corrected again. Functional correctness of the gate itself: it now does what its own docstring says.

New runtime dependency? Why + alternatives considered

n/a

Rollback plan

Revert the two commits. The gate goes back to counting docstrings as code. The provider docstring can stay corrected only if the revert also brings back the KNOWN_GAP entry, or if that commit is kept with a Tunes-Against: trailer.

Prepared with AI assistance; reviewed before submission.

A docstring is an Expr(Constant(str)) statement, so ast.dump carried it and
a prose-only edit to a tunable module moved the fingerprint exactly as a
ranking change does. That demanded a Tunes-Against trailer no honest commit
can carry.

Every Python comparison the gate makes now drops the leading string of each
module, class and function body before dumping. Bare strings elsewhere,
f-strings and bytes literals stay in the tree. The premise (nothing in the
package reads a docstring) is pinned by a test, and both directions are
covered on every fingerprint path: docstring-only edits pass, same-size
edits to the adjacent statement still fail.

Closes #174
…ring

The module docstring told readers that `pip install sprout` runs end to end
with no extras. That installs an unrelated library. Before the previous commit
the tuning-scope gate counted this one-word prose fix as tuning, so it was
held back as the single KNOWN_GAP entry in test_install_instructions.py. It
is corrected now, and the exemption and the test that pinned it are removed
as their own comment said they should be.

Refs #174
@ChelseaKR
ChelseaKR merged commit 23cad03 into main Oct 2, 2026
15 checks passed
@ChelseaKR
ChelseaKR deleted the fix/tuning-fingerprint-ignores-docstrings branch October 2, 2026 17:34
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

A docstring-only change to a tunable module cannot land: the tuning-scope fingerprint counts docstrings as code

1 participant