Repository navigation
fix(eval): tuning-scope fingerprint ignores docstrings - #210
Merged
Merged
Conversation
A docstring is an Expr(Constant(str)) statement, so ast.dump carried it and a prose-only edit to a tunable module moved the fingerprint exactly as a ranking change does. That demanded a Tunes-Against trailer no honest commit can carry. Every Python comparison the gate makes now drops the leading string of each module, class and function body before dumping. Bare strings elsewhere, f-strings and bytes literals stay in the tree. The premise (nothing in the package reads a docstring) is pinned by a test, and both directions are covered on every fingerprint path: docstring-only edits pass, same-size edits to the adjacent statement still fail. Closes #174
…ring The module docstring told readers that `pip install sprout` runs end to end with no extras. That installs an unrelated library. Before the previous commit the tuning-scope gate counted this one-word prose fix as tuning, so it was held back as the single KNOWN_GAP entry in test_install_instructions.py. It is corrected now, and the exemption and the test that pinned it are removed as their own comment said they should be. Refs #174
2 tasks
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What & why
A change confined to a docstring in a tunable module could not land.
sprout check-tuning-scopetreated it as a behavior change and asked for aTunes-Against:trailer, and no honest trailer exists for prose. The cause was_python_fingerprintinsrc/sprout/eval/tuning_scope.py. It says it is comment-insensitive, and that holds for#comments, whichastdrops. A docstring is anExpr(Constant(str))statement, though, so it stayed in theast.dump.This PR adds
_DropDocstrings, which removes only the leading string constant of a module, class or function body. All four Python comparisons the gate makes now use it: the default fingerprint plus theanswer.py,config.pyand provider-factory fingerprints. A bare string anywhere else, an f-string and a bytes literal all stay in the tree.The exemption meets the same standard as the existing three. Nothing in the package reads a docstring, and the new
test_package_never_reads_a_docstringpins that. It scanssrc/sprout/for__doc__,getdoc,get_docstring,model_json_schemaanduse_attribute_docstrings, so it fails the day one of those appears.A second commit fixes the docstring from the issue. It is in
src/sprout/providers/__init__.pyand told readers topip install sprout, which installs an unrelated library. That commit also removes theKNOWN_GAPentry andtest_the_known_gap_is_still_a_gapfromtests/test_install_instructions.py, as the entry's comment said should happen.Closes #174
How it was verified
sprout check-tuning-scope --base origin/mainfailed and asked for aTunes-Against:trailer."deterministic"provider literal) in a throwaway commit still fails the check, so the real-repo check works in both directions.tests/test_tuning_scope.pyrun once for each fingerprint path. A docstring-only edit to a module, class, method, async method or docstring-only function passes. A same-size edit to the statement next to it fails, with or without a docstring edit in the same commit. Editing a bare string that is not in docstring position also fails. Unit cases cover the edge cases in both directions: moving a docstring below a statement, f-string and bytes literals, and cutting a function body down to its docstring.tuning_scope.pyrestored from main, 9 of the 20 new tests fail (the four docstring-only gate tests and the five fingerprint-ignores-docstring cases). The other 11 are controls that pass either way, as they should. With the fix, all 20 pass.make verifyis green: lint, mypy strict, 1333 tests at 94.84% coverage, pip-audit, semgrep, gitleaks, eval, a11y, the site, claims and other checks, and 247 web-static tests. The local pre-push hook script also passed.Definition of done
make verifyis green locally (lint · type · test ≥90% · security · eval · a11y)guards.py,confidence.py,eval/), an ADR is linked and a CODEOWNER reviewed. This touchessrc/sprout/eval/tuning_scope.py. The issue calls widening what the gate accepts a governance decision for the owner. No ADR is included, and this needs owner review.check-tuning-scopehelp text and an Unreleased CHANGELOG entry all now describe the docstring exemption.ISO 25010 characteristic(s) this PR touches
Maintainability (modifiability): prose in tunable modules can be corrected again. Functional correctness of the gate itself: it now does what its own docstring says.
New runtime dependency? Why + alternatives considered
n/a
Rollback plan
Revert the two commits. The gate goes back to counting docstrings as code. The provider docstring can stay corrected only if the revert also brings back the
KNOWN_GAPentry, or if that commit is kept with aTunes-Against:trailer.Prepared with AI assistance; reviewed before submission.