Repository navigation
Refuse vague licence names ourselves, and decide precedence in one place - #34
Merged
Merged
Conversation
NEVER_RESOLVE held "gpl" and "gpl+". Every other name the module
promises never to guess at -- "BSD", "LGPL", "Apache", "Public Domain",
"BSD-like" -- stayed unresolved only because ospac happened not to carry
it.
That is a coincidence, not a promise. ospac is actively filling in folk
spellings at our own request (ospac#88, #89), and a release adding
`bsd -> BSD-3-Clause` would have made normalize("BSD") answer with a
confident wrong identifier. The test protecting that behaviour would
have failed after it changed rather than instead of it, and anyone
running a build in between would have got the wrong licence.
The family names are on the denylist now, checked before any lookup, so
no upstream data reaches them.
An adopter overlay still lifts a refusal for a name it maps explicitly.
That behaviour is documented and tested, and the existing suite caught
the first version of this change removing it -- the denylist exists to
stop this tool guessing, and to stop somebody else's data guessing on
its behalf, not to overrule the person running it. `never_resolve` in
the same overlay is the more specific statement and still wins.
Sweep unchanged: 65,870 findings, zero different. Nothing that resolved
before stops resolving, because none of these names resolved before.
Both found by review, both the same hole one step further along. `descriptive_key` strips a leading "The" and the word "Version", not the word "License". So "Apache License" is a different key from "apache", and denylisting only the bare token left the spelling SBOMs actually use protected by upstream omission -- exactly the state this list was widened to end. The families written out are refused too, and only where the family is ambiguous: "MIT License", "ISC License" and "Zlib License" name one licence each and still resolve, and a version makes any family specific again. An overlay that maps a name and refuses it in the same file had its refusal subtracted in the exact key space only, so `never_resolve: ["The MIT License"]` left the descriptive key lifted and "MIT License" resolved through the alias the same overlay had just refused. Both spellings of a refusal are subtracted now. Nothing that named a licence stops resolving: sweep unchanged at 65,870 findings, and the versioned and single-licence spellings are pinned by tests rather than checked by hand.
Third round of review, third spelling missed: abbreviations, then "<family> License", now the written-out names -- "Eclipse Public License", "Mozilla Public License", "Common Development and Distribution License". Each fix closed the example and left the class, which is what enumeration does. The missing names are added, and a test now generates the shapes from the families rather than trusting the list to be complete. For every family whose SPDX identifiers differ only by version, each spelling is checked bare, title-cased, upper-cased and with a leading "The". The list and the test have to agree, so the next gap fails a test instead of shipping. The other half is generated too: a version turns any of these back into a name that identifies one licence, and those must keep resolving. That is what real SBOMs carry, and it is the reason refusing the family is safe. Sweep unchanged: 65,870 findings, zero different.
Overrides were applied by subtracting the adopter's names from the denylist, which put them after the SPDX parser and split them across two key spaces. Three bugs came out of that in one review round: an overlay mapping "gpl" got the parser's deprecated GPL-1.0-or-later instead of the identifier the adopter chose, and lifting one spelling left the other refused, so the same name answered two ways. The adopter's own tables are consulted first now, before anything else looks at the string. Their never_resolve, then their aliases, both matched in either spelling. No subtraction, no lifting, nothing to drift apart. That also settles a rule that was inconsistent. Ambiguity used to beat an adopter's alias while the denylist did not, which is two different answers to one question. There is one now: an explicit statement by the adopter beats every refusal this tool would make, because they are describing their corpus rather than guessing at it. Their never_resolve still beats their own alias, being the more specific of the two. Two tests written earlier today asserted the old split and were changed rather than the code, which is called out in their names. Sweep unchanged: 65,870 findings, zero different.
A refusal was matched against the whole string, so an expression walked straight past it. "GPL" was refused and "GPL/MIT" returned `GPL-1.0-or-later AND MIT` -- the confident wrong answer the refusal exists to prevent, in the spelling SBOMs most often use for dual licensing. "(GPL)" did it with one token and no operator at all. An adopter's own never_resolve was bypassed the same way. Operands are checked now, on the parsed form and on the separator-rewritten form, since "GPL/MIT" only shows its operands once the separator has become an operator. Only text that parses as an expression is split, so a name that legitimately contains punctuation stays one name: "Apache License, Version 2.0" is unaffected. Also fixes the back-application, which rewrote the merged alias table and not the adopter's own -- the one normalize reads first. With two overlays declaring keys that share a descriptive key, "Apache License, Version 2.0" answered with the earlier overlay's value while "The Apache License, Version 2.0" answered with the later one. One existing test asserted `Apache-2.0 plus MIT` resolves while the same overlay refused MIT -- it was asserting the hole. It now checks the separator rewrite on an operand the overlay says nothing about, and asserts the refusal reaches inside the compound. Sweep unchanged: 65,870 findings, zero different.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
The bug
licenses.pypromises that a name identifying a licence family but not a licence is never guessed at — "BSD" could be any of four, so the honest answer is that it cannot be resolved.That promise was being kept by accident.
NEVER_RESOLVEheld two entries. "BSD", "Apache", "LGPL", "Public Domain" and the rest were unresolved only because ospac did not happen to carry them.ospac is now filling in folk spellings — at our own request, in SemClone/ospac#88 and #89. A release adding
bsd -> BSD-3-Clausewould have madenormalize("BSD")answer with a confident wrong identifier, silently. The test guarding it would have failed after the behaviour changed rather than instead of it.A promise kept by someone else's omission is not kept.
The fix
Two parts.
The refusal is ours. The family names are on the denylist, checked before any lookup, so no upstream data reaches them. Only families whose SPDX identifiers differ by version: "MIT License", "ISC License" and "Zlib License" name one licence each and still resolve, and a version makes any family specific again.
Precedence is decided in one place. An adopter overlay still overrides a refusal — they are describing their corpus, not guessing. That was first implemented by subtracting their names from the denylist, which put the override after the SPDX parser and split it across two key spaces. Three bugs came out of that in a single review round:
The adopter's own tables are consulted first now, before anything else looks at the string — their
never_resolve, then their aliases, both matched in either spelling. No subtraction, no lifting, nothing to drift.One rule, where there were two
Ambiguity used to beat an adopter's alias while the denylist did not. There is one rule now: an explicit statement by the adopter beats every refusal this tool would make. Their
never_resolvestill beats their own alias, being the more specific of the two.Two tests written earlier today asserted the old split. They were changed rather than the code, and their names say so.
Four rounds, and what they cost
Review found a new instance each round — the abbreviations, then
<family> License, then the written-out names, then the precedence bugs. Each fix closed the example and left the class, which is what enumeration does.The list stopped growing at round three. A test now generates the shapes from the families rather than trusting the list: every family whose SPDX ids differ only by version, each spelling checked bare, title-cased, upper-cased and article-prefixed, plus the inverse — a version must make it resolve again. The list and the test have to agree, so the next gap fails a test instead of shipping.
Also here
A test asserted which lookup path resolved a real licence, which is ospac's data to decide. ospac 1.7.0 moved one spelling between paths and
mainwent red on a release nobody here made. It builds its own alias now and asserts the mechanism rather than upstream's contents.Verification
Nothing that named a licence stops resolving — none of the refused names ever did.