Convert license URL to license ID

[bact]

In some metadata schema, URL is used for license.
Have to convert it to license ID.

For example, these URLs

• https://opensource.org/license/Apache-2.0
• https://opensource.org/license/apache-2.0
• https://www.apache.org/licenses/LICENSE-2.0
• https://spdx.org/licenses/Apache-2.0
• https://spdx.org/licenses/Apache-2.0.html
• https://scancode-licensedb.aboutcode.org/apache-2.0.html
• spdx:Apache-2.0

should all be converted to

Apache-2.0

[bact]

Eventually, we need a dataset for this and a lookup table in the implementation.

[bact]

https://github.com/aboutcode-org/scancode-licensedb has this kind of info.

See https://github.com/aboutcode-org/scancode-licensedb/blob/main/docs/apache-2.0.yml

[bact]

With the URL input, the license identification flow will be like this:

• If text looks like a URL, resolve the license ID by the URL first.
• If text looks like an SPDX License Expression, parse it first.
• If text looks like an SPDX License ID, lookup by ID to check its validility (and normalize it if necessary).
• If text is very short, could be a license name, lookup by name.
• If text does not look like pure legal/license text, try license marker detection.

[bact]

Note the URL changes in SPDX License List XML spdx/license-list-XML@8da21bb, we may like to have both old and new URLs for matching purpose.