diff --git a/README.md b/README.md index 84375f3e..322d210d 100644 --- a/README.md +++ b/README.md @@ -93,12 +93,12 @@ llvm.toolchain( ) ``` -Prereleases must be requested explicitly. Following the common SemVer range -convention, they are excluded from `latest`, `first`, and stable-only version -requirements. An exact `llvm_version = "23.1.0-rc3"` selects that release, and -a requirement opts in by naming a prerelease with the same -major/minor/patch tuple. Thus `latest:>=23.1.0-rc1` can select `23.1.0-rc3`, -while `latest:>=23` and `latest:>22` cannot select it. +Automatic `latest` and `first` selection excludes prereleases by default, including +when a requirement explicitly mentions a prerelease. Set `llvm_version_allow_prereleases = True` +to make matching LLVM prereleases eligible, or +`linker_version_allow_prereleases = True` for catalogued linkers, including mold. Exact pins +such as `llvm_version = "23.1.0-rc3"` select that release directly regardless of the +option. `extra_llvm_distributions` is not limited to official LLVM artifacts. Its keys may also be complete download URLs or absolute local paths, so internally @@ -236,25 +236,105 @@ for how CC toolchains work in general. ### Requirements -Version attributes can be requirements of the form `first`, `first:`, -`latest` or `latest:`. +Version requirements let projects bind automatic LLVM upgrades to deliberate `toolchains_llvm` +release upgrades. A `toolchains_llvm` release can add new LLVM release identifiers to its +checksummed catalogue. Upgrading the pinned module then makes those releases eligible without a +separate edit to an exact `llvm_version`. Requirements restrict which of those additions a project +will adopt, for example by keeping LLVM within major version 23: -In case of `latest`, the latest distribution matching the optional `condition` -will be selected. +```starlark +llvm.toolchain( + name = "llvm_toolchain", + llvm_version = "latest:major==23", +) +``` -In case of `first`, the first distribution matching the optional `condition` -will be selected. +With this policy, upgrading `toolchains_llvm` to a release that adds a newer stable LLVM 23 +version also upgrades the selected compiler, when an archive is available for the execution host. +Even a module release that only adds LLVM release identifiers can therefore update LLVM as part +of the same dependency upgrade. Adding LLVM 24 to the catalogue leaves this selection within +LLVM 23; adopting LLVM 24 requires an explicit selector change. Bare `latest` permits major +upgrades introduced by module upgrades as well. Exact versions remain useful when an LLVM +release must match another artifact, such as an instrumented libc++ build. + +Selection uses the catalogue bundled with the pinned `toolchains_llvm` module plus explicitly +configured extra distributions. Builds do not query upstream for newly published LLVM releases. +With the same module, configured catalogue and execution host, the selected release stays the +same. A module upgrade or an explicit catalogue change is what makes additional releases eligible. + +Version attributes accept exact versions or selectors of the form `first`, `first:`, +`latest`, or `latest:`. `latest` picks the newest matching release available for the +execution host; `first` picks the oldest. The same clause syntax applies to LLVM and catalogued +linker versions, and lets one configuration accommodate hosts whose available archives differ. + +Conditions are comma-separated and combined with AND. toolchains_llvm supplies `major`, `minor`, +and `patch` as numeric release-component constants, so `latest:major==23,minor>=2` stays within +LLVM 23 and requires a minor version of at least 2. Comparisons and function expressions use +[mboworks_bzl's SemVer mode](https://mboworks.github.io/bzl/site/tag/0.7.0/docs/versions.html#semver). + + +| Selector | Selection policy | +| --------------------------- | ---------------------------------------------- | +| `latest:major==23` | Newest stable LLVM 23 release for the host | +| `latest:major==23,minor==1` | Newest stable 23.1 release for the host | +| `latest:~=23.1.0,!=23.1.2` | Newest stable compatible release except 23.1.2 | +| `first:>=22,<23` | Oldest stable LLVM 22 release for the host | + +For all comparison operators, wildcard matching and compatible-release boundaries, see +[Compatible and matching requirements](https://mboworks.github.io/bzl/site/tag/0.7.0/docs/versions.html#compatible-and-matching-requirements). +Function clauses such as `major()==23` and more advanced inspection are documented under +[Requirement expressions and constants](https://mboworks.github.io/bzl/site/tag/0.7.0/docs/versions.html#requirement-expressions-and-constants). + +`llvm_version_allow_prereleases` and `linker_version_allow_prereleases` each +default to `False`. They independently control bare and constrained +`first`/`latest` selectors for LLVM and catalogued linkers, respectively. Requirements always +apply; enabling prereleases does not bypass ranges or host availability. Naming +an RC in a requirement does not implicitly enable prereleases. Exact version pins +are unaffected. -The condition consists of a comma separated list of semver version comparisons -supporting `<`, `<=`, `>`, `>=`, `==`, `!=`. Examples: +```starlark +llvm.toolchain( + name = "llvm_rc", + llvm_version = "latest:>=23.1.0-rc1,<23.1.0", + llvm_version_allow_prereleases = True, +) +``` + +This selects the newest matching release candidate available for the execution +host. With `llvm_version_allow_prereleases = False`, that RC-only range has no matching release. +With the option enabled, expressions such as `is_prerelease()==True` can further +restrict selection to prereleases; `is_prerelease()==False` can restrict a +particular selector to stable releases. + +Linker selectors use the same bzl SemVer comparisons, requirement expressions, +projection constants, and first/latest selection as LLVM. For example: + +```starlark +llvm.toolchain( + name = "llvm_with_mold_rc", + llvm_version = "latest:major==23", + llvm_version_allow_prereleases = False, + linker = {"": "mold"}, + linker_version = "latest:major==3,is_prerelease()==True", + linker_version_allow_prereleases = True, +) +``` -- `latest` -- `latest:>=20.1.0` -- `latest:>17.0.4,!=19.1.7,<=20.1.0` -- `first:>=15.0.6,<16` +This selects stable LLVM and a mold prerelease independently. A matching mold +prerelease must exist in the configured linker catalogue (for example, through +`extra_linker_distribution_files`); selection does not discover live releases. +Linker candidates are restricted to the execution platform before version selection, +just as LLVM candidates are restricted to compatible archives. An exact +`linker_version = "3.0.0-rc1"` does not require enabling prereleases. Bare `mold` +without a version derives an exact pin from the injected mold module. The linker +flag has no effect on bundled linkers, `auto`, or absolute executable paths. +Both flags also work in feature overrides; changing only the linker flag can +reuse the base LLVM archive. It is further possible to provide the version or requirement from an environment -variable with a fallback version or requirement. In this case it is important to +variable with a fallback version or requirement. This works for both LLVM and +linker versions, for example `linker_version = "getenv(MOLD_VERSION,latest:major==3)"` +with `--repo_env=MOLD_VERSION=3.0.0`. In this case it is important to also use the bazel flag `--repo_env=LLVM_VERSION=version_or_requirement`. It is important to use both correctly because otherwise the resulting builds are not reproducible. The main purpose of using an environment variable to encode the diff --git a/toolchain/deps.bzl b/toolchain/deps.bzl index beb02c3e..bb37af6c 100644 --- a/toolchain/deps.bzl +++ b/toolchain/deps.bzl @@ -59,7 +59,6 @@ def bazel_toolchain_dependencies(): url = "https://github.com/bazel-contrib/bazel_features/releases/download/v1.52.0/bazel_features-v1.52.0.tar.gz", ) - # Load mboworks_bzl for version comparisons. if not native.existing_rule("mboworks_bzl"): http_archive( name = "mboworks_bzl", diff --git a/toolchain/internal/BUILD.bazel b/toolchain/internal/BUILD.bazel index eab79151..dab3b963 100644 --- a/toolchain/internal/BUILD.bazel +++ b/toolchain/internal/BUILD.bazel @@ -15,12 +15,15 @@ load("@bazel_skylib//rules:diff_test.bzl", "diff_test") load(":feature_overrides_test.bzl", "feature_overrides_test_suite") load("linker_distributions.bzl", "linker_version_test_writer") -load("llvm_distributions.bzl", "distributions_test_writer", "prerelease_test_writer", "requirements_test_writer") +load(":llvm_distributions_test.bzl", "distributions_test_writer", "prerelease_test_writer", "requirements_test_writer") +load(":version_selectors_test.bzl", "version_selectors_test_suite") exports_files(["template.modulemap"]) feature_overrides_test_suite(name = "feature_overrides_tests") +version_selectors_test_suite(name = "version_selectors_tests") + linker_version_test_writer( name = "linker_version_test_output", testonly = True, diff --git a/toolchain/internal/feature_overrides_test.bzl b/toolchain/internal/feature_overrides_test.bzl index d1d02732..f8f365e5 100644 --- a/toolchain/internal/feature_overrides_test.bzl +++ b/toolchain/internal/feature_overrides_test.bzl @@ -113,6 +113,19 @@ def _archive_sharing_test_impl(ctx): asserts.false(env, can_share_llvm_archive(base, {}, reset = ["distribution"])) asserts.false(env, can_share_llvm_archive(base, dict(toolchain_roots = {"": "/opt/llvm"}))) + asserts.true(env, can_share_llvm_archive(base, dict(linker_version_allow_prereleases = True))) + asserts.true(env, can_share_llvm_archive(base, {}, reset = ["linker_version_allow_prereleases"])) + asserts.false(env, can_share_llvm_archive(base, dict(llvm_version_allow_prereleases = True))) + asserts.false(env, can_share_llvm_archive(base, {}, reset = ["llvm_version_allow_prereleases"])) + for llvm_allow in [False, True]: + for linker_allow in [False, True]: + settings = dict(llvm_version_allow_prereleases = llvm_allow, linker_version_allow_prereleases = linker_allow) + override = llvm_feature_override(features = ["test"], **settings) + merged = merge_override(dict(base, llvm_version_allow_prereleases = not llvm_allow, linker_version_allow_prereleases = not linker_allow), settings) + asserts.equals(env, llvm_allow, merged["llvm_version_allow_prereleases"]) + asserts.equals(env, linker_allow, merged["linker_version_allow_prereleases"]) + asserts.equals(env, settings, override["settings"]) + # Resetting an external root must create a new download: the base has no # generated LLVM archive repository that the variant could share. external_base = dict(base, toolchain_roots = {"": "/opt/llvm"}) diff --git a/toolchain/internal/linker_distributions.bzl b/toolchain/internal/linker_distributions.bzl index 711603d6..2c285a52 100644 --- a/toolchain/internal/linker_distributions.bzl +++ b/toolchain/internal/linker_distributions.bzl @@ -17,7 +17,9 @@ load("@mboworks_bzl//bzl/versions:versions.bzl", "versions") load("//toolchain/internal:common.bzl", "is_absolute_path") load("//toolchain/internal:distributions_repo.bzl", "load_jsonc") -load("//toolchain/internal:llvm_distributions.bzl", "resolve_version") +load("//toolchain/internal:version_selectors.bzl", "version_selectors") + +_semver = versions.with_mode("semver") def catalogued_linker_reference(selection, version): """Return the catalogue reference for a linker selection, or None.""" @@ -31,7 +33,7 @@ def llvm_23_macos_uses_bundled_linker(llvm_version, linker_dict): """Whether any Darwin target retains a bundled LLVM 23 linker without arm64e.x1 support.""" # LLVM 23.1.3 backports arm64e.x1 support to ld64.lld (llvm/llvm-project#224185). - if not llvm_version.startswith("23.") or not versions.lt(llvm_version, "23.1.3"): + if not llvm_version.startswith("23.") or not _semver.lt(llvm_version, "23.1.3"): return False default = linker_dict.get("", "") return any([ @@ -80,7 +82,7 @@ def _merge_catalogue(catalogue, extra): merged_versions[version] = merged_platforms merged_platforms.update(platforms) -def resolve_linker_distribution(catalogue, linker, version_selection, platform): +def resolve_linker_distribution(catalogue, linker, version_selection, platform, *, allow_prereleases = False): """Resolve a version against releases available for an execution platform.""" versions = catalogue.get(linker) if not versions: @@ -92,7 +94,7 @@ def resolve_linker_distribution(catalogue, linker, version_selection, platform): ] if not available_versions: fail("linker '{}' has no executable for the {} execution platform".format(linker, platform)) - version = resolve_version(version_selection, available_versions) + version = version_selectors.resolve(version_selection, available_versions, allow_prereleases = allow_prereleases) return version, versions[version][platform] def _mold_version(rctx): @@ -124,6 +126,7 @@ def _linker_distributions_repository_impl(rctx): linker, version_selection = "mold", _mold_version(rctx) else: linker, version_selection = _parse_reference(reference) + version_selection = version_selectors.resolve_environment(rctx, version_selection) versions = catalogue.get(linker) if not versions: fail("unknown linker '{}'; known linkers: {}".format(linker, ", ".join(sorted(catalogue.keys())))) @@ -136,7 +139,7 @@ def _linker_distributions_repository_impl(rctx): exec_os = platform.split("-")[0] if all([target and target.split("-")[0] != exec_os for target in references[reference]]): continue - _, distribution = resolve_linker_distribution(catalogue, linker, version_selection, platform) + _, distribution = resolve_linker_distribution(catalogue, linker, version_selection, platform, allow_prereleases = rctx.attr.linker_version_allow_prereleases) extraction_dir = "_extract/{}".format(index) rctx.download_and_extract( @@ -167,6 +170,7 @@ exports_files(["linkers.json"]) linker_distributions_repository = repository_rule( implementation = _linker_distributions_repository_impl, attrs = { + "linker_version_allow_prereleases": attr.bool(default = False), "catalogue": attr.label( allow_single_file = [".json", ".jsonc"], default = Label("//toolchain/distributions:linkers.jsonc"), @@ -191,7 +195,7 @@ def _linker_version_test_writer_impl(ctx): "latest:>=2.40.0,!=2.42.1", ] version_results = [ - "{} -> {}".format(selection, resolve_version(selection, available)) + "{} -> {}".format(selection, version_selectors.resolve(selection, available)) for selection in selections ] reference_results = [ diff --git a/toolchain/internal/llvm_distributions.bzl b/toolchain/internal/llvm_distributions.bzl index f76faf38..d34a068e 100644 --- a/toolchain/internal/llvm_distributions.bzl +++ b/toolchain/internal/llvm_distributions.bzl @@ -26,6 +26,7 @@ load( "host_info", ) load("//toolchain/internal:distributions_repo.bzl", "load_distribution_files") +load("//toolchain/internal:version_selectors.bzl", "version_selectors") # The merged distribution table is built at module-load time by the # `llvm_distributions` module extension (see @@ -42,6 +43,8 @@ load("//toolchain/internal:distributions_repo.bzl", "load_distribution_files") # - extra.jsonc - empty by default; downstream slot for additional # bundled distributions (private mirrors, internal # builds, etc.). +_semver = versions.with_mode("semver") + _llvm_distributions = LLVM_DISTRIBUTIONS # Per-basename download URL map, fully expanded from the `_meta.base_url` @@ -61,21 +64,7 @@ _llvm_distribution_urls = LLVM_DISTRIBUTION_URLS _DEFAULT_URL_TEMPLATE = "https://github.com/llvm/llvm-project/releases/download/llvmorg-{version}/" def _parse_version(v): - return tuple(versions.parse(v)) - -def _version_string(version): - core = [] - suffix = "" - for part in version: - if part in ["-", "+"]: - suffix += part - elif suffix: - # The versions helper tokenizes LLVM's `rc3` identifier as - # `"rc", 3`; LLVM release tags use the compact spelling. - suffix += str(part) - else: - core.append(str(part)) - return ".".join(core) + suffix + return tuple(_semver.parse(v)) def _distribution_basename(distribution): return distribution.split("?", 1)[0].split("#", 1)[0].split("/")[-1].split("\\")[-1].replace("%2B", "+") @@ -90,46 +79,6 @@ def _distribution_version_string(distribution): version += "-" + parts[2] return version -def _is_prerelease(version): - return "-" in version - -def _parsed_version_core(parsed): - core = [] - for part in parsed: - if part in ["-", "+"]: - break - core.append(part) - return tuple(core) - -def _version_core(version): - return _parsed_version_core(_parse_version(version)) - -def _requirements_allow_prerelease(version, requirements): - if not _is_prerelease(version): - return True - if not requirements: - return False - core = _version_core(version) - return any([ - "-" in requirement.version and _parsed_version_core(requirement.version) == core - for requirement in requirements - ]) - -def _sort_versions(version_strings, *, reverse): - result = [] - for version in version_strings: - inserted = False - for pos in range(len(result)): - if versions.cmp(result[pos], version) >= 0: - result = result[:pos] + [version] + result[pos:] - inserted = True - break - if not inserted: - result.append(version) - if reverse: - result = result[::-1] - return result - def _distribution_version(distribution): # Return the version string of a distribution. return _parse_version(_distribution_version_string(distribution)) @@ -606,103 +555,29 @@ def _is_requirement(version_or_requirements): return True return False -def _parse_version_or_requirements(version_or_requirements): - for prefix in ["latest:", "first:"]: - if version_or_requirements.startswith(prefix): - return versions.parse_requirements(version_or_requirements.removeprefix(prefix)) - if version_or_requirements in ["latest", "first"]: - return None - if not _is_requirement(version_or_requirements): - return None - fail("ERROR: Invalid version requirements: '{version_or_requirements}'.".format( - version_or_requirements = version_or_requirements, - )) - -def resolve_version(version_or_requirements, available_versions): - """Resolve an exact version or first/latest requirement against versions.""" - if not version_or_requirements: - fail("ERROR: Empty version selection.") +def _required_llvm_release_name(*, version_or_requirements, all_llvm_distributions, host_info, allow_prereleases = False): if not _is_requirement(version_or_requirements): - if version_or_requirements not in available_versions: - fail("ERROR: Version '{}' is not available; supported versions: {}".format( - version_or_requirements, - ", ".join(_sort_versions(available_versions)), - )) - return version_or_requirements - - requirements = _parse_version_or_requirements(version_or_requirements) - for version in _sort_versions(available_versions, reverse = version_or_requirements.startswith("latest")): - if not requirements or versions.check_all_requirements(version, requirements): - return version - fail("ERROR: No version matches '{}' in: {}".format( + basename, error = _find_llvm_basename_or_error(version_or_requirements, all_llvm_distributions, host_info) + return version_or_requirements, basename, error + + # Restrict to unambiguous host-compatible archives before selecting a version. + available = {} + for distribution in all_llvm_distributions: + version = _distribution_version_string(distribution) + if version not in available: + available[version] = _find_llvm_basename_list(version, all_llvm_distributions, host_info) + selected = version_selectors.select( version_or_requirements, - ", ".join(_sort_versions(available_versions)), - )) - -def _get_version_from_distribution(distribution): - # We assume here that the `distribution` is a basename of the form `LLVM--...` or - # `clang+llvm--...`. - return _distribution_version_string(distribution) - -def _get_llvm_versions(*, version_or_requirements, all_llvm_distributions): - requirements = _parse_version_or_requirements(version_or_requirements) - llvm_version_dict = {} - for distribution in all_llvm_distributions.keys(): - version = _get_version_from_distribution(distribution) - - # Match the common SemVer range convention: prereleases are excluded - # unless a comparator explicitly mentions a prerelease with the same - # major/minor/patch tuple. - if not _requirements_allow_prerelease(version, requirements): - continue - llvm_version_dict[version] = None - - return _sort_versions(llvm_version_dict.keys(), reverse = version_or_requirements.startswith("latest")) - -def _required_llvm_release_name(*, version_or_requirements, all_llvm_distributions, host_info): - llvm_versions = _get_llvm_versions(version_or_requirements = version_or_requirements, all_llvm_distributions = all_llvm_distributions) - requirements = _parse_version_or_requirements(version_or_requirements) - for llvm_version in llvm_versions: - if requirements and not versions.check_all_requirements(llvm_version, requirements): - continue - basenames = _find_llvm_basename_list(llvm_version, all_llvm_distributions, host_info) - if len(basenames) == 1: - return llvm_version, basenames[0], None + [(version, names[0]) for version, names in available.items() if len(names) == 1], + key = lambda item: item[0], + allow_prereleases = allow_prereleases, + ) + if selected != None: + return selected[0], selected[1], None return None, None, "ERROR: No matching distribution found." -def _resolve_llvm_version_rctx_env(rctx, llvm_version): - if llvm_version.startswith("getenv(") and llvm_version.endswith(")"): - env_var = llvm_version[len("getenv("):-1] - if env_var.find(",") >= 0: - env_name, env_default = env_var.split(",", 1) - else: - env_name = env_var - env_default = None - - env_name = env_name.strip(" ") - if not env_var: - fail("ERROR: Bad getenv parameter '{env_var}'.".format( - env_var = env_var, - )) - - # We prefer 'repository_ctx.getenv' if it is available (~7.1+) and default - # to accessing the environment directly. The latter breaks "hermeticity". - if hasattr(rctx, "getenv"): - llvm_version = rctx.getenv(env_name, env_default) - elif env_name in rctx.os.environ: - llvm_version = rctx.os.environ[env_name] - else: - llvm_version = env_default - - if not llvm_version: - fail("ERROR: Empty getenv lookup for '{env_var}'.".format( - env_var = env_var, - )) - - return llvm_version - def _required_llvm_release_name_rctx(rctx, llvm_version): - llvm_version = _resolve_llvm_version_rctx_env(rctx, llvm_version) + llvm_version = version_selectors.resolve_environment(rctx, llvm_version) builtin_distributions, user_distributions, _ = _configured_distribution_data(rctx) all_llvm_distributions = _get_all_llvm_distributions( llvm_distributions = builtin_distributions, @@ -713,6 +588,7 @@ def _required_llvm_release_name_rctx(rctx, llvm_version): version_or_requirements = llvm_version, all_llvm_distributions = all_llvm_distributions, host_info = host_info(rctx), + allow_prereleases = rctx.attr.llvm_version_allow_prereleases, ) def required_llvm_version_rctx(rctx): @@ -733,7 +609,7 @@ def _filter_llvm_distributions(*, llvm_version, all_llvm_distributions): """Return (distribution: sha) entries from `all_llvm_distributions` that match `llvm_version`.""" result = {} for k, v in all_llvm_distributions.items(): - if _get_version_from_distribution(k) == llvm_version: + if _distribution_version_string(k) == llvm_version: result[k] = v return result @@ -750,12 +626,13 @@ def _distribution_urls(rctx): if rctx.attr.distribution == "auto": rctx_host_info = host_info(rctx) - llvm_version = _resolve_llvm_version_rctx_env(rctx, llvm_version) + llvm_version = version_selectors.resolve_environment(rctx, llvm_version) if _is_requirement(llvm_version): llvm_version, basename, error = _required_llvm_release_name( version_or_requirements = llvm_version, all_llvm_distributions = all_llvm_distributions, host_info = rctx_host_info, + allow_prereleases = rctx.attr.llvm_version_allow_prereleases, ) else: basename, error = _find_llvm_basename_or_error(llvm_version, all_llvm_distributions, rctx_host_info) @@ -809,387 +686,15 @@ def _distribution_urls(rctx): return urls, sha256, strip_prefix -def _distributions_test_writer_impl(ctx): - """Analyze the configured versions and write to a file for test consumption. - - The test generated file '.out' contains the following lines: - - a 'del:' line denotes a llvm distribution basename that was not found. - - a 'add:' line denotes a version that was predicted but does not exist. - - Lines of type `add:` should never occur as the algorithm is supposed to - verify that predicted distributions have been configured. Otherwise the - algorithm could not know the hash value. - """ - use_llvm_distributions = _llvm_distributions - - # Inject version '0.0.0' that verifies additional behavior using `extra_llvm_distributions`. - extra_llvm_distributions = { - # Keep both formats to verify that automatic selection avoids zstd - # until Bazel supports the large windows used by LLVM 23 archives. - "LLVM-0.0.0-Linux-ARM64.tar.zst": "d6b8679be46bdaa383e0c7f13a473ca8f7a4f87233f2cc0e0a7ab19e1b6265e7", - "LLVM-0.0.0-Linux-ARM64.tar.xz": "a6b8679be46bdaa383e0c7f13a473ca8f7a4f87233f2cc0e0a7ab19e1b6265e7", - "/foo/bar/LLVM-0.0.0-Linux-X64.tar.xz?xyz": "0a764a8ca521606532ca9ec4e5745c933b16b7d30f4701a47ee851d448fcdb74", - "http://server/foo/bar/LLVM-0.0.0-macOS-ARM64.tar.xz#xyz": "9da86f64a99f5ce9b679caf54e938736ca269c5e069d0c94ad08b995c5f25c16", - "http://server/foo/bar/LLVM-0.0.0-macOS-X64.tar.xz": "264f2f1e8b67f066749349ae8b4943d346cd44e099464164ef21b42a57663540", - "http://server/clang%2Bllvm-0.0.0-aarch64-pc-windows-msvc.tar.xz": "5916d93bf80e3ae504022cdd8cb8887be001f9b68a7a08bd268727e8d858afa4", - "http://server/path-to-file/clang%2Bllvm-0.0.0-x86_64-pc-windows-msvc.tar.xz#bla": "5916d93bf80e3ae504022cdd8cb8887be001f9b68a7a08bd268727e8d858afa4", - } - - arch_list = [ - "aarch64", - "armv7a", - "mips", - "mipsel", - "powerpc64", - "powerpc64le", - "sparc64", - "sparcv9", - "x86_32", - "x86_64", - ] - arch_alias_dict = { - "sparc64": ["sparc64", "sparcv9"], - "sparcv9": ["sparcv9", "sparc64"], - } - os_list = [ - "darwin", - "linux", - "windows", - ] - ANY_VERSION = "0" # Version does not matter, but must be a valid integer - dist_dict_list = { - "linux": [ - # keep sorted - struct(name = "amzn", version = ANY_VERSION), - struct(name = "arch", version = ANY_VERSION), - struct(name = "centos", version = "6"), - struct(name = "centos", version = "7"), - struct(name = "chainguard", version = ANY_VERSION), - struct(name = "debian", version = "0"), - struct(name = "debian", version = "8"), - struct(name = "debian", version = "9"), - struct(name = "fedora", version = "26"), - struct(name = "fedora", version = "27"), - struct(name = "fedora", version = "42"), - struct(name = "freebsd", version = "10"), - struct(name = "freebsd", version = "11"), - struct(name = "freebsd", version = "12"), - struct(name = "freebsd", version = "13"), - struct(name = "ibm-aix", version = "7.2"), - struct(name = "linuxmint", version = "18"), - struct(name = "linuxmint", version = "19"), - struct(name = "pc-solaris", version = "2.11"), - struct(name = "raspbian", version = ANY_VERSION), - struct(name = "rhel", version = ANY_VERSION), - struct(name = "sun-solaris", version = "2.11"), - struct(name = "suse", version = "11.3"), - struct(name = "suse", version = "12.2"), - struct(name = "suse", version = "12.3"), - struct(name = "suse", version = "12.4"), - struct(name = "suse", version = "15.5"), - struct(name = "suse", version = "16.0"), - struct(name = "suse", version = "17.0"), - struct(name = "ubuntu", version = "14.04"), - struct(name = "ubuntu", version = "16.04"), - struct(name = "ubuntu", version = "18.04.5"), - struct(name = "ubuntu", version = "18.04.6"), - struct(name = "ubuntu", version = "18.04"), - struct(name = "ubuntu", version = "20.04"), - struct(name = "ubuntu", version = "20.10"), - struct(name = "ubuntu", version = "22.04"), - struct(name = "ubuntu", version = "24.04"), - struct(name = "void", version = ANY_VERSION), - struct(name = "wolfi", version = ANY_VERSION), - ], - } - - # Define the min real version. For earlier injected versions we do no perform dist testing. - MIN_VERSION = _parse_version("6.0.0") - - # Additional output will be generated for versions up to and including `MAX_VERSION` - MAX_VERSION = _parse_version("20.1.3") - version_dict = { - _distribution_version(basename): None - for basename in use_llvm_distributions.keys() + extra_llvm_distributions.keys() - } - versions = sorted(version_dict.keys()) - - # Write versions to output to check which versions we take into account. - output = [] - select = [] - for version in versions: - output.append("version: " + _version_string(version)) - - # We keep track of versions in `not_found` and remove the ones we found. - # So at the end all version that were not found remain, hence the name. - not_found = { - _distribution_basename(distribution): None - for distribution in use_llvm_distributions.keys() + extra_llvm_distributions.keys() - } - - # While computing we add predicted versions that are not configured as True. - # At the end we add the not-found versions as False. - result = {} - - # Collect cases that produce duplicates (or multiple) basenames. - dupes = [] - - # For all versions X arch X os check if we can compute the distribution. - for version in versions: - all_llvm_distributions = _get_all_llvm_distributions( - llvm_distributions = use_llvm_distributions, - extra_llvm_distributions = extra_llvm_distributions, - parsed_llvm_version = version, - ) - for basename, distribution in all_llvm_distributions.items(): - _, shaerr = _normalize_and_check_sha256(distribution.sha256) - if shaerr: - output.append("err: {basename}: bad sha256: {shaerr}".format( - basename = basename, - shaerr = shaerr, - )) - for arch in arch_list: - for os in os_list: - if version < MIN_VERSION: - # Limit the injected version checks - if arch not in ["aarch64", "x86_64"]: - break - dist_list = [struct(name = os, version = "")] - else: - dist_list = dist_dict_list.get(os, [struct(name = os, version = "")]) - for dist in dist_list: - if arch == "sparc64" and dist.name != "sun-solaris": - # Sparc64 and SparcV9 are handled in the same way, just different precedence. - # One is the architecture th other the ISA. Restrict to one to limit output. - continue - host_info = struct( - arch = arch, - os = os, - dist = dist, - ) - basenames = _find_llvm_basename_list(_version_string(version), all_llvm_distributions, host_info) - if version <= MAX_VERSION: - predicted, error = _find_llvm_basename_or_error( - _version_string(version), - all_llvm_distributions, - host_info, - ) - skip_output = False - if error: - if error.startswith("ERROR: No matching config could be found for version"): - skip_output = True - else: - if predicted.endswith(".exe"): - error = "ERROR: Windows .exe is not supported: " + predicted - elif predicted not in all_llvm_distributions: - error = "ERROR: Unavailable prediction: " + predicted - elif len(basenames) == 0: - skip_output = True - elif len(basenames) == 1: - predicted = basenames[0] - else: - error = "ERROR: Multiple selections" - if not error: - arch_found = [arch for arch in arch_list if arch in predicted] - if len(arch_found) == 1 and arch_found[0] not in arch_alias_dict.get(arch, [arch]): - error = "ERROR: Bad arch selection: " + predicted - if not skip_output: - select.append("{version}-{arch}-{os}/{dist_name}/{dist_version} -> {basename}".format( - version = _version_string(version), - arch = arch, - os = os, - dist_name = dist.name, - dist_version = dist.version, - basename = error or predicted, - )) - if len(basenames) != 1: - if basenames: - dupes.append("dup: {version}-{arch}-{os}-{dist_name}-{dist_version} -> {count}".format( - version = _version_string(version), - arch = arch, - os = os, - dist_name = dist.name, - dist_version = dist.version, - count = len(basenames), - )) - dupes.extend([" : " + basename for basename in basenames]) - continue - basename = basenames[0] - if basename in all_llvm_distributions: - if basename in not_found: - not_found.pop(basename) - else: - result[basename] = True - - # Build result - for dist in not_found: - result[dist] = False - output += [("add: " if found else "del: ") + dist for dist, found in result.items()] - output += dupes - ctx.actions.write(ctx.outputs.output, "\n".join(output) + "\n") - ctx.actions.write(ctx.outputs.select, "\n".join(select) + "\n") - -distributions_test_writer = rule( - implementation = _distributions_test_writer_impl, - attrs = { - "output": attr.output(mandatory = True), - "select": attr.output(mandatory = True), - }, -) - -def _requirements_test_writer_impl(ctx): - """Analyze the configured versions and write to a file for test consumption. - The test generated file '.out' contains the following lines: - [,,]: - """ - all_llvm_distributions = { - # In order to prevent new distributions to interfere we cut at 20.1.3. - k: v - for k, v in _llvm_distributions.items() - if _parse_version(_get_version_from_distribution(k)) <= (20, 1, 3) - } - requirement_list = [ - "latest:<=20.1.0", - "latest:<=20.1.0,>17.0.4,!=19.1.7", - "latest:<20.1.0,>17.0.4,!=19.1.7", - "latest:<20.1.0,>17.0.4", - "latest:>=15.0.6,<16", - "first:>=15.0.6,<16", - "latest", - "first", - ] - arch_list = [ - "aarch64", - "armv7a", - "x86_64", - ] - os_list = [ - "darwin", - "linux", - "windows", - ] - ANY_VERSION = "0" # Version does not matter, but must be a valid integer - dist_dict_list = { - "linux": [ - # keep sorted - struct(name = "ubuntu", version = ANY_VERSION), - struct(name = "raspbian", version = ANY_VERSION), - struct(name = "rhel", version = ANY_VERSION), - ], - } - result = [] - for arch in arch_list: - for os in os_list: - dist_list = dist_dict_list.get(os, [struct(name = os, version = "")]) - for dist in dist_list: - for requirement in requirement_list: - host_info = struct( - arch = arch, - os = os, - dist = dist, - ) - llvm_version, basename, error = _required_llvm_release_name( - version_or_requirements = requirement, - all_llvm_distributions = all_llvm_distributions, - host_info = host_info, - ) - if llvm_version and basename: - result.append("[{arch},{os}{dist_name}{dist_version},'{requirement}']: {llvm_version} = {basename}".format( - arch = arch, - os = os, - dist_name = "," + dist.name if os == "linux" else "", - dist_version = "," + dist.version if os == "linux" else "", - requirement = requirement, - llvm_version = llvm_version, - basename = basename, - )) - else: - result.append("[{arch},{os},\"{requirement}\"]: {error}".format( - arch = arch, - os = os, - requirement = requirement, - llvm_version = llvm_version, - basename = basename, - error = error or "ERROR: N/A", - )) - ctx.actions.write(ctx.outputs.result, "\n".join(result) + "\n") - -requirements_test_writer = rule( - implementation = _requirements_test_writer_impl, - attrs = { - "result": attr.output(mandatory = True), - }, -) - -def _prerelease_test_writer_impl(ctx): - # Exercise the checked-in release table so this also proves that published - # prereleases are bundled, not merely supported through user-supplied data. - distributions = _llvm_distributions - host = struct( - arch = "x86_64", - os = "linux", - dist = struct(name = "ubuntu", version = "24.04"), - ) - result = [] - - for requested in ["23.1.0-rc3", "22.1.8"]: - available = _get_all_llvm_distributions( - llvm_distributions = distributions, - extra_llvm_distributions = {}, - parsed_llvm_version = _parse_version(requested), - ) - basename, error = _find_llvm_basename_or_error(requested, available, host) - result.append("exact {version}: {result}".format( - version = requested, - result = error or basename, - )) - - all_distributions = _get_all_llvm_distributions( - llvm_distributions = distributions, - extra_llvm_distributions = {}, - parsed_llvm_version = None, - ) - for requested in ["latest", "latest:>=23", "latest:>22", "latest:>=23.1.0-rc1"]: - version, basename, error = _required_llvm_release_name( - version_or_requirements = requested, - all_llvm_distributions = all_distributions, - host_info = host, - ) - result.append("{requested}: {result}".format( - requested = requested, - result = error or "{} = {}".format(version, basename), - )) - - # With built-ins suppressed, version resolution must see only the user - # entries. In particular, `latest` must not leak a newer bundled release. - custom_only = _get_all_llvm_distributions( - llvm_distributions = {}, - extra_llvm_distributions = { - "https://example.com/LLVM-20.1.4-Linux-X64.tar.xz": "3" * 64, - }, - parsed_llvm_version = None, - ) - version, basename, error = _required_llvm_release_name( - version_or_requirements = "latest", - all_llvm_distributions = custom_only, - host_info = host, - ) - result.append("custom-only latest: {result}".format( - result = error or "{} = {}".format(version, basename), - )) - builtin_basename = "LLVM-22.1.8-Linux-X64.tar.xz" - result.append("custom-only bundled URL: {}".format( - _configured_distribution_url( - basename = builtin_basename, - use_builtin_llvm_distributions = False, - user_distribution_urls = {}, - ) or "suppressed", - )) - ctx.actions.write(ctx.outputs.result, "\n".join(result) + "\n") - -prerelease_test_writer = rule( - implementation = _prerelease_test_writer_impl, - attrs = { - "result": attr.output(mandatory = True), - }, +# Internal test access only; not a supported toolchain API. Keep production +# helpers private and test implementations in llvm_distributions_test.bzl. +testing = struct( + configured_distribution_url = _configured_distribution_url, + distribution_basename = _distribution_basename, + distribution_version_string = _distribution_version_string, + find_llvm_basename_list = _find_llvm_basename_list, + find_llvm_basename_or_error = _find_llvm_basename_or_error, + get_all_llvm_distributions = _get_all_llvm_distributions, + normalize_and_check_sha256 = _normalize_and_check_sha256, + required_llvm_release_name = _required_llvm_release_name, ) diff --git a/toolchain/internal/llvm_distributions_test.bzl b/toolchain/internal/llvm_distributions_test.bzl new file mode 100644 index 00000000..35ea361d --- /dev/null +++ b/toolchain/internal/llvm_distributions_test.bzl @@ -0,0 +1,455 @@ +# Copyright 2018 The Bazel Authors. +# +# Licensed under the Apache License, Version 2.0 (the "License"); +# you may not use this file except in compliance with the License. +# You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. + +"""Golden-output tests for LLVM distribution lookup and version selection.""" + +load("@llvm_distributions_data//:data.bzl", "LLVM_DISTRIBUTIONS") +load("@mboworks_bzl//bzl/versions:versions.bzl", "versions") +load(":llvm_distributions.bzl", _testing = "testing") + +_semver = versions.with_mode("semver") + +def _parse_version(version): + return tuple(_semver.parse(version)) + +def _distributions_test_writer_impl(ctx): + """Analyze the configured versions and write to a file for test consumption. + + The test generated file '.out' contains the following lines: + - a 'del:' line denotes a llvm distribution basename that was not found. + - a 'add:' line denotes a version that was predicted but does not exist. + + Lines of type `add:` should never occur as the algorithm is supposed to + verify that predicted distributions have been configured. Otherwise the + algorithm could not know the hash value. + """ + use_llvm_distributions = LLVM_DISTRIBUTIONS + + # Inject version '0.0.0' that verifies additional behavior using `extra_llvm_distributions`. + extra_llvm_distributions = { + # Keep both formats to verify that automatic selection avoids zstd + # until Bazel supports the large windows used by LLVM 23 archives. + "LLVM-0.0.0-Linux-ARM64.tar.zst": "d6b8679be46bdaa383e0c7f13a473ca8f7a4f87233f2cc0e0a7ab19e1b6265e7", + "LLVM-0.0.0-Linux-ARM64.tar.xz": "a6b8679be46bdaa383e0c7f13a473ca8f7a4f87233f2cc0e0a7ab19e1b6265e7", + "/foo/bar/LLVM-0.0.0-Linux-X64.tar.xz?xyz": "0a764a8ca521606532ca9ec4e5745c933b16b7d30f4701a47ee851d448fcdb74", + "http://server/foo/bar/LLVM-0.0.0-macOS-ARM64.tar.xz#xyz": "9da86f64a99f5ce9b679caf54e938736ca269c5e069d0c94ad08b995c5f25c16", + "http://server/foo/bar/LLVM-0.0.0-macOS-X64.tar.xz": "264f2f1e8b67f066749349ae8b4943d346cd44e099464164ef21b42a57663540", + "http://server/clang%2Bllvm-0.0.0-aarch64-pc-windows-msvc.tar.xz": "5916d93bf80e3ae504022cdd8cb8887be001f9b68a7a08bd268727e8d858afa4", + "http://server/path-to-file/clang%2Bllvm-0.0.0-x86_64-pc-windows-msvc.tar.xz#bla": "5916d93bf80e3ae504022cdd8cb8887be001f9b68a7a08bd268727e8d858afa4", + } + + arch_list = [ + "aarch64", + "armv7a", + "mips", + "mipsel", + "powerpc64", + "powerpc64le", + "sparc64", + "sparcv9", + "x86_32", + "x86_64", + ] + arch_alias_dict = { + "sparc64": ["sparc64", "sparcv9"], + "sparcv9": ["sparcv9", "sparc64"], + } + os_list = [ + "darwin", + "linux", + "windows", + ] + ANY_VERSION = "0" # Version does not matter, but must be a valid integer + dist_dict_list = { + "linux": [ + # keep sorted + struct(name = "amzn", version = ANY_VERSION), + struct(name = "arch", version = ANY_VERSION), + struct(name = "centos", version = "6"), + struct(name = "centos", version = "7"), + struct(name = "chainguard", version = ANY_VERSION), + struct(name = "debian", version = "0"), + struct(name = "debian", version = "8"), + struct(name = "debian", version = "9"), + struct(name = "fedora", version = "26"), + struct(name = "fedora", version = "27"), + struct(name = "fedora", version = "42"), + struct(name = "freebsd", version = "10"), + struct(name = "freebsd", version = "11"), + struct(name = "freebsd", version = "12"), + struct(name = "freebsd", version = "13"), + struct(name = "ibm-aix", version = "7.2"), + struct(name = "linuxmint", version = "18"), + struct(name = "linuxmint", version = "19"), + struct(name = "pc-solaris", version = "2.11"), + struct(name = "raspbian", version = ANY_VERSION), + struct(name = "rhel", version = ANY_VERSION), + struct(name = "sun-solaris", version = "2.11"), + struct(name = "suse", version = "11.3"), + struct(name = "suse", version = "12.2"), + struct(name = "suse", version = "12.3"), + struct(name = "suse", version = "12.4"), + struct(name = "suse", version = "15.5"), + struct(name = "suse", version = "16.0"), + struct(name = "suse", version = "17.0"), + struct(name = "ubuntu", version = "14.04"), + struct(name = "ubuntu", version = "16.04"), + struct(name = "ubuntu", version = "18.04.5"), + struct(name = "ubuntu", version = "18.04.6"), + struct(name = "ubuntu", version = "18.04"), + struct(name = "ubuntu", version = "20.04"), + struct(name = "ubuntu", version = "20.10"), + struct(name = "ubuntu", version = "22.04"), + struct(name = "ubuntu", version = "24.04"), + struct(name = "void", version = ANY_VERSION), + struct(name = "wolfi", version = ANY_VERSION), + ], + } + + # Define the min real version. For earlier injected versions we do no perform dist testing. + MIN_VERSION = _parse_version("6.0.0") + + # Additional output will be generated for versions up to and including `MAX_VERSION` + MAX_VERSION = _parse_version("20.1.3") + version_dict = { + _parse_version(_testing.distribution_version_string(basename)): None + for basename in use_llvm_distributions.keys() + extra_llvm_distributions.keys() + } + versions = sorted(version_dict.keys()) + + # Write versions to output to check which versions we take into account. + output = [] + select = [] + for version in versions: + output.append("version: " + _semver.normalize_string(version)) + + # We keep track of versions in `not_found` and remove the ones we found. + # So at the end all version that were not found remain, hence the name. + not_found = { + _testing.distribution_basename(distribution): None + for distribution in use_llvm_distributions.keys() + extra_llvm_distributions.keys() + } + + # While computing we add predicted versions that are not configured as True. + # At the end we add the not-found versions as False. + result = {} + + # Collect cases that produce duplicates (or multiple) basenames. + dupes = [] + + # For all versions X arch X os check if we can compute the distribution. + for version in versions: + all_llvm_distributions = _testing.get_all_llvm_distributions( + llvm_distributions = use_llvm_distributions, + extra_llvm_distributions = extra_llvm_distributions, + parsed_llvm_version = version, + ) + for basename, distribution in all_llvm_distributions.items(): + _, shaerr = _testing.normalize_and_check_sha256(distribution.sha256) + if shaerr: + output.append("err: {basename}: bad sha256: {shaerr}".format( + basename = basename, + shaerr = shaerr, + )) + for arch in arch_list: + for os in os_list: + if version < MIN_VERSION: + # Limit the injected version checks + if arch not in ["aarch64", "x86_64"]: + break + dist_list = [struct(name = os, version = "")] + else: + dist_list = dist_dict_list.get(os, [struct(name = os, version = "")]) + for dist in dist_list: + if arch == "sparc64" and dist.name != "sun-solaris": + # Sparc64 and SparcV9 are handled in the same way, just different precedence. + # One is the architecture th other the ISA. Restrict to one to limit output. + continue + host_info = struct( + arch = arch, + os = os, + dist = dist, + ) + basenames = _testing.find_llvm_basename_list(_semver.normalize_string(version), all_llvm_distributions, host_info) + if version <= MAX_VERSION: + predicted, error = _testing.find_llvm_basename_or_error( + _semver.normalize_string(version), + all_llvm_distributions, + host_info, + ) + skip_output = False + if error: + if error.startswith("ERROR: No matching config could be found for version"): + skip_output = True + else: + if predicted.endswith(".exe"): + error = "ERROR: Windows .exe is not supported: " + predicted + elif predicted not in all_llvm_distributions: + error = "ERROR: Unavailable prediction: " + predicted + elif len(basenames) == 0: + skip_output = True + elif len(basenames) == 1: + predicted = basenames[0] + else: + error = "ERROR: Multiple selections" + if not error: + arch_found = [arch for arch in arch_list if arch in predicted] + if len(arch_found) == 1 and arch_found[0] not in arch_alias_dict.get(arch, [arch]): + error = "ERROR: Bad arch selection: " + predicted + if not skip_output: + select.append("{version}-{arch}-{os}/{dist_name}/{dist_version} -> {basename}".format( + version = _semver.normalize_string(version), + arch = arch, + os = os, + dist_name = dist.name, + dist_version = dist.version, + basename = error or predicted, + )) + if len(basenames) != 1: + if basenames: + dupes.append("dup: {version}-{arch}-{os}-{dist_name}-{dist_version} -> {count}".format( + version = _semver.normalize_string(version), + arch = arch, + os = os, + dist_name = dist.name, + dist_version = dist.version, + count = len(basenames), + )) + dupes.extend([" : " + basename for basename in basenames]) + continue + basename = basenames[0] + if basename in all_llvm_distributions: + if basename in not_found: + not_found.pop(basename) + else: + result[basename] = True + + # Build result + for dist in not_found: + result[dist] = False + output += [("add: " if found else "del: ") + dist for dist, found in result.items()] + output += dupes + ctx.actions.write(ctx.outputs.output, "\n".join(output) + "\n") + ctx.actions.write(ctx.outputs.select, "\n".join(select) + "\n") + +distributions_test_writer = rule( + implementation = _distributions_test_writer_impl, + attrs = { + "output": attr.output(mandatory = True), + "select": attr.output(mandatory = True), + }, +) + +def _requirements_test_writer_impl(ctx): + """Analyze the configured versions and write to a file for test consumption. + The test generated file '.out' contains the following lines: + [,,]: + """ + all_llvm_distributions = { + # In order to prevent new distributions to interfere we cut at 20.1.3. + k: v + for k, v in LLVM_DISTRIBUTIONS.items() + if _parse_version(_testing.distribution_version_string(k)) <= (20, 1, 3) + } + requirement_list = [ + "latest:<=20.1.0", + "latest:<=20.1.0,>17.0.4,!=19.1.7", + "latest:<20.1.0,>17.0.4,!=19.1.7", + "latest:<20.1.0,>17.0.4", + "latest:>=15.0.6,<16", + "first:>=15.0.6,<16", + "latest", + "first", + ] + arch_list = [ + "aarch64", + "armv7a", + "x86_64", + ] + os_list = [ + "darwin", + "linux", + "windows", + ] + ANY_VERSION = "0" # Version does not matter, but must be a valid integer + dist_dict_list = { + "linux": [ + # keep sorted + struct(name = "ubuntu", version = ANY_VERSION), + struct(name = "raspbian", version = ANY_VERSION), + struct(name = "rhel", version = ANY_VERSION), + ], + } + result = [] + for arch in arch_list: + for os in os_list: + dist_list = dist_dict_list.get(os, [struct(name = os, version = "")]) + for dist in dist_list: + for requirement in requirement_list: + host_info = struct( + arch = arch, + os = os, + dist = dist, + ) + llvm_version, basename, error = _testing.required_llvm_release_name( + version_or_requirements = requirement, + all_llvm_distributions = all_llvm_distributions, + host_info = host_info, + ) + if llvm_version and basename: + result.append("[{arch},{os}{dist_name}{dist_version},'{requirement}']: {llvm_version} = {basename}".format( + arch = arch, + os = os, + dist_name = "," + dist.name if os == "linux" else "", + dist_version = "," + dist.version if os == "linux" else "", + requirement = requirement, + llvm_version = llvm_version, + basename = basename, + )) + else: + result.append("[{arch},{os},\"{requirement}\"]: {error}".format( + arch = arch, + os = os, + requirement = requirement, + llvm_version = llvm_version, + basename = basename, + error = error or "ERROR: N/A", + )) + ctx.actions.write(ctx.outputs.result, "\n".join(result) + "\n") + +requirements_test_writer = rule( + implementation = _requirements_test_writer_impl, + attrs = { + "result": attr.output(mandatory = True), + }, +) + +def _prerelease_test_writer_impl(ctx): + # Exercise the checked-in release table so this also proves that published + # prereleases are bundled, not merely supported through user-supplied data. + distributions = LLVM_DISTRIBUTIONS + host = struct( + arch = "x86_64", + os = "linux", + dist = struct(name = "ubuntu", version = "24.04"), + ) + result = [] + + for requested in ["23.1.0-rc3", "22.1.8"]: + available = _testing.get_all_llvm_distributions( + llvm_distributions = distributions, + extra_llvm_distributions = {}, + parsed_llvm_version = _parse_version(requested), + ) + _, basename, error = _testing.required_llvm_release_name( + version_or_requirements = requested, + all_llvm_distributions = available, + host_info = host, + ) + result.append("exact {version}: {result}".format( + version = requested, + result = error or basename, + )) + + all_distributions = _testing.get_all_llvm_distributions( + llvm_distributions = distributions, + extra_llvm_distributions = {}, + parsed_llvm_version = None, + ) + for allow_prereleases in [False, True]: + for requested in ["latest", "latest:>=23", "latest:>22", "latest:>=23.1.0-rc1"]: + version, basename, error = _testing.required_llvm_release_name( + version_or_requirements = requested, + all_llvm_distributions = all_distributions, + host_info = host, + allow_prereleases = allow_prereleases, + ) + result.append("allow_prereleases={allow_prereleases} {requested}: {result}".format( + requested = requested, + allow_prereleases = allow_prereleases, + result = error or "{} = {}".format(version, basename), + )) + + # Fixed synthetic catalogue: latest is host-specific, constrained by the + # release series, and must not admit an otherwise newer prerelease. + selector_distributions = _testing.get_all_llvm_distributions( + llvm_distributions = {}, + extra_llvm_distributions = { + "LLVM-23.1.2-Linux-X64.tar.xz": "1" * 64, + "LLVM-23.1.2-macOS-ARM64.tar.xz": "2" * 64, + "LLVM-23.1.3-Linux-X64.tar.xz": "3" * 64, + "LLVM-23.2.0-Linux-X64.tar.xz": "4" * 64, + "LLVM-23.3.0-rc1-Linux-X64.tar.xz": "5" * 64, + "LLVM-24.1.0-Linux-X64.tar.xz": "6" * 64, + }, + parsed_llvm_version = None, + ) + for selector_host in [host, struct(arch = "aarch64", os = "darwin", dist = struct(name = "darwin", version = ""))]: + for allow_prereleases in [False, True]: + for requested in [ + "latest:major==23", + "latest:major()==23", + "latest:major==23,minor==1", + "first:major==23", + "latest:~=23.1.0", + "latest:~~=23.1.*", + "latest:!~=23.2.0,!~~=24.*", + "latest:==23.1.*", + ]: + version, basename, error = _testing.required_llvm_release_name( + version_or_requirements = requested, + all_llvm_distributions = selector_distributions, + host_info = selector_host, + allow_prereleases = allow_prereleases, + ) + result.append("synthetic {} allow_prereleases={} {}: {}".format( + selector_host.os, + allow_prereleases, + requested, + error or "{} = {}".format(version, basename), + )) + + # With built-ins suppressed, version resolution must see only the user + # entries. In particular, `latest` must not leak a newer bundled release. + custom_only = _testing.get_all_llvm_distributions( + llvm_distributions = {}, + extra_llvm_distributions = { + "https://example.com/LLVM-20.1.4-Linux-X64.tar.xz": "3" * 64, + }, + parsed_llvm_version = None, + ) + version, basename, error = _testing.required_llvm_release_name( + version_or_requirements = "latest", + all_llvm_distributions = custom_only, + host_info = host, + ) + result.append("custom-only latest: {result}".format( + result = error or "{} = {}".format(version, basename), + )) + builtin_basename = "LLVM-22.1.8-Linux-X64.tar.xz" + result.append("custom-only bundled URL: {}".format( + _testing.configured_distribution_url( + basename = builtin_basename, + use_builtin_llvm_distributions = False, + user_distribution_urls = {}, + ) or "suppressed", + )) + ctx.actions.write(ctx.outputs.result, "\n".join(result) + "\n") + +prerelease_test_writer = rule( + implementation = _prerelease_test_writer_impl, + attrs = { + "result": attr.output(mandatory = True), + }, +) diff --git a/toolchain/internal/llvm_prerelease_test.golden.txt b/toolchain/internal/llvm_prerelease_test.golden.txt index 102ea571..7dba32a3 100644 --- a/toolchain/internal/llvm_prerelease_test.golden.txt +++ b/toolchain/internal/llvm_prerelease_test.golden.txt @@ -1,8 +1,44 @@ exact 23.1.0-rc3: LLVM-23.1.0-rc3-Linux-X64.tar.xz exact 22.1.8: LLVM-22.1.8-Linux-X64.tar.xz -latest: 23.1.4 = LLVM-23.1.4-Linux-X64.tar.xz -latest:>=23: 23.1.4 = LLVM-23.1.4-Linux-X64.tar.xz -latest:>22: 23.1.4 = LLVM-23.1.4-Linux-X64.tar.xz -latest:>=23.1.0-rc1: 23.1.4 = LLVM-23.1.4-Linux-X64.tar.xz +allow_prereleases=False latest: 23.1.4 = LLVM-23.1.4-Linux-X64.tar.xz +allow_prereleases=False latest:>=23: 23.1.4 = LLVM-23.1.4-Linux-X64.tar.xz +allow_prereleases=False latest:>22: 23.1.4 = LLVM-23.1.4-Linux-X64.tar.xz +allow_prereleases=False latest:>=23.1.0-rc1: 23.1.4 = LLVM-23.1.4-Linux-X64.tar.xz +allow_prereleases=True latest: 23.1.4 = LLVM-23.1.4-Linux-X64.tar.xz +allow_prereleases=True latest:>=23: 23.1.4 = LLVM-23.1.4-Linux-X64.tar.xz +allow_prereleases=True latest:>22: 23.1.4 = LLVM-23.1.4-Linux-X64.tar.xz +allow_prereleases=True latest:>=23.1.0-rc1: 23.1.4 = LLVM-23.1.4-Linux-X64.tar.xz +synthetic linux allow_prereleases=False latest:major==23: 23.2.0 = LLVM-23.2.0-Linux-X64.tar.xz +synthetic linux allow_prereleases=False latest:major()==23: 23.2.0 = LLVM-23.2.0-Linux-X64.tar.xz +synthetic linux allow_prereleases=False latest:major==23,minor==1: 23.1.3 = LLVM-23.1.3-Linux-X64.tar.xz +synthetic linux allow_prereleases=False first:major==23: 23.1.2 = LLVM-23.1.2-Linux-X64.tar.xz +synthetic linux allow_prereleases=False latest:~=23.1.0: 23.1.3 = LLVM-23.1.3-Linux-X64.tar.xz +synthetic linux allow_prereleases=False latest:~~=23.1.*: 23.1.3 = LLVM-23.1.3-Linux-X64.tar.xz +synthetic linux allow_prereleases=False latest:!~=23.2.0,!~~=24.*: 23.1.3 = LLVM-23.1.3-Linux-X64.tar.xz +synthetic linux allow_prereleases=False latest:==23.1.*: 23.1.3 = LLVM-23.1.3-Linux-X64.tar.xz +synthetic linux allow_prereleases=True latest:major==23: 23.3.0-rc1 = LLVM-23.3.0-rc1-Linux-X64.tar.xz +synthetic linux allow_prereleases=True latest:major()==23: 23.3.0-rc1 = LLVM-23.3.0-rc1-Linux-X64.tar.xz +synthetic linux allow_prereleases=True latest:major==23,minor==1: 23.1.3 = LLVM-23.1.3-Linux-X64.tar.xz +synthetic linux allow_prereleases=True first:major==23: 23.1.2 = LLVM-23.1.2-Linux-X64.tar.xz +synthetic linux allow_prereleases=True latest:~=23.1.0: 23.1.3 = LLVM-23.1.3-Linux-X64.tar.xz +synthetic linux allow_prereleases=True latest:~~=23.1.*: 23.1.3 = LLVM-23.1.3-Linux-X64.tar.xz +synthetic linux allow_prereleases=True latest:!~=23.2.0,!~~=24.*: 23.3.0-rc1 = LLVM-23.3.0-rc1-Linux-X64.tar.xz +synthetic linux allow_prereleases=True latest:==23.1.*: 23.1.3 = LLVM-23.1.3-Linux-X64.tar.xz +synthetic darwin allow_prereleases=False latest:major==23: 23.1.2 = LLVM-23.1.2-macOS-ARM64.tar.xz +synthetic darwin allow_prereleases=False latest:major()==23: 23.1.2 = LLVM-23.1.2-macOS-ARM64.tar.xz +synthetic darwin allow_prereleases=False latest:major==23,minor==1: 23.1.2 = LLVM-23.1.2-macOS-ARM64.tar.xz +synthetic darwin allow_prereleases=False first:major==23: 23.1.2 = LLVM-23.1.2-macOS-ARM64.tar.xz +synthetic darwin allow_prereleases=False latest:~=23.1.0: 23.1.2 = LLVM-23.1.2-macOS-ARM64.tar.xz +synthetic darwin allow_prereleases=False latest:~~=23.1.*: 23.1.2 = LLVM-23.1.2-macOS-ARM64.tar.xz +synthetic darwin allow_prereleases=False latest:!~=23.2.0,!~~=24.*: 23.1.2 = LLVM-23.1.2-macOS-ARM64.tar.xz +synthetic darwin allow_prereleases=False latest:==23.1.*: 23.1.2 = LLVM-23.1.2-macOS-ARM64.tar.xz +synthetic darwin allow_prereleases=True latest:major==23: 23.1.2 = LLVM-23.1.2-macOS-ARM64.tar.xz +synthetic darwin allow_prereleases=True latest:major()==23: 23.1.2 = LLVM-23.1.2-macOS-ARM64.tar.xz +synthetic darwin allow_prereleases=True latest:major==23,minor==1: 23.1.2 = LLVM-23.1.2-macOS-ARM64.tar.xz +synthetic darwin allow_prereleases=True first:major==23: 23.1.2 = LLVM-23.1.2-macOS-ARM64.tar.xz +synthetic darwin allow_prereleases=True latest:~=23.1.0: 23.1.2 = LLVM-23.1.2-macOS-ARM64.tar.xz +synthetic darwin allow_prereleases=True latest:~~=23.1.*: 23.1.2 = LLVM-23.1.2-macOS-ARM64.tar.xz +synthetic darwin allow_prereleases=True latest:!~=23.2.0,!~~=24.*: 23.1.2 = LLVM-23.1.2-macOS-ARM64.tar.xz +synthetic darwin allow_prereleases=True latest:==23.1.*: 23.1.2 = LLVM-23.1.2-macOS-ARM64.tar.xz custom-only latest: 20.1.4 = LLVM-20.1.4-Linux-X64.tar.xz custom-only bundled URL: suppressed diff --git a/toolchain/internal/repo.bzl b/toolchain/internal/repo.bzl index 5cbabd1e..b78775af 100644 --- a/toolchain/internal/repo.bzl +++ b/toolchain/internal/repo.bzl @@ -27,6 +27,11 @@ _target_pairs = ", ".join(_supported_os_arch_keys()) # Atributes common to both `llvm` and `toolchain` repository rules. common_attrs = { + "llvm_version_allow_prereleases": attr.bool( + default = False, + doc = ("Allow prereleases in first/latest LLVM selection only. " + + "Requirements still apply. Exact version pins are unaffected."), + ), "llvm_versions": attr.string_dict( mandatory = False, doc = ("LLVM version strings, keyed by host OS release name and architecture, " + @@ -74,7 +79,8 @@ llvm_repo_attrs.update({ "for the `llvm_versions` attribute. This value can be set to `first` or " + "`latest` in order to find the respective first or latest LLVM version " + "supported on the OS/arch. Further, requirements can be provided, e.g. " + - "`latest:>=17.0.4,!=19.0.7`."), + "`latest:major==23`, `latest:major==23,minor==1`, or `latest:~=23.1.0,!=23.1.2`. " + + "Selectors use the pinned catalogue and configured extras, not live release discovery."), ), "urls": attr.string_list_dict( mandatory = False, @@ -465,6 +471,11 @@ llvm_config_attrs.update({ "version from an injected mold module. An empty key applies to all " + "targets.").format(_target_pairs), ), + "linker_version_allow_prereleases": attr.bool( + default = False, + doc = ("Allow prereleases in first/latest catalogued linker selection, including mold. " + + "Requirements still apply. Exact pins and LLVM selection are unaffected."), + ), "linker_version": attr.string( doc = ("Default version or version requirement for catalogued linkers, " + "for example `2.42.1` or `latest:>=2.40.0,<3`. If unset, mold " + diff --git a/toolchain/internal/version_selectors.bzl b/toolchain/internal/version_selectors.bzl new file mode 100644 index 00000000..635ddb6d --- /dev/null +++ b/toolchain/internal/version_selectors.bzl @@ -0,0 +1,108 @@ +# Copyright 2026 The Bazel Authors. +# +# Licensed under the Apache License, Version 2.0 (the "License"); +# you may not use this file except in compliance with the License. +# You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. + +"""Shared SemVer requirement clauses.""" + +load("@mboworks_bzl//bzl/versions:versions.bzl", "versions") + +_semver = versions.with_mode("semver") + +def _matches(version, requirements): + """Whether version satisfies every clause, using SemVer release precedence.""" + return _semver.check_all_requirements( + version, + requirements, + constants = { + "major": _semver.major(version), + "minor": _semver.minor(version), + "patch": _semver.patch(version), + }, + ) + +def _is_selector(selection): + return selection in ["first", "latest"] or selection.startswith("first:") or selection.startswith("latest:") + +def _select(selection, candidates, *, key = None, allow_prereleases = False): + """Pick a compatible candidate with the same bzl policy for LLVM and linkers.""" + if not _is_selector(selection): + fail("ERROR: Invalid version requirements: '{}'".format(selection)) + expression = selection.partition(":")[2] + requirements = _semver.parse_requirements(expression) if ":" in selection else None + pick = _semver.latest if selection.startswith("latest") else _semver.earliest + return pick( + candidates, + key = key, + include_requirements = lambda v: not requirements or _matches(v, requirements), + exclude_requirements = None if allow_prereleases else _semver.is_prerelease, + ) + +def _resolve(version_or_requirements, available_versions, *, allow_prereleases = False): + """Resolve an exact version or first/latest requirement against versions.""" + if not version_or_requirements: + fail("ERROR: Empty version selection.") + if not _is_selector(version_or_requirements): + if version_or_requirements not in available_versions: + fail("ERROR: Version '{}' is not available; supported versions: {}".format( + version_or_requirements, + ", ".join(_semver.sorted(available_versions)), + )) + return version_or_requirements + + selected = _select(version_or_requirements, available_versions, allow_prereleases = allow_prereleases) + if selected != None: + return selected + fail("ERROR: No version matches '{}' in: {}".format( + version_or_requirements, + ", ".join(_semver.sorted(available_versions)), + )) + +def _resolve_environment(rctx, selection): + """Expand an environment-provided exact version or selector for either tool.""" + if selection.startswith("getenv(") and selection.endswith(")"): + env_var = selection[len("getenv("):-1] + if env_var.find(",") >= 0: + env_name, env_default = env_var.split(",", 1) + else: + env_name = env_var + env_default = None + + env_name = env_name.strip(" ") + if not env_name: + fail("ERROR: Bad getenv parameter '{env_var}'.".format( + env_var = env_var, + )) + + # We prefer 'repository_ctx.getenv' if it is available (~7.1+) and default + # to accessing the environment directly. The latter breaks "hermeticity". + if hasattr(rctx, "getenv"): + selection = rctx.getenv(env_name, env_default) + elif env_name in rctx.os.environ: + selection = rctx.os.environ[env_name] + else: + selection = env_default + + if not selection: + fail("ERROR: Empty getenv lookup for '{env_var}'.".format( + env_var = env_var, + )) + + return selection + +version_selectors = struct( + matches = _matches, + resolve = _resolve, + resolve_environment = _resolve_environment, + select = _select, + parse_requirements = _semver.parse_requirements, +) diff --git a/toolchain/internal/version_selectors_test.bzl b/toolchain/internal/version_selectors_test.bzl new file mode 100644 index 00000000..e0e03dc5 --- /dev/null +++ b/toolchain/internal/version_selectors_test.bzl @@ -0,0 +1,231 @@ +# Copyright 2026 The Bazel Authors. +# +# Licensed under the Apache License, Version 2.0 (the "License"); +# you may not use this file except in compliance with the License. +# You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. + +"""Version-selector regression tests independent of the mutable LLVM catalogue.""" + +load("@bazel_skylib//lib:unittest.bzl", "analysistest", "asserts", "unittest") +load(":linker_distributions.bzl", "resolve_linker_distribution") +load(":version_selectors.bzl", "version_selectors") + +def _requirements_test_impl(ctx): + env = unittest.begin(ctx) + for candidate, expression, expected in [ + ("23.1.2", "major()==23", True), + ("23.1.0-rc1", 'prerelease()=="rc1"', True), + ("23.1.0-rc1", 'prerelease ()=="rc,1"', False), + ("23.1.0-rc1", 'prerelease()=="rc,,1"', False), + ("23.1.0-rc1", 'prerelease()!="rc, ,1"', True), + ("23.1.0-rc1", 'has_prerelease_prefix("rc,,")==False', True), + ("23.1.0-rc1", 'prerelease()=="rc.1"', False), + ("23.1.0-rc1", "is_prerelease()==True", True), + ("23.1.0-rc1", 'has_prerelease_prefix("rc")==True', True), + ("23.1.2", "component_count()==3,component(-1)==2", True), + ("23.1.2", "major==23", True), + ("24.1.0", "major==23", False), + ("23.2.1", "major==23,minor>=2,patch==1", True), + ("23.1.2", "major==23,minor>=2", False), + ("23", "minor==0,patch==0", True), + ("23.1.2", "major==23,major()==23,prefix(minor)==23", True), + ("23.1.2", "prefix(2)==23.1", True), + ("23.1.2", "components(start=1,end=3)==1.2", True), + ("23.1.2", "major(prefix(2))==23", True), + ("23.1.2", 'prefix("24.3.2",2)==24.3', True), + ("24.1.0", "major()==23", False), + ("23.2.0", "major()==23,minor()==1", False), + ("23.1.9", "major()==23,minor()==1", True), + ("23.1.2.7", "major()==23,minor()==1,patch()==2", True), + ("23.1.3", "major()==23,minor()==1,patch()==2", False), + ("23", "minor()==0", True), + ("23.1", "patch()==0", True), + ("23.2.0", "major()==23,minor()>=1,minor()<3", True), + ("23.1.2", "major()>=23,patch()!=1", True), + ("23.2.1", "major()==23,minor()>=2", True), + ("23.1.9", "major()==23,minor()>=2", False), + ("23.1.2", "~=23.1", True), + ("23.2.0", "~=23.1.0", False), + ("23.2.0", "!~=23.1.0", True), + ("23.1.2", "~~=23.1.*", True), + ("23.2.0", "!~~=23.1.*", True), + ("23.1.2", "==23.1.*", True), + ("23.1.2", "!=23.1.2", False), + ("23.1.2", "==23.1", False), + ("23.1.0", "==23.1", True), + ("23.1.0-rc.1", "~~=23.1.*", False), + ("23.1.0-rc.1", "==23.1.*", True), + ("23.1.0-rc10", "<23.1.0-rc2", True), + ("23.1.2+build", " major() == 23 , minor() == 1 ", True), + ]: + asserts.equals(env, expected, version_selectors.matches(candidate, version_selectors.parse_requirements(expression)), "{} {}".format(candidate, expression)) + return unittest.end(env) + +def _resolution_test_impl(ctx): + env = unittest.begin(ctx) + available = ["24.1.0", "23.1.0-rc2", "23.1.3", "22.1.8", "23.1.2", "23.1.0-rc10", "25.1.0-rc1"] + for selection, allow, expected in [ + ("latest:major()==23", False, "23.1.3"), + ("latest:major==23", False, "23.1.3"), + ('latest:major==23,prerelease()!=",,"', False, "23.1.3"), + ("first:major==23", False, "23.1.2"), + ("latest:major==23,minor==1,patch!=3", False, "23.1.2"), + ("latest:prefix(2)==23.1", False, "23.1.3"), + ("latest:major()==23,components(start=1,end=2)==1", False, "23.1.3"), + ("first:major()==23", False, "23.1.2"), + ("latest:major()==23,minor()==1,patch()!=3", False, "23.1.2"), + ("latest:~=23.1.0,!=23.1.3", False, "23.1.2"), + ("latest:!~=23.1.0", False, "24.1.0"), + ("latest:~~=23.1.*,!=23.1.3", False, "23.1.2"), + ("latest:!~~=23.1.*", False, "24.1.0"), + ("latest:==23.1.*", False, "23.1.3"), + ("latest", False, "24.1.0"), + ("first", False, "22.1.8"), + ("first:>=23.1.0-rc10,<23.1.0", True, "23.1.0-rc10"), + ("latest:>=23.1.0-rc10,<23.1.0", True, "23.1.0-rc2"), + ('latest:>=23.1.0-rc10,<23.1.0,has_prerelease_prefix("rc")==True', True, "23.1.0-rc2"), + ('latest:>=23.1.0-rc10,<23.1.0,prerelease()=="rc10"', True, "23.1.0-rc10"), + ("23.1.0-rc2", True, "23.1.0-rc2"), + ]: + asserts.equals(env, expected, version_selectors.resolve(selection, available, allow_prereleases = allow), selection) + catalogue = {"test": { + "23.1.2": {"linux-x86_64": {"binary": "linux-old"}, "darwin-aarch64": {"binary": "darwin"}}, + "23.1.3": {"linux-x86_64": {"binary": "linux-new"}}, + "24.1.0": {"linux-x86_64": {"binary": "next-major"}}, + }} + for platform, expected in [("linux-x86_64", "23.1.3"), ("darwin-aarch64", "23.1.2")]: + selected, _distribution = resolve_linker_distribution(catalogue, "test", "latest:major==23", platform) + asserts.equals(env, expected, selected) + return unittest.end(env) + +def _prerelease_test_impl(ctx): + env = unittest.begin(ctx) + available = ["23.1.0-rc1", "23.1.0-rc2", "23.1.0", "24.0.0-rc1"] + for selection, allow, expected in [ + ("latest", False, "23.1.0"), + ("first", False, "23.1.0"), + ("latest", True, "24.0.0-rc1"), + ("first", True, "23.1.0-rc1"), + ("latest:>=23", False, "23.1.0"), + ("latest:>=23", True, "24.0.0-rc1"), + ("latest:>=23.1.0-rc1", False, "23.1.0"), + ("latest:>=23.1.0-rc1", True, "24.0.0-rc1"), + ("latest:is_prerelease()==False", True, "23.1.0"), + ("latest:is_prerelease()==True", True, "24.0.0-rc1"), + ("first:major()==23", True, "23.1.0-rc1"), + ("latest:>=23.1.0-rc1,<23.1.0", True, "23.1.0-rc2"), + ("23.1.0-rc1", False, "23.1.0-rc1"), + ]: + asserts.equals(env, expected, version_selectors.resolve(selection, available, allow_prereleases = allow), repr((selection, allow))) + catalogue = {"test": {v: {"linux-x86_64": {"binary": v}} for v in available}} + selected, distribution = resolve_linker_distribution(catalogue, "test", selection, "linux-x86_64", allow_prereleases = allow) + asserts.equals(env, expected, selected) + asserts.equals(env, expected, distribution["binary"]) + return unittest.end(env) + +def _linker_parity_test_impl(ctx): + env = unittest.begin(ctx) + available = ["3.0.0-rc1", "3.0.0", "3.1.0", "4.0.0-rc1"] + catalogue = {"mold": {v: {"linux-x86_64": {"binary": v}} for v in available}} + + # A newer release on another platform must not affect selection. + catalogue["mold"]["5.0.0"] = {"darwin-aarch64": {"binary": "other"}} + for allow in [False, True]: + for selection in [ + "first", + "latest", + "first:major==3", + "latest:major()==3", + "latest:>=3,<4", + "latest:~=3.0.0", + "latest:~~=3.*", + "latest:==3.*,!=3.1.0", + "latest:!~=4.0.0,!~~=5.*", + "3.0.0-rc1", + ]: + expected = version_selectors.select(selection, [(v, v) for v in available], key = lambda item: item[0], allow_prereleases = allow)[0] if selection != "3.0.0-rc1" else selection + actual, artifact = resolve_linker_distribution(catalogue, "mold", selection, "linux-x86_64", allow_prereleases = allow) + asserts.equals(env, expected, actual, repr((selection, allow))) + asserts.equals(env, actual, artifact["binary"]) + return unittest.end(env) + +def _environment_test_impl(ctx): + env = unittest.begin(ctx) + values = {"VERSION": "latest:major==3,minor==0", "PIN": "3.0.0-rc1"} + for rctx in [ + struct(getenv = lambda name, default: values.get(name, default)), + struct(os = struct(environ = values)), + ]: + for selection, expected in [ + ("latest", "latest"), + ("3.0.0-rc1", "3.0.0-rc1"), + ("getenv(VERSION)", values["VERSION"]), + ("getenv(PIN,latest)", values["PIN"]), + ("getenv(MISSING,latest:major==3,minor==0)", values["VERSION"]), + ]: + asserts.equals(env, expected, version_selectors.resolve_environment(rctx, selection)) + return unittest.end(env) + +def _invalid_impl(ctx): + if ctx.attr.selection.startswith("getenv("): + version_selectors.resolve_environment(struct(getenv = lambda name, default: default), ctx.attr.selection) + elif ctx.attr.selection: + version_selectors.resolve(ctx.attr.selection, ["23.1.0-rc1", "23.1.2", "24.1.0"]) + else: + version_selectors.parse_requirements(ctx.attr.expression) + return [] + +_invalid = rule( + implementation = _invalid_impl, + attrs = {"expression": attr.string(), "selection": attr.string()}, +) + +def _invalid_test_impl(ctx): + env = analysistest.begin(ctx) + asserts.expect_failure(env, ctx.attr.message) + return analysistest.end(env) + +_invalid_test = analysistest.make(_invalid_test_impl, expect_failure = True, attrs = {"message": attr.string()}) +_requirements_test = unittest.make(_requirements_test_impl) +_resolution_test = unittest.make(_resolution_test_impl) +_prerelease_test = unittest.make(_prerelease_test_impl) +_linker_parity_test = unittest.make(_linker_parity_test_impl) +_environment_test = unittest.make(_environment_test_impl) + +def version_selectors_test_suite(name): + """Instantiate selector, platform filtering, and invalid-input tests.""" + unittest.suite(name, _requirements_test, _resolution_test, _prerelease_test, _linker_parity_test, _environment_test) + for suffix, expression, selection, message in [ + ("empty", "", "", "non-negative numeric release components"), + ("unknown_constant", "", "latest:mjor==23", "Unknown version constant"), + ("empty_clause", ">=23,,<24", "", "non-negative numeric release components"), + ("leading_empty_clause", ",>=23", "", "non-negative numeric release components"), + ("trailing_empty_clause", ">=23,", "", "non-negative numeric release components"), + ("spaced_empty_clause", ">=23, ,<24", "", "non-negative numeric release components"), + ("projection_operator", "major()=23", "", "non-negative numeric release components"), + ("major_width", "major()==23.1", "", "non-negative integer operand"), + ("minor_width", "minor()==23.1.2", "", "non-negative integer operand"), + ("projection_suffix", "patch()>=23.1.0-rc1", "", "non-negative integer operand"), + ("bad_wildcard", "==23.*.1", "", "single trailing"), + ("bool_operand", "", "latest:is_prerelease()==1", "bool operand"), + ("text_operand", "", "latest:prerelease()==rc1", "arguments must be"), + ("local_mode", "", 'latest:local()==""', "requires PEP 440 mode"), + ("prerelease_disabled", "", "latest:>=23.1.0-rc1,<23.1.0", "No version matches"), + ("prerelease_predicate_disabled", "", "latest:is_prerelease()==True", "No version matches"), + ("no_match", "", "latest:major()==25", "No version matches"), + ("unavailable", "", "25.1.0", "not available"), + ("empty_env_name", "", "getenv( ,latest)", "Bad getenv parameter"), + ("missing_env", "", "getenv(MISSING)", "Empty getenv lookup"), + ("empty_selection", "", "latest:", "non-negative numeric release components"), + ]: + target = name + "_" + suffix + _invalid(name = target, expression = expression, selection = selection, tags = ["manual"]) + _invalid_test(name = target + "_test", target_under_test = ":" + target, message = message) diff --git a/toolchain/rules.bzl b/toolchain/rules.bzl index c1e10119..ba5c92d5 100644 --- a/toolchain/rules.bzl +++ b/toolchain/rules.bzl @@ -116,6 +116,7 @@ def _llvm_toolchain(name, **kwargs): if catalogued_linkers: linker_distributions_repository( name = name + "_linkers", + linker_version_allow_prereleases = kwargs.get("linker_version_allow_prereleases", False), exec_arch = kwargs.get("exec_arch", ""), exec_os = kwargs.get("exec_os", ""), extra_catalogues = kwargs.get("extra_linker_distribution_files", []),