Add fallback to default url

README.md

@@ -73,15 +73,25 @@ Additionally, a few special version names are supported for our official release
 
 ## Where does Bazelisk get Bazel from?
 
-By default Bazelisk retrieves Bazel releases, release candidates and binaries built at green commits from Google Cloud Storage.
+By default Bazelisk retrieves Bazel releases, release candidates and binaries built at green commits from Google Cloud Storage. The downloaded artifacts are validated against the SHA256 value recorded in `BAZELISK_VERIFY_SHA256` if this variable is set in the configuration file.
 
 As mentioned in the previous section, the `<FORK>/<VERSION>` version format allows you to use your own Bazel fork hosted on GitHub:
 
-If you want to create a fork with your own releases, you have to follow the naming conventions that we use in `bazelbuild/bazel` for the binary file names.
+If you want to create a fork with your own releases, you should follow the naming conventions that we use in `bazelbuild/bazel` for the binary file names as this results in predictable URLs that are similar to the official ones.
 The URL format looks like `https://github.com/<FORK>/bazel/releases/download/<VERSION>/<FILENAME>`.
 
 You can also override the URL by setting the environment variable `$BAZELISK_BASE_URL`. Bazelisk will then append `/<VERSION>/<FILENAME>` to the base URL instead of using the official release server. Bazelisk will read file [`~/.netrc`](https://everything.curl.dev/usingcurl/netrc) for credentials for Basic authentication.
 
+If for any reason none of this works, you can also override the URL format altogether by setting the environment variable `$BAZELISK_FORMAT_URL`. This variable takes a format-like string with placeholders and performs the following replacements to compute the download URL:
+
+- `%e`: Extension suffix, such as the empty string or `.exe`.
+- `%h`: Value of `BAZELISK_VERIFY_SHA256`, respecting uppercase/lowercase characters.
+- `%m`: Machine architecture name, such as `arm64` or `x86_64`.
+- `%o`: Operating system name, such as `darwin` or `linux`.
+- `%v`: Bazel version as determined by Bazelisk.
+- `%%`: Literal `%` for escaping purposes.
+- All other characters after `%` are reserved for future use and result in a processing error.
+
 ## Ensuring that your developers use Bazelisk rather than Bazel
 
 Bazel installers typically provide Bazel's [shell wrapper script] as the `bazel` on the PATH.
@@ -100,25 +110,52 @@ require users update their bazel.
 [shell wrapper script]: https://github.com/bazelbuild/bazel/blob/master/scripts/packages/bazel.sh
 ## Other features
 
-The Go version of Bazelisk offers two new flags.
+The Go version of Bazelisk offers three new flags.
+
+### --strict
 
 `--strict` expands to the set of incompatible flags which may be enabled for the given version of Bazel.
 
 ```shell
 bazelisk --strict build //...
 ```
 
+### --migrate
+
 `--migrate` will run Bazel multiple times to help you identify compatibility issues.
 If the code fails with `--strict`, the flag `--migrate` will run Bazel with each one of the flag separately, and print a report at the end.
 This will show you which flags can safely enabled, and which flags require a migration.
 
+
+### --bisect
+
+`--bisect` flag allows you to bisect Bazel versions to find which version introduced a build failure. You can specify the range of versions to bisect with `--bisect=<GOOD>..<BAD>`, where GOOD is the last known working Bazel version and BAD is the first known non-working Bazel version. Bazelisk uses [GitHub's compare API](https://docs.github.com/en/rest/commits/commits#compare-two-commits) to get the list of commits to bisect. When GOOD is not an ancestor of BAD, GOOD is reset to their merge base commit.
+
+Examples:
+```shell
+# Bisect between 6.0.0 and Bazel at HEAD
+bazelisk --bisect=6.0.0..HEAD test //foo:bar_test
+
+# Bisect between 6.1.0 and the second release candidate of Bazel 6.2.0
+bazelisk --bisect=6.1.0..release-6.2.0rc2 test //foo:bar_test
+
+# Bisect between two commits on the main branch (or branches with `release-` prefix) of the Bazel GitHub repository.
+bazelisk --bisect=<good commit hash>..<bad commit hash> test //foo:bar_test
+```
+
+Note that, Bazelisk uses prebuilt Bazel binaries at commits on the main and release branches, therefore you cannot bisect your local commits.
+
+### Useful environment variables for --migrate and --bisect
+
 You can set `BAZELISK_INCOMPATIBLE_FLAGS` to set a list of incompatible flags (separated by `,`) to be tested, otherwise Bazelisk tests all flags starting with `--incompatible_`.
 
 You can set `BAZELISK_GITHUB_TOKEN` to set a GitHub access token to use for API requests to avoid rate limiting when on shared networks.
 
-You can set `BAZELISK_SHUTDOWN` to run `shutdown` between builds when migrating if you suspect this affects your results.
+You can set `BAZELISK_SHUTDOWN` to run `shutdown` between builds when migrating or bisecting if you suspect this affects your results.
+
+You can set `BAZELISK_CLEAN` to run `clean --expunge` between builds when migrating or bisecting if you suspect this affects your results.
 
-You can set `BAZELISK_CLEAN` to run `clean --expunge` between builds when migrating if you suspect this affects your results.
+## tools/bazel
 
 If `tools/bazel` exists in your workspace root and is executable, Bazelisk will run this file, instead of the Bazel version it downloaded.
 It will set the environment variable `BAZEL_REAL` to the path of the downloaded Bazel binary.
@@ -149,6 +186,7 @@ The following variables can be set:
 - `BAZELISK_SHUTDOWN`
 - `BAZELISK_SKIP_WRAPPER`
 - `BAZELISK_USER_AGENT`
+- `BAZELISK_VERIFY_SHA256`
 - `USE_BAZEL_VERSION`
 
 Configuration variables are evaluated with precedence order. The preferred values are derived in order from highest to lowest precedence as follows:

bazelisk.go

@@ -25,11 +25,12 @@ import (
 
 func main() {
 	gcs := &repositories.GCSRepo{}
-	gitHub := repositories.CreateGitHubRepo(core.GetEnvOrConfig("BAZELISK_GITHUB_TOKEN"))
+	config := core.MakeDefaultConfig()
+	gitHub := repositories.CreateGitHubRepo(config.Get("BAZELISK_GITHUB_TOKEN"))
 	// Fetch LTS releases, release candidates, rolling releases and Bazel-at-commits from GCS, forks from GitHub.
 	repos := core.CreateRepositories(gcs, gcs, gitHub, gcs, gcs, true)
 
-	exitCode, err := core.RunBazelisk(os.Args[1:], repos)
+	exitCode, err := core.RunBazeliskWithArgsFuncAndConfig(func(string) []string { return os.Args[1:] }, repos, config)
 	if err != nil {
 		log.Fatal(err)
 	}

bazelisk.py

@@ -97,6 +97,8 @@ def find_workspace_root(root=None):
         root = os.getcwd()
     if os.path.exists(os.path.join(root, "WORKSPACE")):
         return root
+    if os.path.exists(os.path.join(root, "WORKSPACE.bazel")):
+        return root
     new_root = os.path.dirname(root)
     return find_workspace_root(new_root) if new_root != root else None
 
@@ -178,8 +180,7 @@ def get_version_history(bazelisk_directory):
         ),
         # This only handles versions with numeric components, but that is fine
         # since prerelease versions have been excluded.
-        key=lambda version: tuple(int(component)
-                                  for component in version.split('.')),
+        key=lambda version: tuple(int(component) for component in version.split(".")),
         reverse=True,
     )
 
@@ -306,6 +307,8 @@ def download_bazel_into_directory(version, is_commit, directory):
 
     sha256_path = destination_path + ".sha256"
     expected_hash = ""
+    matcher = re.compile(r"(\d*\.\d*(?:\.\d*)?)(rc\d+)?")
+    matched = matcher.match(version)
     if not os.path.exists(sha256_path):
         try:
             download(bazel_url + ".sha256", sha256_path)
@@ -314,7 +317,20 @@ def download_bazel_into_directory(version, is_commit, directory):
                 sys.stderr.write(
                     "The Bazel mirror does not have a checksum file; skipping checksum verification."
                 )
-                return destination_path
+                if "https://releases.bazel.build" in bazel_url or not matched:
+                    return destination_path
+                if matched:
+                    (version, rc) = matched.groups()
+                    fallback_url = "https://releases.bazel.build/{}/{}/{}".format(
+                        version, rc if rc else "release", bazel_filename
+                    )
+                    try:
+                        download("{}.sha256".format(fallback_url), sha256_path)
+                        os.remove(destination_path)
+                        download(fallback_url, destination_path)
+                    except HTTPError:
+                        return destination_path
+                    os.chmod(destination_path, 0o755)
             raise e
     with open(sha256_path, "r") as sha_file:
         expected_hash = sha_file.read().split()[0]
@@ -324,16 +340,24 @@ def download_bazel_into_directory(version, is_commit, directory):
             sha256_hash.update(byte_block)
     actual_hash = sha256_hash.hexdigest()
     if actual_hash != expected_hash:
-        os.remove(destination_path)
         os.remove(sha256_path)
+        os.remove(destination_path)
         print(
-            "The downloaded Bazel binary is corrupted. Expected SHA-256 {}, got {}. Please try again.".format(
+            "The downloaded Bazel binary is corrupted. Expected SHA-256 {}, got {}. Fallback to default releases.bazel.build url.".format(
                 expected_hash, actual_hash
             )
         )
-        # Exiting with a special exit code not used by Bazel, so the calling process may retry based on that.
-        # https://docs.bazel.build/versions/0.21.0/guide.html#what-exit-code-will-i-get
-        sys.exit(22)
+        if matched:
+            (version, rc) = matched.groups()
+            fallback_url = "https://releases.bazel.build/{}/{}/{}".format(
+                version, rc if rc else "release", bazel_filename
+            )
+            try:
+                download("{}.sha256".format(fallback_url), sha256_path)
+                download(fallback_url, destination_path)
+            except HTTPError:
+                exit(22)
+            os.chmod(destination_path, 0o755)
     return destination_path
 
 

bazelisk_test.sh

@@ -197,7 +197,20 @@ function test_bazel_version_from_file() {
       (echo "FAIL: Expected to find 'Build label: 5.0.0' in the output of 'bazelisk version'"; exit 1)
 }
 
-function test_bazel_version_from_url() {
+function test_bazel_version_from_format_url() {
+  setup
+
+  echo "0.19.0" > .bazelversion
+
+  BAZELISK_FORMAT_URL="https://github.com/bazelbuild/bazel/releases/download/%v/bazel-%v-%o-%m%e" \
+      BAZELISK_HOME="$BAZELISK_HOME" \
+          bazelisk version 2>&1 | tee log
+
+  grep "Build label: 0.19.0" log || \
+      (echo "FAIL: Expected to find 'Build label: 0.19.0' in the output of 'bazelisk version'"; exit 1)
+}
+
+function test_bazel_version_from_base_url() {
   setup
 
   echo "0.19.0" > .bazelversion
@@ -286,6 +299,52 @@ EOF
       (echo "FAIL: Expected to find 'BAZELISK_SKIP_WRAPPER=true' in the output of 'bazelisk version'"; exit 1)
 }
 
+function test_path_is_consistent_regardless_of_base_url() {
+  setup
+
+  echo 6.2.0 > .bazelversion
+
+  cat >WORKSPACE <<EOF
+load("//:print_path.bzl", "print_path")
+
+print_path(name = "print_path")
+
+load("@print_path//:defs.bzl", "noop")
+
+noop()
+EOF
+
+cat >print_path.bzl <<EOF
+def _print_path_impl(rctx):
+    print("PATH is: {}".format(rctx.os.environ["PATH"]))
+
+    rctx.file("WORKSPACE", "")
+    rctx.file("BUILD", "")
+    rctx.file("defs.bzl", "def noop(): pass")
+
+print_path = repository_rule(
+    implementation = _print_path_impl,
+)
+EOF
+
+  BAZELISK_HOME="$BAZELISK_HOME" bazelisk sync --only=print_path 2>&1 | tee log1
+
+  BAZELISK_HOME="$BAZELISK_HOME" bazelisk clean --expunge 2>&1
+
+  # We need a separate mirror of bazel binaries, which has identical files.
+  # Ideally we wouldn't depend on sourceforge for test runtime, but hey, it exists and it works.
+  BAZELISK_HOME="$BAZELISK_HOME" BAZELISK_BASE_URL=https://downloads.sourceforge.net/project/bazel.mirror bazelisk sync --only=print_path 2>&1 | tee log2
+
+  path1="$(grep "PATH is:" log1)"
+  path2="$(grep "PATH is:" log2)"
+
+  [[ -n "${path1}" && -n "${path2}" ]] || \
+      (echo "FAIL: Expected PATH to be non-empty, got path1=${path1}, path2=${path2}"; exit 1)
+
+  [[ "${path1}" == "${path2}" ]] || \
+      (echo "FAIL: Expected PATH to be the same regardless of which mirror was used, got path1=${path1}, path2=${path2}"; exit 1)
+}
+
 function test_skip_wrapper() {
   setup
 
@@ -314,10 +373,54 @@ function test_bazel_download_path_go() {
   BAZELISK_HOME="$BAZELISK_HOME" \
       bazelisk version 2>&1 | tee log
 
-  find "$BAZELISK_HOME/downloads/bazelbuild" 2>&1 | tee log
+  find "$BAZELISK_HOME/downloads/metadata/bazelbuild" 2>&1 | tee log
 
-  grep "^$BAZELISK_HOME/downloads/bazelbuild/bazel-[0-9][0-9]*.[0-9][0-9]*.[0-9][0-9]*-[a-z0-9_-]*/bin/bazel\(.exe\)\?$" log || \
-      (echo "FAIL: Expected to download bazel binary into specific path."; exit 1)
+  grep "^$BAZELISK_HOME/downloads/metadata/bazelbuild/bazel-[0-9][0-9]*.[0-9][0-9]*.[0-9][0-9]*-[a-z0-9_-]*$" log || \
+      (echo "FAIL: Expected to download bazel metadata in specific path."; exit 1)
+}
+
+function test_bazel_verify_sha256() {
+  setup
+
+  echo "6.1.1" > .bazelversion
+
+  # First try to download and expect an invalid hash (it doesn't matter what it is).
+  if BAZELISK_HOME="$BAZELISK_HOME" BAZELISK_VERIFY_SHA256="invalid-hash" \
+      bazelisk version 2>&1 | tee log; then
+    echo "FAIL: Command should have errored out"; exit 1
+  fi
+
+  grep "need sha256=invalid-hash" log || \
+      (echo "FAIL: Expected to find hash mismatch"; exit 1)
+
+  # IMPORTANT: The mixture of lowercase and uppercase letters in the hashes below is
+  # intentional to ensure the variable contents are normalized before comparison.
+  # If updating these values, re-introduce randomness.
+  local os="$(uname -s | tr A-Z a-z)"
+  case "${os}" in
+    darwin)
+      expected_sha256="038e95BAE998340812562ab8d6ada1a187729630bc4940a4cd7920cc78acf156"
+      ;;
+    linux)
+      expected_sha256="651a20d85531325df406b38f38A1c2578c49D5e61128fba034f5b6abdb3d303f"
+      ;;
+    msys*|mingw*|cygwin*)
+      expected_sha256="1d997D344936a1d98784ae58db1152d083569556f85cd845e6e340EE855357f9"
+      ;;
+    *)
+      echo "FAIL: Unknown OS ${os} in test"
+      exit 1
+      ;;
+  esac
+
+  # Now try the same download as before but with the correct hash expectation. Note that the
+  # hash has a random uppercase / lowercase mixture to ensure this does not impact equality
+  # checks.
+  BAZELISK_HOME="$BAZELISK_HOME" BAZELISK_VERIFY_SHA256="${expected_sha256}" \
+      bazelisk version 2>&1 | tee log
+
+  grep "Build label:" log || \
+      (echo "FAIL: Expected to find 'Build label' in the output of 'bazelisk version'"; exit 1)
 }
 
 function test_bazel_download_path_py() {
@@ -338,8 +441,26 @@ function test_bazel_prepend_binary_directory_to_path_go() {
   BAZELISK_HOME="$BAZELISK_HOME" \
       bazelisk --print_env 2>&1 | tee log
 
-  PATTERN=$(echo "^PATH=$BAZELISK_HOME/downloads/bazelbuild/bazel-[0-9][0-9]*.[0-9][0-9]*.[0-9][0-9]*-[a-z0-9_-]*/bin[:;]" | sed -e 's/\//\[\/\\\\\]/g')
-  grep "$PATTERN" log || \
+  local os="$(uname -s | tr A-Z a-z)"
+  case "${os}" in
+      darwin|linux)
+        path_entry_delimiter=":"
+        path_delimiter="/"
+        extension=""
+        ;;
+      msys*|mingw*|cygwin*)
+        path_entry_delimiter=";"
+        path_delimiter="\\"
+        extension=".exe"
+        ;;
+      *)
+        echo "FAIL: Unknown OS ${os} in test"
+        exit 1
+        ;;
+    esac
+  path_entry="$(grep "^PATH=" log | cut -d= -f2- | cut -d"${path_entry_delimiter}" -f1)"
+
+  [[ -x "${path_entry}${path_delimiter}bazel${extension}" ]] || \
       (echo "FAIL: Expected PATH to contains bazel binary directory."; exit 1)
 }
 
@@ -383,8 +504,12 @@ if [[ $BAZELISK_VERSION == "GO" ]]; then
   test_bazel_last_rc
   echo
 
-  echo "# test_bazel_version_from_url"
-  test_bazel_version_from_url
+  echo "# test_bazel_version_from_format_url"
+  test_bazel_version_from_format_url
+  echo
+
+  echo "# test_bazel_version_from_base_url"
+  test_bazel_version_from_base_url
   echo
 
   echo "# test_bazel_version_prefer_environment_to_bazeliskrc"
@@ -411,10 +536,18 @@ if [[ $BAZELISK_VERSION == "GO" ]]; then
   test_bazel_download_path_go
   echo
 
+  echo '# test_bazel_verify_sha256'
+  test_bazel_verify_sha256
+  echo
+
   echo "# test_bazel_prepend_binary_directory_to_path_go"
   test_bazel_prepend_binary_directory_to_path_go
   echo
 
+  echo "# test_path_is_consistent_regardless_of_base_url"
+  test_path_is_consistent_regardless_of_base_url
+  echo
+
   case "$(uname -s)" in
     MSYS*)
       # The tests are currently not compatible with Windows.

config/BUILD

@@ -0,0 +1,12 @@
+load("@io_bazel_rules_go//go:def.bzl", "go_library")
+
+go_library(
+    name = "go_default_library",
+    srcs = ["config.go"],
+    importpath = "github.com/bazelbuild/bazelisk/config",
+    visibility = ["//visibility:public"],
+    deps = [
+        "//ws:go_default_library",
+        "@com_github_mitchellh_go_homedir//:go_default_library",
+    ],
+)