Skip to content

[proposal] Multiple cache wheel servers #1065

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
tiran wants to merge 3 commits into
base: main
Choose a base branch
from
Open

[proposal] Multiple cache wheel servers #1065

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
3 commits
Select commit Hold shift + click to select a range
91f6bf0
[proposal] Multiple cache wheel servers
tiran Apr 16, 2026
b9a7d28
Merge branch 'main' into proposal-multiple-cache-wheel-servers
mergify[bot] Apr 16, 2026
35e2e70
Merge branch 'main' into proposal-multiple-cache-wheel-servers
mergify[bot] Apr 17, 2026
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
1 change: 1 addition & 0 deletions docs/proposals/index.rst
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -4,5 +4,6 @@ Fromager Enhancement Proposals
.. toctree::
:maxdepth: 1

multiple-cache-wheel-servers
new-patcher-config
new-resolver-config
131 changes: 131 additions & 0 deletions docs/proposals/multiple-cache-wheel-servers.md
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,131 @@
# Multiple cache wheel servers

- Author: Christian Heimes
- Created: 2026-04-16
- Status: Open

## What

Support multiple cache wheel server URLs instead of a single
`--cache-wheel-server-url`. Caches are checked in order, similar to pip's
`--extra-index-url` or uv's `unsafe-best-match` index strategy.

Comment on lines +10 to +12
Copy link
Copy Markdown

@coderabbitai coderabbitai Bot Apr 16, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

⚠️ Potential issue | 🟠 Major

🧩 Analysis chain

🌐 Web query:

`Using official documentation, what are the exact selection semantics for:

  1. pip with --index-url + multiple --extra-index-url
  2. uv with UV_INDEX_STRATEGY=unsafe-best-match
    Specifically, do they use first-index-hit ordering or best-candidate selection across all indexes?`

💡 Result:

  1. pip with --index-url + multiple --extra-index-url: best-candidate selection across all indexes. Pip collects package files/links from all specified indexes (including the primary --index-url and all --extra-index-urls), evaluates them together as a unified set of candidates, and selects the best match based on version, compatibility, and preferences (e.g., wheels over sdists). There is no strict first-index-hit ordering; indexes are merged before resolution. 2) uv with UV_INDEX_STRATEGY=unsafe-best-match: best-candidate selection across all indexes. Unlike uv's default first-index (which uses first-index-hit), unsafe-best-match searches all indexes for a package, considers all versions from all indexes as candidates, and selects the "best" (typically highest compatible) version. If the same version appears in multiple indexes, it uses the one from the first index.

Citations:

Fix factual claim about pip's index selection behavior.

Lines 10–12 and 82–83 incorrectly state that the proposal uses "first-match semantics, same as --extra-index-url". In reality, pip's --extra-index-url performs best-candidate selection across all indexes, not first-match ordering. This contradicts the accurate description in lines 90–93, which correctly states that uv's unsafe-best-match chooses the best match across all indexes.

Clarify whether fromager's cache lookup uses first-match (first cache hit wins) or best-candidate selection (evaluate candidates from all caches), and correct the comparison to pip's actual behavior.

🤖 Prompt for AI Agents 
Verify each finding against the current code and only fix it if needed.

In `@docs/proposals/multiple-cache-wheel-servers.md` around lines 10 - 12, Update
the proposal text to correct the wrong comparison to pip: remove the claim that
`--extra-index-url` uses first-match semantics and state that pip performs
best-candidate selection across all indexes; then clearly state whether
Fromager's `--cache-wheel-server-url` lookup uses first-match (first cache hit
wins) or best-candidate selection, and make the comparison consistent with uv's
`unsafe-best-match` wording used later. Reference and edit the sections
mentioning `--cache-wheel-server-url`, `--extra-index-url`, `unsafe-best-match`,
and the Fromager cache lookup description so the behavior is unambiguous and
consistent across lines 10–12, 82–83, and 90–93.

## Why

In downstream, each variant and CPU architecture has its own build cache in
a GitLab PyPI registry. During bootstrap and build, fromager uses a
read/write cache in GitLab to avoid rebuilding wheels that were already built
for the current architecture. After a successful bootstrap and build, wheels
for all architectures are copied into a variant-specific cache in Pulp.

Currently fromager only supports a single `--cache-wheel-server-url`. To
pull from both the arch-specific GitLab cache **and** the variant-wide Pulp
cache, we need ordered multi-cache support.

All cache indexes are fully trusted and controlled by us, so security
concerns around index mixing (as with pip's `--extra-index-url`) do not
apply.

## Goals

- Accept multiple cache wheel server URLs on the CLI and in
`WorkContext`.
- Check caches in order: local wheel server first, then each cache URL in
Copy link
Copy Markdown
Contributor

@rd4398 rd4398 Apr 22, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

If _download_wheel_from_cache() uses first-match (stop at first cache that has any matching wheel), the order of -c flags matters. A stale wheel in cache 1 will shadow a newer one in cache 2. Is this something we should worry about?

the order specified.
- Remain backwards-compatible: a single `--cache-wheel-server-url` keeps
working.
- Integrate with `build_environment.py` so build-dependency installation
also searches multiple caches.

## Current state

`bootstrap`, `build-sequence`, and `build-parallel` each accept a single
`-c` / `--cache-wheel-server-url` option. The cache URL is not stored on
`WorkContext`; it is threaded through function arguments.

`wheels.get_wheel_server_urls()` builds an ordered list for resolution:

1. Per-package variant `wheel_server_url` (if set, used exclusively)
2. `ctx.wheel_server_url` (local wheel server)
3. `cache_wheel_server_url` (single cache, appended last)

Build-dependency installation (`build_environment.BuildEnvironment`) only
uses the local wheel server (`--index-url`). The cache is not passed to
pip/uv when installing build dependencies.

`Bootstrapper._download_wheel_from_cache()` checks the single cache URL
and downloads the first matching wheel.

## Proposed changes

### 1. CLI: accept multiple `--cache-wheel-server-url` values

Make the existing `-c` / `--cache-wheel-server-url` click option
repeatable (`multiple=True`). The internal parameter name changes from
`cache_wheel_server_url` (singular) to `cache_wheel_server_urls`
(sequence). A single `-c URL` keeps working.

### 2. Make `cache_wheel_server_urls` arguments a sequence

All internal call sites that currently pass a single
`cache_wheel_server_url: str | None` change to accept a sequence.
`wheels.get_wheel_server_urls()` appends all cache URLs in order instead
of a single one.
Copy link
Copy Markdown
Contributor

@rd4398 rd4398 Apr 22, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

If cache 1 returns a connection error or timeout, does fromager skip to cache 2, or does it fail the entire build?


Affected modules: `commands/bootstrap.py`, `commands/build.py`,
`bootstrapper.py`, `bootstrap_requirement_resolver.py`, `wheels.py`.

### 3. Bootstrapper: iterate over caches

`Bootstrapper._download_wheel_from_cache()` iterates over all cache URLs
instead of checking a single one. The first cache that has a matching
wheel wins (first-match semantics, same as `--extra-index-url`).

### 4. Build environment: pass caches to pip/uv

Store `cache_wheel_server_urls` on `WorkContext` and extend
`pip_wheel_server_args` to emit `--extra-index-url` for each cache URL.
`BuildEnvironment.install()` then picks up caches automatically.

Set `UV_INDEX_STRATEGY=unsafe-best-match` in the build environment so uv
considers all indexes equally and picks the best matching version across
all of them, rather than stopping at the first index that has any version
of a package.

`WorkContext` already carries the local wheel server URL, so the cache
URLs belong there too -- they share the same lifetime.

### 5. Resolution order

The final resolution order for wheel lookups:

1. **Per-package variant `wheel_server_url`** -- if set, used exclusively
(no change).
2. **Local wheel server** (`ctx.wheel_server_url`) -- just-built wheels.
3. **Cache 1** (first `-c` URL) -- e.g., variant-wide Pulp cache
(read-only).
4. **Cache 2** (second `-c` URL) -- e.g., arch-specific GitLab PyPI
registry (read/write).
5. ... additional caches in CLI order.

This matches the use case: check local builds first, then the Pulp cache
with already-published wheels, then the arch-specific GitLab cache.

## Example usage

```bash
fromager \
--variant cuda \
bootstrap \
-c https://pulp.example/content/rhel-ai-cuda/simple \
-c https://gitlab.example/api/v4/projects/123/packages/pypi/simple \
-r requirements.txt
```

## Migration

- The CLI flag name stays the same (`--cache-wheel-server-url` / `-c`).
- Existing invocations with zero or one `-c` flag work without changes.
- The singular parameter name `cache_wheel_server_url` is renamed to
`cache_wheel_server_urls` in internal APIs. This is a breaking change
for any downstream code calling these functions directly.
3 changes: 3 additions & 0 deletions docs/spelling_wordlist.txt
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -2,6 +2,7 @@ args
backend
backends
backoff
bootstrapper
canonicalize
canonicalized
changelog
Expand Down Expand Up @@ -34,6 +35,7 @@ lexicographically
libcurl
linter
localhost
lookups
matcher
mypy
namespace
Expand Down Expand Up @@ -73,6 +75,7 @@ txt
unshare
url
urls
uv
vendored
vendoring
versionless
Expand Down
Loading