Skip to content

Support for uv lockfiles as an alternative to pex lockfiles (cherry-pick of #23302)#23320

Merged
benjyw merged 2 commits intopantsbuild:2.32.xfrom
benjyw:cherry-pick-23302-to-2.32.x
May 6, 2026
Merged

Support for uv lockfiles as an alternative to pex lockfiles (cherry-pick of #23302)#23320
benjyw merged 2 commits intopantsbuild:2.32.xfrom
benjyw:cherry-pick-23302-to-2.32.x

Conversation

@benjyw
Copy link
Copy Markdown
Contributor

@benjyw benjyw commented May 5, 2026

uv lockfiles are first-class entities that users check in.
When a pex is needed, we use uv to create a venv, which
it does very efficiently, and then use pex's
--venv-repository feature to build a pex from
(a subset of) that venv's requirements.

Note that we still create VenvPexes to run tools, and
in some cases this might be unnecessary, since we
already have the uv venv. But utilizing those venvs
directly would be an even bigger change (and this one
is already massive), and there are caching tradeoffs
to figure out . So we leave that for the future (and
probably for the new python backend).

Note also that the dynamics of lockfile generation remain
pants's (you must generate explicitly) and not uv's
(lockfiles are automatically updated for you when
input requirements change). The latter is desirable, but
again this is for the future.

The general thrust of the changes is:

  • Add uv as a third lockfile format (after pex and the deprecated
    legacy constraints file "locking")
  • Update the lockfile metadata to include the format (and
    also the name of the resolve that created the lockfile).
  • Make lockfile metadata reading much more robust
    and useful, so we can get the format from it reliably.
    Unfortunately this logic is tricky since it has to handle
    pex lockfiles both with and without separate metadata files.
    For uv we simply require separate metadata files and do
    not add any metadata inside the lockfile itself, as we
    foolishly did for pex lockfiles at one point.
  • Update the lockfile diffing code to support diffing
    across all combinations of old/new and pex/uv.
  • Update the existing uv subsystem for more general use.
  • Add src/python/pants/backend/python/util_rules/uv.py,
    which runs uv, first generating ephemeral pyproject.toml
    and uv.toml config files. We handle environments correctly,
    which requires some realpath shenanigans.
  • Update some iffy logic in PythonToolBase that parses lockfiles
    to find useful info for help messages. It was previously relying
    on pex lockfile internals (which are not guaranteed to be stable)
    and it now also relies on uv lockfile internals (similarly not stable).
    However now at least it fails gracefully if the lockfile parsing fails
    and just prints a less useful error message.
  • Use all this in src/python/pants/backend/python/util_rules/pex.py,
    instead of the previous, uv pex building solution.

Note that this supersedes the more limited (but much appreciated)
change in #23197 that implemented only the uv venv -> pex part.
It also supersedes the competing pending changes in #22949
and #23212, which implemented partial support for uv resolution.

…ld#23302)

uv lockfiles are first-class entities that users check in.
When a pex is needed, we use uv to create a venv, which
it does very efficiently, and then use pex's
`--venv-repository` feature to build a pex from
(a subset of) that venv's requirements.

Note that we still create VenvPexes to run tools, and
in some cases this might be unnecessary, since we
already have the uv venv. But utilizing those venvs
directly would be an even bigger change (and this one
is already massive), and there are caching tradeoffs
to figure out . So we leave that for the future (and
probably for the new python  backend).

Note also that the dynamics of lockfile generation remain
pants's (you must generate explicitly) and not uv's
(lockfiles are automatically updated for you when
input requirements change). The latter is desirable, but
again this is for the future.

The general thrust of the changes is:

- Add uv as a third lockfile format (after pex and the deprecated
  legacy constraints file "locking")
- Update the lockfile metadata to include the format (and
  also the name of the resolve that created the lockfile).
- Make lockfile metadata reading much more robust
  and useful, so we can get the format from it reliably.
  Unfortunately this logic is tricky since it has to handle
  pex lockfiles both with and without separate metadata files.
  For uv we simply require separate metadata files and do
  not add any metadata inside the lockfile itself, as we
  foolishly did for pex lockfiles at one point.
- Update the lockfile diffing code to support diffing
  across all combinations of old/new and pex/uv.
- Update the existing `uv` subsystem for more general use.
- Add src/python/pants/backend/python/util_rules/uv.py,
  which runs `uv`, first generating ephemeral pyproject.toml
  and uv.toml config files. We handle environments correctly,
  which requires some `realpath` shenanigans.
- Update some iffy logic in PythonToolBase that parses lockfiles
  to find useful info for help messages. It was previously relying
  on pex lockfile internals (which are not guaranteed to be stable)
and it now also relies on uv lockfile internals (similarly not stable).
  However now at least it fails gracefully if the lockfile parsing fails
   and just prints a less useful error message.
- Use all this in src/python/pants/backend/python/util_rules/pex.py,
  instead of the previous, uv pex building solution.

Note that this supersedes the more limited (but much appreciated)
change in pantsbuild#23197 that implemented only the uv venv -> pex part.
It also supersedes the competing pending changes in pantsbuild#22949
and pantsbuild#23212, which implemented partial support for uv resolution.
@benjyw benjyw changed the title Support for uv lockfiles as an alternative to pex lockfiles (#23302) Support for uv lockfiles as an alternative to pex lockfiles (cherry-pick of #23302) May 5, 2026
@benjyw benjyw requested review from cburroughs and sureshjoshi May 5, 2026 22:13
@benjyw benjyw merged commit a4a20b6 into pantsbuild:2.32.x May 6, 2026
25 checks passed
@benjyw benjyw deleted the cherry-pick-23302-to-2.32.x branch May 6, 2026 04:38
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants