-
Notifications
You must be signed in to change notification settings - Fork 23
Support semver version ranges for dependencies #311
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Open
heatonmatthew
wants to merge
10
commits into
helsing-ai:main
Choose a base branch
from
heatonmatthew:versioning
base: main
Could not load branches
Branch not found: {{ refName }}
Loading
Could not load tags
Nothing to show
Loading
Are you sure you want to change the base?
Some commits from the old base branch may be removed from the timeline,
and old review comments may become outdated.
Open
Changes from all commits
Commits
Show all changes
10 commits
Select commit
Hold shift + click to select a range
de6d421
feat(cli): [WIP] Support semantic version ranges.
heatonmatthew ce2b42f
feat(cli): [WIP] Additional unit tests for semantic version ranges.
heatonmatthew 99ed6e7
feat(cli): Update documentation
heatonmatthew 95326c5
fix(cli): Incomplete handling of parts in "name version" Deserialize
heatonmatthew a9ae96c
fix(cli): Incorrect handling of CWD when installing a workspace.
heatonmatthew 65ce199
chore: fix lint fmt
heatonmatthew 46604d8
chore(cli): re-add debug traces for Artifactory calls
heatonmatthew d33ebc7
doc: Re-add incorrectly removed content
heatonmatthew 19596ee
Remove unused function.
heatonmatthew 09e53cf
Tidy-up imports
heatonmatthew File filter
Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
There are no files selected for viewing
Some generated files are not rendered by default. Learn more about how customized files appear on GitHub.
Oops, something went wrong.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -1 +1,71 @@ | ||
| # Dependency Resolution | ||
|
|
||
| When you run `buffrs install`, the resolver builds a complete dependency graph | ||
| for your project — including all transitive dependencies — and determines the | ||
| concrete version to install for each package. | ||
|
|
||
| ## Resolution algorithm | ||
|
|
||
| For each dependency (direct or transitive), the resolver follows this priority | ||
| order: | ||
|
|
||
| 1. **Lockfile hit** — if `Proto.lock` already records a version of the package | ||
| that satisfies the requirement, that version is used immediately without | ||
| contacting the registry. This makes repeated installs fast and reproducible. | ||
|
|
||
| 2. **Registry resolution** — if no matching locked version exists, the resolver | ||
| queries the registry for all available versions of the package, then selects | ||
| the **highest** version that satisfies the requirement. | ||
|
|
||
| 3. **Download and cache** — the resolved version is downloaded, stored in the | ||
| local cache, and its digest is recorded in the lockfile for future installs. | ||
|
|
||
| Transitive dependencies are discovered by reading the `Proto.toml` bundled | ||
| inside each downloaded package archive, then resolved recursively using the | ||
| same steps above. | ||
|
|
||
| ## Version conflict detection | ||
|
|
||
| If the same package is required by more than one path in the dependency tree, | ||
| the resolver checks that the already-resolved version satisfies all | ||
| requirements. If it does not, the install fails with a version conflict error: | ||
|
|
||
| ``` | ||
| version conflict for leaf-lib: requirement ^2.0.0 is not satisfied by | ||
| resolved version 1.5.0 (chosen to satisfy ^1.0.0) | ||
| ``` | ||
|
|
||
| To fix a conflict, update the requiring packages so their version requirements | ||
| overlap, or introduce a package that bridges the incompatible requirements. | ||
|
|
||
| Note that this conflict detection operates **within a single package's | ||
| dependency graph**. In a workspace, different members may independently resolve | ||
| different versions of the same package — the workspace lockfile records them | ||
| separately using a `(name, version)` composite key. | ||
|
|
||
| ## Workspace resolution | ||
|
|
||
| In a workspace, each member package's dependency graph is resolved | ||
| independently. The workspace lockfile (`Proto.lock` at the workspace root) | ||
| accumulates all resolved packages across all members. Because the workspace | ||
| lockfile allows multiple versions of the same package, two members that require | ||
| incompatible versions of a shared library can co-exist. | ||
|
|
||
| If a subsequent install finds a workspace lockfile, it reuses those locked | ||
| versions (subject to satisfying each member's requirements) to avoid redundant | ||
| registry queries. | ||
|
|
||
| ## Topological ordering | ||
|
|
||
| After the full graph is built, packages are sorted topologically so that each | ||
| dependency is installed before its dependants. This guarantees that vendored | ||
| proto sources are available in the correct order during compilation. | ||
|
|
||
| ## Determinism and the lockfile | ||
|
|
||
| The resolver always picks the **highest** satisfying version when multiple | ||
| candidates exist. This is deterministic given the same set of available | ||
| registry versions. Once a version is recorded in `Proto.lock`, it is used | ||
| as-is on all subsequent installs, regardless of newer versions that may have | ||
| been published since. Run `buffrs install` after deleting or modifying | ||
| `Proto.lock` to re-resolve against the current registry state. |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -1 +1,88 @@ | ||
| # SemVer Compatibility | ||
|
|
||
| buffrs uses [Semantic Versioning](https://semver.org/) for all packages. | ||
| Version requirements in `Proto.toml` follow the same syntax as | ||
| [Cargo](https://doc.rust-lang.org/cargo/reference/specifying-dependencies.html). | ||
|
|
||
| ## Version requirement syntax | ||
|
|
||
| A version requirement is placed in the `version` field of a dependency: | ||
|
|
||
| ```toml | ||
| [dependencies.my-lib] | ||
| version = "^1.2.0" | ||
| registry = "https://my-registry.example.com" | ||
| repository = "my-repo" | ||
| ``` | ||
|
|
||
| The following operators are supported: | ||
|
|
||
| ### Caret (`^`) — default for ranges | ||
|
|
||
| Allows minor and patch updates within the same major version. | ||
| This is the recommended operator for most dependencies. | ||
|
|
||
| | Requirement | Resolves versions | | ||
| |-------------|-------------------| | ||
| | `^1.2.3` | `>=1.2.3, <2.0.0` | | ||
| | `^1.2` | `>=1.2.0, <2.0.0` | | ||
| | `^1` | `>=1.0.0, <2.0.0` | | ||
| | `^0.2.3` | `>=0.2.3, <0.3.0` | | ||
| | `^0.0.3` | `>=0.0.3, <0.0.4` | | ||
|
|
||
| Note that `0.x` versions are treated as unstable: `^0.2` only allows `0.2.x`, | ||
| not `0.3.x`, since breaking changes are expected in pre-1.0 packages. | ||
|
|
||
| ### Tilde (`~`) — patch-level updates only | ||
|
|
||
| Allows patch updates within the same minor version. | ||
|
|
||
| | Requirement | Resolves versions | | ||
| |-------------|-------------------| | ||
| | `~1.2.3` | `>=1.2.3, <1.3.0` | | ||
| | `~1.2` | `>=1.2.0, <1.3.0` | | ||
| | `~1` | `>=1.0.0, <2.0.0` | | ||
|
|
||
| ### Exact (`=`) — pin to a specific version | ||
|
|
||
| Resolves to exactly the stated version, with no flexibility. | ||
|
|
||
| ```toml | ||
| version = "=1.2.3" | ||
| ``` | ||
|
|
||
| Use exact pins when you need bit-for-bit reproducibility in the manifest | ||
| itself, or when you are distributing a library whose consumers should be | ||
| in full control of the version. | ||
|
|
||
| ### Comparison operators | ||
|
|
||
| For more control, the standard comparison operators are available: | ||
|
|
||
| | Requirement | Meaning | | ||
| |-----------------|----------------------------------| | ||
| | `>=1.2.0` | Any version at or above 1.2.0 | | ||
| | `>1.2.0` | Any version strictly above 1.2.0 | | ||
| | `<2.0.0` | Any version strictly below 2.0.0 | | ||
| | `<=2.0.0` | Any version at or below 2.0.0 | | ||
| | `>=1.0.0, <2.0.0` | Intersection (multiple constraints) | | ||
|
|
||
| ## How the resolver picks a version | ||
|
|
||
| When a requirement matches more than one available version, buffrs always | ||
| selects the **highest** satisfying version. The resolved concrete version is | ||
| written to `Proto.lock` to ensure reproducible installs — re-running | ||
| `buffrs install` will use the locked version rather than querying the registry | ||
| again. | ||
|
|
||
| See [Dependency Resolution](./resolver.md) for a full description of the | ||
| resolution algorithm. | ||
|
|
||
| ## Choosing between pinning and ranges | ||
|
|
||
| | Situation | Recommended style | | ||
| |-----------|-------------------| | ||
| | Public library — let consumers decide | `^1.0.0` | | ||
| | Internal service — stable dependency set | `^1.0.0` or `~1.2.0` | | ||
| | Security patch must be applied exactly | `=1.2.5` | | ||
| | Compatibility ceiling known | `>=1.0.0, <3.0.0` | |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -1,52 +1,86 @@ | ||
| # Specifying Dependencies | ||
|
|
||
| Dependencies are declared in the `[dependencies]` section of the `Proto.toml` | ||
| manifest. Each entry maps a dependency package name to a dependency | ||
| specification object. | ||
| Dependencies are declared in the `[dependencies]` section of `Proto.toml`. | ||
| Each entry names the package and provides a version requirement, registry URL, | ||
| and repository name. | ||
|
|
||
| ## Remote Dependencies | ||
|
|
||
| Remote dependencies are downloaded from an Artifactory registry during | ||
| [`buffrs install`](../commands/buffrs-install.md). | ||
| ## Inline table syntax | ||
|
|
||
| ```toml | ||
| [dependencies] | ||
| my-package = { registry = "https://your.registry/artifactory", repository = "my-repo", version = "1.2.3" } | ||
| [dependencies.my-lib] | ||
| version = "^1.0.0" | ||
| registry = "https://my-registry.example.com" | ||
| repository = "my-repo" | ||
| ``` | ||
|
|
||
| The `version` field must be an exact semantic version (e.g. `"1.2.3"`). | ||
| Version ranges or operators (`^`, `~`, `<`, `>`) are not currently supported. | ||
| The three required fields for a remote dependency are: | ||
|
|
||
| | Field | Description | | ||
| |-------|-------------| | ||
| | `version` | A SemVer requirement — see [SemVer Compatibility](./semver.md) | | ||
| | `registry` | Base URL of the Artifactory registry | | ||
| | `repository` | Repository name within that registry | | ||
|
|
||
| ## Adding dependencies via the CLI | ||
|
|
||
| Use [`buffrs add`](../commands/buffrs-add.md) to add a remote dependency from | ||
| the command line: | ||
| The `buffrs add` command writes the manifest entry for you: | ||
|
|
||
| ```bash | ||
| # Caret range (recommended): resolves to the highest 1.x.y | ||
| buffrs add --registry https://my-registry.example.com my-repo/my-lib@^1.0.0 | ||
|
|
||
| # Exact pin: resolves to exactly 1.2.3 | ||
| buffrs add --registry https://my-registry.example.com my-repo/my-lib@=1.2.3 | ||
|
|
||
| # Latest: omitting the version resolves to the latest available | ||
| buffrs add --registry https://my-registry.example.com my-repo/my-lib | ||
| ``` | ||
| buffrs add --registry https://your.registry/artifactory my-repo/my-package@1.2.3 | ||
|
|
||
| After adding a dependency, run `buffrs install` to resolve and download it. | ||
|
|
||
| ## Version requirements | ||
|
|
||
| buffrs supports the full range of SemVer requirement operators: | ||
|
|
||
| ```toml | ||
| version = "^1.0.0" # >=1.0.0, <2.0.0 (recommended for most deps) | ||
| version = "~1.2.0" # >=1.2.0, <1.3.0 (patch updates only) | ||
| version = ">=1.5.0" # any version at or above 1.5.0 | ||
| version = "=1.2.3" # exactly 1.2.3 | ||
| version = ">=1.0, <2.0" # explicit intersection | ||
| ``` | ||
|
|
||
| ## Local Dependencies | ||
| The resolver queries the registry and selects the **highest** available version | ||
| satisfying the requirement. See [SemVer Compatibility](./semver.md) for the | ||
| full operator reference. | ||
|
|
||
| ## Local dependencies | ||
|
|
||
| Local dependencies are resolved from the local filesystem relative to the | ||
| manifest. They are useful for multi-package repositories where packages depend | ||
| on each other without going through a remote registry. | ||
| You can depend on a package in a local directory (useful in monorepos or during | ||
| development): | ||
|
|
||
| ```toml | ||
| [dependencies] | ||
| my-lib = { path = "../my-lib" } | ||
| [dependencies.my-lib] | ||
| path = "../my-lib" | ||
| ``` | ||
|
|
||
| The `path` field is a relative path from the manifest file to the dependency's | ||
| root directory (the directory containing the dependency's `Proto.toml`). | ||
|
|
||
| Local dependencies do not have a version requirement — the package at that | ||
| path is used as-is. They cannot be mixed with a remote entry for the same | ||
| package name. | ||
|
|
||
| See [Local Dependencies](../guide/local-dependencies.md) for more information. | ||
|
|
||
| ## Lockfile | ||
| ## The lockfile | ||
|
|
||
| After adding or modifying dependencies in the manifest, run | ||
| [`buffrs install`](../commands/buffrs-install.md) to resolve and lock them. | ||
| The lockfile (`Proto.lock`) records the exact resolved versions and checksums | ||
| and should be committed to version control. | ||
| Once resolved, the concrete version is recorded in `Proto.lock`. Subsequent | ||
| installs use the locked version without re-querying the registry, ensuring | ||
| reproducible builds across machines and CI environments. | ||
|
|
||
| See [Manifest vs Lockfile](../guide/manifest-vs-lockfile.md) for more | ||
| information. | ||
| Commit `Proto.lock` to version control for applications and services. For | ||
| libraries intended to be consumed by others, committing the lockfile is | ||
| optional — consumers will resolve their own versions. | ||
|
|
||
| See [Manifest vs Lockfile](../guide/manifest-vs-lockfile.md) for more detail. | ||
Oops, something went wrong.
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Can we keep the removed lines:
Uh oh!
There was an error while loading. Please reload this page.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Good catch. Fixed.
Please check you're happy.