Skip to content
Open
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
21 changes: 21 additions & 0 deletions .config/nextest.toml
Original file line number Diff line number Diff line change
Expand Up @@ -56,3 +56,24 @@ slow-timeout = { period = "10s", terminate-after = 4 }
[[profile.ci.overrides]]
filter = "binary(gate_credit_nextest)"
slow-timeout = { period = "10s", terminate-after = 6 }

# These tests write fresh shell shims and run the `cargo-tops` binary against
# them, for the kit suite only; they match nothing in an adopter. A first run of
# each new script costs a fraction of a second on macOS, so alone the slowest
# ran in 1.9 s to 10 s. Under the full parallel suite at a load of 7 to 10 on
# the build machine, 3 of them timed out at 20 s in 4 runs. They get 60 s; the
# profile timeout is not raised.
[[profile.ci.overrides]]
filter = "binary(lock_verify)"
slow-timeout = { period = "10s", terminate-after = 6 }

# These tests run the whole gate against stub tools several times each, for
# the kit suite only; they match nothing in an adopter. Since the gate asks each
# pinned tool for its version, a gate runs about nine more commands. The slowest
# test of them ran in 2.3 s on the base and 3.4 s to 3.9 s with the pins, alone
# on the build machine. In the parallel run under coverage it timed out at 20 s
# in 3 of 4 gates at a load of 6 to 13. They get 60 s; the profile timeout is not
# raised.
[[profile.ci.overrides]]
filter = "binary(gate_survivors) | binary(gate_exemptions) | binary(gate_base)"
slow-timeout = { period = "10s", terminate-after = 6 }
9 changes: 9 additions & 0 deletions .github/workflows/rust-tops.yml
Original file line number Diff line number Diff line change
Expand Up @@ -385,6 +385,15 @@ jobs:
with:
tool: cargo-mutants@27.1.0
- run: cargo install --path crates/cargo-tops --locked
# The tools just installed are the pinned ones: checked on a fresh
# `init` of a throwaway crate (the lock an adopter gets), at the tiers
# the gate below runs.
- name: Lock verify on a fresh init
run: |
probe="$(mktemp -d)"
cargo new --lib --vcs none "$probe/probe"
cargo-tops init --path "$probe/probe" --class library-core --name probe
cargo-tops lock --verify --max-tier 1 --path "$probe/probe"
- run: cargo tops gate --max-tier 1 --base "origin/${{ github.base_ref || github.event.repository.default_branch }}"
env:
# The gate's own tests include a run of the real cargo-mutants: it
Expand Down
42 changes: 42 additions & 0 deletions PROTOCOL.md
Original file line number Diff line number Diff line change
Expand Up @@ -849,6 +849,45 @@ Every adopting repository keeps a `RUST-TOPS.md` at its root. It is the one page

The template is [`config/drop-in/RUST-TOPS.md`](config/drop-in/RUST-TOPS.md). `cargo tops init` writes it beside the instance, with items 2–4 and 6 filled from that instance. When `init` runs in one crate's directory of a larger repo, move its row into the root record and delete the copy. Keep it current when an instance, an exemption, or the baseline path changes. A record that disagrees with a `rust-tops.yaml` is wrong; the instance wins.

### 14.5 The tool lock (since 1.4.0)

`rust-tops.lock` pins the tools the gate runs, under `[tools]`. A pin the gate does not check is only a comment, so the gate asks each tool for its version and compares the answer with the pin.

**The lock.** A pin is a string of dot-separated digits, such as `0.9.0` or `0.9`. A `[tools]` value that is not a string, a `tools` that is not a table, and a pin of a tool that is not dot-separated digits are errors that name the key: exit 2 in `cargo tops lock`, in `cargo tops check` and in the gate. `cargo tops lock` on a lock with no `[tools]` table is exit 2 too. A lock file that is not there pins nothing.

**What is asked.** One row per tool. The first line of the output is read, and the version is the first word after the tool's name. Each ask has 10 seconds.

| Lock key | Asked | Line starts with |
|---|---|---|
| `rustc` | `rustc --version` | `rustc ` |
| `clippy`, `rustfmt` | judged by the toolchain: `rustc --version`; `cargo clippy --version` and `cargo fmt --version` must also exit 0 | `rustc ` |
| `cargo-nextest` | `cargo nextest --version` | `cargo-nextest ` |
| `cargo-llvm-cov` | `cargo llvm-cov --version` | `cargo-llvm-cov ` |
| `cargo-mutants` | `cargo mutants --version` | `cargo-mutants ` |
| `cargo-crap` | `cargo crap --version` | `cargo-crap ` |
| `cargo-deny` | `cargo deny --version` | `cargo-deny ` |
| `cargo-fuzz` | `cargo fuzz --version` | `cargo-fuzz ` |
| `kani-verifier` | `cargo kani --version` | `Kani Rust Verifier ` |
| `nushell` | `nu --version` | (the version alone) |

Keys that name a library or a runner (`proptest`, `criterion`, `callgrind-runner`) are not tools on `PATH`: they print `skipped`. Any other key is not a tool this protocol knows (a misspelt one, say): `lock --verify` prints `unknown` and does not check it, and the exit code is not changed.

**Matching.** The digits and dots at the front of the first word after the tool's name are the version, and a `+build` part after them is cut. A pin matches when its dot-separated parts are the first parts of the version found: `0.12` matches `0.12.0` and `0.12.3`, and does not match `0.120.0`; `0.12.3` does not match `0.12`. A pre-release (`0.9.0-rc.1`, `1.99.1-nightly`) matches no pin, and the whole word is named as found. A line that does not have the shape above is a mismatch and is named as found; empty output reads `no version line`. A command that cannot be run, does not exit 0, or runs past the limit is a missing tool, and the line names the command and how it failed (`exited 101`, `timed out`).

**In the gate.**
- Before the commands of a layer that runs a command, the gate asks each pinned tool that layer needs, once per gate. The layers and tools are: `static-lint` (`rustc`, `clippy`, `rustfmt`, `cargo-deny`), `unit`, `integration`, `property` and `acceptance` (`cargo-nextest`), `doc` (`rustc`), `coverage` (`cargo-llvm-cov`, `cargo-nextest`), `crap` (`cargo-crap`), `security-advisory` (`cargo-deny`), `mutation` (`cargo-mutants`), `fuzz` (`cargo-fuzz`) and `formal-kani` (`kani-verifier`). `fuzz` and `formal-kani` have no gate command in this release, so the gate does not ask for `cargo-fuzz` or `kani-verifier`.
- A tool that prints another version fails its layer with the verdict `tool-mismatch`. The layer's `detail`, its problem and the line on stdout carry the same words: `mismatch <tool>: pin <pin>, found <found>`. The kind is `environment` and the exit code is 3. None of the layer's commands run.
- A tool that cannot be run is the `tool-missing` verdict, in the same words in the same three places: `missing <tool>: pin <pin>, `<command>` <how it failed>`.
- A layer that takes another layer's run takes its verdict, `tool-mismatch` included.
- A layer above `--max-tier`, or one that is off, asks nothing. A tool with no pin in the lock is not asked. With no lock file the gate asks nothing.
- A `rust-tops.lock` that cannot be read or parsed, or that holds a pin that cannot be read, when a layer in range needs a pinned tool, is a config error: exit 2.

**`cargo tops lock --verify [--max-tier N] [--path P]`.** Asks every tool in the lock, or, with `--max-tier`, only the tools that the enabled layers at or below `N` need (this includes `cargo-fuzz` from tier 3 and `kani-verifier` from tier 4, which the gate does not ask), and prints one line per tool: `ok`, `mismatch`, `missing`, `skipped` or `unknown`. A lock with an empty `[tools]` prints `nothing pinned in [tools]`. Exit 0 when none is a mismatch or missing, 2 when the lock cannot be read, has no `[tools]` table or holds a pin that cannot be read (or the config cannot be loaded for `--max-tier`), 3 otherwise. It never writes a file.

**`cargo tops lock --write [--path P]`.** Replaces the value of each pinned tool whose pin does not match what the tool prints now, with that release version; a partial pin that already matches is left alone, and a tool that cannot be run or prints a pre-release keeps its value. Only a plain `key = "value"` line under `[tools]` is rewritten, and comments, order, spacing and other tables are kept. Any other form of a key to be written (a quoted or dotted key, an inline table, a literal or multi-line string, a header written `[ tools ]`) is exit 2 naming the key, and the lock is not changed. The edited lock is read back before it is written. It prints `wrote` only for a value it changed.

The reference workflow runs `cargo tops lock --verify` after the tools are installed and before the gate (section 17).

---

## 15. Repository layout
Expand Down Expand Up @@ -937,6 +976,8 @@ Jobs are separated so a 90-minute Miri run cannot hide a 20-second clippy fail.
| `criterion` | dedicated runner optional | no |
| `sanitizers` | schedule | release for ffi/unsafe |

A job that runs `cargo tops gate` runs `cargo tops lock --verify` first, with the same `--max-tier`, after the pinned tools are installed (section 14.5).

---

## 18. Scoring model (optional dashboard)
Expand Down Expand Up @@ -1035,6 +1076,7 @@ cargo test --doc --workspace
cargo llvm-cov nextest --workspace --all-features \
--lcov --output-path target/coverage/lcov.info --fail-under-lines 90
cargo crap
cargo tops lock --verify # the installed tools are the pinned ones (section 14.5)
git diff "$(git merge-base "$BASE" HEAD)" HEAD --unified=0 > /tmp/pr.diff # BASE: section 8.3
cargo mutants --in-diff /tmp/pr.diff

Expand Down
2 changes: 2 additions & 0 deletions config/drop-in/rust-tops.yml
Original file line number Diff line number Diff line change
Expand Up @@ -221,6 +221,8 @@ jobs:
with:
tool: cargo-mutants@27.1.0
fallback: none
# The installed tools are the pinned ones, at the tiers the gate runs.
- run: cargo tops lock --verify --max-tier 1
- run: cargo tops gate --max-tier 1 --base "origin/${{ github.base_ref || github.event.repository.default_branch }}"

# mutants-diff: class overlay. Diff-scoped; no --baseline skip; default
Expand Down
199 changes: 125 additions & 74 deletions crates/cargo-tops/src/gate/execute.rs
Original file line number Diff line number Diff line change
Expand Up @@ -8,11 +8,12 @@ use std::time::{Instant, SystemTime};

use anyhow::{Context, Result};

use super::plan::{GatePlan, Unit, What, CRAP_VERSION};
use super::plan::{GatePlan, Unit, What};
use super::{Fault, Mode};
use crate::config::Layer;
use crate::evidence;
use crate::facts::{Outcome, Run};
use crate::lock::{self, Line};
use crate::mutation::{self, Outcomes};
use crate::steps::{self, Base, Kind, MutationPlan, Step};
use crate::{laws, TOOL_LOCK};
Expand Down Expand Up @@ -82,29 +83,36 @@ const CARGO_BUILTINS: [&str; 6] = ["test", "build", "check", "doc", "run", "benc

/// The first line `<tool> --version` prints, or `None` when the tool does
/// not answer. A `cargo` subcommand asks itself; a built-in asks `cargo`.
fn version_of(
dir: &Path,
argv: &[String],
cache: &mut HashMap<String, Option<String>>,
) -> Option<String> {
fn version_of(dir: &Path, argv: &[String], cache: &mut Answers) -> Option<String> {
let (program, rest) = argv.split_first()?;
let asks = version_ask(program, rest.first().map(String::as_str));
ask_cached(
dir,
&version_ask(program, rest.first().map(String::as_str)),
cache,
)
.ok()
.filter(|line| !line.is_empty())
}

/// What each version command answered: its first non-empty line, or why it did
/// not answer.
type Answers = HashMap<String, Result<String, String>>;

/// The first line `asks` prints (empty when it prints none), or why it did not
/// answer within the limit. Each distinct argv is run once per gate, so a
/// pinned tool the gate checked and the version the report names come from
/// the same answer.
fn ask_cached(dir: &Path, asks: &[&str], cache: &mut Answers) -> Result<String, String> {
cache
.entry(asks.join(" "))
.or_insert_with(|| {
let (head, tail) = asks.split_first()?;
let out = Command::new(head)
.args(tail)
.current_dir(dir)
.output()
.ok()?;
out.status.success().then(|| {
String::from_utf8_lossy(&out.stdout)
.lines()
lock::run_probe(dir, asks, lock::PROBE_LIMIT).map(|out| {
out.lines()
.map(str::trim)
.find(|line| !line.is_empty())
.map(str::to_string)
})?
.unwrap_or_default()
.to_string()
})
})
.clone()
}
Expand Down Expand Up @@ -144,9 +152,13 @@ pub(crate) fn run<'a>(dir: &Path, plan: &GatePlan<'a>, mode: Mode) -> Result<Exe
found: None,
},
versions: HashMap::new(),
asked: HashMap::new(),
};
for unit in &plan.units {
runner.unit(unit)?;
// A layer that needs no pinned tool has no pins to ask.
if !runner.pinned_tools_stop(unit.layer) {
runner.unit(unit)?;
}
}
if runner.live {
runner.take_shared();
Expand All @@ -163,7 +175,9 @@ struct Runner<'p, 'a> {
/// `false` for a dry run: commands are listed, never run.
live: bool,
done: Executed<'a>,
versions: HashMap<String, Option<String>>,
versions: Answers,
/// What each pinned tool answered, asked once per gate.
asked: HashMap<&'static str, Line>,
}

impl<'a> Runner<'_, 'a> {
Expand All @@ -182,6 +196,65 @@ impl<'a> Runner<'_, 'a> {
Ok(())
}

/// Asks each pinned tool `layer` needs for its version, before any of the
/// layer's commands ([`Runner::ask_pins`]). `true` when the layer stopped
/// on one. A dry run only names the pins.
fn pinned_tools_stop(&mut self, layer: &Layer) -> bool {
let pins = self.plan.pins_of(&layer.id);
if pins.is_empty() {
return false;
}
let named: Vec<String> = pins
.iter()
.map(|(row, pin)| format!("{} {pin}", row.key))
.collect();
println!(
"\n==> [{}] {TOOL_LOCK} pins, asked first: {}",
layer.id,
named.join(", ")
);
if !self.live {
return false;
}
let clock = Clock::start();
let Some(run) = self.ask_pins(layer) else {
return false;
};
self.done.record(run, &clock, None);
true
}

/// The run that stands for `layer` when a pinned tool it needs is at
/// another version ([`Outcome::ToolMismatch`]) or will not answer
/// ([`Outcome::ToolMissing`]), the note carrying every such tool in the
/// words of [`lock::Line::text`]; `None` when every tool answers with its
/// pin. Each tool is asked once per gate.
fn ask_pins(&mut self, layer: &Layer) -> Option<Run> {
let dir = self.dir;
let mut stops = Vec::new();
for (row, pin) in self.plan.pins_of(&layer.id) {
let line = self.asked.entry(row.key).or_insert_with(|| {
lock::check_row(row, pin, &mut |argv| {
ask_cached(dir, argv, &mut self.versions)
})
});
stops.extend(line.stop());
}
let first = stops.first()?;
let note: Vec<&str> = stops.iter().map(|stop| stop.text.as_str()).collect();
let outcome = if stops.iter().any(|stop| stop.mismatch) {
Outcome::ToolMismatch
} else {
Outcome::ToolMissing
};
Some(pin_stop_run(
&layer.id,
outcome,
&first.asked,
note.join("; "),
))
}

fn shared(&mut self, layer: &'a Layer, from: &'static str) {
println!(
"\n==> [{}] no command of its own: it takes the {from} run's outcome",
Expand Down Expand Up @@ -227,38 +300,27 @@ impl<'a> Runner<'_, 'a> {
}

fn chain(&mut self, layer: &Layer, chain: &[Step]) -> Result<(), Fault> {
let pin = self.plan.crap_pin.as_deref().filter(|_| layer.id == "crap");
let mut shown: Vec<String> = pin.map(|_| CRAP_VERSION.join(" ")).into_iter().collect();
shown.extend(chain.iter().map(Step::display));
println!("\n==> [{}] {}", layer.id, shown.join(" && "));
if let Some(pin) = pin {
println!(
" `{}` must print `cargo-crap {pin}`, the {TOOL_LOCK} pin",
CRAP_VERSION.join(" ")
);
}
println!(
"\n==> [{}] {}",
layer.id,
chain
.iter()
.map(Step::display)
.collect::<Vec<_>>()
.join(" && ")
);
self.done.listed += 1;
if !self.live {
return Ok(());
}
let clock = Clock::start();
let run = match pin {
Some(pin) => run_pinned_crap(self.dir, &layer.id, chain, pin),
None => run_chain(self.dir, &layer.id, chain),
}
.map_err(Fault::env)?;
let version = match pin {
// The version the gate already verified against the lock.
Some(pin) => run
.cmd
.starts_with(&format!("{} && ", CRAP_VERSION.join(" ")))
.then(|| format!("cargo-crap {pin}")),
None if layer.id == "crap" => None,
// A tool that is not there has no version to ask for.
None => first_tool(chain)
.filter(|_| run.outcome != Outcome::ToolMissing)
.and_then(|argv| version_of(self.dir, argv, &mut self.versions)),
};
let run = run_chain(self.dir, &layer.id, chain).map_err(Fault::env)?;
let version = first_tool(chain)
// A tool that is not there has no version to ask for, and crap's
// is named only when the lock pinned it.
.filter(|_| run.outcome != Outcome::ToolMissing)
.filter(|_| layer.id != "crap" || !self.plan.pins_of("crap").is_empty())
.and_then(|argv| version_of(self.dir, argv, &mut self.versions));
self.done.record(run, &clock, version);
Ok(())
}
Expand Down Expand Up @@ -389,34 +451,23 @@ pub(crate) fn run_chain(dir: &Path, layer: &str, chain: &[Step]) -> Result<Run>
Ok(finish(layer, &ran, outcome))
}

/// The crap chain, run only when `cargo crap --version` prints
/// `cargo-crap <pin>`. It asks the binary that `cargo crap` runs, so another
/// version earlier on `PATH` fails the layer before a report is written. A
/// cargo-crap that is not there is left to the chain, which reports it
/// missing.
pub(crate) fn run_pinned_crap(dir: &Path, layer: &str, chain: &[Step], pin: &str) -> Result<Run> {
let version = CRAP_VERSION.join(" ");
let argv = CRAP_VERSION.map(String::from);
if steps::check_tool(&argv).is_err() {
return run_chain(dir, layer, chain);
}
let output = Command::new(CRAP_VERSION[0])
.args(&CRAP_VERSION[1..])
.current_dir(dir)
.output()
.with_context(|| format!("spawn: {version}"))?;
let stdout = String::from_utf8_lossy(&output.stdout);
let printed = stdout.lines().next().unwrap_or_default().trim();
let want = format!("cargo-crap {pin}");
if !output.status.success() || printed != want {
println!(
"Failed [{layer}]: `{version}` printed `{printed}`, not `{want}` from {TOOL_LOCK}"
);
return Ok(finish(layer, &[version], Outcome::Failed));
/// A layer that stopped because a pinned tool prints another version than its
/// pin (`outcome` is a mismatch) or would not answer (a missing tool). The same `note` is the stdout line,
/// the layer's detail and its problem.
fn pin_stop_run(layer: &str, outcome: Outcome, asked: &str, note: String) -> Run {
let label = if outcome == Outcome::ToolMismatch {
"tool-mismatch"
} else {
"tool-missing"
};
println!("{label} [{layer}]: {note}");
Run {
layer: layer.to_string(),
cmd: String::new(),
step: asked.to_string(),
outcome,
note: Some(note),
}
let mut run = run_chain(dir, layer, chain)?;
run.cmd = format!("{version} && {}", run.cmd);
Ok(run)
}

/// A mutation layer's run, and what its `outcomes.json` held.
Expand Down
Loading
Loading