Skip to content
Closed
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
9 changes: 9 additions & 0 deletions .github/workflows/rust-tops.yml
Original file line number Diff line number Diff line change
Expand Up @@ -385,6 +385,15 @@ jobs:
with:
tool: cargo-mutants@27.1.0
- run: cargo install --path crates/cargo-tops --locked
# The tools just installed are the pinned ones: checked on a fresh
# `init` of a throwaway crate (the lock an adopter gets), at the tiers
# the gate below runs.
- name: Lock verify on a fresh init
run: |
probe="$(mktemp -d)"
cargo new --lib --vcs none "$probe/probe"
cargo-tops init --path "$probe/probe" --class library-core --name probe
cargo-tops lock --verify --max-tier 1 --path "$probe/probe"
- run: cargo tops gate --max-tier 1 --base "origin/${{ github.base_ref || github.event.repository.default_branch }}"
env:
# The gate's own tests include a run of the real cargo-mutants: it
Expand Down
40 changes: 40 additions & 0 deletions PROTOCOL.md
Original file line number Diff line number Diff line change
Expand Up @@ -849,6 +849,43 @@ Every adopting repository keeps a `RUST-TOPS.md` at its root. It is the one page

The template is [`config/drop-in/RUST-TOPS.md`](config/drop-in/RUST-TOPS.md). `cargo tops init` writes it beside the instance, with items 2–4 and 6 filled from that instance. When `init` runs in one crate's directory of a larger repo, move its row into the root record and delete the copy. Keep it current when an instance, an exemption, or the baseline path changes. A record that disagrees with a `rust-tops.yaml` is wrong; the instance wins.

### 14.5 The tool lock (since 1.4.0)

`rust-tops.lock` pins the tools the gate runs, under `[tools]`. A pin the gate does not check is only a comment, so the gate asks each tool for its version and compares the answer with the pin.

**What is asked.** One row per tool. The first line of the output is read, and the version is the first word after the tool's name, cut at the first character that is not a digit or a dot.

| Lock key | Asked | Line starts with |
|---|---|---|
| `rustc` | `rustc --version` | `rustc ` |
| `clippy`, `rustfmt` | judged by the toolchain: `rustc --version`; `cargo clippy --version` and `cargo fmt --version` must also exit 0 | `rustc ` |
| `cargo-nextest` | `cargo nextest --version` | `cargo-nextest ` |
| `cargo-llvm-cov` | `cargo llvm-cov --version` | `cargo-llvm-cov ` |
| `cargo-mutants` | `cargo mutants --version` | `cargo-mutants ` |
| `cargo-crap` | `cargo crap --version` | `cargo-crap ` |
| `cargo-deny` | `cargo deny --version` | `cargo-deny ` |
| `cargo-fuzz` | `cargo fuzz --version` | `cargo-fuzz ` |
| `kani-verifier` | `cargo kani --version` | `Kani Rust Verifier ` |
| `nushell` | `nu --version` | (the version alone) |

Keys that name a library or a runner (`proptest`, `criterion`, `callgrind-runner`) are not tools on `PATH`. They are skipped.

**Matching.** A pin matches when its dot-separated parts are the first parts of the version found. `0.12` matches `0.12.0` and `0.12.3`, and does not match `0.120.0`. `0.12.3` does not match `0.12`. A line that does not have the shape above is a mismatch, and the line is named as found.

**In the gate.**
- Before the commands of a layer that runs, the gate asks each pinned tool that layer needs, once per gate. The layers and tools are: `static-lint` (`rustc`, `clippy`, `rustfmt`, `cargo-deny`), `unit`, `integration`, `property` and `acceptance` (`cargo-nextest`), `doc` (`rustc`), `coverage` (`cargo-llvm-cov`, `cargo-nextest`), `crap` (`cargo-crap`), `security-advisory` (`cargo-deny`), `mutation` (`cargo-mutants`), `fuzz` (`cargo-fuzz`) and `formal-kani` (`kani-verifier`).
- A tool that prints another version fails its layer with the verdict `tool-mismatch`. The layer's `detail`, its problem and the line on stdout carry the same words: `mismatch <tool>: pin <pin>, found <found>`. The kind is `environment` and the exit code is 3. None of the layer's commands run.
- A tool that cannot be run is the `tool-missing` outcome, as before.
- A layer that takes another layer's run takes its verdict, `tool-mismatch` included.
- A layer above `--max-tier`, or one that is off, asks nothing. A tool with no pin in the lock is not asked. With no lock file the gate asks nothing.
- A `rust-tops.lock` that cannot be read or parsed, when a layer in range needs a pinned tool, is a config error: exit 2.

**`cargo tops lock --verify [--max-tier N] [--path P]`.** Asks every tool in the lock, or, with `--max-tier`, only the tools that the enabled layers at or below `N` need, and prints one line per tool: `ok`, `mismatch`, `missing` or `skipped`. Exit 0 when none is a mismatch or missing, 2 when the lock cannot be read (or the config cannot be loaded for `--max-tier`), 3 otherwise. It never writes a file.

**`cargo tops lock --write [--path P]`.** Replaces each `[tools]` value with the version the tool prints now. Comments, order, spacing and other tables are kept. A tool that cannot be run keeps its value.

The reference workflow runs `cargo tops lock --verify` after the tools are installed and before the gate (section 17).

---

## 15. Repository layout
Expand Down Expand Up @@ -937,6 +974,8 @@ Jobs are separated so a 90-minute Miri run cannot hide a 20-second clippy fail.
| `criterion` | dedicated runner optional | no |
| `sanitizers` | schedule | release for ffi/unsafe |

A job that runs `cargo tops gate` runs `cargo tops lock --verify` first, with the same `--max-tier`, after the pinned tools are installed (section 14.5).

---

## 18. Scoring model (optional dashboard)
Expand Down Expand Up @@ -1035,6 +1074,7 @@ cargo test --doc --workspace
cargo llvm-cov nextest --workspace --all-features \
--lcov --output-path target/coverage/lcov.info --fail-under-lines 90
cargo crap
cargo tops lock --verify # the installed tools are the pinned ones (section 14.5)
git diff "$(git merge-base "$BASE" HEAD)" HEAD --unified=0 > /tmp/pr.diff # BASE: section 8.3
cargo mutants --in-diff /tmp/pr.diff

Expand Down
2 changes: 2 additions & 0 deletions config/drop-in/rust-tops.yml
Original file line number Diff line number Diff line change
Expand Up @@ -221,6 +221,8 @@ jobs:
with:
tool: cargo-mutants@27.1.0
fallback: none
# The installed tools are the pinned ones, at the tiers the gate runs.
- run: cargo tops lock --verify --max-tier 1
- run: cargo tops gate --max-tier 1 --base "origin/${{ github.base_ref || github.event.repository.default_branch }}"

# mutants-diff: class overlay. Diff-scoped; no --baseline skip; default
Expand Down
177 changes: 121 additions & 56 deletions crates/cargo-tops/src/gate/execute.rs
Original file line number Diff line number Diff line change
Expand Up @@ -8,11 +8,12 @@ use std::time::{Instant, SystemTime};

use anyhow::{Context, Result};

use super::plan::{GatePlan, Unit, What, CRAP_VERSION};
use super::plan::{GatePlan, Unit, What};
use super::{Fault, Mode};
use crate::config::Layer;
use crate::evidence;
use crate::facts::{Outcome, Run};
use crate::lock::{self, Line, Verdict};
use crate::mutation::{self, Outcomes};
use crate::steps::{self, Base, Kind, MutationPlan, Step};
use crate::{laws, TOOL_LOCK};
Expand Down Expand Up @@ -88,7 +89,21 @@ fn version_of(
cache: &mut HashMap<String, Option<String>>,
) -> Option<String> {
let (program, rest) = argv.split_first()?;
let asks = version_ask(program, rest.first().map(String::as_str));
ask_cached(
dir,
&version_ask(program, rest.first().map(String::as_str)),
cache,
)
}

/// The first line `asks` prints, or `None` when it cannot run or fails. Each
/// distinct argv is run once per gate, so a pinned tool the gate checked and
/// the version the report names come from the same answer.
fn ask_cached(
dir: &Path,
asks: &[&str],
cache: &mut HashMap<String, Option<String>>,
) -> Option<String> {
cache
.entry(asks.join(" "))
.or_insert_with(|| {
Expand Down Expand Up @@ -144,9 +159,13 @@ pub(crate) fn run<'a>(dir: &Path, plan: &GatePlan<'a>, mode: Mode) -> Result<Exe
found: None,
},
versions: HashMap::new(),
asked: HashMap::new(),
};
for unit in &plan.units {
runner.unit(unit)?;
// A layer that needs no pinned tool has no pins to ask.
if !runner.pinned_tools_stop(unit.layer) {
runner.unit(unit)?;
}
}
if runner.live {
runner.take_shared();
Expand All @@ -164,6 +183,8 @@ struct Runner<'p, 'a> {
live: bool,
done: Executed<'a>,
versions: HashMap<String, Option<String>>,
/// What each pinned tool answered, asked once per gate.
asked: HashMap<&'static str, Line>,
}

impl<'a> Runner<'_, 'a> {
Expand All @@ -182,6 +203,63 @@ impl<'a> Runner<'_, 'a> {
Ok(())
}

/// Asks each pinned tool `layer` needs for its version, before any of the
/// layer's commands ([`Runner::ask_pins`]). `true` when the layer stopped
/// on one. A dry run only names the pins.
fn pinned_tools_stop(&mut self, layer: &Layer) -> bool {
let pins = self.plan.pins_of(&layer.id);
if pins.is_empty() {
return false;
}
let named: Vec<String> = pins
.iter()
.map(|(row, pin)| format!("{} {pin}", row.key))
.collect();
println!(
"\n==> [{}] {TOOL_LOCK} pins, asked first: {}",
layer.id,
named.join(", ")
);
if !self.live {
return false;
}
let clock = Clock::start();
let Some(run) = self.ask_pins(layer) else {
return false;
};
self.done.record(run, &clock, None);
true
}

/// The run that stands for `layer` when a pinned tool it needs is at
/// another version ([`Outcome::ToolMismatch`], the note naming tool, pin
/// and found) or will not run ([`Outcome::ToolMissing`]); `None` when
/// every tool answers with its pin. Each tool is asked once per gate.
fn ask_pins(&mut self, layer: &Layer) -> Option<Run> {
let dir = self.dir;
let mut mismatched = Vec::new();
let mut missing: Option<String> = None;
for (row, pin) in self.plan.pins_of(&layer.id) {
let line = self.asked.entry(row.key).or_insert_with(|| {
lock::check_row(row, pin, &mut |argv| {
ask_cached(dir, argv, &mut self.versions)
})
});
match &line.verdict {
Verdict::Mismatch { .. } => mismatched.push((row.argv.join(" "), line.text())),
Verdict::Missing { asked } => {
missing.get_or_insert_with(|| asked.clone());
}
Verdict::Ok { .. } | Verdict::Skipped { .. } => {}
}
}
if let Some((argv, _)) = mismatched.first() {
let notes: Vec<&str> = mismatched.iter().map(|(_, text)| text.as_str()).collect();
return Some(mismatch_run(&layer.id, argv, notes.join("; ")));
}
missing.map(|asked| missing_run(&layer.id, asked))
}

fn shared(&mut self, layer: &'a Layer, from: &'static str) {
println!(
"\n==> [{}] no command of its own: it takes the {from} run's outcome",
Expand Down Expand Up @@ -227,38 +305,27 @@ impl<'a> Runner<'_, 'a> {
}

fn chain(&mut self, layer: &Layer, chain: &[Step]) -> Result<(), Fault> {
let pin = self.plan.crap_pin.as_deref().filter(|_| layer.id == "crap");
let mut shown: Vec<String> = pin.map(|_| CRAP_VERSION.join(" ")).into_iter().collect();
shown.extend(chain.iter().map(Step::display));
println!("\n==> [{}] {}", layer.id, shown.join(" && "));
if let Some(pin) = pin {
println!(
" `{}` must print `cargo-crap {pin}`, the {TOOL_LOCK} pin",
CRAP_VERSION.join(" ")
);
}
println!(
"\n==> [{}] {}",
layer.id,
chain
.iter()
.map(Step::display)
.collect::<Vec<_>>()
.join(" && ")
);
self.done.listed += 1;
if !self.live {
return Ok(());
}
let clock = Clock::start();
let run = match pin {
Some(pin) => run_pinned_crap(self.dir, &layer.id, chain, pin),
None => run_chain(self.dir, &layer.id, chain),
}
.map_err(Fault::env)?;
let version = match pin {
// The version the gate already verified against the lock.
Some(pin) => run
.cmd
.starts_with(&format!("{} && ", CRAP_VERSION.join(" ")))
.then(|| format!("cargo-crap {pin}")),
None if layer.id == "crap" => None,
// A tool that is not there has no version to ask for.
None => first_tool(chain)
.filter(|_| run.outcome != Outcome::ToolMissing)
.and_then(|argv| version_of(self.dir, argv, &mut self.versions)),
};
let run = run_chain(self.dir, &layer.id, chain).map_err(Fault::env)?;
let version = first_tool(chain)
// A tool that is not there has no version to ask for, and crap's
// is named only when the lock pinned it.
.filter(|_| run.outcome != Outcome::ToolMissing)
.filter(|_| layer.id != "crap" || !self.plan.pins_of("crap").is_empty())
.and_then(|argv| version_of(self.dir, argv, &mut self.versions));
self.done.record(run, &clock, version);
Ok(())
}
Expand Down Expand Up @@ -389,34 +456,32 @@ pub(crate) fn run_chain(dir: &Path, layer: &str, chain: &[Step]) -> Result<Run>
Ok(finish(layer, &ran, outcome))
}

/// The crap chain, run only when `cargo crap --version` prints
/// `cargo-crap <pin>`. It asks the binary that `cargo crap` runs, so another
/// version earlier on `PATH` fails the layer before a report is written. A
/// cargo-crap that is not there is left to the chain, which reports it
/// missing.
pub(crate) fn run_pinned_crap(dir: &Path, layer: &str, chain: &[Step], pin: &str) -> Result<Run> {
let version = CRAP_VERSION.join(" ");
let argv = CRAP_VERSION.map(String::from);
if steps::check_tool(&argv).is_err() {
return run_chain(dir, layer, chain);
/// A layer that stopped because a pinned tool prints another version than its
/// pin. The same `note` is the stdout line, the layer's detail and its problem.
fn mismatch_run(layer: &str, argv: &str, note: String) -> Run {
println!("tool-mismatch [{layer}]: {note}");
Run {
layer: layer.to_string(),
cmd: String::new(),
step: argv.to_string(),
outcome: Outcome::ToolMismatch,
note: Some(note),
}
let output = Command::new(CRAP_VERSION[0])
.args(&CRAP_VERSION[1..])
.current_dir(dir)
.output()
.with_context(|| format!("spawn: {version}"))?;
let stdout = String::from_utf8_lossy(&output.stdout);
let printed = stdout.lines().next().unwrap_or_default().trim();
let want = format!("cargo-crap {pin}");
if !output.status.success() || printed != want {
println!(
"Failed [{layer}]: `{version}` printed `{printed}`, not `{want}` from {TOOL_LOCK}"
);
return Ok(finish(layer, &[version], Outcome::Failed));
}

/// A layer that stopped because a pinned tool will not run: the same outcome
/// and words as a command whose tool is not on `PATH`.
fn missing_run(layer: &str, asked: String) -> Run {
let words: Vec<String> = asked.split_whitespace().map(str::to_string).collect();
let binary = steps::tool_binary(&words).unwrap_or_else(|| asked.clone());
println!("{}", steps::MissingTool { binary });
Run {
layer: layer.to_string(),
cmd: String::new(),
step: asked,
outcome: Outcome::ToolMissing,
note: None,
}
let mut run = run_chain(dir, layer, chain)?;
run.cmd = format!("{version} && {}", run.cmd);
Ok(run)
}

/// A mutation layer's run, and what its `outcomes.json` held.
Expand Down
17 changes: 9 additions & 8 deletions crates/cargo-tops/src/gate/judge.rs
Original file line number Diff line number Diff line change
Expand Up @@ -272,23 +272,23 @@ fn finish(
laws_seconds: laws.seconds,
bend_ran: laws.ran,
laws_failed: laws.failed,
environment: done
.runs
.iter()
.any(|run| run.outcome == Outcome::ToolMissing),
environment: done.runs.iter().any(|run| run.outcome.lacks_tool()),
law_total,
survivors,
}
}

/// The verdict of a layer's runs: a failure outranks a missing tool, which
/// outranks a vacuous run.
/// outranks a tool at another version than its pin, which outranks a vacuous
/// run.
fn runs_verdict(runs: &[&Run]) -> &'static str {
let any = |outcome: Outcome| runs.iter().any(|run| run.outcome == outcome);
if any(Outcome::Failed) {
"fail"
} else if any(Outcome::ToolMissing) {
"tool-missing"
} else if any(Outcome::ToolMismatch) {
"tool-mismatch"
} else if any(Outcome::Vacuous) {
"vacuous"
} else {
Expand Down Expand Up @@ -408,7 +408,7 @@ fn judge_laws(
/// judged the laws layer. A layer that took another layer's run ran none.
pub(crate) fn ran_commands(runs: &[Run], bend_ran: bool, shared: &[(&str, &str)]) -> usize {
runs.iter()
.filter(|run| run.outcome != Outcome::ToolMissing && run.ran())
.filter(|run| !run.outcome.lacks_tool() && run.ran())
.filter(|run| !shared.iter().any(|(layer, _)| *layer == run.layer))
.count()
+ usize::from(bend_ran)
Expand Down Expand Up @@ -437,7 +437,7 @@ pub(crate) fn command_count(
"laws" => !bend_ran,
_ => !runs
.iter()
.any(|run| run.layer == *id && run.outcome != Outcome::ToolMissing && run.ran()),
.any(|run| run.layer == *id && !run.outcome.lacks_tool() && run.ran()),
})
.collect();
let mut line = format!(
Expand Down Expand Up @@ -485,7 +485,8 @@ pub(crate) fn laws_evaluated(cases: &Path) -> Option<(usize, usize)> {
pub(crate) fn run_failure(run: &Run) -> Option<String> {
match run.outcome {
Outcome::Passed | Outcome::Vacuous => None,
Outcome::Failed => Some(match &run.note {
// A tool at another version always carries its note: tool, pin, found.
Outcome::Failed | Outcome::ToolMismatch => Some(match &run.note {
Some(note) => format!("{}: {note}", run.layer),
None => format!("{}: `{}` failed", run.layer, run.step),
}),
Expand Down
3 changes: 2 additions & 1 deletion crates/cargo-tops/src/gate/mod.rs
Original file line number Diff line number Diff line change
Expand Up @@ -10,7 +10,8 @@
//! expired exemption); 2 the config or the request cannot be gated
//! (`rust-tops.yaml` missing or failing check, `rust-tops.lock` unreadable,
//! no enabled layer at or below `--max-tier`); 3 the environment (a required
//! tool or `cargo metadata` is missing, or `report.json` cannot be written).
//! tool or `cargo metadata` is missing, a pinned tool prints another version
//! than its pin in `rust-tops.lock`, or `report.json` cannot be written).
//! Where more than one applies, 2 wins over 3, and 3 over 1; every problem is
//! still listed in the report.
use std::path::Path;
Expand Down
Loading
Loading