Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
16 changes: 16 additions & 0 deletions CHANGELOG.md
Original file line number Diff line number Diff line change
Expand Up @@ -17,6 +17,22 @@ workspace carried at those versions without a tag.

## Unreleased

- Every part of a verification names its toolchain by address (SPEC.md
§22.11, §9.4). A shard writes `toolchain.json` (`lexlean/verify-toolchain/1`)
beside its output: the SHA-256 of the `lean`, `lake`, and `leanchecker` its
preflight hashed, of the running `lexlean`, and the address of the `rustc`
that compiled it (its executable and `rustc_driver` library, which the build
script hashes). A part that reads another refuses it (`LLI9001`, naming the
part, the tool, and both addresses) unless all five equal its own. A record
names only the executable it ran, so a part that ran another `lake`,
`lexlean`, or `rustc` was accepted before. `VR-23` plants another `lake`
address in a shard's output, and on unix a join whose own `lake` differs by
one byte. The bootstrap export's custody record becomes
`lexlean/ci-bootstrap-export/2`: it also names the SHA-256 of the pinned
`lake`, `lean`, and `leanchecker` and the address of the `rustc` that built
the exporting binary, and a leg whose own differ refuses the export (`RP-14`
plants another `leanchecker` and another `rustc`). Published directories,
attestations, and `expected/verify` are unchanged.
- Releases and regeneration need no hand steps (SPEC.md §9.4). `regenerate.yml`
gains the job `deliver`: when the regenerated commit heads one open pull
request of this repository, the patch is committed on it, pushed to the pull
Expand Down
2 changes: 1 addition & 1 deletion CONFORMANCE.md
Original file line number Diff line number Diff line change
Expand Up @@ -678,7 +678,7 @@ the error registry does not sanction (R5, `ERRORS.md`).
| `VR-20` | `build` | The verifier process width is an operational resource profile selected only by the environment, defaulting to 1 and refusing any malformed value with LLV7017, and it enters no lock, identity, or verified artifact. |
| `VR-21` | `build` | A multi-module project whose imports form nontrivial frontiers verifies at width 1 and at a larger width to byte-identical verified directories and the same attestation identity, observed axiom sets, and policy rows, and no completion order orders a record or changes which failure is reported. |
| `VR-22` | `build` | Performance telemetry is written only to the CI evidence directory the environment names, a verification with telemetry on publishes byte-identical artifacts to one without, and telemetry planted in a verified artifact fails the gate. |
| `VR-23` | `build` | A project verified as the parts of one run, each shard running the processes its plan names with the oleans of the shards it needs and the join reading every shard's records, publishes a verified directory byte-identical to the whole verification's with the same attestation identity, and a shard whose plan leaves a module without its imports, a join missing a record, or a join reading a record of an executable other than its own, is refused. |
| `VR-23` | `build` | A project verified as the parts of one run, each shard running the processes its plan names with the oleans of the shards it needs and the join reading every shard's records, publishes a verified directory byte-identical to the whole verification's with the same attestation identity, and a shard whose plan leaves a module without its imports, a join missing a record, a join reading a record of an executable other than its own, or a part that ran a toolchain other than its own, is refused. |

## Cited authorities

Expand Down
8 changes: 4 additions & 4 deletions SPEC.md

Large diffs are not rendered by default.

74 changes: 74 additions & 0 deletions crates/conformance/src/cases/operational.rs
Original file line number Diff line number Diff line change
Expand Up @@ -905,6 +905,80 @@ fn parts_equal_the_whole() {
text.contains("from any part of the run") && text.contains(&rest[0]),
"{text}"
);
// A join refuses a part that ran another toolchain: the first shard's
// output names the address of every executable it ran or was built by
// (§22.11), and a `lake` other than the join's is refused, naming the
// part, the tool, and both addresses, although no record names `lake`.
let addresses_path = dir("a").join("toolchain.json");
let addresses = std::fs::read_to_string(addresses_path.as_std_path())
.expect("a shard names its toolchain addresses");
let lake = serde_json::from_str::<serde_json::Value>(&addresses).expect("the addresses parse")
["lake"]
.as_str()
.expect("the addresses name lake")
.to_owned();
let planted_lake = "f".repeat(lake.len());
assert_ne!(lake, planted_lake);
std::fs::write(
addresses_path.as_std_path(),
addresses.replacen(&lake, &planted_lake, 1),
)
.expect("plant another lake");
let error = run_part(VerifyPart::Join {
plan: plan(&rest, &rest),
parts: vec![dir("a"), dir("b")],
})
.map(|_| ())
.expect_err("a part of another lake");
support::expect_code(&error, "LLI9001");
let text = error.to_string();
assert!(
text.contains("ran `lake`") && text.contains(&planted_lake) && text.contains(&lake),
"{text}"
);
std::fs::write(addresses_path.as_std_path(), &addresses).expect("restore the addresses");
// The same refusal from a real binary: a join whose own toolchain's
// `lake` differs by one appended byte from the pinned one the shards ran
// refuses them, naming both addresses. Its preflight hashes the changed
// file, so the join never trusts a part of the pinned toolchain.
#[cfg(unix)]
{
let pinned = support::real_elan_home()
.join("toolchains")
.join(support::mangled_toolchain_name())
.join("bin")
.join("lake");
let mut changed = std::fs::read(&pinned).expect("the pinned lake");
changed.push(0);
let changed_address = lexlean::Sha256Digest::of(&changed).to_hex();
let home = support::fake_elan_home(&[("lake".to_owned(), changed)]);
let home_text = home.path().to_string_lossy().into_owned();
let project = P::copy_example(example);
let error = support::with_env(&[("ELAN_HOME", Some(&home_text))], || {
Operational::new(1, None).expect("width 1").scoped(|| {
lexlean::verify_part(
&project.engine(),
VerifyRequest {
selection: Selection::Entrypoints,
},
VerifyPart::Join {
plan: plan(&rest, &rest),
parts: vec![dir("a"), dir("b")],
},
)
})
})
.map(|_| ())
.expect_err("a join whose own lake changed");
support::expect_code(&error, "LLI9001");
let text = error.to_string();
assert!(
text.contains("ran `lake`") && text.contains(&lake) && text.contains(&changed_address),
"{text}"
);
}
#[cfg(not(unix))]
support::unix_only("VR-23", "symlinked toolchain to plant a changed `lake` in");
// A join refuses a record that another `leanchecker` ran: the second
// shard's replay record of a module names an executable that is not the
// join's, as a part on a different toolchain would have written it.
Expand Down
30 changes: 30 additions & 0 deletions crates/conformance/src/cases/repository.rs
Original file line number Diff line number Diff line change
Expand Up @@ -3968,6 +3968,15 @@ fn bootstrap_custody_decisions() {
.into_iter()
.map(|(name, text)| (name.to_owned(), text.to_owned()))
.collect();
let executables: BTreeMap<String, String> = [
("lake", "5"),
("lean", "6"),
("leanchecker", "7"),
("rustc", "8"),
]
.into_iter()
.map(|(name, digit)| (name.to_owned(), digit.repeat(64)))
.collect();
let here = Identity {
repository: "UOR-Foundation/LexLean".to_owned(),
run_id: "100".to_owned(),
Expand All @@ -3976,6 +3985,7 @@ fn bootstrap_custody_decisions() {
tree: "2".repeat(40),
pins: pins.clone(),
toolchains: toolchains.clone(),
executables: executables.clone(),
};
let bundle = ("d".repeat(64), 162);
let record = Custody {
Expand All @@ -3987,6 +3997,7 @@ fn bootstrap_custody_decisions() {
tree: here.tree.clone(),
pins,
toolchains,
executables,
bundle_path: ".lexlean/production/bundle".to_owned(),
bundle_sha256: bundle.0.clone(),
bundle_files: bundle.1,
Expand Down Expand Up @@ -4022,6 +4033,25 @@ fn bootstrap_custody_decisions() {
.insert("rustc".to_owned(), "rustc 1.97.0".to_owned());
}),
),
// An export whose toolchain reports the pinned versions but whose
// executables are other bytes: a changed `leanchecker`, or a binary
// built by another `rustc` (§22.11).
(
"executables",
Box::new(|record| {
record
.executables
.insert("leanchecker".to_owned(), "9".repeat(64));
}),
),
(
"executables",
Box::new(|record| {
record
.executables
.insert("rustc".to_owned(), "0".repeat(64));
}),
),
(
"bundle",
Box::new(|record| record.bundle_sha256 = "f".repeat(64)),
Expand Down
43 changes: 38 additions & 5 deletions crates/conformance/src/export_custody.rs
Original file line number Diff line number Diff line change
Expand Up @@ -2,8 +2,9 @@
//! (SPEC.md §9.4). The `bootstrap-export` job exports the bootstrap project
//! once, requires its packages to equal the committed packages, and uploads
//! the bundle with a custody record that binds it to the repository, the
//! run and its attempt, the commit and its tree, the toolchain pins and the
//! toolchains that ran, and the bundle's bytes. A conformance case of the same
//! run and its attempt, the commit and its tree, the toolchain pins, the
//! toolchains that ran and the addresses of their executables, and the
//! bundle's bytes. A conformance case of the same
//! run that needs the export takes it from there instead of exporting the
//! project again: the harness names the directory it waits in through
//! [`VARIABLE`], checks the record against its own run and checkout, replays
Expand All @@ -26,7 +27,7 @@ use crate::support::{self, P};
pub const VARIABLE: &str = "LEXLEAN_BOOTSTRAP_EXPORT";

/// The custody record's format.
pub const SPEC: &str = "lexlean/ci-bootstrap-export/1";
pub const SPEC: &str = "lexlean/ci-bootstrap-export/2";

/// The custody record, beside the bundle in the uploaded directory.
pub const CUSTODY_FILE: &str = "custody.json";
Expand All @@ -49,6 +50,10 @@ pub const ARTIFACT_DIR: &str = "artifact";
/// The repository files whose bytes pin the toolchains and the dependencies.
pub const PINS: [&str; 3] = ["rust-toolchain.toml", "lean-toolchain", "Cargo.lock"];

/// The executables of the pinned Lean toolchain a custody record addresses,
/// beside `rustc` (SPEC.md §22.11).
pub const LEAN_EXECUTABLES: [&str; 3] = ["lake", "lean", "leanchecker"];

/// The longest a case waits for the run's export: the export job's own
/// limit, after which the job has failed and the fetcher says so.
const WAIT: Duration = Duration::from_secs(6 * 60 * 60);
Expand All @@ -73,6 +78,10 @@ pub struct Custody {
pub pins: BTreeMap<String, String>,
/// What the toolchains report: `rustc -vV` and `lean --version`.
pub toolchains: BTreeMap<String, String>,
/// The address of each executable: the SHA-256 of `lake`, `lean`, and
/// `leanchecker` of the pinned toolchain, and the address of the `rustc`
/// that compiled the job's own binary (§22.11).
pub executables: BTreeMap<String, String>,
/// The project-relative bundle directory of the export.
pub bundle_path: String,
/// [`bundle_digest`] of the bundle.
Expand All @@ -98,6 +107,8 @@ pub struct Identity {
pub pins: BTreeMap<String, String>,
/// What the toolchains report.
pub toolchains: BTreeMap<String, String>,
/// The address of each executable.
pub executables: BTreeMap<String, String>,
}

fn sha256_hex(bytes: &[u8]) -> String {
Expand Down Expand Up @@ -130,7 +141,8 @@ fn run(root: &Utf8Path, program: &str, arguments: &[&str]) -> Result<String, Str
/// # Errors
///
/// When the run is not named by GitHub's variables, the checkout's tracked
/// files differ from its commit, or a toolchain cannot be asked its version.
/// files differ from its commit, a toolchain cannot be asked its version, or
/// an executable of the pinned toolchain cannot be read.
pub fn identity(root: &Utf8Path) -> Result<Identity, String> {
let variable = |name: &str| -> Result<String, String> {
std::env::var(name)
Expand Down Expand Up @@ -169,6 +181,19 @@ pub fn identity(root: &Utf8Path) -> Result<Identity, String> {
("lean".to_owned(), run(root, "lean", &["--version"])?),
("rustc".to_owned(), run(root, "rustc", &["-vV"])?),
]);
let bin = lexlean::verify::toolchain::toolchain_root()
.map_err(|error| format!("the pinned toolchain: {}", error.message))?
.join("bin");
let mut executables = BTreeMap::new();
for name in LEAN_EXECUTABLES {
let bytes = std::fs::read(bin.join(name).as_std_path())
.map_err(|error| format!("{bin}/{name}: {error}"))?;
executables.insert(name.to_owned(), sha256_hex(&bytes));
}
executables.insert(
"rustc".to_owned(),
lexlean::build_identity::RUSTC_SHA256.to_owned(),
);
Ok(Identity {
repository,
run_id,
Expand All @@ -177,6 +202,7 @@ pub fn identity(root: &Utf8Path) -> Result<Identity, String> {
tree: run(root, "git", &["rev-parse", "HEAD^{tree}"])?,
pins,
toolchains,
executables,
})
}

Expand Down Expand Up @@ -261,6 +287,12 @@ pub fn check(custody: &Custody, current: &Identity, bundle: &(String, u64)) -> R
&current.toolchains,
custody.toolchains == current.toolchains,
);
same(
"executables",
&custody.executables,
&current.executables,
custody.executables == current.executables,
);
same(
"bundle",
&(&custody.bundle_sha256, custody.bundle_files),
Expand Down Expand Up @@ -330,6 +362,7 @@ pub fn publish(output: &Utf8Path) -> Result<Custody, String> {
tree: before.tree,
pins: before.pins,
toolchains: before.toolchains,
executables: before.executables,
bundle_path: exported.output.bundle.to_string(),
bundle_sha256,
bundle_files,
Expand All @@ -349,7 +382,7 @@ pub fn publish(output: &Utf8Path) -> Result<Custody, String> {
/// # Errors
///
/// When the run has no export, the record names another run, commit, tree,
/// pin, toolchain, or bundle, or the bundle does not replay.
/// pin, toolchain, executable, or bundle, or the bundle does not replay.
pub fn take(directory: &Utf8Path) -> Result<Exported, String> {
std::fs::create_dir_all(directory.as_std_path())
.and_then(|()| std::fs::write(directory.join(WANTED).as_std_path(), b""))
Expand Down
5 changes: 5 additions & 0 deletions crates/lexlean/Cargo.toml
Original file line number Diff line number Diff line change
Expand Up @@ -39,6 +39,11 @@ walkdir = { workspace = true }
rustix = { workspace = true }
signal-hook = { workspace = true }

# The build script records the address of the `rustc` that compiles this crate
# (SPEC.md §22.11): the SHA-256 of its executable and driver library.
[build-dependencies]
sha2 = { workspace = true }

# `production-data` embeds the native preservation library
# (`language/preservation-1.2/library/`, 32.2 of the 36.4 MB of language data),
# which only certificate generation reads: `production build`, and `verify`
Expand Down
77 changes: 77 additions & 0 deletions crates/lexlean/build.rs
Original file line number Diff line number Diff line change
Expand Up @@ -97,6 +97,83 @@ fn main() {
errors.display()
);
fs::write(out_dir.join("errors.toml"), bytes).expect("write errors.toml");

rustc_identity(&out_dir);
}

/// The address of the compiler that builds this crate (SPEC.md §22.11): the
/// SHA-256 of one line `<sha256 of the file> <sysroot-relative path>` per
/// file of the `rustc` executable and its driver library, in path order, and
/// what `rustc -vV` reports. The `rustc` executable is a thin launcher; the
/// compiler is the driver library it loads, so both are addressed. A compiler
/// whose sysroot cannot be read is a build failure, never an empty address.
fn rustc_identity(out_dir: &Path) {
use sha2::{Digest, Sha256};
use std::process::Command;

println!("cargo:rerun-if-env-changed=RUSTC");
let rustc = env::var("RUSTC").unwrap_or_else(|_| "rustc".to_owned());
let run = |argument: &str| {
let output = Command::new(&rustc)
.arg(argument)
.output()
.unwrap_or_else(|error| panic!("running `{rustc} {argument}`: {error}"));
assert!(
output.status.success(),
"`{rustc} {argument}` failed: {}",
String::from_utf8_lossy(&output.stderr)
);
String::from_utf8(output.stdout).expect("rustc reports UTF-8")
};
let version = run("-vV").replace("\r\n", "\n");
let sysroot = PathBuf::from(run("--print=sysroot").trim());
let mut files: Vec<(String, PathBuf)> = Vec::new();
for directory in ["bin", "lib"] {
let Ok(entries) = fs::read_dir(sysroot.join(directory)) else {
continue;
};
for entry in entries.filter_map(Result::ok) {
let name = entry.file_name().to_string_lossy().into_owned();
let compiler = name == "rustc"
|| name == "rustc.exe"
|| name.starts_with("rustc_driver-")
|| name.starts_with("librustc_driver-");
if compiler && entry.path().is_file() {
files.push((format!("{directory}/{name}"), entry.path()));
}
}
}
files.sort();
assert!(
files.iter().any(|(path, _)| path.starts_with("bin/rustc")),
"{}: no `rustc` executable in the compiler's sysroot",
sysroot.display()
);
let mut listing = String::new();
for (relative, path) in &files {
println!("cargo:rerun-if-changed={}", path.display());
let bytes =
fs::read(path).unwrap_or_else(|error| panic!("reading {}: {error}", path.display()));
listing.push_str(&hex(&Sha256::digest(&bytes)));
listing.push_str(" ");
listing.push_str(relative);
listing.push('\n');
}
let address = hex(&Sha256::digest(listing.as_bytes()));
fs::write(
out_dir.join("rustc_identity.rs"),
format!(
"/// What `rustc -vV` reported for the compiler that built this crate.\n\
pub const RUSTC_VERSION: &str = {version:?};\n\
/// The address of that compiler's executable and driver library.\n\
pub const RUSTC_SHA256: &str = {address:?};\n"
),
)
.expect("write rustc_identity.rs");
}

fn hex(bytes: &[u8]) -> String {
bytes.iter().map(|byte| format!("{byte:02x}")).collect()
}

/// Resolve a crate-relative normative path: a directory or file as such; a
Expand Down
Loading
Loading