Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
5 changes: 5 additions & 0 deletions .github/workflows/ci.yml
Original file line number Diff line number Diff line change
Expand Up @@ -151,6 +151,11 @@ jobs:
env:
PYTHONDONTWRITEBYTECODE: "1"
run: python3 -m unittest discover -s .github/scripts -p 'test_intel_host_tests.py'
# Stale generated files fail here in minutes, before the expensive
# roots: every input a committed build oracle records, every lock, and
# every committed export package must be current (§9.4). No Lean.
- name: Check that the committed generated files are current
run: cargo xtask check-freshness
- name: Run the honesty part of the acceptance gate
shell: bash
run: |
Expand Down
4 changes: 4 additions & 0 deletions CHANGELOG.md
Original file line number Diff line number Diff line change
Expand Up @@ -17,6 +17,10 @@ workspace carried at those versions without a tag.

## Unreleased

- `cargo xtask check-freshness` (SPEC.md §9.4), run in the honesty job before the expensive roots:
every input a committed build oracle records still hashes to its recorded digest, every lock
passes `lexlean lock --check`, and every committed export package names this compiler's
semantics and its project's committed semantic ID. Stale generated files fail in minutes.
- Declaration IDs (SPEC.md §21.9, `SM-38`; no new diagnostic code, no change to
any compiler-semantics ID). A language-1.2 build publishes
`declarations/<module>.json` (`lexlean/declarations/1`, schema
Expand Down
2 changes: 1 addition & 1 deletion SPEC.md
Original file line number Diff line number Diff line change
Expand Up @@ -703,7 +703,7 @@ Every normative gate job runs on the standard GitHub-hosted runner, `ubuntu-24.0

The Intel host factory is separate from the normative BDD and Atlas partitions above: it transfers only the same-run host test construction and dependency data of §8.3. Construction requires successful honesty; consumers require construction; the aggregate requires all of them and cannot skip an incomplete result. The factory's behavioral tests run in the initial honesty job, before the expensive roots, and exercise inventory completeness, custody and fail-closed result joins without needing an Intel runner. These tests do not substitute for execution of the complete workload on Intel macOS.

`ci.yml` MUST expose the initial `honesty` job (retaining the previous check context) separately so claim drift is distinguishable from code failure. It runs the existing format, model and specification-link recipes and the complete BDD meta-gate target, without Lean. The remaining rows, supported hosts and Atlas prefix MUST require its success; these recipes and meta tests run once in the normative gate. Both the plan's previous-run evidence and release evidence MUST require the honesty row, including its successful conclusion; missing, failed or skipped honesty never establishes full acceptance. The separate `honesty.yml` workflow is subsumed by this same-run job.
`ci.yml` MUST expose the initial `honesty` job (retaining the previous check context) separately so claim drift is distinguishable from code failure. It runs the existing format, model and specification-link recipes and the complete BDD meta-gate target, without Lean. Before them it runs `cargo xtask check-freshness`, also without Lean: every input a committed build oracle (`expected/build/manifest.json`) records still has its recorded digest and length, every such project's lock passes `lexlean lock --check`, every oracle and every committed package under `tests/production-export/` names this compiler's semantics, and every such package names its project's committed semantic ID. A tree committed without regenerating what its change moves fails there in minutes, before the expensive roots; the check adds a precondition and replaces no check of the gate, which still compares every generated byte. The remaining rows, supported hosts and Atlas prefix MUST require its success; these recipes and meta tests run once in the normative gate. Both the plan's previous-run evidence and release evidence MUST require the honesty row, including its successful conclusion; missing, failed or skipped honesty never establishes full acceptance. The separate `honesty.yml` workflow is subsumed by this same-run job.

`reproducibility.yml` MUST perform two builds in distinct absolute directories and compare every platform-independent artifact byte-for-byte after path normalization.

Expand Down
9 changes: 9 additions & 0 deletions crates/conformance/src/cases/repository.rs
Original file line number Diff line number Diff line change
Expand Up @@ -2244,6 +2244,15 @@ pub(crate) fn run(id: &str) {
);
// The honesty and reproducibility workflows stay separate.
let honesty = job("honesty");
// §9.4: stale generated files fail in honesty, before the gate
// command and every expensive root.
let fresh = honesty
.find(" run: cargo xtask check-freshness\n")
.expect("§9.4: honesty checks that the committed generated files are current");
assert!(
fresh < honesty.find("just --timestamp").expect("the gate command"),
"§9.4: the freshness check runs before the honesty recipes"
);
assert!(
honesty.contains("LEXLEAN_BDD_PART: ${{ matrix.bdd_part }}")
&& honesty.contains("fromJSON(needs.plan.outputs.honesty)")
Expand Down
318 changes: 318 additions & 0 deletions crates/conformance/src/freshness.rs
Original file line number Diff line number Diff line change
@@ -0,0 +1,318 @@
//! Whether the committed generated files are current (SPEC.md §9.4), checked
//! by `cargo xtask check-freshness` in the initial honesty job, without Lean
//! and in seconds. A change to a source, a project file, a lock, or the
//! compiler that was committed without regenerating what depends on it fails
//! here, before the expensive roots of the gate start, instead of hours into
//! them. It adds a precondition to the gate and replaces none of its checks:
//! `golden`, `examples`, and the bootstrap and production-export cases still
//! compare every generated byte.
//!
//! What it reads: each committed build oracle `expected/build/manifest.json`
//! records the digest and length of every input its build read and the
//! compiler semantics that built it; each committed package of
//! `tests/production-export/` records, in `provenance.json`, the compiler
//! semantics and the semantic ID of the build it was exported from; each
//! project's `lexlean.lock` is what `lexlean lock --check` accepts.

use std::collections::BTreeMap;

use camino::{Utf8Path, Utf8PathBuf};
use lexlean::{Engine, LockRequest};
use serde_json::Value;
use sha2::{Digest, Sha256};

/// The projects of `tests/production-export/`, each with its directory.
pub const EXPORTED: [(&str, &str); 5] = [
("bootstrap", "bootstrap"),
("helpdesk", "examples/helpdesk"),
("production", "examples/production"),
("production-coverage", "examples/production-coverage"),
("stdlib", "stdlib"),
];

/// What the check read.
#[derive(Debug, Default, Clone, PartialEq, Eq)]
pub struct Summary {
/// Projects with a committed build oracle.
pub projects: usize,
/// Recorded inputs hashed.
pub inputs: usize,
/// Committed export packages read.
pub packages: usize,
}

fn sha256_hex(bytes: &[u8]) -> String {
Sha256::digest(bytes)
.iter()
.map(|byte| format!("{byte:02x}"))
.collect()
}

fn read_json(path: &Utf8Path) -> Result<Value, String> {
let bytes = std::fs::read(path.as_std_path()).map_err(|error| format!("{path}: {error}"))?;
serde_json::from_slice(&bytes).map_err(|error| format!("{path}: {error}"))
}

/// Every project directory under `root` with a committed build oracle.
#[must_use]
pub fn oracle_projects(root: &Utf8Path) -> Vec<Utf8PathBuf> {
let mut found = Vec::new();
for parent in [root.to_owned(), root.join("examples")] {
let Ok(entries) = std::fs::read_dir(parent.as_std_path()) else {
continue;
};
for entry in entries.flatten() {
let Ok(path) = Utf8PathBuf::from_path_buf(entry.path()) else {
continue;
};
if path.join("expected/build/manifest.json").is_file() {
found.push(path);
}
}
}
found.sort();
found
}

/// The source packages a project names, by package, as directories.
fn source_packages(
root: &Utf8Path,
project: &Utf8Path,
) -> Result<BTreeMap<String, Utf8PathBuf>, String> {
let file = project.join("lexlean.toml");
let text =
std::fs::read_to_string(file.as_std_path()).map_err(|error| format!("{file}: {error}"))?;
let config: toml::Value = toml::from_str(&text).map_err(|error| format!("{file}: {error}"))?;
let mut packages = BTreeMap::new();
for row in config
.get("source_package")
.and_then(toml::Value::as_array)
.into_iter()
.flatten()
{
if let (Some(name), Some(path)) = (
row.get("package").and_then(toml::Value::as_str),
row.get("path").and_then(toml::Value::as_str),
) {
packages.insert(name.to_owned(), root.join(path));
}
}
Ok(packages)
}

/// Whether one project's committed build oracle and lock are current. Returns
/// the oracle's semantic ID and the number of inputs hashed.
///
/// # Errors
///
/// Every stale or unreadable input, one per line.
pub fn check_project(root: &Utf8Path, project: &Utf8Path) -> Result<(String, usize), String> {
let mut unmet = Vec::new();
let manifest = read_json(&project.join("expected/build/manifest.json"))?;
let name = project.strip_prefix(root).unwrap_or(project);
let language = manifest["language"].as_str().unwrap_or_default();
let compiler = lexlean::compiler_semantics_id_for(language).to_hex();
if manifest["compiler"]["semantics_id"].as_str() != Some(compiler.as_str()) {
unmet.push(format!(
"{name}: expected/build was built by compiler semantics {}, this compiler is {compiler}",
manifest["compiler"]["semantics_id"]
));
}
let packages = source_packages(root, project)?;
let mut hashed = 0;
for input in manifest["inputs"].as_array().into_iter().flatten() {
let path = input["path"].as_str().unwrap_or_default();
if input["kind"] == "lexicon" {
// A lexicon's digest is its package tree's: the lock records a
// local package's (`lock --check` below), and the compiler's
// semantics ID, compared above, covers an embedded one.
continue;
}
let file = match path.split_once(':') {
Some((package, relative)) => match packages.get(package) {
Some(directory) => directory.join(relative),
None => {
unmet.push(format!(
"{name}: input `{path}` names no source package of lexlean.toml"
));
continue;
}
},
None => project.join(path),
};
hashed += 1;
match std::fs::read(file.as_std_path()) {
Ok(bytes) => {
if input["sha256"].as_str() != Some(sha256_hex(&bytes).as_str())
|| input["byte_length"].as_u64() != Some(bytes.len() as u64)
{
unmet.push(format!(
"{name}: `{path}` changed since expected/build was generated"
));
}
}
Err(error) => unmet.push(format!("{name}: `{path}`: {error}")),
}
}
let lock = Engine::load(&project.join("lexlean.toml")).and_then(|engine| {
engine.lock(LockRequest {
check_only: true,
allow_network: false,
})
});
if let Err(error) = lock {
unmet.push(format!(
"{name}: `lexlean lock --check` fails: {:?}",
error
.diagnostics
.iter()
.map(|diagnostic| diagnostic.message.clone())
.collect::<Vec<_>>()
));
}
if unmet.is_empty() {
Ok((
manifest["semantic_id"]
.as_str()
.unwrap_or_default()
.to_owned(),
hashed,
))
} else {
Err(unmet.join("\n"))
}
}

/// Whether every committed generated file under `root` is current.
///
/// # Errors
///
/// Every stale file, one per line, with the regeneration that renews it.
pub fn check(root: &Utf8Path) -> Result<Summary, String> {
let mut unmet = Vec::new();
let mut summary = Summary::default();
let mut semantic = BTreeMap::new();
for project in oracle_projects(root) {
summary.projects += 1;
match check_project(root, &project) {
Ok((id, hashed)) => {
summary.inputs += hashed;
semantic.insert(project, id);
}
Err(reason) => unmet.push(reason),
}
}
let compiler = lexlean::compiler_semantics_id_for(lexlean::LANGUAGE_1_2).to_hex();
for (exported, directory) in EXPORTED {
let packages = root
.join("tests/production-export")
.join(exported)
.join("packages");
let expected = semantic.get(&root.join(directory));
for entry in walkdir::WalkDir::new(packages.as_std_path())
.into_iter()
.flatten()
{
if entry.file_name() != "provenance.json" {
continue;
}
let Ok(path) = Utf8PathBuf::from_path_buf(entry.into_path()) else {
continue;
};
summary.packages += 1;
let relative = path.strip_prefix(root).unwrap_or(&path).to_owned();
let provenance = match read_json(&path) {
Ok(value) => value,
Err(reason) => {
unmet.push(reason);
continue;
}
};
if provenance["compiler_semantics"].as_str() != Some(compiler.as_str()) {
unmet.push(format!(
"{relative}: exported by compiler semantics {}, this compiler is {compiler}",
provenance["compiler_semantics"]
));
}
let sources: Vec<&str> = provenance["sources"]
.as_array()
.into_iter()
.flatten()
.filter_map(Value::as_str)
.collect();
if expected.is_none_or(|id| sources != [id.as_str()]) {
unmet.push(format!(
"{relative}: exported from {sources:?}, {directory}/expected/build is {}",
expected.map_or("missing", String::as_str)
));
}
}
}
if unmet.is_empty() {
Ok(summary)
} else {
unmet.sort();
Err(format!(
"{} committed generated file(s) are stale; regenerate them (`regenerate.yml`, or the `-write` recipes):\n{}",
unmet.len(),
unmet.join("\n")
))
}
}

#[cfg(test)]
mod tests {
use super::*;

fn copy(from: &Utf8Path, to: &Utf8Path) {
for entry in walkdir::WalkDir::new(from.as_std_path())
.into_iter()
.flatten()
{
let relative = entry
.path()
.strip_prefix(from.as_std_path())
.expect("below");
if relative.starts_with(".lexlean") {
continue;
}
let target = to.as_std_path().join(relative);
if entry.file_type().is_dir() {
std::fs::create_dir_all(&target).expect("dir");
} else if entry.file_type().is_file() {
std::fs::copy(entry.path(), &target).expect("copy");
}
}
}

#[test]
fn a_committed_oracle_is_current_and_a_changed_input_is_stale() {
let root = crate::support::repo_root();
let temp = tempfile::tempdir().expect("tempdir");
let copy_root = Utf8PathBuf::from_path_buf(temp.path().to_owned()).expect("utf8");
let project = copy_root.join("examples/nat-add-zero");
copy(&root.join("examples/nat-add-zero"), &project);
let (semantic, hashed) =
check_project(&copy_root, &project).expect("the committed oracle is current");
assert!(!semantic.is_empty() && hashed >= 2, "{semantic} {hashed}");
let source = walkdir::WalkDir::new(project.join("src").as_std_path())
.into_iter()
.flatten()
.find(|entry| entry.path().extension().is_some_and(|ext| ext == "tex"))
.expect("a source")
.into_path();
let mut text = std::fs::read_to_string(&source).expect("source");
text.push('\n');
std::fs::write(&source, text).expect("write");
let stale = check_project(&copy_root, &project).expect_err("a changed source is stale");
assert!(
stale.contains("changed since expected/build was generated"),
"{stale}"
);
std::fs::remove_file(project.join("expected/build/manifest.json")).expect("remove");
assert!(
check_project(&copy_root, &project).is_err(),
"a missing oracle fails"
);
}
}
1 change: 1 addition & 0 deletions crates/conformance/src/lib.rs
Original file line number Diff line number Diff line change
Expand Up @@ -16,6 +16,7 @@ pub mod certification_library;
pub mod differential;
pub mod evidence;
pub mod fixtures;
pub mod freshness;
pub mod gnaf;
pub mod inventory;
pub mod lcnf_adapter;
Expand Down
2 changes: 1 addition & 1 deletion xtask/Cargo.toml
Original file line number Diff line number Diff line change
@@ -1,6 +1,6 @@
[package]
name = "xtask"
description = "Repository gates: validate-model, validate-spec-links, verify-examples, check-golden, check-reproducibility, check-fixtures, check-calculus, check-production-export, check-package, check-test-inventory, bdd, ci-summary, ci-hints, ci-shards, release-artifacts, release-check, validate."
description = "Repository gates: validate-model, validate-spec-links, verify-examples, check-golden, check-reproducibility, check-fixtures, check-calculus, check-production-export, check-freshness, check-package, check-test-inventory, bdd, ci-summary, ci-hints, ci-shards, release-artifacts, release-check, validate."
version.workspace = true
edition.workspace = true
rust-version.workspace = true
Expand Down
Loading