Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
10 changes: 9 additions & 1 deletion .github/workflows/bootstrap.yml
Original file line number Diff line number Diff line change
Expand Up @@ -27,7 +27,9 @@ jobs:
# formal legs, the merge of an up-to-date pull request for all of them) finds
# the receipt and does not repeat the work; a formal leg that failed on the
# same inputs is refuted in seconds instead of failing again an hour later
# (re-run the job to observe it anyway). Each leg uploads only its outcome;
# (re-run the job to observe it anyway). Before it runs, a leg deletes every
# file it does not declare, so an undeclared input fails the leg instead of
# proving a receipt it should not. Each leg uploads only its outcome;
# factory.yml turns outcomes into receipts from the default branch.
template-gate:
name: acceptance / ${{ matrix.leg }} / ${{ matrix.runner }}
Expand Down Expand Up @@ -64,6 +66,12 @@ jobs:
if: steps.route.outputs.route != 'observe' && steps.route.outputs.route != 'proved'
shell: bash
run: exit 1
- name: Keep only this leg's declared inputs
if: steps.route.outputs.route == 'observe'
shell: bash
env:
OBLIGATION: ${{ matrix.leg }}-${{ matrix.runner }}
run: node scripts/factory.mjs materialize "$OBLIGATION"
- name: Pull the locked SDK, retrying the registry
if: steps.route.outputs.route == 'observe'
shell: bash
Expand Down
33 changes: 26 additions & 7 deletions factory/README.md
Original file line number Diff line number Diff line change
Expand Up @@ -15,18 +15,36 @@ that orders remaining work. Here the decisions are a small Node host
| Obligation | Kind | Reads | Carried out by |
| --- | --- | --- | --- |
| `rest-<runner>` (2) | fetch | everything | bootstrap `template-gate` |
| `artifacts`, `claims`, `varint`, `reproducibility`, `integrity` | hermetic | the formal set | bootstrap `template-gate` |
| `artifacts` | hermetic | the LexLean project: `src/` and its project files | bootstrap `template-gate` |
| `varint` | hermetic | the import closure of `Uorc.Varint`, `Uorc.VarintContract`; the project files, xtask, scripts | bootstrap `template-gate` |
| `integrity` | hermetic | the import closure of every `Uorc.Integrity` module, `Uorc.Wire.ArchiveIntegrity`, `Uorc.Wire.OwnedRead` (99 of 340 modules); `validation/integrity/`, `standards/`; the project files, xtask, scripts | bootstrap `template-gate` |
| `claims`, `reproducibility` | hermetic | the formal set | bootstrap `template-gate` |
| `offline` | fetch | the formal set | offline-validation |
| `export` | fetch, host | the import closure of `Uorc.Client`, the LexLean project files, the export scripts | `node scripts/client-export.mjs` on a workstation |
| `export` | fetch, host | the import closure of `Uorc.Client` (client/src over src), the LexLean project files, the export scripts | `node scripts/client-export.mjs` on a workstation |
| `gate` | gate | the identities of the seven bootstrap legs | proved when all seven are |
| `native-<target>` (2), `wasm` | fetch | the client set (+ `crates/uorc-wasm/`) | release `native`, `wasm` |
| `bench` | fetch | `scripts/bench.mjs`, release.yml, `native-x86_64` | release `bench` |
| `publish` | fetch | everything, `@mode`, `@run` | release `publish`, `pages` |

The formal set is the tree without the product client and the release plumbing
(the same scope the formal legs had before). The client set is what `cargo build
-p uorc` reads: the workspace manifests and lock, the toolchain, `.cargo/`,
`crates/uorc/` and `generated/rust/`.
The formal set is the tree without the product client and the release plumbing.
The legs that build xtask (every formal leg but `artifacts`) read the product
crates' Cargo manifests only (`manifest:` inputs): the closure copy gives their
targets empty files, so cargo resolves the workspace without reading their code.
The client set is what `cargo build -p uorc` reads: the workspace manifests and
lock, the toolchain, `.cargo/`, `crates/uorc/` and `generated/rust/`.

A registry or claims edit therefore re-runs `artifacts`, `claims` and
`reproducibility` (they read all of `src/` or the model), not `integrity` or
`varint`; a change to `model/`, `features/` or `CONFORMANCE.md` does not re-run
`artifacts`, `integrity` or `varint`.

## Closure copy

Before a bootstrap leg runs, `factory.mjs materialize` deletes every tracked
file the leg does not declare (PrismPM runs each obligation in a copy of its
closure). An undeclared input fails the leg instead of proving a receipt it
should not. Adding the copy found one: `just model-write` reads `README.md`,
which the old κ scope left out.

## Identity

Expand All @@ -38,7 +56,8 @@ name <name>
kind hermetic|fetch|gate
environment <runner>
run <command>
input <prefix> one line per declared input (`!prefix` excludes, `@key` is a context, `closure:Module` an import closure)
input <prefix> one line per declared input (`!prefix` excludes, `@key` is a context,
`closure:ROOTS@DIRS` an import closure, `manifest:PATH` a Cargo manifest only)
needs <name> <identity> one line per need
closure <sha256 of the closure>
```
Expand Down
13 changes: 8 additions & 5 deletions factory/graph.json
Original file line number Diff line number Diff line change
@@ -1,19 +1,22 @@
{
"schema": "uorc/factory-graph/1",
"sets": {
"formal": ["", "!client/", "!crates/uorc/", "!crates/uorc-wasm/", "!generated/", "!web/", "!vendor/hologram-brand/", "!scripts/site.", "!Cargo.toml", "!Cargo.lock", "!README.md", "!scripts/bench.mjs", "!scripts/client-export.mjs", "!factory/", "!scripts/factory.", "!.github/workflows/release.yml", "!.github/workflows/pages.yml", "!.github/workflows/factory.yml"],
"formal": ["", "!client/", "!crates/uorc/", "!crates/uorc-wasm/", "!generated/", "!web/", "!vendor/hologram-brand/", "!scripts/site.", "!scripts/bench.mjs", "!scripts/client-export.mjs", "!factory/", "!scripts/factory.", "!.github/workflows/release.yml", "!.github/workflows/pages.yml", "!.github/workflows/factory.yml", "manifest:crates/conformance/Cargo.toml", "manifest:crates/uorc/Cargo.toml", "manifest:crates/uorc-wasm/Cargo.toml", "manifest:generated/rust/uorc_compress/Cargo.toml", "manifest:generated/rust/uorc_decompress/Cargo.toml"],
"lexlean": ["lexlean.toml", "lexlean.lock", "lakefile.toml", "lake-manifest.json", "lean-toolchain", "prismpm.lock", "Justfile", ".github/workflows/bootstrap.yml", "validation/lexlean-verify-hints.json"],
"xtask": ["Cargo.toml", "Cargo.lock", "rust-toolchain.toml", ".cargo/", "xtask/", "crates/model/", "manifest:crates/conformance/Cargo.toml", "manifest:crates/uorc/Cargo.toml", "manifest:crates/uorc-wasm/Cargo.toml", "manifest:generated/rust/uorc_compress/Cargo.toml", "manifest:generated/rust/uorc_decompress/Cargo.toml"],
"scripts": ["scripts/", "!scripts/factory.", "!scripts/site.", "!scripts/bench.mjs", "!scripts/client-export.mjs"],
"client": ["Cargo.toml", "Cargo.lock", "rust-toolchain.toml", ".cargo/", "crates/uorc/", "generated/rust/", ".github/workflows/release.yml"]
},
"obligations": [
{"name": "rest-ubuntu-24.04", "kind": "fetch", "environment": "ubuntu-24.04", "run": "just lexlean-authority template-check fmt-check model lint test features bdd deny", "inputs": [""], "needs": [], "workflow": "bootstrap.yml", "job": "template-gate"},
{"name": "rest-ubuntu-24.04-arm", "kind": "fetch", "environment": "ubuntu-24.04-arm", "run": "just lexlean-authority template-check fmt-check model lint test features bdd deny", "inputs": [""], "needs": [], "workflow": "bootstrap.yml", "job": "template-gate"},
{"name": "artifacts-ubuntu-24.04", "kind": "hermetic", "environment": "ubuntu-24.04", "run": "just lexlean-artifacts", "inputs": ["&formal"], "needs": [], "workflow": "bootstrap.yml", "job": "template-gate"},
{"name": "artifacts-ubuntu-24.04", "kind": "hermetic", "environment": "ubuntu-24.04", "run": "just lexlean-artifacts", "inputs": ["&lexlean", "src/"], "needs": [], "workflow": "bootstrap.yml", "job": "template-gate"},
{"name": "claims-ubuntu-24.04", "kind": "hermetic", "environment": "ubuntu-24.04", "run": "just claim-dispositions formal-tests", "inputs": ["&formal"], "needs": [], "workflow": "bootstrap.yml", "job": "template-gate"},
{"name": "varint-ubuntu-24.04", "kind": "hermetic", "environment": "ubuntu-24.04", "run": "just varint-proofs", "inputs": ["&formal"], "needs": [], "workflow": "bootstrap.yml", "job": "template-gate"},
{"name": "varint-ubuntu-24.04", "kind": "hermetic", "environment": "ubuntu-24.04", "run": "just varint-proofs", "inputs": ["&lexlean", "&xtask", "&scripts", "closure:Uorc.Varint,Uorc.VarintContract"], "needs": [], "workflow": "bootstrap.yml", "job": "template-gate"},
{"name": "reproducibility-ubuntu-24.04", "kind": "hermetic", "environment": "ubuntu-24.04", "run": "just reproducibility", "inputs": ["&formal"], "needs": [], "workflow": "bootstrap.yml", "job": "template-gate"},
{"name": "integrity-ubuntu-24.04", "kind": "hermetic", "environment": "ubuntu-24.04", "run": "just integrity", "inputs": ["&formal"], "needs": [], "workflow": "bootstrap.yml", "job": "template-gate"},
{"name": "integrity-ubuntu-24.04", "kind": "hermetic", "environment": "ubuntu-24.04", "run": "just integrity", "inputs": ["&lexlean", "&xtask", "&scripts", "closure:Uorc.Integrity.*,Uorc.Wire.ArchiveIntegrity,Uorc.Wire.OwnedRead", "validation/integrity/", "standards/", "standards.lock"], "needs": [], "workflow": "bootstrap.yml", "job": "template-gate"},
{"name": "offline", "kind": "fetch", "environment": "ubuntu-24.04", "run": "node scripts/offline-validation.mjs acquire, isolated first and second-distinct-root, compare-offline", "inputs": ["&formal"], "needs": [], "workflow": "offline-validation.yml", "job": "offline-compare"},
{"name": "export", "kind": "fetch", "environment": "host", "run": "node scripts/client-export.mjs", "inputs": ["closure:Uorc.Client", "lexlean.toml", "lakefile.toml", "lake-manifest.json", "lean-toolchain", "scripts/client-export.mjs", "scripts/semantic-source.mjs"], "needs": [], "record": "generated/rust/factory.identity"},
{"name": "export", "kind": "fetch", "environment": "host", "run": "node scripts/client-export.mjs", "inputs": ["closure:Uorc.Client@client/src,src", "lexlean.toml", "lakefile.toml", "lake-manifest.json", "lean-toolchain", "scripts/client-export.mjs", "scripts/semantic-source.mjs"], "needs": [], "record": "generated/rust/factory.identity"},
{"name": "gate", "kind": "gate", "environment": "-", "run": "-", "inputs": [], "needs": ["rest-ubuntu-24.04", "rest-ubuntu-24.04-arm", "artifacts-ubuntu-24.04", "claims-ubuntu-24.04", "varint-ubuntu-24.04", "reproducibility-ubuntu-24.04", "integrity-ubuntu-24.04"], "workflow": "release.yml", "job": "factory"},
{"name": "native-x86_64-unknown-linux-gnu", "kind": "fetch", "environment": "ubuntu-24.04", "run": "cargo test --locked --release -p uorc && cargo build --locked --release -p uorc --target x86_64-unknown-linux-gnu", "inputs": ["&client"], "needs": [], "workflow": "release.yml", "job": "native", "evidence": "factory-native-x86_64-unknown-linux-gnu"},
{"name": "native-aarch64-unknown-linux-gnu", "kind": "fetch", "environment": "ubuntu-24.04-arm", "run": "cargo test --locked --release -p uorc && cargo build --locked --release -p uorc --target aarch64-unknown-linux-gnu", "inputs": ["&client"], "needs": [], "workflow": "release.yml", "job": "native", "evidence": "factory-native-aarch64-unknown-linux-gnu"},
Expand Down
2 changes: 1 addition & 1 deletion generated/rust/factory.identity
Original file line number Diff line number Diff line change
@@ -1 +1 @@
export 2c94bf8be48d5269b913c6ea52fea4ba1d2516072e4100947225037c50d0d0e1
export c128b01d319be60d3ea704e483e337bcec8ade796fe2e9539ea808b42bcf1c3b
61 changes: 55 additions & 6 deletions scripts/factory.mjs
Original file line number Diff line number Diff line change
Expand Up @@ -21,6 +21,7 @@
// node scripts/factory.mjs plan [--rev REV] [--context KEY=VALUE]...
// node scripts/factory.mjs identity NAME [--rev REV] [--context KEY=VALUE]...
// node scripts/factory.mjs route NAME... [--context KEY=VALUE]... [--expect-run TEXT]
// node scripts/factory.mjs materialize NAME (CI: delete what NAME does not declare)
// node scripts/factory.mjs outcome NAME IDENTITY success|failure DIR
// node scripts/factory.mjs publish DIR (factory.yml only: needs GH_TOKEN)
// node scripts/factory.mjs export-check | export-record
Expand Down Expand Up @@ -69,42 +70,61 @@ export function loadGraph(dir) {
return graph;
}

// `closure:Module` is the module's import closure, each module read from
// client/src when it has one there and from src otherwise (as the export stages it).
function closurePaths(module, entries) {
const paths = new Set(entries.map(e => e.path)); const found = []; const seen = new Set(); const stack = [module];
// `closure:ROOTS@DIRS` is the import closure of ROOTS (comma-separated module
// names; `Uorc.Integrity.*` names every module under that directory), each
// module read from the first of DIRS (default `src`) that has it, as the
// export stages client/src over src.
function closurePaths(spec, entries) {
const [rootList, dirList = 'src'] = spec.split('@');
const dirs = dirList.split(',');
const paths = new Set(entries.map(e => e.path)); const found = []; const seen = new Set();
const stack = rootList.split(',').flatMap(r => {
if (!r.endsWith('.*')) return [r];
const prefix = r.slice(0, -2).split('.').join('/') + '/';
return [...paths].flatMap(p => dirs.map(d => `${d}/`).filter(d => p.startsWith(d + prefix) && p.endsWith('.lex.tex')).map(d => p.slice(d.length, -8).split('/').join('.')));
});
while (stack.length) {
const m = stack.pop(); if (seen.has(m)) continue; seen.add(m);
const rel = m.split('.').join('/') + '.lex.tex';
const p = paths.has(`client/src/${rel}`) ? `client/src/${rel}` : paths.has(`src/${rel}`) ? `src/${rel}` : fail(`closure ${module}: module ${m} is not tracked`);
const p = dirs.map(d => `${d}/${rel}`).find(x => paths.has(x)) ?? fail(`closure ${spec}: module ${m} is not tracked`);
found.push(p);
const object = entries.find(e => e.path === p).object.split(':')[1];
for (const i of git(['cat-file', 'blob', object]).toString().matchAll(/\\importmodule\{([^}]+)\}/g)) stack.push(i[1]);
}
return new Set(found);
}

// `manifest:PATH` is a Cargo manifest read for workspace resolution only: the
// manifest is an input, and a closure copy gives each target it declares an
// empty file when the target's own sources are not inputs.
const manifestPath = i => i.slice('manifest:'.length);

/// An entry is covered when the longest inclusion it starts with is longer
/// than the longest exclusion (`!prefix`) it starts with.
export function covered(inputs, p, exact) {
let include = -1, exclude = -1;
for (const i of inputs) {
if (i.startsWith('@') || i.startsWith('closure:')) continue;
if (i.startsWith('manifest:')) { if (p === manifestPath(i)) include = Math.max(include, p.length); continue; }
if (i.startsWith('!')) { if (p.startsWith(i.slice(1))) exclude = Math.max(exclude, i.length - 1); }
else if (p.startsWith(i)) include = Math.max(include, i.length);
}
if (exact.has(p)) include = Math.max(include, p.length);
return include > exclude;
}

function exactPaths(o, entries) {
return new Set(o.inputs.filter(i => i.startsWith('closure:')).flatMap(i => [...closurePaths(i.slice(8), entries)]));
}

/// Identities of every obligation at REV: PrismPM's `prism-factory/obligation/1`.
export function identities(graph, entries, contexts) {
const ids = {};
for (const o of graph.obligations) {
// An obligation that reads a context nobody supplied, or needs one that
// does, has no identity here (a pull request run never plans `publish`).
if (o.inputs.some(i => i.startsWith('@') && contexts[i.slice(1)] === undefined) || o.needs.some(n => ids[n] === null)) { ids[o.name] = null; continue; }
const exact = new Set(o.inputs.filter(i => i.startsWith('closure:')).flatMap(i => [...closurePaths(i.slice(8), entries)]));
const exact = exactPaths(o, entries);
let closure = '';
for (const e of entries) if (covered(o.inputs, e.path, exact)) closure += `file ${e.object} ${e.path}\n`;
for (const i of o.inputs) if (i.startsWith('@')) {
Expand Down Expand Up @@ -268,6 +288,35 @@ function main(argv) {
fs.writeFileSync(path.join(out, `${name}.outcome`), `outcome ${name} ${identity} ${result === 'success' ? 'pass' : 'fail'} ${git(['rev-parse', 'HEAD']).toString().trim()}\n`);
return;
}
case 'materialize': {
// A closure copy in place (PrismPM runs each obligation in a copy of its
// closure): delete every tracked file the obligation does not declare,
// so an undeclared input fails the run instead of proving a false receipt.
const o = byName(opts.names[0]);
if (o.inputs.includes('') && !o.inputs.some(i => i.startsWith('!'))) { console.log(`${o.name} reads the whole tree`); return; }
const entries = tree('HEAD', dir), exact = exactPaths(o, entries);
let removed = 0, kept = 0;
for (const e of entries) {
if (covered(o.inputs, e.path, exact)) { kept++; continue; }
if (e.path.startsWith('factory/') || e.path.startsWith('scripts/factory.')) continue; // the host itself, for the outcome step
fs.rmSync(path.join(dir, e.path), {force: true}); removed++;
}
const stubs = [];
for (const i of o.inputs.filter(x => x.startsWith('manifest:'))) {
const manifest = manifestPath(i), crate = path.dirname(manifest);
const text = fs.readFileSync(path.join(dir, manifest), 'utf8');
const targets = [...text.matchAll(/^path\s*=\s*"([^"]+)"/gm)].map(m => m[1]);
if (!targets.length) targets.push('src/lib.rs');
for (const t of targets) {
const file = path.join(dir, crate, t);
if (fs.existsSync(file)) continue;
fs.mkdirSync(path.dirname(file), {recursive: true}); fs.writeFileSync(file, '');
stubs.push(path.join(crate, t));
}
}
console.log(`${o.name}: kept ${kept} declared files, removed ${removed}, stubbed ${stubs.length} target(s) of manifest-only crates`);
return;
}
case 'publish': return publish(graph, dir, opts.names[0]);
case 'export-check': {
const ids = identities(graph, tree('HEAD', dir), plannedContexts);
Expand Down
7 changes: 7 additions & 0 deletions scripts/factory.test.mjs
Original file line number Diff line number Diff line change
Expand Up @@ -60,3 +60,10 @@ test('weights stay in 1..16: a failure raises priority, a pass lowers it, and sc
const r = route(graph, ids, parseStore(''));
assert.deepEqual(schedule(graph, r, w).map(s => s.name), ['leg', 'build', 'bench', 'net']);
});

test('a manifest input covers only that manifest, even under an exclusion', () => {
const inputs = ['', '!crates/uorc/', 'manifest:crates/uorc/Cargo.toml'];
assert.equal(covered(inputs, 'crates/uorc/Cargo.toml', new Set()), true);
assert.equal(covered(inputs, 'crates/uorc/src/lib.rs', new Set()), false);
assert.equal(covered(inputs, 'crates/model/src/lib.rs', new Set()), true);
});
Loading
Loading