diff --git a/README.md b/README.md index 016d0dc..0f90109 100644 --- a/README.md +++ b/README.md @@ -14,9 +14,11 @@ A workflow file mixes two things. The orchestration (triggers, job graph, matrix **Run directly.** `run` executes the pipeline through an in-process runner that asks an injected executor to run each command. Nothing is generated. A single job, a step range or one matrix combination can be run alone. Around each step the runner applies the provider's file commands, as a hosted runner does: the step writes to `GITHUB_OUTPUT`, `GITHUB_ENV`, `GITHUB_PATH` and `GITHUB_STEP_SUMMARY`, and its stdout is read for `::add-mask::` and the log-level commands, so outputs reach the step's record, environment and paths reach the job's later steps, and masked values are not echoed. +GitHub's own parts of a pipeline run on the host too. A provider step carrying a `uses:` reference is resolved, a composite action's steps are interpreted and a node action's entry runs under the CLI's own node, with the action's `pre` and `post` phases placed around the job's steps the way the hosted runner places them. `${{ }}` expressions in command text, environments and conditions are evaluated with the run's context: step outputs and environment written by earlier steps, the results of needed jobs, and secrets supplied by `--secret name=value` or `--secret name`, which reads the environment variable of that name and never puts the value on a command line. Local `./` references resolve against the workspace and an `owner/repo@ref` reference is fetched as a tarball and extracted for the run's lifetime; the platform token, wherever the run has one, is supplied to expressions as `secrets.GITHUB_TOKEN` and sent with the fetch. What needs a container stays the provider's: a `docker://` reference is skipped by `--skip-provider-steps` and fails the run without it, and the same flag skips whole actions, for a host that has already supplied the toolchain those steps would install. + **Query the platform.** A separate read-only contract lists runs, checks, logs and the checks a branch requires. It does not depend on the pipeline model. -**Import a native file.** `import` reads a provider's file into a pipeline, as TypeScript or JSON, and `convert` writes another provider's files from it. What the model cannot express is rejected with the file and pointer, never approximated. +**Import a native file.** `import` reads a provider's file into a pipeline, as TypeScript or JSON, and `convert` writes another provider's files from it. What the model cannot express is rejected with the file and pointer, never approximated. A GitHub workflow imported as a pipeline runs through `import` then `run --local`: its actions, expressions, output hand-offs and `if` gates execute on the host for the supported subset, and what the host cannot run is rejected naming the construct rather than approximated. ## Authoring @@ -100,7 +102,7 @@ node packages/cli/src/bin.ts check-names --pipeline ci/ci.pipeline.ts --provider node packages/cli/src/bin.ts platform runs --repo OWNER/REPO --sha SHA ``` -`--pipeline` takes a `.ts` module or a `.json` document. `run` takes its context (event, ref, sha and so on) from the environment when it detects GitHub Actions or GitLab CI, or from `--context key=value`. A missing value is an error, not a guess. `--local` supplies the context of a manual run on a branch called `local`. +`--pipeline` takes a `.ts` module or a `.json` document. `run` takes its context (event, ref, sha and so on) from the environment when it detects GitHub Actions or GitLab CI, or from `--context key=value`. A missing value is an error, not a guess. `--local` supplies the context of a manual run on a branch called `local`. `run` takes secrets with `--secret name=value`, or `--secret name` to read the environment variable of that name; a secret an expression reads but the run was not given is an error naming it, and secret values are never printed, logged or stored. Platform commands read the token from `GH_TOKEN`; the adapter reads no files or other variables for it. diff --git a/docs/providers.md b/docs/providers.md index 21ac65e..d32ccea 100644 --- a/docs/providers.md +++ b/docs/providers.md @@ -63,6 +63,14 @@ A job that names no runner label has an empty `runsOn`, which means the provider Every adapter can render a job in the wrapper form: orchestration stays native and each contiguous run of portable steps becomes one call to this tool's runner for exactly those step indexes. A run step with `form: "native"` stays in the provider's syntax, which is how an install step runs before the tool exists. A matrix combination is passed as a quoted `--matrix "axis=value"` argument: as a shell-quoted literal where the adapter expands the matrix itself, and through the provider's own substitution (`${{ matrix.node }}`, `$node`, `<< parameters.node >>`) where the provider does. The run context the runner reads comes from the provider's environment variables; a value that provider does not expose (Bitbucket cannot tell a push from a manual run, CircleCI exposes no workflow name) is absent, and the runner fails only if a step or condition reads it. The runner's own file commands (`GITHUB_OUTPUT` and its siblings) apply between the steps of one wrapper call and are not forwarded to the provider's runner, so a native step after the call does not see what an inner step wrote. +## GitHub's own steps on the host + +A `run` that reaches a provider step of GitHub's resolves it on the host, in two moments that mirror the hosted runner's own. Before a job runs, every action the job's `uses` steps name, whose step this run will execute, is resolved and its `pre` and `post` phases are placed: `pre` before the job's own steps in the order the steps name their actions, `post` after them in the reverse of that order, which is where the hosted runner runs them. When the run reaches a step, it is resolved against what the job's earlier steps by then produced, so a `with:` mapping or an `if:` reading a step output, or command text reading the environment a step wrote, works as written. A composite action's steps run as commands; a node action's entry runs under the CLI's own node with its inputs in the environment as `INPUT_`; a local `./` reference resolves against the workspace and an `owner/repo@ref` reference is fetched as a tarball, extracted once per reference for the run's lifetime. + +Expressions evaluate with the run's context: the neutral context values mapped onto the `github` context's own names, the environment the next command inherits, the outputs and conclusions of the job's steps so far (keyed by the `id` the workflow gave them), the results of needed jobs, and secrets from `--secret`. The platform token, wherever the run holds one, is supplied as `secrets.GITHUB_TOKEN`. + +What stays provider-owned: a `docker://` reference, an action whose `runs.using` needs a container, a workflow step naming a shell other than `bash` or `sh` or a setting the model has no step form for (`continue-on-error`, `timeout-minutes`), and every step of another provider. Each is skipped by `--skip-provider-steps` and fails the run without it, naming the construct; `--skip-provider-steps` skips whole actions too, for a host that has already supplied the toolchain those steps would install. An action's declared `outputs` are read as data but not evaluated into the steps context, because the model records no outputs an action declares, and a composite action's step conditions see the view the step was reached with, not the outcomes of the composite's own later steps. + ## Check names `checkNames` returns, per job, the names under which the provider reports it, one per matrix combination: `Test (22)` on GitHub, `Test: [22]` on GitLab, and the step name on Bitbucket, which reports one status per pipeline. CircleCI names a matrix job from a templated name that the schema accepts and that was not run on CircleCI. diff --git a/eslint.config.ts b/eslint.config.ts index 041364f..6d3048b 100644 --- a/eslint.config.ts +++ b/eslint.config.ts @@ -18,6 +18,8 @@ export default defineConfig( "schemas", // Golden files are the exact bytes an adapter must emit. "packages/*/test/golden", + // Action fixtures are files a host run executes as data, like the tarball a fetched action arrives in. + "packages/cli/test/fixtures", ], }, { diff --git a/packages/cli/src/actions.ts b/packages/cli/src/actions.ts new file mode 100644 index 0000000..01605a6 --- /dev/null +++ b/packages/cli/src/actions.ts @@ -0,0 +1,580 @@ +import { mkdir, mkdtemp, readFile, rm, writeFile } from "node:fs/promises"; +import { tmpdir } from "node:os"; +import { dirname, join, resolve } from "node:path"; +import { gunzipSync } from "node:zlib"; + +import type { + ContextKey, + JobExpansion, + JobView, + PreparedStep, + RunContext, + Step, + StepHost, + StepResolution, + StepView, +} from "@pipewright/pipeline-contract"; +import { + ActionReferenceError, + EVENT_NAME, + PROVIDER, + evaluateCondition, + evaluateTemplate, + evaluationContextSchema, + expandAction, + parseActionReference, + type ActionResolver, + type ActionTree, + type EvaluationContext, + type NodeRunner, +} from "@pipewright/pipeline-github-actions"; + +/** The shells the host's own executor runs as written; a workflow step naming another interpreter names a command this host cannot run faithfully. */ +const POSIX_SHELLS = new Set(["bash", "sh"]); + +/** The GitHub event name of each neutral event; a name outside the table passes through as written. */ +const EVENT_NAMES: ReadonlyMap = new Map(Object.entries(EVENT_NAME)); + +/** + * The `github` context field each neutral context key maps onto. It mirrors the adapter's own expression table (`CONTEXT_EXPRESSION`), which the compiler writes and the importer reads, and a test holds the two to the same correspondence. + */ +export const GITHUB_FIELD: Readonly> = { + event: "event_name", + refName: "ref_name", + baseRefName: "base_ref", + sha: "sha", + repository: "repository", + workflow: "workflow", + runId: "run_id", +}; + +/** A provider step of GitHub's, whatever its native mapping carries. */ +type GitHubStep = Extract & { readonly provider: "github-actions" }; + +function isGitHubStep(step: Step): step is GitHubStep { + return step.kind === "provider" && step.provider === PROVIDER; +} + +/** The `uses` value a provider step carries, or undefined when it carries none or one that is not text. */ +function usesOf(step: Step): string | undefined { + return isGitHubStep(step) ? textOf(step.native, "uses") : undefined; +} + +/** A string a provider's native mapping carried, or undefined when the key is absent or holds anything else. */ +function textOf(native: Readonly>, key: string): string | undefined { + const value = native[key]; + + return typeof value === "string" ? value : undefined; +} + +/** Whether a value is a scalar a provider's expressions substitute, which every value an imported mapping carries is. */ +function isScalar(value: unknown): value is string | number | boolean { + return typeof value === "string" || typeof value === "number" || typeof value === "boolean"; +} + +function scalarText(value: string | number | boolean): string { + return typeof value === "string" ? value : String(value); +} + +/** Resolves a local reference's path inside a root, rejecting one that climbs above it, because an action cannot name files outside the repository or workspace its own reference was resolved from. */ +function within(root: string, path: string): string { + const resolved: string[] = []; + for (const segment of path.split("/")) { + if (segment === "" || segment === ".") continue; + if (segment === "..") { + if (resolved.pop() === undefined) { + throw new Error(`the local reference ./${path}, which climbs above the workspace`); + } + continue; + } + resolved.push(segment); + } + + return resolve(root, resolved.join("/")); +} + +/** A tree over a directory the host already holds: the workspace for a local reference, an extracted tarball for a remote one. */ +function directoryTree(directory: string): ActionTree { + const read = async (relative: string): Promise => { + try { + return await readFile(join(directory, relative), "utf8"); + } catch (error) { + if (error instanceof Error && "code" in error && error.code === "ENOENT") return undefined; + throw error; + } + }; + + return { + path: directory, + read, + local: async (relative) => Promise.resolve(directoryTree(within(directory, relative))), + }; +} + +/** The tar header's field offsets, in bytes, as the format fixes them. */ +const NAME_END = 100; +const SIZE_AT = 124; +const SIZE_END = 136; +const TYPE_AT = 156; +const PREFIX_AT = 345; +const PREFIX_END = 500; +/** Every block of a tar stream, header and data alike, is this size. */ +const BLOCK = 512; +/** The space that separates a pax record's length from its key. */ +const SPACE = 0x20; + +/** Reads a NUL- or space-padded string field of a tar header. */ +function fieldOf(header: Uint8Array, from: number, to: number): string { + const end = header.subarray(from, to).indexOf(0); + const field = header.subarray(from, end === -1 ? to : from + end); + + return new TextDecoder().decode(field); +} + +/** Reads a size field as the octal ASCII the format defines, rejecting the GNU base-256 form rather than misreading it. */ +function sizeOf(header: Uint8Array): number { + const text = fieldOf(header, SIZE_AT, SIZE_END).trim(); + if (!/^[0-7]*$/.test(text)) { + throw new Error("a tar entry whose size is not octal text, which this reader does not carry"); + } + + return text === "" ? 0 : Number.parseInt(text, 8); +} + +/** One entry of a tar stream: a file's bytes, or null for a directory. */ +interface TarEntry { + readonly name: string; + readonly bytes: Uint8Array | null; +} + +/** + * Reads the entries of a tar stream, the container GitHub serves a repository tarball in. A repository tarball starts with a pax global header naming its commit and carries a path too long for the header's name field in a pax extended header, so both are read: the global header's records are ignored unless they rewrite a path, and an extended header's `path` names the entry that follows it. A size a header's octal field cannot carry never appears in a repository tarball, so a pax `size` record is read only to be rejected. + */ +function readTar(tar: Uint8Array): TarEntry[] { + const entries: TarEntry[] = []; + let offset = 0; + let longName: string | undefined; + while (offset + BLOCK <= tar.length) { + const header = tar.subarray(offset, offset + BLOCK); + if (header.every((byte) => byte === 0)) break; + const size = sizeOf(header); + const data = tar.subarray(offset + BLOCK, offset + BLOCK + size); + offset += BLOCK + Math.ceil(size / BLOCK) * BLOCK; + const prefix = fieldOf(header, PREFIX_AT, PREFIX_END); + const name = + longName ?? [...(prefix === "" ? [] : [prefix]), fieldOf(header, 0, NAME_END)].join("/"); + const type = String.fromCharCode(header[TYPE_AT] ?? 0); + if (type === "x" || type === "g") { + const records = paxRecords(data); + if (records.has("size")) { + throw new Error("a pax header that rewrites a size, which this reader does not carry"); + } + const path = records.get("path"); + if (type === "g") { + if (path !== undefined) { + throw new Error( + "a pax global header that rewrites a path, which this reader does not carry", + ); + } + continue; + } + if (path !== undefined) longName = path; + continue; + } + if (type === "L") { + longName = new TextDecoder().decode(data).replace(/\0+$/, ""); + continue; + } + + longName = undefined; + if (type === "5" || name.endsWith("/")) { + entries.push({ name: name, bytes: null }); + continue; + } + + entries.push({ name: name, bytes: data }); + } + + return entries; +} + +/** Reads the key-value records of a pax header, each a length-prefixed `key=value` line. */ +function paxRecords(data: Uint8Array): Map { + const records = new Map(); + let offset = 0; + while (offset < data.length) { + const space = data.subarray(offset).indexOf(SPACE); + if (space === -1) throw new Error("a pax header record with no key"); + const length = Number.parseInt( + new TextDecoder().decode(data.subarray(offset, offset + space)), + 10, + ); + if (!Number.isInteger(length) || length <= 0) { + throw new Error("a pax header record whose length is not decimal text"); + } + const record = new TextDecoder().decode(data.subarray(offset, offset + length)); + const equals = record.indexOf("="); + const newline = record.lastIndexOf("\n"); + if (equals === -1 || newline === -1) { + throw new Error("a pax header record that is not key=value"); + } + records.set(record.slice(space + 1, equals), record.slice(equals + 1, newline)); + offset += length; + } + + return records; +} + +/** Writes a tarball's entries under a directory, stripping the repository root every entry name starts with. */ +async function extractTarball(root: string, tarball: Uint8Array): Promise { + await readTar(gunzipSync(tarball)).reduce>(async (previous, entry) => { + await previous; + // Every entry of a repository tarball starts with the repository's own root directory, whose name the action never sees, and a directory entry's name ends with the slash that ends it. + const whole = entry.name.endsWith("/") ? entry.name.slice(0, -1) : entry.name; + const segments = whole.split("/").slice(1); + const climbs = segments.some( + (segment) => segment === "" || segment === "." || segment === "..", + ); + if (segments.length === 0 || climbs) { + if (entry.bytes === null) return; + throw new Error(`a tarball entry outside its repository root: ${entry.name}`); + } + const target = join(root, ...segments); + if (entry.bytes === null) { + await mkdir(target, { recursive: true }); + + return; + } + await mkdir(dirname(target), { recursive: true }); + await writeFile(target, entry.bytes); + }, Promise.resolve()); +} + +/** The tarball endpoint a remote reference is fetched from, which answers a redirect to the download itself. */ +const TARBALL = "https://api.github.com/repos"; + +/** What the host needs from the outside world to resolve and run actions. */ +export interface HostOptions { + /** The workspace a local `./` reference resolves against. */ + readonly cwd: string; + readonly fetch: typeof fetch; + /** The run's context, whose values the `github` context maps onto GitHub's own names. */ + readonly context: RunContext; + /** The platform token, sent with action fetches and supplied to expressions as `secrets.GITHUB_TOKEN` and `github.token`; absent, both read as absent and only public repositories can be fetched. */ + readonly token: string | undefined; + /** Secrets supplied for the run, by name. Held for the run's lifetime, passed only to the evaluator, and never written anywhere. */ + readonly secrets: Readonly>; + /** Whether a provider step this host cannot run is reported skipped rather than failing the run. */ + readonly skipProviderSteps: boolean; +} + +/** + * Fetches a repository tarball, following the one redirect GitHub answers with and re-attaching the token, which a cross-origin redirect would otherwise drop and a private repository needs on the second hop. + * @see https://docs.github.com/en/rest/repos/contents + * @see https://web.archive.org/web/20261008201445/https://docs.github.com/en/rest/repos/contents + */ +async function fetchTarball( + options: HostOptions, + owner: string, + repo: string, + ref: string | undefined, +): Promise { + const url = `${TARBALL}/${owner}/${repo}/tarball${ref === undefined ? "" : `/${encodeURIComponent(ref)}`}`; + const headers: Record = { Accept: "application/vnd.github+json" }; + if (options.token !== undefined) headers.Authorization = `Bearer ${options.token}`; + let response = await options.fetch(url, { headers, redirect: "manual" }); + const location = response.headers.get("location"); + // GitHub answers a redirect to the download itself, one hop, and the token goes along: a cross-origin redirect would otherwise drop it and a private repository needs it there. + if (location !== null && !response.ok && new URL(location, url).hostname.endsWith("github.com")) { + response = await options.fetch(location, { headers, redirect: "manual" }); + } + if (!response.ok) { + throw new Error( + `fetching ${owner}/${repo}${ref === undefined ? "" : `@${ref}`} answered ${String(response.status)}`, + ); + } + + return new Uint8Array(await response.arrayBuffer()); +} + +/** The GitHub Actions host for a `run`: the resolver and node port the library's action expansion asks for, and the step host the runner calls. */ +export interface ActionHost { + readonly resolver: ActionResolver; + readonly node: NodeRunner; + readonly steps: StepHost; + /** Removes what the run fetched; called when the run ends, so nothing a run downloaded outlives it. */ + readonly dispose: () => Promise; +} + +/** The `github` context under GitHub's own names: each neutral context key mapped onto its field, plus the token wherever the run holds one. A key the run did not supply is absent, so an expression reading it reads null, as GitHub's own absent fields do. */ +function githubContext(context: RunContext, token: string | undefined): Record { + const github: Record = {}; + if (context.event !== undefined) { + // An event name this model does not know passes through as written, the same pass-through the importer's own event table applies. + github.event_name = EVENT_NAMES.get(context.event) ?? context.event; + } + if (context.refName !== undefined) github.ref_name = context.refName; + if (context.baseRefName !== undefined) github.base_ref = context.baseRefName; + if (context.sha !== undefined) github.sha = context.sha; + if (context.repository !== undefined) github.repository = context.repository; + if (context.workflow !== undefined) github.workflow = context.workflow; + if (context.runId !== undefined) github.run_id = context.runId; + if (token !== undefined) github.token = token; + + return github; +} + +/** The evaluation context for a view of the run: the neutral context mapped onto GitHub's names, the environment the next command inherits, and the steps and needed jobs as GitHub reports them. */ +function evaluationContextOf( + view: JobView | StepView, + github: Readonly>, + secrets: Readonly>, +): EvaluationContext { + const steps: Record }> = {}; + const records = "steps" in view ? view.steps : []; + for (const record of records) { + // Only a step the workflow gave an id can be read by one, and an imported step with an id stays a provider step that carries it. + const id = record.step.kind === "provider" ? textOf(record.step.native, "id") : undefined; + if (id === undefined || id in steps) continue; + steps[id] = { conclusion: record.outcome, outputs: { ...record.outputs } }; + } + const failed = records.some((record) => record.outcome === "failure"); + + return evaluationContextSchema.parse({ + github, + env: { ...view.env }, + job: { status: failed ? "failure" : "success" }, + steps, + needs: Object.fromEntries( + Object.entries(view.needs).map(([id, result]) => [id, { result, outputs: {} }]), + ), + matrix: { ...view.matrix }, + // The token GitHub supplies to every run, wherever the host holds one, so the expressions a workflow writes against it work unchanged. + secrets: { ...secrets, ...(github.token === undefined ? {} : { GITHUB_TOKEN: github.token }) }, + inputs: {}, + }); +} + +/** The environment a step's own mapping declares, evaluated against the view the step is reached with. */ +function stepEnvironment( + mapping: unknown, + context: EvaluationContext, + fail: (construct: string) => never, +): Record { + if (mapping === undefined) return {}; + if (typeof mapping !== "object" || mapping === null || Array.isArray(mapping)) { + return fail("a step environment that is not a mapping"); + } + const evaluated: Record = {}; + for (const [key, value] of Object.entries(mapping)) { + if (!isScalar(value)) { + return fail(`the step environment "${key}" with a value that is not a scalar`); + } + evaluated[key] = evaluateTemplate(scalarText(value), context); + } + + return evaluated; +} + +/** Why a `uses` value names no tree this host can read, or undefined when it parses. */ +function unparsable(uses: string): string | undefined { + try { + parseActionReference(uses); + + return undefined; + } catch (error) { + if (error instanceof ActionReferenceError) return error.construct; + throw error; + } +} + +/** The name a resolved step reports under: the step's own name, else its command text, as the runner does for a neutral run step. */ +function stepNameOf(step: Step, command: string): string { + return step.name ?? command; +} + +/** + * Builds the host that runs a pipeline's GitHub Actions parts on the machine `run` executes on. Local `./` references resolve against the workspace; a remote `owner/repo@ref` is fetched as a tarball, extracted once per reference for the run's lifetime, and served as a tree. A node action's entry runs under the same node the CLI itself runs under. + * + * The step host the runner calls works in two moments. Before a job runs it expands every action the job's `uses` steps name, with the pre-run view, and places the pre and post phases the expansion returned around the job's own steps, which is the placement the library documents from the runner's own phases; the expanded steps themselves are discarded, because the same action expands again when the step is reached, against what the job's earlier steps by then produced, which is the view GitHub evaluates a `with:` mapping and a step condition with. A provider step that carries a `run` instead resolves through the evaluator directly: its condition decides whether it runs, and its command text, environment and working directory are evaluated as written. + * + * What this host cannot run it leaves to the runner's own stance: a `docker://` reference and a step of another provider are reported skipped when the run was told to skip provider steps and fail the run otherwise, and a workflow step naming a shell or a setting the model has no step form for fails the run naming it. + */ +export function createActionHost(options: HostOptions): ActionHost { + const github = githubContext(options.context, options.token); + /** Fetched references by identity, so one action used twice is downloaded once. */ + const fetched = new Map>(); + let rootPromise: Promise | undefined; + const rootOf = async (): Promise => { + const created = rootPromise ?? mkdtemp(join(tmpdir(), "pipewright-actions-")); + rootPromise = created; + + return created; + }; + + const materialise = async ( + owner: string, + repo: string, + ref: string | undefined, + ): Promise => { + const directory = await rootOf(); + const target = join(directory, owner, repo, ref ?? "-"); + await mkdir(target, { recursive: true }); + await extractTarball(target, await fetchTarball(options, owner, repo, ref)); + + return target; + }; + + const resolver: ActionResolver = async (reference) => { + if (reference.kind === "local") { + return Promise.resolve(directoryTree(within(options.cwd, reference.path))); + } + const identity = `${reference.owner}/${reference.repo}@${reference.ref ?? "-"}`; + let directory = fetched.get(identity); + if (directory === undefined) { + directory = materialise(reference.owner, reference.repo, reference.ref); + fetched.set(identity, directory); + } + + return directoryTree(within(await directory, reference.subpath ?? ".")); + }; + + const node: NodeRunner = async () => + // The entry runs under the node the CLI itself runs under, which is the one runtime the host knows is present. + Promise.resolve({ command: process.execPath, args: [] }); + + const steps: StepHost = { + async expand(job, view): Promise { + const raw = job.native?.[PROVIDER]?.env; + // A run told to skip provider steps runs none of their actions, so it fetches nothing and places no phases; the job's own environment is still evaluated, because the steps that do run read it. + const uses = options.skipProviderSteps + ? [] + : job.steps.filter( + (step, index) => + usesOf(step) !== undefined && + (view.range === undefined || (index >= view.range.from && index <= view.range.to)), + ); + // A job that carries no action and no environment of GitHub's own prepares nothing and fetches nothing. + if (uses.length === 0 && raw === undefined) { + return { steps: job.steps, env: {} }; + } + const context = evaluationContextOf(view, github, options.secrets); + const env = stepEnvironment(raw, context, (construct) => { + throw new Error(`job ${JSON.stringify(job.name)} carries ${construct}`); + }); + const pre: Step[] = []; + const post: Step[] = []; + // The phases are placed in the order the steps name their actions, because that is the order the runner's own phases run them in. + const expansions = await uses.reduce>( + async (previous, step) => { + const placed = await previous; + const reference = usesOf(step); + // A reference the library cannot parse names no tree to read, so it has no phases to place; the step stays in place for resolution to reject or skip. + if (reference === undefined || unparsable(reference) !== undefined) return placed; + // Only the phases are kept: the steps the expansion produced are derived again when the step is reached, against the view that moment holds. + const expansion = await expandAction(step, { resolve: resolver, node, context }); + + return { + pre: [...placed.pre, ...expansion.pre], + post: [...placed.post, ...expansion.post], + }; + }, + Promise.resolve({ pre, post }), + ); + pre.push(...expansions.pre); + post.push(...expansions.post); + + return { steps: [...pre, ...job.steps, ...[...post].reverse()], env }; + }, + + async resolve(step, view): Promise { + if (step.kind === "provider" && options.skipProviderSteps) { + // The run was told its host has supplied the toolchain the provider's steps would install, so they are all skipped, an action's own included. + return { kind: "skipped" }; + } + if (step.kind === "provider" && step.provider !== PROVIDER) { + return options.skipProviderSteps + ? { kind: "skipped" } + : { kind: "unhandled", construct: `a step owned by ${step.provider}` }; + } + if (!isGitHubStep(step)) { + throw new Error("asked to resolve a step that is not one of GitHub's provider steps"); + } + const context = evaluationContextOf(view, github, options.secrets); + const condition = textOf(step.native, "if"); + if (condition !== undefined && !evaluateCondition(condition, context)) { + return { kind: "skipped" }; + } + const uses = textOf(step.native, "uses"); + if (uses !== undefined) { + const construct = unparsable(uses); + if (construct !== undefined) { + return options.skipProviderSteps + ? { kind: "skipped" } + : { kind: "unhandled", construct: `${uses}, ${construct}` }; + } + const expansion = await expandAction(step, { resolve: resolver, node, context }); + const prepared = expansion.steps.map((inner) => { + if (inner.kind !== "run") { + throw new Error("the expansion produced a step the runner does not run"); + } + + return { + name: inner.name ?? inner.run, + command: inner.run, + env: { ...inner.env }, + workingDirectory: inner.workingDirectory, + }; + }); + + return { kind: "steps", steps: prepared }; + } + const run = textOf(step.native, "run"); + if (run === undefined) { + return { + kind: "unhandled", + construct: "a provider step whose native carries neither run nor uses", + }; + } + const fail = (construct: string): never => { + throw new Error(`the step ${JSON.stringify(stepNameOf(step, run))} carries ${construct}`); + }; + const shell = textOf(step.native, "shell"); + if (shell !== undefined && !POSIX_SHELLS.has(shell)) { + return { + kind: "unhandled", + construct: `the shell "${shell}", which names an interpreter the host's own shell would not run as written`, + }; + } + for (const key of ["continue-on-error", "timeout-minutes"]) { + if (step.native[key] !== undefined) { + return { kind: "unhandled", construct: `${key}, which the model has no step form for` }; + } + } + const env = stepEnvironment(step.native.env, context, fail); + // The step's own environment is visible to the expressions of its command and condition, which is the view the library evaluates a composite step with. + const own = { ...context, env: { ...context.env, ...env } }; + const directory = textOf(step.native, "working-directory"); + const workingDirectory = + directory === undefined ? undefined : evaluateTemplate(directory, own); + const command = evaluateTemplate(run, own); + + return { + kind: "steps", + steps: [{ name: stepNameOf(step, command), command, env, workingDirectory }], + }; + }, + }; + + return { + resolver, + node, + steps, + async dispose() { + // A run that fetched nothing created no directory, so there is nothing to remove. + if (rootPromise !== undefined) { + await rm(await rootPromise, { recursive: true, force: true }); + } + }, + }; +} diff --git a/packages/cli/src/index.ts b/packages/cli/src/index.ts index 63629d6..1cacd27 100644 --- a/packages/cli/src/index.ts +++ b/packages/cli/src/index.ts @@ -1,2 +1,3 @@ +export { createActionHost, type ActionHost, type HostOptions } from "./actions.ts"; export { main, type Io } from "./main.ts"; export { resolveContext } from "./context.ts"; diff --git a/packages/cli/src/main.ts b/packages/cli/src/main.ts index 3cb353d..aa87734 100644 --- a/packages/cli/src/main.ts +++ b/packages/cli/src/main.ts @@ -43,6 +43,7 @@ import { createGitLabCiCompiler, createGitLabCiImporter } from "@pipewright/pipe import { createLocalRunner } from "@pipewright/pipeline-local"; import { createGitHubReader } from "@pipewright/platform-github"; +import { createActionHost } from "./actions.ts"; import { resolveContext } from "./context.ts"; /** @@ -88,7 +89,7 @@ Commands emit --pipeline --provider [--tool-command ] write the provider's native files check --pipeline --provider [--tool-command ] fail when a written file has drifted run --pipeline [--job ]... [--steps -] [--matrix =]... [--context =]... - [--local] [--skip-provider-steps] + [--secret [=]]... [--local] [--skip-provider-steps] execute the pipeline directly, no emitted files import --provider --file [--output ] [--format ts|json] [--schema ] read a provider's native file into a pipeline @@ -200,6 +201,34 @@ function contextOverrides( return overrides; } +/** + * The secrets a run was given, by name. `--secret name=value` supplies one outright and `--secret name` reads the environment variable of that name, so a secret's value never has to appear on a command line. The values are held for the run's lifetime, passed only to the expression evaluator, and never printed, logged or stored. + */ +function secretsOf( + entries: readonly string[] | undefined, + env: Readonly>, +): Record { + const secrets: Record = {}; + for (const entry of entries ?? []) { + const separator = entry.indexOf("="); + if (separator > 0) { + secrets[entry.slice(0, separator)] = entry.slice(separator + 1); + continue; + } + if (separator === -1 && /^[A-Za-z_][A-Za-z0-9_]*$/.test(entry)) { + const value = env[entry]; + if (value === undefined || value === "") { + throw new Error(`--secret names ${entry}, which the environment does not carry`); + } + secrets[entry] = value; + continue; + } + throw new Error(`--secret expects = or , got "${entry}"`); + } + + return secrets; +} + /** * The context of a developer running the pipeline on their own machine: a manual run on a branch called `local`. Explicit `--context` values still win, so a condition can be exercised against any other value. */ @@ -365,25 +394,42 @@ export async function main(argv: readonly string[], io: Io): Promise { steps: { type: "string" }, matrix: { type: "string", multiple: true }, context: { type: "string", multiple: true }, + secret: { type: "string", multiple: true }, local: { type: "boolean" }, "skip-provider-steps": { type: "boolean" }, }, }); const { pipeline } = await load(io, required(values.pipeline, "--pipeline")); const matrix: MatrixCombination = Object.fromEntries(keyValues(values.matrix, "--matrix")); - const report = await createLocalRunner().run(pipeline, { - executor: io.executor, - scratch: io.scratch, - output: cliRunOutput(io), - context: resolveContext(io.env, { - ...(values.local === true ? localContext(pipeline.name) : {}), - ...contextOverrides(values.context), - }), - ...(values.job === undefined ? {} : { only: values.job }), - ...(values["skip-provider-steps"] === true ? { skipProviderSteps: true } : {}), - ...(values.steps === undefined ? {} : { steps: parseRange(values.steps) }), - ...(Object.keys(matrix).length === 0 ? {} : { matrix }), + const context = resolveContext(io.env, { + ...(values.local === true ? localContext(pipeline.name) : {}), + ...contextOverrides(values.context), }); + const name = io.env.CI_ADAPTERS_TOKEN_VARIABLE ?? "GH_TOKEN"; + const host = createActionHost({ + cwd: io.cwd, + fetch: io.fetch, + context, + token: io.env[name], + secrets: secretsOf(values.secret, io.env), + skipProviderSteps: values["skip-provider-steps"] === true, + }); + let report; + try { + report = await createLocalRunner().run(pipeline, { + executor: io.executor, + scratch: io.scratch, + output: cliRunOutput(io), + context, + host: host.steps, + ...(values.job === undefined ? {} : { only: values.job }), + ...(values["skip-provider-steps"] === true ? { skipProviderSteps: true } : {}), + ...(values.steps === undefined ? {} : { steps: parseRange(values.steps) }), + ...(Object.keys(matrix).length === 0 ? {} : { matrix }), + }); + } finally { + await host.dispose(); + } for (const job of report.jobs) { const label = Object.keys(job.matrix).length === 0 ? "" : ` ${JSON.stringify(job.matrix)}`; diff --git a/packages/cli/test/fixtures/host-actions/actions/composite/action.yml b/packages/cli/test/fixtures/host-actions/actions/composite/action.yml new file mode 100644 index 0000000..07a0c1d --- /dev/null +++ b/packages/cli/test/fixtures/host-actions/actions/composite/action.yml @@ -0,0 +1,18 @@ +name: Composite +description: Writes what it was handed, the way an imported composite action runs on the host +inputs: + message: + description: The text to carry + required: true +runs: + using: composite + steps: + - name: Record the input + run: | + echo "input=${{ inputs.message }}" >> composite.txt + echo "ref=${{ github.ref_name }}" >> composite.txt + shell: bash + - name: Release only + if: inputs.message == 'release' + run: echo "release" >> composite.txt + shell: bash diff --git a/packages/cli/test/fixtures/host-actions/actions/node/action.yml b/packages/cli/test/fixtures/host-actions/actions/node/action.yml new file mode 100644 index 0000000..5d8a079 --- /dev/null +++ b/packages/cli/test/fixtures/host-actions/actions/node/action.yml @@ -0,0 +1,9 @@ +name: Write +description: Writes the file a node action was handed, the way an imported node action runs on the host +inputs: + text: + description: The text to write + required: true +runs: + using: node20 + main: index.js diff --git a/packages/cli/test/fixtures/host-actions/actions/node/index.js b/packages/cli/test/fixtures/host-actions/actions/node/index.js new file mode 100644 index 0000000..97323d4 --- /dev/null +++ b/packages/cli/test/fixtures/host-actions/actions/node/index.js @@ -0,0 +1,5 @@ +// Writes the effect a host run is asserted on, so the test checks what the action really received. +const fs = require("node:fs"); +const text = process.env.INPUT_TEXT; +if (text === undefined) throw new Error("INPUT_TEXT was not supplied"); +fs.writeFileSync("node.txt", `${text}\n`); diff --git a/packages/cli/test/fixtures/host-actions/workflow.yml b/packages/cli/test/fixtures/host-actions/workflow.yml new file mode 100644 index 0000000..f9d192a --- /dev/null +++ b/packages/cli/test/fixtures/host-actions/workflow.yml @@ -0,0 +1,27 @@ +name: Host actions +on: push +jobs: + build: + runs-on: ubuntu-latest + steps: + - name: Generate + id: generate + run: | + echo "flag=yes" >> "$GITHUB_OUTPUT" + echo "token-present=${{ secrets.API_TOKEN != '' }}" >> "$GITHUB_OUTPUT" + - name: Use the composite action + uses: ./actions/composite + with: + message: from-${{ steps.generate.outputs.flag }} + - name: Use the node action + uses: ./actions/node + with: + text: wrote-${{ github.ref_name }} + - name: Record the hand-off + if: steps.generate.outputs.flag == 'yes' + run: echo "hand-off=${{ steps.generate.outputs.flag }} token=${{ steps.generate.outputs.token-present }}" >> hand-off.txt + - name: Never taken + if: steps.generate.outputs.flag == 'no' + run: | + touch never.txt + exit 1 diff --git a/packages/cli/test/host-actions.integration.test.ts b/packages/cli/test/host-actions.integration.test.ts new file mode 100644 index 0000000..ee1c4fb --- /dev/null +++ b/packages/cli/test/host-actions.integration.test.ts @@ -0,0 +1,41 @@ +import { mkdtemp } from "node:fs/promises"; +import { tmpdir } from "node:os"; +import { join } from "node:path"; + +import { describe, expect, it } from "vitest"; + +import { CHECKOUT_ACTION, nodeActionSchema } from "@pipewright/pipeline-github-actions"; +import { parse } from "yaml"; + +import { createActionHost } from "../src/actions.ts"; + +/** + * Resolves the pinned checkout action the adapter's own pipelines use, over the network, through the same fetching resolver a `run` supplies: the reference is fetched as a tarball, extracted, and its metadata reads as the node action it is. The token goes along when the environment holds one, so a private clone of this repository resolves it too. + */ +describe("a fetched action over the network", () => { + it("resolves the pinned checkout action and reads its metadata and entry", async () => { + const host = createActionHost({ + cwd: await mkdtemp(join(tmpdir(), "pipewright-fetch-")), + fetch, + context: {}, + token: process.env.GH_TOKEN, + secrets: {}, + skipProviderSteps: false, + }); + try { + const [locator, ref] = CHECKOUT_ACTION.split("@"); + const [owner, repo] = locator?.split("/") ?? []; + if (owner === undefined || repo === undefined || ref === undefined) { + throw new Error(`the pinned checkout action is not owner/repo@ref: ${CHECKOUT_ACTION}`); + } + const tree = await host.resolver({ kind: "remote", owner, repo, subpath: undefined, ref }); + const metadata = await tree.read("action.yml"); + if (metadata === undefined) throw new Error("the checkout action holds no action.yml"); + const action = nodeActionSchema.parse(parse(metadata)); + const entry = await tree.read(action.runs.main); + expect(entry).toBeDefined(); + } finally { + await host.dispose(); + } + }); +}); diff --git a/packages/cli/test/host-actions.unit.test.ts b/packages/cli/test/host-actions.unit.test.ts new file mode 100644 index 0000000..3868db7 --- /dev/null +++ b/packages/cli/test/host-actions.unit.test.ts @@ -0,0 +1,529 @@ +import { spawn } from "node:child_process"; +import { cp, mkdir, mkdtemp, readFile, rm, writeFile } from "node:fs/promises"; +import { tmpdir } from "node:os"; +import { dirname, join, resolve } from "node:path"; +import { fileURLToPath } from "node:url"; +import { gzipSync } from "node:zlib"; + +import { CONTEXT_EXPRESSION } from "@pipewright/pipeline-github-actions"; +import type { + CommandExecution, + CommandExecutor, + CommandResult, + StepCommandFiles, + StepFileContents, + StepScratch, +} from "@pipewright/pipeline-contract"; +import { afterAll, describe, expect, it } from "vitest"; + +import { GITHUB_FIELD } from "../src/actions.ts"; +import { main, type Io } from "../src/index.ts"; + +/** The schema reference a JSON pipeline document must declare. */ +const SCHEMA = "https://unpkg.com/@pipewright/pipeline-contract/dist/pipeline.schema.json"; + +/** Where the demonstration fixtures live, beside this test file. */ +const FIXTURES = resolve(dirname(fileURLToPath(import.meta.url)), "fixtures/host-actions"); + +/** The workspaces and scratch roots this test created, removed once every case has run. */ +const created: string[] = []; + +afterAll(async () => { + await Promise.all(created.map(async (path) => rm(path, { recursive: true, force: true }))); +}); + +/** A fresh directory under the system temporary directory, tracked for removal when the test ends. */ +async function directory(): Promise { + const path = await mkdtemp(join(tmpdir(), "pipewright-host-")); + created.push(path); + + return path; +} + +/** The executor the real CLI uses: each command through `sh -e -c`, both streams read back. */ +function executor(cwd: string): CommandExecutor { + return { + async execute(execution: CommandExecution, signal?: AbortSignal): Promise { + return new Promise((done, fail) => { + const child = spawn("sh", ["-e", "-c", execution.command], { + stdio: ["inherit", "pipe", "pipe"], + cwd: + execution.workingDirectory === undefined + ? cwd + : resolve(cwd, execution.workingDirectory), + env: { + ...process.env, + ...execution.env, + PATH: [execution.pathEntries.join(":"), process.env.PATH] + .filter((entry): entry is string => entry !== undefined) + .join(":"), + }, + ...(signal === undefined ? {} : { signal }), + }); + child.stdout.setEncoding("utf8"); + child.stderr.setEncoding("utf8"); + let stdout = ""; + let stderr = ""; + child.stdout.on("data", (chunk: string) => { + stdout += chunk; + }); + child.stderr.on("data", (chunk: string) => { + stderr += chunk; + }); + child.on("error", fail); + child.on("close", (code) => { + done({ exitCode: code ?? 1, stdout, stderr }); + }); + }); + }, + }; +} + +/** The scratch the real CLI uses: one directory per run, one empty file per command, removed once read. */ +async function scratch(): Promise { + const root = await directory(); + let stepCount = 0; + + return { + async create(): Promise { + const step = join(root, String(stepCount)); + stepCount += 1; + await mkdir(step, { recursive: true }); + const files = { + output: join(step, "output"), + env: join(step, "env"), + path: join(step, "path"), + summary: join(step, "summary"), + }; + await Promise.all([ + writeFile(files.output, ""), + writeFile(files.env, ""), + writeFile(files.path, ""), + writeFile(files.summary, ""), + ]); + + return files; + }, + async read(files): Promise { + return { + output: await readFile(files.output, "utf8"), + env: await readFile(files.env, "utf8"), + path: await readFile(files.path, "utf8"), + summary: await readFile(files.summary, "utf8"), + }; + }, + async release(files) { + await rm(dirname(files.output), { recursive: true, force: true }); + }, + async dispose() { + await rm(root, { recursive: true, force: true }); + }, + }; +} + +interface Harness { + readonly io: Io; + readonly out: () => string; + readonly err: () => string; + readonly workspace: string; +} + +/** A run of the real CLI over a real workspace: the fixture tree copied to a fresh directory, a real executor and scratch, output collected. */ +async function harness(fetch: typeof globalThis.fetch = globalThis.fetch): Promise { + const workspace = await directory(); + await cp(FIXTURES, workspace, { recursive: true }); + const out: string[] = []; + const err: string[] = []; + + return { + workspace, + out: () => out.join(""), + err: () => err.join(""), + io: { + cwd: workspace, + env: {}, + stdout(text) { + out.push(text); + }, + stderr(text) { + err.push(text); + }, + reader: { read: async (path) => readFile(resolve(workspace, path), "utf8") }, + readOptional: async (path) => { + try { + return await readFile(resolve(workspace, path), "utf8"); + } catch (error) { + if (error instanceof Error && "code" in error && error.code === "ENOENT") { + return undefined; + } + throw error; + } + }, + async writeFile(path, content) { + const target = resolve(workspace, path); + await mkdir(dirname(target), { recursive: true }); + await writeFile(target, content); + }, + scratch: await scratch(), + executor: executor(workspace), + importModule: async () => Promise.reject(new Error("this test loads no TypeScript module")), + fetch, + }, + }; +} + +/** A pipeline document authored directly, written into the workspace as JSON. */ +async function authored(harness_: Harness, document: unknown, name: string): Promise { + const path = join(harness_.workspace, name); + await writeFile(path, `${JSON.stringify(document, null, 2)}\n`); + + return name; +} + +describe("run through the real CLI", () => { + it("imports a workflow and runs its actions end to end on the host", async () => { + const run = await harness(); + const imported = await main( + [ + "import", + "--provider", + "github-actions", + "--file", + "workflow.yml", + "--output", + "ci.pipeline.json", + "--format", + "json", + ], + run.io, + ); + expect(imported).toBe(0); + const code = await main( + [ + "run", + "--pipeline", + "ci.pipeline.json", + "--local", + "--secret", + "API_TOKEN=demo-secret-value", + ], + run.io, + ); + expect(code).toBe(0); + expect(run.out()).toContain("success: build"); + // The composite action's steps ran with the input mapped from the workflow's own step output and the run's context. + await expect(readFile(join(run.workspace, "composite.txt"), "utf8")).resolves.toBe( + "input=from-yes\nref=local\n", + ); + // The node action's entry ran under the host's node with its input in the environment. + await expect(readFile(join(run.workspace, "node.txt"), "utf8")).resolves.toBe("wrote-local\n"); + // The value one step wrote through GITHUB_OUTPUT reached a later step's expression, and the secret it read never appeared. + await expect(readFile(join(run.workspace, "hand-off.txt"), "utf8")).resolves.toBe( + "hand-off=yes token=true\n", + ); + await expect(readFile(join(run.workspace, "never.txt"), "utf8")).rejects.toThrow(); + expect(run.out()).not.toContain("demo-secret-value"); + }); + + it("fails the run naming a secret it was not given", async () => { + const run = await harness(); + await main( + [ + "import", + "--provider", + "github-actions", + "--file", + "workflow.yml", + "--output", + "ci.pipeline.json", + "--format", + "json", + ], + run.io, + ); + const code = await main(["run", "--pipeline", "ci.pipeline.json", "--local"], run.io); + expect(code).toBe(1); + expect(run.err()).toContain("API_TOKEN"); + expect(run.err()).not.toContain("demo-secret-value"); + }); + + it("leaves a container reference to the provider, skipped only when asked", async () => { + const run = await harness(); + const pipeline = await authored( + run, + { + $schema: SCHEMA, + name: "Container", + triggers: [{ kind: "push" }], + jobs: { + build: { + name: "Build", + runsOn: ["ubuntu-latest"], + steps: [ + { kind: "run", run: "echo ran >> ran.txt" }, + { + kind: "provider", + provider: "github-actions", + native: { uses: "docker://alpine:3.20" }, + }, + ], + }, + }, + }, + "container.pipeline.json", + ); + const rejected = await main(["run", "--pipeline", pipeline, "--local"], run.io); + expect(rejected).toBe(1); + expect(run.err()).toContain("docker://alpine:3.20"); + expect(run.err()).toContain("container runtime"); + // The step before the container reference ran before the run rejected on it. + await expect(readFile(join(run.workspace, "ran.txt"), "utf8")).resolves.toBe("ran\n"); + const skipped = await main( + ["run", "--pipeline", pipeline, "--local", "--skip-provider-steps"], + run.io, + ); + expect(skipped).toBe(0); + await expect(readFile(join(run.workspace, "ran.txt"), "utf8")).resolves.toBe("ran\nran\n"); + }); + + it("rejects a workflow step whose shell the host would not run as written", async () => { + const run = await harness(); + const pipeline = await authored( + run, + { + $schema: SCHEMA, + name: "Shell", + triggers: [{ kind: "push" }], + jobs: { + build: { + name: "Build", + runsOn: ["ubuntu-latest"], + steps: [ + { + kind: "provider", + provider: "github-actions", + native: { run: "Write-Output done", shell: "pwsh" }, + }, + ], + }, + }, + }, + "shell.pipeline.json", + ); + const code = await main(["run", "--pipeline", pipeline, "--local"], run.io); + expect(code).toBe(1); + expect(run.err()).toContain("pwsh"); + }); +}); + +/** The tar header's field offsets, in bytes, as the format fixes them; the reader this feed exercises reads the same fields. */ +const HEADER = 512; +const NAME_LIMIT = 100; +const MODE_AT = 100; +const OWNER_AT = 108; +const GROUP_AT = 116; +const SIZE_AT = 124; +const SIZE_DIGITS = 11; +const TIME_AT = 136; +const CHECK_AT = 148; +const CHECK_DIGITS = 6; +const TYPE_AT = 156; +const MAGIC_AT = 257; +const VERSION_AT = 264; +/** The octal radix the format's numeric fields are written in. */ +const OCTAL = 8; + +/** One tar entry: a header block of the given type flag, an optional name, and the data, padded to a full block. */ +function tarEntry(typeflag: string, data: string, name = ""): number[] { + const header = Buffer.alloc(HEADER); + header.write(name, 0, "ascii"); + header.write("0000644", MODE_AT, "ascii"); + header.write("0000000", OWNER_AT, "ascii"); + header.write("0000000", GROUP_AT, "ascii"); + header.write(data.length.toString(OCTAL).padStart(SIZE_DIGITS, "0"), SIZE_AT, "ascii"); + header.write("00000000000", TIME_AT, "ascii"); + header.fill(" ", CHECK_AT, CHECK_AT + CHECK_DIGITS + 2); + header.write(typeflag, TYPE_AT, "ascii"); + header.write("ustar", MAGIC_AT, "ascii"); + header.write("00", VERSION_AT, "ascii"); + const sum = header.reduce((total, byte) => total + byte, 0); + header.write(`${sum.toString(OCTAL).padStart(CHECK_DIGITS, "0")}\0 `, CHECK_AT, "ascii"); + const bytes = [...Buffer.from(data, "utf8")]; + const padding = (HEADER - (bytes.length % HEADER)) % HEADER; + + return [...header, ...bytes, ...new Array(padding).fill(0)]; +} + +/** One pax header entry: `x` for the entry that follows it, `g` for the whole archive, over the records given. */ +function paxEntry(typeflag: "x" | "g", records: Readonly>): number[] { + const data = Object.entries(records) + .map(([key, value]) => { + const record = `${key}=${value}\n`; + // A record's length prefix counts the digits and the space it is written with, so the length is found by widening until it is stable. + let digits = 1; + for (;;) { + const length = String(record.length + digits + 1).length; + if (length === digits) break; + digits = length; + } + + return `${String(record.length + digits + 1)} ${record}`; + }) + .join(""); + + return tarEntry(typeflag, data, typeflag === "g" ? "pax_global_header" : ""); +} + +/** A repository tarball over the files given, in the shape GitHub serves one: a pax global header first, a path too long for the header's own field carried in an extended header, gzip over the whole. */ +function tarballOf(files: Readonly>): Uint8Array { + const entries = [ + paxEntry("g", { comment: "1234567890abcdef1234567890abcdef12345678" }), + ...Object.entries(files).flatMap(([name, content]) => [ + ...(name.length > NAME_LIMIT ? [paxEntry("x", { path: name })] : []), + tarEntry("0", content, name.length > NAME_LIMIT ? "" : name), + ]), + ].flat(); + + return gzipSync(new Uint8Array(entries)); +} + +describe("a fetched action", () => { + /** How many directory segments the long-path fixture nests, past the header's hundred-byte name field. */ + const PATH_DEPTH = 25; + /** A path past the tar header's name field, carried in a pax extended header as a real tarball does. */ + const LONG_PATH = `phased-v1/${"docs/".repeat(PATH_DEPTH)}notes.md`; + + /** A node action whose phases write one file each, so the order the run placed them in is the file's order. */ + const ACTION: Readonly> = { + "phased-v1/action.yml": `name: Phased +description: Runs around the job's own steps +inputs: + text: + description: The text to carry + required: true +runs: + using: node20 + main: index.js + pre: pre.js + post: post.js +`, + "phased-v1/index.js": `const fs = require("node:fs"); +const path = require("node:path"); +const text = process.env.INPUT_TEXT; +if (text === undefined) throw new Error("INPUT_TEXT was not supplied"); +fs.appendFileSync("phases.txt", \`main=\${text}\\n\`); +const deep = Array(25).fill("docs").join("/") + "/notes.md"; +const carried = fs.readFileSync(path.join(process.env.GITHUB_ACTION_PATH, deep), "utf8"); +fs.appendFileSync("phases.txt", \`long=\${carried.trim()}\\n\`); +`, + "phased-v1/pre.js": `const fs = require("node:fs"); +fs.appendFileSync("phases.txt", "pre\\n"); +`, + "phased-v1/post.js": `const fs = require("node:fs"); +fs.appendFileSync("phases.txt", "post\\n"); +`, + [LONG_PATH]: "a long path\n", + }; + // A path past the header's name field, so the tarball carries it in a pax extended header as a real one does. + + it("is fetched once, extracted, and its phases placed around the job's steps", async () => { + const tarball = tarballOf(ACTION); + const fetched: string[] = []; + const serving: typeof globalThis.fetch = async (input) => { + const url = typeof input === "string" ? input : input instanceof URL ? input.href : input.url; + fetched.push(url); + + return Promise.resolve( + new Response(tarball, { + status: 200, + headers: { "content-type": "application/x-gzip" }, + }), + ); + }; + const run = await harness(serving); + const pipeline = await authored( + run, + { + $schema: SCHEMA, + name: "Phased", + triggers: [{ kind: "push" }], + jobs: { + build: { + name: "Build", + runsOn: ["ubuntu-latest"], + steps: [ + { kind: "run", run: `echo "before" >> phases.txt` }, + { + kind: "provider", + provider: "github-actions", + native: { uses: "owner/repo@v1", with: { text: "carried" } }, + }, + { kind: "run", run: `echo "after" >> phases.txt` }, + ], + }, + }, + }, + "phased.pipeline.json", + ); + const code = await main(["run", "--pipeline", pipeline, "--local"], run.io); + if (code !== 0) throw new Error(run.err()); + expect(code).toBe(0); + expect(fetched).toEqual(["https://api.github.com/repos/owner/repo/tarball/v1"]); + // The entry read the long-path file out of its own tree, so the pax header that carried the path was extracted with it. + await expect(readFile(join(run.workspace, "phases.txt"), "utf8")).resolves.toBe( + "pre\nbefore\nmain=carried\nlong=a long path\nafter\npost\n", + ); + }); + + it("skips the action entirely when the run was told to skip provider steps", async () => { + const fetched: string[] = []; + const serving: typeof globalThis.fetch = async (input) => { + const url = typeof input === "string" ? input : input instanceof URL ? input.href : input.url; + fetched.push(url); + + return Promise.reject(new Error("a skipped run fetches nothing")); + }; + const run = await harness(serving); + const pipeline = await authored( + run, + { + $schema: SCHEMA, + name: "Skipped", + triggers: [{ kind: "push" }], + jobs: { + build: { + name: "Build", + runsOn: ["ubuntu-latest"], + steps: [ + { kind: "run", run: "echo before >> skipped.txt" }, + { + kind: "provider", + provider: "github-actions", + native: { uses: "owner/repo@v1", with: { text: "carried" } }, + }, + ], + }, + }, + }, + "skipped.pipeline.json", + ); + const code = await main( + ["run", "--pipeline", pipeline, "--local", "--skip-provider-steps"], + run.io, + ); + expect(code).toBe(0); + expect(fetched).toEqual([]); + await expect(readFile(join(run.workspace, "skipped.txt"), "utf8")).resolves.toBe("before\n"); + await expect(readFile(join(run.workspace, "phases.txt"), "utf8")).rejects.toThrow(); + }); +}); + +describe("the host's context table", () => { + it("maps each neutral key onto the field the adapter's own expression table names", () => { + for (const [key, expression] of Object.entries(CONTEXT_EXPRESSION)) { + expect(GITHUB_FIELD[key as keyof typeof GITHUB_FIELD]).toBe( + expression.slice("github.".length), + ); + } + }); +}); diff --git a/packages/pipeline-contract/src/index.ts b/packages/pipeline-contract/src/index.ts index 3c0dab0..2ef1d81 100644 --- a/packages/pipeline-contract/src/index.ts +++ b/packages/pipeline-contract/src/index.ts @@ -69,20 +69,27 @@ export { type CommandExecution, type CommandExecutor, type CommandResult, + type JobExpansion, type JobReport, + type JobView, type LogEntry, type LogLevel, type Outcome, type PipelineRunner, + type PreparedStep, type RunContext, type RunOptions, type RunOutput, type RunReport, type StepCommandFiles, type StepFileContents, + type StepHost, + type StepRecord, type StepReport, + type StepResolution, type StepScratch, type StepSummary, + type StepView, } from "./runner.ts"; export { describeRunnerConformance } from "./runner-conformance.ts"; export type { PredicateNode } from "trilean"; diff --git a/packages/pipeline-contract/src/runner-conformance.ts b/packages/pipeline-contract/src/runner-conformance.ts index 9245a45..2a8e1b7 100644 --- a/packages/pipeline-contract/src/runner-conformance.ts +++ b/packages/pipeline-contract/src/runner-conformance.ts @@ -11,10 +11,11 @@ import type { StepScratch, StepSummary, } from "./runner.ts"; +import { describeStepHostConformance } from "./runner-host-conformance.ts"; import { pipelineSchema, type Pipeline } from "./schema.ts"; import { samplePipeline, type TestApi } from "./conformance.ts"; -const CONTEXT: RunContext = { +export const CONTEXT: RunContext = { event: "push", refName: "main", baseRefName: "", @@ -32,7 +33,7 @@ const FILE_VARIABLES: Readonly> = { summary: "GITHUB_STEP_SUMMARY", }; -interface RecordingExecutor extends CommandExecutor { +export interface RecordingExecutor extends CommandExecutor { readonly executed: CommandExecution[]; } @@ -51,7 +52,7 @@ interface ExecutorBehaviour { readonly script?: (execution: CommandExecution) => StepScript; } -function recordingExecutor( +export function recordingExecutor( scratch: MemoryScratch, behaviour: ExecutorBehaviour = {}, ): RecordingExecutor { @@ -81,12 +82,12 @@ function recordingExecutor( }; } -interface MemoryScratch extends StepScratch { +export interface MemoryScratch extends StepScratch { /** The scratch's store, keyed by path, so the scripted executor can stand in for the command that writes its files. */ readonly written: Map; } -function memoryScratch(): MemoryScratch { +export function memoryScratch(): MemoryScratch { const written = new Map(); let stepCount = 0; @@ -136,12 +137,12 @@ function memoryScratch(): MemoryScratch { }; } -interface RecordingOutput extends RunOutput { +export interface RecordingOutput extends RunOutput { readonly entries: LogEntry[]; readonly summaries: StepSummary[]; } -function recordingOutput(): RecordingOutput { +export function recordingOutput(): RecordingOutput { const entries: LogEntry[] = []; const summaries: StepSummary[] = []; @@ -158,11 +159,11 @@ function recordingOutput(): RecordingOutput { } /** The ports a run needs besides its executor, fresh for each case so no state leaks between them. */ -function ports(): { scratch: MemoryScratch; output: RecordingOutput } { +export function ports(): { scratch: MemoryScratch; output: RecordingOutput } { return { scratch: memoryScratch(), output: recordingOutput() }; } -function check(condition: boolean, message: string): void { +export function check(condition: boolean, message: string): void { if (!condition) throw new Error(message); } @@ -170,6 +171,7 @@ function check(condition: boolean, message: string): void { * Registers the behavioural spec every `PipelineRunner` must meet, driven through a recording executor so no process is spawned. */ export function describeRunnerConformance(api: TestApi, makeRunner: () => PipelineRunner): void { + describeStepHostConformance(api, makeRunner); api.describe("PipelineRunner conformance", () => { api.it("runs jobs in needs order", async () => { const { scratch, output } = ports(); diff --git a/packages/pipeline-contract/src/runner-host-conformance.ts b/packages/pipeline-contract/src/runner-host-conformance.ts new file mode 100644 index 0000000..155b1f1 --- /dev/null +++ b/packages/pipeline-contract/src/runner-host-conformance.ts @@ -0,0 +1,313 @@ +import { UnsupportedFeatureError } from "./errors.ts"; +import { pipelineSchema, type Pipeline } from "./schema.ts"; +import type { TestApi } from "./conformance.ts"; +import type { PipelineRunner, PreparedStep, StepHost, StepView } from "./runner.ts"; +import type { Step } from "./schema.ts"; + +import { CONTEXT, check, ports, recordingExecutor } from "./runner-conformance.ts"; + +/** + * Registers the behavioural spec every `PipelineRunner` must meet for the step host port, driven through a recording executor so no process is spawned. The port's two moments are both covered: preparation, which places the host's steps around the job's own, and resolution, which decides what runs where a provider step stood. + */ +export function describeStepHostConformance(api: TestApi, makeRunner: () => PipelineRunner): void { + api.describe("PipelineRunner step host", () => { + /** A pipeline whose one job holds a provider step between two run steps, the smallest shape over which a host's placement and resolution are both exercised. */ + const hostedPipeline = (): Pipeline => + pipelineSchema.parse({ + name: "CI", + triggers: [{ kind: "push" }], + jobs: { + build: { + name: "Build", + runsOn: ["x"], + steps: [ + { kind: "run", run: "before" }, + { + kind: "provider", + provider: "github-actions", + native: { uses: "example/action@v1" }, + }, + { kind: "run", run: "after" }, + ], + }, + }, + }); + + /** A prepared step with a distinct command, so the order the runner ran them in is the order of the commands. */ + const prepared = (command: string): PreparedStep => ({ + name: command, + command, + env: {}, + workingDirectory: undefined, + }); + + api.it( + "places a host's steps around the job's own and its environment under them", + async () => { + const { scratch, output } = ports(); + const executor = recordingExecutor(scratch); + const host: StepHost = { + async expand(job) { + return Promise.resolve({ + steps: [{ kind: "run", run: "pre" }, ...job.steps, { kind: "run", run: "post" }], + env: { HOSTED: "yes" }, + }); + }, + async resolve() { + return Promise.resolve({ kind: "steps", steps: [prepared("resolved")] }); + }, + }; + await makeRunner().run(hostedPipeline(), { + executor, + scratch, + output, + context: CONTEXT, + host, + }); + const commands = executor.executed.map((entry) => entry.command); + check(commands.join(",") === "pre,before,resolved,after,post", `ran ${commands.join(",")}`); + const resolved = executor.executed.find((entry) => entry.command === "resolved"); + check(resolved?.env.HOSTED === "yes", "the host's job environment did not reach its steps"); + }, + ); + + api.it("gives each prepared step its own report entry and file commands", async () => { + const { scratch, output } = ports(); + const executor = recordingExecutor(scratch, { + script: (execution) => ({ + stdout: "", + writes: execution.command === "inner one" ? { output: "inner=1\n" } : {}, + }), + }); + const host: StepHost = { + async expand(job) { + return Promise.resolve({ steps: job.steps, env: {} }); + }, + async resolve() { + return Promise.resolve({ + kind: "steps", + steps: [prepared("inner one"), prepared("inner two")], + }); + }, + }; + const report = await makeRunner().run(hostedPipeline(), { + executor, + scratch, + output, + context: CONTEXT, + host, + }); + const names = report.jobs[0]?.steps.map((step) => step.name); + check( + names !== undefined && names.includes("inner one") && names.includes("inner two"), + `prepared steps were reported as ${JSON.stringify(names)}`, + ); + const inner = report.jobs[0]?.steps.find((step) => step.name === "inner one"); + check(inner?.outputs.inner === "1", "a prepared step's outputs were not recorded"); + }); + + api.it("stops a resolved step's commands at the first failure", async () => { + const { scratch, output } = ports(); + const executor = recordingExecutor(scratch, { + failing: (command) => command === "inner one", + }); + const host: StepHost = { + async expand(job) { + return Promise.resolve({ steps: job.steps, env: {} }); + }, + async resolve() { + return Promise.resolve({ + kind: "steps", + steps: [prepared("inner one"), prepared("inner two")], + }); + }, + }; + const report = await makeRunner().run(hostedPipeline(), { + executor, + scratch, + output, + context: CONTEXT, + host, + }); + check(!report.ok, "a failing prepared step was reported as ok"); + check( + !executor.executed.some((entry) => entry.command === "inner two"), + "the command after a failing one still ran", + ); + check( + !executor.executed.some((entry) => entry.command === "after"), + "a step after the failed provider step still ran", + ); + }); + + api.it("resolves against what the job's earlier steps produced", async () => { + const { scratch, output } = ports(); + const executor = recordingExecutor(scratch, { + script: (execution) => ({ + stdout: "", + writes: + execution.command === "before" ? { output: "flag=held\n", env: "SET=here\n" } : {}, + }), + }); + let seen: StepView | undefined; + const host: StepHost = { + async expand(job) { + return Promise.resolve({ steps: job.steps, env: {} }); + }, + async resolve(_step, view) { + seen = view; + + return Promise.resolve({ kind: "steps", steps: [prepared("resolved")] }); + }, + }; + await makeRunner().run(hostedPipeline(), { + executor, + scratch, + output, + context: CONTEXT, + host, + }); + const before = seen?.steps.find( + (record) => record.step.kind === "run" && record.step.run === "before", + ); + check( + before?.outputs.flag === "held", + `the resolver saw ${JSON.stringify(seen?.steps.map((record) => record.outputs))}`, + ); + check(seen?.env.SET === "here", "the resolver did not see the written environment"); + const resolved = executor.executed.find((entry) => entry.command === "resolved"); + check(resolved?.env.SET === "here", "a resolved step did not run with the written entry"); + }); + + api.it("reports a provider step the host skips, and asks once per step", async () => { + const { scratch, output } = ports(); + let asked = 0; + const host: StepHost = { + async expand(job) { + return Promise.resolve({ steps: job.steps, env: {} }); + }, + async resolve() { + asked += 1; + + return Promise.resolve({ kind: "skipped" }); + }, + }; + const report = await makeRunner().run(hostedPipeline(), { + executor: recordingExecutor(scratch), + scratch, + output, + context: CONTEXT, + host, + }); + check(asked === 1, `the host was asked ${String(asked)} times`); + const reported = report.jobs[0]?.steps.find( + (step) => step.outcome === "skipped" && step.name === "provider", + ); + check(reported !== undefined, "the skipped provider step was not reported"); + }); + + api.it("does not prepare a job that does not run", async () => { + const { scratch, output } = ports(); + const pipeline = hostedPipeline(); + const build = pipeline.jobs.build; + if (build === undefined) throw new Error("sample pipeline lost its build job"); + build.when = { + kind: "textCompare", + op: "equals", + left: { kind: "reference", key: "refName" }, + right: { kind: "textLiteral", value: "release" }, + }; + let preparedCount = 0; + const host: StepHost = { + async expand(job) { + preparedCount += 1; + + return Promise.resolve({ steps: job.steps, env: {} }); + }, + async resolve() { + return Promise.resolve({ kind: "skipped" }); + }, + }; + await makeRunner().run(pipeline, { + executor: recordingExecutor(scratch), + scratch, + output, + context: CONTEXT, + host, + }); + check(preparedCount === 0, "a job that did not run was prepared"); + }); + + api.it("places no phases for steps outside the range the run was given", async () => { + const { scratch, output } = ports(); + const executor = recordingExecutor(scratch); + let expanded: number | undefined; + const host: StepHost = { + async expand(job, view) { + // A host resolves the actions of the steps that will run, which is what the range in the view states. + expanded = view.range === undefined ? undefined : view.range.from; + const within = job.steps.filter( + (step, index) => + view.range === undefined || (index >= view.range.from && index <= view.range.to), + ); + + return Promise.resolve({ + steps: [ + ...within.flatMap((step): Step[] => + step.kind === "provider" ? [{ kind: "run", run: "pre" }] : [], + ), + ...job.steps, + ], + env: {}, + }); + }, + async resolve() { + return Promise.resolve({ kind: "skipped" }); + }, + }; + await makeRunner().run(hostedPipeline(), { + executor, + scratch, + output, + context: CONTEXT, + host, + only: ["build"], + steps: { from: 1, to: 1 }, + }); + check(expanded === 1, `the host was told the range starts at ${String(expanded)}`); + const commands = executor.executed.map((entry) => entry.command); + check( + !commands.includes("pre") && commands.includes("before"), + `a step outside the range contributed a phase: ${commands.join(",")}`, + ); + }); + + api.it("rejects a provider step the host does not handle", async () => { + const { scratch, output } = ports(); + const host: StepHost = { + async expand(job) { + return Promise.resolve({ steps: job.steps, env: {} }); + }, + async resolve() { + return Promise.resolve({ kind: "unhandled", construct: "a container image" }); + }, + }; + let thrown: unknown; + try { + await makeRunner().run(hostedPipeline(), { + executor: recordingExecutor(scratch), + scratch, + output, + context: CONTEXT, + host, + }); + } catch (error) { + thrown = error; + } + check( + thrown instanceof UnsupportedFeatureError && thrown.feature.includes("container image"), + `the unhandled step was rejected with ${String(thrown)}`, + ); + }); + }); +} diff --git a/packages/pipeline-contract/src/runner.ts b/packages/pipeline-contract/src/runner.ts index e2b0dbd..fb57804 100644 --- a/packages/pipeline-contract/src/runner.ts +++ b/packages/pipeline-contract/src/runner.ts @@ -1,6 +1,6 @@ import type { ContextKey } from "./context.ts"; import type { MatrixCombination } from "./matrix.ts"; -import type { Pipeline } from "./schema.ts"; +import type { Job, Pipeline, Step } from "./schema.ts"; /** * One command a runner asks its host to run. The runner never spawns processes itself, so the same runner executes in a CI job, on a laptop or under test. @@ -88,6 +88,69 @@ export interface CommandExecutor { execute: (command: CommandExecution, signal?: AbortSignal) => Promise; } +/** + * What one step of the job produced, kept beside the step itself so a host can read the identity a provider gave it, which the report alone no longer carries once a provider step can resolve to several. + */ +export interface StepRecord { + readonly step: Step; + readonly outcome: Outcome; + /** The outputs the step declared through its output file; for a step a host resolved into several, every output they wrote. */ + readonly outputs: Readonly>; +} + +/** The run's state a job's steps are prepared against, before any of them has run. */ +export interface JobView { + readonly context: RunContext; + readonly matrix: MatrixCombination; + /** The result of each job this one needs that has run, by job id. A job this run does not execute is absent, so a provider's expression reads it the way an absent value reads. */ + readonly needs: Readonly>; + /** The environment the job's steps inherit before any of them adds to it: the pipeline's and the job's own values, rendered for this combination. */ + readonly env: Readonly>; + /** The inclusive range of the job's own step indexes this run will execute, or undefined when it will execute them all. Preparation resolves actions whose steps run, so a step outside the range contributes nothing, not even the phases its action declares. */ + readonly range: { readonly from: number; readonly to: number } | undefined; +} + +/** A job view plus what the job's steps have left behind so far. */ +export interface StepView extends JobView { + /** Every step of this job that has settled, in order, each with the outcome and outputs the run recorded. */ + readonly steps: readonly StepRecord[]; +} + +/** One command a host resolved a provider step into: everything the runner needs, with no expression of the provider's left unevaluated. */ +export interface PreparedStep { + /** The name the step's report entries and log lines carry. */ + readonly name: string; + readonly command: string; + /** The step's own environment, final values; the runner composes the pipeline's, the job's and what earlier steps wrote around it. */ + readonly env: Readonly>; + readonly workingDirectory: string | undefined; +} + +/** What a host answers when the run reaches a provider step. */ +export type StepResolution = + /** The steps to run where the provider step stood, in order; each gets the runner's file commands and its own report entry, and the first failure stops the rest. */ + | { readonly kind: "steps"; readonly steps: readonly PreparedStep[] } + | { readonly kind: "skipped" } + /** A provider step this host does not run; the runner rejects the run naming the construct. */ + | { readonly kind: "unhandled"; readonly construct: string }; + +/** A job's steps as the host prepares them: the list to run and any environment the host adds at job level. */ +export interface JobExpansion { + readonly steps: readonly Step[]; + /** Job-level environment the host supplies for this provider's steps, final values, composed over the pipeline's and the job's own. */ + readonly env: Readonly>; +} + +/** + * The host's port for provider steps, which have no meaning outside their provider. A host that supplies one takes over both moments a provider step is reached: before the job runs, when the steps are prepared (a host places the phases an imported action declares around the job's own steps and adds whatever job-level environment its provider carries), and when the run reaches one of the steps the preparation left in place, when the host resolves it into what runs. The runner calls no other port in between, so a host's expressions evaluate against exactly the view each call receives. + */ +export interface StepHost { + /** Called once per job and matrix combination that will run, before any of its steps. */ + readonly expand: (job: Job, view: JobView) => Promise; + /** Called when the run reaches a provider step the preparation left in place. */ + readonly resolve: (step: Step, view: StepView) => Promise; +} + /** * The neutral context values for one run, supplied by whoever hosts the runner. A host supplies the values its provider can, and a value a step or condition actually reads but the host did not supply is an error at that point, so a command that needs no context runs wherever it is hosted. */ @@ -132,6 +195,8 @@ export interface RunOptions { readonly steps?: { readonly from: number; readonly to: number }; /** Report steps owned by a provider as skipped instead of rejecting them, for a host that has supplied the toolchain those steps would install. */ readonly skipProviderSteps?: boolean; + /** The host's port for provider steps. Supplied, it replaces the upfront rejection of a provider step: the host prepares the job's steps before it runs and resolves each provider step when the run reaches it. */ + readonly host?: StepHost; readonly signal?: AbortSignal; } diff --git a/packages/pipeline-github-actions/src/action.ts b/packages/pipeline-github-actions/src/action.ts index 22d491b..9f52f09 100644 --- a/packages/pipeline-github-actions/src/action.ts +++ b/packages/pipeline-github-actions/src/action.ts @@ -18,12 +18,12 @@ const envSchema = z.record(z.string().min(1), scalarSchema); const INPUT_ID = /^[A-Za-z_][A-Za-z0-9_-]*$/; /** - * One declared input. GitHub documents no type for an action's inputs (unlike a reusable workflow's `workflow_call` inputs): every value an action receives is a string, and `default` is a string or a YAML scalar that reads as one. `required` is a request, not an enforcement: the runner supplies an empty string when a required input arrives unset, so enforcement is this package's decision, taken in `composite.ts`. + * One declared input. GitHub documents no type for an action's inputs (unlike a reusable workflow's `workflow_call` inputs): every value an action receives is a string, and `default` is a string or a YAML scalar that reads as one. A `default` written with no value reads as null and means no default, which actions in wide use declare (`actions/checkout` among them). `required` is a request, not an enforcement: the runner supplies an empty string when a required input arrives unset, so enforcement is this package's decision, taken in `composite.ts`. */ const inputSchema = z.strictObject({ description: z.string(), required: z.boolean().optional(), - default: scalarSchema.optional(), + default: z.union([scalarSchema, z.null()]).optional(), deprecationMessage: z.string().optional(), }); export type ActionInput = z.output; @@ -39,10 +39,25 @@ const continueOnErrorSchema = z.union([z.boolean(), z.string()]); const actionKeys = { name: z.string().min(1), description: z.string(), + // The author the syntax page documents as optional, carried as data: nothing that runs reads it. + author: z.string().optional(), inputs: z.record(z.string().regex(INPUT_ID), inputSchema).optional(), branding: z.strictObject({ icon: z.string(), color: z.string() }).optional(), }; +/** + * A composite action's outputs, each a `${{ }}` expression over the steps that produced them, declared under `runs`. Carried as data only: evaluating one needs the outputs those steps wrote at run time, which no expansion of the steps can know beforehand. + */ +const compositeOutputsSchema = z.record( + z.string().min(1), + z.strictObject({ value: z.string(), description: z.string().optional() }), +); + +/** + * A node action's outputs, declared at the root of its metadata. Each carries a description only: the entry sets the value at run time through its output file, so there is no expression to read. + */ +const nodeOutputsSchema = z.record(z.string().min(1), z.strictObject({ description: z.string() })); + /** * A step that runs a command. `shell` is required whenever `run` is written, which is GitHub's own rule for composite steps and the reason it is not optional here. */ @@ -77,22 +92,14 @@ export const compositeStepSchema = z.union([runStepSchema, usesStepSchema]); export type CompositeStep = z.output; /** - * The metadata of an action whose `runs.using` is `composite`. The keys GitHub documents and this package does not read (`branding`, the declared `inputs`' descriptions, `runs.outputs`) are carried as data, because they describe the action rather than change how its steps run. Keys GitHub does not document for a composite action (a step `timeout-minutes`, `runs.image`) are rejected by the strict objects, which is the same stance the importer takes with a workflow. + * The metadata of an action whose `runs.using` is `composite`. The keys GitHub documents and this package does not read (`author`, `branding`, the declared `inputs`' descriptions, `runs.outputs`) are carried as data, because they describe the action rather than change how its steps run. Keys GitHub does not document for a composite action (a step `timeout-minutes`, `runs.image`) are rejected by the strict objects, which is the same stance the importer takes with a workflow. */ export const compositeActionSchema = z.strictObject({ ...actionKeys, runs: z.strictObject({ using: z.literal("composite"), steps: z.array(compositeStepSchema).min(1), - /** - * The action's outputs, each a `${{ }}` expression over the steps that produced them. They are carried as data only: evaluating one needs the outputs those steps wrote at run time, which no expansion of the steps can know beforehand. - */ - outputs: z - .record( - z.string().min(1), - z.strictObject({ value: z.string(), description: z.string().optional() }), - ) - .optional(), + outputs: compositeOutputsSchema.optional(), }), }); export type CompositeAction = z.output; @@ -104,12 +111,14 @@ export type CompositeAction = z.output; const NODE_RUNTIMES = ["node20", "node16", "node12", "node24"] as const; /** + * An action's outputs, each a `${{ }}` expression over the steps that produced them. A node action declares them at the root of its metadata and a composite action under its `runs`, which is where GitHub's own syntax page puts each, and they are carried as data only: evaluating one needs the outputs those steps wrote at run time, which no expansion of the steps can know beforehand. * The metadata of an action whose `runs.using` names a node runtime: `main` is the entry file the runtime executes, and `pre` and `post` are entry files that run around the job's steps when `pre-if` and `post-if` allow, both conditions defaulting to `always()`. A `pre` a local action declares is rejected where the expansion places it, because GitHub documents `pre` as not supported for local actions. Keys the node form does not document (`runs.image`, `runs.steps`) are rejected by the strict object, which is the same stance the composite schema takes. * @see https://docs.github.com/en/actions/sharing-automations/creating-actions/metadata-syntax-for-github-actions * @see https://web.archive.org/web/20250615225859/https://docs.github.com/en/actions/sharing-automations/creating-actions/metadata-syntax-for-github-actions */ export const nodeActionSchema = z.strictObject({ ...actionKeys, + outputs: nodeOutputsSchema.optional(), runs: z.strictObject({ using: z.enum(NODE_RUNTIMES), main: z.string().min(1), diff --git a/packages/pipeline-github-actions/src/composite.ts b/packages/pipeline-github-actions/src/composite.ts index d1d3fa8..4301367 100644 --- a/packages/pipeline-github-actions/src/composite.ts +++ b/packages/pipeline-github-actions/src/composite.ts @@ -264,7 +264,8 @@ function mapInputs( // The first matching key in document order wins when two differ only by case, which is the closest a mapping without duplicate keys can come to the runner's dictionary. value ??= entry; } - if (value === undefined && input.default !== undefined) { + // A `default` written with no value reads as null and means no default, so it is absent rather than the text "null". + if (value === undefined && input.default !== undefined && input.default !== null) { value = evaluateTemplate(scalarText(input.default), ambient); } if (value === undefined && input.required === true) { diff --git a/packages/pipeline-github-actions/src/index.ts b/packages/pipeline-github-actions/src/index.ts index 92e06eb..e9b57e7 100644 --- a/packages/pipeline-github-actions/src/index.ts +++ b/packages/pipeline-github-actions/src/index.ts @@ -30,3 +30,4 @@ export { type ExpressionValue, } from "./evaluation.ts"; export { createGitHubActionsImporter } from "./importer.ts"; +export { CHECKOUT_ACTION, CONTEXT_EXPRESSION, EVENT_NAME, PROVIDER } from "./vocabulary.ts"; diff --git a/packages/pipeline-github-actions/test/composite.unit.test.ts b/packages/pipeline-github-actions/test/composite.unit.test.ts index 3d1b390..a12c97e 100644 --- a/packages/pipeline-github-actions/test/composite.unit.test.ts +++ b/packages/pipeline-github-actions/test/composite.unit.test.ts @@ -159,6 +159,36 @@ describe("expandAction", () => { }); }); + it("treats a default written with no value as no default", async () => { + const files: Files = { + "action.yml": ` +name: Sparse +description: Maps inputs the way published actions declare them +inputs: + mode: + description: How to run + default: + extra: + description: An extra flag + required: true +runs: + using: composite + steps: + - run: echo mode=\${{ inputs.mode }} extra=\${{ inputs.extra }} + shell: bash +`, + }; + const { steps } = await expanded( + usesStep("exadev/sparse@v1", { extra: "--fast" }), + resolverOf({}, { "exadev/sparse@v1": files }), + ); + expect(steps[0]).toMatchObject({ run: "echo mode= extra=--fast" }); + const rejected = await rejectionOf( + expanded(usesStep("exadev/sparse@v1"), resolverOf({}, { "exadev/sparse@v1": files })), + ); + expect(rejected.message).toContain('the required input "extra"'); + }); + it("expands a remote action's nesting, running a nested node action and carrying a docker reference", async () => { const OUTER: Files = { "action.yml": ` diff --git a/packages/pipeline-github-actions/test/node.unit.test.ts b/packages/pipeline-github-actions/test/node.unit.test.ts index 6959374..01ccac3 100644 --- a/packages/pipeline-github-actions/test/node.unit.test.ts +++ b/packages/pipeline-github-actions/test/node.unit.test.ts @@ -34,6 +34,38 @@ runs: `, }; + it("accepts the root outputs and a null default a published action declares", async () => { + // The shapes actions/checkout itself declares: outputs at the root with descriptions only, and inputs whose default is written with no value. + const files: Files = { + "action.yml": ` +name: Checkout +description: Checks a repository out +author: GitHub +outputs: + ref: + description: The ref that was checked out + commit: + description: The commit SHA that was checked out +inputs: + repository: + description: Repository name with owner + default: + ref: + description: The ref to checkout +runs: + using: node24 + main: dist/index.js + post: dist/index.js +`, + }; + const { steps, post } = await expanded( + usesStep("exadev/checkout@v1", { repository: "exadev/pipewright" }), + resolverOf({}, { "exadev/checkout@v1": files }), + ); + expect(steps[0]?.kind).toBe("run"); + expect(post).toHaveLength(1); + }); + it("expands the main entry into a run step with the inputs and the action path", async () => { const { steps, pre, post } = await expanded( usesStep("exadev/greet@v1", { "cache-key": "build-main" }), diff --git a/packages/pipeline-local/src/runner.ts b/packages/pipeline-local/src/runner.ts index 02cbbe8..824c8ba 100644 --- a/packages/pipeline-local/src/runner.ts +++ b/packages/pipeline-local/src/runner.ts @@ -13,10 +13,12 @@ import { type PipelineRunner, type PredicateHandlers, type PredicateNode, + type PreparedStep, type RunContext, type RunOptions, type Step, type StepFileContents, + type StepRecord, type StepReport, } from "@pipewright/pipeline-contract"; @@ -145,7 +147,8 @@ interface JobPlan { readonly pipeline: Pipeline; readonly options: RunOptions; readonly jobCondition: Condition; - readonly stepConditions: readonly Condition[]; + /** The result of each job this one needs that this run executed, by id; a job this run does not carry is absent, so a host reading it sees an absent value. */ + readonly needs: Readonly>; /** False when an earlier job failed or was skipped and this job does not run regardless. */ readonly mayRun: boolean; /** A gate reports failure when a job it needs did not succeed, rather than being skipped. */ @@ -162,21 +165,17 @@ interface JobRunState { readonly masks: string[]; } -/** The state one matrix combination runs against: what its steps read and what they accumulate. */ -interface CombinationRun { - readonly scope: Scope; - readonly state: JobRunState; -} - -/** What running one step produced: its report entry and what it handed to the job's later steps. */ +/** What running one command produced: its report entry and what it handed to the job's later steps. */ interface StepRun { + /** The name the command's report entry and log lines carry. */ + readonly name: string; readonly outcome: Outcome; readonly outputs: Readonly>; - /** Entries the step wrote to its environment file, applied to the job's later steps. */ + /** Entries the command wrote to its environment file, applied to the job's later steps. */ readonly env: readonly FileEntry[]; - /** Directories the step added, applied to the job's later steps. */ + /** Directories the command added, applied to the job's later steps. */ readonly paths: readonly string[]; - /** The summary the step wrote, masked, when it wrote one. */ + /** The summary the command wrote, masked, when it wrote one. */ readonly summary: string | undefined; } @@ -188,13 +187,10 @@ function recordOf(entries: readonly FileEntry[]): Record { return Object.fromEntries(entries.map((entry) => [entry.key, entry.value])); } -/** - * Reads the workflow command stream from one step's stdout and echoes the rest through the run's output, masked. A mask covers only the lines after the command that registered it, as a runner reading the stream live does. - */ /** * A step's stderr carries no workflow commands, but it can carry a masked value, so each line is emitted through the masks the run has registered. */ -function processStderr(plan: JobPlan, state: JobRunState, step: Step, stderr: string): void { +function processStderr(plan: JobPlan, state: JobRunState, step: string, stderr: string): void { if (stderr === "") return; const lines = stderr.split("\n"); if (lines[lines.length - 1] === "") lines.pop(); @@ -202,14 +198,14 @@ function processStderr(plan: JobPlan, state: JobRunState, step: Step, stderr: st const line = ended.endsWith("\r") ? ended.slice(0, -1) : ended; plan.options.output.log({ job: plan.job.name, - step: stepName(step), + step, level: "info", message: applyMasks(line, state.masks), }); } } -function processStdout(plan: JobPlan, state: JobRunState, step: Step, stdout: string): void { +function processStdout(plan: JobPlan, state: JobRunState, step: string, stdout: string): void { const lines = stdout.split("\n"); // A trailing newline ends the last line rather than starting an empty one, and a line's own CR is dropped, so a command emitted with CRLF line endings is still read. if (lines[lines.length - 1] === "") lines.pop(); @@ -224,7 +220,7 @@ function processStdout(plan: JobPlan, state: JobRunState, step: Step, stdout: st } plan.options.output.log({ job: plan.job.name, - step: stepName(step), + step, level: parsed.command.level, message: applyMasks(parsed.command.message, state.masks), ...(parsed.command.annotation === undefined @@ -246,71 +242,73 @@ function processStdout(plan: JobPlan, state: JobRunState, step: Step, stdout: st } plan.options.output.log({ job: plan.job.name, - step: stepName(step), + step, level: "info", message: applyMasks(parsed.text, state.masks), }); } } -async function runStep( +/** + * Builds the runner of one command of the job: it runs the command with the runner's file commands around it, fresh command files and the job's composed environment, then reads the stream and the files back. The command's own environment arrives as final values, because a neutral step's are rendered here and a host's resolved steps come already evaluated. + */ +function commandRunner( plan: JobPlan, - combination: CombinationRun, - step: Step, - index: number, -): Promise { - const { scope, state } = combination; - const condition = plan.stepConditions[index]; - if (condition === undefined || !inRange(plan.options, index) || !condition(scope)) { - return { outcome: "skipped", outputs: {}, env: [], paths: [], summary: undefined }; - } - if (step.kind === "provider") { - return { outcome: "skipped", outputs: {}, env: [], paths: [], summary: undefined }; - } - if (step.kind !== "run") { - return { outcome: "success", outputs: {}, env: [], paths: [], summary: undefined }; - } - const files = await plan.options.scratch.create(); - const result = await plan.options.executor.execute( - { - command: step.run, - env: { - ...state.env, - ...resolveEnv(plan.pipeline.env, scope), - ...resolveEnv(plan.job.env, scope), - ...resolveEnv(step.env, scope), - [OUTPUT_VARIABLE]: files.output, - [ENV_VARIABLE]: files.env, - [PATH_VARIABLE]: files.path, - [SUMMARY_VARIABLE]: files.summary, + scope: Scope, + state: JobRunState, + declared: Readonly>, +): (step: PreparedStep) => Promise { + return async (step) => { + const files = await plan.options.scratch.create(); + const result = await plan.options.executor.execute( + { + command: step.command, + env: { + ...state.env, + ...declared, + ...step.env, + [OUTPUT_VARIABLE]: files.output, + [ENV_VARIABLE]: files.env, + [PATH_VARIABLE]: files.path, + [SUMMARY_VARIABLE]: files.summary, + }, + workingDirectory: step.workingDirectory, + pathEntries: [...state.pathEntries], }, - workingDirectory: step.workingDirectory, - pathEntries: [...state.pathEntries], - }, - plan.options.signal, - ); - let contents: StepFileContents; - try { - processStdout(plan, state, step, result.stdout); - processStderr(plan, state, step, result.stderr); - contents = await plan.options.scratch.read(files); - } finally { - // The files go even when reading or the command stream throws, so a failure leaves nothing a later process can read. - await plan.options.scratch.release(files); - } + plan.options.signal, + ); + let contents: StepFileContents; + try { + processStdout(plan, state, step.name, result.stdout); + processStderr(plan, state, step.name, result.stderr); + contents = await plan.options.scratch.read(files); + } finally { + // The files go even when reading or the command stream throws, so a failure leaves nothing a later process can read. + await plan.options.scratch.release(files); + } - return { - outcome: result.exitCode === 0 ? "success" : "failure", - outputs: recordOf(parseKeyValueFile(files.output, contents.output)), - env: parseKeyValueFile(files.env, contents.env), - paths: parsePathFile(contents.path), - summary: contents.summary === "" ? undefined : applyMasks(contents.summary, state.masks), + return { + name: step.name, + outcome: result.exitCode === 0 ? "success" : "failure", + outputs: recordOf(parseKeyValueFile(files.output, contents.output)), + env: parseKeyValueFile(files.env, contents.env), + paths: parsePathFile(contents.path), + summary: contents.summary === "" ? undefined : applyMasks(contents.summary, state.masks), + }; }; } +/** The conclusion one step of the job settled to, from the report entries its commands produced: one failure fails it, one skipped entry alone skips it, and anything else succeeded. */ +function concluded(produced: readonly StepReport[]): Outcome { + if (produced.some((report) => report.outcome === "failure")) return "failure"; + if (produced.length === 1 && produced[0]?.outcome === "skipped") return "skipped"; + + return "success"; +} + async function runCombination(plan: JobPlan, matrix: MatrixCombination): Promise { const state: JobRunState = { env: {}, pathEntries: [], masks: [] }; - const combination: CombinationRun = { scope: { context: plan.options.context, matrix }, state }; + const scope: Scope = { context: plan.options.context, matrix }; const { job, id } = plan; const skippedSteps = job.steps.map((step) => ({ name: stepName(step), @@ -323,11 +321,11 @@ async function runCombination(plan: JobPlan, matrix: MatrixCombination): Promise id, name: job.name, matrix, - outcome: plan.jobCondition(combination.scope) ? "failure" : "skipped", + outcome: plan.jobCondition(scope) ? "failure" : "skipped", steps: skippedSteps, }; } - if (!plan.mayRun || !plan.jobCondition(combination.scope)) { + if (!plan.mayRun || !plan.jobCondition(scope)) { return { id, name: job.name, @@ -337,27 +335,105 @@ async function runCombination(plan: JobPlan, matrix: MatrixCombination): Promise }; } - const steps = await job.steps.reduce>(async (previous, step, index) => { - const reports = await previous; - const failed = reports.some((report) => report.outcome === "failure"); - const run = failed - ? { outcome: "skipped" as const, outputs: {}, env: [], paths: [], summary: undefined } - : await runStep(plan, combination, step, index); - for (const entry of run.env) state.env[entry.key] = entry.value; - state.pathEntries = preferredPaths(run.paths, state.pathEntries); - if (run.summary !== undefined) { - plan.options.output.summary({ job: job.name, step: stepName(step), text: run.summary }); + // Preparation happens only for a job that runs, because a host resolves actions, which reads the filesystem and may fetch, and a job that does not run must not reach outside the run. + const range = plan.options.steps === undefined ? undefined : { ...plan.options.steps }; + const host = plan.options.host; + let steps: readonly Step[] = job.steps; + let declared: Record = { + ...resolveEnv(plan.pipeline.env, scope), + ...resolveEnv(plan.job.env, scope), + }; + if (host === undefined) { + checkPortable(id, steps, plan.options); + } else { + const expansion = await host.expand(job, { + context: scope.context, + matrix, + needs: plan.needs, + env: { ...declared }, + range, + }); + steps = expansion.steps; + declared = { ...declared, ...expansion.env }; + } + const conditions: readonly Condition[] = steps.map((step, index) => + conditionOf(step.when, `/jobs/${id}/steps/${String(index)}/when`), + ); + const records: StepRecord[] = []; + const command = commandRunner(plan, scope, state, declared); + + const reports = await steps.reduce>(async (previous, step, index) => { + const done = await previous; + const skip = (): StepReport[] => [{ name: stepName(step), outcome: "skipped", outputs: {} }]; + // The command a host resolved stops at its own first failure, and a failed step skips the rest of the job. + if (done.some((report) => report.outcome === "failure")) return [...done, ...skip()]; + if (!inRange(plan.options, index) || !(conditions[index] ?? (() => true))(scope)) { + return [...done, ...skip()]; + } + const produced: StepReport[] = []; + const run = async (prepared: PreparedStep): Promise => { + const executed = await command(prepared); + for (const entry of executed.env) state.env[entry.key] = entry.value; + state.pathEntries = preferredPaths(executed.paths, state.pathEntries); + if (executed.summary !== undefined) { + plan.options.output.summary({ job: job.name, step: executed.name, text: executed.summary }); + } + produced.push({ name: executed.name, outcome: executed.outcome, outputs: executed.outputs }); + }; + if (step.kind === "provider" && host !== undefined) { + const resolution = await host.resolve(step, { + context: scope.context, + matrix, + needs: plan.needs, + env: { ...state.env, ...declared }, + range, + steps: [...records], + }); + if (resolution.kind === "unhandled") { + throw new UnsupportedFeatureError( + PROVIDER, + `/jobs/${id}/steps/${String(index)}`, + resolution.construct, + ); + } + // The commands a host resolves run one at a time and the first failure stops the rest, exactly as the steps of an action do on the provider's own runner. + if (resolution.kind === "steps") { + await resolution.steps.reduce>(async (previousCommand, prepared) => { + await previousCommand; + if (produced.some((report) => report.outcome === "failure")) return; + await run(prepared); + }, Promise.resolve()); + } else { + produced.push(...skip()); + } + } else if (step.kind === "provider") { + produced.push(...skip()); + } else if (step.kind === "run") { + await run({ + name: stepName(step), + command: step.run, + env: resolveEnv(step.env, scope), + workingDirectory: step.workingDirectory, + }); + } else { + // `checkout` is a no-op, because the host has already placed the working tree. + produced.push({ name: stepName(step), outcome: "success", outputs: {} }); } + records.push({ + step, + outcome: concluded(produced), + outputs: Object.fromEntries(produced.flatMap((report) => Object.entries(report.outputs))), + }); - return [...reports, { name: stepName(step), outcome: run.outcome, outputs: run.outputs }]; + return [...done, ...produced]; }, Promise.resolve([])); return { id, name: job.name, matrix, - outcome: steps.some((report) => report.outcome === "failure") ? "failure" : "success", - steps, + outcome: reports.some((report) => report.outcome === "failure") ? "failure" : "success", + steps: reports, }; } @@ -374,8 +450,8 @@ function combinationsOf(job: Job, id: string, options: RunOptions): MatrixCombin return combinations; } -function checkPortable(id: string, job: Job, options: RunOptions): void { - for (const [index, step] of job.steps.entries()) { +function checkPortable(id: string, steps: readonly Step[], options: RunOptions): void { + for (const [index, step] of steps.entries()) { if (step.kind === "provider" && inRange(options, index) && options.skipProviderSteps !== true) { throw new UnsupportedFeatureError( PROVIDER, @@ -398,7 +474,9 @@ function aggregate(reports: readonly JobReport[]): Outcome { * * A job with a matrix runs once per combination, sequentially. `checkout` is a no-op, because the host has already placed the working tree. * - * Around each step the runner applies the provider's file commands: it asks the host's scratch for fresh command files and passes their paths in the command's environment, then reads what the step wrote, records its outputs, and carries its environment and path entries to the job's later steps. The step's stdout is read for workflow commands, whose masks and levels are applied to the run's output. + * Around each command the runner applies the provider's file commands: it asks the host's scratch for fresh command files and passes their paths in the command's environment, then reads what the command wrote, records its outputs, and carries its environment and path entries to the job's later commands. The command's stdout is read for workflow commands, whose masks and levels are applied to the run's output. + * + * A `steps` host takes over the provider steps: it is asked to prepare each job that runs, placing its own steps around the job's, and to resolve each provider step the preparation left in place, which may be several commands each receiving the same file-command treatment as a native step. */ export function createLocalRunner(): PipelineRunner { return { @@ -414,7 +492,6 @@ export function createLocalRunner(): PipelineRunner { jobReports = await inSequence(ids, async (id) => { const job = pipeline.jobs[id]; if (job === undefined) throw new Error(`unknown job ${id}`); - checkPortable(id, job, options); const needsSatisfied = selected !== undefined || @@ -426,8 +503,12 @@ export function createLocalRunner(): PipelineRunner { options, jobCondition: selected === undefined ? conditionOf(job.when, `/jobs/${id}/when`) : () => true, - stepConditions: job.steps.map((step, index) => - conditionOf(step.when, `/jobs/${id}/steps/${String(index)}/when`), + needs: Object.fromEntries( + (job.needs ?? []).flatMap((needed) => { + const outcome = outcomes.get(needed); + + return outcome === undefined ? [] : [[needed, outcome]]; + }), ), mayRun: job.needsPolicy === "always" || needsSatisfied, failsGate: job.needsPolicy === "gate" && !needsSatisfied,