diff --git a/architecture.md b/architecture.md index cf3ec78b5..a6395a055 100644 --- a/architecture.md +++ b/architecture.md @@ -448,6 +448,33 @@ durable is written. `undefined`, and a replay reconstructs the same validated prop set through ordinary expansion. +### The query engine a component loads + +`` and `` select and render through one published package rather +than through an engine this project owns. Three properties make that an +ordinary component concern rather than a new piece of architecture. + +- **It is loaded by the invocation that needs it.** A literal + `until(import("yq.rs"))` keeps the specifier in the module graph a compiled + binary and a published package are built from, and makes the load an ordinary + operation of the element that asked: cancellation stops that element's + continuation, its binding and its output. The package's root export is + already-ready — its own top-level await compiles the embedded WebAssembly — + so nothing here owns an instance, and loading twice is the module cache's + concern. Module compilation that finishes after cancellation authorizes no + result: the cancelled invocation publishes nothing. +- **Capabilities arrive as arguments, so none is given.** The evaluator reaches + the environment, the filesystem and the clock only through its `env`, `files` + and `now` options. Both call sites pass empty maps and omit `now`, which is + what makes an environment read an empty string, a file load a refusal and a + clock a refusal — and no process is ever started. A document that wants data + in a query hands it over as the value. +- **Acceptance is the engine's existing strict JSON.** The evaluation's whole + output is parsed as one JSON value and then crosses `parseJson`, so a + selection joins the same value domain as everything else a document holds. + Zero and several results are refusals rather than guesses, and a refusal + binds nothing. + ### Expansion identity Core describes the executable element currently being expanded: diff --git a/deno.json b/deno.json index 292fd4016..2219f7c1e 100644 --- a/deno.json +++ b/deno.json @@ -49,7 +49,8 @@ "typescript": "npm:typescript@^5.0.0", "unist-util-select": "npm:unist-util-select@^5", "zod": "npm:zod@^4.3.6", - "mdast-util-to-string": "npm:mdast-util-to-string@^4" + "mdast-util-to-string": "npm:mdast-util-to-string@^4", + "yq.rs": "npm:yq.rs@0.1.4" }, "tasks": { "xmd": "deno run --allow-all packages/cli/src/deno.ts", diff --git a/deno.lock b/deno.lock index 011bf7f1a..ef0eaacd5 100644 --- a/deno.lock +++ b/deno.lock @@ -115,6 +115,7 @@ "npm:unist-util-select@5": "5.1.0", "npm:vite@^7.1.3": "7.3.6_@types+node@24.13.3_tsx@4.23.1", "npm:vite@^7.1.4": "7.3.6_@types+node@24.13.3_tsx@4.23.1", + "npm:yq.rs@0.1.4": "0.1.4", "npm:zod@4": "4.6.6", "npm:zod@^4.3.6": "4.6.6" }, @@ -4301,6 +4302,10 @@ "yargs-parser" ] }, + "yq.rs@0.1.4": { + "integrity": "sha512-GOGCtD020Nowq9djd7lKrgSz9a+sVHv11IRwEE4vt15yKmen9GOBTTdub/5w6jUmcDxN8q3+dAWcGGaq8dYy6Q==", + "bin": true + }, "zod@4.6.6": { "integrity": "sha512-4UmzQUF5fxvfcYHlbjBmejU+w+hbDnWFx4aS8kBx7nPBHcO2SqPbpAG8snbdg+Xse4pRefR98hgjFgIMCI/UhA==" }, @@ -4339,6 +4344,7 @@ "npm:semver@^7.8.5", "npm:typescript@5", "npm:unist-util-select@5", + "npm:yq.rs@0.1.4", "npm:zod@^4.3.6" ], "packageJson": { @@ -4438,6 +4444,7 @@ "npm:remark@15", "npm:remend@^1.2.2", "npm:unist-util-select@5", + "npm:yq.rs@0.1.4", "npm:zod@^4.3.6" ], "packageJson": { @@ -4465,7 +4472,8 @@ "npm:mdast-util-to-string@4", "npm:remark@15", "npm:remend@^1.2.2", - "npm:unist-util-select@5" + "npm:unist-util-select@5", + "npm:yq.rs@0.1.4" ] } }, diff --git a/packages/cli/tests/repl-entries.test.ts b/packages/cli/tests/repl-entries.test.ts index a7ed1fcb7..3201e911b 100644 --- a/packages/cli/tests/repl-entries.test.ts +++ b/packages/cli/tests/repl-entries.test.ts @@ -142,6 +142,33 @@ const INHERITING = [ "", ].join("\n"); +/** + * One entry that selects out of data it computed, renders it both ways, and + * publishes the selected value for the next entry. + */ +const SELECTS_AND_RENDERS = [ + "```js eval", + 'const report = { service: "api", checks: [{ name: "boot", ok: true }, { name: "ping", ok: false }] };', + "```", + "", + ``, + "", + "", + "", + "", + "", + "```js eval", + "const selected = failing;", + "const picked = selected[0];", + "```", + "", + "One: {picked}", + "", +].join("\n"); + +/** One entry that queries what it inherited and prints the result unbound. */ +const QUERIES_INHERITED = ['', ""].join("\n"); + /** One entry that publishes a nested value a later entry will edit. */ const PUBLISHES_NESTED = [ "```js eval", @@ -777,6 +804,60 @@ describe("REPL entries: what each entry owns and what the next one inherits", () expect(value(after.entries[1].bindings, "seen")).toBe("draft/build/2"); }); + it("EP1: a selected value is retained, inherited and queried again", function* () { + const physical = new InMemoryStream(); + const first = yield* runEntry(physical, SELECTS_AND_RENDERS); + expect(first.failure).toBe(undefined); + + // Both renderers wrote the same selected value in their own spelling, and + // the entry's own output is what a person read. + expect(first.output).toContain('[\n "ping"\n]'); + expect(first.output).toContain("- ping\n"); + expect(first.output).toContain("One: ping"); + + // What the entry retained is the structure, not either rendering of it — + // the selected binding itself is ordinary root data, beside the value it + // was selected from. + const before = projected(yield* physical.readAll()); + expect(value(before.entries[0].bindings, "failing")).toEqual(["ping"]); + expect(value(before.entries[0].bindings, "selected")).toEqual(["ping"]); + const supplied = entryInitialBindings(before); + expect(supplied).toEqual({ + report: { + service: "api", + checks: [ + { name: "boot", ok: true }, + { name: "ping", ok: false }, + ], + }, + failing: ["ping"], + selected: ["ping"], + picked: "ping", + }); + + // The next entry inherits that value and queries it without `as`, so what + // it prints is JSON — a selected string arrives quoted. + const second = yield* runEntry(physical, QUERIES_INHERITED, supplied); + expect(second.failure).toBe(undefined); + expect(second.output).toContain('"ping"'); + + // Entry 1 is where it was: its own retained value did not move, and the + // record handed over is unchanged. + const after = projected(yield* physical.readAll()); + expect(after.entries).toHaveLength(2); + expect(value(after.entries[0].bindings, "selected")).toEqual(["ping"]); + expect(supplied.selected).toEqual(["ping"]); + // The second entry computed nothing of its own to publish, so what it + // retains is exactly the names it inherited: querying a value is not an + // effect with a record of its own. + expect(after.entries[1].bindings.map((binding) => binding.name).sort()).toEqual([ + "failing", + "picked", + "report", + "selected", + ]); + }); + it("EP1: a retained `props` is the entry's own and reaches no successor", function* () { const physical = new InMemoryStream(); const run = yield* runEntry(physical, PUBLISHES_PROPS); diff --git a/packages/core/deno.json b/packages/core/deno.json index abdb91ecd..955fa0756 100644 --- a/packages/core/deno.json +++ b/packages/core/deno.json @@ -22,6 +22,7 @@ "zod": "npm:zod@^4.3.6", "unist-util-select": "npm:unist-util-select@^5", "mdast-util-to-string": "npm:mdast-util-to-string@^4", - "@secretlint/profiler": "npm:@secretlint/profiler@13.0.4" + "@secretlint/profiler": "npm:@secretlint/profiler@13.0.4", + "yq.rs": "npm:yq.rs@0.1.4" } } diff --git a/packages/core/package.json b/packages/core/package.json index 087e5b35f..4f76bbaf6 100644 --- a/packages/core/package.json +++ b/packages/core/package.json @@ -34,6 +34,7 @@ "mdast-util-to-string": "^4", "remark": "15", "remend": "^1.2.2", - "unist-util-select": "^5" + "unist-util-select": "^5", + "yq.rs": "0.1.4" } } diff --git a/packages/core/src/components/Query.ts b/packages/core/src/components/Query.ts new file mode 100644 index 000000000..123a94104 --- /dev/null +++ b/packages/core/src/components/Query.ts @@ -0,0 +1,98 @@ +/** + * `` — select one JSON value out of a value the document already has + * (specs/executable-mdx-spec.md §6.22). + * + * `` reads text, `` lists paths, an eval block computes data; this + * takes one of those values and answers with the part of it the author asked + * for. The expression language is `yq`'s, and the whole of the request is the + * value and the expression: + * + * ```xmd + * + * + * ``` + * + * With `as` the selected value is bound as data and nothing is written. Without + * it the selected value is written as JSON — every type, including strings, + * which are quoted and escaped the way `` writes them. One spelling, one + * reading: an author never has to work out whether what they are looking at was + * a string before it was printed. + * + * ## It queries a value, and runs nothing to get one + * + * The value arrives as an ordinary prop, so whatever produced it has already + * finished and been bound. Querying it twice asks the same question of the same + * data twice: no file is read again, no request is repeated, and a paged + * producer's next page is not fetched because this never learns there is one. + * + * Text stays text. A JSON-looking string selected out of a value is a string, + * and `from_json` or `from_yaml` in the expression is how an author says to + * decode it — which is also what keeps a document that meant to match a quoted + * body from silently matching its contents instead. + * + * ## One value is the answer + * + * `null`, `false` and `[]` are answers. No answer at all and several answers + * are not: both refuse, and both say to collect with `[ … ]`, because an + * expression that yielded three results would otherwise have to pick one and an + * expression that yielded none would have to invent one. + * + * A refused selection binds nothing and prints nothing selected. The previous + * value of a binding this element would have written stays what it was — a + * failed query leaves the document with the data it already had rather than + * with the input it was handed. + */ + +import type { Operation, Result } from "effection"; +import { hasBinding, hasContent } from "../component-api.ts"; +import { printErrors } from "../component-failures.ts"; +import { parseJson } from "../json.ts"; +import { select } from "./query-engine.ts"; +import type { Json } from "../types.ts"; + +export const props = { + type: "object", + properties: { + value: {}, + yq: { type: "string", minLength: 1 }, + }, + required: ["value", "yq"], + additionalProperties: false, +}; + +/** An invocation `` cannot run, or a selection it cannot accept. */ +export class QueryError extends Error { + constructor(message: string, options?: ErrorOptions) { + super(message, options); + this.name = "QueryError"; + } +} + +const CONTENT = + ' selects from the value it is given, not from content: write .'; +const FAILED = " could not evaluate this expression."; + +export default printErrors(function* Query(props: Record): Operation { + if (yield* hasContent()) { + throw new QueryError(CONTENT); + } + // Asked before the selection, because the answer belongs to how the element + // was written rather than to what the expression found. + const bound = yield* hasBinding(); + const expression = String(props.yq); + + let selection: Result; + try { + selection = yield* select(expression, JSON.stringify(props.value ?? null)); + } catch (error) { + throw new QueryError(FAILED, { cause: error }); + } + if (!selection.ok) { + throw new QueryError(selection.error.message, { cause: selection.error }); + } + + // The whole answer, through the engine's own strict acceptance: a binding + // receives the domain every other value in a document belongs to. + const selected = parseJson(JSON.parse(selection.value)); + return bound ? selected : JSON.stringify(selected, null, 2); +}); diff --git a/packages/core/src/components/Yaml.ts b/packages/core/src/components/Yaml.ts new file mode 100644 index 000000000..b0b0c8114 --- /dev/null +++ b/packages/core/src/components/Yaml.ts @@ -0,0 +1,94 @@ +/** + * `` — render a structured value as YAML text + * (specs/executable-mdx-spec.md §6.23). + * + * ``'s sibling, for the other spelling of the same value: + * + * ```xmd + * + * ``` + * + * One supplied value becomes one piece of YAML — written where the element is, + * or captured by `as`. The text is the query package's own rendering, returned + * exactly as it produced it, including the newline it ends with: a reader + * comparing this against `yq`'s output elsewhere is looking at the same + * renderer. + * + * The operand is a **capture**, for the reason ``'s is (§6.5): an + * ordinary prop would cross the component JSON boundary first and serialize the + * value before this ever saw it, which would decide the diagnostics below + * somewhere else. + * + * ## Shape is decided before the operand runs + * + * Content, then the syntactic presence of `value`, and only then the captured + * operand — so a malformed invocation costs nothing. `capture()` stays outside + * the `try`, because an expression that throws before it produces a value is + * that invocation's ordinary captured-expression failure rather than a + * rendering failure. + * + * ## One serialization, then one rendering + * + * `JSON.stringify` runs once, and its failures stay the two different facts + * they are: no JSON text at all for a value that has none, and a thrown + * exception for a serialization that started and could not finish. A value + * carrying a getter or a `toJSON` hook belongs to the document, and asking it + * twice would run somebody else's code twice. + * + * What the renderer does with the serialized value — key order, scalar quoting, + * block style, how `null` and `false` are written — is the package's, and this + * restates none of it. The value a selection bound stays data; rendering it + * here does not change it. + */ + +import type { Operation } from "effection"; +import { capture, hasCapture, hasContent } from "../component-api.ts"; +import { printErrors } from "../component-failures.ts"; +import { yaml } from "./query-engine.ts"; + +export const props = { + type: "object", + properties: {}, + additionalProperties: false, +}; + +/** An invocation `` cannot render, or a value it cannot turn into text. */ +export class YamlRenderError extends Error { + constructor(message: string, options?: ErrorOptions) { + super(message, options); + this.name = "YamlRenderError"; + } +} + +const CONTENT = " renders the value it is given, not content: write ."; +const MISSING = " requires a `value` prop: write ."; +const NO_TEXT = " serialization produced no JSON text for this value."; +const THREW = " serialization of this value failed."; +const RENDER = " could not render this value as YAML."; + +export default printErrors(function* Yaml(): Operation { + if (yield* hasContent()) { + throw new YamlRenderError(CONTENT); + } + if (!(yield* hasCapture("value"))) { + throw new YamlRenderError(MISSING); + } + + const value = yield* capture("value"); + + let json: unknown; + try { + json = JSON.stringify(value); + } catch (error) { + throw new YamlRenderError(THREW, { cause: error }); + } + if (typeof json !== "string") { + throw new YamlRenderError(NO_TEXT); + } + + try { + return yield* yaml(json); + } catch (error) { + throw new YamlRenderError(RENDER, { cause: error }); + } +}); diff --git a/packages/core/src/components/components.md b/packages/core/src/components/components.md index f05e609cc..7910c9da8 100644 --- a/packages/core/src/components/components.md +++ b/packages/core/src/components/components.md @@ -357,6 +357,48 @@ Writes the JSON where the element is written, or binds it with `as`. The counterpart of ``: this turns a value into text, that turns text into a value. +## Yaml + +Renders a value as YAML text. + +```mdx + +``` + +Writes the YAML where the element is written, or binds it with `as`. ``'s +sibling: the same value, the other spelling, for a document going somewhere that +reads YAML. + +## Query + +Selects part of a value with a `yq` expression. + +```mdx + + + +``` + +The value is one a document already has — a file it read, a list of paths it +globbed, data an eval block computed — and the expression says which part of it +to keep. With `as` the selected value is bound as data, so ``, `` +and another `` can go on working with it. Without `as` the selected value +is written as JSON, whatever its type: an object as JSON, `false` as `false`, +`[]` as `[]`, and a string quoted and escaped the way JSON writes one. + +Text stays text until the expression says otherwise. `from_json` and `from_yaml` +decode it, which is how a document that meant to search a file's contents says +so, and how one that meant to match the text itself still can. + +One value is the answer. `null`, `false` and an empty array are answers; +selecting nothing and selecting several both refuse and say to collect results +with `[ … ]` instead. A refused selection binds nothing, so a name it would have +written keeps the value it already had. + +It queries the value it was given and nothing else. No file is read again for +it, and a page of results is the page it was handed — if a producer reported +more elsewhere, ask that producer for them. + ## Parse Parses JSON text against a schema, and errors on invalid content. diff --git a/packages/core/src/components/query-engine.ts b/packages/core/src/components/query-engine.ts new file mode 100644 index 000000000..33a24f4fc --- /dev/null +++ b/packages/core/src/components/query-engine.ts @@ -0,0 +1,108 @@ +/** + * The query engine `` and `` select and render with. + * + * One published package, loaded the first time an interpretation actually needs + * it. The literal specifier is what keeps it in the module graph a compiled + * binary and a published package are built from, and `until` is what makes the + * import an ordinary operation of the invocation that asked — so a cancelled + * element stops waiting for it the same way it stops waiting for anything else. + * + * The package's root export is a ready evaluator: its own top-level await + * compiles the embedded WebAssembly once and hands back something already + * usable. Loading it twice is the module cache's job, so there is no instance + * to own, revoke or tear down here, and nothing retains the invocation that + * triggered the load. + * + * ## Capabilities arrive as options, so none is given + * + * The evaluator reaches the environment, the filesystem and the clock only + * through `env`, `files` and `now`. Both call sites below pass empty maps and + * omit `now`, which is what makes `strenv(HOME)` an empty string rather than a + * host reading, `load("/etc/passwd")` a refusal rather than a file, and `now` a + * refusal rather than a timestamp. A document that wants data gives it to the + * component as a value. + * + * ## One answer, or none + * + * `evaluateWithStatus` reports `matched`, which describes truthiness: a + * selection of `false` or `null` is a successful selection that did not match. + * Success is the absence of a thrown error, so that flag is read by nobody here. + * + * What comes back is text, and how many results it holds is decided by parsing + * it whole. One JSON value parses. No results is an empty string, and several + * results are several JSON values the parser rejects — so neither can be + * mistaken for the other, and neither becomes a value by cutting the text into + * lines. + */ + +import { Err, Ok, until } from "effection"; +import type { Operation, Result } from "effection"; +import type { EvalOptions, Format, Yq } from "yq.rs"; + +/** Everything both components fix about an evaluation, as the package takes it. */ +function options(outputFormat: Format): EvalOptions { + return { + inputFormat: "json", + outputFormat, + indent: 2, + // A selected string arrives as JSON rather than bare, which is what lets + // the whole answer be parsed as one value. + unwrapScalar: false, + printDocSeparators: false, + env: {}, + files: {}, + }; +} + +function* evaluator(): Operation { + const { yq } = yield* until(import("yq.rs")); + return yq; +} + +/** A selection that is not one JSON value. */ +export class QuerySelectionError extends Error { + constructor(message: string) { + super(message); + this.name = "QuerySelectionError"; + } +} + +const NONE = + " expression selected nothing, and one JSON value is the answer: collect results with " + + "`[ … ]` to select an array, which may be empty."; +const SEVERAL = + " expression selected more than one value, and one JSON value is the answer: collect " + + "results with `[ … ]` to select them as an array."; + +/** + * Evaluate `expression` over `input` and answer with one JSON text. + * + * `input` is JSON the caller already serialized. A thrown evaluation failure is + * the package's and travels out of here unchanged, so the component can present + * it as its own refusal with that cause. + */ +export function* select(expression: string, input: string): Operation> { + const yq = yield* evaluator(); + const { output } = yq.evaluateWithStatus(expression, input, options("json")); + if (output.trim() === "") { + return Err(new QuerySelectionError(NONE)); + } + try { + JSON.parse(output); + } catch { + return Err(new QuerySelectionError(SEVERAL)); + } + return Ok(output); +} + +/** + * Render `input` as the YAML text of the same value. + * + * The expression is identity, so what comes back is the package's own complete + * rendering of what it was given — its indentation, its scalar quoting and its + * terminating newline — returned without a second pass over the text. + */ +export function* yaml(input: string): Operation { + const yq = yield* evaluator(); + return yq.evaluateWithStatus(".", input, options("yaml")).output; +} diff --git a/packages/core/src/components/registry.ts b/packages/core/src/components/registry.ts index c87faac53..33fb3f11b 100644 --- a/packages/core/src/components/registry.ts +++ b/packages/core/src/components/registry.ts @@ -37,6 +37,8 @@ import Glob, { returns as globReturns, } from "./Glob.ts"; import Json, { props as jsonProps } from "./Json.ts"; +import Query, { props as queryProps } from "./Query.ts"; +import Yaml, { props as yamlProps } from "./Yaml.ts"; import Parse, { props as parseProps, returns as parseReturns } from "./Parse.ts"; import SafeParse, { props as safeParseProps, returns as safeParseReturns } from "./SafeParse.ts"; import Slice, { props as sliceProps } from "./Slice.ts"; @@ -263,6 +265,35 @@ export const CORE_REGISTRY: ComponentRegistry = new Map([ }, { captures: ["value"], forms: ["self-closing"] }, ), + core( + "Query", + Query, + parseJsonObject(queryProps), + { + description: + "Select part of a value with a yq expression. " + + '`` binds the ' + + "scripts object; the same element without `as` writes the selected value as JSON. " + + "Text is decoded only where the expression says `from_json` or `from_yaml`, and an " + + "expression that selects nothing or several values says to collect them with `[ … ]`.", + as: "Optional. Binds the selected value as data instead of writing it as JSON.", + context: null, + }, + { forms: ["self-closing"] }, + ), + core( + "Yaml", + Yaml, + parseJsonObject(yamlProps), + { + description: + "Render a value as YAML text. `` writes the YAML where you " + + "put it.", + as: "Optional. Captures the YAML text instead of emitting it.", + context: null, + }, + { captures: ["value"], forms: ["self-closing"] }, + ), core( "Parse", Parse, diff --git a/packages/core/tests/query-component.test.ts b/packages/core/tests/query-component.test.ts new file mode 100644 index 000000000..0a9dfde95 --- /dev/null +++ b/packages/core/tests/query-component.test.ts @@ -0,0 +1,504 @@ +/** + * Tier QRY — `` (spec §6.22). + * + * Four questions, in the order an author meets them. What the two modes write + * and bind; what the expression language is allowed to decode and how many + * answers it may give; what the engine can reach while it evaluates; and what + * happens when the value under query came from a producer that already ran. + * + * Expansion output is read raw where framing is the point, and real execution + * is used where registration, binding and producers are the point — a selection + * that only ever ran through a hand-built invocation would prove nothing about + * the component a document resolves. + */ + +import { describe, it } from "@executablemd/test-support/bdd"; +import { expect } from "@executablemd/test-support/expect"; +import { ensure, resource, scoped, until } from "effection"; +import type { Operation } from "effection"; +import { mkdtemp, realpath } from "node:fs/promises"; +import { tmpdir } from "node:os"; +import { join } from "node:path"; +import process from "node:process"; +import { exists, readTextFile, rm, writeTextFile } from "@effectionx/fs"; +import { API, env, Files, useHostFiles } from "@executablemd/runtime"; +import { InMemoryStream } from "@executablemd/durable-streams"; +import { Component } from "../src/component-api.ts"; +import { collect } from "../src/collect.ts"; +import { execute } from "../src/execute.ts"; +import { expandSegments } from "../src/expand.ts"; +import { renderSegments } from "../src/render.ts"; +import { scanSegments } from "../src/scanner.ts"; +import { CORE_REGISTRY } from "../src/components/registry.ts"; +import type { + ComponentFailure, + ErrorSegment, + EvalEnv, + FunctionComponentDefinition, + Json, +} from "../src/types.ts"; + +/** Core's own registered definition, so what runs is what a document resolves. */ +function core(name: string): FunctionComponentDefinition { + const entry = CORE_REGISTRY.get(name)?.default?.definition; + if (entry === undefined || entry.kind !== "function") { + throw new Error(`core supplies no <${name}> definition`); + } + return entry; +} + +interface Run { + output: string; + observed: ErrorSegment[]; + /** Every failure offered to the `handleFailure` chain, from outside the run. */ + offered: ComponentFailure[]; +} + +/** Expand `source` with core's own ``/``/`` and `values` in scope. */ +function run(source: string, values: Record = {}): Operation { + return scoped(function* () { + const observed: ErrorSegment[] = []; + const offered: ComponentFailure[] = []; + yield* Component.around({ + *raise([segment], next) { + observed.push(segment); + return yield* next(segment); + }, + *handleFailure([failure], next) { + offered.push(failure); + return yield* next(failure); + }, + }); + const env: EvalEnv = { values }; + yield* Component.around({ env: () => env }, { at: "min" }); + yield* Component.around( + { + // deno-lint-ignore require-yield + *importComponent([name]) { + if (name !== "Query" && name !== "Yaml" && name !== "Json") { + throw new Error(`Cannot resolve component: ${name}`); + } + return core(name); + }, + }, + { at: "min" }, + ); + const origin = { path: "doc.md", baseOffset: 0, baseLine: 1 }; + const segments = yield* expandSegments(scanSegments(source, origin), {}, {}, new Set()); + return { output: renderSegments(segments), observed, offered }; + }); +} + +/** Whether an error named `name` is reachable from `error` by its causes. */ +function reaches(error: unknown, name: string, seen = new Set()): boolean { + if (error instanceof Error && error.name === name) { + return true; + } + if (typeof error !== "object" || error === null || seen.has(error)) { + return false; + } + seen.add(error); + return error instanceof Error && error.cause !== undefined + ? reaches(error.cause, name, seen) + : false; +} + +/** Everything a run reported, as one string. */ +function reported(result: Run): string { + return result.observed.map((segment) => segment.message).join("\n"); +} + +/** A temporary directory of this case's own, removed with its scope. */ +function useFixture(): Operation { + return resource(function* (provide) { + const dir = yield* until(mkdtemp(join(tmpdir(), "query-component-"))); + yield* ensure(() => rm(dir, { recursive: true, force: true })); + yield* provide(yield* until(realpath(dir))); + }); +} + +describe("Tier QRY — the two ordinary modes", () => { + const DATA: Record = { + name: "widget", + count: 2, + tags: ["a", "b"], + nothing: null, + off: false, + empty: [], + nested: { inner: { deep: true } }, + quoted: '{"a":1}', + lines: "first\nsecond {brace}", + }; + + it("Q1: every unbound type is written as JSON", function* () { + const result = yield* run( + [ + '', + '', + '', + '', + '', + '', + '', + "", + ].join("\n\n"), + { data: DATA }, + ); + + expect(reported(result)).toBe(""); + // A string is JSON-quoted like any other JSON value: one reading, whatever + // the selected type turns out to be. + expect(result.output).toContain('"widget"'); + expect(result.output).toContain("\n2\n"); + expect(result.output).toContain('[\n "a",\n "b"\n]'); + expect(result.output).toContain("\nnull\n"); + expect(result.output).toContain("\nfalse\n"); + expect(result.output).toContain("\n[]\n"); + expect(result.output).toContain('{\n "inner": {\n "deep": true\n }\n}'); + }); + + it("Q1: a quoted and a multiline string keep their JSON escaping", function* () { + const result = yield* run( + '\n\n\n', + { data: DATA }, + ); + + expect(reported(result)).toBe(""); + expect(result.output).toContain('"{\\"a\\":1}"'); + expect(result.output).toContain('"first\\nsecond {brace}"'); + }); + + it("Q1: a bound selection keeps its type and writes nothing", function* () { + const result = yield* run( + [ + '', + "", + '', + "flag is {flag}", + '', + "", + "", + ].join("\n\n"), + { data: DATA }, + ); + + expect(reported(result)).toBe(""); + // The structured value reached the binding: `` renders the object it + // was given rather than text somebody printed. + expect(result.output).toContain('{\n "inner": {\n "deep": true\n }\n}'); + expect(result.output).toContain("flag is false"); + expect(result.output).toContain("[]"); + // And the bound elements themselves wrote nothing of their own. + expect(result.output).not.toContain('"inner": {\n "deep": true\n }\n}\n{'); + }); + + it("Q1: a missing value, a missing yq, an alias, content and bad as all refuse", function* () { + const alone = yield* run('\n', { data: DATA }); + expect(reported(alone)).toContain("value"); + + const unqueried = yield* run("\n", { data: DATA }); + expect(reported(unqueried)).toContain("yq"); + + const aliased = yield* run('\n', { data: DATA }); + expect(reported(aliased)).toContain("expression"); + + const withContent = yield* run('body\n', { data: DATA }); + expect(reported(withContent)).toContain("not from content"); + + const badBinding = yield* run('\n', { + data: DATA, + }); + expect(reported(badBinding)).not.toBe(""); + }); + + it("Q1: a refused selection leaves an earlier binding as it was", function* () { + const result = yield* run( + [ + '', + '', + "picked is {picked}", + "", + ].join("\n\n"), + { data: DATA }, + ); + + expect(reported(result)).not.toBe(""); + // The prior data, not the input it was handed and not an empty value. + expect(result.output).toContain("picked is widget"); + }); +}); + +describe("Tier QRY — what the expression may decode, and how many answers it may give", () => { + it("Q2: identity over a JSON-looking string keeps it a string", function* () { + const result = yield* run('\n', { text: '{"a":1}' }); + + expect(reported(result)).toBe(""); + expect(result.output.trim()).toBe('"{\\"a\\":1}"'); + }); + + it("Q2: from_json and from_yaml decode one text each", function* () { + const result = yield* run( + [ + '', + '', + "", + ].join("\n\n"), + { json: '{"a":[1,2]}', yaml: "b: widget\n" }, + ); + + expect(reported(result)).toBe(""); + expect(result.output).toContain("[\n 1,\n 2\n]"); + expect(result.output).toContain('"widget"'); + }); + + it("Q2: a filter and a projection select through the pinned engine", function* () { + const records = { + records: [ + { name: "a", status: "ok" }, + { name: "b", status: "failed" }, + ], + }; + const result = yield* run( + `\n`, + { data: records }, + ); + + expect(reported(result)).toBe(""); + expect(result.output.trim()).toBe('[\n {\n "name": "b"\n }\n]'); + }); + + it("Q2: no answer and several answers both say to collect with brackets", function* () { + const none = yield* run('\n', { pair: [1, 2] }); + expect(reported(none)).toContain("[ … ]"); + expect(reported(none)).toContain("selected nothing"); + expect(none.output).not.toContain("1"); + + const several = yield* run('\n', { pair: [1, 2] }); + expect(reported(several)).toContain("[ … ]"); + expect(reported(several)).toContain("more than one"); + expect(several.output).not.toContain("1\n2"); + + // Collecting is what the guidance says to write, and it works. + const collected = yield* run('\n', { pair: [1, 2] }); + expect(reported(collected)).toBe(""); + expect(collected.output.trim()).toBe("[\n 1,\n 2\n]"); + }); + + it("Q2: an invalid expression fails with its own cause and binds nothing", function* () { + const result = yield* run('\n\n[{picked}]\n', { + data: { a: 1 }, + }); + + const message = reported(result); + expect(message).toContain("could not evaluate"); + // No fallback: the input is not the answer to a question that failed. + expect(result.output).not.toContain('"a": 1'); + expect(result.output).toContain("[{picked}]"); + }); + + it("Q2: null, false and an empty array are answers, not failures", function* () { + const result = yield* run( + [ + '', + '', + '', + "", + "", + "", + "", + ].join("\n\n"), + { data: { nothing: null, off: false } }, + ); + + expect(reported(result)).toBe(""); + expect(result.output).toContain("null"); + expect(result.output).toContain("false"); + expect(result.output).toContain("[]"); + }); +}); + +describe("Tier QRY — what the engine can reach", () => { + /** + * One host environment value, set for this case and removed with it. + * + * The canary has to be real: an environment read that answers nothing + * because nothing was there would discriminate nothing. + */ + function useEnvironmentCanary(name: string, value: string): Operation { + return resource(function* (provide) { + const before = process.env[name]; + // Registered before the value is set, so a scope that ends between the + // two still puts back what this process had. + yield* ensure(() => { + if (before === undefined) { + delete process.env[name]; + } else { + process.env[name] = before; + } + }); + process.env[name] = value; + yield* provide(value); + }); + } + + /** One actual readable file of this case's own, removed with its scope. */ + function* useFileCanary(contents: string): Operation { + const dir = yield* useFixture(); + const file = join(dir, "query-canary.txt"); + yield* writeTextFile(file, contents); + return file; + } + + it("Q3: an environment read answers empty rather than the host's own value", function* () { + const canary = yield* useEnvironmentCanary("QUERY_CANARY", "host-visible-secret"); + // The host really holds it, read through the same host surface a document + // reaches the environment by. + expect(yield* env("QUERY_CANARY")).toBe(canary); + + const result = yield* run( + '\n\n[{seen}]\n', + { data: { a: 1 } }, + ); + + expect(reported(result)).toBe(""); + // The empty environment answered, so what the binding holds is the empty + // string and not what this process has. + expect(result.output).toContain("[]"); + expect(result.output).not.toContain("host-visible-secret"); + }); + + it("Q3: loading an actual existing file refuses, and binds nothing", function* () { + const file = yield* useFileCanary("canary contents\n"); + + const result = yield* run( + `\n\n[{loaded}]\n`, + { data: { a: 1 } }, + ); + + const message = reported(result); + expect(message).toContain("could not evaluate"); + // The engine's own refusal is the cause, and the file's contents are + // nowhere in the document. + expect(reaches(result.offered[0]?.error, "YqError")).toBe(true); + expect(result.output).not.toContain("canary contents"); + // Nothing was bound: the name is still unresolved text. + expect(result.output).toContain("[{loaded}]"); + }); + + it("Q3: reading the clock refuses, and binds nothing", function* () { + const result = yield* run('\n\n[{when}]\n', { + data: { a: 1 }, + }); + + expect(reported(result)).toContain("could not evaluate"); + expect(reaches(result.offered[0]?.error, "YqError")).toBe(true); + expect(result.output).toContain("[{when}]"); + }); + + it("Q3: the disabled process operation refuses, runs nothing and binds nothing", function* () { + const dir = yield* useFixture(); + const marker = join(dir, "system-ran.txt"); + + const result = yield* run( + `\n\n[{ran}]\n`, + { data: { a: 1 } }, + ); + + expect(reported(result)).toContain("could not evaluate"); + expect(reaches(result.offered[0]?.error, "YqError")).toBe(true); + // No process ran: the marker the command would have created is absent. + expect(yield* exists(marker)).toBe(false); + expect(result.output).toContain("[{ran}]"); + }); +}); + +describe("Tier QRY — two producers, each run once, over the page they supplied", () => { + const FIXTURE = JSON.stringify( + { + records: [ + { name: "a", status: "ok" }, + { name: "b", status: "failed" }, + ], + next: "cursor-2", + }, + null, + 2, + ); + + /** Every provider call this case's own fixture saw. */ + interface Calls { + reads: string[]; + globs: string[]; + } + + function watched(calls: Calls): Operation { + return Files.around({ + *readTextFile([input], next) { + const answer = yield* next(input); + calls.reads.push(String(input.path)); + return answer; + }, + *globFiles([input], next) { + calls.globs.push([...input.include].join(",")); + return yield* next(input); + }, + }); + } + + it("Q4: a file read once and a glob listed once answer several queries", function* () { + const dir = yield* useFixture(); + yield* writeTextFile(join(dir, "input.json"), FIXTURE); + yield* writeTextFile( + join(dir, "doc.md"), + [ + '', + ``, + '', + '', + ``, + "", + "", + "", + "", + ].join("\n\n"), + ); + + const calls: Calls = { reads: [], globs: [] }; + const stream = new InMemoryStream(); + const output = yield* scoped(function* () { + // The document's own working directory, installed explicitly the way an + // entrypoint does: `` and `` resolve against `Env.cwd` + // rather than against this process's. + yield* API.Env.around( + { + // deno-lint-ignore require-yield + *cwd() { + return dir; + }, + }, + { at: "min" }, + ); + yield* useHostFiles(); + yield* watched(calls); + return yield* collect(yield* execute({ path: join(dir, "doc.md"), stream, includes: [dir] })); + }); + + const rendered = String(output); + expect(rendered).toContain('{\n "name": "b"\n }'); + expect(rendered).toContain('[\n "a",\n "b"\n]'); + expect(rendered).toContain('"path": "input.json"'); + // Each producer crossed its provider exactly once, however many questions + // were asked of what it produced. + expect(calls.reads).toEqual(["input.json"]); + expect(calls.globs).toHaveLength(1); + // The next marker is in the value and stays there: nothing followed it and + // nothing claimed the records were complete. + expect(rendered).not.toContain("cursor-2"); + expect( + yield* scoped(function* () { + yield* useHostFiles(); + return yield* readTextFile(join(dir, "input.json")); + }), + ).toBe(FIXTURE); + }); +}); diff --git a/packages/core/tests/yaml-component.test.ts b/packages/core/tests/yaml-component.test.ts new file mode 100644 index 000000000..d93fb3839 --- /dev/null +++ b/packages/core/tests/yaml-component.test.ts @@ -0,0 +1,223 @@ +/** + * Tier YAML — `` (spec §6.23). + * + * ``'s sibling, read the same way: what the text says, and what a + * document cannot safely write. The mixed operand below is one value rendered + * twice — once as JSON by the existing component, once as YAML by this one — + * and the comparison is made by decoding both texts back and asking whether + * they describe what is still bound. + * + * The hostile operands are here for the reason ``'s are: a getter that + * counts its own reads and a cycle cannot be written in Markdown without an + * eval block that would teach the wrong thing. + */ + +import { describe, it } from "@executablemd/test-support/bdd"; +import { expect } from "@executablemd/test-support/expect"; +import { scoped } from "effection"; +import type { Operation } from "effection"; +import { Component } from "../src/component-api.ts"; +import { expandSegments } from "../src/expand.ts"; +import { renderSegments } from "../src/render.ts"; +import { scanSegments } from "../src/scanner.ts"; +import { CORE_REGISTRY } from "../src/components/registry.ts"; +import { select } from "../src/components/query-engine.ts"; +import type { + ComponentFailure, + ErrorSegment, + EvalEnv, + FunctionComponentDefinition, +} from "../src/types.ts"; + +function core(name: string): FunctionComponentDefinition { + const entry = CORE_REGISTRY.get(name)?.default?.definition; + if (entry === undefined || entry.kind !== "function") { + throw new Error(`core supplies no <${name}> definition`); + } + return entry; +} + +interface Run { + output: string; + observed: ErrorSegment[]; + /** Every failure offered to the `handleFailure` chain, from outside the run. */ + offered: ComponentFailure[]; +} + +function run(source: string, values: Record = {}): Operation { + return scoped(function* () { + const observed: ErrorSegment[] = []; + const offered: ComponentFailure[] = []; + yield* Component.around({ + *raise([segment], next) { + observed.push(segment); + return yield* next(segment); + }, + *handleFailure([failure], next) { + offered.push(failure); + return yield* next(failure); + }, + }); + const env: EvalEnv = { values }; + yield* Component.around({ env: () => env }, { at: "min" }); + yield* Component.around( + { + // deno-lint-ignore require-yield + *importComponent([name]) { + if (name !== "Yaml" && name !== "Json" && name !== "Query") { + throw new Error(`Cannot resolve component: ${name}`); + } + return core(name); + }, + }, + { at: "min" }, + ); + const origin = { path: "doc.md", baseOffset: 0, baseLine: 1 }; + const segments = yield* expandSegments(scanSegments(source, origin), {}, {}, new Set()); + return { output: renderSegments(segments), observed, offered }; + }); +} + +function reported(result: Run): string { + return result.observed.map((segment) => segment.message).join("\n"); +} + +/** Whether `target` is reachable from `error` by identity. */ +function reaches(error: unknown, target: unknown, seen = new Set()): boolean { + if (error === target) { + return true; + } + if (typeof error !== "object" || error === null || seen.has(error)) { + return false; + } + seen.add(error); + return error instanceof Error && error.cause !== undefined + ? reaches(error.cause, target, seen) + : false; +} + +/** The mixed value both renderers are asked about. */ +const MIXED = { + name: "widget", + count: 2, + numeric: "7", + lines: "first\nsecond", + tags: ["a", "b"], + nothing: null, + off: false, + nested: { inner: [1, { deep: true }] }, +}; + +describe("Tier YAML — what the text says", () => { + it("Q5: one value renders as JSON and as YAML, and both decode back to it", function* () { + const result = yield* run( + [ + '', + '', + "JSON:{json}", + "YAML:{yaml}", + "", + ].join("\n\n"), + { mixed: MIXED }, + ); + + expect(reported(result)).toBe(""); + const json = result.output.split("JSON:")[1]?.split("YAML:")[0] ?? ""; + const yaml = result.output.split("YAML:")[1] ?? ""; + + // The existing component's own shape, unchanged: two spaces, no trailing + // newline of its own. + expect(json.trim()).toBe(JSON.stringify(MIXED, null, 2)); + // The renderer's complete YAML text, two-space and without document + // separators, ending in the newline it produced. + expect(yaml).toContain("name: widget\n"); + expect(yaml).toContain("nested:\n inner:\n - 1\n - deep: true\n"); + expect(yaml).not.toContain("---"); + expect(yaml.endsWith("\n")).toBe(true); + // A numeric-looking string stays a string, and a multiline string survives. + expect(yaml).toContain('numeric: "7"'); + expect(yaml).toContain("lines: "); + expect(yaml).toContain("nothing:"); + expect(yaml).toContain("off: false"); + + // And the two texts describe the same value: the YAML is decoded by the + // expression language the document would use, not by a parser this suite + // brought along. + const decoded = yield* select("from_yaml", JSON.stringify(yaml)); + expect(decoded.ok).toBe(true); + if (decoded.ok) { + expect(JSON.parse(decoded.value)).toEqual(MIXED); + } + expect(JSON.parse(json.trim())).toEqual(MIXED); + }); + + it("Q5: a selected binding stays data while both renderers write text", function* () { + const result = yield* run( + [ + '', + '', + "", + "captured:{text}", + "", + ].join("\n\n"), + { mixed: MIXED }, + ); + + expect(reported(result)).toBe(""); + // The binding is still the structure: `` renders an object, not the + // YAML text the other element captured. + expect(result.output).toContain( + '{\n "inner": [\n 1,\n {\n "deep": true\n }\n ]\n}', + ); + expect(result.output).toContain("captured:inner:\n - 1\n - deep: true\n"); + }); +}); + +describe("Tier YAML — what a document cannot write", () => { + it("Q5: the operand is serialized once, and capture failures stay apart", function* () { + let reads = 0; + const counted = { + get name(): string { + reads += 1; + return "widget"; + }, + }; + + const rendered = yield* run("\n", { payload: counted }); + expect(reported(rendered)).toBe(""); + expect(rendered.output).toContain("name: widget"); + // Once: the renderer serializes the operand a single time, and nothing here + // reads it again to describe what it did. + expect(reads).toBe(1); + + const cycle: Record = {}; + cycle.self = cycle; + const cyclic = yield* run('\n\n[{text}]\n', { + payload: cycle, + }); + expect(reported(cyclic)).toContain("serialization of this value failed"); + // Atomically: nothing partial was captured under that name. + expect(cyclic.output).toContain("[{text}]"); + + const thrown = new Error("the hook refused"); + const hostile = yield* run("\n", { + payload: { + toJSON() { + throw thrown; + }, + }, + }); + // The document's own thrown hook, reachable as the cause of what this + // component refused with. + expect(reaches(hostile.offered[0]?.error, thrown)).toBe(true); + }); + + it("Q5: missing value and content refuse before anything is rendered", function* () { + const missing = yield* run("\n"); + expect(reported(missing)).toContain("requires a `value` prop"); + + const withContent = yield* run("body\n", { payload: { a: 1 } }); + expect(reported(withContent)).toContain("not content"); + expect(withContent.output).not.toContain("a: 1"); + }); +}); diff --git a/pnpm-lock.yaml b/pnpm-lock.yaml index 9b0a09b2b..5aa5dc4c2 100644 --- a/pnpm-lock.yaml +++ b/pnpm-lock.yaml @@ -316,6 +316,9 @@ importers: unist-util-select: specifier: ^5 version: 5.1.0 + yq.rs: + specifier: 0.1.4 + version: 0.1.4 packages/durable-streams: dependencies: @@ -3005,6 +3008,11 @@ packages: resolution: {integrity: sha512-D1mvvtDG0L5ft/jGWkLpG1+m0eQxOfaBvTNELraWj22wSVUMWxZUvYgJYcKh6jGGIkJFhH4IZPQhR4TKpc8mBw==} engines: {node: '>=10'} + yq.rs@0.1.4: + resolution: {integrity: sha512-GOGCtD020Nowq9djd7lKrgSz9a+sVHv11IRwEE4vt15yKmen9GOBTTdub/5w6jUmcDxN8q3+dAWcGGaq8dYy6Q==} + engines: {node: '>=22'} + hasBin: true + zod@4.3.6: resolution: {integrity: sha512-rftlrkhHZOcjDwkGlnUtZZkvaPHCsDATp4pGpuOOMDaTdDDXF91wuVDJoWoPsKX/3YPQ5fHuF3STjcYyKr+Qhg==} @@ -5267,6 +5275,8 @@ snapshots: y18n: 5.0.8 yargs-parser: 20.2.9 + yq.rs@0.1.4: {} + zod@4.3.6: {} zod@4.4.3: {} diff --git a/scripts/runtime-test-exclusions.ts b/scripts/runtime-test-exclusions.ts index 869ac10c5..36007b097 100644 --- a/scripts/runtime-test-exclusions.ts +++ b/scripts/runtime-test-exclusions.ts @@ -71,6 +71,12 @@ const DENO_ONLY_TOOLING: RuntimeExclusion[] = [ "builds packages/acp with dnt before driving the artifact under Node, and that build is spawned as ` run -A scripts/build-npm.ts` — an argument vector only the Deno CLI understands; the materialization it then proves is covered portably by Tier AM", issue: DERIVED_SCOPE, }, + { + path: "scripts/tests/query-distribution.test.ts", + reason: + "it is the harness that drives the query engine into every distribution, and the driving is Deno's own: it starts `Deno.execPath()` for the source arm, compiles the host binary through the canonical `compile()`, and builds the CLI's npm closure with dnt before packing and installing it. The Node and Bun consumers it exists to prove are launched by it — source Node through the repository's `tsx` loader, source Bun, and the packed closure under both — so excluding the harness from those runtimes drops none of that consumption. What the engine does inside a document is proven on every runtime by packages/core/tests/query-component.test.ts", + issue: DERIVED_SCOPE, + }, { path: "scripts/tests/cloudflare-dofs-vendor.test.ts", reason: diff --git a/scripts/tests/query-distribution.test.ts b/scripts/tests/query-distribution.test.ts new file mode 100644 index 000000000..45dc3a932 --- /dev/null +++ b/scripts/tests/query-distribution.test.ts @@ -0,0 +1,499 @@ +/** + * Tier QDIST — the query engine in every distribution this repository ships. + * + * `` and `` reach their expression language through one published + * package whose root export compiles embedded WebAssembly during its own + * top-level await, loaded through a dynamic import. Each of those is a thing a + * distribution can lose: a module graph may omit a specifier nothing statically + * imports, a compiled binary has no `node_modules` to fall back to, and a + * published package carries whatever its emitted manifest declares. + * + * So one document is run five ways and compared against the source reading of + * it: under Deno, under Node through the repository's own loader, under Bun, + * from one compiled host binary, and from the built npm closure packed and + * installed into a directory of its own. The document reads a file, decodes it + * explicitly, queries a globbed list, renders both ways and prints each unbound + * type — so a distribution that loaded the engine but lost a capability fails + * here rather than at a reader's first document. + * + * One compile and one closure build, reused by every consumption below. The + * cwd of each consumption holds the fixture and nothing else: no checkout, no + * `node_modules` beside it, and — for the compiled binary — no executable on + * `PATH` at all, because an engine that shelled out to a `yq` on the machine + * would otherwise pass this. + */ + +import { describe, it } from "@executablemd/test-support/bdd"; +import { expect } from "@executablemd/test-support/expect"; +import { useTempDirectory } from "@executablemd/test-support/temp"; +import { scoped, until } from "effection"; +import type { Operation } from "effection"; +import { exec, Stdio } from "@effectionx/process"; +import type { ProcessResult } from "@effectionx/process"; +import { timebox } from "@effectionx/timebox"; +import { ensureDir, exists, readTextFile, writeTextFile } from "@effectionx/fs"; +import { createHash } from "node:crypto"; +import { readFile, readdir } from "node:fs/promises"; +import path from "node:path"; +import { fileURLToPath } from "node:url"; +import { compile } from "../lib/compile.ts"; +import { useNpmWorkspace } from "./npm-output.ts"; + +const ROOT = fileURLToPath(new URL("../../", import.meta.url)); + +/** dnt's install and type-check dominate the closure build. */ +const TIMEOUT = 900_000; + +/** The one document every distribution runs. */ +const DOCUMENT = [ + '', + "", + ``, + "", + '', + "", + ``, + "", + "JSON:", + "", + "YAML:", + "", + 'OBJ:', + "", + 'FALSE:', + "", + 'NULL:', + "", + 'EMPTY:', + "", +].join("\n"); + +/** Its input, with a next marker nothing is allowed to follow. */ +const INPUT = JSON.stringify( + { + service: "api", + checks: [ + { name: "boot", ok: true }, + { name: "ping", ok: false }, + ], + next: "cursor-2", + }, + null, + 2, +); + +/** A directory holding the fixture and nothing else. */ +function* useDocument(): Operation { + const dir = yield* useTempDirectory("xmd-query-dist-"); + yield* writeTextFile(path.join(dir, "input.json"), INPUT); + yield* writeTextFile(path.join(dir, "doc.md"), DOCUMENT); + return dir; +} + +function* run( + command: string, + args: readonly string[], + cwd: string, + env?: Record, +): Operation { + const attempt = yield* timebox(TIMEOUT, function* () { + return yield* exec(command, { + arguments: [...args], + cwd, + ...(env === undefined ? {} : { env }), + }).join(); + }); + if (attempt.timeout) { + throw new Error(`${command} timed out`); + } + return attempt.value; +} + +/** What each distribution has to say, read as the operator reads it. */ +function reading(result: ProcessResult): string { + if (result.code !== 0) { + throw new Error(`the distribution exited ${result.code}\n${result.stderr}`); + } + expect(result.stdout).not.toContain("ERROR"); + return result.stdout.trim(); +} + +/** Every selected value and both renderings, as one comparable text. */ +function selections(output: string): string[] { + return output + .split("\n") + .map((line) => line.trimEnd()) + .filter((line) => line !== ""); +} + +function digest(bytes: Uint8Array): string { + return createHash("sha256").update(bytes).digest("hex"); +} + +/** + * What one launch was, as the log keeps it. + * + * The identities belong in the output rather than only in an assertion: a + * passing consumption says that some artifact ran, and this says which one, + * from where, under which runtime, before the owned scratch is released. + */ +interface Launch { + readonly distribution: string; + readonly command: string; + readonly argv: readonly string[]; + readonly cwd: string; + readonly runtime: string; + /** A child killed by a signal reports none, which is not a success. */ + readonly exit: number | undefined; +} + +/** Everything one run of this tier observed, printed as its own evidence. */ +interface Evidence { + readonly artifacts: Record; + readonly emitted: Record; + readonly launches: Launch[]; +} + +function record( + evidence: Evidence, + distribution: string, + command: string, + argv: readonly string[], + cwd: string, + runtime: string, + result: ProcessResult, +): string { + evidence.launches.push({ distribution, command, argv, cwd, runtime, exit: result.code }); + return reading(result); +} + +/** The version string one runtime reports for itself. */ +function* runtimeVersion(command: string, args: readonly string[], cwd: string): Operation { + const reported = yield* run(command, args, cwd); + return reported.stdout.trim().split("\n")[0] ?? ""; +} + +function report(evidence: Evidence): void { + console.log(`QDIST evidence: ${JSON.stringify(evidence, null, 2)}`); +} + +describe( + "Tier QDIST — one document, every distribution", + { + sanitizeOps: false, + sanitizeResources: false, + }, + () => { + it("QD1: the three source runtimes select, render and print the same values", function* () { + const dir = yield* useDocument(); + const evidence: Evidence = { artifacts: {}, emitted: {}, launches: [] }; + + const denoArgv = [ + "run", + "--allow-all", + "--cached-only", + "--frozen", + "--config", + path.join(ROOT, "deno.json"), + path.join(ROOT, "packages/cli/src/deno.ts"), + "run", + "doc.md", + ]; + const deno = record( + evidence, + "source deno", + Deno.execPath(), + denoArgv, + dir, + yield* runtimeVersion(Deno.execPath(), ["--version"], dir), + yield* run(Deno.execPath(), denoArgv, dir), + ); + + // The selected values, both renderings and each unbound type. + expect(selections(deno)).toEqual([ + "JSON:[", + ' "ping"', + "]", + "YAML:- ping", + "OBJ:{", + ' "path": "input.json"', + "}", + "FALSE:false", + "NULL:null", + "EMPTY:[]", + ]); + // The page it was given, and no next page: the marker is in the file and + // nothing followed it. + expect(deno).not.toContain("cursor-2"); + + const nodeArgv = [ + "--tsconfig", + path.join(ROOT, "tsconfig.node.json"), + path.join(ROOT, "packages/cli/src/node.ts"), + "run", + "doc.md", + ]; + const node = record( + evidence, + "source node (repository tsx loader)", + path.join(ROOT, "node_modules/.bin/tsx"), + nodeArgv, + dir, + yield* runtimeVersion("node", ["--version"], dir), + yield* run(path.join(ROOT, "node_modules/.bin/tsx"), nodeArgv, dir), + ); + + const bunArgv = [path.join(ROOT, "packages/cli/src/bun.ts"), "run", "doc.md"]; + const bun = record( + evidence, + "source bun", + "bun", + bunArgv, + dir, + yield* runtimeVersion("bun", ["--version"], dir), + yield* run("bun", bunArgv, dir), + ); + + expect(selections(node)).toEqual(selections(deno)); + expect(selections(bun)).toEqual(selections(deno)); + expect(evidence.launches.map((launch) => launch.exit)).toEqual([0, 0, 0]); + report(evidence); + }); + + it("QD2: one compiled host binary runs it with no checkout and no PATH", function* () { + const dir = yield* useDocument(); + const scratch = yield* useTempDirectory("xmd-query-binary-"); + const binary = path.join(scratch, "xmd"); + + // The canonical compile, once, with the repository's own flags. + yield* scoped(function* () { + yield* Stdio.around({ + *stdout() {}, + *stderr() {}, + }); + yield* compile({ output: binary }); + }); + + const evidence: Evidence = { artifacts: {}, emitted: {}, launches: [] }; + const bytes = yield* until(readFile(binary)); + // Which binary ran, by its own bytes, recorded while it still exists. + evidence.artifacts[path.basename(binary)] = { + bytes: bytes.byteLength, + sha256: digest(bytes), + }; + + // Nothing on PATH, so an engine that reached for a `yq` executable has + // none to find, and nothing beside the document to resolve from. + const compiled = record( + evidence, + "compiled host binary", + binary, + ["run", "doc.md"], + dir, + yield* runtimeVersion(binary, ["--version"], dir), + yield* run(binary, ["run", "doc.md"], dir, { HOME: scratch, PATH: "" }), + ); + + expect(selections(compiled)).toEqual([ + "JSON:[", + ' "ping"', + "]", + "YAML:- ping", + "OBJ:{", + ' "path": "input.json"', + "}", + "FALSE:false", + "NULL:null", + "EMPTY:[]", + ]); + + // The same bytes afterwards: the hash above names what actually ran. + const after = yield* until(readFile(binary)); + expect(digest(after)).toBe(evidence.artifacts[path.basename(binary)]?.sha256); + expect(evidence.launches.map((launch) => launch.exit)).toEqual([0]); + report(evidence); + }); + + it("QD3: the built npm closure declares the engine and runs it under Node and Bun", function* () { + const dir = yield* useDocument(); + const workspace = yield* useNpmWorkspace(); + const evidence: Evidence = { artifacts: {}, emitted: {}, launches: [] }; + const version = "0.0.0-query-distribution"; + + const built = yield* scoped(function* () { + yield* Stdio.around({ + *stdout() {}, + *stderr() {}, + }); + return yield* run( + Deno.execPath(), + [ + "run", + "-A", + "--node-modules-dir=manual", + "--cached-only", + "--frozen", + "scripts/build-npm.ts", + "packages/cli", + version, + ], + workspace, + { ...Deno.env.toObject(), NPM_CONFIG_LEGACY_PEER_DEPS: "true" }, + ); + }); + if (built.code !== 0) { + throw new Error(`build-npm.ts exited ${built.code}\n${built.stderr}`); + } + + // The emitted manifest declares the exact pinned engine, which is what a + // consumer installs: a build that dropped it would resolve nothing. + const emitted = JSON.parse( + yield* readTextFile(path.join(workspace, "packages/core/npm/package.json")), + ); + expect(emitted.dependencies["yq.rs"]).toBe("0.1.4"); + evidence.emitted["core dependency"] = `yq.rs@${emitted.dependencies["yq.rs"]}`; + + // The specifier survived the build: a graph that dropped the literal + // import would install the engine and never reach it. + const emittedEngine = yield* readTextFile( + path.join(workspace, "packages/core/npm/esm/src/components/query-engine.js"), + ); + expect(emittedEngine).toContain('import("yq.rs")'); + evidence.emitted["core root import"] = 'import("yq.rs")'; + + // The closure, packed as artifacts and installed into a directory of its + // own — never resolved from the registry, and never from this checkout. + const scratch = yield* useTempDirectory("xmd-query-closure-"); + const tarballs = path.join(scratch, "tarballs"); + const consumer = path.join(scratch, "consumer"); + yield* ensureDir(tarballs); + yield* ensureDir(consumer); + yield* writeTextFile( + path.join(consumer, "package.json"), + // React is the peer `@executablemd/web`'s form surface reaches through + // `@rjsf`, and the CLI's module graph loads that surface at startup. An + // ordinary consumer receives it as a resolved peer; this one declares + // it, because the install below is told to resolve none. + `${JSON.stringify( + { + name: "query-consumer", + private: true, + type: "module", + dependencies: { react: "19.2.0", "react-dom": "19.2.0" }, + }, + null, + 2, + )}\n`, + ); + + const members = (yield* until( + readdir(path.join(workspace, "packages"), { + withFileTypes: true, + }), + )) + .filter((entry) => entry.isDirectory()) + .map((entry) => path.join(workspace, "packages", entry.name, "npm")); + for (const member of members) { + const manifest = path.join(member, "package.json"); + if (!(yield* exists(manifest))) { + continue; + } + const packed = yield* run( + "npm", + ["pack", "--silent", "--pack-destination", tarballs], + member, + ); + if (packed.code !== 0) { + throw new Error(`npm pack failed in ${member}\n${packed.stderr}`); + } + } + + const artifacts = (yield* until(readdir(tarballs))).filter((name) => name.endsWith(".tgz")); + for (const name of artifacts) { + const packedBytes = yield* until(readFile(path.join(tarballs, name))); + evidence.artifacts[name] = { + bytes: packedBytes.byteLength, + sha256: digest(packedBytes), + }; + } + // Every workspace member the CLI's closure holds, as artifacts this build + // produced. + expect(artifacts.length).toBeGreaterThanOrEqual(10); + expect(artifacts).toContain(`executablemd-cli-${version}.tgz`); + + const installed = yield* run( + "npm", + [ + "install", + "--no-audit", + "--no-fund", + "--legacy-peer-deps", + ...artifacts.map((name) => path.join(tarballs, name)), + "react@19.2.0", + "react-dom@19.2.0", + ], + consumer, + ); + if (installed.code !== 0) { + throw new Error(`npm install failed\n${installed.stderr}`); + } + + const bin = path.join(consumer, "node_modules/@executablemd/cli/esm/src/node.js"); + const consumed = JSON.parse( + yield* readTextFile(path.join(consumer, "node_modules/@executablemd/core/package.json")), + ); + // The installed copy, not the checkout: its own manifest names the version + // this build emitted and the engine it declared. + expect(consumed.dependencies["yq.rs"]).toBe("0.1.4"); + + evidence.emitted["installed core dependency"] = `yq.rs@${consumed.dependencies["yq.rs"]}`; + evidence.emitted["installed cli version"] = String( + JSON.parse( + yield* readTextFile(path.join(consumer, "node_modules/@executablemd/cli/package.json")), + ).version, + ); + + const node = record( + evidence, + "packed npm closure under node", + "node", + [bin, "run", "doc.md"], + dir, + yield* runtimeVersion("node", ["--version"], dir), + yield* run("node", [bin, "run", "doc.md"], dir), + ); + const bun = record( + evidence, + "packed npm closure under bun", + "bun", + [bin, "run", "doc.md"], + dir, + yield* runtimeVersion("bun", ["--version"], dir), + yield* run("bun", [bin, "run", "doc.md"], dir), + ); + + expect(selections(node)).toEqual([ + "JSON:[", + ' "ping"', + "]", + "YAML:- ping", + "OBJ:{", + ' "path": "input.json"', + "}", + "FALSE:false", + "NULL:null", + "EMPTY:[]", + ]); + expect(selections(bun)).toEqual(selections(node)); + + // Every artifact this build produced is named by its own hash, and the + // two launches are recorded, before the owned scratch is released. + expect(Object.keys(evidence.artifacts).length).toBe(artifacts.length); + for (const artifact of Object.values(evidence.artifacts)) { + expect(artifact.sha256).toMatch(/^[0-9a-f]{64}$/); + } + expect(evidence.launches.map((launch) => launch.exit)).toEqual([0, 0]); + report(evidence); + }); + }, +); diff --git a/specs/executable-mdx-spec.md b/specs/executable-mdx-spec.md index 438ef7c1e..bc65bb4ce 100644 --- a/specs/executable-mdx-spec.md +++ b/specs/executable-mdx-spec.md @@ -10262,6 +10262,91 @@ a partial replay both reach it through normal expansion and take a window of the text that execution reconstructed; completed-root terminal reuse is unchanged. +### 6.22 Selecting part of a value: `` + +A document that read a file, globbed a list or computed data usually wants part +of what it has rather than all of it. `` is core's own component (§5.3) +for that: + +```md + + + +``` + +It takes two ordinary props, both required: `value` is any JSON value, and `yq` +is the expression, written in the syntax `yq.rs` implements. `as` is optional +and ordinary (§6.10): with it the selected value is bound and the invocation +emits nothing, and without it the selected value is written as JSON. + +**Every unbound type is written as JSON**, through the same formatting `` +uses (§6.12): an object and an array as JSON, `false` as `false`, `null` as +`null`, `[]` as `[]`, and a string JSON-quoted and escaped. There is no +raw-text branch, so what a reader sees never depends on which type the +expression happened to select. The text carries no added newline or framing of +its own; enclosing output and normalization rules still apply. + +The supplied value is queried as it arrives. A string stays a string until the +expression decodes it with `from_json` or `from_yaml`, so a document that meant +to match a quoted body matches the body and one that meant to match the text +matches the text. Nothing is re-read or re-requested: the producer that bound +the value has already finished, several queries over one value cost one +production, and a producer's own next-page marker is ordinary data this never +follows. + +**Exactly one JSON value is an answer.** `null`, `false` and `[]` are answers. +An expression that selects nothing and one that selects several both refuse, +and both say to collect results with `[ … ]`. The whole evaluation output is +accepted as one JSON value or not at all — it is never split into lines — and a +refusal binds nothing: a name the invocation would have written keeps the value +it already had, and no input is substituted as a fallback. + +An invalid expression and an evaluation failure refuse with the engine's own +failure as the cause. Refusals follow ordinary component error policy (§6.8), +so a printed diagnostic does not make a rejected selection successful. + +The engine reaches nothing of the host. It is evaluated with an empty +environment and an empty file map and with no clock, so `strenv` reads an empty +string, `load` and `now` refuse, and no process is started. + +#### Resolution and replay + +The registration is an ordinary overridable default, not a reserved name: a +repository `Query.md` is chosen ahead of it. Inspection reports the +self-closing form, the props schema above, no captures and no `returns` — a +declared return contract would make `as` required, and `as` is optional here. +The component has no durable effect of its own: a live run and a replay both +reach it through normal expansion and select again from the value expansion +reconstructed. + +### 6.23 Rendering a value as YAML: `` + +``'s sibling (§6.12), for a document going somewhere that reads YAML: + +```md + +``` + +The operand is a **capture** (§6.5), as ``'s is, so the exact evaluation +result arrives by reference. Shape is decided before the operand runs: content +refuses, then a missing `value` refuses, and only then is the operand evaluated +and serialized. `JSON.stringify` runs once; a value with no JSON text and a +serialization that threw are different refusals, and the thrown value is +preserved as the cause. + +The text is the query engine's own rendering of that value with an identity +expression and YAML output: two-space indentation, no document separators, +scalar quoting as the renderer writes it, and the terminating newline it +produced. It is returned unchanged. `as` captures that text instead of emitting +it, and a value bound elsewhere stays data — rendering it here transforms +nothing. + +#### Resolution and replay + +An ordinary overridable default, like ``: inspection reports the +self-closing form, the `value` capture and text return mode, and the component +has no durable effect of its own. + ## 7. Entry point ### 8.1 `execute`