diff --git a/README.md b/README.md index dcbcc74..5fd6021 100644 --- a/README.md +++ b/README.md @@ -66,11 +66,13 @@ The internal packages, private and bundled into `process-graph`: | [`graph-dag`](packages/core/graph/dag/README.md) | Model-independent dependency-graph primitives over a node set and a `needs` accessor: layers, ancestors, descendants, cycles, transitive reduction, critical path. | | [`graph-schema`](packages/core/graph/schema/README.md) | The process model: typed identifier spaces, `defineModel` with compile-time reference checking, structural validation. | | [`graph-projection`](packages/core/graph/projection/README.md) | Projections of the model into a flowchart, swimlane, state machine and sequence diagram, plus derived hand-offs, and the inverses that turn an edited view into a diff against the model. | +| [`graph-export`](packages/core/graph/export/README.md) | Text exports of a projected view: Mermaid and Graphviz DOT for every notation and the projection as JSON, with every label from a source made safe to show; the one writer the command line and the web app share. | | [`graph-domain`](packages/core/graph/domain/README.md) | Reasoning over the model and a run overlay: control graph, illegal cycles, three-valued status, ready set, blocked-by, impact, what-if and critical path. | | [`graph-layout`](packages/core/graph/layout/README.md) | Layered layout of a projected flowchart, swimlane or state machine into positions in the model's `views` shape. | | [`graph-source-contract`](packages/core/graph/source-contract/README.md) | The `GraphSource` port as an oRPC contract: versioned, paged and streamed sync of bound activities, participants, transitions and run state, resumable by cursor, typed filters and scope listings, with an in-memory fake and a conformance kit. | | [`graph-source-github-issues`](packages/adapters/graph/source-github-issues/README.md) | The GitHub issues adapter for the `GraphSource` port: issues, `blockedBy` and `blocking`, sub-issues and assignees across organisations, repositories and Projects v2 boards, with server-side filters, a scope listing and per-owner tokens. | | [`graph-store-contract`](packages/core/graph/store-contract/README.md) | The persistence port as an oRPC contract: save, load and delete models, set run state and the facts a run reads (provided artefacts, condition values), upsert items and relations from a source, ready, blocked-chain, impact and hierarchy queries, a change feed; with an in-memory fake and a conformance kit. | +| [`graph-sync`](packages/core/graph/sync/README.md) | Syncs `GraphSource`s into a `GraphStore` page by page: participants, bound activities, transitions, relations and run state, resumable from a checkpoint and narrowed by a typed filter; the one sync the command line and the web app share. | | [`graph-store-sqlite`](packages/adapters/graph/store-sqlite/README.md) | The store port on SQLite through Drizzle: one table definition, committed drizzle-kit migrations applied the same way in every runtime, recursive SQL for the structural queries, against Drizzle's async SQLite database so any driver plugs in. | | [`graph-sqlite-node`](packages/adapters/graph/sqlite-node/README.md) | The SQLite store on Node's built-in `node:sqlite` through `drizzle-orm/node-sqlite`, for the command line and servers. | | [`graph-sqlite-wasm`](packages/adapters/graph/sqlite-wasm/README.md) | The SQLite store in the browser: SQLite wasm in a dedicated worker on OPFS `opfs-sahpool`, shared between tabs under a Web Lock with no special headers, served to the page over oRPC. | diff --git a/knip.config.ts b/knip.config.ts index ff8a968..7a348a5 100644 --- a/knip.config.ts +++ b/knip.config.ts @@ -21,20 +21,12 @@ const config: KnipConfig = { entry: ["src/index.{ts,tsx}"], project: ["src/**/*.{ts,tsx}!"], }, - // The contracts' Zod schemas embed the model's, which are built on trilean's expression and JSON schemas, so their emitted declarations import trilean although no source file names it, and knip traces source alone. - ...Object.fromEntries( - [ - "packages/core/graph/source-contract", - "packages/core/graph/store-contract", - ].map((workspace) => [ - workspace, - { - entry: ["src/index.{ts,tsx}"], - project: ["src/**/*.{ts,tsx}!"], - ignoreDependencies: ["trilean"], - }, - ]), - ), + // The source contract's Zod schemas embed the model's, which are built on trilean's expression and JSON schemas, so its emitted declarations import trilean although no source file names it, and knip traces source alone. The store contract names trilean in its own source, so knip sees that use. + "packages/core/graph/source-contract": { + entry: ["src/index.{ts,tsx}"], + project: ["src/**/*.{ts,tsx}!"], + ignoreDependencies: ["trilean"], + }, "packages/adapters/*/*": { entry: ["src/index.{ts,tsx}"], project: ["src/**/*.{ts,tsx}!"], diff --git a/packages/core/graph/export/AGENTS.md b/packages/core/graph/export/AGENTS.md new file mode 120000 index 0000000..42061c0 --- /dev/null +++ b/packages/core/graph/export/AGENTS.md @@ -0,0 +1 @@ +README.md \ No newline at end of file diff --git a/packages/core/graph/export/CLAUDE.md b/packages/core/graph/export/CLAUDE.md new file mode 120000 index 0000000..42061c0 --- /dev/null +++ b/packages/core/graph/export/CLAUDE.md @@ -0,0 +1 @@ +README.md \ No newline at end of file diff --git a/packages/core/graph/export/README.md b/packages/core/graph/export/README.md new file mode 100644 index 0000000..c7d4a9a --- /dev/null +++ b/packages/core/graph/export/README.md @@ -0,0 +1,40 @@ +# @exadev/process-graph-export + +[![GitHub](https://img.shields.io/badge/GitHub-181717?logo=github&logoColor=white)](https://github.com/ExaDev/process-graph) [![CI](https://img.shields.io/github/actions/workflow/status/ExaDev/process-graph/ci.yml?branch=main)](https://github.com/ExaDev/process-graph/actions) + +A private workspace package, not published on its own: it is bundled into [`process-graph`](../../../targets/process-graph/README.md), which exposes it as its exports at the root entry, `process-graph`. + +Text exports of a view projected by [`graph-projection`](../projection/README.md): Mermaid and Graphviz DOT for the flowchart, swimlane, state machine and sequence diagram, and the projection itself as JSON. The command line's `show` and the web app's downloads both write through `viewText`, so the same view gives the same text in both. `src/` imports nothing Node-only. + +```ts +import { project } from "@exadev/process-graph-projection"; +import { viewText } from "@exadev/process-graph-export"; + +const projected = project(model, { id: "lanes", notation: "swimlane" }); +viewText(model, projected, "mermaid"); // flowchart LR ... with a subgraph per lane +viewText(model, projected, "dot"); // digraph "swimlane" { ... } with a cluster per lane +viewText(model, projected, "json"); // the swimlane projection, indented +``` + +## Functions + +| Export | What it is | +| --------------------------------------------------------------------------------------- | ----------------------------------------------------------------------------------------------------------------------------------------------------------- | +| `viewText(model, projected, format)` | A `ProjectedView` as `mermaid`, `dot` or `json` text; the model gives a sequence diagram's participant labels, an undeclared actor shown by its identifier. | +| `exportFormats`, `ExportFormat` | The formats `viewText` writes. | +| `flowchartToMermaid`, `swimlaneToMermaid`, `stateMachineToMermaid`, `sequenceToMermaid` | Each notation as Mermaid source, ending with a line break. | +| `flowchartToDot`, `swimlaneToDot`, `stateMachineToDot`, `sequenceToDot` | Each notation as Graphviz DOT source, ending with a line break. | +| `terminalText(text)`, `terminalJson(value)`, `isUnsafeCharacter(character)` | Text and JSON made safe to show: every control, format, separator, unassigned and private-use character written as a visible escape of its code point. | + +## Safe labels + +A label from a source (an issue title, a repository name) can carry characters that break a diagram or act on whatever shows it. The Mermaid emitters name nodes by their position in the view, since Mermaid does not accept identifiers such as `github:org/repo#12`, and write every character Mermaid reads as syntax as its entity code. The DOT emitters keep the model identifiers and quote every one. In both, every character in the Unicode general categories `Cc`, `Cf`, `Zl`, `Zp`, `Cn` and `Co` is written as a visible escape such as `\u{1b}`, apart from line feed, tab and the zero-width joiner and non-joiner, which text in many scripts needs. Hand-offs are thick (Mermaid) or bold (DOT) edges, and DOT, which has no sequence notation, draws a sequence diagram as a grid of lifelines with one numbered row per message. + +## Commands + +```sh +pnpm build # tsc -b -> dist/ (ESM + .d.ts) +pnpm test # unit tests +pnpm test:smoke # the built dist/ exercised as a real artefact +pnpm test:mutation # Stryker +``` diff --git a/packages/core/graph/export/eslint.config.ts b/packages/core/graph/export/eslint.config.ts new file mode 100644 index 0000000..4c43201 --- /dev/null +++ b/packages/core/graph/export/eslint.config.ts @@ -0,0 +1,9 @@ +import { defineConfig } from "eslint/config"; +import { packageLintConfig } from "../../../../eslint.shared.ts"; + +export default defineConfig( + packageLintConfig({ + tsconfigRootDir: import.meta.dirname, + isomorphic: true, + }), +); diff --git a/packages/core/graph/export/package.json b/packages/core/graph/export/package.json new file mode 100644 index 0000000..8471974 --- /dev/null +++ b/packages/core/graph/export/package.json @@ -0,0 +1,70 @@ +{ + "name": "@exadev/process-graph-export", + "version": "0.0.0", + "description": "Text exports of a projected view: Mermaid and Graphviz DOT for every notation, and the projection as JSON, with every label from a source made safe to show.", + "private": true, + "type": "module", + "repository": { + "type": "git", + "url": "git+https://github.com/ExaDev/process-graph.git", + "directory": "packages/core/graph/export" + }, + "homepage": "https://github.com/ExaDev/process-graph/tree/main/packages/core/graph/export#readme", + "bugs": { + "url": "https://github.com/ExaDev/process-graph/issues" + }, + "exports": { + ".": { + "types": "./dist/index.d.ts", + "import": "./dist/index.js" + } + }, + "types": "./dist/index.d.ts", + "files": [ + "dist" + ], + "sideEffects": false, + "license": "MIT", + "scripts": { + "build": "turbo run _build", + "_build": "tsc -b tsconfig.build.json", + "lint": "turbo run _lint", + "_lint": "eslint . --cache --max-warnings 0", + "_lint:fix": "eslint . --fix --cache --max-warnings 0", + "typecheck": "turbo run _typecheck _typecheck:attw", + "_typecheck": "tsc -p tsconfig.json --noEmit && tsc -p tsconfig.node.json --noEmit", + "_typecheck:attw": "attw --pack --profile esm-only", + "test": "turbo run _test", + "_test": "vitest run --project unit", + "test:coverage": "turbo run _test:coverage", + "_test:coverage": "vitest run --project unit --coverage", + "test:smoke": "turbo run _test:smoke", + "_test:smoke": "vitest run --project smoke", + "test:mutation": "turbo run _test:mutation", + "_test:mutation": "stryker run stryker.config.ts", + "prepublishOnly": "pnpm run lint && pnpm run typecheck && pnpm run test && pnpm run build && publint && attw --pack --profile esm-only" + }, + "keywords": [ + "process-model", + "mermaid", + "graphviz", + "dot", + "export" + ], + "devDependencies": { + "@arethetypeswrong/cli": "catalog:", + "@stryker-mutator/core": "catalog:", + "@stryker-mutator/typescript-checker": "catalog:", + "@stryker-mutator/vitest-runner": "catalog:", + "@types/node": "catalog:", + "@vitest/coverage-v8": "catalog:", + "eslint": "catalog:", + "publint": "catalog:", + "typescript": "catalog:", + "vitest": "catalog:" + }, + "dependencies": { + "@exadev/process-graph-projection": "workspace:*", + "@exadev/process-graph-schema": "workspace:*" + } +} diff --git a/packages/targets/graph-cli/src/emit/dot.ts b/packages/core/graph/export/src/dot.ts similarity index 100% rename from packages/targets/graph-cli/src/emit/dot.ts rename to packages/core/graph/export/src/dot.ts diff --git a/packages/targets/graph-cli/src/emit/emit.unit.test.ts b/packages/core/graph/export/src/emit.unit.test.ts similarity index 100% rename from packages/targets/graph-cli/src/emit/emit.unit.test.ts rename to packages/core/graph/export/src/emit.unit.test.ts diff --git a/packages/core/graph/export/src/index.ts b/packages/core/graph/export/src/index.ts new file mode 100644 index 0000000..ce8951e --- /dev/null +++ b/packages/core/graph/export/src/index.ts @@ -0,0 +1,21 @@ +/** + * Text exports of a projected view: Mermaid and Graphviz DOT for the flowchart, swimlane, state machine and sequence diagram, and the projection itself as JSON. Every label from a source is made safe to show: escaped for the notation, and with every control and invisible format character written as a visible escape. The command line's `show` and the web app's downloads both write through `viewText`. + */ +export { exportFormats, viewText, type ExportFormat } from "./view.ts"; +export { + flowchartToMermaid, + sequenceToMermaid, + stateMachineToMermaid, + swimlaneToMermaid, +} from "./mermaid.ts"; +export { + flowchartToDot, + sequenceToDot, + stateMachineToDot, + swimlaneToDot, +} from "./dot.ts"; +export { + isUnsafeCharacter, + terminalJson, + terminalText, +} from "./terminal-text.ts"; diff --git a/packages/targets/graph-cli/src/emit/mermaid.ts b/packages/core/graph/export/src/mermaid.ts similarity index 100% rename from packages/targets/graph-cli/src/emit/mermaid.ts rename to packages/core/graph/export/src/mermaid.ts diff --git a/packages/targets/graph-cli/src/terminal-text.ts b/packages/core/graph/export/src/terminal-text.ts similarity index 100% rename from packages/targets/graph-cli/src/terminal-text.ts rename to packages/core/graph/export/src/terminal-text.ts diff --git a/packages/core/graph/export/src/terminal-text.unit.test.ts b/packages/core/graph/export/src/terminal-text.unit.test.ts new file mode 100644 index 0000000..0ab806b --- /dev/null +++ b/packages/core/graph/export/src/terminal-text.unit.test.ts @@ -0,0 +1,98 @@ +import { describe, expect, test } from "vitest"; +import { + isUnsafeCharacter, + terminalJson, + terminalText, +} from "./terminal-text.ts"; + +/** A title as a hostile source could write it: clear the screen (CSI 2J), set the clipboard (OSC 52, ended by BEL), reverse the rest of the line (a right-to-left override), and a NUL. */ +const hostile = + "Fix\u001b[2J\u001b]52;c;aGVsbG8=\u0007 \u202Egnp.exe\u0000 now"; +/** The same title as the command line writes it. */ +const escaped = + "Fix\\u{1b}[2J\\u{1b}]52;c;aGVsbG8=\\u{7} \\u{202e}gnp.exe\\u{0} now"; +/** Ordinary text from anywhere: accents, CJK and an emoji, none of which is changed. */ +const ordinary = "Café 日本語 🚀"; + +/** The characters of an output that should never be written raw; empty for safe output. */ +const unsafeIn = (output: string): string[] => + Array.from(output).filter((character) => isUnsafeCharacter(character)); + +/** One example of each kind of character the general-category rule escapes beyond the C0 and C1 controls, with the escape it is written as. */ +const categoryExamples: readonly (readonly [string, string, string])[] = [ + ["the Arabic letter mark (Cf)", "\u061C", "\\u{61c}"], + ["the left-to-right mark (Cf)", "\u200E", "\\u{200e}"], + ["the right-to-left mark (Cf)", "\u200F", "\\u{200f}"], + ["the line separator (Zl)", "\u2028", "\\u{2028}"], + ["the paragraph separator (Zp)", "\u2029", "\\u{2029}"], + ["the soft hyphen (Cf)", "\u00AD", "\\u{ad}"], + ["a tag character (Cf)", "\u{E0041}", "\\u{e0041}"], + ["the interlinear annotation anchor (Cf)", "\uFFF9", "\\u{fff9}"], + ["a private-use character (Co)", "\uE000", "\\u{e000}"], + ["an unassigned code point (Cn)", "\u2065", "\\u{2065}"], +]; + +/** Text that must pass through unchanged: a family emoji joined by zero-width joiners, CJK, accents, Arabic, and Persian with its zero-width non-joiner. */ +const scripts = [ + "\u{1F468}\u200D\u{1F469}\u200D\u{1F467}", + "日本語の文", + "crème brûlée", + "مرحبا بالعالم", + "\u0645\u06CC\u200C\u062E\u0648\u0627\u0647\u0645", +]; + +describe("terminalText", () => { + test("escapes every control and format character class it guards, visibly and by code point", () => { + expect( + terminalText( + "\u0000\u0008\u000B\u001F\u007F\u0080\u009F\u200B\u200F\u202A\u202E\u2060\u2064\u2066\u2069\uFEFF", + ), + ).toBe( + "\\u{0}\\u{8}\\u{b}\\u{1f}\\u{7f}\\u{80}\\u{9f}\\u{200b}\\u{200f}\\u{202a}\\u{202e}\\u{2060}\\u{2064}\\u{2066}\\u{2069}\\u{feff}", + ); + }); + + test("leaves line feed, tab and ordinary text in any script exactly as written, joiners included", () => { + const text = `a\tb\n${ordinary}${scripts.join(" ")}`; + + expect(terminalText(text)).toBe(text); + }); + + test.each(categoryExamples)( + "escapes %s visibly by its code point", + (_name, character, escape) => { + expect(terminalText(`a${character}b`)).toBe(`a${escape}b`); + }, + ); + + test("escapes a hostile title", () => { + expect(terminalText(hostile)).toBe(escaped); + }); +}); + +describe("terminalJson", () => { + test.each(categoryExamples)( + "writes %s as a JSON escape that parses back to it", + (_name, character) => { + const json = terminalJson({ title: `a${character}b` }); + + expect(unsafeIn(json)).toEqual([]); + expect(json).toMatch(/\\u[0-9a-f]{4}/u); + expect(JSON.parse(json)).toEqual({ title: `a${character}b` }); + }, + ); + + test("writes ordinary text in any script, joiners included, as it is", () => { + const value = { scripts }; + + expect(terminalJson(value)).toBe(`${JSON.stringify(value, null, 2)}\n`); + }); + + test("writes no unsafe character raw, and parses back to the same value", () => { + const value = { title: hostile, c1: "\u009B2J", del: "\u007F", ordinary }; + const json = terminalJson(value); + + expect(unsafeIn(json)).toEqual([]); + expect(JSON.parse(json)).toEqual(value); + }); +}); diff --git a/packages/targets/graph-cli/src/emit/text.ts b/packages/core/graph/export/src/text.ts similarity index 97% rename from packages/targets/graph-cli/src/emit/text.ts rename to packages/core/graph/export/src/text.ts index 9df2a91..26b3d56 100644 --- a/packages/targets/graph-cli/src/emit/text.ts +++ b/packages/core/graph/export/src/text.ts @@ -1,4 +1,4 @@ -import { terminalText } from "../terminal-text.ts"; +import { terminalText } from "./terminal-text.ts"; /** * The Mermaid entity code for a character that would end or change the meaning of a label: quotes, statement and edge-label delimiters, the entity marker itself and HTML. diff --git a/packages/core/graph/export/src/view.ts b/packages/core/graph/export/src/view.ts new file mode 100644 index 0000000..b6119c0 --- /dev/null +++ b/packages/core/graph/export/src/view.ts @@ -0,0 +1,66 @@ +import type { ProjectedView } from "@exadev/process-graph-projection"; +import type { Model, OpenIds } from "@exadev/process-graph-schema"; +import { + flowchartToDot, + sequenceToDot, + stateMachineToDot, + swimlaneToDot, +} from "./dot.ts"; +import { + flowchartToMermaid, + sequenceToMermaid, + stateMachineToMermaid, + swimlaneToMermaid, +} from "./mermaid.ts"; +import { terminalJson } from "./terminal-text.ts"; + +/** The text formats a view is written in: Mermaid, Graphviz DOT, and the projection itself as JSON. */ +export const exportFormats = ["mermaid", "dot", "json"] as const; + +/** One of the text formats a view is written in. */ +export type ExportFormat = (typeof exportFormats)[number]; + +/** + * A projected view as text: Mermaid or Graphviz DOT source, or the projection as JSON indented by two spaces and ending in a line break, with every control, format, separator, unassigned and private-use character in it written as a JSON `\u` escape (see `terminalJson`), so the text shows nothing hidden or reordered wherever it is saved or printed and parses back to exactly the projection. This is the one writer of a view's text, so the command line's `show` and a download from the web app give the same text for the same view. + * @param model - The model the view was projected from, for the labels of a sequence diagram's participants. + * @param projected - The view, as graph-projection's `project` gives it. + * @param format - The format to write. + * @returns The text. + */ +export const viewText = ( + model: Model, + projected: ProjectedView, + format: ExportFormat, +): string => { + if (format === "json") { + return terminalJson(projected.view); + } + const mermaid = format === "mermaid"; + if (projected.notation === "flowchart") { + return mermaid + ? flowchartToMermaid(projected.view) + : flowchartToDot(projected.view); + } + if (projected.notation === "swimlane") { + return mermaid + ? swimlaneToMermaid(projected.view) + : swimlaneToDot(projected.view); + } + if (projected.notation === "stateMachine") { + return mermaid + ? stateMachineToMermaid(projected.view) + : stateMachineToDot(projected.view); + } + const labels = new Map( + model.participants.map((participant) => [ + participant.id, + participant.label, + ]), + ); + // An actor a loaded model names without declaring has no label of its own, so it is shown by its identifier, as the swimlane projection labels its lane. + const participantLabel = (id: string): string => labels.get(id) ?? id; + + return mermaid + ? sequenceToMermaid(projected.view, participantLabel) + : sequenceToDot(projected.view, participantLabel); +}; diff --git a/packages/core/graph/export/src/view.unit.test.ts b/packages/core/graph/export/src/view.unit.test.ts new file mode 100644 index 0000000..cc6e713 --- /dev/null +++ b/packages/core/graph/export/src/view.unit.test.ts @@ -0,0 +1,136 @@ +import { + project, + toFlowchart, + toSequence, + toStateMachine, + toSwimlane, +} from "@exadev/process-graph-projection"; +import type { Model, OpenIds } from "@exadev/process-graph-schema"; +import { describe, expect, test } from "vitest"; +import { + flowchartToDot, + sequenceToDot, + stateMachineToDot, + swimlaneToDot, +} from "./dot.ts"; +import { + flowchartToMermaid, + sequenceToMermaid, + stateMachineToMermaid, + swimlaneToMermaid, +} from "./mermaid.ts"; +import { isUnsafeCharacter } from "./terminal-text.ts"; +import { viewText } from "./view.ts"; + +/** An order taken by a shop, followed through its states, with a payment provider the model names as an actor without declaring it, as a model loaded from a store can. */ +const model: Model = { + participants: [ + { id: "customer", label: "Customer", kind: "role" }, + { id: "shop", label: "Shop", kind: "system" }, + { id: "order", label: "Order", kind: "entity" }, + ], + events: [], + actions: [], + artefacts: [], + elements: [ + { kind: "activity", id: "submit", label: "Submit", actor: "customer" }, + { kind: "activity", id: "pack", label: "Pack", actor: "shop" }, + { kind: "activity", id: "pay", label: "Pay", actor: "provider" }, + { kind: "state", id: "placed", label: "Placed" }, + { kind: "state", id: "packed", label: "Packed" }, + ], + transitions: [ + { id: "t1", source: "submit", target: "pack" }, + { id: "t2", source: "pack", target: "pay" }, + { id: "s1", source: "placed", target: "packed" }, + ], + fragments: [], + projections: [], +}; + +describe("viewText", () => { + test("writes each notation in Mermaid and DOT with that notation's emitter", () => { + const flowchart = toFlowchart(model); + const swimlane = toSwimlane(model); + const states = toStateMachine(model, "order"); + + expect( + viewText(model, { notation: "flowchart", view: flowchart }, "mermaid"), + ).toBe(flowchartToMermaid(flowchart)); + expect( + viewText(model, { notation: "flowchart", view: flowchart }, "dot"), + ).toBe(flowchartToDot(flowchart)); + expect( + viewText(model, { notation: "swimlane", view: swimlane }, "mermaid"), + ).toBe(swimlaneToMermaid(swimlane)); + expect( + viewText(model, { notation: "swimlane", view: swimlane }, "dot"), + ).toBe(swimlaneToDot(swimlane)); + expect( + viewText(model, { notation: "stateMachine", view: states }, "mermaid"), + ).toBe(stateMachineToMermaid(states)); + expect( + viewText(model, { notation: "stateMachine", view: states }, "dot"), + ).toBe(stateMachineToDot(states)); + }); + + test("labels a sequence's lifelines with their participants' labels, and an undeclared actor by its identifier", () => { + const sequence = toSequence(model, ["t1", "t2"]); + const labels: Readonly> = { + customer: "Customer", + shop: "Shop", + provider: "provider", + }; + const label = (id: string): string => labels[id] ?? "unexpected"; + + expect( + viewText(model, { notation: "sequence", view: sequence }, "mermaid"), + ).toBe(sequenceToMermaid(sequence, label)); + expect( + viewText(model, { notation: "sequence", view: sequence }, "dot"), + ).toBe(sequenceToDot(sequence, label)); + expect( + viewText(model, { notation: "sequence", view: sequence }, "mermaid"), + ).toContain("participant p2 as provider"); + }); + + test("writes the projection itself as JSON, indented and ending in a line break", () => { + const projected = project(model, { id: "lanes", notation: "swimlane" }); + const text = viewText(model, projected, "json"); + + expect(text.endsWith("}\n")).toBe(true); + expect(text).toContain('\n "lanes": ['); + const parsed: unknown = JSON.parse(text); + expect(parsed).toEqual(toSwimlane(model)); + }); + + test("writes every unsafe character of a hostile title as a JSON escape, and parses back to the same projection", () => { + // A C1 control sequence introducer, a right-to-left override, a zero-width space format character, a line separator, a private-use character and ESC. + const hostile = "Pay\u009B2J \u202Edrawkcab\u200B\u2028\uE000\u001B[0m"; + const titled: Model = { + ...model, + elements: model.elements.map((element) => + element.id === "pay" && element.kind === "activity" + ? { ...element, label: hostile } + : element, + ), + }; + const projected = project(titled, { id: "lanes", notation: "swimlane" }); + + const text = viewText(titled, projected, "json"); + + expect(Array.from(text).filter(isUnsafeCharacter)).toEqual([]); + for (const escape of [ + "\\u009b", + "\\u202e", + "\\u200b", + "\\u2028", + "\\ue000", + "\\u001b", + ]) { + expect(text).toContain(escape); + } + const parsed: unknown = JSON.parse(text); + expect(parsed).toEqual(projected.view); + }); +}); diff --git a/packages/core/graph/export/stryker.config.ts b/packages/core/graph/export/stryker.config.ts new file mode 100644 index 0000000..6429b27 --- /dev/null +++ b/packages/core/graph/export/stryker.config.ts @@ -0,0 +1,3 @@ +import { packageStrykerConfig } from "../../../../stryker.shared.ts"; + +export default packageStrykerConfig(); diff --git a/packages/core/graph/export/test/package.smoke.test.ts b/packages/core/graph/export/test/package.smoke.test.ts new file mode 100644 index 0000000..c39da54 --- /dev/null +++ b/packages/core/graph/export/test/package.smoke.test.ts @@ -0,0 +1,29 @@ +import { project } from "@exadev/process-graph-projection"; +import { defineModel } from "@exadev/process-graph-schema"; +import { expect, test } from "vitest"; +// Deliberately imports the built dist/, not src/: this is the check that catches a wrong exports map or a build misconfiguration that a unit test importing straight from src/ never touches. +import { terminalText, viewText } from "../dist/index.js"; + +test("the built package writes a projected view as Mermaid, DOT and JSON", () => { + const model = defineModel({ + participants: [{ id: "shop", label: "Shop", kind: "system" }], + events: [], + actions: [], + artefacts: [], + elements: [ + { kind: "activity", id: "pack", label: "Pack", actor: "shop" }, + { kind: "activity", id: "ship", label: "Ship", actor: "shop" }, + ], + transitions: [{ id: "t1", source: "pack", target: "ship" }], + fragments: [], + projections: [], + }); + const projected = project(model, { id: "flow", notation: "flowchart" }); + + expect(viewText(model, projected, "mermaid")).toContain('n0["Pack"]'); + expect(viewText(model, projected, "dot")).toContain('"pack" -> "ship"'); + expect(JSON.parse(viewText(model, projected, "json"))).toEqual( + projected.view, + ); + expect(terminalText("a\u001bb")).toBe("a\\u{1b}b"); +}); diff --git a/packages/core/graph/export/tsconfig.build.json b/packages/core/graph/export/tsconfig.build.json new file mode 100644 index 0000000..93341fb --- /dev/null +++ b/packages/core/graph/export/tsconfig.build.json @@ -0,0 +1,20 @@ +{ + "extends": "./tsconfig.json", + "compilerOptions": { + "composite": true, + "noEmit": false, + "declaration": true, + "declarationMap": false, + "rootDir": "src", + "outDir": "dist", + "rewriteRelativeImportExtensions": true + }, + "references": [ + { + "path": "../schema/tsconfig.build.json" + }, + { + "path": "../projection/tsconfig.build.json" + } + ] +} diff --git a/packages/core/graph/export/tsconfig.json b/packages/core/graph/export/tsconfig.json new file mode 100644 index 0000000..5645111 --- /dev/null +++ b/packages/core/graph/export/tsconfig.json @@ -0,0 +1,9 @@ +{ + "extends": "../../../../tsconfig.base.json", + "compilerOptions": { + "lib": ["ES2024"], + "types": [] + }, + "include": ["src/**/*.ts"], + "exclude": ["src/**/*.test.ts", "src/test-support/**"] +} diff --git a/packages/core/graph/export/tsconfig.node.json b/packages/core/graph/export/tsconfig.node.json new file mode 100644 index 0000000..8a460f4 --- /dev/null +++ b/packages/core/graph/export/tsconfig.node.json @@ -0,0 +1,8 @@ +{ + "extends": "../../../../tsconfig.base.json", + "compilerOptions": { + "lib": ["ES2024"], + "types": ["node"] + }, + "include": ["src/**/*.ts", "test/**/*.ts", "scripts/**/*.ts", "*.config.ts"] +} diff --git a/packages/core/graph/export/vitest.config.ts b/packages/core/graph/export/vitest.config.ts new file mode 100644 index 0000000..01b9391 --- /dev/null +++ b/packages/core/graph/export/vitest.config.ts @@ -0,0 +1,16 @@ +import { defineConfig } from "vitest/config"; + +export default defineConfig({ + test: { + projects: [ + { + extends: true, + test: { name: "unit", include: ["src/**/*.test.ts"] }, + }, + { + extends: true, + test: { name: "smoke", include: ["test/package.smoke.test.ts"] }, + }, + ], + }, +}); diff --git a/packages/core/graph/projection/README.md b/packages/core/graph/projection/README.md index 0a7614f..94b62a6 100644 --- a/packages/core/graph/projection/README.md +++ b/packages/core/graph/projection/README.md @@ -6,13 +6,14 @@ A private workspace package, not published on its own: it is bundled into [`proc Projections of the process model from `@exadev/process-graph-schema` into each notation. Every projection is a pure function of the model plus the extra input its notation needs, and none of them reads or preserves layout. `src/` imports nothing Node-only. -| Function | Extra input | Result | -| -------------------------------- | ------------------------ | --------------------------------------------------------------- | -| `handoffs(model)` | none | Transitions between elements whose actors differ. | -| `toFlowchart(model)` | none | Nodes (process, decision, terminator, bar) and guarded edges. | -| `toSwimlane(model)` | none | The flowchart plus lanes by actor, unassigned nodes, hand-offs. | -| `toStateMachine(model, subject)` | a subject entity | States and transitions with decisions folded into guards. | -| `toSequence(model, scenario)` | a path of transition ids | Lifelines in order of first appearance, messages and fragments. | +| Function | Extra input | Result | +| -------------------------------- | ------------------------ | ------------------------------------------------------------------------------------------------------------------------- | +| `handoffs(model)` | none | Transitions between elements whose actors differ. | +| `toFlowchart(model)` | none | Nodes (process, decision, terminator, bar) and guarded edges. | +| `toSwimlane(model)` | none | The flowchart plus lanes by actor, unassigned nodes, hand-offs. | +| `toStateMachine(model, subject)` | a subject entity | States and transitions with decisions folded into guards. | +| `toSequence(model, scenario)` | a path of transition ids | Lifelines in order of first appearance, messages and fragments. | +| `project(model, projection)` | a declared projection | The view the projection's notation gives, as `{ notation, view }` (a `ProjectedView`), following its subject or scenario. | ```ts import { handoffs, toSequence } from "@exadev/process-graph-projection"; diff --git a/packages/core/graph/projection/src/index.ts b/packages/core/graph/projection/src/index.ts index 7430e41..7db65af 100644 --- a/packages/core/graph/projection/src/index.ts +++ b/packages/core/graph/projection/src/index.ts @@ -36,6 +36,11 @@ export { type TransitionChange, type TransitionFieldChange, } from "./diff.ts"; +export { + project, + type ProjectedView, + type ViewsByNotation, +} from "./project.ts"; export { diffFlowchart, diffSwimlane } from "./inverse-flowchart.ts"; export { diffStateMachine } from "./inverse-state-machine.ts"; export { diffSequence } from "./inverse-sequence.ts"; diff --git a/packages/core/graph/projection/src/project.ts b/packages/core/graph/projection/src/project.ts new file mode 100644 index 0000000..4fbab23 --- /dev/null +++ b/packages/core/graph/projection/src/project.ts @@ -0,0 +1,50 @@ +import type { IdSpace, Model, Projection } from "@exadev/process-graph-schema"; +import { toFlowchart, type Flowchart } from "./flowchart.ts"; +import { toSequence, type SequenceDiagram } from "./sequence.ts"; +import { toStateMachine, type StateMachine } from "./state-machine.ts"; +import { toSwimlane, type Swimlane } from "./swimlane.ts"; + +/** The view each projection notation produces. */ +export interface ViewsByNotation { + readonly flowchart: Flowchart; + readonly swimlane: Swimlane; + readonly stateMachine: StateMachine; + readonly sequence: SequenceDiagram; +} + +/** A projection request together with the view it produced, discriminated by notation. */ +export type ProjectedView = { + readonly [N in keyof ViewsByNotation]: { + readonly notation: N; + readonly view: ViewsByNotation[N]; + }; +}[keyof ViewsByNotation]; + +/** + * Projects a model into the notation a projection names, with the input it records (a subject, a scenario). + * @param model - The model to project. + * @param projection - One of the model's declared projections, or one built the same way. + * @returns The notation and its view. + */ +export const project = ( + model: Model, + projection: Projection, +): ProjectedView => { + if (projection.notation === "stateMachine") { + return { + notation: "stateMachine", + view: toStateMachine(model, projection.subject), + }; + } + if (projection.notation === "sequence") { + return { + notation: "sequence", + view: toSequence(model, projection.scenario), + }; + } + + // A flowchart and a swimlane share one variant, which records nothing beyond its notation. + return projection.notation === "swimlane" + ? { notation: "swimlane", view: toSwimlane(model) } + : { notation: "flowchart", view: toFlowchart(model) }; +}; diff --git a/packages/core/graph/projection/src/project.unit.test.ts b/packages/core/graph/projection/src/project.unit.test.ts new file mode 100644 index 0000000..75dc70f --- /dev/null +++ b/packages/core/graph/projection/src/project.unit.test.ts @@ -0,0 +1,40 @@ +import { describe, expect, test } from "vitest"; +import { toFlowchart } from "./flowchart.ts"; +import { project } from "./project.ts"; +import { toSequence } from "./sequence.ts"; +import { toStateMachine } from "./state-machine.ts"; +import { toSwimlane } from "./swimlane.ts"; +import { order } from "./test-support/order.ts"; + +describe("project", () => { + test("projects each declared projection into its notation, following its subject or scenario", () => { + const [, , state, sequence] = order.projections; + + expect( + order.projections.map( + (projection) => project(order, projection).notation, + ), + ).toEqual(["flowchart", "swimlane", "stateMachine", "sequence"]); + expect(state === undefined ? undefined : project(order, state)).toEqual({ + notation: "stateMachine", + view: toStateMachine(order, "order"), + }); + expect( + sequence === undefined ? undefined : project(order, sequence), + ).toEqual({ + notation: "sequence", + view: toSequence(order, ["t1", "t2", "t3", "t4", "t5", "t7"]), + }); + }); + + test("tells a flowchart from a swimlane, which record nothing else", () => { + expect(project(order, { id: "f", notation: "flowchart" })).toEqual({ + notation: "flowchart", + view: toFlowchart(order), + }); + expect(project(order, { id: "s", notation: "swimlane" })).toEqual({ + notation: "swimlane", + view: toSwimlane(order), + }); + }); +}); diff --git a/packages/core/graph/sync/AGENTS.md b/packages/core/graph/sync/AGENTS.md new file mode 120000 index 0000000..42061c0 --- /dev/null +++ b/packages/core/graph/sync/AGENTS.md @@ -0,0 +1 @@ +README.md \ No newline at end of file diff --git a/packages/core/graph/sync/CLAUDE.md b/packages/core/graph/sync/CLAUDE.md new file mode 120000 index 0000000..42061c0 --- /dev/null +++ b/packages/core/graph/sync/CLAUDE.md @@ -0,0 +1 @@ +README.md \ No newline at end of file diff --git a/packages/core/graph/sync/README.md b/packages/core/graph/sync/README.md new file mode 100644 index 0000000..0c8d043 --- /dev/null +++ b/packages/core/graph/sync/README.md @@ -0,0 +1,60 @@ +# @exadev/process-graph-sync + +[![GitHub](https://img.shields.io/badge/GitHub-181717?logo=github&logoColor=white)](https://github.com/ExaDev/process-graph) [![CI](https://img.shields.io/github/actions/workflow/status/ExaDev/process-graph/ci.yml?branch=main)](https://github.com/ExaDev/process-graph/actions) + +A private workspace package, not published on its own: it is bundled into [`process-graph`](../../../targets/process-graph/README.md), which exposes it as `process-graph/sync`. + +Syncs work from `GraphSource`s ([`graph-source-contract`](../source-contract/README.md)) into a `GraphStore` ([`graph-store-contract`](../store-contract/README.md)). The command line's `sync` and the web app's sync both call it, so a sync is written the same way wherever it runs. `src/` imports nothing Node-only. + +```ts +import { syncScopes } from "@exadev/process-graph-sync"; + +const result = await syncScopes({ + store, + sources: (prefix) => (prefix === "github" ? github : undefined), + modelId: "acme", + scopes: ["github:acme", "github:partner/sdk"], + filter: { states: ["open"], labels: ["bug"] }, + signal, + onProgress: (progress) => { + record(progress.checkpoint); + }, +}); +``` + +## What a sync writes + +Scopes are grouped by their source prefix and read one source after another, in the order each source first appears. Each page is merged into the stored model through `items.upsert` as soon as it is read: participants, bound activities, transitions, relations (`child-of`, `assigned-to`, `part-of`) and run state. A model the store does not have is created by the first write. + +A store rejects a write that leaves a transition or relation naming an element or participant the model lacks, and a source may report a dependency or a sub-issue whose other end a later page brings. A transition or relation is therefore held back until the model has both of its ends, and written with the page that completes it. One whose end never arrives is written with the last page, where the store's rejection names the missing reference. + +## Progress and resuming + +After every page, `onProgress` receives the pages written, the revision of the last write, the distinct participants, activities, transitions and relations written, the gaps met and a `SyncCheckpoint`. A checkpoint names the model, the scopes, the filter, the source to read next and the cursor after the last page written within it; passing it back as `resume` reads only what is left. A source's cursor is valid only with the scopes and filter it was issued for, which is why the checkpoint carries both. `syncCheckpointSchema` is its Zod schema, for a consumer that stores it. The final progress has no checkpoint. + +A failure part way leaves every page before it written. A source that fails while it is read, an abort included, ends the sync with an `InterruptedSyncError`: its message and cause are the source's failure, `resume` is the checkpoint that continues the failing source after the last page read from it, `remaining` the scopes of the sources after it that were not reached, and `saved` whether any page was written. A store's failure is thrown as the store threw it. + +## Filters + +`filter` is the source contract's `SyncFilter`: run states, labels and an order. It is passed to every source the sync reads and recorded in every checkpoint. A source that cannot apply a filter to a scope refuses the sync with `BAD_REQUEST`; the GitHub issues adapter does so for a Projects v2 board. + +## Functions + +| Export | What it is | +| ---------------------- | -------------------------------------------------------------------------------------------------------------------------- | +| `syncScopes(request)` | Syncs the scopes into the store and resolves with the final progress. | +| `scopesBySource` | Groups source-qualified scopes by source prefix, in the order a sync reads them. | +| `syncCheckpointSchema` | The Zod schema of a `SyncCheckpoint`. | +| `InterruptedSyncError` | Thrown when a source fails while it is read; carries `resume`, `remaining`, `saved` and the source's failure as its cause. | +| `UnknownSourceError` | Thrown for a scope whose source prefix no reader serves; carries the `prefix`. | +| `InvalidScopeError` | Thrown for a scope that names no source; carries the `scope`. | +| `SyncStore` | The part of the `GraphStore` client a sync calls: `models.load` and `items.upsert`. | + +## Commands + +```sh +pnpm build # tsc -b -> dist/ (ESM + .d.ts) +pnpm test # unit tests against the source and store contracts' in-memory fakes +pnpm test:smoke # the built dist/ exercised as a real artefact +pnpm test:mutation # Stryker +``` diff --git a/packages/core/graph/sync/eslint.config.ts b/packages/core/graph/sync/eslint.config.ts new file mode 100644 index 0000000..4c43201 --- /dev/null +++ b/packages/core/graph/sync/eslint.config.ts @@ -0,0 +1,9 @@ +import { defineConfig } from "eslint/config"; +import { packageLintConfig } from "../../../../eslint.shared.ts"; + +export default defineConfig( + packageLintConfig({ + tsconfigRootDir: import.meta.dirname, + isomorphic: true, + }), +); diff --git a/packages/core/graph/sync/package.json b/packages/core/graph/sync/package.json new file mode 100644 index 0000000..b292253 --- /dev/null +++ b/packages/core/graph/sync/package.json @@ -0,0 +1,72 @@ +{ + "name": "@exadev/process-graph-sync", + "version": "0.0.0", + "description": "Syncs a GraphSource into a GraphStore page by page: participants, bound activities, transitions, relations and run state, resumable from a checkpoint, with an optional typed filter.", + "private": true, + "type": "module", + "repository": { + "type": "git", + "url": "git+https://github.com/ExaDev/process-graph.git", + "directory": "packages/core/graph/sync" + }, + "homepage": "https://github.com/ExaDev/process-graph/tree/main/packages/core/graph/sync#readme", + "bugs": { + "url": "https://github.com/ExaDev/process-graph/issues" + }, + "exports": { + ".": { + "types": "./dist/index.d.ts", + "import": "./dist/index.js" + } + }, + "types": "./dist/index.d.ts", + "files": [ + "dist" + ], + "sideEffects": false, + "license": "MIT", + "scripts": { + "build": "turbo run _build", + "_build": "tsc -b tsconfig.build.json", + "lint": "turbo run _lint", + "_lint": "eslint . --cache --max-warnings 0", + "_lint:fix": "eslint . --fix --cache --max-warnings 0", + "typecheck": "turbo run _typecheck _typecheck:attw", + "_typecheck": "tsc -p tsconfig.json --noEmit && tsc -p tsconfig.node.json --noEmit", + "_typecheck:attw": "attw --pack --profile esm-only", + "test": "turbo run _test", + "_test": "vitest run --project unit", + "test:coverage": "turbo run _test:coverage", + "_test:coverage": "vitest run --project unit --coverage", + "test:smoke": "turbo run _test:smoke", + "_test:smoke": "vitest run --project smoke", + "test:mutation": "turbo run _test:mutation", + "_test:mutation": "stryker run stryker.config.ts", + "prepublishOnly": "pnpm run lint && pnpm run typecheck && pnpm run test && pnpm run build && publint && attw --pack --profile esm-only" + }, + "keywords": [ + "process-model", + "sync", + "github-issues", + "resumable" + ], + "devDependencies": { + "@arethetypeswrong/cli": "catalog:", + "@orpc/server": "catalog:", + "@stryker-mutator/core": "catalog:", + "@stryker-mutator/typescript-checker": "catalog:", + "@stryker-mutator/vitest-runner": "catalog:", + "@types/node": "catalog:", + "@vitest/coverage-v8": "catalog:", + "eslint": "catalog:", + "publint": "catalog:", + "typescript": "catalog:", + "vitest": "catalog:" + }, + "dependencies": { + "@exadev/process-graph-source-contract": "workspace:*", + "@exadev/process-graph-store-contract": "workspace:*", + "@orpc/client": "catalog:", + "zod": "catalog:" + } +} diff --git a/packages/core/graph/sync/src/checkpoint.ts b/packages/core/graph/sync/src/checkpoint.ts new file mode 100644 index 0000000..7343db1 --- /dev/null +++ b/packages/core/graph/sync/src/checkpoint.ts @@ -0,0 +1,17 @@ +import { syncFilterSchema } from "@exadev/process-graph-source-contract"; +import { z } from "zod"; + +/** + * Where a sync stopped: the model, every scope it was asked for in order, the filter it read them with, the prefix of the source to read next, and the cursor after the last page it wrote within that source's scopes (absent to read them from the start). Everything before it is in the store, so a sync started from it reads only what is left. A source's cursor is valid only with the scopes and filter it was issued for, which is why both are kept with it. + */ +export const syncCheckpointSchema = z + .strictObject({ + modelId: z.string().min(1), + scopes: z.array(z.string().min(1)).readonly(), + filter: syncFilterSchema.optional(), + source: z.string().min(1), + after: z.string().min(1).optional(), + }) + .readonly(); + +export type SyncCheckpoint = z.infer; diff --git a/packages/core/graph/sync/src/index.ts b/packages/core/graph/sync/src/index.ts new file mode 100644 index 0000000..6cce107 --- /dev/null +++ b/packages/core/graph/sync/src/index.ts @@ -0,0 +1,19 @@ +/** + * Syncs work from `GraphSource`s into a `GraphStore`: page by page, resumable from a checkpoint, with an optional typed filter, writing participants, bound activities, transitions, relations and run state. One function serves every consumer, so the command line and the web app write a sync the same way. + */ +export { + InterruptedSyncError, + syncScopes, + UnknownSourceError, + type SyncInterruption, + type SourceLookup, + type SyncProgress, + type SyncRequest, + type SyncStore, +} from "./sync.ts"; +export { syncCheckpointSchema, type SyncCheckpoint } from "./checkpoint.ts"; +export { + InvalidScopeError, + scopesBySource, + type SourceScopes, +} from "./scopes.ts"; diff --git a/packages/core/graph/sync/src/scopes.ts b/packages/core/graph/sync/src/scopes.ts new file mode 100644 index 0000000..7a9c203 --- /dev/null +++ b/packages/core/graph/sync/src/scopes.ts @@ -0,0 +1,48 @@ +/** The separator between a scope's source prefix and the rest, as in `github:owner/repo`. */ +const sourceSeparator = ":"; + +/** The scopes of one source, in the order they were given. */ +export interface SourceScopes { + /** The source's identifier prefix, such as `github`. */ + readonly prefix: string; + readonly scopes: readonly string[]; +} + +/** Thrown for a scope that names no source: it has no prefix before a colon. */ +export class InvalidScopeError extends Error { + /** The scope as it was given. */ + readonly scope: string; + + /** + * @param scope - The scope that names no source. + */ + constructor(scope: string) { + super( + `"${scope}" is not a scope: write :, such as github:owner/repo`, + ); + this.name = "InvalidScopeError"; + this.scope = scope; + } +} + +/** + * Groups source-qualified scopes by their source prefix, keeping each prefix where it first appears and each source's scopes in the order given. A sync reads the groups in this order, one source after another. + * @param scopes - Source-qualified scopes, such as `github:owner` or `github:owner/repo`. + * @returns One group per source prefix. + * @throws InvalidScopeError for a scope with nothing before its first colon, or no colon at all. + */ +export const scopesBySource = ( + scopes: readonly string[], +): readonly SourceScopes[] => { + const groups = new Map(); + for (const scope of scopes) { + const separator = scope.indexOf(sourceSeparator); + if (separator <= 0) { + throw new InvalidScopeError(scope); + } + const prefix = scope.slice(0, separator); + groups.set(prefix, [...(groups.get(prefix) ?? []), scope]); + } + + return [...groups].map(([prefix, grouped]) => ({ prefix, scopes: grouped })); +}; diff --git a/packages/core/graph/sync/src/sync.ts b/packages/core/graph/sync/src/sync.ts new file mode 100644 index 0000000..3ec7104 --- /dev/null +++ b/packages/core/graph/sync/src/sync.ts @@ -0,0 +1,341 @@ +import type { + Gap, + GraphSourceClient, + Relation, + SourceTransition, + SyncFilter, + SyncPage, +} from "@exadev/process-graph-source-contract"; +import type { GraphStoreClient } from "@exadev/process-graph-store-contract"; +import { ORPCError } from "@orpc/client"; +import type { SyncCheckpoint } from "./checkpoint.ts"; +import { scopesBySource } from "./scopes.ts"; + +/** Finds the source that reads scopes with an identifier prefix, such as `github`, or nothing when no source does. */ +export type SourceLookup = (prefix: string) => GraphSourceClient | undefined; + +/** The part of the `GraphStore` client a sync calls: loading the model it merges into and upserting each page. */ +export interface SyncStore { + readonly v1: { + readonly models: Pick; + readonly items: Pick; + }; +} + +/** How far a sync has got. Each count is of distinct identifiers (relations by kind and both ends) written by this sync. */ +export interface SyncProgress { + /** Pages read and written. */ + readonly pages: number; + /** The revision the last write took. */ + readonly revision: number; + readonly participants: number; + /** Bound activities. */ + readonly activities: number; + readonly transitions: number; + /** Relations: sub-issues to their parents, assignees, repositories to their owners. */ + readonly relations: number; + /** The parts of the scope the sources could not read, in the order met. */ + readonly gaps: readonly Gap[]; + /** Where to resume; absent once every scope has been read. */ + readonly checkpoint: SyncCheckpoint | undefined; +} + +/** What to sync and where to. */ +export interface SyncRequest { + readonly store: SyncStore; + readonly sources: SourceLookup; + readonly modelId: string; + /** Source-qualified scopes, read in order, one source after another. */ + readonly scopes: readonly string[]; + /** Narrows and orders what each scope lists. It is recorded in every checkpoint, since a cursor is valid only with the filter it was issued under. */ + readonly filter?: SyncFilter | undefined; + /** A checkpoint from an interrupted sync of the same model, scopes and filter: the sync carries on from it. */ + readonly resume?: SyncCheckpoint | undefined; + /** Ends the sync between or during reads; what was written stays written. */ + readonly signal?: AbortSignal | undefined; + /** Called after every page is written, with the progress so far. */ + readonly onProgress?: ((progress: SyncProgress) => void) | undefined; +} + +/** Thrown when a scope names a source no reader serves. */ +export class UnknownSourceError extends Error { + /** The source prefix no reader serves. */ + readonly prefix: string; + + /** + * @param prefix - The source prefix no reader serves. + */ + constructor(prefix: string) { + super(`No source reads "${prefix}" scopes.`); + this.name = "UnknownSourceError"; + this.prefix = prefix; + } +} + +/** Where a sync a source interrupted carries on, as data. */ +export interface SyncInterruption { + /** The checkpoint that continues the failing source: its prefix, after the cursor of the last page read from it, or from the start of its scopes when none was read. */ + readonly resume: SyncCheckpoint; + /** The scopes of the sources after the failing one, which were not read. */ + readonly remaining: readonly string[]; + /** Whether this sync wrote any pages before the failure. */ + readonly saved: boolean; +} + +/** + * Thrown by `syncScopes` when a source fails while it is being read: opening its stream or reading a page. Its message is the source's, its cause is what the source threw, and where to carry on is data in `resume` and `remaining`. A failure of the store, or of the request itself (an unknown source, a malformed scope), is thrown as it is. + */ +export class InterruptedSyncError extends Error implements SyncInterruption { + readonly resume: SyncCheckpoint; + + readonly remaining: readonly string[]; + + readonly saved: boolean; + + /** + * @param interruption - Where the sync carries on. + * @param cause - What the source threw. + */ + constructor(interruption: SyncInterruption, cause: unknown) { + super(cause instanceof Error ? cause.message : String(cause), { cause }); + this.name = "InterruptedSyncError"; + this.resume = interruption.resume; + this.remaining = interruption.remaining; + this.saved = interruption.saved; + } +} + +/** The oRPC client options carrying a signal, leaving the key out when there is none, since the options type does not accept an explicit `undefined`. */ +const callOptions = ( + signal: AbortSignal | undefined, +): { readonly signal?: AbortSignal } => + signal === undefined ? {} : { signal }; + +/** The identifiers of the elements and participants a model in the store already has, or none when the store has no such model. */ +const storedIdentifiers = async ( + store: SyncStore, + modelId: string, + signal: AbortSignal | undefined, +): Promise> => { + try { + const stored = await store.v1.models.load({ modelId }, callOptions(signal)); + + return new Set([ + ...stored.model.elements.map((element) => element.id), + ...stored.model.participants.map((participant) => participant.id), + ]); + } catch (error) { + if (error instanceof ORPCError && error.code === "NOT_FOUND") { + return new Set(); + } + throw error; + } +}; + +const relationKey = (relation: Readonly): string => + JSON.stringify([relation.kind, relation.from, relation.to]); + +/** Splits what is waiting to be written into what can be written now, whose ends are all known, and what must wait for a later page. */ +const settle = ( + pending: readonly T[], + endsOf: (item: T) => readonly string[], + known: ReadonlySet, + last: boolean, +): { readonly ready: readonly T[]; readonly held: readonly T[] } => { + const ready = pending.filter( + (item) => last || endsOf(item).every((id) => known.has(id)), + ); + + return { ready, held: pending.filter((item) => !ready.includes(item)) }; +}; + +/** + * Reads the scopes from their sources page by page and merges each page into the stored model as soon as it is read: participants, bound activities, transitions, relations and run state. Progress is kept as it is made, so a sync that is cancelled, rate limited or interrupted leaves everything it read in the store, and a checkpoint to carry on from. + * + * A store rejects a write that leaves a transition or a relation naming an element or participant the model lacks, and a source may report a dependency or a sub-issue whose other end a later page brings. A transition or relation is therefore held back until the model has both of its ends, and written with the page that completes it. One whose end never arrives is written with the last page, where the store's rejection says which reference is missing. + * @param request - The store, sources, model, scopes, an optional filter and checkpoint, the signal and a progress listener. + * @returns The final progress, with no checkpoint. + * @throws InvalidScopeError for a scope that names no source; UnknownSourceError for a scope whose source no reader serves; an Error for a checkpoint naming a source none of the scopes use, and for a sync of no scopes; InterruptedSyncError, after the pages before it are written, when a source fails while it is read, an abort included; and whatever the store throws. + */ +export const syncScopes = async ( + request: SyncRequest, +): Promise => { + const { store, sources, modelId, scopes, filter, resume, signal } = request; + const groups = scopesBySource(scopes).map((group) => { + const client = sources(group.prefix); + if (client === undefined) { + throw new UnknownSourceError(group.prefix); + } + + return { ...group, client }; + }); + if (groups.length === 0) { + throw new Error("A sync needs at least one scope."); + } + const startGroup = + resume === undefined + ? 0 + : groups.findIndex((group) => group.prefix === resume.source); + if (startGroup === -1) { + throw new Error( + "The checkpoint names a source that none of the scopes use; start the sync again.", + ); + } + + const known = new Set(await storedIdentifiers(store, modelId, signal)); + const written = { + participants: new Set(), + activities: new Set(), + transitions: new Set(), + relations: new Set(), + }; + const gaps: Gap[] = []; + let heldTransitions: readonly SourceTransition[] = []; + let heldRelations: readonly Relation[] = []; + let pages = 0; + let latest: SyncProgress | undefined; + + const checkpointAt = ( + source: string, + after: string | undefined, + ): SyncCheckpoint => ({ + modelId, + scopes, + ...(filter === undefined ? {} : { filter }), + source, + ...(after === undefined ? {} : { after }), + }); + + const write = async ( + page: SyncPage, + checkpoint: SyncCheckpoint | undefined, + ): Promise => { + for (const record of [...page.participants, ...page.activities]) { + known.add(record.id); + } + const last = checkpoint === undefined; + const transitions = settle( + [...heldTransitions, ...page.transitions], + (transition) => [transition.source, transition.target], + known, + last, + ); + const relations = settle( + [...heldRelations, ...page.relations], + (relation) => [relation.from, relation.to], + known, + last, + ); + heldTransitions = transitions.held; + heldRelations = relations.held; + const { revision } = await store.v1.items.upsert( + { + modelId, + items: { + participants: page.participants, + activities: page.activities, + transitions: transitions.ready, + relations: relations.ready, + overlay: page.overlay, + }, + }, + callOptions(signal), + ); + pages += 1; + for (const participant of page.participants) { + written.participants.add(participant.id); + } + for (const activity of page.activities) { + written.activities.add(activity.id); + } + for (const transition of transitions.ready) { + written.transitions.add(transition.id); + } + for (const relation of relations.ready) { + written.relations.add(relationKey(relation)); + } + gaps.push(...page.gaps); + latest = { + pages, + revision, + participants: written.participants.size, + activities: written.activities.size, + transitions: written.transitions.size, + relations: written.relations.size, + gaps: [...gaps], + checkpoint, + }; + request.onProgress?.(latest); + }; + + /** Reads the sources in order, each only after the one before has finished, since a source's failure must stop the ones after it. */ + const readFrom = async (index: number): Promise => { + const group = groups[index]; + if (group === undefined) { + return; + } + // The cursor after the last page read from this source, which is where it carries on if it fails. + let after = index === startGroup ? resume?.after : undefined; + const following = groups[index + 1]; + const interrupted = (cause: unknown): InterruptedSyncError => + new InterruptedSyncError( + { + resume: checkpointAt(group.prefix, after), + remaining: groups.slice(index + 1).flatMap((next) => next.scopes), + saved: pages > 0, + }, + cause, + ); + const stream = await group.client.v1.sync + .stream( + { + scope: group.scopes, + ...(after === undefined ? {} : { after }), + ...(filter === undefined ? {} : { filter }), + }, + callOptions(signal), + ) + .catch((cause: unknown) => { + throw interrupted(cause); + }); + const iterator = stream[Symbol.asyncIterator](); + + /** Writes each page the source yields, in order, each only after the one before is written. */ + const drain = async (): Promise => { + const next = await iterator.next().catch((cause: unknown) => { + throw interrupted(cause); + }); + if (next.done === true) { + return; + } + const page = next.value; + await write( + page, + page.next !== undefined + ? checkpointAt(group.prefix, page.next) + : following === undefined + ? undefined + : checkpointAt(following.prefix, undefined), + ); + after = page.next; + await drain(); + }; + try { + await drain(); + } catch (error) { + // Closing the stream stops the source reading pages nobody will write. + await iterator.return(); + throw error; + } + await readFrom(index + 1); + }; + await readFrom(startGroup); + + if (latest === undefined) { + throw new Error( + "unreachable: every source answers a sync with at least one page", + ); + } + + return latest; +}; diff --git a/packages/core/graph/sync/src/sync.unit.test.ts b/packages/core/graph/sync/src/sync.unit.test.ts new file mode 100644 index 0000000..72b9dbc --- /dev/null +++ b/packages/core/graph/sync/src/sync.unit.test.ts @@ -0,0 +1,519 @@ +import { + createFakeGraphSource, + GraphSourceError, + type GraphSourcePager, + type SyncFilter, +} from "@exadev/process-graph-source-contract"; +import { describe, expect, test } from "vitest"; +import { InvalidScopeError, scopesBySource } from "./scopes.ts"; +import { + InterruptedSyncError, + syncScopes, + UnknownSourceError, + type SyncProgress, +} from "./sync.ts"; +import { + apiPage, + fakeScopes, + lookup, + newStore, + pageOf, +} from "./test-support/fixtures.ts"; + +const fakeGitHub = createFakeGraphSource({ + source: "github", + scopes: fakeScopes, +}); + +const modelId = "github:acme"; + +describe("syncScopes", () => { + test("writes every page as it is read, reports progress after each, and ends with no checkpoint", async () => { + const store = newStore(); + const progress: SyncProgress[] = []; + + const result = await syncScopes({ + store, + sources: lookup(fakeGitHub), + modelId, + scopes: ["github:acme"], + onProgress: (step) => { + progress.push(step); + }, + }); + + expect( + progress.map((step) => [step.pages, step.checkpoint?.after]), + ).toEqual([ + [1, "github%3Aacme@0"], + [2, undefined], + ]); + expect(result).toEqual({ + pages: 2, + revision: progress.at(-1)?.revision, + participants: 4, + activities: 4, + transitions: 2, + relations: 5, + gaps: [], + checkpoint: undefined, + }); + const stored = await store.v1.models.load({ modelId }); + expect( + stored.model.elements.map((element) => element.id).toSorted(), + ).toEqual([ + "github:acme/api#1", + "github:acme/api#2", + "github:acme/web#1", + "github:acme/web#9", + ]); + expect(stored.overlay).toMatchObject({ + "github:acme/api#1": "done", + "github:acme/web#9": "open", + }); + }); + + test("keeps the relations a source reports, so the store answers hierarchy queries", async () => { + const store = newStore(); + + await syncScopes({ + store, + sources: lookup(fakeGitHub), + modelId, + scopes: ["github:acme"], + }); + + const { relations } = await store.v1.relations.list({ modelId }); + expect(relations).toEqual([ + { kind: "part-of", from: "github:acme/api", to: "github:@acme" }, + { kind: "assigned-to", from: "github:acme/api#1", to: "github:@alice" }, + { kind: "child-of", from: "github:acme/api#2", to: "github:acme/web#9" }, + { kind: "part-of", from: "github:acme/web", to: "github:@acme" }, + { kind: "child-of", from: "github:acme/web#1", to: "github:acme/web#9" }, + ]); + expect( + await store.v1.query.children({ modelId, element: "github:acme/web#9" }), + ).toEqual({ children: ["github:acme/api#2", "github:acme/web#1"] }); + expect( + await store.v1.query.ancestors({ modelId, element: "github:acme/api#2" }), + ).toEqual({ + ancestors: [{ element: "github:acme/web#9", distance: 1 }], + }); + }); + + test("holds a transition and a relation back until the page that brings their other end", async () => { + const store = newStore(); + const early = pageOf([ + { + id: "github:acme/web#1", + label: "Build the page", + repository: "github:acme/web", + state: "open", + blockedBy: ["github:acme/api#2"], + parent: "github:acme/api#2", + }, + ]); + const late = pageOf([ + { + id: "github:acme/api#2", + label: "Build the endpoint", + repository: "github:acme/api", + state: "open", + }, + ]); + const progress: SyncProgress[] = []; + + await syncScopes({ + store, + sources: lookup( + createFakeGraphSource({ + source: "github", + scopes: { "github:acme": [early, late] }, + }), + ), + modelId, + scopes: ["github:acme"], + onProgress: (step) => { + progress.push(step); + }, + }); + + expect(progress.map((step) => step.transitions)).toEqual([0, 1]); + // The first page's own repository membership is written at once; its sub-issue waits for the parent the second page brings. + expect(progress.map((step) => step.relations)).toEqual([ + early.relations.length - 1, + early.relations.length + late.relations.length, + ]); + const stored = await store.v1.models.load({ modelId }); + expect(stored.model.transitions).toEqual([ + { + id: "github:acme/api#2->github:acme/web#1", + source: "github:acme/api#2", + target: "github:acme/web#1", + }, + ]); + expect( + await store.v1.query.children({ modelId, element: "github:acme/api#2" }), + ).toEqual({ children: ["github:acme/web#1"] }); + }); + + test("writes a relation whose end never arrives with the last page, where the store names the missing reference", async () => { + const orphan = pageOf([ + { + id: "github:acme/web#1", + label: "Build the page", + repository: "github:acme/web", + state: "open", + parent: "github:acme/web#404", + }, + ]); + + await expect( + syncScopes({ + store: newStore(), + sources: lookup( + createFakeGraphSource({ + source: "github", + scopes: { "github:acme": [orphan] }, + }), + ), + modelId, + scopes: ["github:acme"], + }), + ).rejects.toMatchObject({ + code: "UNPROCESSABLE_CONTENT", + data: { + issues: [expect.objectContaining({ code: "unknown-reference" })], + }, + }); + }); + + test("resumes from a checkpoint with only the pages after it", async () => { + const store = newStore(); + const read: string[] = []; + const counting: GraphSourcePager = { + source: "github", + page: async (request) => { + read.push(request.cursor ?? "start"); + + return fakeGitHub.page(request); + }, + }; + await store.v1.items.upsert({ + modelId, + items: { + participants: apiPage.participants, + activities: apiPage.activities, + transitions: apiPage.transitions, + overlay: apiPage.overlay, + }, + }); + + const result = await syncScopes({ + store, + sources: lookup(counting), + modelId, + scopes: ["github:acme"], + resume: { + modelId, + scopes: ["github:acme"], + source: "github", + after: "github%3Aacme@0", + }, + }); + + expect(read).toEqual(["github%3Aacme@0"]); + expect(result.pages).toBe(1); + const stored = await store.v1.models.load({ modelId }); + expect( + stored.model.transitions.map((transition) => transition.id), + ).toContain("github:acme/api#2->github:acme/web#1"); + }); + + test("reads with the filter it is given and records it in every checkpoint", async () => { + const filters: (SyncFilter | undefined)[] = []; + const recording: GraphSourcePager = { + source: "github", + page: async (request) => { + filters.push(request.filter); + + return fakeGitHub.page(request); + }, + }; + const filter: SyncFilter = { + states: ["open"], + labels: ["bug"], + order: { field: "updated", direction: "descending" }, + }; + const progress: SyncProgress[] = []; + + await syncScopes({ + store: newStore(), + sources: lookup(recording), + modelId, + scopes: ["github:acme"], + filter, + onProgress: (step) => { + progress.push(step); + }, + }); + + expect(filters).toEqual([filter, filter]); + // The fake source's cursor carries the filter it was issued under, as the GitHub adapter's does. + const checkpoint = progress[0]?.checkpoint; + expect(checkpoint?.filter).toEqual(filter); + expect(checkpoint?.after?.startsWith("github%3Aacme@0;")).toBe(true); + }); + + test("keeps the pages before a failure, reports where to resume, and passes the failure on", async () => { + const store = newStore(); + const progress: SyncProgress[] = []; + const limited: GraphSourcePager = { + source: "github", + page: async (request) => { + if (request.cursor !== undefined) { + throw new GraphSourceError({ + kind: "rate-limited", + message: "rate limited", + resetAt: "2100-01-01T00:00:00.000Z", + }); + } + + return fakeGitHub.page(request); + }, + }; + + await expect( + syncScopes({ + store, + sources: lookup(limited), + modelId, + scopes: ["github:acme"], + onProgress: (step) => { + progress.push(step); + }, + }), + ).rejects.toMatchObject({ + name: "InterruptedSyncError", + message: "rate limited", + resume: { + modelId, + scopes: ["github:acme"], + source: "github", + after: "github%3Aacme@0", + }, + remaining: [], + saved: true, + cause: { code: "TOO_MANY_REQUESTS" }, + }); + expect(progress.at(-1)?.checkpoint).toEqual({ + modelId, + scopes: ["github:acme"], + source: "github", + after: "github%3Aacme@0", + }); + const stored = await store.v1.models.load({ modelId }); + expect(stored.model.elements.map((element) => element.id)).toEqual([ + "github:acme/api#1", + "github:acme/api#2", + ]); + }); + + test("reports a source that fails before its first page with the cursor it was resumed from, and the sources it did not reach", async () => { + const refusing: GraphSourcePager = { + source: "github", + page: async () => + Promise.reject( + new GraphSourceError({ kind: "unauthorised", message: "bad token" }), + ), + }; + const jira = createFakeGraphSource({ + source: "jira", + scopes: { "jira:ACME": [] }, + }); + + const failure = await syncScopes({ + store: newStore(), + sources: lookup(refusing, jira), + modelId, + scopes: ["github:acme", "jira:ACME", "github:partner"], + resume: { + modelId, + scopes: ["github:acme", "jira:ACME", "github:partner"], + source: "github", + after: "github%3Aacme@0", + }, + }).then( + () => undefined, + (error: unknown) => error, + ); + + expect(failure).toBeInstanceOf(InterruptedSyncError); + expect(failure).toMatchObject({ + message: "bad token", + resume: { + source: "github", + scopes: ["github:acme", "jira:ACME", "github:partner"], + after: "github%3Aacme@0", + }, + remaining: ["jira:ACME"], + saved: false, + cause: { code: "UNAUTHORIZED" }, + }); + }); + + test("passes a store's refusal on as it is, not as an interruption of the source", async () => { + const orphan = pageOf([ + { + id: "github:acme/web#1", + label: "Build the page", + repository: "github:acme/web", + state: "open", + parent: "github:acme/web#404", + }, + ]); + + const failure = await syncScopes({ + store: newStore(), + sources: lookup( + createFakeGraphSource({ + source: "github", + scopes: { "github:acme": [orphan] }, + }), + ), + modelId, + scopes: ["github:acme"], + }).then( + () => undefined, + (error: unknown) => error, + ); + + expect(failure).not.toBeInstanceOf(InterruptedSyncError); + expect(failure).toMatchObject({ code: "UNPROCESSABLE_CONTENT" }); + }); + + test("reads the sources in order and checkpoints the start of the next source between them", async () => { + const progress: SyncProgress[] = []; + const jira = createFakeGraphSource({ + source: "jira", + scopes: { + "jira:ACME": [ + pageOf([ + { + id: "jira:ACME-1", + label: "Plan", + repository: "jira:ACME/board", + state: "open", + }, + ]), + ], + }, + }); + + await syncScopes({ + store: newStore(), + sources: lookup(fakeGitHub, jira), + modelId, + scopes: ["github:acme", "jira:ACME"], + onProgress: (step) => { + progress.push(step); + }, + }); + + expect(progress.map((step) => step.checkpoint)).toEqual([ + { + modelId, + scopes: ["github:acme", "jira:ACME"], + source: "github", + after: "github%3Aacme@0", + }, + { modelId, scopes: ["github:acme", "jira:ACME"], source: "jira" }, + undefined, + ]); + }); + + test("reports the parts of the scope the source could not read", async () => { + const result = await syncScopes({ + store: newStore(), + sources: lookup(fakeGitHub), + modelId: "github:acme github:ghost", + scopes: ["github:acme", "github:ghost"], + }); + + expect(result.gaps).toEqual([ + { + target: "github:ghost", + reason: "not-found", + message: "github:ghost does not exist", + }, + ]); + }); + + test("stops when its signal is aborted, keeping what it wrote", async () => { + const store = newStore(); + const controller = new AbortController(); + + const outcome = syncScopes({ + store, + sources: lookup(fakeGitHub), + modelId, + scopes: ["github:acme"], + signal: controller.signal, + onProgress: () => { + controller.abort(); + }, + }); + + await expect(outcome).rejects.toThrow(); + const stored = await store.v1.models.load({ modelId }); + expect(stored.model.elements).toHaveLength(2); + }); + + test("refuses a scope whose source no reader serves, and one that names no source", async () => { + await expect( + syncScopes({ + store: newStore(), + sources: lookup(fakeGitHub), + modelId: "linear:team", + scopes: ["linear:team"], + }), + ).rejects.toThrow(UnknownSourceError); + await expect( + syncScopes({ + store: newStore(), + sources: lookup(fakeGitHub), + modelId, + scopes: ["acme/app"], + }), + ).rejects.toThrow(InvalidScopeError); + }); + + test("refuses a checkpoint for a source none of the scopes use", async () => { + await expect( + syncScopes({ + store: newStore(), + sources: lookup(fakeGitHub), + modelId, + scopes: ["github:acme"], + resume: { modelId, scopes: ["github:acme"], source: "jira" }, + }), + ).rejects.toThrow("The checkpoint names a source"); + }); +}); + +describe("scopesBySource", () => { + test("groups scopes by source, each source where it first appears, its scopes in order", () => { + expect( + scopesBySource(["github:a", "jira:X", "github:b/c", "jira:Y"]), + ).toEqual([ + { prefix: "github", scopes: ["github:a", "github:b/c"] }, + { prefix: "jira", scopes: ["jira:X", "jira:Y"] }, + ]); + }); + + test("names a scope with nothing before its colon", () => { + expect(() => scopesBySource([":a"])).toThrow( + '":a" is not a scope: write :, such as github:owner/repo', + ); + }); +}); diff --git a/packages/core/graph/sync/src/test-support/fixtures.ts b/packages/core/graph/sync/src/test-support/fixtures.ts new file mode 100644 index 0000000..e9fe46a --- /dev/null +++ b/packages/core/graph/sync/src/test-support/fixtures.ts @@ -0,0 +1,155 @@ +import { + createGraphSourceRouter, + type FakePage, + type GraphSourcePager, + type Relation, +} from "@exadev/process-graph-source-contract"; +import { + createFakeGraphStore, + createGraphStoreRouter, + type GraphStoreClient, +} from "@exadev/process-graph-store-contract"; +import { createRouterClient } from "@orpc/server"; +import type { SourceLookup } from "../sync.ts"; + +/** One issue of the fake GitHub source. */ +export interface Issue { + readonly id: string; + readonly label: string; + readonly repository: string; + readonly state: "done" | "open"; + readonly blockedBy?: readonly string[]; + /** The parent issue, for a sub-issue. */ + readonly parent?: string; + /** The people assigned to it, as account participants. */ + readonly assignees?: readonly string[]; +} + +const ownerOf = (repository: string): string => + `github:@${repository.slice(repository.indexOf(":") + 1, repository.indexOf("/"))}`; + +/** + * A page of the fake GitHub source, built as the real adapter builds one: each issue a bound activity whose actor is its repository, each repository `part-of` its owner, each blocker a transition to the issue it blocks, each sub-issue `child-of` its parent, each assignee a role participant the issue is `assigned-to`, and the state in the overlay. + * @param issues - The page's issues. + * @returns The page. + */ +export const pageOf = (issues: readonly Issue[]): FakePage => { + const repositories = [...new Set(issues.map((issue) => issue.repository))]; + const assignees = [ + ...new Set(issues.flatMap((issue) => issue.assignees ?? [])), + ]; + const relations: Relation[] = [ + ...repositories.map((repository): Relation => ({ + kind: "part-of", + from: repository, + to: ownerOf(repository), + })), + ...issues.flatMap((issue): Relation[] => [ + ...(issue.parent === undefined + ? [] + : [{ kind: "child-of", from: issue.id, to: issue.parent } as const]), + ...(issue.assignees ?? []).map((person): Relation => ({ + kind: "assigned-to", + from: issue.id, + to: person, + })), + ]), + ]; + + return { + participants: [ + ...[...new Set(repositories.map(ownerOf))].map((id) => ({ + id, + label: id.slice(id.indexOf("@") + 1), + kind: "system" as const, + })), + ...repositories.map((id) => ({ + id, + label: id.slice(id.indexOf(":") + 1), + kind: "system" as const, + })), + ...assignees.map((id) => ({ + id, + label: id.slice(id.indexOf("@") + 1), + kind: "role" as const, + })), + ], + activities: issues.map((issue) => ({ + kind: "activity", + id: issue.id, + label: issue.label, + source: issue.id, + actor: issue.repository, + })), + transitions: issues.flatMap((issue) => + (issue.blockedBy ?? []).map((blocker) => ({ + id: `${blocker}->${issue.id}`, + source: blocker, + target: issue.id, + })), + ), + relations, + overlay: Object.fromEntries(issues.map((issue) => [issue.id, issue.state])), + gaps: [], + }; +}; + +/** The first page of the fake `github:acme` scope: the API repository's issues, the second a sub-issue of an epic the next page brings. */ +export const apiPage: FakePage = pageOf([ + { + id: "github:acme/api#1", + label: "Design the schema", + repository: "github:acme/api", + state: "done", + assignees: ["github:@alice"], + }, + { + id: "github:acme/api#2", + label: "Build the endpoint", + repository: "github:acme/api", + state: "open", + blockedBy: ["github:acme/api#1"], + parent: "github:acme/web#9", + }, +]); + +/** The second page of the fake `github:acme` scope: the web repository's issues, the epic among them, with a blocker in the API repository. */ +const webPage: FakePage = pageOf([ + { + id: "github:acme/web#1", + label: "Build the page", + repository: "github:acme/web", + state: "open", + blockedBy: ["github:acme/api#2"], + parent: "github:acme/web#9", + }, + { + id: "github:acme/web#9", + label: "Launch", + repository: "github:acme/web", + state: "open", + }, +]); + +/** The fake GitHub source's scopes. */ +export const fakeScopes: Readonly> = { + "github:acme": [apiPage, webPage], +}; + +/** A fresh in-memory store, through its oRPC client. */ +export const newStore = (): GraphStoreClient => + createRouterClient(createGraphStoreRouter(createFakeGraphStore())); + +/** + * The source lookup that serves each pager's scopes, by its source prefix, through its oRPC client. + * @param pagers - The pagers. + */ +export const lookup = + (...pagers: readonly GraphSourcePager[]): SourceLookup => + (prefix) => { + const pager = pagers.find((candidate) => candidate.source === prefix); + + return pager === undefined + ? undefined + : createRouterClient(createGraphSourceRouter(pager)); + }; diff --git a/packages/core/graph/sync/stryker.config.ts b/packages/core/graph/sync/stryker.config.ts new file mode 100644 index 0000000..6429b27 --- /dev/null +++ b/packages/core/graph/sync/stryker.config.ts @@ -0,0 +1,3 @@ +import { packageStrykerConfig } from "../../../../stryker.shared.ts"; + +export default packageStrykerConfig(); diff --git a/packages/core/graph/sync/test/package.smoke.test.ts b/packages/core/graph/sync/test/package.smoke.test.ts new file mode 100644 index 0000000..735274f --- /dev/null +++ b/packages/core/graph/sync/test/package.smoke.test.ts @@ -0,0 +1,73 @@ +import { + createFakeGraphSource, + createGraphSourceRouter, +} from "@exadev/process-graph-source-contract"; +import { + createFakeGraphStore, + createGraphStoreRouter, +} from "@exadev/process-graph-store-contract"; +import { createRouterClient } from "@orpc/server"; +import { expect, test } from "vitest"; +// Deliberately imports the built dist/, not src/: this is the check that catches a wrong exports map or a build misconfiguration that a unit test importing straight from src/ never touches. +import { syncCheckpointSchema, syncScopes } from "../dist/index.js"; + +test("the built package syncs a source's pages and relations into a store", async () => { + const store = createRouterClient( + createGraphStoreRouter(createFakeGraphStore()), + ); + const source = createRouterClient( + createGraphSourceRouter( + createFakeGraphSource({ + source: "fake", + scopes: { + "fake:team": [ + { + participants: [ + { id: "fake:team", label: "team", kind: "system" }, + ], + activities: [ + { + kind: "activity", + id: "fake:1", + source: "fake:1", + label: "Parent", + actor: "fake:team", + }, + { + kind: "activity", + id: "fake:2", + source: "fake:2", + label: "Child", + actor: "fake:team", + }, + ], + transitions: [], + relations: [{ kind: "child-of", from: "fake:2", to: "fake:1" }], + overlay: { "fake:1": "open", "fake:2": "open" }, + gaps: [], + }, + ], + }, + }), + ), + ); + + const result = await syncScopes({ + store, + sources: (prefix) => (prefix === "fake" ? source : undefined), + modelId: "team", + scopes: ["fake:team"], + }); + + expect(result.relations).toBe(1); + expect( + await store.v1.query.children({ modelId: "team", element: "fake:1" }), + ).toEqual({ children: ["fake:2"] }); + expect( + syncCheckpointSchema.parse({ + modelId: "team", + scopes: ["fake:team"], + source: "fake", + }), + ).toEqual({ modelId: "team", scopes: ["fake:team"], source: "fake" }); +}); diff --git a/packages/core/graph/sync/tsconfig.build.json b/packages/core/graph/sync/tsconfig.build.json new file mode 100644 index 0000000..ba95c1f --- /dev/null +++ b/packages/core/graph/sync/tsconfig.build.json @@ -0,0 +1,20 @@ +{ + "extends": "./tsconfig.json", + "compilerOptions": { + "composite": true, + "noEmit": false, + "declaration": true, + "declarationMap": false, + "rootDir": "src", + "outDir": "dist", + "rewriteRelativeImportExtensions": true + }, + "references": [ + { + "path": "../source-contract/tsconfig.build.json" + }, + { + "path": "../store-contract/tsconfig.build.json" + } + ] +} diff --git a/packages/core/graph/sync/tsconfig.json b/packages/core/graph/sync/tsconfig.json new file mode 100644 index 0000000..8895dd1 --- /dev/null +++ b/packages/core/graph/sync/tsconfig.json @@ -0,0 +1,9 @@ +{ + "extends": "../../../../tsconfig.base.json", + "compilerOptions": { + "lib": ["ES2024", "WebWorker"], + "types": [] + }, + "include": ["src/**/*.ts"], + "exclude": ["src/**/*.test.ts", "src/test-support/**"] +} diff --git a/packages/core/graph/sync/tsconfig.node.json b/packages/core/graph/sync/tsconfig.node.json new file mode 100644 index 0000000..8a460f4 --- /dev/null +++ b/packages/core/graph/sync/tsconfig.node.json @@ -0,0 +1,8 @@ +{ + "extends": "../../../../tsconfig.base.json", + "compilerOptions": { + "lib": ["ES2024"], + "types": ["node"] + }, + "include": ["src/**/*.ts", "test/**/*.ts", "scripts/**/*.ts", "*.config.ts"] +} diff --git a/packages/core/graph/sync/vitest.config.ts b/packages/core/graph/sync/vitest.config.ts new file mode 100644 index 0000000..01b9391 --- /dev/null +++ b/packages/core/graph/sync/vitest.config.ts @@ -0,0 +1,16 @@ +import { defineConfig } from "vitest/config"; + +export default defineConfig({ + test: { + projects: [ + { + extends: true, + test: { name: "unit", include: ["src/**/*.test.ts"] }, + }, + { + extends: true, + test: { name: "smoke", include: ["test/package.smoke.test.ts"] }, + }, + ], + }, +}); diff --git a/packages/targets/graph-cli/README.md b/packages/targets/graph-cli/README.md index 16e20a9..b425596 100644 --- a/packages/targets/graph-cli/README.md +++ b/packages/targets/graph-cli/README.md @@ -17,29 +17,31 @@ process-graph show --view swimlane --format mermaid ## Commands -| Command | What it does | -| --------------------------------------------------------------------------------------------------------- | --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | -| `sync [--after ]` | Reads the scopes from their sources and merges them into the model in one write; prints what it wrote and any gaps. | -| `ready` | The work that can start now. | -| `blocked ` | The open work an element is waiting on: the ends of its blocked-by chain. | -| `impact ` | The live work that depends on an element. | -| `path` | The critical path: the longest chain of work still ahead, each element counting one step. | -| `cycles` | The cycles no decision closes. | -| `handoffs` | The transitions from one actor's work to another's, such as a dependency across repositories or organisations. | -| `why ` | The rule that decided an element's status, its needs and artefacts, and the blocked-by chain behind it. | -| `show [--view ] [--format ]` | Draws the model. Views: `flowchart` (default), `swimlane`, `state`, `sequence`. Formats: `mermaid` (default), `dot`, `json` (the projection itself). | -| `add participant