Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
9 changes: 8 additions & 1 deletion content/docs/guide/ci-cd-pipeline.md
Original file line number Diff line number Diff line change
Expand Up @@ -63,10 +63,17 @@ keys in both directions, so a job added or removed without touching this table i
(This section used to open with a hard-coded count and list a seventh job, `dev-server`, that had
been deleted three months earlier — [#3451](https://github.com/objectstack-ai/objectui/issues/3451).)

The **What it runs** column is pinned one level further down, by command
([#3653](https://github.com/objectstack-ai/objectui/issues/3653)): every first-party command a job
runs — a `node scripts/*.mjs` invocation, a root `package.json` script, or a `turbo run` task — must
be named in that job's row, and a row may not name one its job does not run. Until that pin landed
this page was judged job by job only, so a `run:` step added to an existing job left every check on
it green — which is how two of `type-check`'s gates came to be missing from this column.

| Job key | Appears as | What it runs | When |
|---|---|---|---|
| `changeset-check` | Changeset Fixed Group Check | `scripts/check-changeset-fixed.mjs` — every workspace package must be in the changeset `fixed` group or explicitly ignored. It checks group *membership*; it does **not** check whether the PR added a changeset. | Every run |
| `type-check` | Type Check | `scripts/check-type-check-coverage.mjs`, then `pnpm check:spec-symbols`, then `pnpm type-check:scripts`, then `pnpm type-check`, then `pnpm type-check:vitest-setup`. The coverage guard runs first because turbo silently skips packages that have no `type-check` script, so a package without one would otherwise read as passing (#2911). `pnpm type-check:scripts` (`tsconfig.scripts.json`) covers `scripts/**/*.ts`, which `pnpm type-check` cannot reach at all — `scripts/` has no package.json, so turbo never walks it, and the coverage guard decides coverage per *package*. Until [#3494](https://github.com/objectstack-ai/objectui/issues/3494) that left the pin tests in `scripts/__tests__/` — including the one pinning this very page — compiled by nothing. `pnpm type-check:vitest-setup` (`tsconfig.vitest-setup.json`) closes the same gap for the four repo-root `vitest.setup.*` files, uncovered until [#3515](https://github.com/objectstack-ai/objectui/issues/3515); it runs *last*, after `pnpm type-check`, because `vitest.setup.dom.tsx` side-effect-imports four `@object-ui/*` packages and resolves them through the declarations that turbo's `^build` produces. | Every run |
| `type-check` | Type Check | `scripts/check-type-check-coverage.mjs`, then `pnpm check:spec-symbols`, then `pnpm check:i18n-keys`, then `pnpm check:i18n-drift`, then `pnpm type-check:scripts`, then `pnpm type-check`, then `pnpm type-check:vitest-setup`. The coverage guard runs first because turbo silently skips packages that have no `type-check` script, so a package without one would otherwise read as passing (#2911). The two locale gates sit in the middle because both parse the sources with `typescript`: they need the install and nothing built. `pnpm check:i18n-keys` fails when a `t()` call site asks for a key the `en` pack does not define ([#3530](https://github.com/objectstack-ai/objectui/issues/3530)); `pnpm check:i18n-drift` fails when a change to an `en` string is not accompanied by the nine translation packs ([#3650](https://github.com/objectstack-ai/objectui/issues/3650)), and it is why this job's checkout sets `fetch-depth: 0` — it diffs against the merge base, which a depth-1 clone cannot resolve. `pnpm type-check:scripts` (`tsconfig.scripts.json`) covers `scripts/**/*.ts`, which `pnpm type-check` cannot reach at all — `scripts/` has no package.json, so turbo never walks it, and the coverage guard decides coverage per *package*. Until [#3494](https://github.com/objectstack-ai/objectui/issues/3494) that left the pin tests in `scripts/__tests__/` — including the one pinning this very page — compiled by nothing. `pnpm type-check:vitest-setup` (`tsconfig.vitest-setup.json`) closes the same gap for the four repo-root `vitest.setup.*` files, uncovered until [#3515](https://github.com/objectstack-ai/objectui/issues/3515); it runs *last*, after `pnpm type-check`, because `vitest.setup.dom.tsx` side-effect-imports four `@object-ui/*` packages and resolves them through the declarations that turbo's `^build` produces. | Every run |
| `test` | Test (shard N/4) | `pnpm test --shard=N/4` across a 4-runner matrix with `fail-fast: false`, so every shard reports its own failures. No coverage instrumentation — v8 adds 40–100% overhead. | **Pull requests only** |
| `test-coverage` | Test (coverage) | One unsharded `pnpm test:coverage`, uploaded to Codecov. Nothing blocks on it, which is why it is not sharded. | **Push only** |
| `e2e` | Build & E2E | Builds the console with `vite build` (`VITE_BASE_PATH=/console/`), verifies the artifact, then `pnpm test:e2e --project=chromium`. Uploads the Playwright report on failure. | Every run |
Expand Down
170 changes: 166 additions & 4 deletions scripts/__tests__/ci-cd-pipeline-doc.test.ts
Original file line number Diff line number Diff line change
Expand Up @@ -268,18 +268,18 @@ describe('ci-cd-pipeline.md — ci.yml job table', () => {

const JOB_TABLE_HEADER = '| Job key | Appears as | What it runs | When |';

/** First column of the job table, in page order. */
function docJobRows(): { key: string; appearsAs: string }[] {
/** The job table's rows in page order: job key, `Appears as`, `What it runs`. */
function docJobRows(): { key: string; appearsAs: string; runs: string }[] {
const section = coreCiSection();
const at = section.indexOf(JOB_TABLE_HEADER);
expect(at, `the job table must keep the header \`${JOB_TABLE_HEADER}\``).toBeGreaterThan(-1);
const lines = section.slice(at).split('\n').slice(1);
const rows: { key: string; appearsAs: string }[] = [];
const rows: { key: string; appearsAs: string; runs: string }[] = [];
for (const line of lines) {
if (!line.startsWith('|')) break;
if (/^\|[\s|:-]+\|$/.test(line)) continue; // separator
const cells = line.split('|').slice(1, -1).map((c) => c.trim());
rows.push({ key: cells[0].replace(/`/g, '').trim(), appearsAs: cells[1] ?? '' });
rows.push({ key: cells[0].replace(/`/g, '').trim(), appearsAs: cells[1] ?? '', runs: cells[2] ?? '' });
}
return rows;
}
Expand Down Expand Up @@ -374,4 +374,166 @@ describe('ci-cd-pipeline.md — ci.yml job table', () => {
'still denies it (objectui#3451).',
).toBe(true);
});

/**
* objectui#3653: every pin above judges `ci.yml` at JOB granularity — the set of
* job keys, the `name:` each one reports under, the absence of a count. None of
* them reads what a job *runs*, so a `run:` step added to an existing job left
* this whole file green. Two such steps had landed unlisted: `pnpm
* check:i18n-keys` (objectui#3530, PR #3547) and `pnpm check:i18n-drift`
* (objectui#3650, PR #3659) both ran in the `type-check` job while its row on the
* page still listed five commands.
*
* Both directions are asserted, because the column is a claim in both: a command
* the job runs and the row omits is a contributor who cannot learn from this page
* that a gate exists; a command the row names and the job does not run is the
* objectui#3451 shape one level down — a page advertising a guardrail that is not
* there.
*
* What counts as a command is deliberately narrower than "every step", and the
* boundary is *derived* rather than hand-listed: a step counts when it names
* something this repository owns — a `scripts/*.mjs` file, a script in the root
* `package.json`, or a `turbo run` task. Environment setup drops out on its own
* because it names none of those (`corepack enable`, `pnpm --version`, `pnpm
* install --frozen-lockfile`, `pnpm exec playwright install`, `pnpm --filter …
* exec vite build`), which keeps this column a summary of the gates rather than a
* transcript of the YAML: the `e2e` job's artifact check and Playwright cache are
* real steps that no reader of this page needs enumerated.
*
* The hole that leaves, stated so nobody mistakes it for coverage: a gate written
* as an inline shell block names no first-party command and is invisible here.
* Gates in this repository land as a root `package.json` script or a
* `scripts/*.mjs` file — both covered — and that is the only reason the narrower
* rule is enough.
*
* Steps are read from `run:` values only, never from the surrounding YAML. The
* `type-check` job's comments alone mention `pnpm type-check`, `turbo run
* type-check` and `pnpm check:i18n-drift`; a scan of the raw block would take all
* three for steps and this pin would then be describing its own comments.
*
* Whether a given step still EXISTS in `ci.yml` is pinned where that step was
* introduced — `check-i18n-call-site-keys.test.ts` and
* `check-i18n-en-drift.test.ts` each hold their own, as do
* `scripts-type-check.test.ts` and `vitest-setup-type-check.test.ts`. This block
* does not repeat those assertions; it pins the *pairing* between the YAML and
* this page, which is the part nothing owned.
*/
describe('what each job runs', () => {
const rootScripts = new Set(
Object.keys(
(
JSON.parse(fs.readFileSync(path.join(repoRoot, 'package.json'), 'utf8')) as {
scripts: Record<string, string>;
}
).scripts,
),
);

/** One job's YAML block, from its key line up to the next thing at that indent. */
function jobBlock(key: string): string {
const body = ciWorkflow.slice(ciWorkflow.search(/^jobs:[ \t]*$/m));
const at = body.search(new RegExp(`^ {2}${key}:[ \\t]*$`, 'm'));
expect(at, `ci.yml must still define a \`${key}:\` job`).toBeGreaterThan(-1);
const rest = body.slice(at + 1);
// A job's own comments are indented four spaces or more; the two-space ones
// introduce the *next* job, so stopping at any two-space line is right.
const next = rest.search(/^ {2}\S/m);
return next === -1 ? rest : rest.slice(0, next);
}

/** Every `run:` step body in a job block — single-line and block scalar alike. */
function runSteps(block: string): string[] {
const lines = block.split('\n');
const steps: string[] = [];
for (let i = 0; i < lines.length; i++) {
const at = lines[i].indexOf('run:');
// `run:` must be the key of the line, not text inside another value.
if (at === -1 || !/^[\s-]*$/.test(lines[i].slice(0, at))) continue;
const value = lines[i].slice(at + 'run:'.length).trim();
if (!/^[|>][-+]?$/.test(value)) {
steps.push(value);
continue;
}
const body: string[] = [];
for (let j = i + 1; j < lines.length; j++) {
if (lines[j].trim() === '') continue;
if (lines[j].search(/\S/) <= at) break;
body.push(lines[j].trim());
}
steps.push(body.join('\n'));
}
return steps;
}

/**
* The first-party commands named in a piece of text — applied to a job's `run:`
* bodies on one side and to its `What it runs` cell on the other, so the two
* sides are compared by the same rule rather than by two spellings of it.
*/
function firstPartyCommands(text: string): Set<string> {
const found = new Set<string>();
// A gate that lives in this repo's `scripts/` tree. The `node ` prefix is not
// required: ci.yml writes `node scripts/x.mjs`, the page writes the path.
for (const m of text.matchAll(/scripts\/[\w./-]+\.mjs/g)) found.add(m[0]);
// A root `package.json` script. `install`, `--version`, `exec` and `--filter`
// are not scripts, so the setup steps need no exemption list.
for (const m of text.matchAll(/\bpnpm\s+([\w:.-]+)/g)) {
if (rootScripts.has(m[1])) found.add(`pnpm ${m[1]}`);
}
// The build graph, invoked through the task runner instead of a script.
for (const m of text.matchAll(/\bturbo\s+run\s+([\w:-]+)/g)) found.add(`turbo run ${m[1]}`);
return found;
}

/** `job key -> commands it actually runs`, and the same from the page's table. */
function commandsByJob(): { key: string; ran: Set<string>; named: Set<string> }[] {
return docJobRows().map((row) => ({
key: row.key,
ran: firstPartyCommands(runSteps(jobBlock(row.key)).join('\n')),
named: firstPartyCommands(row.runs),
}));
}

it('names every first-party command the job actually runs', () => {
const jobs = commandsByJob();

// A parser that matched nothing would make both directions vacuously green —
// the exact failure the `dev-server` job demonstrated one level up.
const ran = jobs.reduce((n, j) => n + j.ran.size, 0);
expect(ran, 'the ci.yml `run:` parse found implausibly few first-party commands').toBeGreaterThan(8);

const missing = jobs.flatMap((j) => [...j.ran].filter((c) => !j.named.has(c)).map((c) => `${j.key}: ${c}`));

expect(
missing,
`.github/workflows/ci.yml runs commands that the job table in ` +
`content/docs/guide/ci-cd-pipeline.md does not name:\n` +
missing.map((m) => ` - ${m}`).join('\n') +
`\n\nAdd each one to that job's "What it runs" cell, in the order ci.yml runs it. ` +
`A gate nobody wrote down is a build failure contributors meet without knowing what ` +
`produced it — objectui#3653: two locale gates ran in \`type-check\` unlisted, because ` +
`the pins on this page read job keys and job names but never read the steps.`,
).toEqual([]);
});

it('credits no job with a first-party command it does not run', () => {
const jobs = commandsByJob();

const named = jobs.reduce((n, j) => n + j.named.size, 0);
expect(named, 'the job table parse found implausibly few commands in "What it runs"').toBeGreaterThan(8);

const phantom = jobs.flatMap((j) => [...j.named].filter((c) => !j.ran.has(c)).map((c) => `${j.key}: ${c}`));

expect(
phantom,
`content/docs/guide/ci-cd-pipeline.md's job table credits jobs with commands that ` +
`.github/workflows/ci.yml does not run there:\n` +
phantom.map((p) => ` - ${p}`).join('\n') +
`\n\nEither the step was removed and the cell is stale, or the command runs in a ` +
`different job and belongs in that row. A "What it runs" cell reads as this job's ` +
`gate list, so naming a command for contrast inside it makes the page claim a ` +
`guardrail — the objectui#3451 mistake, one level down.`,
).toEqual([]);
});
});
});
Loading