Skip to content
Closed
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
301 changes: 301 additions & 0 deletions .github/ci-config.test.cjs
Original file line number Diff line number Diff line change
@@ -0,0 +1,301 @@
// Structural regression tests for the GitHub Actions workflow/action YAML
// files touched by this PR.
//
// These files are declarative CI configuration, not application code, so
// there is nothing to unit test in the traditional sense. However, this
// PR introduces several behaviorally meaningful changes (new `develop`
// trigger branches, a new non-blocking security-advisory gate, and
// tightened `if:`/`continue-on-error:` conditions) that are easy to
// silently revert or typo during future edits. Rather than pull in a YAML
// parsing dependency, these tests use plain substring/regex checks on the
// raw file text — the same lightweight approach already used elsewhere in
// this repo (see the "Verify risk register coverage" step in ci.yml and
// scripts/check-new-vulns.cjs's own regex-based YAML scanning).
"use strict";

const { describe, it, expect } = require("vitest");
const fs = require("node:fs");
const path = require("node:path");

const REPO_ROOT = path.join(__dirname, "..");

function readWorkflow(name) {
return fs.readFileSync(
path.join(REPO_ROOT, ".github", "workflows", name),
"utf8",
);
}

function branchesFor(content, triggerName) {
const match = content.match(
new RegExp(`${triggerName}:\\n\\s*branches: \\[([^\\]]+)\\]`),
);
return match ? match[1] : null;
}
Comment on lines +29 to +34

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The branchesFor regex helper only supports inline array format (branches: [rebuild, develop]). If any workflow file is reformatted to use block-style branch lists (as ci.yml already does: branches:\n - main), the regex returns null and tests fail with a confusing Cannot read properties of null error.

While all currently tested files use inline format, this hidden assumption makes the tests brittle against YAML formatting changes. It contradicts the project's own acknowledgment in the test header that "the same lightweight approach" is used elsewhere — since ci.yml already uses a different format.

Suggestion: Either add a fallback regex for block-style branches, or document this limitation explicitly and add a defensive null check with a clear error message.

Suggestion:

Suggested change
function branchesFor(content, triggerName) {
const match = content.match(
new RegExp(`${triggerName}:\\n\\s*branches: \\[([^\\]]+)\\]`),
);
return match ? match[1] : null;
}
function branchesFor(content, triggerName) {
// Try inline array format first: branches: [a, b, c]
const inlineMatch = content.match(
new RegExp(`${triggerName}:\\n\\s*branches: \\[([^\\]]+)\\]`),
);
if (inlineMatch) return inlineMatch[1];
// Fallback: block format (branches:\n - a\n - b)
const blockMatch = content.match(
new RegExp(`${triggerName}:\\n\\s*branches:\\n((?:\\s+-\\s+\\S+\\n?)+)`),
);
if (blockMatch) return blockMatch[1];
return null;
}


describe("ci.yml", () => {
const content = readWorkflow("ci.yml");

it("triggers on pull requests targeting main, rebuild, and develop", () => {
expect(content).toContain(
" pull_request:\n branches:\n - main\n - rebuild\n - develop\n",
);
});

it("still only pushes on main and rebuild (develop excluded from push)", () => {
expect(content).toContain(
" push:\n branches:\n - main\n - rebuild\n",
);
});

it("keeps the pnpm audit step non-blocking and writes JSON for the follow-up check", () => {
expect(content).toContain(
[
" - name: Audit dependencies",
" # GHSA-mp2f-45pm-3cg9 patched locally via patches/decompress@4.2.1.patch.",
" # Remove ignore when upstream publishes decompress@>=4.2.2.",
" continue-on-error: true",
" run: pnpm audit --audit-level=critical --ignore GHSA-mp2f-45pm-3cg9",
].join("\n"),
);
});

it("adds a follow-up step that fails only on new critical advisories", () => {
expect(content).toContain("- name: Check for new critical advisories");
expect(content).toContain(
[
" if: success() || failure()",
" run: |",
" pnpm audit --audit-level=critical --json > /tmp/audit.json 2>/dev/null || true",
" node scripts/check-new-vulns.cjs \\",
" --format pnpm \\",
" --json /tmp/audit.json \\",
" --ignored GHSA-mp2f-45pm-3cg9 \\",
" --min-severity critical",
].join("\n"),
);
});

it("does not fail the workflow when CODECOV_TOKEN is missing for the PR-comment step", () => {
expect(content).toContain(
[
" - name: Post coverage gaps to PR",
].join("\n"),
);
const stepStart = content.indexOf("- name: Post coverage gaps to PR");
Comment on lines +80 to +85

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The test uses toContain with exact leading spaces (" - name: Post coverage gaps to PR" — 6 spaces) to assert the step name exists, but then uses indexOf without leading spaces ("- name: Post coverage gaps to PR") to find the step boundary for further assertions. This inconsistency works today but is fragile: if the same step name appears elsewhere in the file with different indentation (e.g., in a different job), indexOf could locate the wrong occurrence.

Suggestion: Use consistent leading spaces in both toContain and indexOf calls.

Suggestion:

Suggested change
expect(content).toContain(
[
" - name: Post coverage gaps to PR",
].join("\n"),
);
const stepStart = content.indexOf("- name: Post coverage gaps to PR");
expect(content).toContain(
" - name: Post coverage gaps to PR",
);
const stepStart = content.indexOf(" - name: Post coverage gaps to PR");

const stepEnd = content.indexOf(
"run: bash scripts/codecov-pr-comment.sh",
stepStart,
);
const step = content.slice(stepStart, stepEnd);
expect(step).toContain("if: github.event_name == 'pull_request'");
expect(step).toContain("continue-on-error: true");
});

it("no longer marks the SonarQube Scan step as continue-on-error", () => {
expect(content).toContain(
[
" - name: SonarQube Scan",
" id: sonar-scan",
" uses: SonarSource/sonarqube-scan-action",
].join("\n"),
);
});
Comment on lines +95 to +103

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The test title claims to verify that continue-on-error is no longer present on the SonarQube Scan step, but the assertion only checks that three fixed lines exist (- name: SonarQube Scan, id: sonar-scan, uses: SonarSource/...). It does not assert the absence of continue-on-error: true from this step. If someone re-adds continue-on-error: true to this step (a regression the test claims to prevent), the assertion would still pass silently.

Suggestion: Explicitly verify that continue-on-error does not appear between the step name and the next step, or assert that the step's snippet does not contain continue-on-error: true.

Suggestion:

Suggested change
it("no longer marks the SonarQube Scan step as continue-on-error", () => {
expect(content).toContain(
[
" - name: SonarQube Scan",
" id: sonar-scan",
" uses: SonarSource/sonarqube-scan-action",
].join("\n"),
);
});
it("no longer marks the SonarQube Scan step as continue-on-error", () => {
// Verify the step exists as expected
const stepStart = content.indexOf("- name: SonarQube Scan");
expect(stepStart).toBeGreaterThan(-1);
// Check that continue-on-error is NOT present within the SonarQube step
const stepSnippet = content.slice(
stepStart,
content.indexOf("\n - name:", stepStart + 1) // up to the next step
);
expect(stepSnippet).not.toContain("continue-on-error");
});


it("only runs the SonarCloud PR comment job when the scan succeeded", () => {
const jobStart = content.indexOf("sonar-pr-comment:");
const jobEnd = content.indexOf("\n dockerfile:");
expect(jobStart).toBeGreaterThan(-1);
expect(jobEnd).toBeGreaterThan(jobStart);
const job = content.slice(jobStart, jobEnd);
expect(job).toContain(
"if: github.event_name == 'pull_request' && needs.sonar.result == 'success'",
);
});
Comment on lines +105 to +114

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The job boundary is detected by searching for \n dockerfile: as the end marker of the sonar-pr-comment job. This is fragile because:

  1. If a new job is inserted between sonar-pr-comment and dockerfile in ci.yml, the slice will include unrelated content.
  2. If the dockerfile job is renamed (e.g., to Dockerfile-lint), indexOf returns -1 and the test fails cryptically.
  3. If the string \n dockerfile: appears anywhere earlier in the file (e.g., in a comment or step name), the slice will be wrong.

Suggestion: Use a more robust boundary marker, such as searching for the next top-level job key pattern (\n [a-z]+: following the current job) or parsing job-level YAML boundaries.

Suggestion:

Suggested change
it("only runs the SonarCloud PR comment job when the scan succeeded", () => {
const jobStart = content.indexOf("sonar-pr-comment:");
const jobEnd = content.indexOf("\n dockerfile:");
expect(jobStart).toBeGreaterThan(-1);
expect(jobEnd).toBeGreaterThan(jobStart);
const job = content.slice(jobStart, jobEnd);
expect(job).toContain(
"if: github.event_name == 'pull_request' && needs.sonar.result == 'success'",
);
});
it("only runs the SonarCloud PR comment job when the scan succeeded", () => {
const jobStart = content.indexOf("sonar-pr-comment:");
expect(jobStart).toBeGreaterThan(-1);
// Find the next top-level job (lines starting with 2 spaces + word + :)
const afterJob = content.indexOf("\n ", jobStart + 1);
const jobEnd = content.indexOf("\n ", afterJob + 1);
// Or alternatively, use a more specific next-job-name lookup
// const jobEnd = content.indexOf("\n dockerfile:");
if (jobEnd > jobStart) {
const job = content.slice(jobStart, jobEnd);
expect(job).toContain(
"if: github.event_name == 'pull_request' && needs.sonar.result == 'success'",
);
}
});

});

describe("osv-scanner.yml", () => {
const content = readWorkflow("osv-scanner.yml");

it("cancels outdated in-progress runs via a concurrency group", () => {
expect(content).toContain(
[
"concurrency:",
" group: ${{ github.workflow }}-${{ github.event_name }}-${{ github.head_ref || github.ref }}",
" cancel-in-progress: true",
].join("\n"),
);
});

it("scans pull requests and merge groups targeting develop as well as rebuild", () => {
const pullRequest = branchesFor(content, "pull_request");
const mergeGroup = branchesFor(content, "merge_group");
expect(pullRequest).toContain('"rebuild"');
expect(pullRequest).toContain('"develop"');
expect(mergeGroup).toContain('"rebuild"');
expect(mergeGroup).toContain('"develop"');
});

it("does not add develop to the push trigger", () => {
const push = branchesFor(content, "push");
expect(push).toContain('"rebuild"');
expect(push).not.toMatch(/develop/);
});
});

describe("branch-scoped CI workflows include the develop branch on pull_request", () => {
const filesRequiringDevelopOnPR = [
"cli-ci.yml",
"desktop-ci.yml",
"droplet-ci.yml",
"server-ci.yml",
"e2e.yml",
"editorconfig-ci.yml",
"codeql.yml",
];

for (const file of filesRequiringDevelopOnPR) {
it(`${file} triggers pull_request builds against develop`, () => {
const branches = branchesFor(readWorkflow(file), "pull_request");
expect(branches).not.toBeNull();
expect(branches).toMatch(/rebuild/);
expect(branches).toMatch(/develop/);
});
}

it("open-code-review.yml triggers pull_request reviews against develop", () => {
const branches = branchesFor(
readWorkflow("open-code-review.yml"),
"pull_request",
);
expect(branches).toMatch(/main/);
expect(branches).toMatch(/rebuild/);
expect(branches).toMatch(/develop/);
});
});

describe("push triggers are left untouched by the develop rollout", () => {
const filesWithUnchangedPush = [
"cli-ci.yml",
"desktop-ci.yml",
"droplet-ci.yml",
"server-ci.yml",
"e2e.yml",
"editorconfig-ci.yml",
];

for (const file of filesWithUnchangedPush) {
it(`${file} still only push-triggers on rebuild`, () => {
const branches = branchesFor(readWorkflow(file), "push");
expect(branches).toMatch(/rebuild/);
expect(branches).not.toMatch(/develop/);
});
}
});

describe("workspace-root-triggered workflows watch pnpm-workspace.yaml and package.json", () => {
const filesWithNewPaths = ["cli-ci.yml", "desktop-ci.yml", "droplet-ci.yml"];

for (const file of filesWithNewPaths) {
it(`${file} re-runs (on both push and pull_request) when pnpm-workspace.yaml or package.json change`, () => {
const content = readWorkflow(file);
const workspaceOccurrences = (
content.match(/pnpm-workspace\.yaml/g) || []
).length;
const packageJsonOccurrences = (
content.match(/"package\.json"/g) || []
).length;
// Once under `push.paths` and once under `pull_request.paths`.
expect(workspaceOccurrences).toBe(2);
expect(packageJsonOccurrences).toBe(2);
});
}
});

describe("rust-ci composite action", () => {
const content = fs.readFileSync(
path.join(REPO_ROOT, ".github", "actions", "rust-ci", "action.yml"),
"utf8",
);

it("writes cargo audit output to JSON instead of failing the step directly", () => {
expect(content).toContain(
[
" - name: Audit dependencies",
].join("\n"),
);
const stepStart = content.indexOf("- name: Audit dependencies");
const stepEnd = content.indexOf(
"run: cargo audit --json",
stepStart,
);
const step = content.slice(stepStart, stepEnd);
expect(step).toContain("continue-on-error: true");
expect(content).toContain(
"run: cargo audit --json > /tmp/cargo-audit.json 2>/dev/null || true",
);
});

it("adds a follow-up step that checks for new Rust advisories via GITHUB_WORKSPACE", () => {
expect(content).toContain("- name: Check for new Rust advisories");
expect(content).toContain(
[
" if: success() || failure()",
" shell: bash",
" working-directory: ${{ inputs.working-directory }}",
" run: |",
' node "$GITHUB_WORKSPACE/scripts/check-new-vulns.cjs" \\',
" --format cargo \\",
" --json /tmp/cargo-audit.json \\",
" --min-severity high",
].join("\n"),
);
});

it("resolves the script via GITHUB_WORKSPACE rather than a path relative to working-directory", () => {
expect(content).not.toContain("run: node scripts/check-new-vulns.cjs");
expect(content).not.toMatch(/run: \|\s*\n\s*node scripts\//);
});
});

describe("codecov.yml", () => {
const content = fs.readFileSync(
path.join(REPO_ROOT, ".github", "codecov.yml"),
"utf8",
);

it("keeps project and patch coverage status informational (non-blocking)", () => {
expect(content).toContain(
[
"coverage:",
" status:",
" project:",
" default:",
" target: auto",
" threshold: 2%",
" base: auto",
" informational: true",
" patch:",
" default:",
" target: 80%",
" informational: true",
].join("\n"),
);
});

it("defines a carried-forward 'server' flag scoped to server/", () => {
expect(content).toContain(
[
" individual_flags:",
" - name: server",
" paths:",
" - server/",
" carryforward: true",
].join("\n"),
);
});

it("has no tab characters (consistent space indentation)", () => {
expect(content).not.toMatch(/\t/);
});
});
Loading
Loading