Skip to content

fix(engine): warn of a _layering or _remove that nothing reads - #605

Merged
Vivswan merged 8 commits into
mainfrom
fix/engine-single-document-directives
Nov 12, 2026
Merged

Vivswan merged 8 commits into
mainfrom
fix/engine-single-document-directives

Conversation

@Vivswan

@Vivswan Vivswan commented Nov 12, 2026 •

Copy link
Copy Markdown
Owner
input                                  today                              now
_layering: banana, top of a single     applied, nothing said              applied, one warning names the
document                               (as a layer: refused, unchanged)   values and the fix
_remove: true on an entry of a list    applied or rendered, nothing said  the same run, one warning names
no key holds (bypass_actors, ...)      GitHub is sent a field "_remove"   the file and the entry's path
  • No file that ran before stops running.
  • Both refusals wait behind COMPAT(v4), and each warning says the next major version refuses the input. Taking either earlier is your decision.
  • A refused document or fold draws neither warning.

Before

# on main, with this branch's three scenario files copied into test/e2e/scenarios/
$ bun test/e2e/run.ts --scenario apply-layering-outside-values-warns --print-stdout | grep -E '^        [^ ]' | grep -v '::debug::'
        labels: created label "bug"
        result: applied
$ bun test/e2e/run.ts --scenario apply-unkeyed-remove-warns --print-stdout | grep -E '^        [^ ]' | grep -v '::debug::'
        rulesets: created ruleset "r"
        result: applied
$ bun test/e2e/run.ts --scenario apply-unkeyed-remove-warns --print-stdout | grep -o 'payload: .*'
payload: {"name":"r","target":"branch","enforcement":"active","bypass_actors":[{"actor_id":1,"actor_type":"Team","bypass_mode":"always","_remove":true}]}
$ bun test/e2e/run.ts --scenario render-unkeyed-remove-warns --print-stdout | grep -E '^        [^ ]' | grep -v '::debug::'
        ::notice::/repo/settings.yml: rulesets[0] carries _remove: true and dropped the entry a lower layer declared under its key
        rendered 2 layers into /repo/rendered.yml
        result: rendered

After

$ bun test/e2e/run.ts --scenario apply-layering-outside-values-warns --print-stdout | grep -E '^        [^ ]' | grep -v '::debug::'
        ::warning::/repo/settings.yml: _layering must be one of "replace", "shallow", "deep"; got a string that is none of them. A single document is not folded, so the directive is not read and the run goes on; mode: render refuses this file as a layer. Write one of the values, or remove the key; the next major version refuses it here too
        labels: created label "bug"
        result: applied
$ bun test/e2e/run.ts --scenario apply-unkeyed-remove-warns --print-stdout | grep -E '^        [^ ]' | grep -v '::debug::'
        ::warning::/repo/settings.yml: rulesets[0].bypass_actors[0]._remove: _remove belongs on an entry of a keyed list, and this list is not keyed, so it removes nothing and passes through as a field of the entry. Remove the key; the next major version refuses it
        rulesets: created ruleset "r"
        result: applied
$ bun test/e2e/run.ts --scenario apply-unkeyed-remove-warns --print-stdout | grep -o 'payload: .*'
payload: {"name":"r","target":"branch","enforcement":"active","bypass_actors":[{"actor_id":1,"actor_type":"Team","bypass_mode":"always","_remove":true}]}
$ bun test/e2e/run.ts --scenario render-unkeyed-remove-warns --print-stdout | grep -E '^        [^ ]' | grep -v '::debug::'
        ::warning::/repo/settings.yml: rulesets[1].bypass_actors[0]._remove: _remove belongs on an entry of a keyed list, and this list is not keyed, so it removes nothing and passes through as a field of the entry. Remove the key; the next major version refuses it
        ::warning::the rendered settings document: rulesets.entries[0].bypass_actors[0]._remove: _remove belongs on an entry of a keyed list, and this list is not keyed, so it removes nothing and passes through as a field of the entry. Remove the key; the next major version refuses it
        ::notice::/repo/settings.yml: rulesets[0] carries _remove: true and dropped the entry a lower layer declared under its key
        rendered 2 layers into /repo/rendered.yml
        result: rendered

How

  • One walk. separateRemovals also gathers every _remove on an entry no list keys.
  • Shared subtrees. One that holds no marker is walked once.
  • One boundary. validateSettingsDoc prints both warnings, once the document has no issue.
  • The fold. foldLayers prints the warnings only once it stands.
  • The file's index. A layer's view hands over its way back.
  • Library: searched the repository. separateRemovals owns the walk, zod's toDotPath the spelling.

Proof

  • Tests: 83 pass, 0 fail in the three touched test files. 25 cases are new.
  • Scenarios: the three pass here. On main each fails only for its missing warning.
  • Gates: typecheck, lint, lint:arch, knip, check:compat, build:check exit 0.
Technical details

Why a warning, not a refusal

  • Both inputs complete a run on 3.0.0, so neither is refused before the next major.
  • The two accepting lines in validateSettingsDoc carry COMPAT(v4) markers, and bun run check:compat lists them.
  • The e2e mock checks every request body against GitHub's OpenAPI description and reported no violation for the body above.
  • The description closes none of these entry shapes:
$ jq -c '[.paths["/repos/{owner}/{repo}/rulesets"].post.requestBody.content["application/json"].schema.properties.bypass_actors.items | {additionalProperties, properties: (.properties | keys)}]' node_modules/@octokit/openapi/generated/api.github.com.deref.json
[{"additionalProperties":null,"properties":["actor_id","actor_type","bypass_mode"]}]

The class: every list of mappings no key holds

list                                                              today            now
rulesets[].bypass_actors                                          sent as a field  warned
rulesets[].rules[].parameters.required_status_checks              sent as a field  warned
rulesets[].rules[].parameters.workflows                           sent as a field  warned
rulesets[].rules[].parameters.code_scanning_tools                 sent as a field  warned
rulesets[].rules[].parameters.required_reviewers                  sent as a field  warned
rulesets[].rules[].parameters.dismissal_restriction.allowed_actors sent as a field warned
check_suite_preferences.auto_trigger_checks                       sent as a field  warned
a list under an open passthrough key (two tried, below)          sent as a field  warned
branches[].protection.required_status_checks.checks               refused (closed) refused, no warning
repository.security_and_analysis.secret_scanning_delegated_bypass_options.reviewers  refused (closed)  refused, no warning
  • The published schema declares 34 lists of mappings under the sections: 19 keyed list sections, 6 nested keyed lists, 9 neither.
  • The 9 are the first seven rows and the last two of the table: 7 open, 2 closed.
  • The fold owns _remove in the 19 and the 6.
  • "Sent as a field" is one real-bundle apply per row: each write's body carried "_remove":true and the mock answered 2xx.
  • The two passthrough lists tried are repository.custom_list and an unknown key of a milestones entry.
  • An environment's reviewers is a nested keyed list (by type and id), so it is not in the class.
  • _remove: false and _remove: null on such an entry draw the warning too: the bundle sends both as a field, so the text is true of them.
The census script and its output

Save as census.mts in the checkout, then bun run build:schema && bun census.mts.

// Every list of mappings the published schema declares under a section, and who owns `_remove` on its entries.
import { readFileSync } from "node:fs";
import { join } from "node:path";

type Node = Record<string, unknown>;
const root = process.cwd();
const schema = JSON.parse(readFileSync(join(root, "lib/settings.schema.json"), "utf8")) as Node;
const definitions = schema.definitions as Record<string, Node>;
const { LIST_SECTIONS, SECTION_KEYS } = await import(join(root, "src/schema.ts"));
const { listLayering } = await import(join(root, "src/sections/registry.ts"));

/** A node's own shapes: its $ref followed, its anyOf/oneOf/allOf branches flattened. */
function shapes(node: Node): Node[] {
  const ref = node.$ref as string | undefined;
  const at = ref === undefined ? node : (definitions[decodeURIComponent(ref.split("/").pop() ?? "")] as Node);
  const branches = ["anyOf", "oneOf", "allOf"].flatMap((key) => (at[key] as Node[] | undefined) ?? []);
  return [at, ...branches.flatMap(shapes)];
}

const lists = new Map<string, boolean>();
function walk(node: Node, path: string, depth = 0): void {
  for (const shape of depth > 12 ? [] : shapes(node)) {
    const items = shape.items as Node | undefined;
    const mappings = items === undefined ? [] : shapes(items).filter((item) => item.properties !== undefined);
    if (mappings.length > 0) {
      lists.set(path, (lists.get(path) ?? false) || mappings.some((item) => item.additionalProperties !== false));
    }
    if (items !== undefined) {
      walk(items, path, depth + 1);
    }
    for (const [key, child] of Object.entries((shape.properties ?? {}) as Record<string, Node>)) {
      // A wrapper's `entries` is the list itself, so both forms land on one path.
      walk(child, key === "entries" ? path : `${path}.${key}`, depth + 1);
    }
  }
}
for (const key of SECTION_KEYS as string[]) {
  walk(((shapes(schema)[0] as Node).properties as Record<string, Node>)[key] as Node, key);
}

function owner(path: string): string {
  const [section, ...fields] = path.split(".") as [string, ...string[]];
  if (!(LIST_SECTIONS as string[]).includes(section)) {
    return "neither";
  }
  let keyed = listLayering(section);
  for (const field of fields) {
    keyed = keyed?.nested?.[field];
  }
  return keyed === undefined ? "neither" : fields.length === 0 ? "keyed section" : "nested keyed";
}

const count = (who: string): number => [...lists.keys()].filter((path) => owner(path) === who).length;
console.log(`${lists.size} lists of mappings: ${count("keyed section")} keyed sections, ${count("nested keyed")} nested keyed, ${count("neither")} neither`);
for (const [path, open] of [...lists].filter(([path]) => owner(path) === "neither").sort()) {
  console.log(`  ${open ? "open  " : "closed"} ${path}`);
}
$ bun census.mts
34 lists of mappings: 19 keyed sections, 6 nested keyed, 9 neither
  closed branches.protection.required_status_checks.checks
  open   check_suite_preferences.auto_trigger_checks
  closed repository.security_and_analysis.secret_scanning_delegated_bypass_options.reviewers
  open   rulesets.bypass_actors
  open   rulesets.rules.parameters.code_scanning_tools
  open   rulesets.rules.parameters.dismissal_restriction.allowed_actors
  open   rulesets.rules.parameters.required_reviewers
  open   rulesets.rules.parameters.required_status_checks
  open   rulesets.rules.parameters.workflows

The walk

  • A path is spelled by zod's toDotPath, as an issue's path is, so a key that only looks like a list step ("rules[0]") is quoted and never renumbered.
  • A mapping or list walked to its end with no marker found is remembered and skipped on its next path. A depth-24 shared document costs 25 visits, where it cost 33,554,431.
  • A subtree that holds a marker is walked once per path: each path is its own warning.
  • A container on a cycle is never remembered, since what the cut hid is not known.

The layered path

  • admit.ts never judged this marker: a layer's unkeyed list replaces the lower one whole, marker included.
  • A layer is validated as its standalone view, whose keyed removals are gone, so the view's index is not the file's.
  • validateLayer passes view.asWritten to validateSettingsDoc through one new ValidateOptions member.
  • The view also sets aside what the fold alone judges, so foldLayers holds every validation's lines in capturingIo and replays them in order once the fold stands.
  • A layered run that stands names a marker once per layer file that writes it, and once for the rendered document, which is validated too. No option silences the second.

Public surface

  • ValidateOptions in src/engine/orchestrate.ts is the engine's own. The library's type of that name is declared in src/flows/library.ts.
  • bun run build:lib on main and on this branch gives an identical lib/pkg/index.d.ts, the shared chunk's hashed file name aside.
  • The declarations chunk both entries import differs by the member's two lines and three added comment lines on foldLayers. Only lib/pkg/internal.d.ts exports validateSettingsDoc and foldLayers.

Tests

  • No existing test was re-expected.
  • bun test test/engine/orchestrate.test.ts test/flows/layers.test.ts test/engine/separate-removals.test.ts runs 83 tests here. The first two files run 58 on main, and the third is new.
  • New in test/engine/orchestrate.test.ts (17): the _layering warning (a string, null), its legal-value control, the pinned _remove warning, eight more rows (six other declared lists, a passthrough list, a wrapper's entry), two refused documents that draw no warning, a key spelled like a list step, a wrapper's entry in a layer's view, and a cyclic value control.
  • New in test/flows/layers.test.ts (5): the layer whose keyed removal precedes the marked entry prints rulesets[1], with the warnings in layer order, and four refused folds print none.
  • New in test/engine/separate-removals.test.ts (3): the shared document's visit count, a cycle cut short, and a shared list with a marker.
  • Controls already in the tree: render-remove-entry and render-file-directive (bun test/e2e/run.ts --scenario <name>), and "a removal entry under %s is refused by its site".

Scenarios

  • test/e2e/scenarios/{apply-layering-outside-values-warns,apply-unkeyed-remove-warns,render-unkeyed-remove-warns}.yml pin each warning line beside the result main also gives.
  • The first is raw text (settings_raw). As a settings mapping it fails test/schema-corpus.test.ts with "schema REJECTS but runtime accepts": the published schema's enum refuses the value the run accepts until the next major.

Page

  • docs/operate/layering.md, the paragraph opening "Inside a keyed entry": one sentence added.

Recorded, not built

  • A _remove on a mapping that is no list entry (repository: {_remove: true}) still passes through unannounced by this walk.
  • One wording for "must be one of ...; got ..." needs src/problem.ts, which already holds two copies.
  • validateLayer still renumbers its issues itself. Only the warnings' paths go through the new option.
  • A document both cyclic and heavily shared is still walked per path. A YAML file with a cycle is refused when it is read, so only a library caller's object reaches this.
  • The walk reads the document as written, so it follows a key the shapes later drop (an own __proto__).
  • Other pages that speak of these directives are untouched: docs/reference/architecture.md ("One downgrade"), docs/schema.docs.yml (SettingsFile._layering), and the rendered-file bullet "No _remove anywhere" in docs/operate/layering.md, which a marker in an unkeyed list contradicts on main too.

Files

  • src/engine/{separate-removals.ts,directives.ts,orchestrate.ts,admit.ts}, src/flows/layers.ts
  • test/engine/{orchestrate.test.ts,separate-removals.test.ts}, test/flows/layers.test.ts, test/e2e/scenarios/{apply-layering-outside-values-warns,apply-unkeyed-remove-warns,render-unkeyed-remove-warns}.yml
  • docs/operate/layering.md

Two directive spellings ran with nothing said. A single document whose top-level _layering is outside "replace", "shallow", "deep" completed silently, because nothing folds a single document and so nothing read the value, while mode: render refused the same file as a layer. A _remove on an entry of a list the registry does not key (a ruleset's bypass_actors, a rule's parameter lists, check_suite_preferences.auto_trigger_checks, any list under a passthrough key) removed nothing and passed through to GitHub as a field named _remove, from a single document and from a layer alike.

Both runs complete today, so neither is refused before the next major: each now draws a warning naming the source and the path, and the two accepting paths in validateSettingsDoc carry COMPAT(v4) markers naming the refusal that replaces the warning.

separateRemovals, the one walk over a document's removals, now also gathers every _remove on an entry no list keys: below the mapping sections, and below each kept keyed entry outside its declared nested lists. Each path is spelled from its steps as validation spells an issue's, so a key that only looks like a list step is quoted and the way back to the document as written never renumbers through it. A layer is judged as its standalone view, so validateLayer hands validateSettingsDoc the view's way back and a layer's warning names the entry by its index in the file.

The wording lives in engine/directives.ts beside the directive keys, and admit.ts now reads the one isLayering predicate from there.
The two warnings printed before the document's issues were known, so a refused document drew them too. For a marker in a closed entry shape (a branch protection's required_status_checks.checks item) the run printed the shape's refusal and, beside it, a warning that the key "passes through as a field" and that "the next major version refuses it": the first untrue of a key just refused, the second already done.

validateSettingsDoc now prints both warnings once the document has no issue. A document that validates has no removal site, so its paths are the document's own and separateRemovals no longer renumbers them itself; a layer's standalone view still hands over its way back. The case that expected a warning beside a refused keyed removal is replaced by two cases pinning that a refused document draws none, and the quoted-key case now runs through a layer's view, the one place a removal may stand beside the marker in a document that validates.

Two lines over the 256-character cap are brought under it: a doc comment, and the pinned warning text in the test.
…nt's

The page said of a _remove on an entry of a list that is not keyed that "the run warns". A document the shapes refuse draws no warning, since a closed entry shape names the marker in its own refusal, so the sentence now says a document that validates draws it.
Copilot AI balanced review requested due to automatic review settings November 12, 2026 04:27
@github-actions

github-actions Bot commented Nov 12, 2026 •

Copy link
Copy Markdown
Contributor

File size check

0 over a hard cap (fails), 6 warning(s).

File Size Tier Cap
src/engine/diff.ts:56 11 comment lines warn 10
src/github/api-error.ts:9 11 comment lines warn 10
src/problem.ts:73 18 comment lines warn 10
src/sections/contract/errors.ts:18 11 comment lines warn 10
src/sections/contract/module.ts:55 12 comment lines warn 10
src/sections/shared/list/decl.ts:265 14 comment lines warn 10

Split the file, wrap the line, shorten or exempt the comment, or list the path in .file-size-allow.local with a # reason.

5 managed file(s) skipped; repo-platform owns them.

@github-actions

github-actions Bot commented Nov 12, 2026 •

Copy link
Copy Markdown
Contributor

Member shape check

16 finding(s).

File Line Rule
src/action/io.ts 24 one-line-comment
src/cli/actions.ts 12 one-line-comment
src/github/api.ts 55 one-line-comment
src/sections/pages/index.ts 71 one-line-comment
test/cli/actions.test.ts 45 one-line-comment
test/scripts/release-pipeline-fixture.test.ts 25 one-line-comment
test/scripts/release-pipeline-fixture.ts 53 one-line-comment
test/sections/contract/validate-hook.test.ts 80 all-or-none
test/sections/teams/teams.test.ts 20 all-or-none
test/sections/teams/teams.test.ts 23 all-or-none
test/sections/teams/teams.test.ts 24 all-or-none
test/sections/teams/teams.test.ts 25 all-or-none
test/sections/teams/teams.test.ts 26 all-or-none
test/sections/teams/teams.test.ts 28 all-or-none
test/sections/teams/teams.test.ts 31 all-or-none
test/sections/teams/teams.test.ts 34 constant-spacing

Add the blank lines, make the comment one line directly above the item, comment every field or none, or list the path in .member-shape-allow.local with a # reason.

4 generated file(s) skipped.

Copilot AI left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🟡 Changes recommended

Shared passthrough objects cause exponential traversal, and rejected layers incorrectly emit _remove warnings before validation.

2 open issues
What changed in this PR

This PR adds warnings for ignored _layering values and ineffective _remove markers in GitHub Settings as Code, while preserving current acceptance behavior until the next major version.

Changes:

  • Detects _remove markers in unkeyed lists and emits warnings after document validation.
  • Maps layer warning paths back to the original file's entry indices.
  • Adds regression tests and documents the new warning behavior.
File Description
test/​flows/​layers.test.ts Tests original-file indices and rendered-document warnings.
test/​engine/​orchestrate.test.ts Tests warnings, refusal controls, quoted paths, and cycles.
src/​flows/​layers.ts Passes original-file path mapping into validation.
src/​engine/​separate-removals.ts Collects markers in unkeyed lists during traversal.
src/​engine/​orchestrate.ts Emits warnings after document checks succeed.
src/​engine/​directives.ts Defines warning messages and shared layering validation.
src/​engine/​admit.ts Reuses the shared layering predicate.
docs/​operate/​layering.md Documents warnings for ineffective removal markers.

Review effort: Balanced

Comment thread src/engine/separate-removals.ts Resolved
Comment thread src/flows/layers.ts Resolved
A layer is validated as its standalone view, which sets aside what the fold alone judges, so its warning for a _remove no list keys went out before the fold refused the layer: a top-level _layering outside its values, a keyed marker that is not true, a wrapper's bad _layering, or another layer the shapes refuse. The line said the key "passes through as a field" of a render that then wrote nothing.

foldLayers now routes every validation of the fold, each layer's and the merged document's, through capturingIo and replays the held lines in their recorded order once the whole fold stands; a refused fold drops them. A fold that stands prints what it printed before, in the same order: the layers' warnings in layer order, then the merged document's.

Three scenarios pin the two warnings in the shipped bundle: a single document whose top-level _layering is outside its values, a single document with a _remove on an entry no list keys, and a layered render where a keyed removal precedes such an entry, so its path counts the removal. The first is raw text: the published schema's enum refuses the value the run still accepts until the next major, and the corpus's schema check holds every settings mapping to that schema.
…keyed _remove

The walk for a _remove on an entry no list keys visited a shared object once per path. A mapping aliased under two keys, 24 levels deep, is 25 mappings and 33,554,431 paths, so a document holding no marker at all took seconds to say so. The YAML reader caps aliases, but validateSettings is a library call that takes a parsed object.

gatherUnkeyed now returns whether a value is known to hold no marker and remembers each such mapping or list, to skip it when another path meets it again, as the neighbouring walk in validate.ts does. A subtree that holds a marker is still walked per path, so every path keeps its own warning. A container on a cycle is never remembered: the descent ends where the cycle closes, so what lies beyond the cut is not known, and a document whose cycle the shapes drop (an own __proto__ key) still validates.
Copilot AI balanced review requested due to automatic review settings November 12, 2026 05:26

Copilot AI left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🟢 Approval recommended

No active review issues are recorded for the change adding warnings about ignored layering values and ineffective removal markers.

No open issues 🎉

2 resolved since last review

Review effort: Balanced

Copilot AI balanced review requested due to automatic review settings November 12, 2026 05:46

Copilot AI left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🟢 Approval recommended

No active review issues are reported for the compatibility-preserving directive-warning changes.

No open issues 🎉

Review effort: Balanced

Copilot AI balanced review requested due to automatic review settings November 12, 2026 06:02

Copilot AI left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🟢 Approval recommended

No active review issues remain for the compatibility-warning changes.

No open issues 🎉

Review effort: Balanced

@Vivswan
Vivswan marked this pull request as ready for review November 12, 2026 06:09
@Vivswan Vivswan added the merge-when-green Owner approved: merge once every gate is green label Nov 12, 2026
@Vivswan
Vivswan merged commit bc81ce2 into main Nov 12, 2026
32 checks passed
@Vivswan
Vivswan deleted the fix/engine-single-document-directives branch November 12, 2026 06:39
This was referenced Nov 12, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

merge-when-green Owner approved: merge once every gate is green

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants