Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
15 commits
Select commit Hold shift + click to select a range
71b4098
Pin the fork at the fail-closed CMDB loader; test a malformed interlo…
djbclark Nov 3, 2026
341f37f
Move the fork pin to c3afaac26 (acceptance 08/09 updated for all-or-n…
djbclark Nov 3, 2026
d4e8c1f
Pin the fork at 8e70831f3; check 4b probes for both containers, adds …
djbclark Nov 6, 2026
c9813af
Move the fork pin to d32cade13 (rollback free() fix)
djbclark Nov 6, 2026
236302a
Correct the loader statements the all-or-nothing pin makes false (run…
djbclark Nov 14, 2026
59b7253
Validate proposal: record the 2026-11-02 rulings (Q10 option 3, R13 o…
djbclark Nov 14, 2026
0b41cb5
Overlay B-5b: note that the fork's all-or-nothing loader supersedes i…
djbclark Nov 14, 2026
867591e
Policy comment: the verify_vars.c reading was at b8abf28408b1; the pi…
djbclark Nov 14, 2026
219f9fc
Check 9 variables collision: assert what the pinned loader does (refu…
djbclark Nov 14, 2026
7d43372
Pin the fork at 6505f488a (strict CMDB parse, class insertion fails c…
djbclark Nov 14, 2026
c2a1131
run.sh: capability-gate fork-only assertions (SKIP, never silent; CI …
djbclark Nov 14, 2026
c2e372d
projector: E-6/E-7 note distinguishes upstream CFEngine from the pinn…
djbclark Nov 14, 2026
d90c78a
Docs and policy comments: qualify the all-or-nothing claims against t…
djbclark Nov 14, 2026
fd13eb3
run.sh: gate the strict-parse integration assertions (repeated member…
djbclark Nov 14, 2026
beddad9
Qualify the cross-check refusal claim (empty lenient view: no actuati…
djbclark Nov 14, 2026
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
4 changes: 4 additions & 0 deletions .github/actions/build-fork/action.yml
Original file line number Diff line number Diff line change
Expand Up @@ -38,6 +38,10 @@ runs:
git -C /tmp/core/libntech checkout -q FETCH_HEAD
echo "core=$core" >> "$GITHUB_OUTPUT"
echo "libntech=$libntech" >> "$GITHUB_OUTPUT"
# In the job log, not only the summary: the libntech commit core's
# submodule pointer names exists on the fork, not on the upstream URL
# in .gitmodules, and is fetched from `libntech.repo` above.
echo "build-fork: core $core, libntech $libntech (from $libntech_repo)"

- name: Build dependencies (core's own script; Debian/Ubuntu branch)
shell: bash
Expand Down
2 changes: 2 additions & 0 deletions .github/workflows/linux.yml
Original file line number Diff line number Diff line change
Expand Up @@ -42,6 +42,7 @@ jobs:
- name: Generic-bundle tests (systemd lifecycle in enforce mode)
env:
TENDCF_TEST_SYSTEMD: "1"
TENDCF_REQUIRE_FORK_LOADER: "1"
run: tests/policy/run.sh --bindir /opt/tendcf/bin
# Schema v5 S1 (schema-v5-dpkg-reconciliation-2026-11-02.md section 6):
# the read-only unit classifier against the real manager and real
Expand Down Expand Up @@ -92,6 +93,7 @@ jobs:
- name: Generic-bundle tests (plus the NixOS-shape refusal)
env:
TENDCF_TEST_NIXOS_SHAPE: "1"
TENDCF_REQUIRE_FORK_LOADER: "1"
run: tests/policy/run.sh --bindir /opt/tendcf/bin
# librsync (and its libb2) is a build-time pick of core's configure
# and not part of a minimal Debian; pastebin lacked it (2026-10-29).
Expand Down
14 changes: 13 additions & 1 deletion AGENTS.md
Original file line number Diff line number Diff line change
Expand Up @@ -242,7 +242,19 @@ a book: `site-private/AGENTS.md` ("Book knowledge base").

- `tests/policy/run.sh --bindir DIR` runs the generic bundle against a real
`cf-agent`/`cf-promises` (Mac dev build: `~/opt/cfengine-dev/bin`); it is
platform-independent and is the test CI runs on Linux. Run it through
the test CI runs on Linux. Policy-layer assertions that hold on any loader
(most of the file) run against any `cf-agent`. Two kinds hold only with the
pinned fork's all-or-nothing, strict-parse loader: loader assertions (its log
strings, "this container was not installed") and loader+policy integration
assertions (9b: the cross-check's refusal and no actuation for a repeated
member, a repeated `vars`, trailing input; a lenient loader and the lenient
`readjson` see the same surviving data, so there is nothing to refuse). A
probe in check 4b decides once; on any other build each of those prints
`SKIP (loader not all-or-nothing): <name>`, never passes silently, and the
run is expected to pass with SKIP lines and a count (not yet run on the Mac:
derived from the parser behaviour). The Mac dev build stays a target
(operator, 2026-11-14, "keep Mac, SKIPs"). The Linux CI legs set
`TENDCF_REQUIRE_FORK_LOADER=1`, which makes a skip a failure. Run it through
`~/ops/site-private/bin/bg`. Two checks are opt-in because they reshape the
host and need root: `TENDCF_TEST_SYSTEMD=1` (check 6, the real-systemd
lifecycle; the ubuntu VM jobs) and `TENDCF_TEST_NIXOS_SHAPE=1` (check 7,
Expand Down
11 changes: 8 additions & 3 deletions bin/projector.py
Original file line number Diff line number Diff line change
Expand Up @@ -409,10 +409,15 @@ def flag(rule: str, msg: str) -> None:
# E-6/E-7: these are LEGAL to CFEngine, which is exactly why the
# negative exists — `variables` loads after `vars` and overwrites
# the same name, so a checker that only asks "is `vars` right?"
# never sees the document that replaced it.
# never sees the document that replaced it. (The fork pinned in
# overlay/fork.json instead refuses a file in which `variables`
# names a variable `vars` already names; upstream replaces it.)
extra = (
" — legal to CFEngine, loaded after `vars` and overwriting the "
"same names, which is why envelope-shape checking misses it"
" — legal to upstream CFEngine, which loads it after `vars` and "
"lets it overwrite the same names, which is why envelope-shape "
"checking misses it; the fork pinned in overlay/fork.json "
"instead refuses a file in which `variables` or `classes` is "
"invalid or names a variable `vars` already names"
)
flag(
RULE_PROJECTION,
Expand Down
64 changes: 45 additions & 19 deletions docs/architecture/tendcf-validate-proposal-2026-11-03.md
Original file line number Diff line number Diff line change
Expand Up @@ -15,8 +15,9 @@ TUF); map §11 (timeout is deny); [`change-process.md`](change-process.md)
of [`projector-reconciliation-2026-08-16.md`](projector-reconciliation-2026-08-16.md).

How the CFEngine claims were checked: the fork source at the commit
`overlay/fork.json` pins (`b8abf28408b1b54e285d372a9bc5f2d850c960ad`,
read with `git show` in `~/src/cfengine-core`). The book (*Learning
`overlay/fork.json` pinned when this was written
(`b8abf28408b1b54e285d372a9bc5f2d850c960ad`, read with `git show` in
`~/src/cfengine-core`; PR #41 later moved the pin, see §13). The book (*Learning
CFEngine*, 3.12) was searched first. It covers cf-execd's role and five-minute
default schedule (ch. 23) and `cf-runagent` (ch. 35). `book-kb query`
finds neither `host_specific` nor `exec_command` in it: both postdate or
Expand All @@ -33,8 +34,9 @@ Each was read in the pinned fork source or in this repository on
(`libpromises/generic_agent.c:1234`) logs a failed load ("Failed to
load CMDB data") and the run continues. `cf-agent
--no-host-specific-data` skips the load entirely.
2. **On the pinned fork, a bad entry is skipped on its own, not with the
whole file.** `ReadCMDBVars` `continue`s past an entry carrying an
2. **On the fork at the pin of the time (`b8abf28408b1`), a bad entry is
skipped on its own, not with the whole file.** (Superseded: the pin
now carries an all-or-nothing loader, PR #41, §13.) `ReadCMDBVars` `continue`s past an entry carrying an
unexpanded variable reference or an invalid key, and still installs
the others. The cause is fork commit `8f0076b81` ("Fixed one bad CMDB
entry discarding every entry in its section"), merged into
Expand Down Expand Up @@ -981,8 +983,9 @@ that hold the stale text, per its brief.
a function of.
5. **Map §12 item 1** ("deleting or withholding the projection is not a
demonstrated fail-closed launch protocol") is right, and fact 2
strengthens it: on the pinned fork, a *malformed* projection is not
even all-or-nothing.
strengthens it: on the fork pinned when this was written (`b8abf28408b1`),
a *malformed* projection was not even all-or-nothing (the pin has since
moved to the fail-closed loader, §13).
6. **`deploy/linux/tendcf-install.sh` `put_data`** is not atomic against
a concurrent cf-agent (fact 8). This matters once a scheduler runs,
which stage B enables.
Expand Down Expand Up @@ -1022,8 +1025,11 @@ projection that `tendcf-install.sh install --data` puts on pastebin
(deploy/linux/README.md) carries both containers: its `vars` keys are
`tendcf_interlock` and `tendcf_service`, checked 2026-11-02. Any skip of
the first leaves the second ungated. No skip is known to happen with the
current bytes, which pass the N-series. It is flagged for djbclark, not
decided here.
current bytes, which pass the N-series. It was flagged for djbclark.
**Ruled 2026-11-02:** option 3 (Q10), implemented in PR #42, which records it
below; and option 5 (R13, Q49), the fork's all-or-nothing loader pinned by PR
#41 ([`upstream-register.md`](upstream-register.md) B-5b note). Each holds
without the other.

**(a) Question.** When the CMDB loader installs `tendcf_service` but
not `tendcf_interlock` (fact 2), how does the policy refuse, instead of
Expand Down Expand Up @@ -1143,9 +1149,11 @@ cannot differ on a consistent file; a `vars` value that is not an object is
refused when it reads back differently: the loader installs a scalar as a
string, which `storejson()` cannot read, and a list of numbers as an slist of
strings, while a list of strings reads back equal and is accepted.) It
also refuses any top-level key of the file other than `vars` (the fork's
loader also applies `variables`, loaded after `vars`, and `classes`, which can
define `interlock_ok_*`; the projector emits neither, N-2), any `vars` key that
also refuses any top-level key of the file other than `vars` (the loader also
applies `variables`, loaded after `vars` (the fork at the pin refuses a file
in which `variables` names a variable `vars` already names; upstream replaces
it), and `classes`, which can define `interlock_ok_*`; the projector emits
neither, N-2), any `vars` key that
is not a plain identifier `[A-Za-z0-9_]+` (the loader reads a key as a variable
reference, `GetCMDBVariableRef`, so `default:tendcf_services.service_ids`
presets a variable of the policy's own bundles), a loaded `data:variables.*`
Expand Down Expand Up @@ -1204,18 +1212,30 @@ rvalue does redefine a preset, a function call does not), and the two
no pre_action, no service, no tombstone. The vars and classes of a
`bundle common` are pre-evaluated before the vars of any non-common bundle,
whatever `bundlesequence` says (`PolicyResolve` and `BundleResolve`,
`libpromises/expand.c` lines 846-871 and 1145-1177 at the pinned fork commit
`b8abf28408b1`, [expand.c](https://github.com/djbclark/core/blob/b8abf28408b1b54e285d372a9bc5f2d850c960ad/libpromises/expand.c#L846)),
`libpromises/expand.c` lines 846-871 and 1145-1177 at fork commit
`b8abf28408b1` (the pin when this was written), [expand.c](https://github.com/djbclark/core/blob/b8abf28408b1b54e285d372a9bc5f2d850c960ad/libpromises/expand.c#L846)),
so `tendcf_cmdb_refused` is decided before a gate is first read. That is a
statement about vars and classes only, not that every promise type of a common
bundle completes first, and the classes inside the check rely on their written
order within the one pass. `bundlesequence` guard classes in `promises.cf` are
not needed for this check. This is a consistency check, not a sandbox: a CMDB
author (root) already chooses each pre_action executable, `/bin/sh` included,
and the unit commands, so it does not stop a hostile but self-consistent file.
The fork-side all-or-nothing loader (PR #41) and this check are independent:
with the loader failing closed, the file's keys are not loaded and the check
refuses and reports it, which is correct and louder.
The fork-side all-or-nothing loader (PR #41) and this check hold each without
the other: with the loader failing closed, the file's keys are not loaded and
the check refuses and reports it, which is correct and louder. The independence
is limited, not complete: the loader's strict parse (a repeated member or
trailing input refuses the file; fork `6505f488a`) belongs to the CMDB loader
only, while this check's `readjson` stays lenient and sees the last of a
repeated member or `vars` section, so an earlier container the loader refused
is not named by the check. The check reports a refusal when that lenient view
still lists variables the loader did not install (or another feature it refuses
on its own); when the lenient view is empty (`{"vars":{},"vars":{}}`, or
`{"vars":{}}` followed by non-whitespace) nothing is installed and nothing
actuates, but the check has nothing to report, so no actuation is not a refusal
report and the loader's own error is the only trace (`run.sh` 9b, empty-view
controls). The check does not detect such a file when the loader is lenient and
the repeated member is invisible to both parsers.

Two findings from the implementation, both about the proof above. (1) The
Mac dev build (`3.29.0a.086d27749`) does not carry the per-entry loader
Expand All @@ -1224,9 +1244,15 @@ makes it install nothing at all, so the fail-open cannot be reproduced there
with a file. `tests/policy/sim-loader.cf` therefore installs
`data:variables.*` in the loader's place, from the same file, to emulate the
per-entry skip; the original policy was run against it first and did
process the service with its interlock skipped. The Linux CI legs build the
pinned fork and so also exercise the real per-entry skip with the literal
fixture (`tests/policy/fixtures/interlock-malformed.json`). (2) The check
process the service with its interlock skipped. When this was written the
Linux CI legs built the fork at `b8abf28408b1` and so also exercised the real
per-entry skip with the literal fixture
(`tests/policy/fixtures/interlock-malformed.json`). That no longer holds: PR
#41 moved the pin to the all-or-nothing loader, which installs nothing from
that fixture, so check 9 sees both keys missing and the partial states stay
tested only through the stand-in (check 10), which remains valid because the
check compares the file with what was installed whatever the loader's policy.
(2) The check
needs `readjson` and `variablesmatching` to compose on the loader's
installed variables; they do on the dev build (checks 9-10 of
`tests/policy/run.sh`), so option 4 stays unneeded. The policy-tree digest
Expand Down
11 changes: 11 additions & 0 deletions docs/architecture/upstream-overlay.md
Original file line number Diff line number Diff line change
Expand Up @@ -220,6 +220,17 @@ drops it.
| `retired --strict` against upstream master | nothing to retire | nothing to retire |
| `self-test` | 13/13 | |

**Since 2026-11-02 (R13, PR #41) the core column's B-5b is not what the fork
the tests run does.** The pin in `overlay/fork.json` carries an all-or-nothing
CMDB loader that replaces B-5b's per-entry skip (a skipped interlock leaves the
services it guards unguarded). The manifest entry still reproduces `8f0076b81`
and carries a `superseded_on_fork` note; the table above is the 2026-10-09
state and is not updated. Upstream plus this overlay skips a bad entry; the
pinned fork installs nothing from the file. The operator ruled on 2026-11-14
to replace the patch: a later, separate PR makes the overlay carry the
all-or-nothing loader in its place ([`upstream-register.md`](upstream-register.md)
B-5b note).

So the overlay reproduces exactly the inputs whose build was verified on
2026-08-19, from this repository alone, and would still apply to a build
cut from today's upstream.
Expand Down
27 changes: 27 additions & 0 deletions docs/architecture/upstream-register.md
Original file line number Diff line number Diff line change
Expand Up @@ -344,6 +344,33 @@ unmerged-libntech submodule dependency, and
[#1](https://github.com/djbclark/core/issues/1) is the
investigation trail behind P-1/P-2 and is not itself a defect.

**B-5b reversed on the integration branch, 2026-11-02 (operator ruling R13).**
`8f0076b81` made the CMDB loader skip one bad entry and install the rest. For
tendcf that is fail-open: an interlock entry dropped at load leaves the
services it guards actuated unguarded. `tendcf-integration` now carries
`cmdb-fail-closed` (`a750b1dfe`, `dbf9dede6`, `61065e551`, `736becc07`, merged
`d32cade13`; hardened by djbclark/core PR #98 and a strict libntech JSON parse,
the pin now `6505f488a`): `LoadCMDBData()` validates every entry of
`vars`/`variables`/`classes` first and installs nothing from
`host_specific.json` if any one is invalid (also an unknown top-level key, a
null section, a key with index brackets, a frame-dependent special scope, two
entries naming one variable, a variable that exists already, a class name with
whitespace at either end), logging each by name and returning false; the agent
then runs with no CMDB variables. It reads the file with a strict parse
(`JsonParseFileStrict`, libntech `cc892efae`): a repeated object member and
anything but whitespace after the document refuse the file before validation.
A class that exists already is left as it is and the rest is installed. Strict
parsing is the CMDB loader's alone: `readjson` and `def.json` stay lenient.
Upstream PRs #6315/#6320
stay closed unmerged. The overlay manifest's B-5b entry (`overlay/overlay.json`)
still describes and reproduces `8f0076b81`, not this behaviour; it now carries
a `superseded_on_fork` note saying so, and
[`upstream-overlay.md`](upstream-overlay.md) section 7 repeats it. Whether the
overlay keeps, retires or replaces that patch was ruled on 2026-11-14 (Replace:
the overlay will carry the all-or-nothing loader instead of the per-entry skip),
and is done in a later, separate PR because it changes what the overlay
reproduces.

B-2 through B-7 are described, measured and sourced in
[`cfengine-upstream-candidates-2026-08-16.md`](cfengine-upstream-candidates-2026-08-16.md);
B-1's full evidence is in
Expand Down
2 changes: 1 addition & 1 deletion overlay/fork.json
Original file line number Diff line number Diff line change
Expand Up @@ -3,7 +3,7 @@
"core": {
"repo": "https://github.com/djbclark/core",
"branch": "tendcf-integration",
"commit": "b8abf28408b1b54e285d372a9bc5f2d850c960ad"
"commit": "6505f488a80ab3e18b089c7d8e32571c88128866"
},
"libntech": {
"repo": "https://github.com/djbclark/libntech"
Expand Down
1 change: 1 addition & 0 deletions overlay/overlay.json
Original file line number Diff line number Diff line change
Expand Up @@ -32,6 +32,7 @@
"retire_when": "an upstream commit carries the same change (`bin/overlay.py retired` says RETIRE), or the code it protects is no longer shipped",
"role": "load-bearing",
"subject": "Fixed one bad CMDB entry discarding every entry in its section",
"superseded_on_fork": "2026-11-02, operator ruling R13 (PR #41): the fork's tendcf-integration branch, which overlay/fork.json pins, replaces this patch's per-entry skip with an all-or-nothing loader (branch cmdb-fail-closed: a750b1dfe, dbf9dede6, 61065e551, 736becc07; merged d32cade13, hardened by djbclark/core PR #98 and a strict libntech parse, pin 6505f488a). A skipped interlock entry leaves the services it guards unguarded. This entry still describes, and apply reproduces, commit 8f0076b81 exactly, so upstream plus the overlay skips a bad entry and does NOT behave like the pinned fork. Operator ruling 2026-11-14: Replace. A later, separate PR makes the overlay carry the all-or-nothing loader instead of this per-entry skip, because that changes what the overlay reproduces (upstream-overlay.md section 7); this PR leaves the patch and its manifest entry as they are.",
"upstream_pr": "https://github.com/cfengine/core/pull/6320",
"upstream_pr_state": "closed unmerged 2026-08-19"
}
Expand Down
Loading
Loading