Skip to content

Overlay: replace B-5b with the all-or-nothing CMDB loader; libntech strict parse; overlay-build CI - #65

Merged
djbclark merged 6 commits into
masterfrom
overlay/cmdb-all-or-nothing
Nov 15, 2026
Merged

djbclark merged 6 commits into
masterfrom
overlay/cmdb-all-or-nothing

Conversation

@djbclark

@djbclark djbclark commented Nov 15, 2026 •

Copy link
Copy Markdown
Collaborator

What changed

The overlay now carries the fork's all-or-nothing CMDB loader (operator ruling R13, "Replace" of 2026-11-14) instead of B-5b's per-entry skip, and a CI job builds and tests it.

  1. Core overlay/core/0002 replaces B-5b (8f0076b81): a squash of the fork's loader (4443504, the pin) onto overlay base a0bca6aaf + B-5a. Branch overlay/cmdb-all-or-nothing on djbclark/core, squash commit 4158906f2a5b3f202af1615fb80357a9a85987cc, tag tendcf-pin-4158906 (B-5a kept as its original patch, applied with git am).
  2. libntech 0005, 0006: fork PR Added JsonParseStrict() and JsonParseFileStrict() djbclark/libntech#11 (dc3fba2 JsonParseStrict()/JsonParseFileStrict(), 8a57e55 null byte fix), as-is. New register id B-24.
  3. overlay/overlay.json: core entry is load-bearing, register B-5b (replaced), with B-7 as a passenger and B-20 as load-bearing; superseded_on_fork dropped; two libntech entries (B-24).
  4. CI: new overlay-build job (one ubuntu-24.04 leg) in linux.yml: fetch core and libntech at the bases, bin/overlay.py apply both, autogen/configure/make (same flags as build-fork), then cmdb_test, libntech json_test and mustache_test, the 06_host_specific_data acceptance directory, and tests/policy/run.sh with TENDCF_REQUIRE_FORK_LOADER=1.
  5. Docs: upstream-overlay.md sections 2, 4, 5 (explicit R1 caveat), 7; upstream-register.md (B-5b row and note rewritten as replaced, new B-24 row, ruling 2 status); validate proposal section 13; AGENTS.md; sim-loader.cf and run.sh header comments.
  6. run.sh (PR 61 carry-overs): the comment calling the negative greps "the only assertions left on a non-fork run" is gone (the marker ensures they cannot pass on an invocation that never reached the policy); policy_completed is anchored to ^R: tendcf-test-done: (the fixture tests/policy/fixtures/test-done.cf prints exactly tendcf-test-done: the entry policy evaluated ... as a reports: promise, like the existing ^R: tendcf: ... greps). Not run: Linux CI is the first run.

Proofs for the squash (O2)

a. git diff 4443504 4158906f2 -- libpromises/cmdb.c libpromises/cmdb.h tests/unit/cmdb_test.c tests/acceptance/00_basics/06_host_specific_data is three lines in two files, not one: cmdb.c:1040 as the design said, and also tests/unit/cmdb_test.c:70 and :85, both EvalContextVariableGetPlaintext( renamed to EvalContextVariableGet(. The design counted only cmdb.c; the function does not exist at the base, so the test would not compile without the rename. Same symbol, same fix, no new 4443504-only symbol. The acceptance directory's file list equals 4443504's. tests/unit/Makefile.am gains the one cmdb_test \ line (checked: it is the only cmdb line in 4443504's Makefile.am).
b. Against the design's scratch patch (f4570cc8/scratch/cmdb-allornothing-squash.patch): identical except the two cmdb_test.c lines above (and the blob hash on that file's index line).
c. Dependency check (identifiers of the new cmdb.c/cmdb_test.c against the base core tree, overlay + strict libntech, and 4443504): one candidate beyond GetPlaintext, JsonArrayContainsNull, is static and defined in cmdb.c itself (:314). Header diff base→4443504 for the identifiers used shows only the EvalContextVariableGet* change; libntech headers cc892efae vs overlay+strict show nothing for the identifiers used. This is a symbol analysis, not a build.

B-7 and B-20

The fork's final cmdb.c contains B-20 (null crash, 87490bb8f) and B-7 (dotted-key warning, ee1f9d60c), so they ride in the squash. B-20 is load-bearing (the loader's null validation relies on it); B-7 is a passenger (a warning). Both are recorded in the manifest and in the register B-5b note.

Results (run locally, hermetic)

  • bin/overlay.py lint: 0 findings. self-test: 21 assertions pass (round 3).
  • apply core at a0bca6aaf: HEAD 76e9339, tree 5c9f3ce18a6b290903b6610b58cb535638453822 = the tree of 4158906f2. apply libntech at 0c0620d6c5f8 (6 patches): tree 8d5fda1f9938c6b69bb9bb4d8a6e3be6c174f9e0. git apply --check of core 0002 on base+B-5a, libntech 0005 on 0004, 0006 on 0005: all rc 0.
  • check at base 2/2 and 6/6; check --against upstream master (core a8ac0dc7e, libntech 3c21479) 2/2 and 6/6; retired --strict: nothing to retire; xref_lint: 0 findings; shellcheck on run.sh: clean; actionlint on linux.yml: only findings that were already there (ubuntu-26.04 label, debian-13 SC2046).

Caveats

  • R1. The squash compiles at the overlay base only. Upstream master has a four-argument EvalContextVariableGet(…, get_secret) (since 671d53ff0), so the series applies there textually (check green) but should not compile. The weekly overlay-drift job cannot see that; documented in upstream-overlay.md section 5. Not built at that commit.
  • R5. The libntech overlay result lacks the fork's other json.c changes (the fork pin's json.c differs by 82 lines). A strict-parse test that needs them would fail only in overlay-build.
  • run.sh capability probe (check 4b) reads the same cmdb.c behaviour, and the CMDB call site in generic_agent.c (it logs Failed to load CMDB data at ERR and goes on to evaluate policy) is the same block at the base and at 4443504 (I compared that block, not the whole file), so it needs no change for the overlay build. Not verified until CI runs.
  • Nothing was built or run on the Mac; the overlay-build CI leg is the proof the overlay builds and passes (green at 2f90f33; result at the merge head below).
  • Known difference from the pinned fork: JSON numbers (Fable A1 / adversary MED1, round 2). The overlay does not carry libntech B-4/B-10 or core's rlist.c number fix, so a hand-edited CMDB file with a top-level or primitive-array number such as 1e5 or an integer past long makes cf-agent exit in the install step, and a real installs as %.2f. Projected data never reaches it (bin/projector.py refuses floats). Documented in upstream-overlay.md section 7 and overlay.json. Whether to carry those fixes in the overlay is open with djbclark; the default is to document and accept, so it does not block this PR.

Prior art / docs checked: docs/architecture/upstream-overlay.md, upstream-register.md, overlay/overlay.json, check.yml, linux.yml, the build-fork action, tests/policy/run.sh, the design report (overlay-replace-a.md). Closing search: exempt (fork-overlay housekeeping).

Review rounds

  1. Round 1 at 75db441: Fable changes-requested (A1: the "behaves like the fork" claims overreached on JSON numbers; A2: no applied-tree assertion; NITs A3-A7), Argon approve (one NIT), adversary approve with no blocking findings.
  2. Round 2 (ba60acf, tidy 2f90f33): claims reworded to what CI proves plus the known-difference block; applied_tree recorded and asserted by bin/overlay.py apply; setup-uv dropped for python3; NITs A3-A7 and Argon's NIT; register rows B-22/B-23/B-24 fixed to nine cells; ruff clean.
  3. Round 3 (1f379b5): the tree is compared only when HEAD is the pinned base (--allow-base-mismatch reports "not compared" instead of failing), applied_tree is a required manifest key, self-test count and the overlay-build wording in upstream-overlay.md corrected. bin/overlay.py was also run through ruff format (whole file), so its diff is mostly formatting; the logic change is about 40 lines.

Reviewers' verdicts

  1. Fable (claude-fable-5-2, high): round 1 changes-requested; round 2 approve; round 3 approve, two NITs accepted without a further round (the "no applied_tree" refusal does not say the patches are left applied, which lint catches first anyway; one self-test clause can never be false).
  2. Argon (gemini-4-argon, high), standing in for Astra (gpt-6-astra), whose Codex quota is below the tendcf floor until 2026-11-18: round 1 approve; round 2 approve (its quotes of bin/overlay.py did not match the file, so only its docs findings were counted); round 3 approve, no findings (quotes checked against the file this round: all match).
  3. Adversary (Opus 5.5 sub-agent, security review at 75db441): approve, no blocking findings; two medium items, one fixed in round 2 (the claims), one pre-existing and filed as a follow-up.

Admission note: the round-1 Claude usage report was rate-limited at admission; each later re-review was re-planned against fresh quota first.

Follow-ups

  1. libntech JsonParseAsString reads past the NUL on a trailing backslash #66 (trailing-backslash over-read in JsonParseAsString, pre-existing on base and fork)
  2. Strict JSON parse: accepted non-JSON forms, O(n^2) duplicate check, no depth cap #67 (strict-parse gaps, quadratic duplicate check, no depth cap)
  3. CMDB loader ignores unknown keys inside class/variable metadata objects #68 (unknown class/variable metadata keys ignored)
  4. Loader tests: ASan/UBSan pass and a loader-level NUL fixture #69 (ASan/UBSan pass and NUL fixture)
  5. Open with djbclark: carry B-4/B-10 and the core rlist.c number fix in the overlay (follow-up PR), or keep the documented difference.

🤖 Generated with Claude Code


Note

High Risk
Changes CMDB loading and strict JSON parsing for host-specific data—enforcement-critical paths—with behaviour still not fully proven equal to the pinned fork for JSON numbers (documented gap).

Overview
Replaces the overlay’s per-entry-skip CMDB patch (8f0076b81) with the fork’s all-or-nothing loader (overlay/core/0002), plus libntech B-24 strict parse patches (0005/0006) that the loader calls via JsonParseFileStrict(). The manifest records expected post-apply trees and retires the old B-5b reproduction story.

bin/overlay.py now requires applied_tree per component: apply asserts the git tree after git am, with lint and expanded self-tests around mismatches.

Linux CI gains an overlay-build job that checks out upstream at overlay bases, applies patches, compiles, and runs cmdb_test, libntech JSON/mustache tests, 06_host_specific_data, and tests/policy/run.sh with TENDCF_REQUIRE_FORK_LOADER=1—proving the overlay builds, not just that weekly check applies textually (R1 accessor caveat documented).

Docs (upstream-overlay.md, upstream-register.md, AGENTS.md, validate proposal) and small run.sh fixes (policy_completed anchored to ^R:) align with the new overlay behaviour.

Reviewed by Cursor Bugbot for commit 1f379b5. Bugbot is set up for automated code reviews on this repo. Configure here.

djbclark and others added 2 commits November 14, 2026 22:02
…ch strict parse (B-24)

Core patch 0002 is now a squash of the fork's CMDB loader (djbclark/core
overlay/cmdb-all-or-nothing 4158906f2, tag tendcf-pin-4158906) onto the overlay base
plus B-5a; it carries B-7 and B-20. libntech 0005/0006 are fork PR 11's
dc3fba2 and 8a57e55 (B-24). The docs and CI follow in later commits.

Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
Docs: upstream-overlay.md sections 2, 4, 5 (R1: check is textual only) and 7;
upstream-register.md B-5b row and note, new B-24 row, ruling 2; validate
proposal section 13 sentences; AGENTS.md; sim-loader.cf and run.sh header
comments. CI: overlay-build job in linux.yml (upstream at the overlay bases
plus overlay/, cmdb_test, json_test, mustache_test, 06_host_specific_data,
run.sh with TENDCF_REQUIRE_FORK_LOADER=1). run.sh: the completion marker
comment no longer claims the negative greps are the only assertions left, and
policy_completed anchors on '^R: tendcf-test-done:'.

Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
@cursor

cursor Bot commented Nov 15, 2026

Copy link
Copy Markdown

Bugbot couldn't run - usage limit reached

Bugbot is counted against Cursor usage for this user or team, and this run hit a usage or spend limit.

A user or team admin can review and increase usage limits in the Cursor dashboard.

(requestId: 5a998c6f-f8dd-411c-871d-1bc73c58c7d9)

@chatgpt-codex-connector

chatgpt-codex-connector Bot commented Nov 15, 2026 •

Copy link
Copy Markdown
Codex Review Summary: ✅ Completed

This comment shows the latest Codex review activity on this pull request.

Review Status Commit Review trigger
📝 Code Review ✅ Completed 2026-11-15T03:09:45.966791Z 911bd84 PR opened
🔒 Security Review ✅ Completed 2026-11-15T03:12:41.714113Z 911bd84 PR opened
ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review" or "@codex security review".

Codex reacts with 👀 while any review is running, comments if it has suggestions, and reacts with 👍 once all reviews finish with no findings.

Same astral-sh/setup-uv@v5 step as check.yml.

Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
@cursor

cursor Bot commented Nov 15, 2026

Copy link
Copy Markdown

Bugbot couldn't run - usage limit reached

Bugbot is counted against Cursor usage for this user or team, and this run hit a usage or spend limit.

A user or team admin can review and increase usage limits in the Cursor dashboard.

(requestId: 6ef811e7-36a8-4a4c-b90e-eb891e0a3282)

…p setup-uv

- docs/workflow/manifest: the overlay loader is the fork's source bar the R1
  accessor; its tests pass on both builds; name the JSON-number difference
  (B-4, B-10, core rlist.c) with its effect and reach; open with djbclark
- overlay.json applied_tree per component; bin/overlay.py apply fails on a
  mismatch (self-test +3 assertions, lint checks the id shape)
- overlay-build: python3 bin/overlay.py, no setup-uv; summary from the manifest
- nits: B-20 null guards wording, mustache_test and the env var in the §4
  sentence, 0002 test names, register B-23/B-24 cell counts

Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
@cursor

cursor Bot commented Nov 15, 2026

Copy link
Copy Markdown

Bugbot couldn't run - usage limit reached

Bugbot is counted against Cursor usage for this user or team, and this run hit a usage or spend limit.

A user or team admin can review and increase usage limits in the Cursor dashboard.

(requestId: deb810b0-321e-4f4d-bba6-c7903ef79028)

…o 9 cells

- drop unused shutil import; parenthesise the two implicit string
  concatenations in Finding lists (DRIFT and TREE MISMATCH)
- B-22 row: fold the trailing '(test-only, no ticket)' cell into the Upstream
  cell so the row matches the 9-column header

Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
@cursor

cursor Bot commented Nov 15, 2026

Copy link
Copy Markdown

Bugbot couldn't run - usage limit reached

Bugbot is counted against Cursor usage for this user or team, and this run hit a usage or spend limit.

A user or team admin can review and increase usage limits in the Cursor dashboard.

(requestId: 593e6b23-93f7-4df2-b7db-6ff864f06624)

…ired; ruff format; doc nits

- apply_series: compare applied_tree only when HEAD is the pinned base, else
  report 'not compared'; a missing applied_tree on the base is a Finding
- applied_tree is a required component key (lint); self-test sample manifest
  carries it; new cases: missing key, off-base not compared (21 assertions)
- ruff format bin/overlay.py (whole-file, mechanical); ruff check clean
- upstream-overlay.md: self-test count; overlay-build runs build-fork's steps,
  not the hand recipe

Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
@cursor

cursor Bot commented Nov 15, 2026

Copy link
Copy Markdown

Bugbot couldn't run - usage limit reached

Bugbot is counted against Cursor usage for this user or team, and this run hit a usage or spend limit.

A user or team admin can review and increase usage limits in the Cursor dashboard.

(requestId: 4d7d24f7-fb9d-4afc-9fda-8103190a8c47)

@djbclark
djbclark merged commit f542384 into master Nov 15, 2026
9 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant