Skip to content

tally: serve --evaluator-lock from a packaged fixed-argv evaluator, and stop grading a slow seat read as stale - #419

Open
mecattaf wants to merge 3 commits into
mainfrom
tally/evaluator-lock-seat-feeder
Open

mecattaf wants to merge 3 commits into
mainfrom
tally/evaluator-lock-seat-feeder

Conversation

@mecattaf

@mecattaf mecattaf commented Sep 17, 2026

Copy link
Copy Markdown
Owner

Two independent commits so they can be cherry-picked apart: the lock (c27e553) and the feeder (af56e72). Base main. flake.nix, home/tally-uplink.nix and tests/tally-uplink/** are deliberately untouched.

1. --evaluator-lock is served (Int-G6, DF-U-D13-2 discharged)

Why. Without the flag the served kernel derives no verdict at all (tally docs/socket.md §4), so the admission → lease → execute → witness → verdict → receipt loop stopped one rung short of a receipt. DEFERRED.md DF-U-D13-2 deferred passing it on one condition — apps/evaluator existing in the lake to be locked. It exists, in the pinned tally-lake input's own store path.

What.

  • pkgs/tally-evaluator/default.nix (new) — a writeShellApplication (runtimeInputs [jq coreutils git gnused]) whose whole text is exec /bin/sh ${tallyLake}/tools/e2e-evaluator.sh "$@". The argv the kernel hashes is one store word, not the two-word checkout form the audit's probe C ran: a checkout path in a locked argv is a lock over a file git pull can move. Everything per-evaluation still travels on stdin (T7-3) — the only reason an argv can be pinned at all. node is deliberately out of the closure: the stdin item names the interpreter by absolute path.
  • modules/tally-b.nixtallyEvaluatorLock is a runCommand over the kernel's own ${inputs.tally-b}/tools/make-evaluator-lock.sh with --root ${inputs.tally-lake}, --argv ${tallyEvaluator}/bin/tally-evaluator and seven --file rows. No digest is typed into this repo, so the lock cannot drift from either pin without the derivation changing. evaluatorLock's default becomes that derivation; null is still accepted and still means "derive no verdict".
  • tests/tally-b/probe-u-d13-guards.sh — G3 flips from "null omits the flag" to: the default lock builds to a store path, scripts/verify-evaluator-lock.sh --lock … --root … --argv … is rc 0, the same verifier is rc 4 under the pre-wrapper two-word argv (so part C is known to compare something), ExecStart carries that exact lock, and null still omits the flag.
  • tests/tally-b/test-tally-b-input.sh — clause E gains "--evaluator-lock names a store path".
  • docs/local-ai/tally-b-input.md — a section on the lock; rev b3a040e (flake.nix already pinned it and is not edited here).

The campaign is untouched. The CUBS and eval(build:LOCAL-SMOKE) kit entries stay /bin/sh -c true no-ops — new DEFERRED.md row DF-U-D13-4 [SCOPE]. The 58 hand-driven runs were graded under those entries, so flipping them in the same act would change the campaign's semantics with no ruling behind it. Tom's line: integration G7 — is the CUBS validation exit plus revert-and-rerun AC#4's mechanical verdict? The other half is mechanical and is D-A's: an eval(...) entry must hand the launcher a whole per-item JSON object on stdin, which is the render tool's second step.

2. The seat feeder (Int-G11)

MEASURED over ~/.local/state/tally-rewrite/uplink/events.jsonl, 2026-09-16T04Z → 09-17T04Z: cc read STALE-MEASURED on 135 of 286 wakes.

  • CLAUDE_READER_TIMEOUT_SECONDS 12 → 16. D-B54's bound is unchanged (30 + 1 + 20 = 51 < 60); the reads are concurrent, so three 16-second reads fit in one window inside the 20 s TimeoutStartSec, leaving four seconds for shaping and one os.replace per row.
  • On a failed read the retained reading is taken from the reader's own .window-cache-<seat>.json in preference to this feeder's last published row when the two name the same measured instant. The cache is the measurement; the row is a projection that drops cells, so re-publishing the row degraded the retained reading further on every failed read. reading_age_seconds is now the cache's true age — which is what reading_source already claimed by naming that file. stale_reason unchanged.
  • tests/tally-b/probe-seat-feeder-timeout.sh (new) — fake reader, scratch meters dir, no credential opened, the live meters dir neither read nor written, no unit started or restarted. 14 s read over a 30 s-old cache → MEASURED, age 30, no stale_reason; 20 s read → cut off at 16 s, STALE-MEASURED, stale_reason naming TimeoutExpired, reading_source the cache path, age 300.
  • home/seat-feeder.nix carries the new delivered sha256 and the corrected arithmetic comment; docs/local-ai/seat-feeder.md says both halves.

This cannot manufacture a reading from an expired token: re-logging in on cc and cc3 stays Tom's (DF-U-D12-3).

Oracle

nix build --offline --no-link .#checks.x86_64-linux.tally-b-topology \
  .#checks.x86_64-linux.util-sampler-topology                              rc 0
bash tests/tally-b/probe-u-d13-guards.sh                                   rc 0  (11 [P], 0 [F])
bash tests/tally-b/test-tally-b-input.sh                                   rc 1  (see below)
ExecStart | grep -c -- '--evaluator-lock /nix/store/'                      == 1
bash tests/tally-b/probe-seat-feeder-timeout.sh                            rc 0  (14 [P], 0 [F])
grep -c 26d7580 docs/local-ai/tally-b-input.md                             == 0
nix flake check --offline --no-build                                       rc 1  (see below)

Also green offline: tally-uplink-topology, tally-filler-topology, tally-pump-topology, raw-dotfiles-guard.

The one red, reported verbatim and NOT laundered. nix flake check --offline --no-build is rc 1 — and it is the same rc 1 on a detached worktree at main (202d9c3), MEASURED both ways, byte-identical. The tail that reproduces on this branch and on main alike:

… while checking the derivation 'checks.x86_64-linux.nas-topology'
  at flake.nix:1559:9
error: assertion '(! ((builtins).elem 8731 (coordinator).networking.firewall.interfaces.wlp192s0.allowedTCPPorts))' failed
  at flake.nix:1694:11

That is inherited, not this branch's: an eval assertion about a coordinator firewall port, belonging to whoever opened 8731 on wlp192s0. This branch edits no flake.nix and no firewall or NAS configuration — git diff --name-only main...HEAD names 11 files, none of them flake.nix. It is not an offline-input problem, and an earlier revision of this body wrongly characterised it as one.

An earlier pass of this item reached a different check first:

checking derivation checks.x86_64-linux.nas-personal-tailnet...
error: path 'pl6rmijq3dkwqw9cf16wzpycsc7gb9m2-86byf0qaz7f0vgj7x4km4zc9q446skd0-source' is not valid

Which of the two surfaces first depends on evaluation order, so a rerun may show either tail. Both are now carried by DEFERRED.md DF-FLAKE-1, with who discharges them; branch fix/fleet-connectivity-artifact-count is already on it. Clause A of test-tally-b-input.sh IS that command, which is why that script is rc 1 with every other clause [P] — and no [ENV] fence was added to any probe or oracle to turn that rc green.

Negative controls

  • G3's verifier taken red (rc 4) under the pre-wrapper argv, inside the probe.
  • probe-seat-feeder-timeout.sh run against a main worktree goes red in exactly four places — the bound, the landed 14 s read, a spurious stale_reason on a successful read, and reading_source naming the published row. Transcribed in the probe's header.

Left out

🤖 Generated with Claude Code

mecattaf and others added 3 commits September 17, 2026 07:17
…(DF-U-D13-2 discharged)

U-D13 (dotfiles#316), integration gap Int-G6. Without `--evaluator-lock` the
served kernel derives NO verdict at all (tally docs/socket.md §4), so the
admission -> lease -> execute -> witness -> verdict -> receipt loop stopped one
rung short of a receipt. DEFERRED.md DF-U-D13-2 deferred passing it on one
condition — `apps/evaluator` existing in the lake to be locked. It exists, in
the pinned `tally-lake` input's store path, so the row is discharged and the
flag is served.

pkgs/tally-evaluator/default.nix is the FIXED ARGV: a writeShellApplication
(runtimeInputs jq/coreutils/git/gnused) whose text is
`exec /bin/sh ${tallyLake}/tools/e2e-evaluator.sh "$@"`. The argv the kernel
hashes is therefore ONE store word, not the two-word checkout form
(`/bin/sh <checkout>/tools/e2e-evaluator.sh`) the audit's probe C ran: a
checkout path in a locked argv is a lock over a file `git pull` can move.
Everything per-evaluation still travels on stdin (T7-3), which is the only
reason an argv can be pinned at all.

modules/tally-b.nix builds the lock rather than transcribing one: a runCommand
over the KERNEL's own ${inputs.tally-b}/tools/make-evaluator-lock.sh with
--root ${inputs.tally-lake}, --argv ${tallyEvaluator}/bin/tally-evaluator and
the seven --file rows (tools/e2e-evaluator.sh plus apps/evaluator's
bin/evaluate.mjs and src/{evaluate,env,normalize,usage,receipt-ingestion}.mjs).
No digest is typed into this repository, so the lock cannot drift from either
pin without the derivation changing. `evaluatorLock`'s default becomes that
derivation; null is still accepted and still means "derive no verdict".

Guard G3 of tests/tally-b/probe-u-d13-guards.sh flips with it: it builds the
lock offline and runs the kernel's own scripts/verify-evaluator-lock.sh
--lock/--root/--argv (rc 0 — every file row recomputed from the pinned lake,
the argv row recomputed from the wrapper), then takes the same verifier RED
(rc 4) under the pre-wrapper two-word argv so part C is known to compare. The
null branch is kept as the other half of the wire. test-tally-b-input.sh gains
clause E: the flag reaches ExecStart and names a STORE path.

docs/local-ai/tally-b-input.md: a new section on the lock, and the rev is now
b3a040e (flake.nix is untouched — it already pinned it).

DEFERRED.md: DF-U-D13-2 gone, with the discharge recorded; new DF-U-D13-4
[OTHER-REPO] — the CUBS and `eval(build:LOCAL-SMOKE)` kit entries stay
`/bin/sh -c true` no-ops, so serving the lock changes nothing for the campaign
(the 58 hand-driven runs were graded under them). What a mechanical verdict for
CUBS IS is Tom's line (integration G7), and the per-item stdin an `eval(...)`
entry needs is the uplink render tool's second step.

MEASURED: verify-evaluator-lock.sh rc 0 over the built lock; foreign argv rc 4;
probe-u-d13-guards.sh all [P]; tally-b-topology and util-sampler-topology build
offline; `nix flake check --offline --no-build` is rc 1 for the pre-existing
`nas-personal-tailnet` ENV red alone — byte-identical on a `main` worktree.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
…er's own cache

U-D12 (dotfiles#315) / CAP-1 (dotfiles#337), integration gap Int-G11.

MEASURED over ~/.local/state/tally-rewrite/uplink/events.jsonl for
2026-09-16T04Z -> 09-17T04Z: the `cc` row read STALE-MEASURED on 135 of 286
wakes. Two causes, both here.

(1) CLAUDE_READER_TIMEOUT_SECONDS 12 -> 16. D-B54's bound is unchanged
(30 + 1 + 20 = 51 < 60); the three readers are CONCURRENT, so three 16-second
reads fit in ONE window inside the 20-second TimeoutStartSec and leave four
seconds for shaping and one os.replace per row. A read that lands in thirteen
seconds was a MEASURED reading this box threw away for a retained one; the cap
is there so a HUNG reader cannot push the service past its deadline, not to
budget the shaping.

(2) On a failed read the retained reading now comes from the READER'S OWN cache
`.window-cache-<seat>.json` in preference to this feeder's last published row
whenever the two name the same measured instant (`last_measured`'s tuple order
IS the tie-break, and it is now documented as such). The cache is the
measurement; the published row is a projection that drops cells (the five-hour
`resets_at` in the sentinel-window branch, a `model_split` the endpoint
answered null for), so re-publishing the row degraded the retained reading a
little further on every failed read. `reading_age_seconds` is now the cache's
TRUE age — which is what `reading_source` already claimed by naming that file
by path. `stale_reason` is unchanged.

tests/tally-b/probe-seat-feeder-timeout.sh is the fixture: a fake reader and a
scratch meters dir, no credential opened, the live meters dir neither read nor
written, no unit started. A reader that sleeps 14 s over a 30-second-old cache
-> grade MEASURED, reading_age_seconds 30, no stale_reason; a reader that
sleeps 20 s -> cut off at 16 s, STALE-MEASURED, stale_reason naming
TimeoutExpired, reading_source the .window-cache-cc.json path, age 300. Run
against a `main` worktree it goes RED in exactly the four places this change
moves (recorded in its header).

home/seat-feeder.nix carries the new delivered sha256
(59fcc77063f19767d0fdfcf94effb48cafb7f2ef26c4441c92301b147efc8813) and the
corrected arithmetic comment; docs/local-ai/seat-feeder.md says both halves.

Neither change can manufacture a reading from an expired token: re-logging in
on cc and cc3 stays Tom's (DEFERRED.md DF-U-D12-3).

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
…) and defer it

The PR body, commit c27e553 and the item report all called the one red
`checks.x86_64-linux.nas-personal-tailnet` ("path '…-source' is not valid")
and typed it as an offline-input `[ENV]` problem. MEASURED 2026-09-17 on this
branch, `nix flake check --offline --no-build` instead dies at
`checks.x86_64-linux.nas-topology`:

  error: assertion '(! ((builtins).elem 8731
  (coordinator).networking.firewall.interfaces.wlp192s0.allowedTCPPorts))' failed
  at flake.nix:1694

That is a real eval assertion about a coordinator firewall port, not a missing
store path. It is still inherited — this branch edits no flake.nix and no
firewall or NAS configuration, and a detached worktree at main (202d9c3) run
through the same command gives the byte-identical tail — but a reader would
otherwise chase a tail that does not reproduce, and read a real assertion as an
environment excuse.

- DEFERRED.md gains DF-FLAKE-1 `[ENV]`, carrying BOTH tails verbatim, saying
  which surfaces first depends on evaluation order, and naming who discharges
  it (whoever owns the wlp192s0 line that opened 8731; branch
  fix/fleet-connectivity-artifact-count is already on it).
- DECISIONS.md's 2026-09-17 entry gains one paragraph correcting the
  characterisation and pointing at that row.
- DF-U-D13-4 is retyped `[OTHER-REPO]` -> `[SCOPE]`: only the G7 ruling is
  outside this repository, while the actionable half is a reviewed edit to
  home/tally-uplink.nix right here, in D-A's lane.

No code, no test logic and no nix is touched, and no `[ENV]` fence is added to
any probe: clause A of tests/tally-b/test-tally-b-input.sh stays red and the
composite oracle stays rc 1, which is correct and expected.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant