[Birfy]

What

jitagent verify <case-dir> now answers an unusable case directory with one line on stderr, nothing on stdout, and exit code 2 — instead of raising out of Case.load. A valid case directory renders exactly as before.

Fixes #20.

Why

verify reads a hand-edited case.json and was the one command left that trusted it. Measured on main at 34b3d2f (CPython 3.12.16, PYTHONPATH=src, scratch JITAGENT_HOME, stdout empty every time):

case directory	what verify did on 34b3d2f
does not exist / no case.json	FileNotFoundError, 22 stderr lines, rc 1
case.json is invalid JSON	json.decoder.JSONDecodeError, 24 lines
case.json holds an array	AttributeError: 'list' object has no attribute 'get', 16 lines
case.json is UTF-16, not UTF-8	UnicodeDecodeError: 'utf-8' codec can't decode byte 0xff …, 23 lines
no "spec" key	KeyError: 'spec', 16 lines
an example carries "notes" for "note"	TypeError: Example.__init__() got an unexpected keyword argument 'notes', 16 lines
the spec carries "timeout" for "timeout_ms"	TypeError: Spec.__init__() got an unexpected keyword argument 'timeout', 16 lines
the named impl file is missing	FileNotFoundError, 22 lines

Exit code 1 made that worse than noise: cmd_verify already returns 1 for "the implementation failed verification", so a broken case file and a REJECTED verdict were indistinguishable to a script.

This is the half of #13 that was deliberately left out — its body says "verify has the same shape (Case.load on a bad directory raises FileNotFoundError) and is deliberately left out of this change". PR #14 fixed the compile door only, with the same exit code 2.

The contract now

cannot read '/tmp/case/case.json': No such file or directory
cannot read '/tmp/case/case.json': not UTF-8 text ('utf-8' codec can't decode byte 0xff in position 0: invalid start byte)
'/tmp/case/case.json' is not valid JSON: Expecting property name enclosed in double quotes: line 1 column 16 (char 15)
'/tmp/case/case.json' must hold a JSON object with "spec", "examples" and "expect", not an array
'/tmp/case/case.json' is missing "spec"
'/tmp/case/case.json': "spec" has an unknown key "timeout"
'/tmp/case/case.json': example 0 has an unknown key "notes"
'/tmp/case/case.json': "examples" must be a list of examples
'/tmp/case/case.json': example 0 needs both "input" and "output" (missing 'output')
'/tmp/case/case.json' needs "expect" to be an object with a "level"
cannot read '/tmp/case/impl.py': No such file or directory

Paths are printed with repr, so a path containing a newline or a control character still produces a single line.

The unknown-key messages are not cosmetic: parsing with Example.from_dict / Spec(**…) while ignoring extra keys would have turned a typo into silence — the case would run with the intended value missing and the author would never be told.

How

• src/jitagent/cases.py: CaseError(RuntimeError), and Case.load validates what it reads — file, JSON, top-level object, required keys, spec/examples/expect shapes, key names against the dataclass fields, impl file — raising one of the lines above.
• src/jitagent/cli.py: cmd_verify turns CaseError into print(e, file=sys.stderr); return 2. No other line in cli.py changes.
• CHANGELOG.md: [Unreleased] entry.
• tests/test_verify_case_errors.py: 16 tests — 14 refusals (incl. the UTF-16 file, the deep-parse file and both unknown-key shapes) plus 2 pins that keep 0/1 meaning what they meant.

jitagent selftest shares Case.load through load_all. A broken corpus case now raises CaseError instead of KeyError there too, but selftest's own reporting is untouched and deliberately out of scope, the same way #13 scoped verify out.

Beyond the issue (found in review)

Security's pass found one more escape of the same class inside the same function: json.loads raises RecursionError — not JSONDecodeError — on a deeply nested case.json, so a 10,000-deep file still came back as rc 1 with a 25-line traceback, indistinguishable from a REJECTED verdict. That is a realistic input here, not a hypothetical: this repo's own #16 and #18 are deep-nesting reports, so anyone reproducing either of them writes a deep case file and meets this first.

Now caught in the same place, with the reason and no invented depth number (the threshold is stack-dependent): '<path>/case.json' nests too deeply to parse. One test builds the file as text ("[" * 50000 + "]" * 50000) rather than dumping a deep value, so the test itself does not depend on the writer.

Evidence

• pytest -q on this branch → 118 passed (102 on 34b3d2f + 16 new), 0 failed.
• The new file on unmodified 34b3d2f → 14 failed, 2 passed; the two passing are the pins (test_a_valid_case_still_verifies, test_a_wrong_implementation_is_still_rejected). Red is behavioural: each refusal test dies on the exception the old loader raised (the deep-parse one on RecursionError), not on a missing constant.
• jitagent selftest → 6/6; python tools/audit_traps.py → 18/18.
• Every repro dir behind the table above (nine of them, counting a missing directory and a missing case.json separately): rc 2, stdout 0 bytes, exactly 1 stderr line each, no Traceback.
• Valid path unchanged, measured against 34b3d2f after normalising the took: Nms line: src/jitagent/corpus/01_correct → rc 0, 194 stdout bytes, 0 stderr; 02_wrong_no_clean → rc 1, 1687 stdout bytes, 0 stderr. Both diff-clean.
• CI at a28347f: run 41946932814 — 4/4 success (test (3.11), test (3.12), test (3.13), package).

Known, not in this diff

Recorded from the review passes so the next change has them, not fixed here (all pre-existing or below the line for this job):

• jitagent selftest shares Case.load through load_all, so a broken corpus case still tracebacks there. Scoped out deliberately, as jitagent compile answers a bad requirement file with a traceback, not an explanation #13 scoped verify out of Explain a bad requirement file in one line instead of a traceback #14.
• An unknown key is echoed in full, so a 500,000-character key produces a single 500 KB stderr line — one line, but hard to read and easy to log-flood. Truncating to ~60 characters would fix it.
• Strictness is asymmetric: unknown keys inside spec/examples are refused, unknown top-level keys are still ignored, so a top-level typo is silent.
• _SPEC_KEYS/_EXAMPLE_KEYS are hand-kept copies of the dataclass fields; a future Spec/Example field would be rejected as "unknown" until they are updated.
• _read is a plain read_text: a case.json symlinked to a FIFO or /dev/zero blocks until the caller gives up. Pre-existing.
• expect.level is not validated against the three Level values: {"level": "BANANA"} loads and verify exits 0 (it does not consult expect at all). A typo in the expected verdict is silent.
• A UTF-8-BOM case.json (Notepad's default save) is refused in the invalid-JSON branch with CPython's own hint, Unexpected UTF-8 BOM (decode using utf-8-sig) — correct, since RFC 8259 has no BOM, but it is Python-ese in a user-facing line. Reading with utf-8-sig would accept the file instead.
• No test yet for a missing top-level expect, a non-string impl, or a case path that does not exist at all (the missing-file test creates the directory).
• The 0/1/2 contract is stated only in the CHANGELOG: verify --help and the README still describe verify without it, which is where a caller looks.
• impl is joined without confinement, so "impl": "../../x.py" reads a file outside the case directory. Pre-existing, read-only, and not introduced here.

Risk

Low. verify writes nothing; the change only runs on the path taken before a case is loaded, and every existing case-file shape that used to load still loads. Rollback is git revert of the two commits on the branch (three source files, one new test file).

_{Fledge · jitagent · Dev-1 · implemented}

[Birfy]

Review at a28347f — 93/100

Case.load now answers an unusable case directory in one sentence, and cmd_verify maps that to exit 2 while 1 keeps meaning "the implementation failed". I read 65aece2 and re-ran everything at this head; the delta since then is exactly the RecursionError branch and its test.

Verified by running (CPython 3.12.16, PYTHONPATH=src)

• pytest -q → 118 passed. On unmodified 34b3d2f the same new file is 14 failed / 2 passed — the two pins (valid case rc 0, wrong impl rc 1) still pass there, so the rest are real coverage.
• Deep case.json: 9 000 nested arrays parse and are refused as an array in one line; 12 000 / 20 000 / 50 000 → rc 2, exactly one stderr line nests too deeply to parse, empty stdout, no traceback. The escape I flagged on 65aece2 is closed.
• Ordinary authoring mistakes — missing directory, missing case.json, invalid JSON, non-object top level, missing spec/examples/expect, unknown key in spec and in an example, examples not a list, example missing input/output, expect without a level, missing impl, UTF-16 — each give rc 2, one stderr line, 0 stdout, no traceback.
• Valid output unchanged: 01_correct rc 0 and 02_wrong_no_clean rc 1, byte-identical to 34b3d2f after normalising took: (same md5).
• selftest 6/6; audit_traps 18/18; CI 4/4 (41946932814); mergeable_state: clean; base 34b3d2f.

Blocking: none.

Non-blocking

1. cases.py:114-116 and :128: expect is checked only for being an object with a level key, never for the value — {"level": "BANANA"} loads and verifies with rc 0. A typo in the expected verdict is silently accepted; validate against the three Level values.
2. cases.py:69-70: a UTF-8-BOM case.json (Notepad's default) is refused as is not valid JSON: Unexpected UTF-8 BOM (decode using utf-8-sig), while the CHANGELOG says "a file that is not UTF-8" — a BOM'd file is UTF-8. Read with utf-8-sig, or name BOM in the changelog; there is no BOM test (the encoding test at tests/test_verify_case_errors.py:87 covers UTF-16 only).
3. cases.py:15-17: _SPEC_KEYS/_EXAMPLE_KEYS repeat the dataclass fields by hand; when Spec/Example grows a field the message will call a valid key "unknown". Deriving them from dataclasses.fields(...) keeps the list honest and feeds _SPEC_REQUIRED from one place.
4. Coverage gaps: no test for a missing top-level expect (the :86 loop covers it, but tests/test_verify_case_errors.py:125 parametrises only spec/examples), for a non-string impl (cases.py:118-120), or for a case path that does not exist at all (:81 creates the directory).
5. cases.py:121: impl is joined without confinement — "impl": "../../x.py" reads a file outside the case dir. Pre-existing, not introduced here, and only read as source, but the PR now prints that path.
6. The 0/1/2 contract lives only in the CHANGELOG; verify --help (cli.py:309) and the README do not state it, and callers are exactly who must tell rc 2 from rc 1.

Merge note: this branch adds ## [Unreleased] at the same CHANGELOG line as #7/#8/#10 — those four have to merge in sequence.

93/100. COMMENT: right layer, one-line diagnostics, tests genuinely red on base, valid cases byte-identical, and the decode-recursion escape is now fixed with a test that writes text rather than relying on the JSON writer. The remaining points are type validation, message wording and docs — none block. Not approved, not merged.

Ran vs read: ran — tarball at a28347f, diff against 65aece2 (only cases.py:71-72 + one test + one CHANGELOG line), full pytest -q (118 passed), new file on base 34b3d2f (14 failed / 2 passed), deep-JSON sweep 9k/12k/20k/50k, ~10 broken-dir/edge CLI probes (BOM, expect.level, traversal), corpus render comparison vs base, selftest, audit_traps, CI/mergeable via API; read — cases.py/cli.py at line level, the 16 tests, issue #20. Not run: 3.11 (not on this machine). I posted nothing to GitHub.

Posted by the Lead on the Reviewer's behalf: the Reviewer wrote and ran this review and has no write credential for this repository.

_{Fledge · jitagent · Reviewer · reviewed}

[Birfy]

Fledge automation · jitagent · job job_UDAE0akszfaP5g (schedule, iterate)

Automated run: Lead wrote · Product claimed · Dev-1 implemented · Test tested · Reviewer reviewed.

Fledge does not merge by default; a human must review and merge.

• Issue: jitagent verify answers a broken case directory with a traceback, not an explanation #20
• Fledge job: https://fledge.hugoxk.online/repos/repo_al_v9DflQ4qkmoDD/jobs/job_UDAE0akszfaP5g

[Birfy]

Re-review at a28347f — 93/100

Head unchanged since the last review at this SHA; re-ran everything just now and the score stands.

Verified by running (CPython 3.12.16, PYTHONPATH=src)

• pytest -q → 118 passed; on unmodified 34b3d2f the same new test file is 14 failed / 2 passed (the two pins), re-checked today; selftest 6/6; audit_traps 18/18; CI run 41946932814 success (3.11/3.12/3.13/package); mergeable_state: clean; base 34b3d2f.
• Deep case.json: 9 000 nested arrays → rc 2, one line refusing a non-object; 12 000 / 20 000 / 50 000 → rc 2, exactly one stderr line nests too deeply to parse, empty stdout, no traceback.
• A valid case still renders unchanged (same bytes as base after normalising took:).

Blocking: none.

Non-blocking

1. cases.py:114-116, :128: expect is checked for having a level key, never for its value — {"level": "BANANA"} loads and verifies rc 0.
2. cases.py:69-70: a UTF-8-BOM case.json is refused as "not valid JSON" while the CHANGELOG says "not UTF-8"; there is no BOM test (the encoding test covers UTF-16 only).
3. cases.py:15-17: _SPEC_KEYS/_EXAMPLE_KEYS duplicate the dataclass fields by hand.
4. Coverage gaps: missing top-level expect, non-string impl, a nonexistent case path (:81 creates the directory).
5. cases.py:121: impl is joined without confinement — "impl": "../../x.py" reads outside the case dir (pre-existing).
6. The 0/1/2 contract lives only in the CHANGELOG, not in verify --help (cli.py:309) or the README.
7. Merge note: this branch inserts ## [Unreleased] at the same CHANGELOG line as #7/#8/#10.

93/100. COMMENT — not approved, not merged.

_{Fledge · jitagent · Reviewer · reviewed}