Skip to content

Add a second blind fixture, a scoring script and its first scored run - #12

Merged
prathikcannotcode merged 2 commits into
mainfrom
eval/second-matter
Nov 12, 2026
Merged

prathikcannotcode merged 2 commits into
mainfrom
eval/second-matter

Conversation

@prathikcannotcode

@prathikcannotcode prathikcannotcode commented Nov 11, 2026 •

Copy link
Copy Markdown
Owner

Morvali Creek stopped being blind when the November source pack filled the four gaps it exposed, and its run never exercised Aranya's own hooks or skill routing, because the session started in the Vigil folder. This adds a second synthetic matter with an answer key, a run sheet, a scoring script and the first blind run's score, following Vigil's tamarind-lend fixture (prathikcannotcode/Vigil#9).

No skill, script, schema or catalog entry changes.

Run 1

Measure Result
Planted issues found 14 of 14, none partial
Should-not-flag items clean 6 of 6
Findings valid on real catalog ids 16 of 16
Uncatalogued or invented authority relied on 0 (Water Act ss.24, 25, 26, 33A, NGT s.18(2) and the invented case all went to flags)
verify / unverified states carried Yes: 3 findings carry verify; the unverified order is kept out of the prayer
Calculator units caught Both: 7.28 ha and 650000 INR in a labelled second run
Aranya's five skills in the session Yes; env-case-analysis invoked through the Skill tool
Write hooks live Yes: after the run, the guard refused an expert_validated write and a write outside the layout
Web calls / key or run-sheet reads 0 / 0

The run also caught a mistake in the key. W1 said the Unit operated without consent until January 2024, but it was closed from 3 October 2023. The key now has that fix and eight real gaps the run found as known_gaps, including a bad restoration lag in the consultant's inputs. The score stands against the original key.

Read SCORE.md before quoting the number.

  • The same model wrote the record and the key and ran the matter, in separate sessions.
  • The scorer wrote the key.
  • The key was built from the catalog's text.
  • The run used Fail closed on bad hook input and settle the Morvali Creek calls #11's head (918a2e6); on main alone, two of its CPCB findings would fail validation.
  • No lawyer has reviewed any of it.

What's here

  • examples/kanthari-nala/: an invented record. A textile dyeing unit bypassed its effluent plant into a nala in 2023, then pumped effluent into a borewell in 2025. The Board closed it twice and levied compensation twice, and the unit's appeal is pending.
  • examples/kanthari-nala.expected.json: 14 planted issues, 6 should-not-flag items, 8 known gaps, calculator checks, watch terms, sources for every legal claim and an unverified list.
  • examples/kanthari-nala.run-sheet.md: the blind procedure. The session starts in the exported Aranya folder with the record at record/, works offline and ends with a hook probe. A separate session scores.
  • examples/score-check.mjs: works for any Aranya run folder and key, Morvali included. It validates findings as the hook does, tallies flags and calculator runs, checks inputs against calc_checks, prints watch lines and tallies a scorer's map. It judges nothing.
  • examples/kanthari-nala.run-1/: the run's matter folder unchanged, plus score-map.json and SCORE.md.

Order

Merge #11 first. This PR's run depends on it, and its tests here run without it.

Found while building it

catalog/provisions.json points 7 times at catalog/patches/2026-11-source-pack/QUESTIONS-FOR-COUNSEL.md (Q1 to Q3), which release 0.3.0 deleted. The run hit the missing file and reported it. The fix is to restore the file under catalog/, which the layout lets agents write, or to repoint the notes by hand.

Checks

  • npm run validate -- --strict: 0 errors
  • npm test: 29 tests, 27 pass, 2 skipped on Windows (branch cut from main, without Fail closed on bad hook input and settle the Morvali Creek calls #11)
  • score-check.mjs reproduces the run 1 tallies from score-map.json
  • Run transcript searched for the key, run sheet, scoring files, repo path and web calls: none
  • A lawyer's review of the key

Synthetic data only. Not legal advice.

🤖 Generated with Claude Code

prathikcannotcode and others added 2 commits November 12, 2026 01:07
examples/kanthari-nala is a synthetic record with an answer key and a
run sheet for a blind, separately scored run. examples/score-check.mjs
validates and tallies any run folder against a key; the scorer judges.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
examples/kanthari-nala.run-1 holds the run's matter folder unchanged,
the scorer's score-map.json and SCORE.md. 14 of 14 planted issues
found, 6 of 6 should-not-flag items clean, all 16 findings valid. A
post-run probe showed the write hooks live in the run's workspace.

The key gains eight known gaps from the run and a W1 correction the run
pointed out. The run sheet now puts the record inside the workspace and
adds the hook probe, as this run did.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@prathikcannotcode prathikcannotcode changed the title Add a second synthetic matter and a scoring script Add a second blind fixture, a scoring script and its first scored run Nov 12, 2026
@prathikcannotcode
prathikcannotcode merged commit 5abcb73 into main Nov 12, 2026
1 check passed
@prathikcannotcode
prathikcannotcode deleted the eval/second-matter branch November 12, 2026 06:01
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant