Repository navigation
fix/review 2026 11 - #35
Merged
Merged
Conversation
lluvr
commented
Nov 7, 2026
lluvr
commented
Owner
- Classify an empty source as insufficient_input in Verifier.score
- Pin known gaps for changed and fabricated numbers with strict xfail tests
- Cap MCP verify and measure string inputs at 200,000 characters
- Label accuracy, threshold and lift figures with what they measure
- Stop quoting unreproduced AUCs for the MiniCheck and AlignScore modes
An empty or whitespace-only source let the source-dependent layers run against nothing, so the result came back validated with a high probability. It now returns scope insufficient_input, as empty text does. source=None still raises ValueError, matching the documented required str argument. Signed-off-by: lluvr <lovro.lucic@gmail.com>
…ests Swapped digits score below a verbatim copy, an all-fabricated text stays under the default threshold, Layer 11 accepts a digit swap as derived within its tolerance, and short sentences and table rows are never classified. Each case is a strict xfail so it flips when the gap closes. A passing test pins that Layer 4 reports the changed numbers. Signed-off-by: lluvr <lovro.lucic@gmail.com>
The server accepted arbitrarily large arguments, so a multi-megabyte text or source tied it up for seconds per call. Each string argument is now checked against MAX_INPUT_CHARS and rejected with a ToolError naming the field. The library API is unchanged. Signed-off-by: lluvr <lovro.lucic@gmail.com>
The 0.67-0.76 AUC range and the 0.07-0.27 thresholds were attributed to the substrate-only Verifier but come from raw Layer 6 and per-detector scores. The docs now give the Verifier's own AUCs and F1-optimal thresholds, cite the out-of-sample RAGTruth figure and the 70/30 calibration split next to them, and state triage lift as the range across corpora. Lines saying docs ship in the wheel now link to the repository, and the quickstart notes that a changed or fabricated number may not move the score and points to Layer 4. Signed-off-by: lluvr <lovro.lucic@gmail.com>
No benchmark in the repository produces AUC 0.76 or 0.77 for the combined Verifier modes. 0.76 matches MiniCheck's own mean across the three corpora, so the docs now say that and describe the combined modes as not yet evaluated. Signed-off-by: lluvr <lovro.lucic@gmail.com>
Reproduce the shipped substrate_only fit exactly (per-class default_rng(0) 70/30 split of the 900-row RAGTruth Summary file, sklearn LogisticRegression defaults) and refit with the unsupported features bounded at zero. The constrained fit matches the current AUC on the RAGTruth held-out rows, SummEval and HaluEval, but a digit swap still scores the same as a verbatim copy and no xfail case flips, so the default stays. l4_unsourced carries no signal on RAGTruth; its coefficient is noise around zero. Commit the 900-row feature snapshot so the evaluation runs without a corpus download, and report the result in production_readiness 4.2.11. Signed-off-by: lluvr <lovro.lucic@gmail.com>
The 0.633 quoted as out of sample was the eval half of the n=400 interleave split; 141 of its 199 rows are training rows of the 70/30 split the default calibration was fit on. On the 271 held-out rows the shipped calibration scores 0.677 (95% CI 0.599-0.748). Correct the README, API reference, production readiness, the Verifier docstring and the MCP list_modes description, and note that the l4_unsourced sign is noise rather than a learned relationship. Signed-off-by: lluvr <lovro.lucic@gmail.com>
measure() stamps library_version from clarethium_touchstone.__version__ (0.2.2) while the touchstone-mcp distribution is 0.1.11. That is the documented policy (CHANGELOG: package and library versioned independently; list_modes reports all three), so the code stays and the README now explains which number is which. A smoke test pins that library_version is the library version and that the distribution version is reported separately by touchstone_mcp. Signed-off-by: lluvr <lovro.lucic@gmail.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.