[camerontjs-dot]

What does this change?

Adds a self-contained check_with_evidence() example in examples/camerontjs-dot_compare_fn/, comparing a weak token-overlap compare_fn with a pinned local DeBERTa NLI compare_fn on the same ten fixed claim/evidence cases. Prints each arm's BOOTH status, reason, detail, and evidence_agreement, along with passage-level NLI labels. It adds no BOOTH core dependency or behavior change.

Addresses issue #18. Related to issue #2.

Why?

BOOTH deliberately delegates evidence agreement to the caller's comparator. On these illustrative cases, the lexical comparator accepted wrong refund quantities, negation, and scope mutations and rejected a legitimate paraphrase. The NLI comparator made better checkpoint decisions, but its simple max-entailment projection also accepted genuinely conflicting evidence. The NLI model itself assigned high support to one passage and high contradiction to another; the scalar projection discarded that distinction. Both improvements and limitations are part of the example.

Reproduction and evidence

• boothpy==0.5.4 installed from PyPI, not imported from the BOOTH checkout; MoritzLaurer/DeBERTa-v3-base-mnli-fever-anli pinned at revision 6f5cf0a2b59cabb106aca4c287eed12e357e90eb.
• Local CPU runs on CPython 3.9.6/macOS arm64. Two initial runs and the review-revision rerun produced byte-identical stdout in this local environment; full-precision floats and stdout bytes are not guaranteed identical on other hardware.
• Authored expected checkpoint decisions: 4/10 lexical vs 9/10 NLI. The NLI remaining false acceptance is the conflict case, retained and explained.
• The complete real stdout, case-by-case reason codes/scores, exact versions, hashes, and limitations are in RESULTS.md.
• Full checkout-source suite (PYTHONPATH="$PWD/src" python -m pytest -v): 348 passed locally.
• The missing-dependency path was checked and fails with a friendly explanation. Pinned model/tokenizer files total approximately 380 MB in the inspected local cache at that revision; the first run downloads them if not cached.
• Only the requested three files in this contributor's example folder are changed.

Checklist

• I ran the full test suite locally (pytest -v) and it passes.
• I added a new regression test that fails without this change: not applicable, because this is an example-only PR, not a BOOTH behavior fix. The script itself contains ten controlled demonstration cases with printed expectations and outcomes.
• No existing tests or assertions are changed.
• No public BOOTH behavior is changed; no changelog entry required.
• The actual limitation and counterexample are preserved in RESULTS.md.

Anything else worth knowing?

This is a small author-constructed comparison, not a statistical accuracy benchmark, a calibrated probability claim, or an improvement to BOOTH's semantic checker (BOOTH does not bundle one). The lexical and NLI scores use different, explicitly documented uncalibrated thresholds (0.5 vs 0.7).

The scope-swap case also reveals that DeBERTa's fine-grained contradiction label can be wrong even when BOOTH's overall decision to withhold acceptance is appropriate. The example intentionally does not silently add conflict-resolution policy or change the library to eliminate those failures.

[Vedantgitbot]

Thanks @camerontjs-dot, this is a strong example. It's scoped correctly, adds no dependency to BOOTH, and the conflicting-evidence case is documented clearly rather than tuned away.

A few small changes before I merge:

1. Remove the stray "CAL V1" reference in RESULTS.md. It isn't related to BOOTH.
2. Reproduction section: the script uses boothpy==0.5.4 from PyPI, not the checkout. Please say which one the example relies on.
3. Stdout hash: note that it only holds for your environment. The full-precision floats may differ on other hardware, though the statuses should match.
4. Model download: please state the approximate size in the docstring and RESULTS.
5. PR description: please link Add an LLM Provider Integration Example & Benchmark #2 and Reference compare_fn examples: weak keyword matcher vs. a real one #18 there too.

Once those are in I'll run it on my side and merge.

[camerontjs-dot]

Thanks for the review. I addressed all five requests in 6e4b2c1:

1. Removed the unrelated CAL reference from RESULTS.md.
2. Clarified that the example imports boothpy==0.5.4 from PyPI, not BOOTH from the checkout, with clean-environment reproduction commands.
3. Restricted the byte-identical stdout claim to the recorded macOS arm64 environment; full-precision scores and hashes may differ across hardware.
4. Added the approximate first-run download size to both the script docstring and RESULTS.md: 380 MB of observed pinned model/tokenizer files (369 MB weights, 11 MB tokenizer/config).
5. Added explicit links to #18 and #2 in the PR description.

I reran the revised script on Python 3.9.6/macOS arm64 with the PyPI package: exit 0; stdout still matches the recorded hash and ten-case output. The full checkout-source suite (PYTHONPATH="$PWD/src" python -m pytest -v) passed 348 tests; the missing-dependency path still exits with a clear installation message. The final PR touches only the original three example-folder files. The 4/10 vs 9/10 authored-control result, conflicting-evidence false acceptance, and scope-labeling limitation are unchanged. This remains a comparator demonstration, not an accuracy benchmark or a BOOTH core improvement.

GitHub's new-head workflow currently shows action_required, not a reported test failure. Your independent reproduction and merge decision remain pending.

[Vedantgitbot]

Thanks @camerontjs-dot, merged. Great example, and the conflicting-evidence case is a useful addition to the docs.