diff --git a/docs/benchmarks/authority/VALIDATION_REPORT.md b/docs/benchmarks/authority/VALIDATION_REPORT.md index b89581e..313de36 100644 --- a/docs/benchmarks/authority/VALIDATION_REPORT.md +++ b/docs/benchmarks/authority/VALIDATION_REPORT.md @@ -17,6 +17,69 @@ on a fixed corpus"). It never belongs inside a customer scan report ("your project"): customer reports contain no benchmark numbers (enforced by `test_customer_reports_contain_no_benchmark_numbers`). +## Offline HTML presentation + +Generate an HTML view from an existing `evaluation.json` without running +the scanner or recalculating metrics. From the repository root, run this +Python snippet (standard library only; no new dependencies): + +```python +import json +from pathlib import Path +from safeai.benchmark.report import write_html + +source = Path("docs/benchmarks/authority/evaluation.json") +evaluation = json.loads(source.read_text(encoding="utf-8")) +write_html(evaluation, "authority-benchmark.html") +``` + +Open `authority-benchmark.html` in a browser. It is self-contained and +works offline, with no scripts, remote fonts, or other network assets. +Keep generated HTML local; it does not need to be committed. + +The page presents all eight attribution levels, per-class Lane-A +verdicts, and ChangeGuard rates in tables, with redundant bars on a 0–1 +scale. Text retains the exact numbers and qualifications from +`render_markdown(evaluation)`; no percentages are rounded or verdicts +reclassified. Missing values and insufficient evidence have no bar; +measured zero remains `0.0`. Known limitations and the existing `Watch` +categories are highlighted. Tables scroll on narrow screens and remain +readable without color or animation. The expandable **Markdown source** +contains the complete current brief, recoverable byte-for-byte as UTF-8 +after HTML entity decoding. + +For trends, load historical committed evaluations exported to local +JSON files, and supply revision labels in the order to display: + +```python +# These files must already exist; choose revisions whose artifacts you +# want to review. No Git lookup, download, or automatic sorting occurs. +history = [ + ("older-revision", json.loads(Path("older-evaluation.json").read_text(encoding="utf-8"))), + ("newer-revision", json.loads(Path("newer-evaluation.json").read_text(encoding="utf-8"))), +] +write_html(evaluation, "authority-benchmark.html", + history=history, label="current-revision") +``` + +The current evaluation is appended after the supplied history. The trend +table shows revision labels, corpus case counts, Lane-A status, +end-to-end exercised authority attribution coverage (including matched +and expected counts), observed false-escalation rates, and missed-material +rates. Each historical brief is available in an expandable section with +its full qualifications and limitations. With no history, only the +current brief is shown; one historical artifact is sufficient to enable +the trend view. Repeated inputs and ordering produce identical UTF-8 HTML +bytes with LF newlines on every platform. + +These are observations on each artifact's corpus, not an improvement +score. Corpus mix, annotations, or truth can change between revisions; +case counts alone do not establish comparability. A rising coverage bar +does not prove scanner improvement, and zero observed false escalations +does not establish universal safety. The original UNKNOWN vetoes, +PROPOSED targets, and surface-blind-change qualifications remain visible +in the briefs. Metric calculation and comparison are unchanged. + ## Executive summary v2.6 Authority Evidence Validation. Overall status: RESEARCH / diff --git a/safeai/benchmark/report.py b/safeai/benchmark/report.py index 8911165..d619aa7 100644 --- a/safeai/benchmark/report.py +++ b/safeai/benchmark/report.py @@ -1,11 +1,16 @@ """Validation report writers for the authority benchmark. JSON is the machine-readable record; Markdown is the reviewer-facing -brief. Both describe the same aggregate dict produced by -``safeai.benchmark.metrics.aggregate``. Neither writer gates anything. +brief, also presented as offline HTML. All describe the same aggregate +dict produced by ``safeai.benchmark.metrics.aggregate``. No writer gates +anything. """ import json +import os +import re +from collections.abc import Sequence +from html import escape def to_dict(aggregate): @@ -317,3 +322,217 @@ def write_markdown(aggregate, path): with open(path, "w", encoding="utf-8") as fh: fh.write(render_markdown(aggregate)) return path + + +_HTML_STYLE = """ +body { margin: 0; background: #f4f6f8; color: #172b3a; + font: 1rem/1.6 system-ui, sans-serif; } +main { max-width: 76rem; margin: auto; padding: 2rem 1rem; } +h1, h2, h3 { line-height: 1.25; } +h1 { font-size: 2rem; } h2 { margin-top: 2rem; } +section { background: white; padding: 1rem; margin: 1rem 0; + border: 1px solid #cad4de; border-radius: .4rem; } +.table-scroll { overflow-x: auto; } +table { border-collapse: collapse; width: 100%; font-variant-numeric: tabular-nums; } +caption { text-align: left; font-weight: bold; padding: .5rem 0; } +th, td { text-align: left; vertical-align: top; padding: .65rem; + border-bottom: 1px solid #cad4de; overflow-wrap: anywhere; } +thead { background: #edf2f6; } +tbody th { min-width: 10rem; } +meter { display: inline-block; width: 6rem; height: .8rem; margin: 0 .5rem; } +meter::-webkit-meter-optimum-value { background: #526b86; } +meter::-moz-meter-bar { background: #526b86; } +.limitations { border-left: .4rem solid #8a4b08; background: #fff8eb; } +.limitations li { margin: .65rem 0; } +.watch { font-weight: bold; } +summary { cursor: pointer; font-weight: bold; padding: .6rem 0; } +pre { white-space: pre-wrap; overflow-wrap: anywhere; } +@media print { body { background: white; } main { padding: 0; } + .table-scroll { overflow: visible; } } +""" + +# Only decorate measured numeric strings already printed by the Markdown +# writer. Never infer a zero, round a value, or assign a verdict from a bar. +_BAR_VALUE = re.compile( + r"(?P