Release 0.3.0 with the first blind, separately scored run

CHANGELOG.md

@@ -1,6 +1,9 @@
 # Changelog
 
-## Unreleased
+## 0.3.0 (9 November 2026)
+
+### Accuracy
+- First blind, separately scored run: `examples/kadamba-fit.run-1` holds the report and score. 16 of 16 planted issues found, 3 of 13 should-not-flag items flagged. The README's "How accurate is it?" section has the caveats.
 
 ### Skills
 Changes from the first real run of `/onboard` and `/audit`, on `examples/acme-demo`.

CITATION.cff

@@ -6,8 +6,8 @@ authors:
   - name: "Vigil contributors"
 repository-code: "https://github.com/prathikcannotcode/Vigil"
 license: Apache-2.0
-version: 0.2.0
-date-released: 2026-11-02
+version: 0.3.0
+date-released: 2026-11-09
 keywords:
   - dpdp
   - gdpr

README.md

@@ -47,7 +47,20 @@ That picture is **sample-co**, a made-up lending app. Its files were written by
 - [`report.json`](assessments/2026-10-19-sample-co-baseline/report.json): 9 schema-valid findings with owners, due dates and cross-references between DPDP and GDPR
 - [`breach-log.jsonl`](company-profile/sample-co/breach-log.jsonl): an incident from detection to CERT-In (5h 25m) and the EU authority (44h 10m), one append-only line per step
 
-To watch a real run, audit [`examples/acme-demo`](examples/acme-demo), a small synthetic app with planted problems, and score the result against [the list of what it should find](examples/acme-demo.expected.json). Quick start step 5 shows how. That list is also the place to start if you want to measure Vigil's accuracy; no benchmark results are published yet.
+To watch a real run, audit [`examples/acme-demo`](examples/acme-demo), a small synthetic app with planted problems, and score the result against [the list of what it should find](examples/acme-demo.expected.json). Quick start step 5 shows how.
+
+## How accurate is it?
+
+One blind run so far, on [`examples/kadamba-fit`](examples/kadamba-fit): a 55-file synthetic fitness app whose code states none of its gaps. One session ran `/audit` without seeing the answer list, and a separate session scored it. The [score](examples/kadamba-fit.run-1/score.md):
+
+| Measure | Result |
+|---|---|
+| Planted issues found | 16 of 16 |
+| Citation anchors cited exactly | 45 of 48 |
+| Should-not-flag items flagged anyway | 3 of 13; two were mistakes in the answer list, since corrected |
+| Fixture values copied into any output | 0 |
+
+Treat that as a floor check, not a benchmark. It is one run on synthetic code, and Claude wrote the app, ran the audit and scored it. The answer list is now public, so a fair re-run copies the app out of the repo and never opens [`examples/kadamba-fit.expected.json`](examples/kadamba-fit.expected.json) or the [run sheet](examples/kadamba-fit.run-sheet.md).
 
 ## Why Vigil
 
@@ -188,7 +201,7 @@ Skills are Markdown that tell Claude what the law requires and how to check for
 | Schema validation | `validate-on-write.mjs`, `npm run validate` | A finding without a clause or severity, a breach entry without an event, a posture with an unknown status. 9 schemas. |
 | Secrets | `write-guard.mjs` | AWS, GitHub, OpenAI and Anthropic-style keys, private keys, password assignments |
 | No personal data in findings | `write-guard.mjs`, skill rules, schema shape | Aadhaar, PAN and +91 mobile numbers in company or assessment data. Findings carry `file` and `line`, never the value found there. Other identifiers rest on the skill rules. |
-| Tests | `npm test`, GitHub Actions | Regressions in all of the above. 23 tests, Node's built-in runner. |
+| Tests | `npm test`, GitHub Actions | Regressions in all of the above. 24 tests, Node's built-in runner. |
 
 Known limit: hooks guard Claude Code's Write, Edit and MultiEdit tools. A shell command run through Bash is not intercepted, so `.claude/settings.json` only pre-approves the validate and test commands. Details in [SECURITY.md](SECURITY.md).
 

examples/kadamba-fit.run-1/score-check.mjs

@@ -0,0 +1,72 @@
+// Scoring checks for the kadamba-fit cold audit. Run from the repo root:
+// node examples/kadamba-fit.run-1/score-check.mjs examples/kadamba-fit examples/kadamba-fit.expected.json
+import { readFileSync, readdirSync, statSync, existsSync } from 'node:fs';
+import { join } from 'node:path';
+
+const [appDir, keyFile] = process.argv.slice(2);
+const dir = 'examples/kadamba-fit.run-1';
+const r = JSON.parse(readFileSync(join(dir, 'report.json'), 'utf8'));
+const key = JSON.parse(readFileSync(keyFile, 'utf8'));
+const by = Object.fromEntries(r.findings.map(f => [f.id, f]));
+// Which findings cover which planted issue: the scorer's judgement, recorded in score.md
+const map = { P1: ['F-0002'], P2: ['F-0007', 'F-0012'], P3: ['F-0011'], P4: ['F-0010'], P5: ['F-0008'], P6: ['F-0020', 'F-0021', 'F-0022', 'F-0023', 'F-0026'], P7: ['F-0005'], P8: ['F-0017'], P9: ['F-0015'], P10: ['F-0006'], P11: ['F-0004', 'F-0016'], P12: ['F-0024'], P13: ['F-0003'], P14: ['F-0001'], P15: ['F-0027'], P16: ['F-0009'] };
+
+const norm = t => t.replace(/Section\s+/g, 's.').replace(/\s+/g, ' ');
+const tokens = c => c.split(/[;,]/).map(x => x.replace(/^\s*(DPDP|GDPR)\s+/, '').trim()).filter(Boolean).map(x => x.startsWith('CERT-In') ? 'CERT-In' : x);
+const top = t => (t.match(/^(s\.\d+|Rule \d+|Art\. \d+)/) || [t])[0];
+// Token present and not followed by a digit, so s.1 does not match s.11
+const has = (text, tok) => { for (let i = text.indexOf(tok); i !== -1; i = text.indexOf(tok, i + 1)) if (!/[0-9]/.test(text[i + tok.length] ?? '')) return true; return false; };
+
+let exact = 0, level = 0, total = 0, sheetRight = 0;
+console.log('--- citation anchors (exact-or-deeper / top-level)');
+for (const e of key.expected) {
+  const text = norm(map[e.id].map(id => `${by[id].clause}; ${by[id].cross_reference ?? ''}`).join(' ; '));
+  const res = tokens(e.clause).map(t => { total++; const ex = has(text, t), lv = ex || has(text, top(t)); if (ex) exact++; if (lv) level++; return `${t}=${ex ? 'exact' : lv ? 'level' : 'MISS'}`; });
+  if (res.some(x => !x.endsWith('MISS'))) sheetRight++;
+  const files = map[e.id].map(id => by[id].file);
+  console.log(e.id, map[e.id].join(','), '|', res.join('  '), '| file', files.includes(e.file) ? 'match' : 'OTHER:' + files.join(','));
+}
+console.log(`exact ${exact}/${total}, top-level ${level}/${total}, run-sheet citation right ${sheetRight}/${key.expected.length}`);
+const mapped = new Set(Object.values(map).flat());
+console.log('mapped', mapped.size, 'other', r.findings.filter(f => !mapped.has(f.id)).map(f => f.id).join(','));
+
+console.log('--- should-not-flag: findings filed on the file / naming it');
+for (const s of key.should_not_flag) console.log(s.file, '| filed:', r.findings.filter(f => f.file === s.file).map(f => f.id).join(',') || '-', '| named in issue:', r.findings.filter(f => f.file !== s.file && f.issue.includes(s.file)).map(f => f.id).join(',') || '-');
+console.log('Singapore/ap-southeast-1/backup mentions:', r.findings.filter(f => /Singapore|ap-southeast|backup\.yaml/i.test(f.issue + f.recommendation)).map(f => f.id).join(',') || 'none');
+
+console.log('--- paths');
+const isAbs = p => /^[A-Za-z]:/.test(p) || p.startsWith('/') || p.startsWith(String.fromCharCode(92)) || p.startsWith('../');
+const bad = r.findings.filter(f => f.file && (isAbs(f.file) || !existsSync(join(appDir, f.file)) || readFileSync(join(appDir, f.file), 'utf8').split('\n').length < (f.line || 1)));
+console.log('findings with file', r.findings.filter(f => f.file).length, '| absolute, missing or line past end:', bad.map(f => `${f.id} ${f.file}:${f.line}`).join(', ') || 'none');
+
+console.log('--- commencement: DPDP cited but no 13 May 2027 timing in issue');
+console.log(r.findings.filter(f => (f.framework === 'DPDP' || /DPDP|s\.\d|Rule \d/.test(f.cross_reference ?? '')) && !f.issue.includes('13 May 2027')).map(f => f.id).join(',') || 'none');
+
+console.log('--- values (names only, never the values)');
+// Every value is read from the app at run time, so this file holds none of them
+const encoded = readFileSync(join(appDir, 'deploy/secrets.yaml'), 'utf8').match(/DATABASE_URL: (\S+)/)[1];
+const decoded = Buffer.from(encoded, 'base64').toString();
+const values = {
+  secretEncoded: encoded,
+  secretDecoded: decoded,
+  secretUserinfo: decoded.split('@')[0].split('//')[1],
+  secretHost: decoded.split('@')[1].split(':')[0],
+  envExampleUrl: readFileSync(join(appDir, 'config/env.example'), 'utf8').match(/DATABASE_URL=(\S+)/)[1],
+};
+JSON.parse(readFileSync(join(appDir, 'tests/fixtures/members.json'), 'utf8')).forEach((m, i) => Object.assign(values, { [`fixture${i}.email`]: m.email, [`fixture${i}.phone`]: m.phone, [`fixture${i}.name`]: m.full_name }));
+const walk = d => readdirSync(d).flatMap(n => { const p = join(d, n); return statSync(p).isDirectory() ? walk(p) : [p]; });
+const files = walk(dir).filter(f => !f.endsWith('score-check.mjs') && !f.endsWith('score.md'));
+let leaks = 0;
+for (const f of files) { const t = readFileSync(f, 'utf8'); const hits = Object.entries(values).filter(([, v]) => t.includes(v)).map(([k]) => k); if (hits.length) { leaks++; console.log('HIT', f, hits.join(',')); } }
+console.log('files checked', files.length, '| files with a value', leaks);
+
+console.log('--- clauses outside the catalogs');
+const all = r.findings.map(f => `${f.clause} ${f.cross_reference ?? ''}`).join(' ');
+const onMap = n => [[1, 10], [12, 22], [24, 26], [28, 28], [30, 30], [32, 39], [44, 49], [77, 84]].some(([a, b]) => n >= a && n <= b);
+const arts = [...new Set([...all.matchAll(/Art\.\s*(\d+)/g)].map(m => +m[1]))].sort((a, b) => a - b);
+console.log('articles', arts.join(','), '| off map:', arts.filter(a => !onMap(a)).join(',') || 'none');
+console.log('other instruments:', [...new Set([...all.matchAll(/(CERT-In[^;,]*|CCPA[^;,]*|RBI[^;,]*|IT Act[^;,]*)/g)].map(m => m[1].trim()))].join(' | ') || 'none');
+const sar = JSON.parse(readFileSync(join(dir, 'findings.sarif'), 'utf8'));
+const uris = sar.runs[0].results.map(x => x.locations[0].physicalLocation.artifactLocation.uri);
+console.log('SARIF results', uris.length, '| absolute uris', uris.filter(isAbs).length);
+console.log('severity', JSON.stringify(r.findings.reduce((a, f) => (a[f.severity] = (a[f.severity] || 0) + 1, a), {})));

examples/kadamba-fit.run-1/score.md

@@ -0,0 +1,129 @@
+# Score: cold audit of kadamba-fit
+
+Assessment `2026-11-05-kadamba-cold-full`, 40 findings, run on master at `259d7e0` against the app from `b6d1c44`. Scored on 8 November 2026 against `examples/kadamba-fit.expected.json` and the run sheet on branch `claude/trusting-wilbur-d90e22`. The key and app are identical at `b6d1c44` and `3c9e897`. After this score, `a2083c0` corrected the key and run sheet; the app is unchanged. These figures use the earlier key.
+
+The scorer did not write the report, and opened the key only for scoring. `report.json` has the same sha256 before and after scoring: `751bc5a92f9fcb550cf9e8701f0a295d6f1774d6e85a97b96a70efb1cebabb34`.
+
+This folder holds the run's `report.json` and `summary.md`, copied from the gitignored assessment folder. The exports and company profile were not copied. `score-check.mjs` reproduces the mechanical checks:
+
+```bash
+node examples/kadamba-fit.run-1/score-check.mjs examples/kadamba-fit examples/kadamba-fit.expected.json
+```
+
+Run against the corrected key, it prints the same planted-issue and value checks; the should-not-flag notes have changed.
+
+## Result
+
+| Measure | Score |
+|---|---|
+| Planted issues found | 16 of 16 |
+| Partly found or missed | 0 |
+| Citation right, run-sheet rule | 16 of 16 |
+| Citation anchors, exact or more specific | 45 of 48 |
+| Citation anchors, at section, rule or article level | 47 of 48 |
+| Should-not-flag violations, by the key | 3 of 13 |
+| Other findings: real unplanted gaps / disputed / false positive | 14 / 3 / 1 |
+| DPDP findings that read as current violations | 0 |
+| Files containing a fixture or config value | 0 of 11 |
+| Findings with an absolute or missing path | 0 of 40 |
+| Clauses outside the catalogs | None |
+
+## Blindness
+
+The transcript never mentions the kadamba-fit key or run sheet. The only hits for `expected.json` are git status lines naming the acme-demo key.
+
+The run did not follow the run sheet's setup. It audited a `git archive` of the app folder from inside the main checkout, not from an exported copy of Vigil, and it used company id `kadamba-cold`. The key sat on disk in `.claude/worktrees/trusting-wilbur-d90e22/` throughout and was never opened. Before starting, the run confirmed that its working tree matched PR #2's head.
+
+## Planted issues
+
+| ID | Issue | Findings | Verdict | Clause check |
+|---|---|---|---|---|
+| P1 | Purge skips the three `member_ref` tables | F-0002 | Found | All 4 exact |
+| P2 | EU analytics to a US endpoint, no mechanism or DPA | F-0007, F-0012 | Found | All 3 exact |
+| P3 | Campaign sync ignores withdrawal | F-0011 | Found | Both exact |
+| P4 | Insurer sharing required at signup | F-0010 | Found | Both exact |
+| P5 | Dependants never marked minor, so they get ad offers | F-0008 | Found | s.9(3) exact; GDPR Art. 8(1) not cited |
+| P6 | Severity thresholds and the 72-hour clock | F-0020, F-0021, F-0022, F-0023, F-0026 | Found | All 5 exact |
+| P7 | Middleware logs bodies at DEBUG in production | F-0005 | Found | All 3 exact |
+| P8 | Export omits tables and recipients | F-0017 | Found, though it doesn't name `docs/vendors/register.md` | Both exact |
+| P9 | `seal()` stores plaintext without its key | F-0015 | Found | All 3 exact |
+| P10 | Base64 database URL in `deploy/secrets.yaml` | F-0006 | Found, though it doesn't name `config/production.yaml` | All 3 exact |
+| P11 | Unmasked production data in staging | F-0004, F-0016 | Found | All 5 exact |
+| P12 | Logs kept 30 days | F-0024 | Found | Both exact |
+| P13 | No Art. 9(2) condition for health data | F-0003 | Found | Both exact |
+| P14 | Partner export sends the whole row | F-0001 | Found | 3 exact; Art. 5(1)(b) only at article level |
+| P15 | Grievance clock of 120 days | F-0027 | Found | Both exact |
+| P16 | Dependant added with no parental consent | F-0009 | Found | 3 exact; Art. 8(1) only at article level, through Art. 8(2) |
+
+Every found item points at the file the key names.
+
+## Should-not-flag
+
+These are scored by aspect, because `config/support.yaml` is both P15 and a should-not-flag item.
+
+| File | Verdict |
+|---|---|
+| `services/otp.py` | Clean. F-0005 says the middleware defeats its masking, which is true |
+| `db/migrations/0005_billing.sql` | Clean |
+| `docs/vendors/mailwren-dpa.md` | Clean on the key's points. F-0018 says Mailwren has no per-member deletion path, which is weak because DPA clause 5 covers help with requests |
+| `docs/vendors/register.md` | Clean. F-0032 flags missing vendors, not the Bengaluru SCCs |
+| `docs/retention-schedule.md` | **Violated** by F-0037, which faults the nine-year invoice row for the Irish entity |
+| `db/migrations/0002_consent_events.sql` | Clean. F-0033 reports the opposite problem, deletion too early |
+| `deploy/backup.yaml` | Clean, never mentioned |
+| `config/env.example` | Clean. F-0006 says it needs no change |
+| `jobs/reminder_push.py` | Clean. F-0011 credits it |
+| `web/settings.html` | **Violated** by F-0038, which cites Art. 7(3) and s.6(4) on the switches |
+| `config/support.yaml` (rights clock) | Clean. F-0027 says 600 hours is fine |
+| `tests/fixtures/members.json` | Clean, no finding |
+| `web/privacy-notice.html` | **Violated** by F-0028, which reports rights missing from the notice. F-0025 repeats the objection point |
+
+Two of the three violations look like key defects:
+
+- **F-0028:** the notice omits restriction and objection, which Art. 13(2)(b) requires. The key's blanket "rights" protection is too broad.
+- **F-0037:** the schedule justifies nine years with Indian statutes only, and Art. 17(3)(b) and Art. 6(3) need EU or Member State law for the Irish controller.
+
+The third is a real false positive. F-0038's withdrawal claim assumes the switches render without the member's current state, but the file is a static fixture and the finding admits that state may be filled elsewhere. Its separate erasure-link point is a real gap: the notice promises erasure in Settings and the page doesn't offer it.
+
+## Other findings
+
+18 findings fall outside the planted issues. The 3 disputed ones are F-0028, F-0037 and F-0038, covered above.
+
+| Verdict | Findings |
+|---|---|
+| Real unplanted gap, checked against the code | F-0013, F-0014, F-0018, F-0019, F-0025, F-0029, F-0030, F-0031, F-0032, F-0033, F-0034, F-0035, F-0036, F-0039 |
+| False positive | F-0040: an open Art. 22 question with no evidence of a solely automated decision |
+
+Three of the real gaps are code defects the key's author missed:
+
+- **F-0019:** `guardian_id` has no cascade, so erasing a parent fails, and the purge loop has no error handling, so one failure ends the nightly run.
+- **F-0033:** consent proof is deleted by `created_at`, so it can go about 30 days after closure instead of three years later. The key's should-not-flag note claims the purge waits for the end of the period, which is wrong for older consents.
+- **F-0036:** a corrected email leaves the old contact in the CRM, where the erasure call cannot find it.
+
+## The audit session's guess points
+
+| # | Point | Verdict |
+|---|---|---|
+| 1 | One `legal_name` and address for two entities | Real schema gap |
+| 2 | `dpo` holds only a mailbox | Real schema gap, minor |
+| 3 | Whether README claims count as posture evidence | Judgement call |
+| 4 | UK and IRDAI left unknown | Judgement call |
+| 5 | Meaning of "realistic-format" in `/pii-scan` | Wording gap. The outcome was right: zero-filled phones are valid format but plainly synthetic |
+| 6 | Production credential marked synthetic, rated by risk | Judgement call, and the right one |
+| 7 | A DPIA reported but not supplied | Real skill gap: the absence rule doesn't cover it |
+| 8 | Sector check ignores an insurer partner | Real skill gap, low impact |
+| 9 | Breach readiness has no rule for a step that is present but wrong | Real skill gap. It drove P6, and the run handled it anyway |
+| 10 | "Always high" for dark patterns | Wording gap: it should say "at least high" |
+| 11 | One `lawful_basis` value; `transfer_mechanism` has no DPDP value | Real schema gap for the second; the first is handled by `none_identified` |
+| 12 | One `file` and `line` per finding | Real format gap. SARIF could carry related locations |
+| 13 | Own coverage count; one posture dimension per finding | Judgement call, by design |
+| 14 | Sub-skills applied inline | Process note, not a gap |
+
+## Comparison with acme-demo
+
+Acme-demo, self-scored on a fixture whose comments named its bugs, found 9 of 10 with 23 of 27 exact anchors. Kadamba-fit, scored by a separate session on a fixture with no telltale comments, found 16 of 16 with 45 of 48 exact anchors. It also violated 3 of 13 should-not-flag items, two of them arguably defects in the key.
+
+## Limits
+
+- A Claude session wrote the fixture, another ran the audit and a third scored it, so shared blind spots could inflate recall.
+- The skill fixes from the acme-demo score were in the skills this run used. Fixing skills from this score would end kadamba-fit's blindness, and another blind measure would need a new fixture.
+- The run sheet asks for the score and audit output to be committed under `examples/kadamba-fit.run-<n>/`. That wasn't done, because this scoring brief said not to commit.