Skip to content

Fix export validity, re-audit counts and guard bypasses - #1

Merged
prathikcannotcode merged 2 commits into
masterfrom
fix/new-user-audit
Nov 4, 2026
Merged

prathikcannotcode merged 2 commits into
masterfrom
fix/new-user-audit

Conversation

@prathikcannotcode

@prathikcannotcode prathikcannotcode commented Nov 4, 2026 •

Copy link
Copy Markdown
Owner

Fixes from auditing Vigil as a new user would: fresh clones, the official export schemas, and hostile input to the hooks. The full list is under Unreleased in CHANGELOG.md.

What was wrong

  • All 9 sample findings failed NIST's OSCAL 1.1.2 schema, and CI only checked the exit code.
  • SARIF results with no file had no location, and rules lacked text GitHub lists as required, which put quick-start step 6 at risk.
  • A second audit doubled everything: 18 findings instead of 9, and open criticals went from 1 to 2.
  • From a subdirectory the hook crashed with exit 1, which does not block, so the write would go through.
  • The agent could rewrite its own guard, because .claude/** was writable.
  • The README called hand-written sample files real output.

What changes for users

  • Claude can no longer edit .claude/scripts, .claude/schemas or .claude/settings.json. Those are hand-edit only, including for your own Claude sessions on this repo.
  • SARIF leaves out findings with no file and closed findings. The exporter names them, and they stay in report.json, OSCAL and OCSF.
  • Real company data is gitignored by default. Only the synthetic demo is tracked.
  • Finding file paths must be relative. Absolute paths and ../ now fail validation.
  • Exporters and kpis --company exit 1 for a company with no report, instead of reporting 0 findings.

Known limits

Both are documented in SECURITY.md, and both block writes rather than let them through.

  • A session that moved into the repo from elsewhere and then changed into a subdirectory can't find the guard until it returns to the repo root. The hook command looks under ${CLAUDE_PROJECT_DIR}, then relative to the working directory.
  • On Windows without Git Bash, Claude Code runs hooks in PowerShell, which can't parse the command. The write-guard then blocks every write, and post-write validation does not run.

Testing

  • 23 of 23 tests pass on clean Linux (Node 22). On Windows (Node 24) 22 pass and the bash hook test is skipped
  • npm run validate -- --strict passes
  • OSCAL validates against NIST's 1.1.2 schema; SARIF validates against the published 2.1.0 schema
  • All 20 skills parse under a strict YAML parser and show as auto-invocable on a Windows default Git clone
  • Hooks ran in a live Claude Code desktop session (2.1.309), in a session that had moved into the repo. Blocked live: a path outside the layout, an edit to .claude/settings.json, an append-only log rewrite, an Aadhaar number in a report, and a key-shaped string. Allowed live: edits to README.md, SECURITY.md and CHANGELOG.md. The post-write hook reported a finding with no clause and an absolute path
  • Before the fallback, onFailure: block stopped a write live when the guard could not start
  • Live SARIF upload to GitHub code scanning
  • A real /audit run on examples/acme-demo, scored against examples/acme-demo.expected.json

Not in this PR

  • aranya/ stays in layout.json and .gitignore, because the local Aranya checkout sits inside this folder.
  • There is no penalty script and no fixed control list for coverage. /penalty now reports totals per finding and per provision.
  • Tags, releases and GitHub topics.

🤖 Generated with Claude Code

prathikcannotcode and others added 2 commits November 5, 2026 00:36
Found by auditing the repo as a new user would.

Exports:
- OSCAL target-id is a valid token; every finding failed NIST's 1.1.2 schema.
- SARIF rules carry fullDescription and help. Findings with no file and
  closed findings are left out and named by the exporter.
- OCSF maps remediated to Resolved (4), not Suppressed (3).
- Exports and KPIs use each company's latest report, so a re-audit no longer
  doubles findings. Exporters exit 1 for a company with no report.

Guards:
- Hooks run from ${CLAUDE_PROJECT_DIR} and find the workspace from the event's
  cwd. A cd into a subdirectory used to crash the hook with exit 1, which does
  not block, so writes went through.
- The write-guard fails closed, with onFailure: block for a hook that can't start.
- Hooks, schemas and settings are out of the writable layout.
- New Aadhaar, PAN and +91 mobile checks in company and assessment data.
- Secret patterns catch OpenSSH keys, github_pat_ and sk-ant- keys and
  JSON-style passwords, and no longer block "password: hardcoded" in prose.
- validate rejects skill descriptions that are not valid YAML; all 20 are quoted.

Usability:
- examples/acme-demo: a synthetic app with planted issues and an answer list.
- run-headless --target, a 2-hour timeout, transcripts saved on failure.
- kpis --company, onboarding fields the schema used to reject, report provenance
  fields, relative-only finding paths, and /penalty totals per finding and
  per provision.
- Real company data is gitignored; private: true; CITATION.cff.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
${CLAUDE_PROJECT_DIR} is the folder a session started in. A session that moved
into the repo could not find the guard, so onFailure: block stopped every write.
The hook commands now fall back to the relative path. Approved by the maintainer,
since these are guard files agents don't normally edit.

Added a test that runs the real hook commands, skipped on Windows because bash
may resolve to WSL. Documented the two cases that still fail closed in SECURITY.md.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@prathikcannotcode
prathikcannotcode merged commit bfe531a into master Nov 4, 2026
1 check passed
@prathikcannotcode
prathikcannotcode deleted the fix/new-user-audit branch November 9, 2026 17:42
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant