docs(audit): systemic audit of Hoplight by Claude Opus 5 - #8
Closed
Livingooal wants to merge 1 commit into
Closed
Conversation
Adds audit-plans/opus-5/: audit record, plan index, and five executor-ready plans for the five MEDIUM findings. Documentation only, no source changes. 5 MEDIUM, 3 LOW. Three of the five MEDIUMs came from adversarial-review leads and required executing code to confirm. Two revision-1 findings were withdrawn after review disproved them; both are retained with the evidence that killed them. Security detail withheld per SECURITY.md and delivered privately.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What this is
A systemic audit of Hoplight at
80451e6, run with thegrumpy-systemic-auditskill inaudit standard directmode. Documentation only: it addsaudit-plans/opus-5/and changes nosource.
Model disclosure (per CONTRIBUTING.md): Claude Opus 5, conductor-only investigation with a
16-agent adversarial review wave. This was a model benchmark; a Fable 5 session audited the same
commit in parallel in a separate worktree and branch, with no shared notes or artifacts.
Headline
Hoplight is in good shape.
verify:ciis green at HEAD (2250 pass, 0 fail, 25560 expect() calls),the defensive engineering is strong, and the Round-Trip Law the project leads with is enforced
by CI, which I confirmed by deliberately breaking a shipping codec (8 test failures).
Result: 5 MEDIUM, 3 LOW. Plans for the five MEDIUMs; the three LOWs are recorded with rationale
and no plan.
Findings
src/uiare never lintedpackagedSwitchorchestration has no direct testThe three worth reading first
F-06 (
src/formats/pygmalion/) is the most user-visible.coverage.ts:17declaresmedia.portraitcarried;index.ts:150-168only emits a portrait when Pygmalion's own escrowexists and never reads
body.media.portrait. Live-proven on four samples across three sourceformats: portrait lost, and not listed as dropped.
media.assetsis reported dropped in everyone of those runs, which proves the reporting machinery works and that
media.portraitalone isbeing suppressed by the declaration. A silent loss behind a clean report is worse than a loud loss.
F-07 (
eslint.config.mjs:43,package.json:38): both the configfileskey and the npm globscope to
.tsx, so eslint has no configuration matching a.tsfile. Three of the excluded filesdefine React hooks. I planted a conditional
useStateinuse-variants.ts;lint:uiand the fullverify:ciboth passed clean, while the identical code in a.tsxfile errors onreact-hooks/rules-of-hooks.F-08 (
.github/workflows/): no Go step anywhere. All ninesidecar/main_test.gotests guard theremote-access authorization boundary (identity spoofing, non-owner denial, kicked devices,
constant-time secret compare) and none of them run.
go builddoes not run either, so the sidecaris not compile-checked.
This audit corrected itself twice, and that is part of the record
Revision 1 shipped two findings that adversarial review destroyed and that I then disproved myself.
Both are retained in
AUDIT.mdas withdrawn entries with the evidence that killed them._template, whosedetect()returns0against a0.5threshold, so the adapter could never be selected and neverround-tripped anything. The correct experiment, breaking the escrow re-merge in
src/formats/sillytavern/index.ts:128-133, gives 8 failures. I also truncateddocs/ROADMAP.mdin a way that inverted its meaning, and dismissed
receipt.test.tsas cosmetic when it is aregistry-walking gate that was working as designed.
resolveAssetandisAllowedDownloadHostlive inrelease-download.tsand are tested.Severities on three surviving findings were lowered after review (F-02 and F-03 from HIGH to MEDIUM,
F-04 from MEDIUM to LOW), and F-03's security framing was withdrawn: no untrusted actor can reach
that boundary.
The three strongest findings (F-06, F-07, F-08) came from reviewer leads rather than my own pass,
and each required executing code to confirm. That is the honest headline about method.
Security
Per
SECURITY.md, which asks for private disclosure first, security detail is not in this PR.No exploitable vulnerability was found. Notes were delivered to the maintainer privately, covering:
the LAN listener versus the "loopback only" wording (F-04, published here because the fix is a docs
edit and there is no exploit attached), ADR-009's packaged-build isolation measurements being
recorded as not run, and a handful of security-relevant modules without direct tests.
One reviewer-raised lead, a possible archive entry-count denial of service, could not be
reproduced (flat 2-5ms, no scaling, against a reported 967ms). It is recorded in
AUDIT.mdas anopen lead and explicitly not a finding. If pursued it should be handled privately.
Scope, stated plainly
Roughly 60-70% of the repository was never examined. Not audited in depth:
src/ui(489 of 829files), per-adapter mapping fidelity (66 files), sandbox adversarial testing,
sidecar/internals,performance, accessibility, and any live browser journey.
Running
direct(no shards) on 102k lines concentrated depth on authority boundaries and leftbreadth untouched. None of my five original findings required executing anything; all were legible
from structure alone, which is why the first systemic theme I proposed was partly an artifact of
method. Read this as a scoping pass over about a third of the code, not a verdict on it. A sharded
pass with a mandate to execute would find more, and different, defects.
Verification
80451e6:bun run verify:ciEXIT 0, 2250 pass, 0 fail.bun run verify:ciEXIT 0.audit-plans/sits outside every gate's walk roots, andthe artifacts hand-honor the no-em-dash and no-emoji conventions anyway.
contains only
audit-plans/.For the merger
This is docs-only and should not cut a release. Put
[skip release]in the squash or merge commitmessage:
auto-release.ymlreads only the head commit of the push toMainstage, so a marker ina branch commit is ignored.
Nothing here is implemented. The plans are artifacts, not changes.