Skip to content

docs(audit): systemic audit of Hoplight by Claude Opus 5 - #8

Closed
Livingooal wants to merge 1 commit into
Coneja-Chibi:Mainstagefrom
Livingooal:audit/grumpy-opus-5
Closed

docs(audit): systemic audit of Hoplight by Claude Opus 5#8
Livingooal wants to merge 1 commit into
Coneja-Chibi:Mainstagefrom
Livingooal:audit/grumpy-opus-5

Conversation

@Livingooal

Copy link
Copy Markdown
Contributor

What this is

A systemic audit of Hoplight at 80451e6, run with the grumpy-systemic-audit skill in
audit standard direct mode. Documentation only: it adds audit-plans/opus-5/ and changes no
source.

Model disclosure (per CONTRIBUTING.md): Claude Opus 5, conductor-only investigation with a
16-agent adversarial review wave. This was a model benchmark; a Fable 5 session audited the same
commit in parallel in a separate worktree and branch, with no shared notes or artifacts.

Headline

Hoplight is in good shape. verify:ci is green at HEAD (2250 pass, 0 fail, 25560 expect() calls),
the defensive engineering is strong, and the Round-Trip Law the project leads with is enforced
by CI, which I confirmed by deliberately breaking a shipping codec (8 test failures).

Result: 5 MEDIUM, 3 LOW. Plans for the five MEDIUMs; the three LOWs are recorded with rationale
and no plan.

Findings

ID Finding Severity Plan
F-06 Pygmalion export drops portraits and the honesty report conceals it MEDIUM 001
F-07 188 TypeScript files under src/ui are never linted MEDIUM 002
F-08 The sidecar's nine authorization tests never run in CI MEDIUM 003
F-02 CLI cannot convert presets or regex sets that the studio converts MEDIUM 004
F-03 Five character subtrees cross the storage boundary unvalidated MEDIUM 005
F-04 "Loopback only" is inaccurate once LAN access is enabled LOW none
F-01 No gate compels a new adapter to ship round-trip evidence LOW none
F-05 packagedSwitch orchestration has no direct test LOW none

The three worth reading first

F-06 (src/formats/pygmalion/) is the most user-visible. coverage.ts:17 declares
media.portrait carried; index.ts:150-168 only emits a portrait when Pygmalion's own escrow
exists and never reads body.media.portrait. Live-proven on four samples across three source
formats: portrait lost, and not listed as dropped. media.assets is reported dropped in every
one of those runs, which proves the reporting machinery works and that media.portrait alone is
being suppressed by the declaration. A silent loss behind a clean report is worse than a loud loss.

F-07 (eslint.config.mjs:43, package.json:38): both the config files key and the npm glob
scope to .tsx, so eslint has no configuration matching a .ts file. Three of the excluded files
define React hooks. I planted a conditional useState in use-variants.ts; lint:ui and the full
verify:ci both passed clean, while the identical code in a .tsx file errors on
react-hooks/rules-of-hooks.

F-08 (.github/workflows/): no Go step anywhere. All nine sidecar/main_test.go tests guard the
remote-access authorization boundary (identity spoofing, non-owner denial, kicked devices,
constant-time secret compare) and none of them run. go build does not run either, so the sidecar
is not compile-checked.

This audit corrected itself twice, and that is part of the record

Revision 1 shipped two findings that adversarial review destroyed and that I then disproved myself.
Both are retained in AUDIT.md as withdrawn entries with the evidence that killed them.

  • "The Round-Trip Law is not CI-enforced" was false. My probe copied _template, whose
    detect() returns 0 against a 0.5 threshold, so the adapter could never be selected and never
    round-tripped anything. The correct experiment, breaking the escrow re-merge in
    src/formats/sillytavern/index.ts:128-133, gives 8 failures. I also truncated docs/ROADMAP.md
    in a way that inverted its meaning, and dismissed receipt.test.ts as cosmetic when it is a
    registry-walking gate that was working as designed.
  • "The update path has zero tests" was materially false. resolveAsset and
    isAllowedDownloadHost live in release-download.ts and are tested.

Severities on three surviving findings were lowered after review (F-02 and F-03 from HIGH to MEDIUM,
F-04 from MEDIUM to LOW), and F-03's security framing was withdrawn: no untrusted actor can reach
that boundary.

The three strongest findings (F-06, F-07, F-08) came from reviewer leads rather than my own pass,
and each required executing code to confirm. That is the honest headline about method.

Security

Per SECURITY.md, which asks for private disclosure first, security detail is not in this PR.
No exploitable vulnerability was found. Notes were delivered to the maintainer privately, covering:
the LAN listener versus the "loopback only" wording (F-04, published here because the fix is a docs
edit and there is no exploit attached), ADR-009's packaged-build isolation measurements being
recorded as not run, and a handful of security-relevant modules without direct tests.

One reviewer-raised lead, a possible archive entry-count denial of service, could not be
reproduced
(flat 2-5ms, no scaling, against a reported 967ms). It is recorded in AUDIT.md as an
open lead and explicitly not a finding. If pursued it should be handled privately.

Scope, stated plainly

Roughly 60-70% of the repository was never examined. Not audited in depth: src/ui (489 of 829
files), per-adapter mapping fidelity (66 files), sandbox adversarial testing, sidecar/ internals,
performance, accessibility, and any live browser journey.

Running direct (no shards) on 102k lines concentrated depth on authority boundaries and left
breadth untouched. None of my five original findings required executing anything; all were legible
from structure alone, which is why the first systemic theme I proposed was partly an artifact of
method. Read this as a scoping pass over about a third of the code, not a verdict on it. A sharded
pass with a mandate to execute would find more, and different, defects.

Verification

  • Baseline at 80451e6: bun run verify:ci EXIT 0, 2250 pass, 0 fail.
  • This branch: bun run verify:ci EXIT 0. audit-plans/ sits outside every gate's walk roots, and
    the artifacts hand-honor the no-em-dash and no-emoji conventions anyway.
  • Five throwaway probes were run to prove or disprove findings. Every one was reverted; the worktree
    contains only audit-plans/.

For the merger

This is docs-only and should not cut a release. Put [skip release] in the squash or merge commit
message
: auto-release.yml reads only the head commit of the push to Mainstage, so a marker in
a branch commit is ignored.

Nothing here is implemented. The plans are artifacts, not changes.

Adds audit-plans/opus-5/: audit record, plan index, and five executor-ready
plans for the five MEDIUM findings. Documentation only, no source changes.

5 MEDIUM, 3 LOW. Three of the five MEDIUMs came from adversarial-review leads
and required executing code to confirm. Two revision-1 findings were withdrawn
after review disproved them; both are retained with the evidence that killed
them.

Security detail withheld per SECURITY.md and delivered privately.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants