Skip to content

Decode covdata in-process for indexing speed, and fix nine review-found bugs - #1

Merged
emoss08 merged 9 commits into
mainfrom
claude/codebase-indexing-perf-um9eja
Aug 8, 2026
Merged

emoss08 merged 9 commits into
mainfrom
claude/codebase-indexing-perf-um9eja

Conversation

@emoss08

@emoss08 emoss08 commented Aug 8, 2026

Copy link
Copy Markdown
Owner

Indexing performance on large workspaces

assay index shelled out to go tool covdata textfmt once per test: each subprocess re-read the package's whole-closure coverage meta and serialised a text profile dominated by zero-count lines that ParseProfile immediately discarded. That cost scales with tests × dependency-closure size — the product that buries large workspaces. BENCHMARKS.md had already named this follow-up and deferred it because the binary formats are internal to the Go toolchain.

This PR builds the in-process decoder for the covmeta/covcounters binary formats (version-stamped, unchanged since Go 1.20), with the maintenance trap fenced rather than walked into blind:

  • every read is bounds-checked; any structural surprise — future version, foreign magic, meta-hash mismatch, truncation — makes the whole package fall back to the covdata subprocess path, which stays first-class. An incompatible future toolchain costs time, never correctness.
  • a parity test drives the real toolchain end to end (real instrumented build, real harness run) and requires the decoder's blocks to equal the textfmt path's exactly, init window and skipped-test windows included.
  • the existing harness-vs-legacy agreement tests pass unchanged.

Measured on a synthetic monorepo-shaped workspace (30 test packages × 20 tests, each closing over 61 instrumented packages, warm build caches, 4 cores): 13.1s → 8.5s wall, user+sys CPU 41.3s → 25.7s. At a 201-package closure: 8.7s → 5.7s. The per-test conversion tax is gone entirely; what remains is compile, link and test execution. Numbers are recorded in BENCHMARKS.md.

Bug fixes

Each fix carries a regression test (the cgo one is covered by the corpus tests); the leak test was verified red against the old code.

Silent under-selection — the failure mode this tool must never have:

  • selection: editing an existing test's body selected nothing. Test files classify as narrowable, but coverage never instruments _test.go, so attribution found nothing and the package dropped — the edited test never ran. Test-file edits now run the owning package in full (comment-only edits still drop safely).
  • index: fingerprints hashed nothing root-dependent while records intern absolute paths, so two checkouts sharing a cache directory served each other records whose every path missed — and narrowed to nothing. The workspace root is now part of every fingerprint.

Wrong verdicts:

  • watch: narrowing to TestMain ran zero tests and printed a green cycle; it now declines refinement like Benchmark/Example.
  • watch: a crashed run (panic, -test.timeout dump, os.Exit) credited passes to requested tests it never reached, fabricating the exact conflicting evidence the flake journal exists to detect. Passes are credited only when the failing binary printed its FAIL trailer.
  • risk/vcs: git log --numstat prints toplevel-relative paths while risk keys lookups root-relative, so --root <subdir> zeroed all churn and printed a green report produced by a key mismatch. Churn is now keyed relative to the analysis root.
  • flake: journal compaction kept the oldest observations on timestamp ties (batched writes share one clock reading), the opposite of the documented contract.

Resource and attribution bugs:

  • mutate: the deferred schemata cleanup iterated the live slice the harness path nils, leaking one workdir — overlays plus compiled test binaries — per batch of every default-mode run.
  • mutate: syntaxPath fell back to indexing GoFiles positionally for Syntax, two lists that disagree for cgo packages, splicing mutations into the wrong file. The path now comes from the parsed tree's own position info; cgo-generated cache files yield no mutants.
  • watch: a staging .tmp binary left by a killed build could be promoted as fresh when a later build of a now-testless package produced no output. The staging path is cleared before every build.

Smaller:

  • selection/index: a go.mod under testdata (fixture, never read by the build) forced a whole-workspace run and, hashed as unowned, invalidated the entire index on any fixture edit. It now attributes to its owning package.
  • cli: color was decided from stdout's TTY-ness while run/verify/progress write to stderr — assay run 2>log wrote escapes into the log. Stderr-bound output now consults stderr.
  • cli: --jobs help claimed a GOMAXPROCS default; the real default has been half of GOMAXPROCS since the inner build-parallelism split.

Testing

  • go vet ./... clean, full go test ./... -count=1 green.
  • New parity + rejection tests for the decoder; regression tests for each behavioural fix.
  • End-to-end smoke: cold assay index on this repo (17 packages / 349 tests), then a warm rerun reusing all 17 records in 5ms.

🤖 Generated with Claude Code

https://claude.ai/code/session_01211tSDyqM7KnfziwEjPzDF


Generated by Claude Code

claude added 9 commits August 8, 2026 04:12
…cess as fallback

Window assembly shelled out to 'go tool covdata textfmt' once per test, each
invocation re-reading the package's whole-closure meta blob and serialising a
text profile dominated by zero-count lines that ParseProfile then discarded.
On a large workspace that per-test round-trip is most of the indexing wall
clock.

The meta and counter files are now decoded in this process: the meta is read
once per package and each window costs one small counter-file read touching
only the units the test actually executed. Both formats are version-stamped
and unchanged since Go 1.20; every read is bounds-checked, and any structural
surprise — future version, foreign magic, meta-hash mismatch, truncation —
falls back to the covdata subprocess path wholesale, so an incompatible future
toolchain costs time, never correctness.

A parity test drives the real toolchain end to end and requires the decoder's
blocks to equal the textfmt path's exactly, init window and skipped-test
windows included.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01211tSDyqM7KnfziwEjPzDF
Two silent under-selection holes, both found by review:

An edit inside an existing test's body was classified narrowable, but index
records never contain _test.go paths — coverage instruments production code
only — so attribution found nothing, the package was dropped, and the edited
test never ran. Test files now take the blocking path: the owning package runs
in full. Comment-only test edits still drop safely.

A go.mod anywhere in the tree, matched by base name alone, forced a
whole-workspace run — including fixtures under testdata that the build never
reads. Testdata module files now attribute to their owning package like any
other fixture.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01211tSDyqM7KnfziwEjPzDF
… module files

Records intern absolute paths, but the fingerprint hashed nothing
root-dependent: a second checkout sharing the cache directory got
byte-identical fingerprints, hit records whose every path missed, and silently
narrowed to nothing. The root is now part of every fingerprint.

A go.mod under a package's testdata was classified unowned and hashed into
every package's fingerprint, so editing one fixture invalidated the whole
index. It now attributes to its owning package.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01211tSDyqM7KnfziwEjPzDF
Narrowing to TestMain ran zero tests and printed a green cycle: it passes the
Test-prefix check but -test.run never selects it. It now declines refinement
like Benchmark and Example do.

A crashed run — panic, -test.timeout dump, os.Exit — credited passes to every
requested test the crash prevented from running, fabricating the exact
conflicting evidence the flake journal exists to detect. Passes are now
credited only when the failing binary printed its FAIL trailer, the marker
that the whole list executed.

A staging binary left by a killed build could be promoted as fresh when a
later build of a now-testless package produced no output. The staging path is
cleared before every build.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01211tSDyqM7KnfziwEjPzDF
The deferred batch cleanup iterated the live variable that the harness path
nils, so every default-mode run leaked one temp directory per batch — the
overlays plus a compiled test binary per covering test package. The cleanup
now iterates its own copy; the strong-suite test doubles as the leak check.

syntaxPath indexed GoFiles as a positional fallback for Syntax, two lists that
disagree in length and order for any cgo package, splicing mutations into
whatever file sat at that index. The path now comes from the parsed tree's own
position information, and cgo-generated cache files yield no mutants.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01211tSDyqM7KnfziwEjPzDF
Observations batched in one write share a clock reading, and the stable sort
kept file order among ties — so a group trimmed at the cap kept its oldest
evidence and discarded its newest, the opposite of the documented contract.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01211tSDyqM7KnfziwEjPzDF
git prints numstat paths relative to the repository toplevel no matter where
it runs, while risk keys its lookups relative to the analysis root. With
--root pointing anywhere below the toplevel every lookup missed, every
function counted as stable, and the report printed a green verdict produced by
a key mismatch rather than by coverage.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01211tSDyqM7KnfziwEjPzDF
Color was decided from stdout's TTY-ness while run, verify and the progress
line write to stderr: 'assay run 2>log' wrote escapes into the log, and
'assay run >results' stripped color from a summary still heading to a
terminal. Stderr-bound output now consults stderr.

The --jobs help claimed a GOMAXPROCS default; the collector has defaulted to
half of GOMAXPROCS since the inner build parallelism split was introduced.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01211tSDyqM7KnfziwEjPzDF
@emoss08
emoss08 merged commit b8cfb9c into main Aug 8, 2026
1 check passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants