Repository navigation
perf(memory): help optimizer avoid duplicative bounds checks & address masking on load/store - #66
Conversation
The safe multi-byte helpers currently pass mem[addr:] to encoding/binary.
When the memory length is not known at compile time, Go must validate the
suffix slice and then check that it contains enough bytes for the value.
On amd64 this also generates pointer-masking instructions for the suffix.
These instructions add up in translated code that frequently accesses
linear memory, even after the helpers and encoding/binary calls inline.
Pass an exact 2-, 4-, or 8-byte window instead. Its constant length lets
the compiler eliminate encoding/binary's length check and unnecessary
slice-pointer work while retaining the window's runtime bounds checks.
This improves the default safe output without requiring -unsafe, adding
dependencies, or changing little-endian encoding.
Both forms still enforce the same memory boundary. The improvement is
that the compiler emits fewer checks and less slice bookkeeping, rather
than merely moving the same check out of encoding/binary. For a uint32
address and a four-byte load, the old form behaves like this Go pseudocode:
a := uint64(addr)
n := uint64(len(mem))
if a > n {
panic("slice out of bounds") // constructing mem[addr:]
}
remaining := n - a
if remaining < 4 {
panic("index out of bounds") // Uint32's check of b[3]
}
return readLittleEndian32(mem, a)
The new form gives the compiler this simpler condition:
a := uint64(addr)
end := a + 4
if end > uint64(len(mem)) {
panic("slice out of bounds") // constructing the four-byte window
}
return readLittleEndian32(mem, a)
Here readLittleEndian32 represents the word read after validation, not
another checked helper call. Widening a uint32 address before adding four
lets the compiler prove that end >= a, eliminating the slice's lower-bound
check. Once the window exists, its length is exactly four, so Uint32's
check is redundant. The capacity-clamping view adds no runtime check in
this example because len(mem) <= cap(mem) is already a slice invariant.
Arbitrary uint64 addresses still need the overflow handling described below.
Go also emits pointer-masking instructions for a suffix that could have
zero capacity, to keep an empty slice from pointing at the next object.
A successfully constructed fixed-width window has positive capacity,
allowing that masking to be removed. Source references:
https://go.dev/src/encoding/binary/binary.go#L90
https://go.dev/src/cmd/compile/internal/ssagen/ssa.go#L5872
The window must stay within len(mem), the size of valid Wasm memory.
Go checks the upper bound of mem[addr:addr+width] against cap(mem), which
can be larger than len(mem) after memory.grow. For example:
mem := make([]byte, 8, 16)
mem[6:10] // succeeds in Go, but crosses the Wasm memory boundary
Previously, encoding/binary checked the length of mem[addr:], so accesses
past len(mem) already trapped. To preserve that boundary with fixed-width
windows, first take mem[:len(mem):len(mem)]. The third index sets the
view's capacity to len(mem), so the subsequent window cannot reach spare
backing storage. Using mem[:len(mem)] alone would retain the original
capacity. This creates a restricted view without copying or allocating:
mem[:len(mem):len(mem)][6:10] // panics because the view's capacity is 8
Convert addr to uint64 before adding the access width so uint32 addresses
cannot wrap. If a uint64 end overflows, its upper bound is below the
starting address and slicing still panics. A rejected store traps while
evaluating the window, before any bytes are written.
Add differential tests against the original accessors for both address
types and all three widths. Cover every length within several backing
capacities, unaligned accesses, boundary crossings, extreme addresses,
unchanged backing storage after rejected stores, and accesses after growth.
Regenerate the checked-in translated sample and regression fixtures.
Self-contained performance example
This microbenchmark does not exercise wasm2go; it compares only the code
being changed, from suffix slices to fixed-width memory windows.
Each case uses 64 KiB of memory, reads and increments aligned uint32 words,
writes them back, and consumes a checksum. The loop functions are noinline
so their memory length is unknown to the compiler, while the accessors
inline normally. Setup is outside the timer.
To reproduce, temporarily save the following as
helpers/memory_bench_test.go (the benchmark is not part of this patch):
package helpers
import (
"encoding/binary"
"testing"
)
var memoryBenchmarkSum uint32
// BenchmarkMemory32 compares the original suffix slice with the fixed-width
// helpers on the same 64 KiB memory. Addresses vary over aligned words; each
// iteration reads a word, increments it, writes it back, and consumes the result.
// Both forms inline, as they do in translated code. The loop functions stay
// out of line so the compiler cannot infer the memory length from the allocation.
// Setup is outside the timer.
// Run with: go test ./helpers -run '^$' -bench '^BenchmarkMemory32$' -benchmem -count=5
func BenchmarkMemory32(b *testing.B) {
b.Run("suffix", func(b *testing.B) {
mem := make([]byte, 1<<16)
b.SetBytes(8)
b.ReportAllocs()
b.ResetTimer()
memoryBenchmarkSum = memorySuffix32(mem, b.N)
})
b.Run("window", func(b *testing.B) {
mem := make([]byte, 1<<16)
b.SetBytes(8)
b.ReportAllocs()
b.ResetTimer()
memoryBenchmarkSum = memoryWindow32(mem, b.N)
})
}
//go:noinline
func memorySuffix32(mem []byte, n int) uint32 {
var sum uint32
for i := 0; i < n; i++ {
addr := uint32(i&((1<<14)-1)) * 4
val := binary.LittleEndian.Uint32(mem[addr:]) + 1
binary.LittleEndian.PutUint32(mem[addr:], val)
sum += val
}
return sum
}
//go:noinline
func memoryWindow32(mem []byte, n int) uint32 {
var sum uint32
for i := 0; i < n; i++ {
addr := uint32(i&((1<<14)-1)) * 4
val := load32(mem, addr) + 1
store32(mem, addr, val)
sum += val
}
return sum
}
Then run:
go test ./helpers -run '^$' -bench '^BenchmarkMemory32$' -benchmem -count=5
Go 1.27.2, linux/amd64, AMD Ryzen 3 PRO 5350GE; medians of five samples:
suffix: 0.9571 ns/op, 0 B/op, 0 allocs/op
window: 0.7132 ns/op, 0 B/op, 0 allocs/op
That is 25.5% less time per read/write pair, or about 1.34x throughput.
Selected instructions from the two loops, obtained with Go 1.27.2's
objdump on the benchmark binary; branch addresses are replaced with
symbolic labels and unrelated loop instructions are omitted. AX is the
memory pointer and BX its length. In the suffix loop, CX is capacity and
SI starts as the address:
CMPQ BX, SI // compare len(mem) with addr
JB panicSlice // reject addr > len(mem)
MOVQ BX, R9
SUBQ SI, BX // remaining length = len(mem) - addr
MOVL SI, R10
SUBQ CX, SI // addr - cap(mem)
SARQ $0x3f, SI // build mask for nonzero remaining capacity
ANDQ SI, R10 // mask the slice pointer offset
CMPQ BX, $0x3
JA access // remaining length >= 4
JMP panicLoad
access:
MOVL 0(AX)(R10*1), R11 // read the word
In the window loop, DX is the address:
LEAQ 0x4(DX), R8 // end = addr + 4
CMPQ BX, R8
JAE access // len(mem) >= end
JMP panicWindow
access:
MOVL 0(AX)(DX*1), R8 // read the word
The suffix loop has two bounds-check branches and pointer masking; the
window loop has one bounds-check branch and no pointer masking. Both
loops also store the incremented word, with no additional store bounds
check after the successful load. The compiler could theoretically
optimize both forms identically; this change presents an expression the
tested compiler optimizes better. These are microbenchmark results for
this compiler and CPU, not an end-to-end or cross-architecture performance
claim.
Validation: go generate; go test ./... (1,186 passing packages); gofmt and
git diff --check.
|
I'm not doubting this is a win, but this is verbose as hell. Also, skimming, why: // This:
mem[:len(mem):len(mem)][6:10]
// But not this:
mem[6:10:len(mem)]Micro benchmarks can be kinda useless because the problem wasm2go faces is that the optimizer “breaks down” (as it should, to avoid super linear behavior, it's slow enough as it is) on huge functions. Also, to ensure a single check is enough the optimizer needs to prove to itself that That said, I'll look into this, and if I find that it makes a difference on my standard macro benchmark, I'll definitely do something like it and will credit you. |
|
The single slice version should work too.
This patch came out of me trying to determine what I could do to speed up
some rust code compiled to wasm which ran much faster using node than using
wasm2go+go.
For my code, it led to a speed up close to the microbenchmark I provided
here. Which is kind of unsurprising as rust code compiled to wasm can be
quite heavy in loads and stores to memory.
I ran a quick test on go-Marisa to evaluate the change, and it shows A
speed up as well, using `BenchmarkTrie/Words` benchmarks on a dictionary of
466,550 words.
[table below]
I didn't try SQLite, re, etc because they're using unsafe memory which bypasses these.
|
|
(putting the table in a separate reply because apparently email replies can't do markdown)
Edit: Given we're effectively trying to convince the go optimizer to do the right thing, I decided to test some other go versions to see how this change differed:
So it looks like the big effect is in 1.26 & 1.27, but earlier ones are less effected (and 1.19 is especially bad, slowing down with this change from already being much slower. 1.19 doesn't appear to inline, and so repeats checks). Here's a survey using go-marisa's benches so we're less in microbenchmark territory:
Which is similar (big gain for 1.27 and 1.26, limited gain for 1.25) |
codyps
changed the title
|
I'd be curious to see benchmarks on lightningcss (give it bootstrap or some equally large CSS framework) too, since that's a rust one. |
|
I've actually found the earlier versions of go are helped by a 2 slice version: with a tweak, it lets them eliminate the address masking on x86_64 for uint32. Go still does the 2 bounds checks on the earlier go versions though. I'll push an update. go-marisa benchmarks with the current version (c69cb63):
lightningcss (minifying bootstrap-4.6.2.css,
(all these benchmarks are on a AMD Ryzen 3 PRO 5350GE running linux) edit: the microbenchmarks with the current version for reference:
(For the microbenchmarks each iteration performs one 32-bit load, increment, store, and sum, cycling through aligned addresses in 64 KiB of memory. Exercised the functions being switched; didn't run wasm2go or translated Wasm.) edit 2: I also ran the 8 bit (control, unchanged), 16 bit and 64 bit ops in a microbenchmark:
|
codyps
changed the title
|
I haven't really reviewed the code yet, but those results look pretty significant! off-topic: at some point, looking into more creative ways to make stuff build faster with less memory would also be nice... |
|
Just to clarify my understanding: these speedups are always without the Maybe the Go compiler is catching up to us, no reason not to. Still totally worth it, but they're less exciting if |
|
I tried many combinations of this, and yours seems to be the best: func load16[T uint32 | uint64](mem []byte, addr T) uint16 {
a := uint64(addr)
return binary.LittleEndian.Uint16(mem[:0:len(mem)][a : a+2 : a+2])
}Mine is a not too distant second: func load16[T uint32 | uint64](mem []byte, addr T) uint16 {
return binary.LittleEndian.Uint16(mem[uint64(addr) : uint64(addr)+2 : len(mem)])
}And unsafe still beats it. These (well not unsafe) are all ways of specifying the same thing. At some point, maybe a discussion with the Go authors is in order. Yours looks particularly obtuse, IMO. PS: this is no way a snide (I'm glad you found it!), I just hope we agree it looks like something no one would write, ever, so it's weird that it's the version the Go compiler optimizes best. |
|
Here's some other projects (including go's standard image package) doing similar things to convince go's optimizer to generate better code:
(probably some copying of code going on there as it's all image stuff) Here's an old go issue about it: golang/go#27857 Definitely would be ideal if the go optimizer got better at recognizing this so that on future releases with no concern about old go versions folks can use the obvious expression (ie: the one this PR removes and replaces with the one that gets the optimizer to perform better). |
|
Sorry for letting this sit. I redid some measurements, and I'm getting conflicting results. |
This changes checked multi-byte loads and stores to use slices with a fixed length and capacity, helping Go eliminate redundant work around each memory access.
For example, a 32-bit load changes from:
to:
Previously, Go checked the suffix slice and then checked that
encoding/binarycould access its fourth byte. The suffix’s potentially zero capacity also required pointer masking. The new slice has a known length and nonzero capacity of four, allowing the compiler to remove the internal length check and pointer masking.With the tested Go 1.26 and 1.27 compilers, widened
uint32addresses compile to a single bounds check. Go 1.25 still emits two slice checks, but benefits from eliminating the pointer masking. Arbitraryuint64addresses retain the checks needed to detect overflow.The initial
mem[:0:len(mem)]view limits capacity to the logical memory length. Needed because Go permits slicing into backing-array capacity, andmemory.growcan leave spare capacity beyond the valid Wasm memory. Widening the address before adding the access width preventsuint32wraparound. Invalid accesses still trap, and rejected stores trap before writing any bytes.The same pattern applies to 16-, 32-, and 64-bit loads and stores. Generated test fixtures are updated accordingly.
Validation:
go generate ./...andgo test ./...pass.