Widen a squashed placement's bound before the per-brick cull (#649)

[leonardoaraujosantos]

Why

#649: a full brick build disagrees with the document's own raw field (clay_eval_points) at in-band samples. #652 fixed the deformer cull and named two mechanisms that remain. This PR fixes the second one, B: squashed placements are culled as if their field were a distance.

A per-axis scale on an item or on its layer makes the field a bound on the distance. cscale_nu_dist multiplies the local value by min(s), so the value can be short of the true distance by up to q = max(s) / min(s), and the two levels' ratios multiply. The per-brick cull drops an item once its bound is more than band + pad from the brick. That argument only holds out to q * (band + pad). A brick in between dropped an item whose field was still inside the band there.

I reproduced it on origin/main 7023dd9 with the issue's probe before changing anything:

seed (rich)	main	this PR
5743	5,625 in-band samples off, worst 0.06767	0
6290	151 in-band samples off, worst 0.01986	0

Root cause

All three cull tests compare the plain item bound: the item test and the group test in Compiler::compile_list, and the planned-entry survive test (CullIndex entries). None of them knows the field understates under a squash. item_geometry_reach_in_document already refuses a squashed placement for this reason (bounds.h). The cull cannot simply refuse, because then a squashed layer would cull nothing.

Fix

The bound is widened, not exempted. A point D from the geometry reads at least D / q (less the rounding), so the cull's "more than band + pad + w from the geometry" has to become q times that. Here w is what the bound already dilates by: the rounding plus the combine support. The widening is

slope * (band + pad) + reach,   slope = q - 1,   reach = slope * w

A group takes its subtree's widest slope and reach, plus slope times its own blend support. That is scene::CullSquash (item_cull_squash, node_cull_squash). item_bound_dilation is factored out of placed_local_bound, so w is exactly the term the bound adds. Both terms are exactly 0 for a similarity (max/min of three equal floats), so a document with no per-axis scale makes identical cull decisions. A test pins this.

The band is a new input to the cull:

• scene::CullRegion gains float band ("how far inside region the samples lie"). It now has a constructor, so every CullRegion{box} keeps meaning "no band" and GCC's missing-field-initializer warning has nothing to say.
• The band is set wherever it is known:
  • the refill request tapes and the batch plan (widest band)
  • the resume and uniform-brick tasks
  • the frontier jobs and try_band
  • the volume bake
  • gradient-normal meshing
  • clay_eval_grid / _device, which take the band from the lattice's margin inside the host's region. That margin is the band clay_brick_cache_cull_region added.
• CullIndex::Entry caches the squash (40 -> 48 B). Chain::widest records the chain's widest squash.
• CullIndex::plan(region, band = 0): a squashed chain's coarse scan widens the region by its widest entry's widening. The packed scan stays one box test per entry, and per-brick survival is still a subset.
• CullPlan::serves_band: a compile drops a plan made for a narrower band than its region carries, but only when the document holds a squashed placement. About thirty plan(region) callers in tests and benchmarks are therefore never wrong. On a squashed document they lose the plan, not an item.

Regression test (tests/unit/test_squashed_cull.cpp)

Three C ABI cases. Each refills a row of bricks whose band-dilated boxes all miss the node's unsquashed influence bound, in one batch so the plan is used, and compares every in-band sample with clay_eval_points. Each one REQUIREs in-band samples beyond the unsquashed band, so it cannot pass because there was nothing to drop.

case	origin/main (same test source)	this PR
item scaled (4, 1, 1)	652 samples off, worst 0.11	0
layer scaled (4, 1, 1)	652 samples off, worst 0.11	0
both levels at 2, blended, rounded, grouped, smooth chain pad	66 samples off, worst 0.025	0

Three scene-level cases:

• A squashed gnarly_document (layer squash, items squashed at the root, inside nested groups and in the instanced layer). Each brick is planned alone. A per-brick tape with the index and a plan for its band must be byte-identical to the plain one, and so must a tape with a plan for no band, which has to be refused. Band-clamped samples must equal the full tape exactly.
• An unsquashed gnarly_document must produce byte-identical tapes with and without a band.
• item_cull_squash returns zero for a similarity and q - 1 otherwise, and the two levels multiply.

Mutation checks, each reverted afterwards:

• Compiler ignores the squash: 4 cases fail, including the gnarly one.
• Plan does not widen: 4 cases fail. The first draft of the gnarly test used one batch plan over scattered bricks, covered the whole document and passed under this mutation, so each brick is now planned alone.
• serves_band always true: the gnarly case fails on byte-identity.

Measured

RV_RAWCHECK=1 undo_bound_oracle_probe, Release, Apple M-series:

sweep	origin/main 7023dd9	this PR
seeds 5001..6500, rich	35 documents, worst 0.0825	9 documents, worst 0.0170
seeds 1..1500, plain	21 documents, worst 0.0310	21 documents, identical lines

26 documents drop out, 5743 and 6290 among them. Of the 9 left, 3 shrink sharply because B was part of their error: 5483 goes from 10,525 samples off to 14, 5907 from 4,132 to 236, and 5976 from 1,652 to 77. 6055 drops from 3,380 to 3,177 at the same worst value. The other 5 are unchanged. The plain sweep has no document this mechanism reaches.

Cost on unsquashed documents (clay_bench, Release, Apple M-series, CPU). The two binaries were interleaved over 3 rounds of 5 repetitions at --benchmark_min_time=0.2s. Each figure is the median of the per-round medians, in µs:

case	origin/main	this PR	ratio
BM_DeepDocCullPlanned2000	264	268	1.015
BM_DeepDocCullPlanned10000	1296	1310	1.011
BM_DeepDocCull2000 (no plan)	1027	1059	1.031
BM_DeepDocRefillPlanned10000	716	736	1.027
BM_CullIndexRebuild	1150	1196	1.040

All three rounds agree within about 1%, so these small costs are real. They come from the extra test on a plain miss and from CullIndex::Entry growing from 40 to 48 B. The first version tested the squash before the plain bound and called out of line for every dropped item: the unplanned cull was +11% and the planned one +3.5%. The plain test now runs first, and a similarity is decided inline (culled_item). check_bench.py passes every gated case in the run. The only failures it reports are the cases outside the filter, listed as missing from the results.

Not in this PR: mechanism A

Seed 978: the #335 chain-pad envelope applies its N = 75 value below 75 contributors, and that breaks the spec's relative bar. Fixing it is a trade between the pad and the #335 performance win, and it needs the iPad gate. I left it alone. The plain sweep's 21 documents and the rich sweep's remaining 9 are where it shows. This PR is Part of #649. It does not close the issue.

Verification

• cmake --preset cpu-only -DCLAY_BUILD_TESTS=ON, then build and ctest --preset cpu-only: 10/10 passed (four unit shards and the partition check).
• check_layering, check_kernel_dialect, check_licenses, check_doc_latency, and check_c_abi build/cpu-only/libclay_shared.dylib (hygiene + ctypes FFI) all pass. check_test_shards --binary: 3010 cases partitioned.
• npx @fission-ai/openspec@1.12.0 validate --all --strict: 71 passed.
• tools/release_check.py --skip-slow passes every software row: version, configure, build, tests (11/11), parity, layering, dialect, licenses, task-symbols, bindings (it imported the built pyclay), kernels, abi and openspec. The rows that fail are release-time hardware records:
  • device: the iPad gate is stale against files this PR does not touch (CMakeLists.txt, clay.h), so it fails on main too.
  • The four hardware/* waivers were recorded at 59e42cc, and include/clay/eval/bake_volume.h (one line that passes the band) counts as a kernel-path file. They are re-run or re-waived when the release is cut.
• The rich sweep was re-run on the final commit and gives the same 9 documents with identical lines.
• OpenSpec change widen-the-cull-for-a-squashed-placement modifies the scene-model "Per-brick tape culling" requirement. It repeats the four existing scenarios and adds three. docs/05 states the widening.
• No ABI, format or version change. No C ABI struct changes.
• Cognitive complexity: compile_list gets simpler (the group test moved into culled_group, and the planned test is one call). The new functions are small: node_cull_squash has two branches and a loop.

Part of #649.