[Assets] Build a deterministic versioned cooked asset database

Parent: #126

Problem

tools/bloom-cook is currently texture-focused. A high-end renderer needs deterministic offline processing for meshes, meshlets/LODs, textures, materials, animation, environments, and world dependencies. Runtime parsing of source glTF plus uncompressed/general-purpose texture upload cannot support large scenes, virtualized geometry, or predictable platform memory.

Outcome

A versioned, deterministic, content-addressed cooked asset database with platform/quality variants and an incremental CLI.

Build graph

Inputs are source assets plus importer/cooker version, platform profile, quality profile, and relevant settings.
Each output has a stable content hash and explicit dependency hashes.
Rebuild only invalidated nodes; produce identical bytes for identical inputs/settings.
Store provenance, source license/path, warnings, and diagnostic names.
Write atomically; interrupted cooks may not leave valid-looking partial artifacts.

Required processors

Textures

Preserve color-space/normal/data semantics.
Generate coverage-preserving alpha mips and normal-aware mips.
Emit platform formats: desktop BC family (including BC7/BC6H as appropriate), mobile ASTC/ETC fallback, and WebGPU-compatible path; use KTX2/Basis where it is the right transport rather than as a blanket answer.
Record dimensions, mip offsets, format, alpha mode, and streaming page layout.

Meshes

Validate/normalize coordinate, index, tangent, skin, and morph data.
Generate conventional LODs and bounds.
Provide meshlet/cluster hierarchy/page output needed by the virtual-geometry issue.
Preserve material boundaries and stable submesh/material IDs.

Materials/animations/environments

Cook the complete supported glTF material extension data.
Pack animation clips/skeletons with validated joint mappings and compression metadata.
Prefilter environment lighting and record color/exposure conventions.

Package/index

Versioned manifest maps logical asset IDs to variant chunks, dependencies, offsets/sizes, and hashes.
Support loose development output and packed shipping archives without changing logical IDs.
Runtime rejects incompatible versions with an actionable recook message.

CLI/API

Provide commands equivalent to:

bloom-cook build <manifest> --platform macos --quality high
bloom-cook inspect <artifact-or-package>
bloom-cook verify <package>
bloom-cook clean --unreferenced

Exact syntax may differ, but output must be scriptable and diagnostics identify source asset/material/node.

Acceptance criteria

Sponza, Bistro, Damaged Helmet, and a skinned model cook without runtime source glTF parsing in shipping mode.
Two clean cooks with identical inputs/settings produce byte-identical manifests/chunks.
Editing one source texture rebuilds that texture and dependent package index, not unrelated meshes/animations.
Runtime loads the correct adapter/platform variant and reports deliberate fallback.
Cooked texture memory/quality and mesh load time are benchmarked against the current path.
Corrupt chunk/hash/version cases are tested and fail safely.
The cooker emits the cluster/LOD metadata contract required by virtual geometry.
Documentation includes asset manifest examples, cache location, CI use, and migration from direct source loading.

Likely files

tools/bloom-cook/
shared asset-format crate/module (do not duplicate struct layout between cooker/runtime)
native/shared/src/textures.rs, models*.rs, animation/environment loaders
package scripts and release workflow

Non-goals

Runtime streaming policy; covered by the async streaming issue.
An editor asset browser.
Removing source-asset loading in development mode.

Dependencies

Can start independently. Coordinate formats with #131, #134, and #137 before freezing v1.

Dependency checkpoint from #131 / PR #147: bloom-cook now emits a versioned little-endian .bgeo artifact with deterministic leaf meshlets, canonical tables, source/payload/page SHA-256, hard page budgets, hierarchy/error fields reserved in the cluster record, and explicit compatibility records. Damaged Helmet cooks byte-identically twice; corruption/version/range/hash controls are required in the quick quality gate. This intentionally does not check a #136 box yet: the asset database still needs manifests/chunk dependency tracking, platform variants, runtime lookup, and a real coarse LOD hierarchy. Please adopt the format/provenance contract in docs/virtualized-geometry.md rather than creating a competing geometry container.

Dependency handoff from #131 / PR #147: the cooker now has an opt-in --hierarchy-levels 0..=16 setting in addition to meshlet/page limits. #136 provenance/cache keys must include hierarchy level count (and the cooker/format revision), because the source SHA alone intentionally stays constant while leaf-only and hierarchical .bgeo payloads differ. Default level 0 remains byte-identical to the previously qualified artifact. See docs/evidence/issue-131-atomic-hierarchy-v1.json at f8308d1.

Additional #131 handoff (fe92915/c22d89e): hierarchical .bgeo files now guarantee that coarse roots are the cluster/page prefix and expose exact root-page count/bytes. #136 can treat that prefix as the initial required dependency set after validating the archive; it does not need to scan all cluster records. Damaged Helmet currently qualifies at 8 pages / 469,360 logical bytes (524,288 bytes with fixed 64 KiB slots). Runtime staging/allocation overhead remains intentionally unclaimed.

Dependency handoff from #131 / PR #147 (2c51d09 / a9f0135): .bgeo now has an explicit version 2 quantized32 vertex payload while version 1 float32 remains the byte-exact default. The #136 build key must include vertex_format as well as hierarchy levels, meshlet limits, page budget, cooker revision, platform, and quality policy; the same source hash intentionally yields different valid artifacts.

The cook report now supplies exact float32-baseline payload/root-prefix reductions plus reconstruction errors. This gives #136 enough information to enforce per-quality asset policy instead of silently choosing packed data. Sponza is the important control: packing saves 53.06%, normals/tangents stay near 0.0036 degrees, but large tiled UVs report 0.01381 absolute binary16 error. A manifest policy should retain float32 or fail the build when configured error limits are exceeded.

No #136 box is checked yet; manifests, dependency hashes, incremental invalidation, platform variants, and runtime lookup remain outstanding. Contract/evidence: docs/virtualized-geometry.md and docs/evidence/issue-131-quantized-vertices-v2.json.

Content-addressed geometry-store checkpoint is pushed in 289ba31, with docs/evidence in 16c58b0 (PR #147).

What this adds:

geometry-store <logical-id> <source> <store> with canonical recipe keys covering source closure, recipe version, meshlet limits, page budget, hierarchy levels, and vertex format;
immutable chunks/sha256/<artifact>.bgeo plus atomic bloom-asset-manifest-v1 logical manifests;
verified zero-write cache hits;
content deduplication across logical IDs;
standalone asset-inspect that recomputes the manifest key and validates dependencies, chunk/file/payload hashes, source closure, format, and strict .bgeo structure;
fail-closed malformed-manifest and corrupt-chunk behavior;
path-safe logical IDs, including a fix preventing dotted-ID collisions.

Damaged Helmet qualification:

recipe key a5d53b58...;
artifact 6c8f924e..., 1,446,496 bytes;
manifest efe37c5a..., 1,041 bytes;
two clean stores are byte-identical;
first build 0.88 s, verified hit 0.05 s (17.6x observed), with zero hit writes;
a second logical ID reused the same chunk and the store retained one chunk.

Twenty-three focused tests and the complete local quick CI lane pass. Shipping runtime delta remains zero, so no rendered pixels or frame-time paths change.

I checked only the acceptance item for emitting the cluster/LOD metadata contract required by virtual geometry. The complete database remains open: path/license provenance, other asset recipes, platform/quality variants, multi-asset invalidation, package archives, runtime lookup, and streaming are not claimed.

Evidence: Markdown, JSON, and store contract.

Deterministic store-index checkpoint is pushed in 2effe1e, with evidence/docs in 9ad5ca9 (PR #147).

What this adds:

asset-index <store> emits canonical bloom-asset-index-v1 entries sorted by logical ID;
every manifest recipe/dependency and every referenced chunk/file/payload/source/format contract is revalidated before indexing;
entries record logical ID/kind, build key, source hash, manifest path/hash, and immutable artifact path/hash/size/format;
no timestamps or output-root paths, so clean stores are byte-identical;
unchanged generation writes zero indexes;
asset-index-inspect reconstructs the expected index and rejects stale, corrupt, or non-canonical bytes;
symlinks, path escapes, unexpected manifest-tree files, and path/content ID disagreements fail closed;
reports distinguish referenced bytes from deduplicated unique-chunk bytes.

Real store proof: two logical IDs -> two sorted entries -> one shared 1,446,496-byte chunk. The 1,686-byte index hash is abab52af...; an independently populated store is exact. Initial install measured 0.37 s, unchanged fully validated generation 0.03 s with zero writes. A scoped recipe change leaves the unrelated manifest exact, marks the old index stale, and rewrites only the derived index.

Twenty-five focused tests and the complete local quick CI lane pass; runtime/pixel delta remains zero. No additional acceptance box is checked because this remains a geometry-only loose-store index, not the complete multi-asset/platform/runtime database.

Evidence: Markdown and JSON.

Platform/quality variants and explicit resolution landed in PR #147 at 8bfa905; evidence is in docs/evidence/issue-136-asset-variants-v2.{md,json}. I marked the deterministic-clean-cook and corrupt-store acceptance boxes complete.

Delivered:

byte-compatible unprofiled v1 manifests/indexes;
v2 manifests under variants// with profile-separated recipe keys;
content deduplication across compatible profiles;
deterministic v2 index sorting by logical ID/profile;
exact-first asset-resolve with only caller-authored ordered fallbacks and separate legacy opt-in;
full manifest/chunk/index revalidation before selection;
zero-write cache hits and fail-closed path/profile/corruption checks.

A pinned finite Bistro subset qualifies two clean stores in opposite insertion orders: macos/high and portable/high have distinct build keys, share one 13,015,712-byte artifact, and produce byte-identical 1,929-byte indexes. Exact macOS selection and explicit windows/ultra -> portable/high fallback pass. The 64/96 mesh derived sets expose an authored non-finite vertex and correctly fail closed; validation was not weakened.

All 29 release cooker tests, strict Clippy/format, and the file-line gate pass. Runtime index loading/streaming and the remaining asset processors stay open, so this issue is not being closed.

Native loose-store resolution and file-backed virtual-geometry demand paging are now qualified and pushed.

Checkpoints:

c601626 — bounded non-blocking native index/store resolver with exact/caller-ordered fallback policy and fail-closed artifact validation
1cc6f1a — page-range I/O behind GPU feedback, per-page SHA revalidation, atomic uploads, fixed request/byte budgets, and runtime telemetry
c300880 — permanent evidence

Real #131 10M store proof on Apple M1 Max / Metal:

v2 index: 984 bytes, SHA-256 8ad290a5...
immutable artifact: 582,052,704 bytes, SHA-256 45b6aa47...
only metadata + 50 coarse-root pages retained after registration
370 I/O requests / 370 completions / 0 failures
76,226,352 bytes demand-read instead of retaining the 582 MB artifact
fixed 64 MiB GPU pool, 955 resident pages, 905 streamed fine pages
ending I/O reservations/in-flight/ready all zero
maximum caller-side request/poll duration 0.1239 ms
6.7390 ms wall mean, 4.4761 ms GPU mean, 7.8572 ms GPU p95
zero pending groups, fallbacks, missing-current pages, overflows, or invalid records

Qualification: focused virtual geometry 34/34; shared library 465 passed / 1 expected ignored; GPU golden corpus 77 passed / 2 expected ignored; strict shared Clippy; wasm web,models3d; formatting and diff checks. The file-size ratchet remains red only for the same three unrelated existing files.

Evidence: Markdown, JSON, and runtime contract.

No additional #136 acceptance box is claimed yet: automatic adapter-owned variant choice, other asset processors, packed/network/web delivery, GC, provenance, and the full shipping database remain open. The asynchronous cold validation is deliberately complete and off-thread; its 8.439 s result on the 582 MB artifact is recorded as a later optimization target rather than hidden.

Incremental texture-store qualification is pushed at d24f17e, with permanent evidence at 9dfe14f. I marked only the source-texture rebuild-isolation acceptance item complete.

Delivered:

texture-store <logical-id> <source> <store> with content-addressed BC7 DDS chunks and canonical v1/v2 manifests;
build keys covering exact source bytes, recipe version, normal/data/color-space semantics, and optional platform/quality profile;
strict DDS format/dimension/mip/hash validation, zero-write verified hits, atomic manifest install, and fail-closed corruption handling;
mixed geometry/texture canonical indexes; the virtual-geometry runtime validates common identities, ignores declared texture entries, and rejects unknown kinds;
direct texture cooking and store cooking now share one encoder.

The release isolation regression builds a package with two textures and one geometry asset. Editing one texture writes exactly one new texture chunk and one manifest, leaves the other DDS/manifest and the .bgeo/geometry manifest byte-identical, leaves the installed index byte-identical but detectably stale, and makes one explicit index rebuild write exactly one index. DDS corruption is then rejected by hash validation.

Qualification: cooker 37/37 release tests; shared native library 472 passed / 1 intentional ignore; strict cooker/native Clippy; formatting and diff checks. The repository file-line gate remains red only for five unrelated existing oversized files; no changed file crosses the ceiling.

Evidence: Markdown, JSON, and updated store contract.

Still open: cross-kind material dependency graphs, platform-specific texture formats, indexed runtime texture loading, packed archives/GC/provenance, full shipping-scene cooking, and texture memory/quality benchmarks.

The cooked-store workflow guide is complete and pushed at 17e93da; I marked the documentation acceptance item complete.

The guide now includes:

complete illustrative geometry and texture manifest shapes, including recipe/settings/dependency/artifact fields and the v2 profile extension;
an explicit cache-location contract: the third positional argument is the whole store root, there is no hidden global cache, and recommended local roots are now ignored by Git;
a runnable CI sequence covering geometry + texture cooking, canonical index generation, strict index inspection, cache-key guidance, and package publication boundaries;
a staged migration from direct source loading, including the native VirtualGeometryStoreLoader request/poll path, DDS sibling fallback, explicit variant policy, and an honest warning that indexed logical texture resolution is not implemented yet;
recook/version failure guidance and the source/manifests/index/chunks shipping boundary.

I executed the documented mixed-store CI flow on this revision using the checked-in Damaged Helmet GLB and embed-perry/bloomFull.png. It produced a validated v2 index with 2 profiled entries, 2 unique chunks, 2,151,652 referenced/unique bytes, index SHA-256 45b0e837fd2b7447a08753e9609acd307ce9d710fdf2d3991ea6358dd3cb2896, and asset-index-inspect returned validation: pass.

Guide: docs/cooked-asset-store.md.

This does not claim the remaining full-scene shipping, automatic adapter variant, or memory/quality benchmark acceptance items.

Cooked texture memory/quality and mesh load-time qualification is pushed.

Checkpoints:

07e748b — repeatable texture/geometry benchmark commands, texture recipe v2, quality-preserving normal mips, and direct RGBA8 DDS runtime upload
9800713 — permanent benchmark evidence

Accepted measurements on Apple M1 Max / macOS 26.5:

4096² Sponza color: BC7 reduces the complete mip chain from 89,478,484 to 22,369,648 bytes (74.99997%), with SSIM 0.999034, RGB PSNR 52.225 dB, and source-decode/DDS-parse speedup 341.68x.
4096² Sponza normal: recipe v2 deliberately uses RGBA8 after BC7 angular quality failed investigation. Mip-zero RGB and replacement alpha are exact; mean angular error is 0.000000331°, DDS parse is 98.64x faster than source decode, and the recorded VRAM reduction is honestly 0%.
Damaged Helmet: cooked .bgeo is 1,446,496 bytes versus 3,773,916 source bytes; full cooked read/structure/hash validation averages 8.6405 ms versus 97.2481 ms for source glTF import, an 11.2549x speedup.

The normal DDS retains Bloom's vector-filtered direction and accumulated LEADR/Toksvig variance mips. The native DDS path now uploads RGBA8 mips directly on adapters without BC support instead of discarding them through top-mip decode/regeneration.

Qualification: cooker 43/43 release tests and strict Clippy; native shared 474 passed / 1 intentional ignore and strict correctness/suspicious/performance Clippy; formatting and diff checks passed. The file-line ratchet remains red only for the same five unrelated existing files; every changed file is below 2,000 lines.

Evidence: Markdown, JSON, and benchmark/runtime contract.

I marked only the cooked texture memory/quality and mesh load-time acceptance item complete. Full shipping-scene cooking and automatic adapter/platform variant selection remain open.

Adapter-profile foundation is now pushed in d5504aa with permanent evidence in fc7fcf3.

Qualified on Apple M1 Max/macOS 26.5:

recipe v3 portable/high: RGBA8 sRGB, 2,815,824 bytes
recipe v3 macos/high: BC7 sRGB, 705,156 bytes
distinct build keys/chunks; v2 index inspection passed
macOS request resolved exact; missing Windows request resolved only through declared portable/high fallback rank 0
renderer-owned virtual-geometry request policy selects native desktop profiles only with accepted BC support, otherwise portable; mutation is rejected and selection emits structured diagnostics
44 cooker release tests passed; 475 native release tests passed with 1 intentional ignore; both Clippy policies and formatting passed

Evidence: docs/evidence/issue-136-adapter-profile-foundation-v1.{md,json}.

I am deliberately leaving the full runtime adapter/platform acceptance item open. The current indexed runtime store resolves virtual geometry, and DDS can load both variants, but logical-ID texture resolution from index.json is still missing. That generic indexed texture loader is the next gap.

Runtime adapter/platform variant loading is now qualified and the acceptance checkbox is checked.

Implementation:

cc557c0 — bounded indexed cooked-texture worker, renderer-owned adapter request, hash/DDS validation, direct semantic-preserving upload, shared adapter profile plan
9140c96 — production end-to-end smoke executable
280d2ba — permanent evidence in docs/evidence/issue-136-runtime-texture-variants-v1.{md,json}

Real Apple M1 Max/Metal qualification using freshly cooked embed-perry/bloomFull.png stores:

native exact: macos/high -> BC7 sRGB, 705,156 bytes, validated and uploaded
deliberate fallback: missing native -> portable/high at rank 0, RGBA8 sRGB, 2,815,824 bytes, validated and uploaded
stable reason: portable-fallback-after-native-miss

The first BC GPU control also found and fixed a compressed-tail bug: 2x2/1x1 BC mips must upload using their physical 4x4 block extent.

Qualification: 44 cooker release tests passed; 480 native release tests passed with 1 intentional ignore; CI Clippy policies and no-default/models-only/image-extras-only builds passed. The only remaining #136 acceptance item is the full source-free shipping cook for Sponza, Bistro, Damaged Helmet, and a skinned model.

The final source-free shipping-scene acceptance item is qualified and pushed.

Implementation:

1420e5a — shared versioned .bscene format, offline scene/texture-closure cooking, path-aware loose glTF import, indexed native scene selection/reconstruction, texture recipe v4 alpha-coverage mips, and fail-closed validation
c69bb9a — permanent evidence in docs/evidence/issue-136-shipping-scenes-v1.{md,json}

Fresh portable/high results:

Damaged Helmet: 1 primitive / 1 placement / 5 textures; 1,583,687-byte scene
Fox: 1 primitive / 1 placement / 1 texture; 24 joints, 3 clips; 242,225-byte scene
Sponza: 103 primitives / 103 placements / 69 textures; 21,660,807-byte scene
full Bistro (assets/bistrox.gltf, not the generated subset): 551 unique primitives / 2,909 placements / 680 textures; 188,317,493-byte scene and 16,266,801,464 validated texture bytes

All four indexes passed strict inspection. I then removed each store's complete cooker-manifest tree and reran the native smoke from only index.json plus immutable chunks. Exact-profile selection, every scene/DDS hash, texture dependency closure, and runtime model reconstruction passed with source_gltf_reads: 0. Fox proves the skinned/animated path. No scene lost a triangle, primitive, or placement; deterministic non-finite shading-attribute repairs are recorded explicitly in each archive and manifest.

Qualification: scene format 2/2 release tests; cooker 46/46; native shared 480 passed / 1 intentional ignore; strict cooker/format Clippy and native CI Clippy policy passed; no-default, models-only, image-extras-only, and combined feature builds passed. The line ratchet remains red only for the same five unrelated pre-existing files, with no new file above 2,000 lines.

Evidence: Markdown, JSON, and the updated store contract.

All issue #136 acceptance boxes are now complete.

Navigation Menu

Repository navigation

[Assets] Build a deterministic versioned cooked asset database #136

Problem

Outcome

Build graph

Required processors

Textures

Meshes

Materials/animations/environments

Package/index

CLI/API

Acceptance criteria

Likely files

Non-goals

Dependencies

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

[Assets] Build a deterministic versioned cooked asset database #136

Description

Problem

Outcome

Build graph

Required processors

Textures

Meshes

Materials/animations/environments

Package/index

CLI/API

Acceptance criteria

Likely files

Non-goals

Dependencies

Activity

proggeramlug commented on Jul 28, 2026

proggeramlug commented on Jul 28, 2026

proggeramlug commented on Jul 28, 2026

proggeramlug commented on Jul 29, 2026

proggeramlug commented on Jul 29, 2026

proggeramlug commented on Jul 29, 2026

proggeramlug commented on Aug 1, 2026

proggeramlug commented on Aug 27, 2026

proggeramlug commented on Aug 29, 2026

proggeramlug commented on Aug 29, 2026

proggeramlug commented on Aug 29, 2026

proggeramlug commented on Aug 29, 2026

proggeramlug commented on Aug 29, 2026

proggeramlug commented on Aug 29, 2026

Metadata

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

Issue actions