Clone and bulk-read a voxel grid, and state its threading (ABI 0.123.0)

CMakeLists.txt

@@ -1,7 +1,7 @@
 cmake_minimum_required(VERSION 3.24)
 
 project(claycore
-  VERSION 0.122.0
+  VERSION 0.123.0
   DESCRIPTION "Portable C++20 SDF + voxel engine library"
   LANGUAGES C CXX)  # C: ufbx
 

README.md

@@ -392,7 +392,10 @@ volume item that carries the palette per sample, so colour survives). Direct —
 trilinear interpolation between cell centres and **redistanced**, so the result
 carries a Lipschitz bound a raymarcher and a blend can trust. Non-destructive:
 the grid is untouched, and the result is an ordinary volume item in an
-ordinary SDF layer.
+ordinary SDF layer. To convert off the interface thread, `clay_voxel_grid_clone`
+the layer's grid there (a copy of its material chunks, every level and the
+palette, in no document) and convert the clone on a worker; `clay.h` states
+the threading contract beside each call.
 
 **Mesh → voxel** — `VoxelGrid.rasterize_mesh(mesh)` in Python,
 `clay_voxel_rasterize_mesh` in C. An imported model reached an SDF layer in one

bindings/c/clay.h

@@ -24,7 +24,7 @@ extern "C" {
 #endif
 
 #define CLAY_ABI_MAJOR 0
-#define CLAY_ABI_MINOR 122
+#define CLAY_ABI_MINOR 123
 #define CLAY_ABI_PATCH 0
 
 /* Upper bound on the element count of any batch call: points, rays, cells,
@@ -5395,6 +5395,50 @@ clay_voxel_grid* clay_voxel_grid_create(float voxel_size);
  * document layer is owned by that document: this returns
  * CLAY_ERROR_INVALID_ARGUMENT and leaves the document untouched. */
 clay_result clay_voxel_grid_destroy(clay_voxel_grid* grid);
+/* A deep copy of `src` into a grid the CALLER owns, from an owned or a borrowed
+ * handle alike: every resolution level, the active level, the palette and the
+ * sculpt layers. Free it with clay_voxel_grid_destroy — on the clone that is
+ * obeyed even when the source was a document layer's, whose own destroy is
+ * still refused. Added at ABI 0.123.0 (issue #658).
+ *
+ * This is how a borrowed grid leaves the interface thread. Before it, the only
+ * route was a clay_voxel_get per cell of the occupied BOX — empty cells
+ * included, the active level only, the sculpt layers dropped — rebuilt on the
+ * worker with clay_voxel_set. The clone is one container copy, sized by the
+ * chunks that hold material rather than by the box.
+ *
+ * WHAT IS NOT COPIED, deliberately. The clone is in no document: edits to it
+ * reach neither the source nor the document's undo history, and
+ * clay_document_save does not write it. A sculpt layer that is RECORDING in
+ * the source arrives closed (clay_voxel_recording_sculpt_layer reads 0), since
+ * "the next edit belongs to this pass" describes the source's session rather
+ * than its cells; its record, strength and visibility arrive intact. A dirty
+ * chunk drain in progress on the source handle stays there: the clone's dirty
+ * set is EVERY occupied chunk, as a grid read from a file has, because nothing
+ * has drawn it yet. clay_voxel_change_count starts at 0.
+ *
+ * THREADING. Cloning is a READ of the source, on the footing
+ * clay_mesh_sculptor_create documents: it touches no document state, and
+ * clay_last_error is per-thread. A clone of an OWNED grid may run on any
+ * thread. A clone of a BORROWED grid may run on a worker against a const
+ * document, and is NOT safe concurrently with a mutating clay_document_* /
+ * clay_voxel_* call on that document — the host owes that serialization, and
+ * a grid edited while it is copied produces a copy of neither state rather
+ * than a refused call. Calls on ONE handle must be serialized by the host, as
+ * with every handle here.
+ *
+ * ONE MORE HAZARD, and it is easy to miss: two READERS of the same grid can
+ * race. clay_voxel_bounds (and the raycast and repair calls that read the
+ * bounds) fill a cache on the grid lazily, so a const read on a grid whose
+ * cache is cold WRITES it, and a clone running concurrently copies those
+ * fields while they are written. Before a worker clones a borrowed grid,
+ * either warm the cache — one clay_voxel_bounds on the source, on the thread
+ * that owns the document — after which those reads are pure reads, or keep
+ * every other read of the source off the interface thread until the clone
+ * returns. Cloning on the interface thread itself has neither problem, and at
+ * a copy of the material chunks it is cheap enough to (0.07 ms for 89k
+ * cells). The clone's own cache is cold and belongs to the clone. */
+clay_result clay_voxel_grid_clone(const clay_voxel_grid* src, clay_voxel_grid** out_owned);
 
 /* Adds a voxel layer and borrows its grid. A voxel layer carries no SDF
  * content, so clay_add_item and clay_layer_add_item do not apply to it.
@@ -11428,6 +11472,36 @@ clay_result clay_voxel_repair_fill_voids(clay_voxel_grid* grid, const clay_mask*
 
 clay_result clay_voxel_occupied_count(const clay_voxel_grid* grid, size_t* out_count);
 
+/* Every occupied cell of the ACTIVE level and its palette index — the read
+ * counterpart to clay_voxel_set_many. Added at ABI 0.123.0 (issue #658).
+ *
+ * out_xyz receives count*3 int32 values and out_index count values, in the
+ * same ORDER: ascending z, then y, then x, which is the order a box walk with
+ * x innermost visits them. The order is a promise, so two reads of an
+ * unchanged grid agree element for element.
+ *
+ * Size-query pattern: with both buffers NULL, *out_count receives the count
+ * (clay_voxel_occupied_count's) and no cell is walked. Either buffer may be
+ * NULL on its own to read only that half. `capacity` is in CELLS; a capacity
+ * below the count is CLAY_ERROR_BUFFER_TOO_SMALL with the needed count in
+ * *out_count and nothing written. out_count is required.
+ *
+ * The cost follows the chunks, not the box: it walks the 32^3-cell chunks that
+ * hold material, so two cells far apart are two chunks rather than every empty
+ * cell between them, then sorts into the order above. Measured on an M-series
+ * host, a 3-cell-thick sphere shell: 31k cells in 1.1 ms, 89k in 4.1 ms, about
+ * two thirds of it the sort. A clone of the same 89k-cell grid is 0.07 ms, so
+ * to take a grid OFF the interface thread, clone it there and read the clone
+ * on the worker. On a partially refined level the cells the level
+ * INHERITS from its parent are reported where they are, exactly as
+ * clay_voxel_get reads them. Other levels are not read: switch the active
+ * level, or clone the grid and switch the clone's.
+ *
+ * THREADING as clay_voxel_grid_clone above: a read, safe on a worker while the
+ * document is const, and touching no cache. */
+clay_result clay_voxel_get_occupied(const clay_voxel_grid* grid, int32_t* out_xyz,
+                                    int32_t* out_index, size_t capacity, size_t* out_count);
+
 /* Cell writes that actually CHANGED a cell, since the grid was constructed.
  *
  * Why this exists: an edit that is entirely legal and entirely without effect
@@ -11601,10 +11675,33 @@ clay_result clay_voxel_mesh_quads(const clay_voxel_grid* grid, const clay_quad_p
  * how a caller assembles a sculpt one entry at a time by hand. clay_voxel_to_layer does NOT need that: it converts at index 0, whose
  * volume already carries every entry's colour per sample.
  *
- * Free with clay_item_destroy; placing it copies it, as every item does. */
+ * Free with clay_item_destroy; placing it copies it, as every item does.
+ *
+ * THREADING, on the footing clay_mesh_sculptor_create documents. The
+ * conversion is a READ of the grid and writes nothing but the new item: it
+ * reads cells, never the bounds cache clay_voxel_grid_clone warns about.
+ *  - On an OWNED grid it touches no document, so it may run on any thread.
+ *    This is the shape to use for a responsive grid-to-field crossing:
+ *    clay_voxel_grid_clone on the interface thread, this on a worker against
+ *    the clone, then clay_add_item / clay_layer_add_item back on the interface
+ *    thread. The clone costs a copy of the material chunks; the conversion is
+ *    the expensive half and it is the half that leaves.
+ *  - On a BORROWED grid it is a read of the document: it may run on a worker
+ *    against a const document, and is NOT safe concurrently with a mutating
+ *    clay_document_* / clay_voxel_* call. The host owes that serialization,
+ *    and a grid edited mid-conversion yields a volume of neither state rather
+ *    than a refused call — which is why the clone is the recommended route.
+ * Calls on ONE grid handle must be serialized by the host, as everywhere. */
 clay_result clay_item_volume_from_voxels(const clay_voxel_grid* grid, int32_t blur, int32_t index,
                                          clay_item** out_item);
 
+/* THREADING: this ADDS A LAYER, so it mutates the document and belongs on the
+ * thread that owns it, serialized with every other clay_document_* call. The
+ * grid argument is only read, and may be a clone (clay_voxel_grid_clone): a
+ * host that wants the conversion itself off that thread converts the clone with
+ * clay_item_volume_from_voxels at index 0 on a worker, then on this thread
+ * adds the layer (clay_add_sdf_layer) and places the item, bracketed with
+ * clay_document_begin_undo_group so it stays one undo as this call is. */
 clay_result clay_voxel_to_layer(clay_document* doc, const clay_voxel_grid* grid, const char* name,
                                 int32_t blur, clay_layer_id* out_layer);
 

bindings/c/clay_c.cpp

@@ -11384,6 +11384,21 @@ clay_result clay_voxel_grid_destroy(clay_voxel_grid* grid) {
     return CLAY_OK;
 }
 
+// VoxelGrid::clone rather than the copy constructor: a borrowed grid's sink and
+// pass capture point into its document's history, and a member-wise copy would
+// carry them (#658). The new handle is an owner, whatever the source was, and
+// carries no staged drain — that belongs to the source handle's conversation.
+clay_result clay_voxel_grid_clone(const clay_voxel_grid* src, clay_voxel_grid** out_owned) {
+    if (!out_owned) return fail(CLAY_ERROR_INVALID_ARGUMENT, "null out pointer");
+    voxel::VoxelGrid* g = nullptr;
+    clay_result r = resolve(src, &g);
+    if (r != CLAY_OK) return r;
+    auto* handle = new clay_voxel_grid();
+    handle->owned = new voxel::VoxelGrid(g->clone());
+    *out_owned = handle;
+    return CLAY_OK;
+}
+
 clay_result clay_document_add_voxel_layer(clay_document* doc, const char* name, float voxel_size,
                                           clay_layer_id* out_layer, clay_voxel_grid** out_grid) {
     if (!doc || !name) return fail(CLAY_ERROR_INVALID_ARGUMENT, "null document or name");
@@ -13841,6 +13856,32 @@ clay_result clay_voxel_occupied_count(const clay_voxel_grid* grid, size_t* out_c
     return CLAY_OK;
 }
 
+clay_result clay_voxel_get_occupied(const clay_voxel_grid* grid, int32_t* out_xyz,
+                                    int32_t* out_index, size_t capacity, size_t* out_count) {
+    if (!out_count) return fail(CLAY_ERROR_INVALID_ARGUMENT, "null out_count");
+    voxel::VoxelGrid* g = nullptr;
+    clay_result r = resolve(grid, &g);
+    if (r != CLAY_OK) return r;
+    // A count query answers from the per-chunk counters and walks no cell.
+    if (!out_xyz && !out_index) {
+        *out_count = g->occupied_count();
+        return CLAY_OK;
+    }
+    const std::vector<voxel::VoxelGrid::OccupiedCell> cells = g->occupied_cells();
+    *out_count = cells.size();
+    if (capacity < cells.size())
+        return fail(CLAY_ERROR_BUFFER_TOO_SMALL, "capacity is below the occupied cell count");
+    for (std::size_t i = 0; i < cells.size(); ++i) {
+        if (out_xyz) {
+            out_xyz[i * 3] = cells[i].cell.x;
+            out_xyz[i * 3 + 1] = cells[i].cell.y;
+            out_xyz[i * 3 + 2] = cells[i].cell.z;
+        }
+        if (out_index) out_index[i] = cells[i].index;
+    }
+    return CLAY_OK;
+}
+
 clay_result clay_voxel_change_count(const clay_voxel_grid* grid, uint64_t* out_count) {
     voxel::VoxelGrid* g = nullptr;
     clay_result r = resolve(grid, &g);

docs/05-claycore-library.md

@@ -3221,6 +3221,8 @@ windows at every size measured**, and the build follows the history while the
 cache's SIZE follows the model — 1.43 MiB at 5,000 items and at 20,000 alike — so
 a budget is sized from one and a build scheduled from the other.
 
+**A grid can leave the interface thread** (#658, ABI 0.123.0): `clay_voxel_grid_clone` deep-copies an owned *or borrowed* grid — every resolution level, the active level, the palette and the sculpt layers — into a handle the caller owns and frees with `clay_voxel_grid_destroy`. Until then a layer's grid was reachable only as a borrow of its document, and a host converting it on a worker rebuilt it from one `clay_voxel_get` per cell of the occupied *box*: empty cells included, the active level only, the sculpt layers lost — 17–46 ms on the interface thread for 5k–100k cells in ClaySpaceDesktop#285. A clone of an 89k-cell grid is 0.07 ms. It is a copy of the **grid, not the session**: the engine's member-wise copy would also carry the document's undo sink, an open pass capture and the recording flag, so `VoxelGrid::clone()` drops all three — edits to a clone reach neither the document nor its history, and a sculpt layer that was recording arrives closed. Its dirty set is every occupied chunk, as a grid read from a file has. `clay_voxel_get_occupied` is the bulk read `clay_voxel_set_many` lacked: every occupied cell of the active level and its index, sorted by z, then y, then x, by the size-query pattern, walking material chunks rather than the box. **Threading** follows `clay_mesh_sculptor_create`'s wording: the clone, the bulk read and `clay_item_volume_from_voxels` are reads; on an owned grid they touch no document and run on any thread, on a borrowed one they may run against a const document and are not safe concurrently with a mutating `clay_document_*` / `clay_voxel_*` call. One more hazard: `clay_voxel_bounds` fills a lazy cache, so two *readers* of a cold grid race — before a worker clones a borrowed grid, warm the cache with one `clay_voxel_bounds` on the owning thread, or keep every other read off that grid while the clone runs. The recommended crossing is clone on the interface thread, `clay_item_volume_from_voxels` on the worker against the clone, and the placement back on the interface thread.
+
 **And the voxel side has one now too** (#86): `clay_voxel_mesh` is the *whole* grid, always, and it costs every occupied chunk on every call — so displaying a sculpt used to cost the model while editing it cost the dab. Two calls close that, in the brick cache's own vocabulary: `clay_voxel_take_dirty_chunks` drains the chunks a mutation could have changed (capacity in, count out, remainder reported — the `clay_brick_cache_take_dirty` shape, not a size query), and `clay_voxel_mesh_chunks` meshes exactly those keys and reports a `clay_voxel_chunk_mesh_range` per key so a host patches GPU sub-ranges instead of rebuilding the buffer. Every mutation feeds the set through the one cell-write choke point, a write that changes nothing dirties nothing, a write on a chunk *face* also dirties the chunk across it — the exposure test reads that cell — and a chunk emptied to nothing is dropped from the grid and *still* reported, because that is the key whose quads a host must remove. Unlike `clay_brick_mesh_range`, these ranges **partition** the mesh with no shared vertex: a voxel face belongs to one cell in one chunk, so clamping the greedy merge to a chunk boundary emits more, smaller quads over the identical surface — never a crack, and no straddler to attribute the way `mesh_bricks` must (#66). The cost is triangle count at chunk seams (+3.7% measured on a realistic sculpt), which is why `clay_voxel_mesh` stays whole-grid for export. Measured on Linux desktop at the device gate's 0.02 cell size — *not* comparable to the device baseline, per `docs/RELEASE.md`; the ratio transfers, the absolute does not — one dab dirties ~2 of 81 occupied chunks and re-meshes in **0.65 ms against 23.3 ms** for the whole grid.
 
 ## 12. Dependencies (all permissive, all C/C++)

include/clay/voxel/grid.h

@@ -135,6 +135,24 @@ class VoxelGrid {
         palette_.resize(1, kernel::cf3(0, 0, 0));  // index 0 = empty, unused
     }
 
+    // A deep copy of the GRID and none of the SESSION around it (#658): every
+    // level, the palette, the active level and the sculpt layers, with their
+    // strengths and records.
+    //
+    // Use this rather than the copy constructor whenever the source may belong
+    // to a document. The implicit copy is a member-wise copy, so it also takes
+    // the change sink and the pass capture — two pointers into the OWNER's undo
+    // journal — and the recording flag, and an edit to such a copy is written
+    // into the source's history. A clone has no sink and no capture, and no
+    // sculpt layer is recording in it: the layer that was open in the source
+    // is carried as a closed one, because "the next edit belongs to this pass"
+    // is a statement about the source's session, not about its cells.
+    //
+    // It is also a grid nothing has displayed yet: every occupied chunk is
+    // dirty, exactly as a grid read back from a file is, the change counter
+    // starts at zero, and the bounds cache is cold.
+    VoxelGrid clone() const;
+
     // Cell size of the ACTIVE level.
     float voxel_size() const { return levels_[active_].voxel_size; }
 
@@ -586,6 +604,24 @@ class VoxelGrid {
     // -- queries -------------------------------------------------------------
     std::size_t occupied_count() const;
 
+    // Every occupied cell of a level with its palette index, sorted by z, then
+    // y, then x — the order a box walk with x innermost visits them, so a
+    // caller replacing one with this sees the same sequence.
+    //
+    // Walks the chunks that hold material rather than the bounding box, so the
+    // cost follows the cells and not the box: two cells far apart are two
+    // chunks, not the volume between them. A partially refined level reports
+    // the cells it INHERITS from its parent as well as the ones it stores,
+    // which is the same solid level_occupied_count counts. Empty for a level
+    // the grid does not have.
+    struct OccupiedCell {
+        VoxelCoord cell;
+        std::uint8_t index = 0;
+        bool operator==(const OccupiedCell&) const = default;
+    };
+    std::vector<OccupiedCell> occupied_cells() const { return occupied_cells(active_); }
+    std::vector<OccupiedCell> occupied_cells(std::size_t level) const;
+
     // Cell writes that actually changed a cell, since this grid was
     // constructed. Monotone; only the DIFFERENCE between two reads means
     // anything. Every verb writes through set(), so this is how a caller tells
@@ -1110,6 +1146,10 @@ class VoxelGrid {
     void seed_refined_chunks(std::size_t fine);
     const Chunk* inherited_chunk(std::size_t level, VoxelCoord key, int* out_up) const;
     void ensure_bounds() const;
+    // The occupied cells of one material chunk of a level, stored or
+    // inherited, appended to `out` in the chunk's own z, y, x order.
+    void append_chunk_cells(std::size_t level, VoxelCoord key,
+                            std::vector<OccupiedCell>& out) const;
     // Chunk keys where a level HAS material, which is not the chunks it
     // STORES: an unrefined chunk reads its parent, so the parent's material is
     // at this level too. Every enumeration over a level goes through here.