Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
42 changes: 42 additions & 0 deletions CHANGELOG.md
Original file line number Diff line number Diff line change
Expand Up @@ -8,6 +8,48 @@ All notable changes to this project are documented here.

- Local `.copc` and `.copc.laz` inputs in `usd-pointcloud-convert`, including
deterministic generated-cache publication and reuse.
- `PointCloudPayloadOptions::owner`, `PointCloudPayloadOwner`, and a
layer-level `AuthorPointCloudTiledAssetWithPayloads` overload that reports
typed diagnostics. A tiled FileFormat read publishes its payloads in
`<payloadDirectory>/<owner>/<generation>/`, where the owner is a hash of the
source (relative to the payload directory for local files) and the layer's
exact file-format arguments. The layout is in the
[file-format argument contract](docs/architecture/FILE_FORMAT_ARGUMENTS.md#generated-payload-ownership).

### Changed

- Layer authoring builds content in an in-memory stage that loads nothing, and
payload files are exported from in-memory layers. No authoring call renames
a caller's layer or gives a scratch layer a file identity any more.
- Tiled FileFormat payloads move from `payloadDirectory/Tile_*.usdc` into the
layer's own generation directory below it. Generations are staged privately
and published by rename: identical content reuses the published generation,
changed content is published beside it, and the superseded one is removed.
- The COPC adapter authors native-hierarchy tiles through the shared
layer-level entry point instead of creating and transferring its own stage.
- A tiled LAS, LAZ, or COPC read that cannot author reports the reason
alongside `LAS016`, `LAZ007`, or `COPC007` instead of the bare code. PLY
already did so under `PLY007`.

### Fixed

- Reading a tiled LAS, LAZ, PLY, or COPC layer again, for example in a new
process or after the first layer was released, no longer fails because the
payloads the first read generated already exist.
- Tiled COPC reads no longer warn `Could not open asset` for their own payloads
when the working directory differs from the source directory.
- A generated-cache hit copies its payloads into a generation named by the
entry, reusing an intact copy without writing and replacing a damaged one,
instead of keeping whatever file already sat at the payload path.
- The authoring stream benchmark builds on macOS, where `std::uintmax_t` and
`std::uint64_t` are distinct types.

### Compatibility

- Payload files written by earlier tiled reads directly into
`payloadDirectory` are no longer used or touched; a read publishes a new
generation below them. Delete the old `Tile_*.usdc` files to reclaim the
space.

## [0.11.0] - 2026-08-27

Expand Down
51 changes: 50 additions & 1 deletion docs/architecture/FILE_FORMAT_ARGUMENTS.md
Original file line number Diff line number Diff line change
Expand Up @@ -96,7 +96,7 @@ The tiled surface also exposes:
| `tile` | `true` when tiling is enabled |
| `tileSize` | Positive source-coordinate tile size |
| `tileMemoryLimit` | Positive total spool working-set limit in bytes |
| `payloadDirectory` | Payload output directory |
| `payloadDirectory` | Payload output directory; see [generated payload ownership](#generated-payload-ownership) |

The explicit `usd-pointcloud-convert` tool additionally accepts the generic
point-budget arguments `maxPointsPerTile`, `minPointsPerTile`, and `maxDepth`.
Expand Down Expand Up @@ -139,6 +139,52 @@ descriptor identity. When a committed entry exists for the source, normalized
arguments, and reader metadata, the LAS, LAZ, or COPC adapter loads it before
point delivery; a miss follows the normal reader and authoring path.

## Generated Payload Ownership

A tiled read writes its payloads below `payloadDirectory` and authors a root
that references them by relative path. The root lives only in the opened
layer, so every read of the layer writes the payloads again, by generating
them or by materializing a cache hit. Each layer therefore owns a directory of
its own:

```text
<payloadDirectory>/
<owner>/ one per layer
g-<content hash>/ a generated set
c-<entry key>/ a set copied from a USDGEO_CACHE_ROOT entry
.tmp-<token>/ a set still being written
```

- `<owner>` is a 64-bit hash of the layer's identity: the source and the
layer's file-format arguments exactly as it holds them. A local source is
named by its path relative to `payloadDirectory`, both canonicalized, so a
project that is moved or mounted elsewhere as a whole keeps its payloads;
any other source is named by its resolved identifier. Only the hash is
written, so no source path, resolver identifier, or validation token
appears on disk.
- Layers never share or replace each other's payloads, and files already in
`payloadDirectory` are never touched. Different sources, arguments, or
spellings of arguments simply get different owners.
- A read writes into a private staging directory and publishes it by renaming
it into place, so a published generation is never modified. A read that
fails or is cancelled removes only its staging directory and leaves the
published generation, and every root referencing it, intact.
- Reading the layer again with the same content finds that generation already
published and keeps it, so nothing that has those payloads open sees them
replaced. Changed content is published as a new generation, and the root
references the new paths. Publishing removes the owner's superseded
generations; a process that still shows a superseded generation keeps what
it has loaded but cannot load more of it.
- A staging directory left by an interrupted process is removed by the next
read of the same layer once it has gone an hour without being written.
- A cache hit whose copy is already published under the entry's key reuses it
without writing anything, so a published payload directory may be
read-only. A damaged copy is replaced by a fresh one.

`usd-pointcloud-convert` does not share this layout. Its output root and
payload directory must not exist yet, its payloads are written straight into
that directory, and its transaction marker owns recovery.

## Rules

1. **Validate everything.** An unknown key, an unparsable value, or an
Expand Down Expand Up @@ -189,3 +235,6 @@ should not silently change authored output.
- Conflicting combinations are rejected rather than resolved by precedence.
- A default-valued argument and an absent argument produce identical layers.
- LAS and LAZ accept the same arguments with the same meaning.
- A tiled layer read a second time reuses or supersedes its own published
payloads, never touches another layer's or a user's files, and leaves its
published payloads intact when the read fails.
5 changes: 5 additions & 0 deletions docs/architecture/PLUGIN_ADAPTER.md
Original file line number Diff line number Diff line change
Expand Up @@ -62,6 +62,11 @@ Both plugins now pass reader output through the shared `usdlas` point-data and
asset builders, then call the layer-level `usdgeo::AuthorPointCloudAsset`
entry point. The plugins no longer own point fan-out, chunk schema
construction, CRS or bounds conversion, stage metrics, or layer transfer.
COPC native-hierarchy tiles reach the layer-level
`usdgeo::AuthorPointCloudTiledAssetWithPayloads` entry point the same way, so
no adapter creates a stage or transfers layer content. Every tiled adapter
names its layer as the owner of the payloads it generates through
`usdgeo::PointCloudPayloadOwner`.

### Consequence

Expand Down
2 changes: 1 addition & 1 deletion docs/architecture/WORKSPACE.md
Original file line number Diff line number Diff line change
Expand Up @@ -23,7 +23,7 @@ modules implement is fixed in
| `usdPointCloudCore` | `libs/usd-pointcloud-core` | plain CMake/OpenStrata static library | implemented | Format-independent point-cloud contracts: point attribute definitions and types, `PointChunk` / `PointData` / `PointCloudAsset`, point validation, attribute selection, range and chunk read options, deterministic sampling, and the shared LOD hierarchy and tile value types. |
| `usdLas` | `libs/usd-las` | plain CMake/OpenStrata static library | implemented | LAS parsing and decoding: 1.2-1.4 headers, VLR/EVLR, point formats 0-10, waveform packet metadata, GeoTIFF key records, scalar and vector Extra Bytes, metadata-only reads, and chunked and range-based reads behind `LasReader`. |
| `usdLaz` | `libs/usd-laz` | plain CMake/OpenStrata static library | implemented | LAZ decompression through the vendored laz-perf codec, converted into the shared LAS and point-cloud contracts, with metadata-only reads, chunked reads, and stable diagnostics for unsupported compressed formats. Owns the vendored laz-perf subset. |
| `usdPointCloudAuthoring` | `libs/usd-pointcloud-authoring` | plain CMake/OpenStrata static library | implemented | Shared OpenUSD authoring: `UsdGeomPoints` and geospatial metadata authoring, point-attribute authoring, `usdLod` hierarchy authoring, tiled point-cloud authoring, payload-backed tile assets, root layer generation, and stage/layer validation. |
| `usdPointCloudAuthoring` | `libs/usd-pointcloud-authoring` | plain CMake/OpenStrata static library | implemented | Shared OpenUSD authoring: `UsdGeomPoints` and geospatial metadata authoring, point-attribute authoring, `usdLod` hierarchy authoring, tiled point-cloud authoring, payload-backed tile assets and the ownership of generated payload files, root layer generation, and stage/layer validation. |
| `pointcloud-las` | `plugins/pointcloud-las` | OpenStrata plugin bundle (`usd-fileformat`) | implemented | OpenUSD `SdfFileFormat` adapter for `.las`: plugin registration, argument normalization, `LasReader` construction, and authoring through the shared library. Owns its `LASxxx` diagnostic codes. |
| `pointcloud-laz` | `plugins/pointcloud-laz` | OpenStrata plugin bundle (`usd-fileformat`) | implemented | The same adapter shape for `.laz`, using `LazReader` and the laz-perf codec integration. Owns its `LAZxxx` diagnostic codes. |
| `usdCopc` | `libs/usd-copc` | plain CMake/OpenStrata static library | implemented (source-backed) | COPC Info and hierarchy validation, project-owned random-access source integration, selective point-data range decoding through the shared LAZ chunk decoder, and native hierarchy streaming. It remains OpenUSD- and transport-independent. |
Expand Down
4 changes: 3 additions & 1 deletion docs/compatibility/OPENUSD.md
Original file line number Diff line number Diff line change
Expand Up @@ -49,7 +49,9 @@ the plugin bundles require OpenUSD.
The plugins depend on a small, stable part of the API:

- `SdfFileFormat`, `SdfLayer`, and `SdfFileFormat::FindByExtension("usda")`
- `UsdStage::Open` on an anonymous layer, then `SdfLayer::TransferContent`
- `UsdStage::CreateInMemory` with `UsdStage::LoadNone`, then
`SdfLayer::TransferContent` into the layer being read; payload files are
built in in-memory layers and written with `SdfLayer::Export`
- `UsdGeomPoints`, `UsdGeomSetStageUpAxis`, `UsdGeomSetStageMetersPerUnit`
- `TfType` registration through `TF_REGISTRY_FUNCTION` and
`SDF_DEFINE_FILE_FORMAT`
Expand Down
1 change: 1 addition & 0 deletions docs/reference/CAPABILITY_MATRIX.md
Original file line number Diff line number Diff line change
Expand Up @@ -198,6 +198,7 @@ reported separately, because they differ.
| Bounded-memory generation during file open | Implemented for LAS, LAZ, COPC, and PLY tiled reads; generated-corpus benchmark available | Source streams, spool buffers, and one-tile reconstruction are bounded; generated-corpus and checked-in real-data measurements are available |
| LOD file-format arguments | n/a | `lod=off\|preview\|balanced\|quality` |
| Spatial tile file-format arguments | n/a | `tile=true`, `tileSize`, `tileMemoryLimit`, `payloadDirectory` for LAS, LAZ, COPC, and PLY |
| Reopening a tiled read over its own payloads | Supported when the caller names a payload owner | Supported: each layer publishes immutable payload generations in a directory of its own, reusing identical content; see [payload ownership](../architecture/FILE_FORMAT_ARGUMENTS.md#generated-payload-ownership) |

With `lod=off` every read authors one `UsdGeomPoints` prim at `/PointCloud`.
The other profiles author a single non-tiled `usdLod` root with fixed-stride
Expand Down
29 changes: 29 additions & 0 deletions docs/roadmap/implementation-status.md
Original file line number Diff line number Diff line change
Expand Up @@ -401,6 +401,35 @@ Phase 6 — validation and baselines:
Stable resolver identities publish and reuse generated entries; unstable
and unavailable identities convert without cache publication.

#### Generated-layer ownership

The [ecosystem strategy](../design/ECOSYSTEM_STRATEGY.md) asks that temporary
generated USD be reduced where in-memory layers are practical, and that the
ownership, paths, crash recovery, and test isolation of generated files that
remain be explicit.

- [x] Build layer content in in-memory stages that load nothing, so a read
never resolves the payloads it is generating, and stop renaming caller
or scratch layers to anchor payload paths. Tiled COPC reads no longer
report `Could not open asset` for their own payloads.
- [x] Move COPC native-hierarchy tiled authoring onto a layer-level shared
entry point, so no adapter creates a stage or transfers layer content
- [x] Give every FileFormat-generated payload set a directory owned by its
layer, keyed by the source relative to the payload directory and the
exact file-format arguments. Reading a tiled layer again no longer fails
on its earlier payloads, and never touches another layer's or a user's
files; see
[payload ownership](../architecture/FILE_FORMAT_ARGUMENTS.md#generated-payload-ownership)
- [x] Stage each payload generation privately and publish it by renaming it
into place, reusing identical content, so a failed, cancelled, or
interrupted read never modifies payloads a root already references
- [x] Materialize generated-cache hits as a generation named by the cache
entry, reusing an intact copy without writing and replacing a damaged
one
- [ ] Give tile spools an owned, recoverable working location instead of
timestamped directories in the system temporary directory, which an
interrupted process leaves behind

#### Research - runtime streaming (no release gate)

- [x] Add an opt-in COPC runtime-streaming probe that takes a requested node
Expand Down
12 changes: 11 additions & 1 deletion docs/roadmap/streaming-and-tiling.md
Original file line number Diff line number Diff line change
Expand Up @@ -270,14 +270,24 @@ The payload writer produces:
- root `usdLod` metadata;
- a source fingerprint and the generation settings;
- tile bounds and point counts;
- no partially published root asset after failure.
- no partially published root asset after failure;
- explicit ownership of the payload files, so a FileFormat read can read the
layer again without colliding with its earlier payloads, another layer's, or
a user's files.

The conversion tool writes the versioned tile manifest only after all payloads
have been authored successfully. Entries are sorted by tile ID and LOD, and
portable payload paths, duplicate tile/LOD entries, bounds, and point counts
are validated before publication. Cache materialization preserves the tile
manifest beside the cached payloads.

FileFormat reads publish their payloads in a directory owned by the layer, as
a generation that is staged privately and renamed into place, so a failed or
interrupted read never modifies payloads a root already references; the
layout is in the
[file-format argument contract](../architecture/FILE_FORMAT_ARGUMENTS.md#generated-payload-ownership).
The plugins never write the root layer of a FileFormat read to disk.

Final output is committed atomically where the filesystem allows it. The
authored representation stays inside the existing
[tile and LOD contract](../architecture/LOD.md); streaming changes how assets
Expand Down
1 change: 1 addition & 0 deletions libs/usd-pointcloud-authoring/CMakeLists.txt
Original file line number Diff line number Diff line change
Expand Up @@ -9,6 +9,7 @@ find_package(pxr CONFIG REQUIRED)

add_library(usdPointCloudAuthoring STATIC
src/ArAssetRandomAccessSource.cpp
src/PayloadGeneration.cpp
src/PointCloudCache.cpp
src/PointCloudLayer.cpp
src/PointCloudStreaming.cpp)
Expand Down
33 changes: 31 additions & 2 deletions libs/usd-pointcloud-authoring/README.md
Original file line number Diff line number Diff line change
Expand Up @@ -30,7 +30,8 @@ namespace `usdgeo`.
index.
- Tiled authoring: one deterministic `usdLod` root per tile.
- Payload-backed tile assets: one USDC payload per tile and LOD level, with
portable relative asset paths.
portable relative asset paths, and ownership of the generated payload set
when the caller names an owner.
- Layer- and stage-level validation, and the typed
`PointCloudAuthorFailure` a caller maps onto its own codes.

Expand Down Expand Up @@ -61,7 +62,7 @@ usdgeo/PointCloudCache.h
| Metadata only | `AuthorPointCloudMetadata`, `PointCloudSourceMetadata` |
| LOD | `AuthorPointCloudLodAsset` |
| Tiling | `PointCloudTileAsset`, `AuthorPointCloudTiledAsset` |
| Payloads | `PointCloudPayloadOptions`, `AuthorPointCloudTiledAssetWithPayloads` |
| Payloads | `PointCloudPayloadOptions`, `AuthorPointCloudTiledAssetWithPayloads` (stage and layer overloads), `PointCloudPayloadOwner` |
| Generated cache | `TryBuildPointCloudCacheLayout`, `TryLoadPointCloudCache`, `TryBuildResolverSourceIdentity` |
| Failure kinds | `PointCloudAuthorFailure` (`InvalidLayer`, `StageCreation`, `StageMetrics`, `PointCloud`) |

Expand Down Expand Up @@ -139,6 +140,28 @@ One caller-side note that belongs to the bundles rather than to this module:
thread-local state, so a bundle that authors a detached stage and transfers its
content must do so on the same thread that will hand the result back.

The layer overloads build content in an in-memory stage opened with
`UsdStage::LoadNone` and transfer it into the caller's layer. Nothing is
loaded, so a payload arc is never resolved or opened while its layer is being
built, and no layer is renamed to anchor one: the caller's layer keeps its
identifier throughout, and the detached stage never takes a file identity.
Each payload file is built in an in-memory layer and written with
`SdfLayer::Export`, so writing it never registers a layer under the payload
path.

`PointCloudPayloadOptions::owner` decides where payload files go. Left empty,
the payload directory is exclusive: payloads are written straight into it and
any existing payload path refuses the write, which is what
`usd-pointcloud-convert` relies on. Set, typically to
`PointCloudPayloadOwner(resolvedPath, payloadDirectory, arguments)`, payloads
go to `<directory>/<owner key>/<generation>/`. A generation is staged in a
private directory and published by renaming it into place, so a published one
is never modified: a failure removes only the staging directory, identical
content reuses the published generation, and changed content is published
beside it before the superseded one is removed. Nothing outside the owner's
directory is touched. The layout is stated in the
[file-format argument contract](../../docs/architecture/FILE_FORMAT_ARGUMENTS.md#generated-payload-ownership).

## Coordinate-space assumptions

Input positions are **source-space** `usdgeo::Vec3d`. This module applies
Expand Down Expand Up @@ -179,6 +202,12 @@ ctest --test-dir build/cy2026-windows-x86_64-py313-usd -C Release `
- Sampling is fixed-stride, inherited from `usdPointCloudCore`.
- Payload working-set behavior is unmeasured: the library emits payloads, but
no claim is made that a non-selected LOD child's payload stays unloaded.
- A superseded generation is removed on a best-effort basis. On a platform
that refuses to delete an open file, such as Windows, files another stage
still holds stay until a later read of the same layer removes them.
- Generating payloads needs write access to the payload directory even when
the content turns out unchanged; only a cache hit whose copy is already
published reads a directory without writing to it.
- `AuthorPointCloudTiledAssetFromStream` consumes a pull stream, spools points
by source tile, and reconstructs one tile at a time before payload authoring.
The router overload also accepts a planned `PointBudgetTileRouter`, allowing
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -632,7 +632,7 @@ int main(int argc, char** argv) {
if (entry.lod != 0) continue;
const auto entryPayloadPath = outputDirectory / entry.payloadPath;
std::error_code payloadError;
const auto entryPayloadBytes = std::filesystem::file_size(
const std::uint64_t entryPayloadBytes = std::filesystem::file_size(
entryPayloadPath, payloadError);
if (payloadError) continue;
++tileManifestCount;
Expand Down
Loading
Loading