diff --git a/CHANGELOG.md b/CHANGELOG.md index f927b65..4808f67 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -8,6 +8,48 @@ All notable changes to this project are documented here. - Local `.copc` and `.copc.laz` inputs in `usd-pointcloud-convert`, including deterministic generated-cache publication and reuse. +- `PointCloudPayloadOptions::owner`, `PointCloudPayloadOwner`, and a + layer-level `AuthorPointCloudTiledAssetWithPayloads` overload that reports + typed diagnostics. A tiled FileFormat read publishes its payloads in + `///`, where the owner is a hash of the + source (relative to the payload directory for local files) and the layer's + exact file-format arguments. The layout is in the + [file-format argument contract](docs/architecture/FILE_FORMAT_ARGUMENTS.md#generated-payload-ownership). + +### Changed + +- Layer authoring builds content in an in-memory stage that loads nothing, and + payload files are exported from in-memory layers. No authoring call renames + a caller's layer or gives a scratch layer a file identity any more. +- Tiled FileFormat payloads move from `payloadDirectory/Tile_*.usdc` into the + layer's own generation directory below it. Generations are staged privately + and published by rename: identical content reuses the published generation, + changed content is published beside it, and the superseded one is removed. +- The COPC adapter authors native-hierarchy tiles through the shared + layer-level entry point instead of creating and transferring its own stage. +- A tiled LAS, LAZ, or COPC read that cannot author reports the reason + alongside `LAS016`, `LAZ007`, or `COPC007` instead of the bare code. PLY + already did so under `PLY007`. + +### Fixed + +- Reading a tiled LAS, LAZ, PLY, or COPC layer again, for example in a new + process or after the first layer was released, no longer fails because the + payloads the first read generated already exist. +- Tiled COPC reads no longer warn `Could not open asset` for their own payloads + when the working directory differs from the source directory. +- A generated-cache hit copies its payloads into a generation named by the + entry, reusing an intact copy without writing and replacing a damaged one, + instead of keeping whatever file already sat at the payload path. +- The authoring stream benchmark builds on macOS, where `std::uintmax_t` and + `std::uint64_t` are distinct types. + +### Compatibility + +- Payload files written by earlier tiled reads directly into + `payloadDirectory` are no longer used or touched; a read publishes a new + generation below them. Delete the old `Tile_*.usdc` files to reclaim the + space. ## [0.11.0] - 2026-08-27 diff --git a/docs/architecture/FILE_FORMAT_ARGUMENTS.md b/docs/architecture/FILE_FORMAT_ARGUMENTS.md index 8889b6d..1396dd1 100644 --- a/docs/architecture/FILE_FORMAT_ARGUMENTS.md +++ b/docs/architecture/FILE_FORMAT_ARGUMENTS.md @@ -96,7 +96,7 @@ The tiled surface also exposes: | `tile` | `true` when tiling is enabled | | `tileSize` | Positive source-coordinate tile size | | `tileMemoryLimit` | Positive total spool working-set limit in bytes | -| `payloadDirectory` | Payload output directory | +| `payloadDirectory` | Payload output directory; see [generated payload ownership](#generated-payload-ownership) | The explicit `usd-pointcloud-convert` tool additionally accepts the generic point-budget arguments `maxPointsPerTile`, `minPointsPerTile`, and `maxDepth`. @@ -139,6 +139,52 @@ descriptor identity. When a committed entry exists for the source, normalized arguments, and reader metadata, the LAS, LAZ, or COPC adapter loads it before point delivery; a miss follows the normal reader and authoring path. +## Generated Payload Ownership + +A tiled read writes its payloads below `payloadDirectory` and authors a root +that references them by relative path. The root lives only in the opened +layer, so every read of the layer writes the payloads again, by generating +them or by materializing a cache hit. Each layer therefore owns a directory of +its own: + +```text +/ + / one per layer + g-/ a generated set + c-/ a set copied from a USDGEO_CACHE_ROOT entry + .tmp-/ a set still being written +``` + +- `` is a 64-bit hash of the layer's identity: the source and the + layer's file-format arguments exactly as it holds them. A local source is + named by its path relative to `payloadDirectory`, both canonicalized, so a + project that is moved or mounted elsewhere as a whole keeps its payloads; + any other source is named by its resolved identifier. Only the hash is + written, so no source path, resolver identifier, or validation token + appears on disk. +- Layers never share or replace each other's payloads, and files already in + `payloadDirectory` are never touched. Different sources, arguments, or + spellings of arguments simply get different owners. +- A read writes into a private staging directory and publishes it by renaming + it into place, so a published generation is never modified. A read that + fails or is cancelled removes only its staging directory and leaves the + published generation, and every root referencing it, intact. +- Reading the layer again with the same content finds that generation already + published and keeps it, so nothing that has those payloads open sees them + replaced. Changed content is published as a new generation, and the root + references the new paths. Publishing removes the owner's superseded + generations; a process that still shows a superseded generation keeps what + it has loaded but cannot load more of it. +- A staging directory left by an interrupted process is removed by the next + read of the same layer once it has gone an hour without being written. +- A cache hit whose copy is already published under the entry's key reuses it + without writing anything, so a published payload directory may be + read-only. A damaged copy is replaced by a fresh one. + +`usd-pointcloud-convert` does not share this layout. Its output root and +payload directory must not exist yet, its payloads are written straight into +that directory, and its transaction marker owns recovery. + ## Rules 1. **Validate everything.** An unknown key, an unparsable value, or an @@ -189,3 +235,6 @@ should not silently change authored output. - Conflicting combinations are rejected rather than resolved by precedence. - A default-valued argument and an absent argument produce identical layers. - LAS and LAZ accept the same arguments with the same meaning. +- A tiled layer read a second time reuses or supersedes its own published + payloads, never touches another layer's or a user's files, and leaves its + published payloads intact when the read fails. diff --git a/docs/architecture/PLUGIN_ADAPTER.md b/docs/architecture/PLUGIN_ADAPTER.md index d587ae4..b45710b 100644 --- a/docs/architecture/PLUGIN_ADAPTER.md +++ b/docs/architecture/PLUGIN_ADAPTER.md @@ -62,6 +62,11 @@ Both plugins now pass reader output through the shared `usdlas` point-data and asset builders, then call the layer-level `usdgeo::AuthorPointCloudAsset` entry point. The plugins no longer own point fan-out, chunk schema construction, CRS or bounds conversion, stage metrics, or layer transfer. +COPC native-hierarchy tiles reach the layer-level +`usdgeo::AuthorPointCloudTiledAssetWithPayloads` entry point the same way, so +no adapter creates a stage or transfers layer content. Every tiled adapter +names its layer as the owner of the payloads it generates through +`usdgeo::PointCloudPayloadOwner`. ### Consequence diff --git a/docs/architecture/WORKSPACE.md b/docs/architecture/WORKSPACE.md index 139bd40..570c84c 100644 --- a/docs/architecture/WORKSPACE.md +++ b/docs/architecture/WORKSPACE.md @@ -23,7 +23,7 @@ modules implement is fixed in | `usdPointCloudCore` | `libs/usd-pointcloud-core` | plain CMake/OpenStrata static library | implemented | Format-independent point-cloud contracts: point attribute definitions and types, `PointChunk` / `PointData` / `PointCloudAsset`, point validation, attribute selection, range and chunk read options, deterministic sampling, and the shared LOD hierarchy and tile value types. | | `usdLas` | `libs/usd-las` | plain CMake/OpenStrata static library | implemented | LAS parsing and decoding: 1.2-1.4 headers, VLR/EVLR, point formats 0-10, waveform packet metadata, GeoTIFF key records, scalar and vector Extra Bytes, metadata-only reads, and chunked and range-based reads behind `LasReader`. | | `usdLaz` | `libs/usd-laz` | plain CMake/OpenStrata static library | implemented | LAZ decompression through the vendored laz-perf codec, converted into the shared LAS and point-cloud contracts, with metadata-only reads, chunked reads, and stable diagnostics for unsupported compressed formats. Owns the vendored laz-perf subset. | -| `usdPointCloudAuthoring` | `libs/usd-pointcloud-authoring` | plain CMake/OpenStrata static library | implemented | Shared OpenUSD authoring: `UsdGeomPoints` and geospatial metadata authoring, point-attribute authoring, `usdLod` hierarchy authoring, tiled point-cloud authoring, payload-backed tile assets, root layer generation, and stage/layer validation. | +| `usdPointCloudAuthoring` | `libs/usd-pointcloud-authoring` | plain CMake/OpenStrata static library | implemented | Shared OpenUSD authoring: `UsdGeomPoints` and geospatial metadata authoring, point-attribute authoring, `usdLod` hierarchy authoring, tiled point-cloud authoring, payload-backed tile assets and the ownership of generated payload files, root layer generation, and stage/layer validation. | | `pointcloud-las` | `plugins/pointcloud-las` | OpenStrata plugin bundle (`usd-fileformat`) | implemented | OpenUSD `SdfFileFormat` adapter for `.las`: plugin registration, argument normalization, `LasReader` construction, and authoring through the shared library. Owns its `LASxxx` diagnostic codes. | | `pointcloud-laz` | `plugins/pointcloud-laz` | OpenStrata plugin bundle (`usd-fileformat`) | implemented | The same adapter shape for `.laz`, using `LazReader` and the laz-perf codec integration. Owns its `LAZxxx` diagnostic codes. | | `usdCopc` | `libs/usd-copc` | plain CMake/OpenStrata static library | implemented (source-backed) | COPC Info and hierarchy validation, project-owned random-access source integration, selective point-data range decoding through the shared LAZ chunk decoder, and native hierarchy streaming. It remains OpenUSD- and transport-independent. | diff --git a/docs/compatibility/OPENUSD.md b/docs/compatibility/OPENUSD.md index 8aad9d9..f76f266 100644 --- a/docs/compatibility/OPENUSD.md +++ b/docs/compatibility/OPENUSD.md @@ -49,7 +49,9 @@ the plugin bundles require OpenUSD. The plugins depend on a small, stable part of the API: - `SdfFileFormat`, `SdfLayer`, and `SdfFileFormat::FindByExtension("usda")` -- `UsdStage::Open` on an anonymous layer, then `SdfLayer::TransferContent` +- `UsdStage::CreateInMemory` with `UsdStage::LoadNone`, then + `SdfLayer::TransferContent` into the layer being read; payload files are + built in in-memory layers and written with `SdfLayer::Export` - `UsdGeomPoints`, `UsdGeomSetStageUpAxis`, `UsdGeomSetStageMetersPerUnit` - `TfType` registration through `TF_REGISTRY_FUNCTION` and `SDF_DEFINE_FILE_FORMAT` diff --git a/docs/reference/CAPABILITY_MATRIX.md b/docs/reference/CAPABILITY_MATRIX.md index a02a27f..3df4d72 100644 --- a/docs/reference/CAPABILITY_MATRIX.md +++ b/docs/reference/CAPABILITY_MATRIX.md @@ -198,6 +198,7 @@ reported separately, because they differ. | Bounded-memory generation during file open | Implemented for LAS, LAZ, COPC, and PLY tiled reads; generated-corpus benchmark available | Source streams, spool buffers, and one-tile reconstruction are bounded; generated-corpus and checked-in real-data measurements are available | | LOD file-format arguments | n/a | `lod=off\|preview\|balanced\|quality` | | Spatial tile file-format arguments | n/a | `tile=true`, `tileSize`, `tileMemoryLimit`, `payloadDirectory` for LAS, LAZ, COPC, and PLY | +| Reopening a tiled read over its own payloads | Supported when the caller names a payload owner | Supported: each layer publishes immutable payload generations in a directory of its own, reusing identical content; see [payload ownership](../architecture/FILE_FORMAT_ARGUMENTS.md#generated-payload-ownership) | With `lod=off` every read authors one `UsdGeomPoints` prim at `/PointCloud`. The other profiles author a single non-tiled `usdLod` root with fixed-stride diff --git a/docs/roadmap/implementation-status.md b/docs/roadmap/implementation-status.md index 28a3c7f..abc74f0 100644 --- a/docs/roadmap/implementation-status.md +++ b/docs/roadmap/implementation-status.md @@ -401,6 +401,35 @@ Phase 6 — validation and baselines: Stable resolver identities publish and reuse generated entries; unstable and unavailable identities convert without cache publication. +#### Generated-layer ownership + +The [ecosystem strategy](../design/ECOSYSTEM_STRATEGY.md) asks that temporary +generated USD be reduced where in-memory layers are practical, and that the +ownership, paths, crash recovery, and test isolation of generated files that +remain be explicit. + +- [x] Build layer content in in-memory stages that load nothing, so a read + never resolves the payloads it is generating, and stop renaming caller + or scratch layers to anchor payload paths. Tiled COPC reads no longer + report `Could not open asset` for their own payloads. +- [x] Move COPC native-hierarchy tiled authoring onto a layer-level shared + entry point, so no adapter creates a stage or transfers layer content +- [x] Give every FileFormat-generated payload set a directory owned by its + layer, keyed by the source relative to the payload directory and the + exact file-format arguments. Reading a tiled layer again no longer fails + on its earlier payloads, and never touches another layer's or a user's + files; see + [payload ownership](../architecture/FILE_FORMAT_ARGUMENTS.md#generated-payload-ownership) +- [x] Stage each payload generation privately and publish it by renaming it + into place, reusing identical content, so a failed, cancelled, or + interrupted read never modifies payloads a root already references +- [x] Materialize generated-cache hits as a generation named by the cache + entry, reusing an intact copy without writing and replacing a damaged + one +- [ ] Give tile spools an owned, recoverable working location instead of + timestamped directories in the system temporary directory, which an + interrupted process leaves behind + #### Research - runtime streaming (no release gate) - [x] Add an opt-in COPC runtime-streaming probe that takes a requested node diff --git a/docs/roadmap/streaming-and-tiling.md b/docs/roadmap/streaming-and-tiling.md index 904871e..bbebb2a 100644 --- a/docs/roadmap/streaming-and-tiling.md +++ b/docs/roadmap/streaming-and-tiling.md @@ -270,7 +270,10 @@ The payload writer produces: - root `usdLod` metadata; - a source fingerprint and the generation settings; - tile bounds and point counts; -- no partially published root asset after failure. +- no partially published root asset after failure; +- explicit ownership of the payload files, so a FileFormat read can read the + layer again without colliding with its earlier payloads, another layer's, or + a user's files. The conversion tool writes the versioned tile manifest only after all payloads have been authored successfully. Entries are sorted by tile ID and LOD, and @@ -278,6 +281,13 @@ portable payload paths, duplicate tile/LOD entries, bounds, and point counts are validated before publication. Cache materialization preserves the tile manifest beside the cached payloads. +FileFormat reads publish their payloads in a directory owned by the layer, as +a generation that is staged privately and renamed into place, so a failed or +interrupted read never modifies payloads a root already references; the +layout is in the +[file-format argument contract](../architecture/FILE_FORMAT_ARGUMENTS.md#generated-payload-ownership). +The plugins never write the root layer of a FileFormat read to disk. + Final output is committed atomically where the filesystem allows it. The authored representation stays inside the existing [tile and LOD contract](../architecture/LOD.md); streaming changes how assets diff --git a/libs/usd-pointcloud-authoring/CMakeLists.txt b/libs/usd-pointcloud-authoring/CMakeLists.txt index 4c6fbaf..09e35c9 100644 --- a/libs/usd-pointcloud-authoring/CMakeLists.txt +++ b/libs/usd-pointcloud-authoring/CMakeLists.txt @@ -9,6 +9,7 @@ find_package(pxr CONFIG REQUIRED) add_library(usdPointCloudAuthoring STATIC src/ArAssetRandomAccessSource.cpp + src/PayloadGeneration.cpp src/PointCloudCache.cpp src/PointCloudLayer.cpp src/PointCloudStreaming.cpp) diff --git a/libs/usd-pointcloud-authoring/README.md b/libs/usd-pointcloud-authoring/README.md index 0b143cf..6d03102 100644 --- a/libs/usd-pointcloud-authoring/README.md +++ b/libs/usd-pointcloud-authoring/README.md @@ -30,7 +30,8 @@ namespace `usdgeo`. index. - Tiled authoring: one deterministic `usdLod` root per tile. - Payload-backed tile assets: one USDC payload per tile and LOD level, with - portable relative asset paths. + portable relative asset paths, and ownership of the generated payload set + when the caller names an owner. - Layer- and stage-level validation, and the typed `PointCloudAuthorFailure` a caller maps onto its own codes. @@ -61,7 +62,7 @@ usdgeo/PointCloudCache.h | Metadata only | `AuthorPointCloudMetadata`, `PointCloudSourceMetadata` | | LOD | `AuthorPointCloudLodAsset` | | Tiling | `PointCloudTileAsset`, `AuthorPointCloudTiledAsset` | -| Payloads | `PointCloudPayloadOptions`, `AuthorPointCloudTiledAssetWithPayloads` | +| Payloads | `PointCloudPayloadOptions`, `AuthorPointCloudTiledAssetWithPayloads` (stage and layer overloads), `PointCloudPayloadOwner` | | Generated cache | `TryBuildPointCloudCacheLayout`, `TryLoadPointCloudCache`, `TryBuildResolverSourceIdentity` | | Failure kinds | `PointCloudAuthorFailure` (`InvalidLayer`, `StageCreation`, `StageMetrics`, `PointCloud`) | @@ -139,6 +140,28 @@ One caller-side note that belongs to the bundles rather than to this module: thread-local state, so a bundle that authors a detached stage and transfers its content must do so on the same thread that will hand the result back. +The layer overloads build content in an in-memory stage opened with +`UsdStage::LoadNone` and transfer it into the caller's layer. Nothing is +loaded, so a payload arc is never resolved or opened while its layer is being +built, and no layer is renamed to anchor one: the caller's layer keeps its +identifier throughout, and the detached stage never takes a file identity. +Each payload file is built in an in-memory layer and written with +`SdfLayer::Export`, so writing it never registers a layer under the payload +path. + +`PointCloudPayloadOptions::owner` decides where payload files go. Left empty, +the payload directory is exclusive: payloads are written straight into it and +any existing payload path refuses the write, which is what +`usd-pointcloud-convert` relies on. Set, typically to +`PointCloudPayloadOwner(resolvedPath, payloadDirectory, arguments)`, payloads +go to `///`. A generation is staged in a +private directory and published by renaming it into place, so a published one +is never modified: a failure removes only the staging directory, identical +content reuses the published generation, and changed content is published +beside it before the superseded one is removed. Nothing outside the owner's +directory is touched. The layout is stated in the +[file-format argument contract](../../docs/architecture/FILE_FORMAT_ARGUMENTS.md#generated-payload-ownership). + ## Coordinate-space assumptions Input positions are **source-space** `usdgeo::Vec3d`. This module applies @@ -179,6 +202,12 @@ ctest --test-dir build/cy2026-windows-x86_64-py313-usd -C Release ` - Sampling is fixed-stride, inherited from `usdPointCloudCore`. - Payload working-set behavior is unmeasured: the library emits payloads, but no claim is made that a non-selected LOD child's payload stays unloaded. +- A superseded generation is removed on a best-effort basis. On a platform + that refuses to delete an open file, such as Windows, files another stage + still holds stay until a later read of the same layer removes them. +- Generating payloads needs write access to the payload directory even when + the content turns out unchanged; only a cache hit whose copy is already + published reads a directory without writing to it. - `AuthorPointCloudTiledAssetFromStream` consumes a pull stream, spools points by source tile, and reconstructs one tile at a time before payload authoring. The router overload also accepts a planned `PointBudgetTileRouter`, allowing diff --git a/libs/usd-pointcloud-authoring/benchmarks/stream_benchmark.cpp b/libs/usd-pointcloud-authoring/benchmarks/stream_benchmark.cpp index cb2b4dc..87fb6c0 100644 --- a/libs/usd-pointcloud-authoring/benchmarks/stream_benchmark.cpp +++ b/libs/usd-pointcloud-authoring/benchmarks/stream_benchmark.cpp @@ -632,7 +632,7 @@ int main(int argc, char** argv) { if (entry.lod != 0) continue; const auto entryPayloadPath = outputDirectory / entry.payloadPath; std::error_code payloadError; - const auto entryPayloadBytes = std::filesystem::file_size( + const std::uint64_t entryPayloadBytes = std::filesystem::file_size( entryPayloadPath, payloadError); if (payloadError) continue; ++tileManifestCount; diff --git a/libs/usd-pointcloud-authoring/include/usdgeo/PointCloudLayer.h b/libs/usd-pointcloud-authoring/include/usdgeo/PointCloudLayer.h index e146111..ad1f053 100644 --- a/libs/usd-pointcloud-authoring/include/usdgeo/PointCloudLayer.h +++ b/libs/usd-pointcloud-authoring/include/usdgeo/PointCloudLayer.h @@ -131,8 +131,27 @@ struct PointCloudPayloadOptions { std::vector* tileManifestEntries = nullptr; usdpointcloud::SpoolIoStats* spoolIoStats = nullptr; + // Identity of the layer the generated payloads belong to. When empty the + // payload directory is exclusive: payloads are written straight into it + // and any existing payload path refuses the write. When set, payloads + // live in `directory///`, where the owner key is a + // hash of this value: each generation is staged privately and published + // whole, identical content reuses the published generation, and + // superseded generations of the same owner are removed. Nothing outside + // the owner's directory is touched, and the value is never persisted. + std::string owner; }; +// The payload owner for a FileFormat read: which source the layer reads and +// the layer's file-format arguments exactly as it holds them. A local source +// is named relative to the payload directory, so a project that moves as a +// whole keeps its owner; any other source is named by its resolved +// identifier. +std::string PointCloudPayloadOwner( + const std::string& source, + const std::filesystem::path& payloadDirectory, + const pxr::SdfLayer::FileFormatArguments& arguments); + bool AuthorPointCloudTiledAssetFromStream( pxr::SdfLayer* layer, const std::string& primPath, @@ -174,4 +193,14 @@ bool AuthorPointCloudTiledAssetWithPayloads( const PointCloudPayloadOptions& options, std::vector& generatedPayloads); +// Authors the payload-backed tiles into a detached stage and transfers the +// result into `layer`. On failure the layer is unchanged, nothing this call +// wrote remains, and `diagnostics` explains why. +bool AuthorPointCloudTiledAssetWithPayloads( + pxr::SdfLayer* layer, + const std::string& primPath, + const std::vector& tiles, + const PointCloudPayloadOptions& options, + std::vector& diagnostics); + } // namespace usdgeo \ No newline at end of file diff --git a/libs/usd-pointcloud-authoring/src/PayloadGeneration.cpp b/libs/usd-pointcloud-authoring/src/PayloadGeneration.cpp new file mode 100644 index 0000000..722ac4b --- /dev/null +++ b/libs/usd-pointcloud-authoring/src/PayloadGeneration.cpp @@ -0,0 +1,422 @@ +#include "PayloadGeneration.h" + +#include "usdgeo/CacheKey.h" + +#include +#include +#include +#include +#include +#include +#include +#include +#include + +namespace usdgeo::detail { +namespace { + +constexpr const char* kStagingPrefix = ".tmp-"; +constexpr const char* kGeneratedPrefix = "g-"; +constexpr const char* kCachedPrefix = "c-"; + +// A live generation touches its staging directory with every payload it +// writes, so one left untouched this long belongs to an interrupted process. +constexpr auto kStaleStagingAge = std::chrono::hours(1); + +bool StartsWith(const std::string& value, const char* prefix) { + return value.rfind(prefix, 0) == 0; +} + +// Payload names are joined onto the directory the generation owns, so a name +// that could leave it is rejected rather than trusted. +bool IsSafeRelativeName(const std::string& value) { + if (value.empty() || value.front() == '/' || + value.find_first_of("\\:\r\n") != std::string::npos) { + return false; + } + std::size_t start = 0; + for (;;) { + const auto separator = value.find('/', start); + const auto end = + separator == std::string::npos ? value.size() : separator; + const auto component = value.substr(start, end - start); + if (component.empty() || component == "." || component == "..") { + return false; + } + if (separator == std::string::npos) { + return true; + } + start = separator + 1; + } +} + +bool IsGenerationName(const std::string& value) { + return (StartsWith(value, kGeneratedPrefix) || + StartsWith(value, kCachedPrefix)) && + value.find('/') == std::string::npos; +} + +std::string Hex(std::uint64_t value) { + std::ostringstream text; + text << std::hex << std::setfill('0') << std::setw(16) << value; + return text.str(); +} + +std::string NewStagingName() { + static std::atomic_uint64_t sequence{0}; + std::random_device device; + const auto clock = static_cast( + std::chrono::steady_clock::now().time_since_epoch().count()); + const auto value = (static_cast(device()) << 32) ^ + device() ^ clock ^ + (++sequence * 0x9e3779b97f4a7c15ull); + return kStagingPrefix + Hex(value); +} + +void RemoveStaleStaging(const std::filesystem::path& ownerDirectory) { + std::vector stale; + const auto now = std::filesystem::file_time_type::clock::now(); + std::error_code error; + for (std::filesystem::directory_iterator entry(ownerDirectory, error), end; + !error && entry != end; entry.increment(error)) { + std::error_code status; + if (!StartsWith(entry->path().filename().string(), kStagingPrefix) || + !entry->is_directory(status) || status) { + continue; + } + const auto modified = + std::filesystem::last_write_time(entry->path(), status); + if (!status && now - modified > kStaleStagingAge) { + stale.push_back(entry->path()); + } + } + for (const auto& path : stale) { + std::error_code removal; + std::filesystem::remove_all(path, removal); + } +} + +// Names the published generation by what it contains, so regenerating the +// same payloads finds the directory already published. +bool HashPayloads(const std::filesystem::path& directory, + const std::set& names, + std::string& hash, + std::string& error) { + constexpr std::uint64_t offsetBasis = 14695981039346656037ull; + constexpr std::uint64_t prime = 1099511628211ull; + std::uint64_t value = offsetBasis; + const auto mix = [&value](const char* data, std::size_t size) { + for (std::size_t index = 0; index < size; ++index) { + value ^= static_cast(data[index]); + value *= prime; + } + }; + std::array buffer{}; + for (const auto& name : names) { + mix(name.c_str(), name.size() + 1); + std::ifstream input(directory / std::filesystem::path(name), + std::ios::binary); + if (!input) { + error = "unable to read generated payload " + name; + return false; + } + std::uint64_t size = 0; + while (input) { + input.read(buffer.data(), + static_cast(buffer.size())); + const auto count = static_cast(input.gcount()); + mix(buffer.data(), count); + size += count; + } + if (!input.eof()) { + error = "unable to read generated payload " + name; + return false; + } + const auto sizeText = std::to_string(size); + mix(sizeText.c_str(), sizeText.size() + 1); + } + hash = Hex(value); + return true; +} + +} // namespace + +std::string PayloadOwnerKey(const std::string& owner) { + return StableCacheKey({{"payloadOwner", owner}}); +} + +std::string CachedGenerationName(const std::string& entryKey) { + return kCachedPrefix + entryKey; +} + +std::filesystem::path PayloadGeneration::OwnerDirectory( + const std::filesystem::path& directory, const std::string& owner) { + return directory / PayloadOwnerKey(owner); +} + +PayloadGeneration::PayloadGeneration(std::filesystem::path directory, + std::string owner) + : directory_(directory.lexically_normal()), owner_(std::move(owner)) { + // A trailing separator would leave an empty final element that makes the + // payload paths below it ambiguous. + if (!directory_.has_filename() && directory_.has_relative_path()) { + directory_ = directory_.parent_path(); + } + if (!owner_.empty()) { + ownerDirectory_ = OwnerDirectory(directory_, owner_); + } +} + +PayloadGeneration::~PayloadGeneration() { + Rollback(); +} + +bool PayloadGeneration::Begin(const std::vector& names, + std::string& error) { + if (active_) { + error = "payload generation has already begun"; + return false; + } + std::set declared; + for (const auto& name : names) { + if (!IsSafeRelativeName(name) || !declared.insert(name).second) { + error = "payload names must be unique relative paths"; + return false; + } + } + + std::error_code status; + const auto directoryExists = std::filesystem::exists(directory_, status); + if (!status && directoryExists && + !std::filesystem::is_directory(directory_, status)) { + status = std::make_error_code(std::errc::not_a_directory); + } + if (status) { + error = "payload directory is not a usable directory"; + return false; + } + if (owner_.empty() && directoryExists) { + for (const auto& name : declared) { + const auto present = std::filesystem::exists( + directory_ / std::filesystem::path(name), status); + if (status) { + error = "unable to inspect payload " + name; + return false; + } + if (present) { + error = "payload directory already contains " + name; + return false; + } + } + } + + names_ = std::move(declared); + active_ = true; + if (!directoryExists) { + std::filesystem::create_directories(directory_, status); + if (status) { + error = "unable to create payload directory"; + Rollback(); + return false; + } + directoryCreated_ = true; + } + if (owner_.empty()) { + return true; + } + + const auto ownerExists = std::filesystem::exists(ownerDirectory_, status); + if (status) { + error = "unable to inspect the payload owner directory"; + Rollback(); + return false; + } + if (ownerExists) { + RemoveStaleStaging(ownerDirectory_); + } else { + std::filesystem::create_directory(ownerDirectory_, status); + if (status) { + error = "unable to create the payload owner directory"; + Rollback(); + return false; + } + ownerDirectoryCreated_ = true; + } + for (int attempt = 0; attempt < 16 && stagingDirectory_.empty(); + ++attempt) { + const auto name = NewStagingName(); + if (std::filesystem::create_directory(ownerDirectory_ / name, status)) { + stagingName_ = name; + stagingDirectory_ = ownerDirectory_ / name; + } + } + if (stagingDirectory_.empty()) { + error = "unable to create a payload staging directory"; + Rollback(); + return false; + } + return true; +} + +std::filesystem::path PayloadGeneration::PathFor(const std::string& name) const { + if (!active_ || names_.count(name) == 0) { + return {}; + } + const auto& base = owner_.empty() ? directory_ : stagingDirectory_; + return base / std::filesystem::path(name); +} + +std::filesystem::path PayloadGeneration::PublishedPathFor( + const std::string& name) const { + if (owner_.empty()) { + return directory_ / std::filesystem::path(name); + } + if (publishedName_.empty()) { + return {}; + } + return ownerDirectory_ / publishedName_ / std::filesystem::path(name); +} + +bool PayloadGeneration::Commit(std::string& error) { + if (!active_) { + error = "payload generation has not begun"; + return false; + } + if (owner_.empty()) { + active_ = false; + directoryCreated_ = false; + return true; + } + for (const auto& name : names_) { + std::error_code status; + if (!std::filesystem::is_regular_file( + stagingDirectory_ / std::filesystem::path(name), status)) { + error = "generated payload " + name + " was not written"; + return false; + } + } + std::string hash; + if (!HashPayloads(stagingDirectory_, names_, hash, error)) { + return false; + } + return Publish(kGeneratedPrefix + hash, error); +} + +bool PayloadGeneration::CommitAs(const std::string& name, std::string& error) { + if (!active_ || owner_.empty()) { + error = "only an owned payload generation can be published by name"; + return false; + } + if (!IsGenerationName(name) || !IsSafeRelativeName(name)) { + error = "invalid payload generation name"; + return false; + } + return Publish(name, error); +} + +bool PayloadGeneration::IsComplete( + const std::filesystem::path& generation) const { + for (const auto& name : names_) { + std::error_code status; + const auto published = generation / std::filesystem::path(name); + if (!std::filesystem::is_regular_file(published, status) || status) { + return false; + } + const auto publishedSize = std::filesystem::file_size(published, status); + if (status) { + return false; + } + const auto stagedSize = std::filesystem::file_size( + stagingDirectory_ / std::filesystem::path(name), status); + if (status || stagedSize != publishedSize) { + return false; + } + } + return true; +} + +bool PayloadGeneration::Publish(const std::string& name, std::string& error) { + const auto published = ownerDirectory_ / name; + std::error_code status; + if (std::filesystem::exists(published, status) && !IsComplete(published)) { + // A damaged generation is replaced rather than trusted. + std::filesystem::remove_all(published, status); + if (status) { + error = "unable to replace incomplete payload generation " + name; + return false; + } + } + if (std::filesystem::exists(published, status)) { + // The same payloads are already published; keep them in place so + // nothing that has them open sees them replaced. + std::filesystem::remove_all(stagingDirectory_, status); + } else { + std::filesystem::rename(stagingDirectory_, published, status); + if (status) { + // A concurrent generation of the same payloads may have + // published first. + std::error_code check; + if (!std::filesystem::exists(published, check) || + !IsComplete(published)) { + error = "unable to publish payload generation " + name; + return false; + } + std::filesystem::remove_all(stagingDirectory_, check); + } + } + publishedName_ = name; + stagingDirectory_.clear(); + active_ = false; + directoryCreated_ = false; + ownerDirectoryCreated_ = false; + + std::vector superseded; + std::error_code iteration; + for (std::filesystem::directory_iterator entry(ownerDirectory_, iteration), + end; + !iteration && entry != end; entry.increment(iteration)) { + const auto entryName = entry->path().filename().string(); + std::error_code entryStatus; + if (entryName != name && IsGenerationName(entryName) && + entry->is_directory(entryStatus) && !entryStatus) { + superseded.push_back(entry->path()); + } + } + for (const auto& path : superseded) { + // Best effort: a file another process still holds open stays until a + // later generation removes it. + std::error_code removal; + std::filesystem::remove_all(path, removal); + } + return true; +} + +void PayloadGeneration::Rollback() { + if (!active_) { + return; + } + active_ = false; + std::error_code status; + if (owner_.empty()) { + // Begin proved none of these existed, so any that do now were + // written by this generation. + for (const auto& name : names_) { + std::filesystem::remove(directory_ / std::filesystem::path(name), + status); + } + } else if (!stagingDirectory_.empty()) { + std::filesystem::remove_all(stagingDirectory_, status); + stagingDirectory_.clear(); + } + // Removing a directory succeeds only while it is empty. + if (ownerDirectoryCreated_) { + std::filesystem::remove(ownerDirectory_, status); + ownerDirectoryCreated_ = false; + } + if (directoryCreated_) { + std::filesystem::remove(directory_, status); + directoryCreated_ = false; + } +} + +} // namespace usdgeo::detail diff --git a/libs/usd-pointcloud-authoring/src/PayloadGeneration.h b/libs/usd-pointcloud-authoring/src/PayloadGeneration.h new file mode 100644 index 0000000..0332c7a --- /dev/null +++ b/libs/usd-pointcloud-authoring/src/PayloadGeneration.h @@ -0,0 +1,84 @@ +#pragma once + +#include +#include +#include +#include + +namespace usdgeo::detail { + +// The directory name under which one owner's payload generations live. +std::string PayloadOwnerKey(const std::string& owner); + +// The generation name for payloads copied out of a generated-cache entry, +// which the entry's key already identifies. +std::string CachedGenerationName(const std::string& entryKey); + +// One set of payload files written into a payload directory. +// +// Without an owner the directory is exclusive: payloads are written straight +// into it, and any payload name that already exists refuses the generation. +// +// With an owner, payloads live in `///`. +// They are written into a private staging directory and published at Commit +// by renaming it into place, so an existing generation is never modified: a +// failed or cancelled generation leaves the published one untouched, and a +// generation with identical content reuses the published directory instead +// of replacing it. Publishing removes the owner's superseded generations. +// Nothing outside the owner's directory is ever written or removed. +class PayloadGeneration final { +public: + PayloadGeneration(std::filesystem::path directory, std::string owner); + ~PayloadGeneration(); + + PayloadGeneration(const PayloadGeneration&) = delete; + PayloadGeneration& operator=(const PayloadGeneration&) = delete; + + // Declares the payload names this generation writes, relative to where it + // publishes them, and prepares the directory they are written into. + bool Begin(const std::vector& names, std::string& error); + + // Where a declared payload is written before Commit. Empty for a name + // that was not declared. + std::filesystem::path PathFor(const std::string& name) const; + + // Publishes the generation under a name derived from its content. + bool Commit(std::string& error); + + // Publishes the generation under `name`, for content whose identity is + // already known. + bool CommitAs(const std::string& name, std::string& error); + + // Removes what this generation wrote, and the directories it created when + // they are left empty. Does nothing after a commit. + void Rollback(); + + // The staging directory name that payload paths contain until Commit, and + // the generation name that replaces it. Both are empty without an owner. + const std::string& StagingName() const noexcept { return stagingName_; } + const std::string& PublishedName() const noexcept { return publishedName_; } + + // Where a declared payload is after Commit. + std::filesystem::path PublishedPathFor(const std::string& name) const; + + // The directory holding the owner's generations. + static std::filesystem::path OwnerDirectory( + const std::filesystem::path& directory, const std::string& owner); + +private: + bool Publish(const std::string& name, std::string& error); + bool IsComplete(const std::filesystem::path& generation) const; + + std::filesystem::path directory_; + std::string owner_; + std::filesystem::path ownerDirectory_; + std::filesystem::path stagingDirectory_; + std::string stagingName_; + std::string publishedName_; + std::set names_; + bool directoryCreated_ = false; + bool ownerDirectoryCreated_ = false; + bool active_ = false; +}; + +} // namespace usdgeo::detail diff --git a/libs/usd-pointcloud-authoring/src/PointCloudCache.cpp b/libs/usd-pointcloud-authoring/src/PointCloudCache.cpp index 71bfef8..2bc1e41 100644 --- a/libs/usd-pointcloud-authoring/src/PointCloudCache.cpp +++ b/libs/usd-pointcloud-authoring/src/PointCloudCache.cpp @@ -1,13 +1,17 @@ #include "usdgeo/PointCloudCache.h" +#include "usdgeo/CacheKey.h" +#include "usdgeo/PointCloudLayer.h" #include "usdpointcloud/Tiling.h" +#include "PayloadGeneration.h" + #include #include #include #include #include -#include +#include #include #include @@ -162,14 +166,42 @@ bool ValidateCachedPayloads( return valid; } +// A copy published under an entry's key holds that entry's bytes, so it is +// reused as long as every payload is still there at its cached size. +bool HoldsCachedCopy( + const std::filesystem::path& generation, + const std::map& files) { + for (const auto& [name, sourcePath] : files) { + std::error_code status; + const auto target = generation / std::filesystem::path(name); + if (!std::filesystem::is_regular_file(target, status) || status) { + return false; + } + const auto targetSize = std::filesystem::file_size(target, status); + if (status) { + return false; + } + const auto sourceSize = std::filesystem::file_size(sourcePath, status); + if (status || targetSize != sourceSize) { + return false; + } + } + return true; +} + +// Copies the cached payloads into the owner's directory under a generation +// named by the cache entry. An intact copy from an earlier hit is reused +// without writing anything, and a new copy is staged and published whole, +// so no payload another stage has open is ever modified. bool MaterializePayloads( const pxr::SdfLayerHandle& layer, const usdgeo::cache::Layout& layout, - const std::filesystem::path& targetDirectory) { + const std::filesystem::path& targetDirectory, + const std::string& owner, + std::filesystem::path& materializedDirectory) { const auto sourceDirectory = layout.payloadDirectory.lexically_normal(); - const auto normalizedTarget = targetDirectory.lexically_normal(); - std::vector> files; + std::map files; bool valid = true; layer->Traverse(pxr::SdfPath::AbsoluteRootPath(), [&](const pxr::SdfPath& path) { @@ -188,20 +220,15 @@ bool MaterializePayloads( const std::filesystem::path sourcePath = (layout.entryDirectory / assetPath).lexically_normal(); if (std::filesystem::path(assetPath).is_absolute() || - !IsWithin(sourcePath, sourceDirectory)) { - valid = false; - return; - } - const auto relative = - sourcePath.lexically_relative(sourceDirectory); - const auto targetPath = - (normalizedTarget / relative).lexically_normal(); - if (!IsValidCachedPayloadPath(sourcePath, - sourceDirectory)) { + !IsWithin(sourcePath, sourceDirectory) || + !IsValidCachedPayloadPath(sourcePath, sourceDirectory)) { valid = false; return; } - files.emplace_back(sourcePath, targetPath); + files.emplace( + sourcePath.lexically_relative(sourceDirectory) + .generic_string(), + sourcePath); } }; collect(payloads.GetExplicitItems()); @@ -213,37 +240,43 @@ bool MaterializePayloads( return false; } - std::set createdFiles; - std::error_code error; - for (const auto& [sourcePath, targetPath] : files) { - static_cast(sourcePath); - const auto targetExists = std::filesystem::exists(targetPath, error); - if (error || (targetExists && - !std::filesystem::is_regular_file(targetPath, error)) || - error) { - valid = false; - break; - } - if (targetExists) { - continue; - } - std::filesystem::create_directories(targetPath.parent_path(), error); - if (error || !std::filesystem::copy_file( - sourcePath, targetPath, - std::filesystem::copy_options::none, error) || - error) { - valid = false; - break; - } - createdFiles.insert(targetPath); + const auto generationName = detail::CachedGenerationName(StableCacheKey( + {{"cacheEntry", + layout.entryDirectory.parent_path().filename().generic_string() + + "/" + layout.entryDirectory.filename().generic_string()}})); + const auto published = + detail::PayloadGeneration::OwnerDirectory( + targetDirectory.lexically_normal(), owner) / + generationName; + if (HoldsCachedCopy(published, files)) { + materializedDirectory = published; + return true; } - if (!valid) { - for (const auto& path : createdFiles) { - std::filesystem::remove(path, error); - error.clear(); + + std::vector names; + names.reserve(files.size()); + for (const auto& entry : files) { + names.push_back(entry.first); + } + detail::PayloadGeneration generation(targetDirectory, owner); + std::string error; + if (!generation.Begin(names, error)) { + return false; + } + for (const auto& [name, sourcePath] : files) { + const auto target = generation.PathFor(name); + std::error_code status; + std::filesystem::create_directories(target.parent_path(), status); + if (status || !std::filesystem::copy_file(sourcePath, target, status) || + status) { + return false; } } - return valid; + if (!generation.CommitAs(generationName, error)) { + return false; + } + materializedDirectory = published; + return true; } void RebasePayloads(pxr::SdfLayer* layer, @@ -345,38 +378,15 @@ bool TryBuildPointCloudCacheLayout( return true; } -bool TryLoadPointCloudCache( - pxr::SdfLayer* layer, - const std::filesystem::path& sourcePath, - const GeoReference& reference, - const usdpointcloud::PointReadRequest& request, - const std::string& parserVersion, - bool& hit, - std::string& errorMessage, - cache::CacheDecision* decision) { - hit = false; - if (!layer) { - errorMessage = "cache lookup requires a writable layer"; - return false; - } - if (PointCloudCacheRootFromEnvironment().empty()) { - return true; - } - - usdgeo::cache::SourceIdentity sourceIdentity; - if (!usdgeo::cache::TryBuildLocalSourceIdentity( - sourcePath, sourceIdentity, errorMessage)) { - return false; - } - return TryLoadPointCloudCache( - layer, sourceIdentity, sourcePath.parent_path(), reference, request, - parserVersion, hit, errorMessage, decision); -} +namespace { -bool TryLoadPointCloudCache( +// `ownerSource` names the source the way the FileFormat read that generates +// payloads does, so materialized payloads land in that layer's own directory. +bool LoadPointCloudCache( pxr::SdfLayer* layer, const cache::SourceIdentity& sourceIdentity, const std::filesystem::path& payloadBaseDirectory, + const std::string& ownerSource, const GeoReference& reference, const usdpointcloud::PointReadRequest& request, const std::string& parserVersion, @@ -446,10 +456,16 @@ bool TryLoadPointCloudCache( std::error_code error; targetPayloadDirectory = std::filesystem::absolute(targetPayloadDirectory, error); - if (error || !MaterializePayloads( - cachedLayer, layout, targetPayloadDirectory)) { + std::filesystem::path materializedDirectory; + if (error || + !MaterializePayloads( + cachedLayer, layout, targetPayloadDirectory, + PointCloudPayloadOwner(ownerSource, targetPayloadDirectory, + layer->GetFileFormatArguments()), + materializedDirectory)) { return true; } + targetPayloadDirectory = materializedDirectory; } layer->TransferContent(cachedLayer); const auto layerBaseDirectory = @@ -464,4 +480,51 @@ bool TryLoadPointCloudCache( return true; } +} // namespace + +bool TryLoadPointCloudCache( + pxr::SdfLayer* layer, + const std::filesystem::path& sourcePath, + const GeoReference& reference, + const usdpointcloud::PointReadRequest& request, + const std::string& parserVersion, + bool& hit, + std::string& errorMessage, + cache::CacheDecision* decision) { + hit = false; + if (!layer) { + errorMessage = "cache lookup requires a writable layer"; + return false; + } + if (PointCloudCacheRootFromEnvironment().empty()) { + return true; + } + + usdgeo::cache::SourceIdentity sourceIdentity; + if (!usdgeo::cache::TryBuildLocalSourceIdentity( + sourcePath, sourceIdentity, errorMessage)) { + return false; + } + return LoadPointCloudCache( + layer, sourceIdentity, sourcePath.parent_path(), + sourcePath.string(), reference, + request, parserVersion, hit, errorMessage, decision); +} + +bool TryLoadPointCloudCache( + pxr::SdfLayer* layer, + const cache::SourceIdentity& sourceIdentity, + const std::filesystem::path& payloadBaseDirectory, + const GeoReference& reference, + const usdpointcloud::PointReadRequest& request, + const std::string& parserVersion, + bool& hit, + std::string& errorMessage, + cache::CacheDecision* decision) { + return LoadPointCloudCache( + layer, sourceIdentity, payloadBaseDirectory, + sourceIdentity.identifier, reference, + request, parserVersion, hit, errorMessage, decision); +} + } // namespace usdgeo diff --git a/libs/usd-pointcloud-authoring/src/PointCloudLayer.cpp b/libs/usd-pointcloud-authoring/src/PointCloudLayer.cpp index f7c3055..092bea0 100644 --- a/libs/usd-pointcloud-authoring/src/PointCloudLayer.cpp +++ b/libs/usd-pointcloud-authoring/src/PointCloudLayer.cpp @@ -1,7 +1,8 @@ #include "usdgeo/PointCloudLayer.h" +#include "TiledPayloadAuthoring.h" + #include -#include #include #include #include @@ -31,37 +32,6 @@ bool IsValidPrimPath(const std::string& primPath) { return !primPath.empty() && primPath.front() == '/'; } -class ScopedLayerIdentifier final { -public: - ScopedLayerIdentifier(const pxr::SdfLayerHandle& layer, - const std::string& identifier) - : layer_(layer), - originalIdentifier_(layer ? layer->GetIdentifier() : std::string()), - changed_(layer_ && !layer_->IsAnonymous()) { - if (changed_) { - const auto temporaryIdentifier = - identifier + ".usdgeo-authoring-" + - std::to_string(++sequence_); - layer_->SetIdentifier(temporaryIdentifier); - } - } - - ~ScopedLayerIdentifier() { - if (changed_) { - layer_->SetIdentifier(originalIdentifier_); - } - } - - ScopedLayerIdentifier(const ScopedLayerIdentifier&) = delete; - ScopedLayerIdentifier& operator=(const ScopedLayerIdentifier&) = delete; - -private: - pxr::SdfLayerHandle layer_; - std::string originalIdentifier_; - bool changed_ = false; - inline static std::atomic_uint64_t sequence_{0}; -}; - bool SameBounds(const usdgeo::SpatialBounds& left, const usdgeo::SpatialBounds& right) { return left.minimum.x == right.minimum.x && @@ -144,14 +114,21 @@ std::string TilePrimName(const usdpointcloud::PointTileId& id) { "_" + axisName(id.y) + "_" + axisName(id.z); } +// Which tile a payload-backed LOD root belongs to, and the generation that +// decides where its payload files are written. +struct PayloadTarget { + const usdpointcloud::PointTileId& tileId; + const std::filesystem::path& rootLayerPath; + const detail::PayloadGeneration& payloads; +}; + bool AuthorLodRoot( const pxr::UsdStageRefPtr& stage, const std::string& primPath, const std::vector& levels, const usdpointcloud::PointLodHierarchy& hierarchy, const pxr::SdfPath& heuristicPath, - const std::filesystem::path* payloadDirectory = nullptr, - const std::filesystem::path* rootLayerPath = nullptr) { + const PayloadTarget* payloadTarget = nullptr) { if (!stage || !IsValidPrimPath(primPath) || levels.size() != hierarchy.items.size()) { return false; @@ -174,7 +151,7 @@ bool AuthorLodRoot( const auto& level = levels[index]; const auto lodName = "LOD" + std::to_string(index); const auto lodPath = primPath + "/" + lodName; - if (payloadDirectory == nullptr) { + if (payloadTarget == nullptr) { if (!AuthorPointCloudAsset(stage, lodPath, level.reference, level.bounds, level.chunk, level.data)) { return false; @@ -182,23 +159,29 @@ bool AuthorLodRoot( continue; } - const auto payloadPath = *payloadDirectory / - (primPath.substr(primPath.find_last_of('/') + 1) + "_" + - lodName + ".usdc"); + const auto payloadPath = payloadTarget->payloads.PathFor( + detail::TilePayloadName(payloadTarget->tileId, index)); + if (payloadPath.empty()) { + return false; + } std::error_code relativeError; const auto payloadIdentifier = std::filesystem::relative( - payloadPath, rootLayerPath->parent_path(), relativeError); + payloadPath, payloadTarget->rootLayerPath.parent_path(), + relativeError); if (relativeError || payloadIdentifier.empty() || payloadIdentifier.is_absolute()) { return false; } - const auto payloadStage = pxr::UsdStage::CreateNew(payloadPath.string()); + // The payload is built in memory and exported, so writing it never + // registers a layer under the payload path. + const auto payloadStage = + pxr::UsdStage::CreateInMemory("payload.usdc"); if (!payloadStage || !pxr::UsdGeomXform::Define( payloadStage, pxr::SdfPath("/" + lodName)) || !AuthorPointCloudAsset(payloadStage, "/" + lodName + "/Points", level.reference, level.bounds, level.chunk, level.data) || - !payloadStage->GetRootLayer()->Save()) { + !payloadStage->GetRootLayer()->Export(payloadPath.string())) { return false; } const auto lodPrim = pxr::UsdGeomXform::Define( @@ -254,6 +237,135 @@ bool AuthorScreenSizeHeuristic( } // namespace +namespace detail { + +pxr::UsdStageRefPtr CreateDetachedStage() { + return pxr::UsdStage::CreateInMemory(pxr::UsdStage::LoadNone); +} + +std::string TilePayloadName(const usdpointcloud::PointTileId& id, + std::size_t lodIndex) { + return TilePrimName(id) + "_LOD" + std::to_string(lodIndex) + ".usdc"; +} + +std::vector TilePayloadNames( + const std::vector& tiles) { + std::vector names; + for (const auto& tile : tiles) { + for (std::size_t index = 0; index < tile.levels.size(); ++index) { + names.push_back(TilePayloadName(tile.tile.id, index)); + } + } + return names; +} + +namespace { + +std::string ReplacePathComponent(const std::string& path, + const std::string& from, + const std::string& to) { + std::string result; + std::size_t start = 0; + for (;;) { + const auto separator = path.find('/', start); + const auto component = path.substr( + start, separator == std::string::npos ? std::string::npos + : separator - start); + result += component == from ? to : component; + if (separator == std::string::npos) { + return result; + } + result += '/'; + start = separator + 1; + } +} + +} // namespace + +void RepointPayloads(const pxr::SdfLayerHandle& layer, + const std::string& fromComponent, + const std::string& toComponent) { + layer->Traverse(pxr::SdfPath::AbsoluteRootPath(), + [&](const pxr::SdfPath& path) { + const auto prim = layer->GetPrimAtPath(path); + if (!prim || !prim->HasPayloads()) { + return; + } + const auto payloads = prim->GetPayloadList(); + const auto repoint = [&](auto items) { + for (const auto& item : items) { + const pxr::SdfPayload payload = item; + const auto assetPath = ReplacePathComponent( + payload.GetAssetPath(), fromComponent, toComponent); + if (assetPath != payload.GetAssetPath()) { + auto repointed = payload; + repointed.SetAssetPath(assetPath); + items.Replace(payload, repointed); + } + } + }; + repoint(payloads.GetExplicitItems()); + repoint(payloads.GetAddedItems()); + repoint(payloads.GetPrependedItems()); + repoint(payloads.GetAppendedItems()); + }); +} + +bool CommitTilePayloads( + const pxr::SdfLayerHandle& layer, + PayloadGeneration& payloads, + std::vector& manifestEntries, + std::string& error) { + if (!payloads.Commit(error)) { + return false; + } + if (payloads.StagingName().empty()) { + return true; + } + RepointPayloads(layer, payloads.StagingName(), payloads.PublishedName()); + for (auto& entry : manifestEntries) { + entry.payloadPath = ReplacePathComponent( + entry.payloadPath, payloads.StagingName(), payloads.PublishedName()); + } + return true; +} + +} // namespace detail + +std::string PointCloudPayloadOwner( + const std::string& source, + const std::filesystem::path& payloadDirectory, + const pxr::SdfLayer::FileFormatArguments& arguments) { + // A local source is named relative to the payload directory, both + // canonicalized, so a project that moves or is mounted elsewhere as a + // whole keeps its owner. Any other source keeps its resolved identifier; + // the owner is only ever hashed, never written. + std::string owner; + std::error_code error; + if (std::filesystem::is_regular_file(source, error) && !error) { + const auto canonicalSource = + std::filesystem::weakly_canonical(source, error); + std::error_code directoryError; + const auto canonicalDirectory = + std::filesystem::weakly_canonical(payloadDirectory, directoryError); + if (!error && !directoryError) { + owner = "local:" + canonicalSource + .lexically_relative(canonicalDirectory) + .generic_string(); + } + } + if (owner.empty()) { + owner = "resolved:" + source; + } + // The arguments exactly as the layer holds them, which is its identity: + // two layers never share an owner because normalization rounded them + // together. + for (const auto& [key, value] : arguments) { + owner += "\n" + key + "=" + value; + } + return owner; +} + pxr::UsdStageRefPtr PointCloudLayer::CreateStage() { return pxr::UsdStage::CreateInMemory(); } @@ -323,7 +435,7 @@ bool AuthorPointCloudAsset( return false; } - const auto stage = PointCloudLayer::CreateStage(); + const auto stage = detail::CreateDetachedStage(); if (!stage) { failure = PointCloudAuthorFailure::StageCreation; return false; @@ -636,7 +748,7 @@ bool AuthorPointCloudMetadata( return false; } - const auto stage = PointCloudLayer::CreateStage(); + const auto stage = detail::CreateDetachedStage(); if (!stage || !pxr::UsdGeomSetStageUpAxis( stage, pxr::TfToken(reference.stageUpAxis)) || !pxr::UsdGeomSetStageMetersPerUnit(stage, 1.0)) { @@ -750,7 +862,7 @@ bool AuthorPointCloudLodAsset( if (!layer || levels.empty() || !levels.front().reference.IsValid()) { return false; } - const auto stage = PointCloudLayer::CreateStage(); + const auto stage = detail::CreateDetachedStage(); if (!stage || !pxr::UsdGeomSetStageUpAxis( stage, pxr::TfToken(levels.front().reference.stageUpAxis)) || !pxr::UsdGeomSetStageMetersPerUnit(stage, 1.0) || @@ -824,72 +936,28 @@ bool AuthorPointCloudTiledAssetWithPayloads( stage, primPath, tiles, options, generatedPayloads); } -bool AuthorPointCloudTiledAssetWithPayloads( +namespace detail { + +bool AuthorClaimedTilePayloads( const pxr::UsdStageRefPtr& stage, const std::string& primPath, const std::vector& tiles, const PointCloudPayloadOptions& options, - std::vector& generatedPayloads) { + PayloadGeneration& payloads, + std::vector& manifestEntries) { if (!stage || !IsValidPrimPath(primPath) || tiles.empty() || options.directory.empty() || options.rootLayerPath.empty()) { return false; } - const std::filesystem::path payloadDirectory(options.directory); const std::filesystem::path rootLayerPath(options.rootLayerPath); - const auto rootLayer = stage->GetRootLayer(); - if (!rootLayer) { - return false; - } - std::vector payloadPaths; - std::vector manifestEntries; - std::error_code error; - const auto payloadDirectoryExisted = - std::filesystem::exists(payloadDirectory, error); - if (error) { - return false; - } - std::filesystem::create_directories(payloadDirectory, error); - if (error) { - return false; - } - - const auto cleanup = [&payloadPaths, &payloadDirectory, - payloadDirectoryExisted]() { - std::error_code cleanupError; - for (const auto& payloadPath : payloadPaths) { - std::filesystem::remove(payloadPath, cleanupError); - cleanupError.clear(); - } - if (!payloadDirectoryExisted) { - const auto empty = std::filesystem::is_empty( - payloadDirectory, cleanupError); - if (empty && !cleanupError) { - std::filesystem::remove(payloadDirectory, cleanupError); - } - } - }; - for (const auto& tile : tiles) { - const auto tilePath = primPath + "/Tiles/" + TilePrimName(tile.tile.id); - for (std::size_t index = 0; index < tile.levels.size(); ++index) { - const auto payloadPath = payloadDirectory / - (tilePath.substr(tilePath.find_last_of('/') + 1) + "_LOD" + - std::to_string(index) + ".usdc"); - if (std::filesystem::exists(payloadPath)) { - cleanup(); - return false; - } - payloadPaths.push_back(payloadPath); - } - } - + std::vector tileEntries; std::set tileNames; for (const auto& tile : tiles) { std::vector diagnostics; if (!usdpointcloud::ValidatePointTile(tile.tile, diagnostics) || tile.levels.size() != tile.tile.lod.items.size() || !tileNames.insert(tile.tile.id.ToString()).second) { - cleanup(); return false; } for (std::size_t index = 0; index < tile.levels.size(); ++index) { @@ -897,18 +965,20 @@ bool AuthorPointCloudTiledAssetWithPayloads( const auto& item = tile.tile.lod.items[index]; if (!level.IsValid() || level.chunk.pointCount != item.pointCount || !SameBounds(level.bounds, item.bounds)) { - cleanup(); return false; } - const auto payloadPath = payloadPaths[ - manifestEntries.size()]; + const auto payloadPath = + payloads.PathFor(TilePayloadName(tile.tile.id, index)); + if (payloadPath.empty()) { + return false; + } + std::error_code error; const auto relativePayloadPath = std::filesystem::relative( payloadPath, rootLayerPath.parent_path(), error); if (error || relativePayloadPath.empty()) { - cleanup(); return false; } - manifestEntries.push_back({ + tileEntries.push_back({ tile.tile.id, static_cast(index), item.bounds, @@ -918,28 +988,26 @@ bool AuthorPointCloudTiledAssetWithPayloads( } if (options.tileManifestEntries) { std::vector manifestDiagnostics; - const usdpointcloud::PointTileManifest tileManifest{manifestEntries}; + const usdpointcloud::PointTileManifest tileManifest{tileEntries}; if (!usdpointcloud::ValidatePointTileManifest( tileManifest, manifestDiagnostics)) { - cleanup(); return false; } } - const ScopedLayerIdentifier scopedRootIdentifier( - rootLayer, rootLayerPath.generic_string()); if (!pxr::UsdGeomXform::Define(stage, pxr::SdfPath(primPath)) || !pxr::UsdGeomScope::Define( stage, pxr::SdfPath(primPath + "/LodHeuristics")) || !pxr::UsdGeomScope::Define( stage, pxr::SdfPath(primPath + "/Tiles"))) { - cleanup(); return false; } const auto heuristicPath = pxr::SdfPath(primPath + "/LodHeuristics"); for (const auto& tile : tiles) { + const PayloadTarget payloadTarget{tile.tile.id, rootLayerPath, + payloads}; const auto tilePath = pxr::SdfPath( primPath + "/Tiles/" + TilePrimName(tile.tile.id)); const auto tileHeuristicPath = @@ -949,18 +1017,52 @@ bool AuthorPointCloudTiledAssetWithPayloads( !AuthorScreenSizeHeuristic( stage, tileHeuristicPath, tile.levels.front().reference, tile.tile.bounds, tile.tile.lod.screenSizeThresholds)) { - cleanup(); return false; } if (!AuthorLodRoot(stage, tilePath.GetString(), tile.levels, - tile.tile.lod, tileHeuristicPath, - &payloadDirectory, &rootLayerPath)) { - cleanup(); + tile.tile.lod, tileHeuristicPath, &payloadTarget)) { return false; } } - generatedPayloads.insert(generatedPayloads.end(), payloadPaths.begin(), - payloadPaths.end()); + manifestEntries.insert(manifestEntries.end(), tileEntries.begin(), + tileEntries.end()); + return true; +} + +} // namespace detail + +namespace { + +bool AuthorGeneratedTilePayloads( + const pxr::UsdStageRefPtr& stage, + const std::string& primPath, + const std::vector& tiles, + const PointCloudPayloadOptions& options, + std::vector& generatedPayloads, + std::string& error) { + if (!stage || !IsValidPrimPath(primPath) || tiles.empty() || + options.directory.empty() || options.rootLayerPath.empty()) { + error = "invalid tiled point-cloud payload request"; + return false; + } + const auto names = detail::TilePayloadNames(tiles); + detail::PayloadGeneration payloads(options.directory, options.owner); + std::vector manifestEntries; + if (!payloads.Begin(names, error)) { + return false; + } + if (!detail::AuthorClaimedTilePayloads(stage, primPath, tiles, options, + payloads, manifestEntries)) { + error = "unable to author tiled point-cloud payloads"; + return false; + } + if (!detail::CommitTilePayloads(stage->GetRootLayer(), payloads, + manifestEntries, error)) { + return false; + } + for (const auto& name : names) { + generatedPayloads.push_back(payloads.PublishedPathFor(name)); + } if (options.tileManifestEntries) { options.tileManifestEntries->insert( options.tileManifestEntries->end(), manifestEntries.begin(), @@ -969,6 +1071,55 @@ bool AuthorPointCloudTiledAssetWithPayloads( return true; } +} // namespace + +bool AuthorPointCloudTiledAssetWithPayloads( + const pxr::UsdStageRefPtr& stage, + const std::string& primPath, + const std::vector& tiles, + const PointCloudPayloadOptions& options, + std::vector& generatedPayloads) { + std::string error; + return AuthorGeneratedTilePayloads(stage, primPath, tiles, options, + generatedPayloads, error); +} + +bool AuthorPointCloudTiledAssetWithPayloads( + pxr::SdfLayer* layer, + const std::string& primPath, + const std::vector& tiles, + const PointCloudPayloadOptions& options, + std::vector& diagnostics) { + diagnostics.clear(); + const auto fail = [&diagnostics](DiagnosticCode code, + const std::string& message) { + diagnostics.push_back( + {code, Severity::Error, message, std::nullopt, std::nullopt}); + return false; + }; + if (!layer || tiles.empty() || tiles.front().levels.empty() || + !tiles.front().levels.front().reference.IsValid()) { + return fail(DiagnosticCode::InvalidPointTile, + "invalid tiled point-cloud authoring request"); + } + const auto stage = detail::CreateDetachedStage(); + if (!stage || !pxr::UsdGeomSetStageUpAxis( + stage, pxr::TfToken( + tiles.front().levels.front().reference.stageUpAxis)) || + !pxr::UsdGeomSetStageMetersPerUnit(stage, 1.0)) { + return fail(DiagnosticCode::DecodeFailure, + "unable to create tiled point-cloud stage"); + } + std::vector generatedPayloads; + std::string error; + if (!AuthorGeneratedTilePayloads(stage, primPath, tiles, options, + generatedPayloads, error)) { + return fail(DiagnosticCode::DecodeFailure, error); + } + layer->TransferContent(stage->GetRootLayer()); + return true; +} + bool AuthorPointCloudTiledAsset( pxr::SdfLayer* layer, const std::string& primPath, @@ -977,7 +1128,7 @@ bool AuthorPointCloudTiledAsset( !tiles.front().levels.front().reference.IsValid()) { return false; } - const auto stage = PointCloudLayer::CreateStage(); + const auto stage = detail::CreateDetachedStage(); if (!stage || !pxr::UsdGeomSetStageUpAxis( stage, pxr::TfToken( tiles.front().levels.front().reference.stageUpAxis)) || diff --git a/libs/usd-pointcloud-authoring/src/PointCloudStreaming.cpp b/libs/usd-pointcloud-authoring/src/PointCloudStreaming.cpp index af2371d..74c4b7a 100644 --- a/libs/usd-pointcloud-authoring/src/PointCloudStreaming.cpp +++ b/libs/usd-pointcloud-authoring/src/PointCloudStreaming.cpp @@ -1,8 +1,9 @@ #include "usdgeo/PointCloudLayer.h" +#include "TiledPayloadAuthoring.h" + #include #include -#include #include #include #include @@ -232,8 +233,6 @@ struct TileSpool { std::unique_ptr writer; }; -std::atomic_uint64_t streamLayerSequence{0}; - } // namespace bool AuthorPointCloudTiledAssetFromStream( @@ -257,16 +256,13 @@ bool AuthorPointCloudTiledAssetFromStream( if (spoolDirectory.empty()) return false; std::map spools; std::size_t bufferedBytes = 0; - std::vector generatedPayloads; + detail::PayloadGeneration payloads(options.directory, options.owner); + std::vector manifestEntries; const auto cleanup = [&]() { for (auto& entry : spools) entry.second.writer.reset(); std::vector cleanupDiagnostics; usdpointcloud::RemoveSpoolDirectory(spoolDirectory, cleanupDiagnostics); - std::error_code cleanupError; - for (const auto& payloadPath : generatedPayloads) { - std::filesystem::remove(payloadPath, cleanupError); - cleanupError.clear(); - } + payloads.Rollback(); }; const auto isCancelled = [&]() { return options.isCancelled && options.isCancelled(); @@ -408,8 +404,28 @@ bool AuthorPointCloudTiledAssetFromStream( return false; } } + if (spools.empty()) { + AddError(diagnostics, DiagnosticCode::DecodeFailure, + "point stream did not produce any points"); + cleanup(); + return false; + } - const auto stage = PointCloudLayer::CreateStage(); + // The tile set is known once the stream is spooled, so every payload is + // declared before the first one is written. + std::vector payloadNames; + payloadNames.reserve(spools.size()); + for (const auto& entry : spools) { + payloadNames.push_back(detail::TilePayloadName(entry.second.id, 0)); + } + std::string payloadError; + if (!payloads.Begin(payloadNames, payloadError)) { + AddError(diagnostics, DiagnosticCode::DecodeFailure, payloadError); + cleanup(); + return false; + } + + const auto stage = detail::CreateDetachedStage(); if (!stage || !pxr::UsdGeomSetStageUpAxis( stage, pxr::TfToken(reference.stageUpAxis)) || !pxr::UsdGeomSetStageMetersPerUnit(stage, 1.0)) { @@ -418,12 +434,7 @@ bool AuthorPointCloudTiledAssetFromStream( cleanup(); return false; } - const auto streamLayerIdentifier = - options.rootLayerPath + ".usdgeo-stream-" + - std::to_string(++streamLayerSequence); - stage->GetRootLayer()->SetIdentifier(streamLayerIdentifier); - std::size_t tileCount = 0; for (const auto& entry : spools) { if (isCancelled()) { AddError(diagnostics, DiagnosticCode::DecodeFailure, @@ -441,6 +452,7 @@ bool AuthorPointCloudTiledAssetFromStream( usdpointcloud::SpoolSchema tileSchema; if (!reader.Open(entry.second.path, tileId, tileSchema, diagnostics, options.spoolIoStats) || + tileId.ToString() != entry.first || !SameSchema(schema, tileSchema)) { AddError(diagnostics, DiagnosticCode::DecodeFailure, "unable to reopen point tile spool"); @@ -497,8 +509,9 @@ bool AuthorPointCloudTiledAssetFromStream( tile.levels.push_back(std::move(asset)); std::vector singleTile; singleTile.push_back(std::move(tile)); - if (!AuthorPointCloudTiledAssetWithPayloads( - stage, primPath, singleTile, options, generatedPayloads)) { + if (!detail::AuthorClaimedTilePayloads( + stage, primPath, singleTile, options, payloads, + manifestEntries)) { AddError(diagnostics, DiagnosticCode::DecodeFailure, "unable to author tiled point-cloud payloads"); return failTile(); @@ -508,13 +521,6 @@ bool AuthorPointCloudTiledAssetFromStream( "point-cloud authoring cancelled"); return failTile(); } - ++tileCount; - } - if (tileCount == 0) { - AddError(diagnostics, DiagnosticCode::DecodeFailure, - "point stream did not produce any points"); - cleanup(); - return false; } std::vector cleanupDiagnostics; if (!usdpointcloud::RemoveSpoolDirectory(spoolDirectory, @@ -524,6 +530,17 @@ bool AuthorPointCloudTiledAssetFromStream( cleanup(); return false; } + if (!detail::CommitTilePayloads(stage->GetRootLayer(), payloads, + manifestEntries, payloadError)) { + AddError(diagnostics, DiagnosticCode::DecodeFailure, payloadError); + cleanup(); + return false; + } + if (options.tileManifestEntries) { + options.tileManifestEntries->insert( + options.tileManifestEntries->end(), manifestEntries.begin(), + manifestEntries.end()); + } layer->TransferContent(stage->GetRootLayer()); return true; } diff --git a/libs/usd-pointcloud-authoring/src/TiledPayloadAuthoring.h b/libs/usd-pointcloud-authoring/src/TiledPayloadAuthoring.h new file mode 100644 index 0000000..600c2ab --- /dev/null +++ b/libs/usd-pointcloud-authoring/src/TiledPayloadAuthoring.h @@ -0,0 +1,51 @@ +#pragma once + +#include "PayloadGeneration.h" + +#include "usdgeo/PointCloudLayer.h" + +#include +#include +#include + +namespace usdgeo::detail { + +// An in-memory stage whose content is built for transfer into a caller's +// layer. It loads nothing, so a payload arc authored into it is never +// resolved or opened while the content is built, and it never needs a file +// identity to anchor one. +pxr::UsdStageRefPtr CreateDetachedStage(); + +std::string TilePayloadName(const usdpointcloud::PointTileId& id, + std::size_t lodIndex); + +std::vector TilePayloadNames( + const std::vector& tiles); + +// Authors payload-backed tiles whose payload names `payloads` has declared, +// writing each payload where the generation stages it. Manifest entries are +// appended to `manifestEntries` and name the staged payloads until +// CommitTilePayloads repoints them. +bool AuthorClaimedTilePayloads( + const pxr::UsdStageRefPtr& stage, + const std::string& primPath, + const std::vector& tiles, + const PointCloudPayloadOptions& options, + PayloadGeneration& payloads, + std::vector& manifestEntries); + +// Publishes `payloads`, then points the payload arcs in `layer` and the +// manifest entries at the published generation instead of the staging +// directory. +bool CommitTilePayloads( + const pxr::SdfLayerHandle& layer, + PayloadGeneration& payloads, + std::vector& manifestEntries, + std::string& error); + +// Replaces one exact path component in every payload asset path in `layer`. +void RepointPayloads(const pxr::SdfLayerHandle& layer, + const std::string& fromComponent, + const std::string& toComponent); + +} // namespace usdgeo::detail diff --git a/libs/usd-pointcloud-authoring/tests/test_pointcloud_cache.cpp b/libs/usd-pointcloud-authoring/tests/test_pointcloud_cache.cpp index c1b2e1c..54e014c 100644 --- a/libs/usd-pointcloud-authoring/tests/test_pointcloud_cache.cpp +++ b/libs/usd-pointcloud-authoring/tests/test_pointcloud_cache.cpp @@ -30,6 +30,26 @@ void Check(bool condition) { } } +// Every file named `name` below `directory`. +std::vector FindFiles( + const std::filesystem::path& directory, const std::string& name) { + std::vector matches; + for (const auto& entry : + std::filesystem::recursive_directory_iterator(directory)) { + if (entry.is_regular_file() && entry.path().filename() == name) { + matches.push_back(entry.path()); + } + } + return matches; +} + +std::string ReadBytes(const std::filesystem::path& path) { + std::ifstream input(path, std::ios::binary); + Check(input.good()); + return {std::istreambuf_iterator(input), + std::istreambuf_iterator()}; +} + std::string FormatDouble(double value) { std::ostringstream result; result << std::setprecision(17) << value; @@ -197,8 +217,15 @@ void TestPointCloudCacheMissAndMaterialization() { reference, request, "las-reader-1", hit, errorMessage)); Check(hit); Check(errorMessage.empty()); - const auto requestedPayload = testRoot / "requested_payloads" / "tile.usdc"; - Check(std::filesystem::exists(requestedPayload)); + // The copy lands in the layer's own directory under the requested one, + // in a generation named by the cache entry. + const auto requestedPayloads = testRoot / "requested_payloads"; + const auto materialized = FindFiles(requestedPayloads, "tile.usdc"); + Check(materialized.size() == 1); + const auto requestedPayload = materialized.front(); + const auto generation = requestedPayload.parent_path(); + Check(generation.filename().string().rfind("c-", 0) == 0); + Check(generation.parent_path().parent_path() == requestedPayloads); const auto cachedPrim = requestedLayer->GetPrimAtPath( pxr::SdfPath("/PointCloud/Tile")); @@ -206,17 +233,56 @@ void TestPointCloudCacheMissAndMaterialization() { const auto payloadItems = cachedPrim->GetPayloadList().GetPrependedItems(); Check(payloadItems.size() == 1); Check(payloadItems.front().GetAssetPath() == - "../requested_payloads/tile.usdc"); - - const auto requestedStage = pxr::UsdStage::Open(requestedLayer); - Check(requestedStage); - const auto requestedPoints = pxr::UsdGeomPoints::Get( - requestedStage, pxr::SdfPath("/PointCloud/Tile/Points")); - Check(requestedPoints.GetPrim().IsValid()); - pxr::VtVec3fArray requestedPositions; - Check(requestedPoints.GetPointsAttr().Get(&requestedPositions)); - Check(requestedPositions.size() == 1 && - requestedPositions[0] == pxr::GfVec3f(1.0f, 2.0f, 3.0f)); + "../requested_payloads/" + + generation.parent_path().filename().generic_string() + "/" + + generation.filename().generic_string() + "/tile.usdc"); + + { + const auto requestedStage = pxr::UsdStage::Open(requestedLayer); + Check(requestedStage); + const auto requestedPoints = pxr::UsdGeomPoints::Get( + requestedStage, pxr::SdfPath("/PointCloud/Tile/Points")); + Check(requestedPoints.GetPrim().IsValid()); + pxr::VtVec3fArray requestedPositions; + Check(requestedPoints.GetPointsAttr().Get(&requestedPositions)); + Check(requestedPositions.size() == 1 && + requestedPositions[0] == pxr::GfVec3f(1.0f, 2.0f, 3.0f)); + } + + const auto loadRequested = [&]() { + const auto layer = + pxr::SdfLayer::CreateAnonymous("requested-again.usda"); + bool loadedHit = false; + std::string loadError; + Check(usdgeo::TryLoadPointCloudCache( + layer.operator->(), resolverIdentity, sourcePath.parent_path(), + reference, request, "las-reader-1", loadedHit, loadError)); + Check(loadError.empty()); + return loadedHit; + }; + + // An intact copy from an earlier hit is reused without being written. + const auto materializedWrite = + std::filesystem::last_write_time(requestedPayload); + Check(loadRequested()); + Check(FindFiles(requestedPayloads, "tile.usdc") == materialized); + Check(std::filesystem::last_write_time(requestedPayload) == + materializedWrite); + + // A damaged copy is replaced by a fresh one from the cache. + std::ofstream(requestedPayload, std::ios::binary | std::ios::trunc) + << "damaged"; + Check(loadRequested()); + Check(FindFiles(requestedPayloads, "tile.usdc") == materialized); + Check(ReadBytes(requestedPayload) == ReadBytes(payloadPath)); + + // A file someone else placed in the requested directory is never + // touched, because the layer's copies live in its own directory. + std::ofstream(requestedPayloads / "tile.usdc", std::ios::binary) + << "user payload"; + Check(loadRequested()); + Check(ReadBytes(requestedPayloads / "tile.usdc") == "user payload"); + Check(ReadBytes(requestedPayload) == ReadBytes(payloadPath)); std::error_code error; std::filesystem::remove(layout.manifest, error); diff --git a/libs/usd-pointcloud-authoring/tests/test_usd_authoring.cpp b/libs/usd-pointcloud-authoring/tests/test_usd_authoring.cpp index 1a318d6..86df377 100644 --- a/libs/usd-pointcloud-authoring/tests/test_usd_authoring.cpp +++ b/libs/usd-pointcloud-authoring/tests/test_usd_authoring.cpp @@ -1,8 +1,10 @@ +#include "usdgeo/CacheKey.h" #include "usdgeo/PointCloudLayer.h" #include "usdcopc/Copc.h" #include #include +#include #include #include #include @@ -13,10 +15,13 @@ #include #include +#include +#include #include #include #include #include +#include #include #include #include @@ -36,6 +41,72 @@ std::string ReadFile(const std::filesystem::path& path) { std::istreambuf_iterator()}; } +void WriteFile(const std::filesystem::path& path, const std::string& text) { + std::ofstream output(path, std::ios::binary | std::ios::trunc); + output << text; + Check(output.good()); +} + +// A directory no other test or earlier run shares, so payload ownership +// state never leaks between cases. +std::filesystem::path UniqueTestDirectory(const std::string& name) { + const auto stamp = + std::chrono::steady_clock::now().time_since_epoch().count(); + const auto directory = std::filesystem::temp_directory_path() / + (name + "_" + std::to_string(stamp)); + std::filesystem::remove_all(directory); + return directory; +} + +std::filesystem::path OwnerDirectory(const std::filesystem::path& directory, + const std::string& owner) { + return directory / usdgeo::StableCacheKey({{"payloadOwner", owner}}); +} + +std::vector Subdirectories( + const std::filesystem::path& directory, const std::string& prefix) { + std::vector matches; + std::error_code error; + for (const auto& entry : + std::filesystem::directory_iterator(directory, error)) { + if (entry.is_directory() && + entry.path().filename().string().rfind(prefix, 0) == 0) { + matches.push_back(entry.path()); + } + } + std::sort(matches.begin(), matches.end()); + return matches; +} + +// The one published generation an owner has, which a test expects to exist. +std::filesystem::path OnlyGeneration(const std::filesystem::path& ownerDirectory) { + const auto generated = Subdirectories(ownerDirectory, "g-"); + const auto cached = Subdirectories(ownerDirectory, "c-"); + Check(generated.size() + cached.size() == 1); + return generated.empty() ? cached.front() : generated.front(); +} + +class WarningCounter final : public pxr::TfDiagnosticMgr::Delegate { +public: + WarningCounter() { pxr::TfDiagnosticMgr::GetInstance().AddDelegate(this); } + ~WarningCounter() override { + pxr::TfDiagnosticMgr::GetInstance().RemoveDelegate(this); + } + + void IssueError(const pxr::TfError&) override {} + void IssueFatalError(const pxr::TfCallContext&, + const std::string&) override { + _UnhandledAbort(); + } + void IssueStatus(const pxr::TfStatus&) override {} + void IssueWarning(const pxr::TfWarning&) override { ++count_; } + + int Count() const noexcept { return count_; } + +private: + std::atomic_int count_{0}; +}; + std::map ReadDirectoryFiles( const std::filesystem::path& directory) { std::map files; @@ -1169,6 +1240,279 @@ void TestStreamPayloadFailureRollsBackGeneratedPayloads() { std::filesystem::remove_all(payloadDirectory); } +std::pair MakeTestChunk( + const std::vector& positions) { + usdpointcloud::PointData data; + data.positions = positions; + usdgeo::SpatialBounds bounds = usdgeo::SpatialBounds::Empty(); + for (const auto& position : positions) bounds.Expand(position); + return {usdpointcloud::MakePointChunk(data, bounds), std::move(data)}; +} + +// Streams `positions` into unit fixed-grid tiles as `owner`'s payloads. +bool AuthorOwnedStream(pxr::SdfLayer* layer, + const std::filesystem::path& directory, + const std::string& owner, + const std::vector& positions, + std::vector& diagnostics, + std::function isCancelled = {}) { + usdgeo::GeoReference reference; + reference.epsgCode = 26910; + TestPointStream stream({MakeTestChunk(positions)}); + usdgeo::PointCloudPayloadOptions options{ + directory.string(), (directory / "PointCloud.usda").string(), 1, + std::move(isCancelled)}; + options.owner = owner; + return usdgeo::AuthorPointCloudTiledAssetFromStream( + layer, "/PointCloud", stream, reference, {1.0, 0}, options, + diagnostics); +} + +void TestOwnedPayloadsLiveInTheirOwnersGeneration() { + const auto directory = UniqueTestDirectory("usd_geo_owned_payloads"); + const auto ownerDirectory = OwnerDirectory(directory, "layer-a"); + std::vector diagnostics; + + const auto layer = pxr::SdfLayer::CreateAnonymous("owned.usda"); + Check(AuthorOwnedStream(layer.operator->(), directory, "layer-a", + {{0.25, 0.0, 0.0}, {1.25, 0.0, 0.0}}, + diagnostics)); + Check(diagnostics.empty()); + const auto generation = OnlyGeneration(ownerDirectory); + const auto first = generation / "Tile_L0_p0_p0_p0_LOD0.usdc"; + Check(std::filesystem::exists(first) && + std::filesystem::exists(generation / "Tile_L0_p1_p0_p0_LOD0.usdc")); + Check(Subdirectories(ownerDirectory, ".tmp-").empty()); + const auto lod = layer->GetPrimAtPath( + pxr::SdfPath("/PointCloud/Tiles/Tile_L0_p0_p0_p0/LOD0")); + Check(lod != nullptr); + const auto arcs = lod->GetPayloadList().GetPrependedItems(); + Check(arcs.size() == 1 && + arcs.front().GetAssetPath() == + ownerDirectory.filename().generic_string() + "/" + + generation.filename().generic_string() + + "/Tile_L0_p0_p0_p0_LOD0.usdc"); + + // Reading the layer again with identical content keeps the published + // generation as it is instead of replacing files that may be open. + const auto firstWrite = std::filesystem::last_write_time(first); + const auto again = pxr::SdfLayer::CreateAnonymous("again.usda"); + Check(AuthorOwnedStream(again.operator->(), directory, "layer-a", + {{0.25, 0.0, 0.0}, {1.25, 0.0, 0.0}}, + diagnostics)); + Check(OnlyGeneration(ownerDirectory) == generation); + Check(std::filesystem::last_write_time(first) == firstWrite); + + // Changed content is published as a new generation, and the superseded + // one is removed. + const auto changed = pxr::SdfLayer::CreateAnonymous("changed.usda"); + Check(AuthorOwnedStream(changed.operator->(), directory, "layer-a", + {{0.25, 0.0, 0.0}, {0.75, 0.0, 0.0}}, + diagnostics)); + const auto changedGeneration = OnlyGeneration(ownerDirectory); + Check(changedGeneration != generation); + Check(!std::filesystem::exists(generation)); + Check(!std::filesystem::exists( + changedGeneration / "Tile_L0_p1_p0_p0_LOD0.usdc")); + { + const auto rootLayerPath = directory / "PointCloud.usda"; + Check(changed->Export(rootLayerPath.string())); + const auto stage = pxr::UsdStage::Open(rootLayerPath.string()); + Check(stage); + const auto points = pxr::UsdGeomPoints::Get( + stage, + pxr::SdfPath("/PointCloud/Tiles/Tile_L0_p0_p0_p0/LOD0/Points")); + pxr::VtVec3fArray positions; + Check(points.GetPointsAttr().Get(&positions)); + Check(positions.size() == 2); + std::filesystem::remove(rootLayerPath); + } + + // Other layers and user files in the same payload directory never meet + // this owner's payloads: each owner has its own directory. + const auto published = + ReadFile(changedGeneration / "Tile_L0_p0_p0_p0_LOD0.usdc"); + WriteFile(directory / "Tile_L0_p0_p0_p0_LOD0.usdc", "user file"); + const auto other = pxr::SdfLayer::CreateAnonymous("other.usda"); + Check(AuthorOwnedStream(other.operator->(), directory, "layer-b", + {{0.25, 0.0, 0.0}}, diagnostics)); + Check(OnlyGeneration(OwnerDirectory(directory, "layer-b")) != + changedGeneration); + Check(ReadFile(changedGeneration / "Tile_L0_p0_p0_p0_LOD0.usdc") == + published); + Check(ReadFile(directory / "Tile_L0_p0_p0_p0_LOD0.usdc") == "user file"); + + // Without an owner the directory stays exclusive. + Check(!AuthorOwnedStream(other.operator->(), directory, "", + {{0.25, 0.0, 0.0}}, diagnostics)); + Check(ReadFile(directory / "Tile_L0_p0_p0_p0_LOD0.usdc") == "user file"); + std::filesystem::remove_all(directory); +} + +void TestInterruptedOwnedGenerationIsRecovered() { + const auto directory = + UniqueTestDirectory("usd_geo_interrupted_owned_payloads"); + const auto ownerDirectory = OwnerDirectory(directory, "layer-a"); + // What interrupted generations leave behind: staging directories that + // were never published. A live generation keeps its own fresh. + const auto stale = ownerDirectory / ".tmp-stale"; + const auto live = ownerDirectory / ".tmp-live"; + std::filesystem::create_directories(stale); + std::filesystem::create_directories(live); + WriteFile(stale / "Tile_L0_p0_p0_p0_LOD0.usdc", "partial payload"); + std::filesystem::last_write_time( + stale, std::filesystem::file_time_type::clock::now() - + std::chrono::hours(2)); + + std::vector diagnostics; + const auto layer = pxr::SdfLayer::CreateAnonymous("recovered.usda"); + Check(AuthorOwnedStream(layer.operator->(), directory, "layer-a", + {{0.25, 0.0, 0.0}}, diagnostics)); + Check(diagnostics.empty()); + Check(!std::filesystem::exists(stale)); + Check(std::filesystem::exists(live)); + + // A published generation that lost a file is replaced when the same + // content is published again, rather than trusted. + const auto generation = OnlyGeneration(ownerDirectory); + const auto payload = generation / "Tile_L0_p0_p0_p0_LOD0.usdc"; + std::filesystem::remove(payload); + Check(AuthorOwnedStream(layer.operator->(), directory, "layer-a", + {{0.25, 0.0, 0.0}}, diagnostics)); + Check(OnlyGeneration(ownerDirectory) == generation); + Check(std::filesystem::exists(payload)); + std::filesystem::remove_all(directory); +} + +void TestFailedOwnedGenerationKeepsPublishedPayloads() { + const auto directory = + UniqueTestDirectory("usd_geo_failed_owned_payloads"); + const auto ownerDirectory = OwnerDirectory(directory, "layer-a"); + std::vector diagnostics; + + const auto layer = pxr::SdfLayer::CreateAnonymous("failed_owned.usda"); + Check(AuthorOwnedStream(layer.operator->(), directory, "layer-a", + {{0.25, 0.0, 0.0}, {1.25, 0.0, 0.0}}, + diagnostics)); + const auto generation = OnlyGeneration(ownerDirectory); + const auto published = ReadDirectoryFiles(generation); + + // Cancel as soon as the regeneration has staged its first payload. + const auto stagedFirst = [&ownerDirectory]() { + for (const auto& staging : Subdirectories(ownerDirectory, ".tmp-")) { + if (std::filesystem::exists(staging / + "Tile_L0_p0_p0_p0_LOD0.usdc")) { + return true; + } + } + return false; + }; + const auto regenerated = pxr::SdfLayer::CreateAnonymous("cancelled.usda"); + Check(!AuthorOwnedStream(regenerated.operator->(), directory, "layer-a", + {{0.5, 0.0, 0.0}, {1.25, 0.0, 0.0}}, + diagnostics, stagedFirst)); + Check(!diagnostics.empty()); + Check(regenerated->GetPrimAtPath(pxr::SdfPath("/PointCloud")) == nullptr); + Check(OnlyGeneration(ownerDirectory) == generation); + Check(ReadDirectoryFiles(generation) == published); + Check(Subdirectories(ownerDirectory, ".tmp-").empty()); + std::filesystem::remove_all(directory); + + // A cancelled first generation removes every directory it created. + const auto created = UniqueTestDirectory("usd_geo_cancelled_owned_payloads"); + const auto createdOwner = OwnerDirectory(created, "layer-a"); + const auto fresh = pxr::SdfLayer::CreateAnonymous("fresh.usda"); + Check(!AuthorOwnedStream( + fresh.operator->(), created, "layer-a", + {{0.25, 0.0, 0.0}, {1.25, 0.0, 0.0}}, diagnostics, + [&createdOwner]() { + std::error_code error; + for (const auto& staging : + std::filesystem::directory_iterator(createdOwner, error)) { + if (std::filesystem::exists(staging.path() / + "Tile_L0_p0_p0_p0_LOD0.usdc")) { + return true; + } + } + return false; + })); + Check(!std::filesystem::exists(created)); +} + +void TestDetachedAuthoringDoesNotResolvePayloads() { + const auto directory = UniqueTestDirectory("usd_geo_detached_payloads"); + usdgeo::GeoReference reference; + reference.epsgCode = 26910; + + usdgeo::PointCloudTileAsset tile; + tile.tile.id = {0, 0, 0, 0}; + tile.tile.bounds.Expand({0.0, 0.0, 0.0}); + tile.tile.lod.bounds = tile.tile.bounds; + tile.tile.lod.items = {{0, 1, tile.tile.bounds, {0, 1}}}; + usdpointcloud::PointCloudAsset level; + level.reference = reference; + level.bounds = tile.tile.bounds; + level.data.positions = {{0.0, 0.0, 0.0}}; + level.chunk = usdpointcloud::MakePointChunk(level.data, level.bounds); + tile.levels.push_back(std::move(level)); + + // The payload arcs are relative to a root that is not the working + // directory. Building the layer must not try to resolve them: resolving + // from a detached stage is what reported "Could not open asset". + const auto layer = pxr::SdfLayer::CreateAnonymous("detached.usda"); + std::vector diagnostics; + usdgeo::PointCloudPayloadOptions options{ + directory.string(), (directory / "PointCloud.usda").string()}; + options.owner = "detached-layer"; + { + const WarningCounter warnings; + Check(usdgeo::AuthorPointCloudTiledAssetWithPayloads( + layer.operator->(), "/PointCloud", {tile}, options, diagnostics)); + Check(warnings.Count() == 0); + } + Check(diagnostics.empty()); + Check(layer->GetPrimAtPath(pxr::SdfPath( + "/PointCloud/Tiles/Tile_L0_p0_p0_p0/LOD0")) != nullptr); + const auto payload = + OnlyGeneration(OwnerDirectory(directory, "detached-layer")) / + "Tile_L0_p0_p0_p0_LOD0.usdc"; + Check(std::filesystem::exists(payload)); + Check(!pxr::SdfLayer::Find(payload.string())); + + // The same layer is authored again over its own payloads. + Check(usdgeo::AuthorPointCloudTiledAssetWithPayloads( + layer.operator->(), "/PointCloud", {tile}, options, diagnostics)); + std::filesystem::remove_all(directory); + + // A failure reports why and leaves the layer as it was. + WriteFile(directory, "not a directory"); + const auto untouched = pxr::SdfLayer::CreateAnonymous("untouched.usda"); + Check(!usdgeo::AuthorPointCloudTiledAssetWithPayloads( + untouched.operator->(), "/PointCloud", {tile}, options, diagnostics)); + Check(!diagnostics.empty() && + diagnostics.front().message.find("not a usable directory") != + std::string::npos); + Check(untouched->GetPrimAtPath(pxr::SdfPath("/PointCloud")) == nullptr); + std::filesystem::remove(directory); + + const auto streamDirectory = + UniqueTestDirectory("usd_geo_detached_stream_payloads"); + const auto streamLayer = pxr::SdfLayer::CreateAnonymous("stream.usda"); + { + const WarningCounter warnings; + Check(AuthorOwnedStream(streamLayer.operator->(), streamDirectory, + "detached-stream", + {{0.25, 0.0, 0.0}, {1.25, 0.0, 0.0}}, + diagnostics)); + Check(warnings.Count() == 0); + } + Check(!pxr::SdfLayer::Find( + (OnlyGeneration(OwnerDirectory(streamDirectory, "detached-stream")) / + "Tile_L0_p0_p0_p0_LOD0.usdc") + .string())); + std::filesystem::remove_all(streamDirectory); +} + void TestInvalidExtraByteNameDoesNotAuthor() { const auto stage = usdgeo::PointCloudLayer::CreateStage(); usdgeo::GeoReference reference; @@ -1215,5 +1559,9 @@ int main() { TestStreamCancellationDuringSpoolReadCleansSpools(); TestStreamFailureDoesNotMutateLayer(); TestStreamPayloadFailureRollsBackGeneratedPayloads(); + TestOwnedPayloadsLiveInTheirOwnersGeneration(); + TestInterruptedOwnedGenerationIsRecovered(); + TestFailedOwnedGenerationKeepsPublishedPayloads(); + TestDetachedAuthoringDoesNotResolvePayloads(); return 0; } \ No newline at end of file diff --git a/plugins/pointcloud-copc/README.md b/plugins/pointcloud-copc/README.md index 49673f9..a6645e2 100644 --- a/plugins/pointcloud-copc/README.md +++ b/plugins/pointcloud-copc/README.md @@ -27,7 +27,10 @@ while decoded points pass through a bounded buffer. Remote COPC is supported only when the active resolver supplies an asset with efficient random-access reads, such as HTTP byte-range support. The plugin does not implement an HTTP client, transport retries, or a network cache. Remote tiled reads require an -absolute local `payloadDirectory`. +absolute local `payloadDirectory`. Each tiled layer publishes its payloads in +a directory of its own below it, named by a hash, so no resolver identifier is +written. See +[payload ownership](../../docs/architecture/FILE_FORMAT_ARGUMENTS.md#generated-payload-ownership). Generated-USDC cache lookup is enabled for a stable local filesystem identity and for a `Stable` resolver identity. `Unstable` and `Unavailable` identity diff --git a/plugins/pointcloud-copc/src/UsdGeoCopcFileFormat.cpp b/plugins/pointcloud-copc/src/UsdGeoCopcFileFormat.cpp index 627ed02..887307b 100644 --- a/plugins/pointcloud-copc/src/UsdGeoCopcFileFormat.cpp +++ b/plugins/pointcloud-copc/src/UsdGeoCopcFileFormat.cpp @@ -16,7 +16,6 @@ #include #include #include -#include #include #include @@ -572,13 +571,6 @@ bool UsdGeoCopcFileFormat::Read(SdfLayer* layer, return false; } - auto stage = usdgeo::PointCloudLayer::CreateStage(); - if (!stage || !pxr::UsdGeomSetStageUpAxis( - stage, pxr::TfToken(reference.stageUpAxis)) || - !pxr::UsdGeomSetStageMetersPerUnit(stage, 1.0)) { - TF_RUNTIME_ERROR("%s", usdgeocopc::diagnostics::PointCloudAuthorFailed); - return false; - } std::filesystem::path payloadDirectory(request.payloadDirectory); if (payloadDirectory.is_relative()) { if (!IsLocalFileSource(resolvedPath)) { @@ -594,15 +586,22 @@ bool UsdGeoCopcFileFormat::Read(SdfLayer* layer, std::filesystem::path(resolvedPath).parent_path() / payloadDirectory; } - const usdgeo::PointCloudPayloadOptions payloadOptions{ + usdgeo::PointCloudPayloadOptions payloadOptions{ payloadDirectory.string(), resolvedPath, request.tileMemoryLimitBytes}; + payloadOptions.owner = usdgeo::PointCloudPayloadOwner( + resolvedPath, payloadDirectory, layer->GetFileFormatArguments()); if (!usdgeo::AuthorPointCloudTiledAssetWithPayloads( - stage, "/PointCloud", tiles, payloadOptions)) { - TF_RUNTIME_ERROR("%s", usdgeocopc::diagnostics::PointCloudAuthorFailed); + layer, "/PointCloud", tiles, payloadOptions, diagnostics)) { + TF_RUNTIME_ERROR("%s", usdgeocopc::diagnostics::Message( + usdgeocopc::diagnostics::PointCloudAuthorFailed, + "Unable to author tiled COPC point cloud: " + + DiagnosticDetail( + diagnostics, + "tiled authoring failed")) + .c_str()); return false; } - layer->TransferContent(stage->GetRootLayer()); return true; } diff --git a/plugins/pointcloud-copc/tests/test_pointcloud_copc.cpp b/plugins/pointcloud-copc/tests/test_pointcloud_copc.cpp index 3aab50c..e67a7a8 100644 --- a/plugins/pointcloud-copc/tests/test_pointcloud_copc.cpp +++ b/plugins/pointcloud-copc/tests/test_pointcloud_copc.cpp @@ -9,6 +9,7 @@ #include #include #include +#include #include #include #include @@ -22,6 +23,8 @@ #include #include +#include +#include #include #include #include @@ -680,6 +683,69 @@ void TestAuthoredLodEquivalence() { std::filesystem::remove(copcPath, error); } +class WarningCounter final : public pxr::TfDiagnosticMgr::Delegate { +public: + WarningCounter() { pxr::TfDiagnosticMgr::GetInstance().AddDelegate(this); } + ~WarningCounter() override { + pxr::TfDiagnosticMgr::GetInstance().RemoveDelegate(this); + } + + void IssueError(const pxr::TfError&) override {} + void IssueFatalError(const pxr::TfCallContext&, + const std::string&) override { + _UnhandledAbort(); + } + void IssueStatus(const pxr::TfStatus&) override {} + void IssueWarning(const pxr::TfWarning&) override { ++count_; } + + int Count() const noexcept { return count_; } + +private: + std::atomic_int count_{0}; +}; + +void TestTiledReadReopens() { + const auto directory = + std::filesystem::temp_directory_path() / + ("usd_copc_tiled_reopen_" + + std::to_string( + std::chrono::steady_clock::now().time_since_epoch().count())); + std::filesystem::remove_all(directory); + std::filesystem::create_directories(directory); + const auto source = directory / "conformance.copc"; + Check(std::filesystem::copy_file( + std::filesystem::path(USDGEOCOPC_SOURCE_DIR) / "tests" / + "fixtures" / "conformance.copc", + source), + "copy COPC fixture"); + + // The second open reads the layer again once the first is released, over + // the payloads the first open generated. Neither open may resolve those + // payloads while the layer is being built. + const pxr::SdfLayer::FileFormatArguments arguments = { + {"payloadDirectory", "payloads"}, {"tile", "true"}, {"tileSize", "1"}}; + for (int open = 0; open < 2; ++open) { + const WarningCounter warnings; + const auto layer = pxr::SdfLayer::FindOrOpen(source.string(), arguments); + Check(static_cast(layer), "tiled COPC layer opens"); + Check(layer->GetPrimAtPath(pxr::SdfPath("/PointCloud/Tiles")) != nullptr, + "tiled COPC layer authors tiles"); + Check(warnings.Count() == 0, + "tiled COPC read resolved payloads while authoring"); + } + Check(!pxr::SdfLayer::Find(source.string(), arguments), + "tiled COPC layer was released"); + bool hasPayload = false; + for (const auto& entry : std::filesystem::recursive_directory_iterator( + directory / "payloads")) { + hasPayload = hasPayload || + entry.path().filename() == "Tile_L0_p0_p0_p0_LOD0.usdc"; + } + Check(hasPayload, "tiled COPC payload exists"); + std::error_code error; + std::filesystem::remove_all(directory, error); +} + void TestResolverBackedRead() { const auto records = MakeEquivalentRecords(); const auto lazPath = std::filesystem::temp_directory_path() / @@ -981,6 +1047,7 @@ int main() { "plugInfo.json"); TestMetadataIntegration(); TestAuthoredLodEquivalence(); + TestTiledReadReopens(); TestResolverBackedRead(); return 0; } diff --git a/plugins/pointcloud-las/README.md b/plugins/pointcloud-las/README.md index a81908a..51afd43 100644 --- a/plugins/pointcloud-las/README.md +++ b/plugins/pointcloud-las/README.md @@ -63,7 +63,7 @@ canonical map that participates in layer identity. | `tile` | `true` | Routes the pull stream into source-coordinate tile payloads | | `tileSize` | positive source units | Fixed-grid tile width and depth | | `tileMemoryLimit` | positive bytes | Per-tile spool buffer limit | -| `payloadDirectory` | path | Directory for generated USDC payloads | +| `payloadDirectory` | path | Directory for generated USDC payloads; each layer publishes its payloads in a directory of its own below it, so reading the layer again never collides with other layers or files; see [payload ownership](../../docs/architecture/FILE_FORMAT_ARGUMENTS.md#generated-payload-ownership) | ```bash usdcat "sample.las:SDF_FORMAT_ARGS:lod=balanced&attributes=xyz,rgb" diff --git a/plugins/pointcloud-las/src/UsdGeoLasFileFormat.cpp b/plugins/pointcloud-las/src/UsdGeoLasFileFormat.cpp index b8755b4..2b81e01 100644 --- a/plugins/pointcloud-las/src/UsdGeoLasFileFormat.cpp +++ b/plugins/pointcloud-las/src/UsdGeoLasFileFormat.cpp @@ -299,10 +299,18 @@ bool UsdGeoLasFileFormat::Read(SdfLayer* layer, usdgeo::PointCloudPayloadOptions payloadOptions{ payloadDirectory.string(), rootLayerPath, request.tileMemoryLimitBytes}; + payloadOptions.owner = usdgeo::PointCloudPayloadOwner( + resolvedPath, payloadDirectory, layer->GetFileFormatArguments()); if (!usdgeo::AuthorPointCloudTiledAssetFromStream( layer, "/PointCloud", selected, reference, {request.tileSize, 0}, payloadOptions, diagnostics)) { - TF_RUNTIME_ERROR("%s", usdgeolas::diagnostics::PointCloudAuthorFailed); + TF_RUNTIME_ERROR("%s", usdgeolas::diagnostics::Message( + usdgeolas::diagnostics::PointCloudAuthorFailed, + "Unable to author tiled LAS point cloud: " + + DiagnosticDetail( + diagnostics, + "tiled authoring failed")) + .c_str()); return false; } return true; diff --git a/plugins/pointcloud-las/tests/test_pointcloud_las.cpp b/plugins/pointcloud-las/tests/test_pointcloud_las.cpp index 54ab60e..8300f05 100644 --- a/plugins/pointcloud-las/tests/test_pointcloud_las.cpp +++ b/plugins/pointcloud-las/tests/test_pointcloud_las.cpp @@ -91,6 +91,20 @@ std::vector MakeFixture() { return bytes; } +// Tiled reads publish their payloads in a directory of their own below the +// requested one, so tests find a payload by name. +bool HasPayload(const std::filesystem::path& directory, const std::string& name) { + std::error_code error; + for (std::filesystem::recursive_directory_iterator entry(directory, error), + end; + !error && entry != end; entry.increment(error)) { + if (entry->path().filename() == name) { + return true; + } + } + return false; +} + void TestFileFormatIntegration() { const auto plugInfo = std::filesystem::path(USDGEOLAS_SOURCE_DIR) / "plugin" / "resources" / "pointcloud-las" / @@ -203,21 +217,26 @@ void TestFileFormatIntegration() { {"tile", "true"}, {"tileMemoryLimit", "1"}, {"tileSize", "1"}}; - const auto tiledLayer = pxr::SdfLayer::FindOrOpen( - path.string(), tiledArguments); - Check(tiledLayer); - const auto tiledStage = pxr::UsdStage::Open(tiledLayer); - Check(tiledStage); - Check(tiledStage->GetPrimAtPath(pxr::SdfPath( - "/PointCloud/Tiles/Tile_L0_p1000_p2000_p0/LOD0")) - .IsValid()); - Check(tiledStage->GetPrimAtPath(pxr::SdfPath( - "/PointCloud/Tiles/Tile_L0_p1001_p2001_p0/LOD0")) - .IsValid()); - Check(std::filesystem::exists( - tiledPayloadDirectory / "Tile_L0_p1000_p2000_p0_LOD0.usdc")); - Check(std::filesystem::exists( - tiledPayloadDirectory / "Tile_L0_p1001_p2001_p0_LOD0.usdc")); + // The second open reads the layer again once the first is released, over + // the payloads the first open generated. + for (int open = 0; open < 2; ++open) { + const auto tiledLayer = pxr::SdfLayer::FindOrOpen( + path.string(), tiledArguments); + Check(tiledLayer); + const auto tiledStage = pxr::UsdStage::Open(tiledLayer); + Check(tiledStage); + Check(tiledStage->GetPrimAtPath(pxr::SdfPath( + "/PointCloud/Tiles/Tile_L0_p1000_p2000_p0/LOD0")) + .IsValid()); + Check(tiledStage->GetPrimAtPath(pxr::SdfPath( + "/PointCloud/Tiles/Tile_L0_p1001_p2001_p0/LOD0")) + .IsValid()); + Check(HasPayload(tiledPayloadDirectory, + "Tile_L0_p1000_p2000_p0_LOD0.usdc")); + Check(HasPayload(tiledPayloadDirectory, + "Tile_L0_p1001_p2001_p0_LOD0.usdc")); + } + Check(!pxr::SdfLayer::Find(path.string(), tiledArguments)); std::filesystem::remove_all(tiledPayloadDirectory); const auto metadataLayer = diff --git a/plugins/pointcloud-laz/README.md b/plugins/pointcloud-laz/README.md index b0c2c6c..0a97a2c 100644 --- a/plugins/pointcloud-laz/README.md +++ b/plugins/pointcloud-laz/README.md @@ -66,7 +66,7 @@ layer-identity participation. | `tile` | `true` | Routes the pull stream into source-coordinate tile payloads | | `tileSize` | positive source units | Fixed-grid tile width and depth | | `tileMemoryLimit` | positive bytes | Per-tile spool buffer limit | -| `payloadDirectory` | path | Directory for generated USDC payloads | +| `payloadDirectory` | path | Directory for generated USDC payloads; each layer publishes its payloads in a directory of its own below it, so reading the layer again never collides with other layers or files; see [payload ownership](../../docs/architecture/FILE_FORMAT_ARGUMENTS.md#generated-payload-ownership) | ```bash usdcat "sample.laz:SDF_FORMAT_ARGS:lod=preview" diff --git a/plugins/pointcloud-laz/src/UsdGeoLazFileFormat.cpp b/plugins/pointcloud-laz/src/UsdGeoLazFileFormat.cpp index 32d7668..0f4b365 100644 --- a/plugins/pointcloud-laz/src/UsdGeoLazFileFormat.cpp +++ b/plugins/pointcloud-laz/src/UsdGeoLazFileFormat.cpp @@ -279,10 +279,18 @@ bool UsdGeoLazFileFormat::Read(SdfLayer* layer, usdgeo::PointCloudPayloadOptions payloadOptions{ payloadDirectory.string(), rootLayerPath, request.tileMemoryLimitBytes}; + payloadOptions.owner = usdgeo::PointCloudPayloadOwner( + resolvedPath, payloadDirectory, layer->GetFileFormatArguments()); if (!usdgeo::AuthorPointCloudTiledAssetFromStream( layer, "/PointCloud", selected, reference, {request.tileSize, 0}, payloadOptions, diagnostics)) { - TF_RUNTIME_ERROR("%s", usdgeolaz::diagnostics::PointCloudAuthorFailed); + TF_RUNTIME_ERROR("%s", usdgeolaz::diagnostics::Message( + usdgeolaz::diagnostics::PointCloudAuthorFailed, + "Unable to author tiled LAZ point cloud: " + + DiagnosticDetail( + diagnostics, + "tiled authoring failed")) + .c_str()); return false; } return true; diff --git a/plugins/pointcloud-ply/README.md b/plugins/pointcloud-ply/README.md index 14f7dea..d216776 100644 --- a/plugins/pointcloud-ply/README.md +++ b/plugins/pointcloud-ply/README.md @@ -15,7 +15,9 @@ contracts, and requires an explicit CRS because PLY has no embedded CRS model. The `epsg` file-format argument is required. `linearUnit`, `sourceUpAxis`, and `stageUpAxis` are optional. Tiled reads use `tileSize`, `tileMemoryLimit`, and -`payloadDirectory`. +`payloadDirectory`, below which each layer publishes its payloads in a +directory of its own. See +[payload ownership](../../docs/architecture/FILE_FORMAT_ARGUMENTS.md#generated-payload-ownership). Faces, mesh authoring, PLY writing, and metadata-only reads are out of scope. See the [capability matrix](../../docs/reference/CAPABILITY_MATRIX.md) and diff --git a/plugins/pointcloud-ply/src/UsdGeoPlyFileFormat.cpp b/plugins/pointcloud-ply/src/UsdGeoPlyFileFormat.cpp index 77aa183..268b6ef 100644 --- a/plugins/pointcloud-ply/src/UsdGeoPlyFileFormat.cpp +++ b/plugins/pointcloud-ply/src/UsdGeoPlyFileFormat.cpp @@ -207,6 +207,8 @@ bool UsdGeoPlyFileFormat::Read(SdfLayer* layer, usdgeo::PointCloudPayloadOptions payloadOptions{ payloadDirectory.string(), resolvedPath, request.tileMemoryLimitBytes}; + payloadOptions.owner = usdgeo::PointCloudPayloadOwner( + resolvedPath, payloadDirectory, layer->GetFileFormatArguments()); if (!usdgeo::AuthorPointCloudTiledAssetFromStream( layer, "/PointCloud", selected, reference, {request.tileSize, 0}, payloadOptions, diagnostics)) { diff --git a/plugins/pointcloud-ply/tests/test_pointcloud_ply.cpp b/plugins/pointcloud-ply/tests/test_pointcloud_ply.cpp index 3d732f6..cc4ccf6 100644 --- a/plugins/pointcloud-ply/tests/test_pointcloud_ply.cpp +++ b/plugins/pointcloud-ply/tests/test_pointcloud_ply.cpp @@ -103,8 +103,15 @@ void TestFileFormatIntegration() { Check(tiledStage->GetPrimAtPath(pxr::SdfPath( "/PointCloud/Tiles/Tile_L0_p0_p0_p0/LOD0")) .IsValid()); - Check(std::filesystem::exists( - tiledPayloadDirectory / "Tile_L0_p0_p0_p0_LOD0.usdc")); + // The layer publishes its payloads in a directory of its own below the + // requested one. + bool hasPayload = false; + for (const auto& entry : std::filesystem::recursive_directory_iterator( + tiledPayloadDirectory)) { + hasPayload = hasPayload || + entry.path().filename() == "Tile_L0_p0_p0_p0_LOD0.usdc"; + } + Check(hasPayload); std::filesystem::remove_all(tiledPayloadDirectory); }