Skip to content

feat: make generated payload ownership explicit - #146

Merged
snkmcb merged 2 commits into
mainfrom
feat/generated-payload-ownership
Sep 12, 2026
Merged

snkmcb merged 2 commits into
mainfrom
feat/generated-payload-ownership

Conversation

@snkmcb

@snkmcb snkmcb commented Sep 12, 2026 •

Copy link
Copy Markdown
Member

Summary

Implements the "reduce temporary generated-layer ownership" direction from the roadmap and ecosystem strategy, which also asks that ownership, crash recovery, and test isolation of generated files be explicit. The second commit reworks the first commit's design after code review; see below.

  • Tiled FileFormat reads can be read again. Before, reading a tile=true layer a second time (new process, or after the first layer was released) failed with LAS016 / COPC007 because the earlier payloads already existed.
  • No more Could not open asset in tiled COPC reads.
    • Layer content is built in in-memory stages opened with UsdStage::LoadNone, so a read never resolves the payloads it is writing.
    • The ScopedLayerIdentifier / .usdgeo-stream-N identifier rewriting is removed.
    • Payloads are exported from in-memory layers instead of being created under their final path.
  • Each layer owns its payload directory. Owned payloads live in <payloadDirectory>/<owner>/<generation>/.
    • <owner> is a hash of the source (for local files, its path relative to the payload directory, both canonicalized) plus the layer's exact file-format arguments. Layers never share or touch each other's files, and a project that is moved or mounted elsewhere keeps its payloads.
    • Each generation is written into a unique .tmp-<token> staging directory and published by rename. A failed or cancelled read removes only its staging directory, and a published generation is never modified.
    • Identical content reuses the published g-<content hash> generation. Changed content is published beside it, and the root points at the new paths, so layers still registered from the old generation never become stale. The superseded generation is removed on a best-effort basis.
    • Staging left by an interrupted process is swept after an hour without writes.
  • Cache hits materialize as c-<entry key>. An intact copy is reused without writing, so a published directory may be read-only; a damaged copy is replaced. No byte comparisons and no adoption.
  • The converter is unchanged. It uses exclusive flat mode: its output must not exist yet, and its transaction marker owns recovery.
  • Thinner COPC adapter. It uses a new layer-level AuthorPointCloudTiledAssetWithPayloads overload that returns typed diagnostics.
  • Errors carry reasons. Tiled author failures in LAS, LAZ, and COPC now include the reason next to the plugin code.
  • macOS build fix. The stream benchmark now builds on macOS, where uintmax_t and uint64_t differ. CI does not build benchmarks.
  • Docs. The contract is in docs/architecture/FILE_FORMAT_ARGUMENTS.md, "Generated Payload Ownership". Capability matrix, implementation status, streaming roadmap, OpenUSD surface, adapter and workspace contracts, module READMEs, and CHANGELOG are updated to match.

Review follow-up (second commit)

The first commit recorded ownership per payload name in a shared flat directory. Review found that this let:

  • a failed re-read delete payloads a root still referenced;
  • concurrent reads race on fixed .tmp names;
  • stale or adopted records reach another layer's files;
  • std::to_string-rounded arguments collapse distinct layers onto one owner;
  • every cache hit need write access and a byte comparison.

Per-owner, immutable, rename-published generations remove those cases structurally, together with the record file, the adoption rule, and the in-place replacement.

Compatibility

Payload files that earlier tiled reads wrote directly into payloadDirectory are no longer used or touched. Delete the old Tile_*.usdc files to reclaim the space.

Follow-up

Tracked as an open item in implementation status: tile spools still live in timestamped system-temp directories that an interrupted process leaves behind.

Validation (macOS arm64, cy2026/usd)

  • ost build + ost test: 18/18
  • Scratch build with USDGEO_BUILD_BENCHMARKS=ON: 25/25
  • Unit tests:
    • generation published under the owner, and arcs point into it
    • identical content reuses the generation without rewriting files
    • changed content publishes a new generation and removes the old
    • other owners and user files are untouched
    • exclusive mode still refuses existing files
    • stale staging is swept while live staging is kept
    • a damaged generation is replaced
    • a cancelled regeneration leaves the published generation byte-identical
    • a cancelled first generation removes the directories it created
    • detached authoring emits no warnings
  • Cache tests: c-<entry> copy is reused without writing, replaced when damaged, and user files are untouched
  • Integration tests: LAS and COPC tiled layers reopened after release; PLY payload lookup; COPC reads emit no warnings
  • Manual usdcat (LAS, COPC):
    • separate processes publish the same g-<hash> with identical arcs (usdc output is deterministic)
    • different arguments coexist
    • a moved project keeps its owner and generation
    • a changed source swaps generations and leaves no staging behind
  • git diff --check

Windows and Linux are left to CI.

🤖 Generated with Claude Code

snkmcb and others added 2 commits September 12, 2026 13:19
Tiled FileFormat reads wrote their payloads with no owner, so the same layer
could not be read a second time: the payloads from the first read refused
the write (LAS016, COPC007). Layer authoring also built content in scratch
stages that loaded every payload and borrowed file identities to anchor
them, which made tiled COPC reads warn "Could not open asset".

- Build layer content in in-memory stages opened with LoadNone and export
  payloads from in-memory layers; drop the layer identifier rewriting.
- Add a layer-level AuthorPointCloudTiledAssetWithPayloads overload with
  typed diagnostics and move the COPC adapter onto it.
- Record payload ownership per layer identity in
  payload-owner-<hash>.manifest: a read replaces the payloads its layer
  generated before, removes superseded ones, and refuses any other file.
  Ownership is claimed before the first write and restored on failure.
- Apply the same rule when materializing a generated-cache hit, taking
  over files that already hold the cached bytes.
- Report the reason behind tiled author failures in LAS, LAZ, and COPC.
- Fix the stream benchmark build on macOS.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
The review of the ownership record found that sharing one flat set of
payload names between layers let a failed re-read delete payloads a root
still referenced, let concurrent reads and stale or adopted records reach
another layer's files, collapsed near-equal arguments onto one owner, and
required writes on every cache hit.

- Publish owned payloads in <payloadDirectory>/<owner>/<generation>/.
  The owner hashes the source, relative to the payload directory for local
  files, and the layer's exact file-format arguments, so layers never meet
  and a relocated project keeps its payloads.
- Stage each generation in a unique private directory and publish it by
  rename. A failure removes only the staging directory, identical content
  reuses the published generation (g-<content hash>), and changed content is
  published beside it before the superseded one is removed. Stale staging
  left by an interrupted process is swept after an hour.
- Materialize cache hits as c-<entry key>, reusing an intact copy without
  writing and replacing a damaged one.
- Drop the ownership record, adoption, byte comparison, and in-place
  replacement; derive payload names in one place and append manifest
  entries only after the generation is published.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
@snkmcb
snkmcb merged commit 11663ea into main Sep 12, 2026
4 checks passed
@snkmcb
snkmcb deleted the feat/generated-payload-ownership branch September 12, 2026 10:25
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant