Skip to content

feat(generator): Image-to-Splat tab (SHARP) + live Spark splat viewer#1650

Draft
kfarr wants to merge 38 commits into
mainfrom
claude/gifted-cray-bBwO7
Draft

feat(generator): Image-to-Splat tab (SHARP) + live Spark splat viewer#1650
kfarr wants to merge 38 commits into
mainfrom
claude/gifted-cray-bBwO7

Conversation

@kfarr
Copy link
Copy Markdown
Collaborator

@kfarr kfarr commented May 28, 2026

Summary

Ships basic splat support end to end, built on a generalized async generation job queue:

  1. Splats as a first-class asset type.ply/.splat/.spz are recognized on upload, stored as application/octet-stream, rendered via Spark, previewable in a live viewer, and draggable into a scene exactly like a mesh.
  2. Image-to-Splat in the AI Generator — a new Splat tab turns a single photo into a 3D Gaussian Splat (.ply) via the SHARP model (kfarr/sharp-ml) on Replicate, saved server-side to the user's gallery.
  3. A provider-agnostic generationJobs queue underneath it — durable, browser-independent completion (webhook + client poll + scheduled reconciler), atomic token charge with exactly-once refund. Splat is the first consumer; Replicate image/video, fal, and Teleport/Varjo photogrammetry drop in as additional kinds/providers without re-architecting.

Full design: docs/generation-job-queue.md.

What's included

Generalized job queue (Phase 0 + Phase 1)

  • users/{uid}/generationJobs/{jobId} collection — internal uuid jobId as the doc id (provider's prediction id is providerJobId), normalized status vocabulary (queued|running|saving|succeeded|failed|canceled), kind/provider discriminators, per-job webhookSecret. Admin-SDK-only via firestore.rules.
  • Three convergent, idempotent completion paths funnel into one processor (processTerminalPrediction): Replicate webhook (replicateJobWebhook), client poll (getGenerationJobStatus), and a scheduled reconciler (reconcileGenerationJobs, every 10 min). A transactional status → 'saving' claim (TTL-bounded) prevents double-save/double-charge across racing paths.
  • Reconciler (public/functions/scheduled/generation-job-reconcile.js) closes the "webhook dropped + tab closed → token charged but result lost" gap and re-triggers jobs wedged in saving. Give-up rule: non-terminal + >30 min + provider failed/absent → mark failed, refund once. Admin-only triggerReconcileGenerationJobs for manual/dry-run sweeps.
  • Server-side persist streams the .ply from Replicate's CDN straight into the Storage write stream — memory stays flat regardless of splat size.

Image-to-Splat (Generator)

  • generateReplicateSplat callable — create-and-return (SHARP can cold-boot for minutes, so we never hold the callable open), atomic 1 genToken charge, refunded once on failure.
  • Self-contained SplatTab (src/generator/splat.js) with submit → poll loop → gallery refresh, and out-of-tokens button state.
  • sharp-ml registered in public/functions/replicate-models.js.

Splats as first-class assets

  • getAssetKind / FILE_PICKER_ACCEPT / SPLAT_MAX_BYTES recognize .ply/.splat/.spz; assetsService.addAsset forces application/octet-stream and preserves the real extension (the splat loader selects by extension).
  • Gallery cards render a point-cloud placeholder, are draggable into the scene, and emit the splat component (bare src:) instead of gltf-model.
  • Splat/mesh download fix — gallery downloads now preserve the real binary extension (.ply/.splat/.spz/.glb) instead of forcing .png, which previously corrupted the downloaded file. (useAssets.downloadItem, type-aware notification in AssetsContent.)

Live splat viewer

  • public/splat-viewer.html — standalone Spark + three.js viewer (?src=), OrbitControls, auto-frame to bounds, isolated import map. Embedded as an iframe in the Splat tab result panel and the gallery details modal (MeshDetailsModal now serves both meshes and splats).

Status

  • Phase 0 (durable schema/contracts) — done
  • Phase 1 (generic reconciler / robustness backstop) — done (this PR)
  • ⏭️ Phase 2 (extract shared jobs/ framework) — next, pure refactor, splat stays sole consumer
  • Later: fold Replicate image in as a kind (Phase 3), notifications (Phase 4)

Deploy notes (three separate targets)

  • Rules: firebase deploy --only firestore:rules,storage (job-doc lockdown + 100 MB octet-stream cap for generated .ply).
  • Functions (deploy separately from hosting): generateReplicateSplat, getGenerationJobStatus, replicateJobWebhook, reconcileGenerationJobs, triggerReconcileGenerationJobs.
  • Hosting: generator bundle + public/splat-viewer.html (via npm run deploy[:staging]).
  • Secrets: none new — reuses REPLICATE_API_TOKEN.

Roadmap

  • Teleport / Varjo photogrammetry (zip/video → splat) is the fast-follow, gated only on API keys — exercises the queue's large-source-upload + cost-hold paths.
  • Target output format: World Labs / sparkjsdev RAD format with LOD, as a practical replacement for Google 3D Tiles for real-world context. The .ply → RAD conversion slots into the queue's persist adapter for kind: 'splat'.

🤖 Generated with Claude Code

claude and others added 6 commits May 28, 2026 19:13
Adds a new "Splat" tab to the AI Generator that turns a single photo into
a 3D Gaussian Splat (.ply) using the SHARP model (kfarr/sharp-ml) on
Replicate. SHARP completes in ~4 min, so this reuses the existing
synchronous "await the callable" pattern — no async job queue.

Generator/cloud:
- New generateReplicateSplat callable Cloud Function (mirrors image gen:
  auth + token check, stages input image, runs SHARP, atomic token
  deduction, generation/token audit logs). Cost: 1 genToken.
- New SplatTab module + tab nav/content wiring; out-of-tokens button state.

Asset layer (Phase 0 — splats become first-class assets):
- Recognize .ply/.splat/.spz uploads (getAssetKind, FILE_PICKER_ACCEPT,
  SPLAT_MAX_BYTES kept in lockstep with the 50MB octet-stream storage rule).
- assetsService.addAsset: force octet-stream MIME, preserve real extension
  from the source filename, re-wrap blobs so the upload content type passes
  storage rules.
- Gallery cards render a point-cloud placeholder and are draggable into the
  scene; getAssetTypeLabel returns "Splat".
- placeCloudAsset / editor drop swap to the `splat` component (bare `src:`)
  instead of gltf-model, so generated/uploaded splats place like meshes.

Splat rendering (Spark) and the SPLAT asset type/category already existed.

Deferred to v2 (designed in docs/splat-generation.md, not built): Teleport
photogrammetry — large-file upload, cost estimation, and an async
splatJobs queue.
Adds public/splat-viewer.html — a standalone Spark + three.js viewer
(mirrors model-viewer.html) that takes ?src=<splatUrl>, renders with
OrbitControls, and auto-frames the camera to the splat bounds. Pinned to
the bundled three (0.180.0) and Spark (2.0.0) via an import map, hosted in
its own document so its THREE copy doesn't collide with window.THREE.

Embedded as an iframe in:
- the Generator Splat tab result panel (live preview of the new splat)
- the gallery details modal — MeshDetailsModal now serves meshes AND splats,
  choosing the viewer page (model-viewer vs splat-viewer) and type label by
  asset type; AssetsContent routes splat clicks to it.
Uncompressed SHARP .ply output can exceed the previous 50MB
application/octet-stream cap in storage.rules, which would reject the
generator's saveToGallery upload. Raise the server ceiling to 100MB.

User-initiated drag-and-drop uploads stay capped at 50MB client-side
(SPLAT_MAX_BYTES) — generated splats save via assetsService.addAsset and
are gated only by the 100MB rule. Updates docs (storage.rules now requires
deploy) and notes compression (.spz/.ksplat) as the long-term fix.
Move splat generation onto a provider-agnostic async job queue so jobs
survive a closed browser and complete server-side.

- New users/{uid}/generationJobs/{jobId} collection: internal uuid jobId
  (Replicate prediction id stored as providerJobId), normalized status enum
  (queued|running|saving|succeeded|failed|canceled) with raw providerStatus,
  and kind/provider fields for future kinds. Pending doc written before submit.
- generateReplicateSplat writes the pending job and returns jobId immediately;
  completion is handled by replicateJobWebhook (browser-independent) and
  getGenerationJobStatus (poll/fallback), both funneling into one idempotent
  processTerminalPrediction.
- Save claim is time-bounded (savingStartedAt + SAVING_CLAIM_TTL_MS) so a
  crashed save self-heals instead of wedging in 'saving'. Token charged on
  submit, refunded exactly once on failure.
- saveSplatToGallery streams the .ply into Storage (no full-file buffering);
  the two completion functions run at 512MB. Fixes a 256MB OOM that left jobs
  stuck mid-save.
- firestore.rules: lock down generationJobs (Admin-SDK only; protects the
  per-job webhookSecret).
- splat.js: poll loop keyed by jobId, dark preview stage, close-tab-safe copy.
- docs: rename splat-generation.md -> generation-job-queue.md and rewrite as
  the queue design + phased plan.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
kfarr added 7 commits May 30, 2026 22:12
Generation-job reconciler (the dropped-webhook backstop):
- New scheduled/generation-job-reconcile.js — sweeps non-terminal
  generationJobs (skipping fresh ones to avoid racing live webhooks/polls)
  and runs the same idempotent processTerminalPrediction the webhook/poll
  use, so no double-save or double-charge. Gives up + refunds once on jobs
  >30min where the provider also reports failed/absent. Every-10-min PubSub
  schedule + admin-only triggerReconcileGenerationJobs (dryRun default).
- Export processTerminalPrediction/refundSplatToken/cleanupSplatTempFile/
  normalizeReplicateStatus from replicate.js; register both functions.

Fix gallery download naming:
- Splats/meshes kept their real binary extension (.ply/.splat/.spz/.glb)
  instead of being forced to .png, which corrupted the downloaded file.
- Images/videos now derive the extension from the fetched blob's actual
  Content-Type (jpg/webp/gif/avif/mp4/webm/mov), with a widened metadata
  fallback. Previously only the literal 'jpeg' mapped correctly; 'jpg',
  'webp', and unset formats all fell through to .png.
- Type-aware download toast in AssetsContent.

Docs: mark Phase 1 done, Phase 2 next; Teleport reframed as fast-follow;
add World Labs/sparkjsdev RAD+LOD as the target output format (3D Tiles
replacement).
Note that the World Labs / sparkjsdev RAD+LOD output maps onto the existing
GLB original/optimized asset schema (storageUrl = raw .ply, optimizedSource*
= RAD variant) with no schema change, and is reprocessable by rewriting only
the optimized variant.
getGenerationJobStatus and replicateJobWebhook ran at the 60s default. The
streamed .ply save (download from Replicate CDN -> resumable upload to
Storage) can exceed that for a large splat; a kill mid-save never releases
the 'saving' claim and wedges the job. Give the save real headroom (the
stale-claim TTL + reconciler are only the backstop).
The reconciler's collection-group where('status','in',...) sweep was failing
every run with FAILED_PRECONDITION (needs a COLLECTION_GROUP_ASC index), so
stuck jobs were never cleaned up. The gallery's per-user pending-job listener
uses the COLLECTION-scope variant. Declare both scopes (ASC+DESC).
Show in-flight generation jobs (image->splat) as pending cards near the top
of the asset gallery, sourced from a live onSnapshot listener on the user's
generationJobs (the job doc is the source of truth), so they survive a reload
and appear across tabs. A fresh upload keeps the upper-left slot; jobs sit
just after it, ahead of finished assets. When a job leaves the non-terminal
set the grid refreshes, so the card turns into the real asset.

- useAssets: live pendingJobs subscription + auto-refresh on completion
- PendingJobCard: spinner + status label + indeterminate bar (no source
  thumbnail by design)
- firestore.rules: owner-read on generationJobs (writes stay admin-only).
  Safe: read is uid-scoped so the webhookSecret stays confidential from
  third parties; an owner can only re-trigger idempotent processing of their
  own job.
…essing UI

- Drop the transient client-side job store; the gallery's Firestore listener
  now drives the pending card, so it persists across reloads.
- Distinguish the upload phase from server processing: show 'Uploading image...'
  (no 'you can close this tab' text, no timer) while the source image is being
  sent, and only switch to 'Generating splat...' + timer + close-tab message
  once the job is actually queued server-side.
Dev authDomain was the default dev-3dstreet.firebaseapp.com, a different site
from the app origin dev-3dstreet.web.app, so the signInWithPopup handshake
relied on third-party cookies/storage — which mobile Chrome now blocks,
breaking login. Point authDomain at the app's own hosting domain (same-site,
first-party), mirroring prod's use of 3dstreet.app. The auth handler is served
at dev-3dstreet.web.app/__/auth/ by Firebase Hosting; the OAuth client already
authorizes that JS origin + redirect URI.
@kfarr
Copy link
Copy Markdown
Collaborator Author

kfarr commented Jun 1, 2026

feature punch list:

  • preview thumbnail - how to generate a preview image of a splat from a ply file? is it possible?
  • cloud rad - create LOD rad file for splat streaming as "optimized" version analogous to glb optimized assets
  • file size upload limit - needs to be increased for working with splats; free: 50mb; 250mb pro, 1gb max ?
  • user notification for successfully completed queued jobs? optional email? checkbox on the same screen default to yes send email when job complete and tab is closed? always send if job token price > x or time > y?
  • model name - apple sharp-ml instead of kfarr sharp-ml
  • if splat is cached in 3dstreet editor, that same cache should be used for splat preview modal
  • what splat input formats should be supported -- only ply? spz? splat? rad?
  • background jobs - are they setup ok? meta - bg job monitoring / app monitoring in general?

qa punch list:

  • test submit a sharp-ml job and stay in the same tab, does it display correctly after ~5 min when completed?

prod deployment punch list:

  • apply cors headers "CORS headers added (you'll apply them with gsutil cors set public/cors.json gs://dev-3dstreet.appspot.com)."
  • deploy hosting + functions + updated storage rules + updated firestore rules?
  • new index must finish building before the notify sweep can query, so deploy firestore:indexes ahead of (or with) the functions.

kfarr added 13 commits May 31, 2026 22:25
…to queue)

Server-side .ply→.rad (Spark LOD) conversion as the splat analog of the GLB
optimized variant. Lands in optimizedSource* on the asset doc; renderer +
client placement already prefer it, so splats stream paged .rad automatically.

- rad-converter/: Cloud Run service. Multi-stage Dockerfile compiles build-lod
  from Spark v2.1.0; Node handler downloads the .ply from GCS, runs
  build-lod --quality, uploads {assetId}-lod.rad (assetRole: optimized, mirrors
  saveSplatToGallery's URL/token scheme), patches optimizedSource* + writes the
  job's terminal status.
- onSplatAssetCreated (rad-dispatch.js): Firestore onCreate trigger on splat
  asset docs → writes a generationJobs doc {provider:'cloudrun', kind:'splat-rad',
  tokenCost:0} and enqueues a Cloud Task (OIDC) to the converter. Covers both
  uploaded and generator-saved splats.
- reconciler: case 'cloudrun' — re-enqueue a stalled job past RACE_GUARD,
  give up past GIVE_UP (refund is a no-op at tokenCost 0).
- cors.json: expose Accept-Ranges + Content-Range for byte-range streaming.
- firebase.json: stop hosting from serving functions/ source + rules publicly.

Deployed + verified end-to-end on dev-3dstreet. Config in rad-dispatch.js is
hardcoded to dev pending prod rollout.
The async job queue now has two real providers with different completion
models. Update the design doc + CLAUDE.md to cover both:

- generation-job-queue.md: provider comparison table, a dedicated "Second
  provider — Cloud Run RAD" section (worker-writeback vs convergent webhook/poll,
  no providerJobId → reconciler re-enqueues, tokenCost 0), updated the registry-
  seam note (two real cases now), corrected the bottom RAD note (built as a
  separate onCreate-triggered job, not a persistResult adapter), and the deploy
  section.
- CLAUDE.md: register onSplatAssetCreated; add an async-job-queue blurb to the
  Generator section pointing at both design docs.
- splat-viewer.html captures a downscaled JPEG once the splat settles and
  postMessages it up; MeshDetailsModal backfills it as the asset thumbnail
  (owner-only, once per asset) so blank .ply gallery cards get a real preview.
- uploadCapturedThumbnail takes an asset subfolder (meshes|splats).
- Cache-Control: public, max-age=31536000 on the generated .ply
  (saveSplatToGallery) and optimized .rad (rad-converter) uploads, matching
  client uploads — so the editor and the preview-modal iframe reuse the browser
  HTTP cache for the same URL instead of re-downloading.
- test: update size-cap test for the single MAX_FILE_BYTES ceiling.
The preview-modal viewer always used lod:true (full download before first
draw); RAD files have pre-built LOD and should stream via byte-range requests.
Branch on the .rad extension and pass paged:true, mirroring the in-editor splat
component, so the modal draws progressively/fast instead of blocking on a full
download.
Make the splat preview thumbnail robust to user interaction and RAD LOD timing:

- splat-viewer.html: snapshot the auto-framed camera, then at ~3s render THAT
  pose into an offscreen WebGLRenderTarget (reusing the already-loaded splat —
  no reload, just a second camera + render target) and postMessage the JPEG.
  Live orbiting can no longer corrupt the thumbnail, and the longer delay lets
  paged RAD stream higher-detail LOD at no UX cost.
- MeshDetailsModal: synchronous close-capture fallback. If the user closes
  before the auto capture fires, read the iframe canvas (same-origin) while the
  modal is still open and upload that lower-quality frame. Wired through the
  explicit close affordances (X / Esc / backdrop / place) via handleClose, since
  a ref read in an unmount cleanup would already be nulled. Guarded once-per-asset.
Support uploading .ply, .splat, .spz, and .rad splats:

- Add .rad to SPLAT_EXTS so pre-optimized/streamable splats (e.g. existing
  Hetzner-built .rad files) can be uploaded directly. assetsService already
  whitelists 'rad' for the stored extension, so storageUrl ends in .rad and the
  renderer streams it paged with no conversion.
- onSplatAssetCreated: skip RAD conversion when the upload is already .rad
  (nothing to optimize) — previously it would enqueue a pointless re-convert.
- .ply/.splat/.spz uploads convert as before: build-lod content-sniffs the
  bytes, so .splat/.spz work despite the converter's nominal {assetId}.ply
  scratch name (clarified in a comment). .rad never reaches the converter.
- test: getAssetKind recognizes all four splat extensions.
Notifications, built entirely on the generationJobs doc (no separate
notification system):
- Splat tab adds an "Email me when my splat is ready" checkbox (default on),
  passed as notify.email to generateReplicateSplat and stored as
  notify:{email,pending} on the job doc.
- A live poll that sees success stamps notify.clientAckedAt and clears pending
  (tab was open → user saw it), suppressing the email.
- The reconciler's sendReadyNotifications sweep emails opted-in succeeded jobs
  that went unacked past a 3-min grace (Postmark splatReady template, reusing
  sendPostmarkEmail/getUserInfo), then clears pending. At most one email per job.
  Adds a (notify.pending, status) collection-group index.

Monitoring: escalateIfNeeded logs at ERROR when a sweep gives up / errors / fails
to email, so Cloud Error Reporting can alert with no new infra. Binds
POSTMARK_API_KEY to the reconciler.

Docs: Phase 4 + monitoring marked done; adds docs/splat-manual-test-punchlist.md.
The notify mechanism (job doc + reconciler sweep + ack-on-poll) was already
kind-agnostic; only the email template was splat-specific. Generalize
splatReady → generationReady with per-kind copy (splat today; video/image reuse
it for free, falling back to neutral "generation" wording for unknown kinds), so
the next async generations need only an opt-in checkbox, no notification work.
The "your splat is ready" email CTA now links to
  https://3dstreet.app/#asset:OWNER_UID/ASSET_ID
reusing the asset-token shape from issue #1641 (asset:OWNER/ID). The owner uid is
required because assets are addressed as users/{uid}/assets/{assetId} with no
id-only lookup; the reconciler has both at send time.

New AssetDeepLinkModal (editor) watches the hash and mounts the existing
MeshDetailsModal standalone (a portal, panel-independent). MeshDetailsModal
already self-fetches by assetId+ownerUid and renders the splat viewer; assets are
public-read by default, so the splat shows even before the recipient's auth
resolves — auth only upgrades to owner mode. View-only for v1 (no place CTA).

generationReady email template now takes an optional ctaUrl (falls back to the
app link). Closing the modal strips the #asset hash.
Wire onPlace on the email→#asset deep-link modal the same way the editor's
Assets panel does (pickPointOnGroundPlane + placeCloudAsset), so a user arriving
from a "your splat is ready" email can see it and drop it straight into the
current scene. MeshDetailsModal already rendered the button behind the optional
onPlace prop; the editor-side modal just supplies the handler. Reverses the
earlier view-only call — placement turned out to be a 3-line reuse, not the
scene-context lift I'd guessed.
kfarr added 11 commits June 1, 2026 20:24
…o black thumb

Three fixes from dev testing:

1. #asset:OWNER/ID deep links were falling through set-loader-from-hash's
   catch-all to fetchJSON('asset:….json'), surfacing "Could not fetch scene" /
   "Could not connect to server". Bail early on the asset: prefix (like #mcp);
   AssetDeepLinkModal handles it in React.

2. Splat viewer now shows a model-viewer-style "drag to orbit" interaction
   prompt: a hand icon that swipes side-to-side, fading in after the splat frames
   and out on first pointer interaction (auto-retires after 6s).

3. Black-thumbnail guard: the sync close-time capture in MeshDetailsModal could
   read the viewer's still-unrendered (black) GL buffer and persist it, which
   then stuck (thumbnailUrl set → proper offscreen capture never replaces it).
   Reject a blank frame and leave the asset thumbnail-less so a later open
   backfills a real one.
The right-sidebar "Details" button routed only type==='mesh' to MeshDetailsModal;
splats (type==='splat') fell through to the image/video AssetsModal — spinner
instead of the splat, minimum-size frame, and bogus Modify/Create Video buttons.
Route splats to MeshDetailsModal too, matching AssetsContent's gallery routing.
The "which detail modal for which asset type" decision was wired by hand at three
render sites (gallery, editor sidebar Details, email deep link); two checked
'mesh' only and shipped splats to the wrong (image/video) modal. Consolidate into
one shared AssetDetailModal that:
- owns the split via is3dViewerType() (mesh/splat → MeshDetailsModal,
  image/video → AssetsModal) — one place to add a type,
- absorbs the modals' different contracts (MeshDetailsModal self-fetches by id;
  AssetsModal needs a hydrated item), fetching the doc itself when a caller has
  only identity (deep link / sidebar mid-load).

All three sites now render <AssetDetailModal/>. Bonus: the email deep link now
works for any asset type, not just splats (it fetches to learn the type).
The component grew past "upload status": in steady state it's the per-entity
asset info card in the properties sidebar (status dot, source/ownership, Details
button, filename) with upload progress only a transient phase. Rename the
component + file to match; the useAssetUploadStatus hook (shared by AssetUploadDot
and EntityLabel, which are genuinely status-only) keeps its name.
…D job

Three issues surfaced uploading a 368 MB .ply:

- Disable on-the-fly LOD for all non-RAD formats (splat component + standalone
  viewer). For large splats that build cost ~90s, is ephemeral (rebuilt every
  load), and isn't serializable — the cloud RAD pipeline bakes a streamable LOD
  variant the scene prefers on reload. Only RAD (paged) streams baked LOD.

- Don't reload the splat on the post-upload identity swap: the src flips from the
  local blob: URL to the cloud URL of the SAME file, so re-fetch + re-decode was
  pure waste (and re-ran processing). Keep the loaded mesh; a later swap to a
  different format (.rad) still reloads since its oldData.src isn't a blob.

- Stop rendering the silent splat-rad (RAD transcode, tokenCost 0) as a
  "Generating… / Asset generation" card — it read as a confusing duplicate of
  the just-uploaded splat. Still tracked for completion so the grid refreshes to
  the optimized URL; just not shown as a generation card.
… + modal)

Reframe the silent RAD/LOD transcode as a status of the asset, not a generation:
- Asset card shows a subtle "Optimizing…" badge while its splat-rad job is in
  flight (useAssets exposes optimizingAssetIds, threaded through the grid).
- Detail modal shows an "Optimization" row: "Optimizing… (status)" while a job
  runs, "Streaming variant ready (RAD/LOD)" once optimizedSourceUrl exists.
  Backed by assetsService.getAssetJobs (owner-only) — the seed of a per-asset
  transcode/variant list (one source, N jobs).
- PendingJobCard: drop the redundant top progress stripe; spinner + status only.
Top-left collided with the type/source title. Lower-center sits clear of the
title (top) and the delete/download buttons (bottom corners).
…s) + codify rad infra

Thumbnail:
- Capture the LIVE canvas instead of a one-shot offscreen render. Spark streams +
  sorts paged .rad splats against the live animation-loop camera, so the offscreen
  render (different camera) came back black for RAD files. preserveDrawingBuffer
  keeps the live canvas readable; center-crop to a square.
- Retry up to 5× (2s apart) so paged RAD has time to stream something visible.
- Blank guard is now range-based (min→max luminance) so it rejects BOTH pure
  black and the flat grey-background empty — never persists a blank thumbnail
  (splat-viewer + MeshDetailsModal sync fallback).

Infra-as-code:
- Add rad-converter/deploy.sh as the single source of truth for the Cloud Run
  sizing (16Gi/4cpu/900s/concurrency 1) AND the Cloud Tasks queue retry policy
  (3 attempts, 10s–300s backoff). These were ephemeral gcloud state; now they're
  committed + reproducible. README points at the script and stops claiming the
  stale 8Gi/2cpu numbers.
Owner-only escape hatch for a bad/blank auto-thumbnail (splats): clears
thumbnailUrl, re-arms the capture path, and reloads the viewer iframe (nonce
bump) so it re-captures. Closes the loop on thumbnails — any bad capture is now
one click to fix.
…mistic scene loading

- resolveSplatAssetUrls: on scene load, re-resolve splat entities that carry an
  asset identity to getServedUrl (optimizedSourceUrl ?? storageUrl), so saved
  scenes stream the cloud RAD/LOD variant instead of the raw .ply baked in at
  placement time. Covers both load paths (viewer + editor).
- LoadingSceneModal: optimistic dismiss on timeout instead of erroring; only
  genuine fetch/parse failures surface the error modal (fetchJSON onload now
  wrapped in try/catch).
- splat: identify headerless .splat local previews via a `format` hint (blob:
  URLs have no extension to detect), and self-heal the blob->cloud swap by
  gating the no-reload guard on actual render rather than mesh existence, so a
  failed preview still loads the cloud copy in place (no re-drag needed).
- assets: regenerate-thumbnail now captures the live view (no iframe reload /
  file re-download) behind a camera icon (was a refresh icon); splat cards show
  the display name instead of "Upload".
…dify sizing

The 368MB/22M-splat build 504'd at the 900s Cloud Run request timeout, not an
OOM (16Gi held). Root fix is the timeout, not more cores.

- deploy.sh: TIMEOUT 900 -> 3600. Keep 16Gi; revert CPU 8 -> 4 (build-lod is
  single-threaded, so 8 vCPU measured no faster).
- rad-dispatch.js: set the Cloud Task dispatchDeadline to the 1800s max so the
  durable production path doesn't give up mid-build and thrash.
- Dockerfile: build build-lod with RUSTFLAGS target-cpu=x86-64-v3 (AVX2, safe
  on Cloud Run) - the only single-thread speed lever (profile already maxed:
  lto=fat, opt-level=3, codegen-units=1).
- README: updated sizing notes.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Support splat as asset type and image-to-splat with apple sharp-ml research model

2 participants