Skip to content

Verify streaming NGFF readback from S3 - #297

Merged
nclack merged 2 commits into
mainfrom
test/s3-ngff-readback
Oct 4, 2026
Merged

nclack merged 2 commits into
mainfrom
test/s3-ngff-readback

Conversation

@nclack

@nclack nclack commented Oct 4, 2026 •

Copy link
Copy Markdown
Member

Verify every advertised NGFF extent directly from S3 with TensorStore before writer flush or close. CPU and GPU tests share the filesystem fixture, analytic references, metadata checks, and codec table: 30 codec/buffering combinations plus a multipart case per backend.

A test-only aiohttp proxy delays shard completion while the reader checks published extents. Missing/corrupt-shard and premature-metadata probes cover all three levels; explicit read bounds handle concurrent metadata growth. Each run cleans up its own bucket, including failed setup. CTest runs both variants in the existing S3Mock CI jobs.

Validation:

  • Native CPU readback checks: 4/4 passed; GPU: 3/3 passed. All 31 S3 cases passed on each backend.
  • CPU Docker S3 readback passed.
  • Bypassing the publication fence fails on missing shard data; an injected proxy startup failure leaves no bucket.
  • C/Python formatting, new Python test lint, and whitespace checks passed.

Closes #34.

@codecov

codecov Bot commented Oct 4, 2026

Copy link
Copy Markdown

Codecov Report

✅ All modified and coverable lines are covered by tests.
see 3 files with indirect coverage changes

🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

@nclack
nclack merged commit 460921b into main Oct 4, 2026
9 checks passed
@nclack
nclack deleted the test/s3-ngff-readback branch October 4, 2026 23:21
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Add S3 readback tests for streaming NGFF consistency

1 participant