Skip to content

feat(extensions): add private immutable SQLite dataset browser - #5006

Closed
tang-vu wants to merge 1 commit into
Osmantic:public-betafrom
tang-vu:feat/beta-sqlite-dataset-explorer-20260914
Closed

tang-vu wants to merge 1 commit into
Osmantic:public-betafrom
tang-vu:feat/beta-sqlite-dataset-explorer-20260914

Conversation

@tang-vu

@tang-vu tang-vu commented Sep 14, 2026 •

Copy link
Copy Markdown
Contributor

Ready for independent review (2026-09-15). Candidate 1ecec1f0389be25253814eeb0148948d7feb23de has a successful GitHub check rollup and no base-branch conflict at this review snapshot. This supersedes earlier draft-status wording only. All validation gaps, migration requirements, live gates and independent human review requirements below remain in force; this is not a merge-ready approval.

Why this matters

Operators need to inspect local dataset snapshots without opening a writable connection to another ODS application's live database. Add a complete Datasette recipe for private table browsing, filtering, read-only SQL and CSV/JSON export, reachable through the existing extension install lifecycle.

Production contract

  • Pin the official stable datasetteproject/datasette:0.65.4@sha256:04ad0a3eb3d253b8c066d9491b7205ede911074d819c1f56dd40952a611b857d. Its published image is Linux amd64 only: Compose declares that platform and the manifest/README exclude an Apple/ARM compatibility claim.
  • Run UID/GID 1000 with read-only root and data mounts, no capabilities and a bounded temporary directory. An entrypoint passes explicit .db, .sqlite and .sqlite3 snapshots as immutable databases, preserving names with spaces. It never enables directory-mode plugin/config discovery from user data.
  • Require a stable signing secret and permit only the local root actor. Use upstream one-use login URLs, with README instructions to replace the internal authority with the actual browser endpoint. Sessions survive recreation when the secret is retained; an .env rotation invalidates them.
  • Preserve one-second SQL timeout, cap results at 1,000 rows/CSV at 10 MB, disable complete database download, and publish only loopback 7827. The public stylesheet is the unauthenticated HTTP health endpoint; data and version views remain private.
  • Handoff: manifest/catalog -> Extensions installation -> installed entrypoint/metadata/Compose -> immutable snapshots -> authenticated HTTP results. Stop before replacing snapshots or rolling back; source files undergo no migrations. Use SQLite backup APIs for live WAL sources rather than copying their main file alone.
  • Canonical catalog regeneration restores metadata for the already tracked disabled pixel-inference manifest without enabling it.

Overlap check

Searched all open/closed upstream PRs for Datasette and changed production recipe paths in the PR file inventory: none exists. Reviewed shared library README/catalog history. Existing Jupyter/Label Studio/Baserow workflows do not provide this immutable SQLite snapshot browser. New recipe additions in #4984, #4990, #4988 and #5003 are independent services; integration regeneration must retain each entry once and update the total count.

Validation

  • Real pinned image, Linux amd64 under WSL: 2 passed (22-live-public-links, 72.71 seconds of pytest). Healthy empty installation, missing-secret refusal, anonymous denial, one-use root login and replay refusal, persistent session across recreation, real Unicode/space-named SQLite snapshot, table filter, SQL aggregate, CSV quoting, SQL-write rejection and disabled file download. Repeated container replacement leaves the snapshot SHA-256 unchanged and creates no WAL/journal; an adjacent failing Python plugin is never loaded.
  • Early fixture failures were investigated: a YAML flow-sequence incorrectly split tmpfs options (corrected); the private versions endpoint returned 403 to health checks (moved to public stylesheet); immutable database URLs use Datasette's own escaping/hash links (the final test follows the public index API's path). No early failure is presented as a passing runtime check.
  • All 36 library manifests, ShellCheck for the new entrypoint, scoped pre-commit hooks and git diff --check passed. No dashboard implementation changes.
  • make gate reproduces the existing root/WSL no-sudo Docker fallback failures already addressed by open fix(installer): honor unavailable sudo in Docker fallback #3158. It is not a full local gate pass. A new path-filtered GitHub job runs the exact-image boundary test.

Review and limitations

Ready for independent review and CI. No user databases, browser automation, ARM emulation, native Windows/macOS, large datasets or remote TLS were tested. This is a single local operator session, not a multi-user identity system. Signed-in users can read all mounted rows; result limits do not redact data. The upstream image is preserved as published, including its base/dependency choices; further image maintenance should be reviewed separately.

Final candidate 1ecec1f0389be25253814eeb0148948d7feb23de: 33 successful CI checks, 4 conditionally skipped; no pending or failed checks.

Batch integration receipt

This is one of 30 independently branched public-beta PRs. The published synthetic integration head is 61f4c6fccc81bea697f4941683fefb5089ba77f4, built from baseline d49ad4bbc760be319de7f4c9b9a93b1f18fe3922 in this reproducible order:

#4946 → #4948 → #4951 → #4954 → #4972 → #4960 → #4961 → #4957 → #4964 → #4967 → #4968 → #4971 → #4975 → #4976 → #4984 → #4982 → #4990 → #4988 → #4989 → #4997 → #5003 → #5006 → #5007 → #5009 → #5012 → #5011 → #5013 → #5014 → #5020 → #5021

Reconciliation preserved all 146 files owned by a single PR byte-for-byte against their exact candidate heads. The eight shared paths preserve both CLI STT/TTS commands and help, every added Makefile test, both environment schema additions and all Extensions UI behavior. LanguageTool and Kroki both remain headless API entries. Canonical regeneration yields 76 catalog entries and 47 library manifests; all README rows are retained. Shared-file changes need these additive resolutions when merged independently.

Combined validation: 2,852 API tests passed / 1 skipped; 1,082 UI tests passed; 47 manifests passed; all 12 added Makefile commands passed; 12 catalog boundary tests passed / 12 opt-in runtime tests skipped in this combined run. Individual exact-image results remain documented above. UI lint/build, scoped secret hooks, diff checks, four platform smoke tests and installer simulation passed. Compose plans for NVIDIA/AMD/Apple each contain 20 services and no duplicate published bindings across the core plus 12 new recipes; this is static plan evidence, not simultaneous startup or GPU inference.

The first unrestricted UI run timed out in two tests under concurrent load; the complete rerun used --maxWorkers=2 with unchanged assertions and timeouts and passed. ESLint reports zero errors and 603 warnings, including the existing JSX usage-detection pattern. Full local make gate still fails the no-sudo Docker baseline covered by #3158. BATS records 416 passed / 5 failed on this root/WSL host (platform detection, root preflight and a 15 GiB versus 20 GiB free-space expectation). Neither full local gate nor BATS is described as green.

This integration receipt supplements each PR's own regression and does not waive its independent review, live hardware/browser/upgrade gates or rollback requirements. The PRs remain drafts; none was merged upstream.

This was referenced Sep 14, 2026
@tang-vu
tang-vu marked this pull request as ready for review September 15, 2026 01:31
@Lightheartdevs

Copy link
Copy Markdown
Collaborator

Thanks for this contribution. public-beta was promoted into main on 2026-09-24 and no longer receives changes, so we're closing pull requests that target it. This isn't a judgment on the change itself. If it's still needed, please rebase onto main and open a focused PR. See #7253 for details and the contribution policy.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants