Skip to content

Roadmap: sandbox backends, heavy parsers, ledger and dispatch redesign #15

Open

Description

@Deep-Sixed

Tracking issue for the next phase of Stele. Each item is an issue; the PR that implements it says Closes #N.

Before starting #5 (done)

  1. Final review of PR Close remaining containment gaps, fix review findings, add macOS/Windows support #4 at its current head, which includes the review fixes and macOS/Windows support.
  2. Merge PR Harden Stele parser containment baseline #3 (containment baseline).
  3. Point PR Close remaining containment gaps, fix review findings, add macOS/Windows support #4 at main, check the combined result, and merge it.

Sandbox model

Bubblewrap is Linux-only, so each workload runs on the backend that fits it:

Workload Linux macOS / Windows
Lightweight, deterministic extractors Wasm/WASI (#7) Wasm/WASI (#7), the same way
Heavy ML parsers (MinerU, Marker, Docling) bubblewrap + seccomp (#6), or OCI with gVisor (#8) OCI container through the Docker Desktop / Podman VM (#8)

Dependency tree

#5  Backend interface
 ├─ #6  Linux: bubblewrap + seccomp + Landlock
 ├─ #7  Wasm/WASI
 ├─ #8  OCI (runc / gVisor / GPU)
 │   ├─ #9  MinerU
 │   └─ #10 Marker + Docling
 └─ #16 Snapshot archive + content-addressed store
      └─ #12 Ledger redesign
           └─ #13 Durable dispatch + invalidation
                └─ #14 Replay (also needs #7)
#11 Telemetry + diagnostics  ← depends on #6, #7, #8

Layers (phase 1: complete)

Layer Issues
Execution abstraction [x] #5
Isolation engines [x] #6 · [x] #7 · [x] #8
Heavy parser integration [x] #9 · [x] #10
Runtime evidence [x] #11
Immutable evidence foundation [x] #16
Provenance and state model [x] #12
Delivery and invalidation evidence [x] #13
Replay (REPRODUCED / EQUIVALENT / DIVERGED / UNREPLAYABLE) [x] #14 · [x] #29 per-field tolerances · [x] #30 recorded run conditions

Shipped in PR #27 without a separate issue:

  • Append-only, hash-chained event log, with ledger verification and external anchors
  • Canonical extraction contract (stele.extraction/v1) and trusted resolver
  • ChatGPT export adapter: streamed, keeping every conversation branch, run through the Wasm splitter

Next

Item Issue
Identity contract: typed, self-verifying references (Source, Snapshot, Observation, Record, Anchor, Event) [x] #42 (PR #45)
Full-corpus production verification and sign-off. The baseline is today's bubblewrap-only setup. [ ] #43

Follow-ups

Item Issue
Parser images: GPU validation, Marker OCR, arm64 builds, hash-pinned torch [ ] #37
OCI: honour writable_scratch=False; tell a parser's own exit 125–127 apart from an engine failure [ ] #38
Ledger migration: re-verify records instead of leaving them invalidated for missing evidence [ ] #39
Extraction: page and bbox normalizers from the parsers' layout JSON [ ] #40

Deferred

  • Firecracker / Kata Containers, only if Stele becomes multi-tenant
  • A native Windows sandbox (AppContainer)
  • Networked parser execution: only ever as an operator override with its own evidence event, never as a parser manifest option

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions