Skip to content

Identity contract: typed, self-verifying references (#42) - #45

Merged
Deep-Sixed merged 2 commits into
mainfrom
claude/sweet-bardeen-bi3jez
Sep 25, 2026
Merged

Deep-Sixed merged 2 commits into
mainfrom
claude/sweet-bardeen-bi3jez

Conversation

@Deep-Sixed

Copy link
Copy Markdown
Owner

Closes #42.

What

stele.identity gives each kind of evidence one typed reference with one canonical string, plus a resolver that checks the reference against the ledger, the archive and the event chain.

Type Canonical string Resolves to
SourceRef stele:source:<source_id> Source
SnapshotRef stele:snapshot:<file|tree>:<sha256> Snapshot (every blob re-hashed)
RecordRef stele:record:<record_id>@<artifact_hash> ArtifactRecord (SEALED; INVALIDATED only on request)
ObservationRef stele:observation:<record_id>@<sha256> Observation (new)
AnchorRef stele:anchor:<record_id>@<artifact_hash>/<artifact>@<digest>#bytes=S-E or #pointer=P&render=R the anchored text, via the #27 extraction resolver
EventRef stele:event:<seq>@<hash> Event (the chain must verify with it as anchor)
  • Commits to content. Every reference contains a digest, so it resolves only while the evidence it named is unchanged.
  • One spelling each. parse_ref() accepts a string only if formatting the parsed reference gives it back byte for byte. Anchor paths and pointers are percent-encoded.
  • Observation is a new canonical record, {record_id, source_id, snapshot, observed_at}: what one run saw, with its own digest. It resolves in any record state, because an observation stays true even if the run is later invalidated.
  • Citations. ExtractionAdapter chunks now carry metadata["stele_anchor"], so a consumer can later check a quoted passage against the sealed evidence.
  • Other changes:
    • BlobStore.verify_snapshot(): a public wrapper around the existing re-hash check.
    • A new CLI: python -m stele.identity refs|resolve.
    • docs/identity.md, plus a line in the README docs table.

Change from the issue text: the Observation reference includes the record id (stele:observation:<record_id>@<sha256>), not just the digest. That way it can be resolved without an index from digest to observation.

Tests

tests/test_identity.py has 49 tests:

  • Format:
    • every type round-trips through its canonical string, including unicode and reserved characters in artifact paths
    • 23 non-canonical spellings are refused: wrong case, leading zeros, escapes that aren't needed, a literal space, a backwards range, a missing renderer, and more
    • the observation digest covers every field
  • Resolution: every type resolves to what it names against honest evidence.
  • Refusals:
    • forged digests, kinds, records, artifacts and ranges
    • a different observation
    • an invalidated record, unless allowed (its observation still resolves)
    • tampered output and input blobs
    • a rewritten event
  • Citations: every extraction chunk's stele_anchor resolves to that chunk's own text.
  • CLI: refs and resolve, with the exit codes.

Local: 801 passed, 86 skipped. ruff is clean on the changed files.

Limits (also in the docs)

  • A RecordRef checks the archived tree object, not every artifact blob. An AnchorRef re-hashes the one blob it reads.
  • An EventRef needs the whole chain to verify, so a break anywhere stops every event from resolving.
  • As with the event log, a reference proves more when it is held outside the reach of whoever can write the ledger.

🤖 Generated with Claude Code

https://claude.ai/code/session_013p3FU6gUJ1MGinnmEgvHsm


Generated by Claude Code

stele.identity gives each kind of evidence one typed reference with one
canonical string, and a resolver that checks it against the ledger, the
archive and the event chain:

  stele:source:<source_id>
  stele:snapshot:<file|tree>:<sha256>
  stele:record:<record_id>@<artifact_hash>
  stele:observation:<record_id>@<sha256>
  stele:anchor:<record_id>@<artifact_hash>/<artifact>@<digest>#bytes=S-E
                                                      #pointer=P&render=R
  stele:event:<seq>@<hash>

Every reference commits to content, so it resolves only while the evidence
it named is unchanged. parse_ref() is strict: one reference, one spelling.
An Observation (record, source, input snapshot, time) is new: what one run
saw, as a canonical record with its own digest.

ExtractionAdapter chunks now carry their anchor as `stele_anchor`, so a
consumer can check a quoted passage against the sealed evidence later.
BlobStore gains a public verify_snapshot(). CLI: python -m stele.identity
refs|resolve. Docs: docs/identity.md.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_013p3FU6gUJ1MGinnmEgvHsm

Copy link
Copy Markdown
Owner Author

linux-containment on d9daf00 hung, so I cancelled it after an hour. I don't think the failure comes from this PR:

  • Only gVisor-under-Docker tests failed:
    • test_backends.py::TestLiveInputs::test_file_input_hash_matches_what_parser_read[oci-runsc-docker]
    • test_backends.py::TestLiveInputs::test_directory_input_is_visible_and_hash_matches[oci-runsc-docker]
    • test_containment.py::TestAllowedArtifact::test_artifact_written_to_output_dir[oci-runsc-docker]
  • Then it hung on a docker process. Before the hang, 72 tests took 16 minutes, when they normally take seconds.
  • This PR doesn't touch that code. It changes no containment or OCI code: the only change outside stele/identity is a wrapper method on BlobStore and one metadata key in ExtractionAdapter.
  • The same job passed elsewhere: on main at this PR's base (df04b6f), and on another branch at 13:25.
  • No fix exists yet. If gVisor on the hosted runner does turn out to hang like this, the fix would be a per-test timeout on the OCI backend tests, and that belongs in a separate PR.

I'm re-running the workflow once. If it fails the same way again, I'll treat the failure as real and investigate it properly.


Generated by Claude Code


Generated by Claude Code

Copy link
Copy Markdown
Owner Author

Taking over driving this PR to green.

linux-containment on d9daf00 was cancelled after about an hour. It hit three failures about 16% into the suite and then hung until the job was cancelled, so the failure details were never printed. By their position, the three failures are the gVisor variants of existing sandbox tests:

  • test_backends.py::TestLiveInputs::*[oci-runsc-docker]
  • test_containment.py::TestAllowedArtifact::*[oci-runsc-docker]

I don't think they come from this PR:

  • This PR adds pure-Python code (stele/identity, plus a chunk metadata field and a BlobStore.verify_snapshot wrapper). Nothing it changes is on the path of those runs.
  • The same tests passed on Full-corpus production verification and sign-off #46 at 13:29 UTC, against the same base (df04b6f), on the same runner image.
  • Locally at d9daf00 the full suite passes: 828 passed. This host has no gVisor, so the runsc variants skip here.

That points to the runner's docker/gVisor setup during this run, not this PR. There is no fix to port. I'm re-running the job once. If it fails again, I'll treat it as this PR's failure and root-cause it.


Generated by Claude Code

Copy link
Copy Markdown
Owner Author

Update: linux-containment hung again on f989300 (this PR with main at #44 merged in). I cancelled the run after 11 minutes, and I'm not merging.

Evidence:

  • Both hangs look the same. The run stops in a docker process in pytest's second batch of 72 tests (tests 73–144). That batch holds the oci-runsc-docker cases in tests/test_backends.py and tests/test_containment.py. The first hang had three gVisor failures before it stopped. This time it stopped before printing anything for that batch.
  • None of this PR's tests had run yet. They sort later and never started.
  • This PR doesn't touch the code involved. It changes no containment, OCI, capture or workflow code.
  • The rate is lopsided. It hung in 2 of 3 runs on this PR. About 10 other runs today on the same workflow passed, including main at 341820e (Bound captured parser output; tell overlapping delivery attempts apart #44), which finished in 4 minutes.

Why I can't settle it:

  • I've already used my one re-run.
  • This session has no Docker daemon, so I can't reproduce the hang locally.
  • pytest -q output doesn't say which test is stuck.

Proposed separate fix, so the next hang names the test and ends on its own:

  • add timeout-minutes to the linux-containment job;
  • run pytest with a per-test timeout (pytest-timeout, e.g. --timeout=600) and -v output.

Generated by Claude Code


Generated by Claude Code

Copy link
Copy Markdown
Owner Author

On the hang at f989300 (linux-containment, cancelled): I tried to reproduce it with gVisor, using a Docker daemon with runsc installed and registered the way test.yml does it (the runsc apt package, runsc install -- --network=none).

  • Full suite at f989300, twice: 877 passed each time, about 84 seconds per run.
  • pytest -k runsc tests/test_backends.py tests/test_containment.py, 3 times: 17 passed each round, about 10 seconds per round.
  • Code check: every engine call in OciBackend has a time limit (run_bounded, _PROBE_TIMEOUT), so the stuck docker process means the Docker daemon on the runner itself was stuck.
  • The diff: nothing in it touches containment, OCI or the workflow.
  • Other branches: the roughly 20 other runs of this workflow today all passed in 3 to 4 minutes.

This commit's first run was cancelled and hasn't been re-run, so I'm re-running it once. If it hangs again, the per-test timeout and timeout-minutes change proposed above goes in as its own PR, so that the next hang names the stuck test.


Generated by Claude Code

@Deep-Sixed
Deep-Sixed marked this pull request as ready for review September 25, 2026 15:33
@Deep-Sixed
Deep-Sixed merged commit 10bbf3f into main Sep 25, 2026
9 of 10 checks passed
@Deep-Sixed
Deep-Sixed deleted the claude/sweet-bardeen-bi3jez branch September 25, 2026 15:33
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Identity contract: typed, self-verifying references (Source, Snapshot, Observation, Record, Anchor, Event)

2 participants