Identity contract: typed, self-verifying references (#42)

README.md

@@ -156,6 +156,7 @@ Details and known limits: [docs/containment.md](docs/containment.md).
 | [Replay](docs/replay.md) | Validation, replay outcomes, invalidation, ledger views |
 | [Adapter contract](docs/adapter.md) | `SteleAdapter`, `TargetWriter`, the Dispatcher and delivery log, the ChatGPT export adapter |
 | [Extraction contract](docs/extraction.md) | `stele.extraction` v1, normalizers, the trusted resolver, `ExtractionAdapter` |
+| [Identity contract](docs/identity.md) | `stele:` references to sources, snapshots, records, observations, anchors and events, and how they resolve |
 | [Evidence store](docs/archive.md) | Content-addressed Snapshot and artifact archive |
 | [Cloud sandboxes](docs/cloud-sandboxes.md) | Design note for hosted sandbox backends (not implemented) |
 | [Parser images](parsers/README.md) | Building and running MinerU, Marker, Docling |

docs/extraction.md

@@ -127,8 +127,10 @@ A `SteleAdapter` for any bundle that has a normalizer.
   fails the transform, and nothing reaches a target.
 - It emits one chunk per unit. The chunk id is `<record_id>:<unit id>`.
 - The chunk metadata holds `kind`, `order`, `page`, `bbox`, `level`,
-  `parent` (as a chunk id), `anchor`, `attributes`, `schema` and
-  `normalizer`.
+  `parent` (as a chunk id), `anchor`, `attributes`, `schema`,
+  `normalizer`, and `stele_anchor`: the unit's anchor as a
+  [Stele reference](identity.md) (`stele:anchor:...`) that a consumer can
+  resolve to check a citation against the evidence.
 
 ## CLI
 

docs/identity.md

@@ -0,0 +1,117 @@
+# Identity Contract
+
+Each layer of Stele names its evidence in its own way:
+
+- a Snapshot by `(kind, digest)`
+- a Source by `source_id`
+- a ledger record by `record_id` and `artifact_hash`
+- an extraction unit by an anchor into a sealed artifact
+- an event by `(seq, hash)` in the hash-chained log
+
+`stele.identity` gives each of these one typed reference with one canonical
+string. A downstream system can store, pass on and cite that string, and check
+it later.
+
+Every reference **commits to content**: it contains the digest of what it
+names. It therefore resolves only while that evidence is unchanged.
+
+## References (`stele/identity/refs.py`)
+
+| Type | Canonical string | Commits to |
+|------|------------------|------------|
+| `SourceRef` | `stele:source:<source_id>` | the Source record (a descriptive locator) |
+| `SnapshotRef` | `stele:snapshot:<file\|tree>:<sha256>` | the exact bytes a parser saw |
+| `RecordRef` | `stele:record:<record_id>@<artifact_hash>` | one run's sealed output tree |
+| `ObservationRef` | `stele:observation:<record_id>@<sha256>` | one run's Observation (below) |
+| `AnchorRef` | `stele:anchor:<record_id>@<artifact_hash>/<artifact>@<digest>#bytes=<start>-<end>` | a byte range of one sealed artifact |
+| | `stele:anchor:<record_id>@<artifact_hash>/<artifact>@<digest>#pointer=<pointer>&render=<name>` | a JSON value of one sealed artifact, rendered by a named trusted renderer |
+| `EventRef` | `stele:event:<seq>@<hash>` | one event, and through the chain, all history before it |
+
+**An Observation** records that one run saw its input:
+
+```
+{schema: "stele.observation", schema_version: 1,
+ record_id, source_id (or null), snapshot: {kind, digest}, observed_at}
+```
+
+- `observed_at` is the record's `created_at`, in ISO 8601 (UTC).
+- Its reference names the record plus the SHA-256 of that canonical JSON. Each
+  field is covered, so a different source, snapshot or time gives a different
+  reference.
+
+**Canonical strings.** `parse_ref()` accepts a string only if formatting the
+parsed reference gives the same string back, so each reference has exactly one
+spelling.
+
+- In anchors, the artifact path and the JSON pointer are percent-encoded.
+  Unreserved characters and `/` stay literal. Everything else becomes `%XX`
+  with upper-case hex, as `urllib.parse.quote` writes it.
+- A renderer name is encoded the same way, but with `/` escaped too.
+- Numbers have no leading zeros. Digests are lower-case hex.
+
+**Constructors:**
+
+| Function | Returns |
+|---|---|
+| `record_ref(record)` | the `RecordRef` of an `ArtifactRecord` or `SealedBundle` |
+| `observation_of(record)` | the record's `Observation`; its `.ref` is the `ObservationRef` |
+| `anchor_ref(extraction, unit)` | the `AnchorRef` of one extraction unit |
+| `event_ref(event)` | the `EventRef` of an event from `EventLog` |
+
+## Resolution (`stele/identity/resolver.py`)
+
+`IdentityResolver(ledger).resolve(ref)` accepts a reference or its string. It
+returns what the reference names or raises `ResolveError`. `check(ref)` returns
+the problems as a list instead. Each type is checked against the evidence,
+never against the caller's word:
+
+| Reference | Resolves when | Returns |
+|---|---|---|
+| `SourceRef` | The Source record is in the archive and hashes to `source_id`. | `Source` |
+| `SnapshotRef` | The Snapshot record is in the archive, and every blob it covers re-hashes: the file, or the tree object and each file it lists. | `Snapshot` |
+| `RecordRef` | The record exists and is `SEALED` (or `INVALIDATED` with `allow_invalidated=True`). It must have that `artifact_hash`, and the archived tree object must list exactly the record's manifest. | `ArtifactRecord` |
+| `ObservationRef` | The Observation rebuilt from the record has that digest, and its Snapshot and Source resolve. An observation stays true whatever happened to the run later, so it resolves in any record state. | `Observation` |
+| `AnchorRef` | The record resolves as a `RecordRef`. The [extraction resolver](extraction.md) then reads the span from bytes the archive has just re-hashed. | the anchored text |
+| `EventRef` | The chain verifies with this event as its anchor: the event is at `seq` with this hash, and nothing before or after it has been altered, removed or reordered. | `Event` |
+
+**What resolution does not check:**
+- A `RecordRef` checks the tree object but not every artifact blob. An
+  `AnchorRef` re-hashes the one blob it reads.
+- An `EventRef` depends on the whole chain being intact. A chain broken
+  anywhere resolves no event, because none of its hashes can be trusted until
+  the break is explained.
+- As with the event log, anyone with write access to the ledger and archive
+  can rewrite them consistently. A reference held outside that access, such
+  as a published `EventRef`, is what detects that.
+
+## Citations
+
+`ExtractionAdapter` chunks carry their unit's anchor reference in
+`metadata["stele_anchor"]`. A system that quotes a chunk can keep that string
+next to the quote. Resolving it later checks that the quoted text is still
+exactly what the sealed evidence says, and that the record still stands.
+
+## CLI
+
+```bash
+python -m stele.identity refs ledger.db archive/ <record_id>
+    # {"record": "stele:record:...", "observation": "...", "snapshot": "...", "events": [...]}
+python -m stele.identity resolve ledger.db archive/ <ref>... [--allow-invalidated]
+    # one JSON line per reference; exit 1 if any does not resolve
+```
+
+## Tests
+
+`tests/test_identity.py`:
+
+| Property | Tests |
+|---|---|
+| Every type round-trips through one canonical string, and other spellings are refused | `TestFormat` |
+| Every type resolves to what it names | `TestResolve::test_every_type_resolves_to_what_it_names` |
+| Wrong digests, kinds, records, artifacts and ranges are refused | `TestResolve::test_wrong_digests_are_refused` |
+| A different observation is refused | `TestResolve::test_a_different_observation_is_refused` |
+| Invalidated records resolve only on request; their observations still hold | `TestResolve::test_invalidated_record_needs_permission` |
+| Tampered output and input blobs are refused | `TestResolve::test_tampered_evidence_is_refused` |
+| A rewritten event is refused | `TestResolve::test_rewritten_event_is_refused` |
+| Extraction chunks cite resolvable anchors | `test_extraction_chunks_carry_resolvable_anchor_refs` |
+| CLI | `test_cli_refs_and_resolve` |

stele/adapters/extraction.py

@@ -6,8 +6,10 @@
 verified bytes by the resolver before anything is returned, so a normalizer
 bug that misquotes the evidence fails the transform instead of reaching a
 target. Chunk ids are "<record_id>:<unit id>"; metadata carries the unit's
-kind, order, page, bbox, level, parent, anchor and attributes, plus the
-extraction's schema and normalizer.
+kind, order, page, bbox, level, parent, anchor and attributes, the
+extraction's schema and normalizer, and stele_anchor: the unit's anchor as a
+Stele reference (stele.identity), which a consumer can resolve to check a
+citation against the evidence.
 """
 from __future__ import annotations
 
@@ -22,6 +24,7 @@ def transform(self, bundle: SealedBundle) -> list[SteleChunk]:
         # Imported here: the normalizers build on the other adapters in this package.
         from ..extraction.normalizers import normalize
         from ..extraction.resolver import BundleResolver, ResolutionError
+        from ..identity.refs import anchor_ref
 
         extraction = normalize(bundle)
         problems = BundleResolver(bundle).verify(extraction)
@@ -45,6 +48,7 @@ def transform(self, bundle: SealedBundle) -> list[SteleChunk]:
                 level=unit.level,
                 parent=f"{bundle.record_id}:{unit.parent}" if unit.parent else None,
                 anchor=unit.anchor.to_json(),
+                stele_anchor=str(anchor_ref(extraction, unit)),
                 attributes=dict(unit.attributes),
             )
             for unit in extraction.units

stele/archive/store.py

@@ -428,6 +428,10 @@ def has_snapshot(self, digest: str, kind: SnapshotKind | str) -> bool:
         kind = SnapshotKind(kind)
         return self._sharded(self._snapshots / kind.value, _check_digest(digest)).is_file()
 
+    def verify_snapshot(self, snapshot: Snapshot) -> None:
+        """Re-hash every blob a recorded snapshot covers (IntegrityError if any changed)."""
+        self._check_snapshot_content(snapshot)
+
     def _check_snapshot_content(self, snapshot: Snapshot) -> None:
         """Re-hash every blob the snapshot covers; a size match is not enough."""
         if snapshot.kind is SnapshotKind.FILE:

stele/identity/__init__.py

@@ -0,0 +1,23 @@
+"""The identity contract (roadmap #42): typed, self-verifying references to evidence.
+
+refs
+    SourceRef, SnapshotRef, RecordRef, ObservationRef, AnchorRef, EventRef:
+    one canonical string each ("stele:<type>:..."), strict parse_ref(), and
+    constructors from ledger records, extraction units and events.
+resolver
+    IdentityResolver(ledger).resolve(ref) checks a reference against the
+    ledger, the evidence archive and the event chain, and returns what it names.
+
+CLI: python -m stele.identity resolve <ledger.db> <archive-dir> <ref>...
+"""
+from .refs import (
+    PREFIX, AnchorRef, EventRef, Observation, ObservationRef, RecordRef, Ref, RefFormatError,
+    SnapshotRef, SourceRef, anchor_ref, event_ref, observation_of, parse_ref, record_ref,
+)
+from .resolver import IdentityResolver, ResolveError
+
+__all__ = [
+    "PREFIX", "AnchorRef", "EventRef", "Observation", "ObservationRef", "RecordRef", "Ref",
+    "RefFormatError", "SnapshotRef", "SourceRef", "anchor_ref", "event_ref", "observation_of",
+    "parse_ref", "record_ref", "IdentityResolver", "ResolveError",
+]

stele/identity/__main__.py

@@ -0,0 +1,93 @@
+"""
+python -m stele.identity resolve <ledger.db> <archive-dir> <ref>... [--allow-invalidated]
+    Resolve each reference against the evidence and print one JSON result per
+    reference; exit 1 if any does not resolve.
+python -m stele.identity refs <ledger.db> <archive-dir> <record_id>
+    Print the record, observation and event references of one record.
+"""
+from __future__ import annotations
+
+import argparse
+import dataclasses
+import json
+import sys
+from pathlib import Path
+from typing import Any
+
+from ..archive import BlobStore, Snapshot, Source
+from ..ledger.events import EventLog
+from ..ledger.models import ArtifactRecord
+from ..ledger.store import LedgerStore, RecordNotFoundError
+from .refs import Observation, event_ref, observation_of, record_ref
+from .resolver import IdentityResolver, ResolveError
+
+
+def _summary(value: Any) -> Any:
+    """A small JSON description of what a reference resolved to."""
+    if isinstance(value, str):
+        return {"text": value}
+    if isinstance(value, ArtifactRecord):
+        return {"record_id": value.record_id, "state": value.state.value,
+                "artifact_hash": value.artifact_hash, "files": len(value.artifact_manifest)}
+    if isinstance(value, Observation):
+        return json.loads(value.to_canonical())
+    if isinstance(value, (Source, Snapshot)):
+        return json.loads(value.to_canonical())
+    if dataclasses.is_dataclass(value):
+        return {k: v for k, v in dataclasses.asdict(value).items()}
+    return repr(value)
+
+
+def main(argv: list[str] | None = None) -> int:
+    ap = argparse.ArgumentParser(prog="python -m stele.identity",
+                                 description="Stele references: resolve and list them.")
+    sub = ap.add_subparsers(dest="command", required=True)
+    re_ = sub.add_parser("resolve", help="resolve references against the evidence")
+    re_.add_argument("ledger", type=Path)
+    re_.add_argument("archive", type=Path)
+    re_.add_argument("refs", nargs="+")
+    re_.add_argument("--allow-invalidated", action="store_true")
+    rf = sub.add_parser("refs", help="the references of one record")
+    rf.add_argument("ledger", type=Path)
+    rf.add_argument("archive", type=Path)
+    rf.add_argument("record_id")
+    args = ap.parse_args(argv)
+
+    ledger = LedgerStore(args.ledger, BlobStore(args.archive))
+    try:
+        if args.command == "refs":
+            try:
+                record = ledger.get(args.record_id)
+            except RecordNotFoundError:
+                print(f"no record {args.record_id}", file=sys.stderr)
+                return 1
+            out: dict[str, Any] = {"record": str(record_ref(record))}
+            if record.source_hash is not None and record.source_kind is not None:
+                observation = observation_of(record)
+                out["observation"] = str(observation.ref)
+                out["snapshot"] = str(observation.snapshot)
+            log = EventLog(ledger)
+            try:
+                out["events"] = [str(event_ref(e)) for e in log.for_subject(record.record_id)]
+            finally:
+                log.close()
+            print(json.dumps(out, indent=2))
+            return 0
+        resolver = IdentityResolver(ledger, allow_invalidated=args.allow_invalidated)
+        failed = False
+        for ref in args.refs:
+            try:
+                value = resolver.resolve(ref)
+            except ResolveError as exc:
+                failed = True
+                print(json.dumps({"ref": ref, "ok": False, "problem": str(exc)}))
+            else:
+                print(json.dumps({"ref": ref, "ok": True, "value": _summary(value)},
+                                 ensure_ascii=False, default=str))
+        return 1 if failed else 0
+    finally:
+        ledger.close()
+
+
+if __name__ == "__main__":
+    sys.exit(main())