Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
10 changes: 10 additions & 0 deletions CHANGELOG.md
Original file line number Diff line number Diff line change
Expand Up @@ -6,6 +6,16 @@ independent of the on-disk **format** version (`format` in the manifest).

## [Unreleased]

### Added

- **Format 1.1.0** — the manifest gains four optional fields (spec §3.1): `specURL`,
`coverage` (`{from, to}` — earliest/latest episode `eventTime`), `scopes` (sorted
context ids the records reference) and `bundleDigest` (sha256 of the exact `CHECKSUMS`
bytes — one hash for the whole archive). The exporter emits them; the validator
recomputes `bundleDigest` when present. 1.0 bundles remain valid — the shipped 1.0
fixtures double as backward-compatibility tests. `MemCoverage` is exported from the
package.

## [0.2.0] - 2026-09-14

`pip install portable-memory` now installs the **`mem` command line** — paste an
Expand Down
15 changes: 14 additions & 1 deletion Schemas/manifest.schema.json
Original file line number Diff line number Diff line change
Expand Up @@ -2,7 +2,7 @@
"$schema": "https://json-schema.org/draft/2020-12/schema",
"$id": "https://macpaw.github.io/portable-memory/schemas/manifest.schema.json",
"title": "Portable Memory manifest",
"description": "manifest.json — declares format version, capabilities, model tags, counts, and per-file integrity for a .mem bundle.",
"description": "manifest.json — declares format version, capabilities, model tags, counts, per-file integrity and (format 1.1) coverage, scopes and an archive-level digest for a .mem bundle.",
"type": "object",
"required": ["format", "generator", "conformanceLevel", "createdAt", "exportMode", "schemaVersion", "embeddingModel", "embeddingDim", "embeddingsIncluded", "capabilities", "counts", "files"],
"properties": {
Expand All @@ -12,6 +12,19 @@
"createdAt": { "type": "string", "format": "date-time" },
"exportMode": { "type": "string", "enum": ["full", "incremental"] },
"since": { "type": "string", "format": "date-time", "description": "present for incremental exports" },
"specURL": { "type": "string", "format": "uri", "description": "format 1.1 — URL of the specification the bundle follows" },
"coverage": {
"type": "object",
"description": "format 1.1 — earliest and latest episode eventTime in the bundle; absent when there are no episodes",
"required": ["from", "to"],
"properties": {
"from": { "type": "string", "format": "date-time" },
"to": { "type": "string", "format": "date-time" }
},
"additionalProperties": false
},
"scopes": { "type": "array", "items": { "type": "string" }, "uniqueItems": true, "description": "format 1.1 — sorted scope (context) ids the exported records reference; absent when none" },
"bundleDigest": { "type": "string", "pattern": "^[0-9a-f]{64}$", "description": "format 1.1 — sha256 of the exact CHECKSUMS bytes: one hash for the whole archive" },
"schemaVersion": { "type": "integer" },
"embeddingModel": { "type": "string" },
"embeddingDim": { "type": "integer", "minimum": 0 },
Expand Down
38 changes: 34 additions & 4 deletions Spec/portable-memory-spec.md
Original file line number Diff line number Diff line change
@@ -1,7 +1,7 @@
# Portable Memory — Specification v1.0
# Portable Memory — Specification v1.1

> An open, vendor-neutral format for AI memory portability — proposed and stewarded in the open by MacPaw.
> Status: **v1.0**. Reference SDKs: Swift (github.com/MacPaw/portable-memory-swift) and Python (github.com/MacPaw/portable-memory).
> Status: **v1.1** (format `1.1.0`; 1.0 bundles remain valid). Reference SDKs: Swift (github.com/MacPaw/portable-memory-swift) and Python (github.com/MacPaw/portable-memory).
> Paper: [Memory Belongs to the User: Portable Memory, an Open Standard Proposal for Cross-Vendor AI Memory](https://research.macpaw.com/publications/portable-memory) (MacPaw Research, 2026).

**Goal.** Define an open, vendor-neutral format and protocol for carrying AI memory
Expand Down Expand Up @@ -221,25 +221,52 @@ is. Embeddings are an optional accelerator.

```jsonc
{
"format": "1.0.0",
"format": "1.1.0",
"specURL": "https://github.com/MacPaw/portable-memory/blob/main/Spec/portable-memory-spec.md",
"generator": "your-app/1.0",
"conformanceLevel": "L2",
"createdAt": "2026-06-30T00:00:00Z",
"exportMode": "full", // or "incremental" with "since"
"coverage": { "from": "2024-01-03T09:12:00Z", "to": "2026-06-29T18:20:00Z" }, // earliest/latest episode eventTime
"scopes": ["ctx_personal", "ctx_team-platform"], // context ids the records reference
"schemaVersion": 12,
"embeddingModel": "bge-m3",
"embeddingDim": 1024,
"embeddingsIncluded": false, // false ⇒ receiver re-embeds from source text
"capabilities": ["bitemporal", "tombstones", "redaction", "evidence-pack"],
"counts": { "episode": 128, "edge": 64, "tombstone": 3 },
"files": [ { "path": "items/episode.jsonl", "sha256": "…", "bytes": 40213 } ]
"files": [ { "path": "items/episode.jsonl", "sha256": "…", "bytes": 40213 } ],
"bundleDigest": "…" // sha256 of the CHECKSUMS file — one hash for the whole archive
}
```

`format` is semver. `capabilities` declares optional features. Importers negotiate by
capability and **preserve unknown kinds verbatim** + **foreign episode fields via
`ext`** on round-trip (§10), so a newer or other-vendor bundle never loses data here.

### 3.1 Format 1.1 additions (normative)

Format `1.1.0` adds four **optional** manifest fields so a bundle answers "what does
this archive cover, in which scopes, and is it whole?" without opening a stream — the
questions a procurement or erasure audit asks first. A producer at format ≥ 1.1 SHOULD
emit them; a reader MUST accept their absence (every 1.0 bundle is a valid 1.1 bundle).

- **`specURL`** — the URL of the specification the bundle follows.
- **`coverage`** — `{ "from", "to" }`: the earliest and latest `eventTime` among the
episodes in the bundle (RFC 3339, whole-second UTC). Omitted when the bundle carries
no episodes. For an incremental export it describes the episodes *in this bundle*, not
the source store.
- **`scopes`** — the sorted (by Unicode code point), de-duplicated set of scope
identifiers the exported records reference: every episode `contextID` plus the `id`
of every exported `context` record. Omitted when empty. Scopes are how a host expresses
org / team / personal partitioning (§2); a reader can enumerate them before import.
- **`bundleDigest`** — the lowercase-hex SHA-256 of the exact bytes of `CHECKSUMS`
(§1.2). Because `CHECKSUMS` lists every data file's digest, this is a single hash for
the whole archive; a validator that finds it present MUST recompute it from the
on-disk `CHECKSUMS` and report a mismatch as an integrity failure. It complements —
not replaces — the optional manifest signature (§1.3), which is what makes the hash
*authentic* rather than merely *consistent*.

---

## 4. Operations — export & import / merge
Expand Down Expand Up @@ -397,6 +424,9 @@ tombstones* is a real, testable feature — not an aspiration.

## 9. Versioning & capability negotiation

- **Format history** — `1.0.0` (2026-06): initial release. `1.1.0` (2026-09): adds the
optional manifest fields `specURL`, `coverage`, `scopes`, `bundleDigest` (§3.1). A
minor bump: 1.0 readers ignore the new fields; 1.1 readers accept their absence.
- `format` is semver in the manifest. `capabilities[]` declares features (e.g.
`bitemporal`, `tombstones`, `redaction`, `evidence-pack`, `embeddings:<model>`).
- Importers negotiate by capability and **preserve unknown kinds verbatim** and
Expand Down
3 changes: 2 additions & 1 deletion portable_memory/__init__.py
Original file line number Diff line number Diff line change
Expand Up @@ -19,6 +19,7 @@
BundlePath,
ConformanceLevel,
ExportMode,
MemCoverage,
MemFileEntry,
MemFormat,
MemKind,
Expand Down Expand Up @@ -64,7 +65,7 @@
__all__ = [
"canonical_json", "format_timestamp", "parse_timestamp", "to_wire", "from_wire", "to_line",
"MemFormat", "MemKind", "IMPORT_ORDER", "ExportMode", "ConformanceLevel",
"MemFileEntry", "MemManifest", "MemLimits", "BundlePath",
"MemFileEntry", "MemManifest", "MemCoverage", "MemLimits", "BundlePath",
"sha256_hex", "sha256_hex_str", "embedding_cache_key",
"KNOWN_EPISODE_KEYS", "extract_episode_ext", "merge_episode_ext",
"PortableEpisode", "PortableEntity", "PortableEdge", "PortableFact", "PortableFactLink",
Expand Down
44 changes: 39 additions & 5 deletions portable_memory/exporter.py
Original file line number Diff line number Diff line change
Expand Up @@ -18,6 +18,7 @@
from .format import (
ConformanceLevel,
ExportMode,
MemCoverage,
MemFileEntry,
MemFormat,
MemKind,
Expand Down Expand Up @@ -125,8 +126,9 @@ def keep(ts: datetime) -> bool:
counts[MemKind.PROCEDURE.value] = self._write_jsonl(
store.export_procedures(), "items/procedure.jsonl", root, files
)
contexts = store.export_contexts()
counts[MemKind.CONTEXT.value] = self._write_jsonl(
store.export_contexts(), "items/context.jsonl", root, files
contexts, "items/context.jsonl", root, files
)
counts[MemKind.COMMUNITY.value] = self._write_jsonl(
store.export_communities(), "items/community.jsonl", root, files
Expand Down Expand Up @@ -166,7 +168,7 @@ def keep(ts: datetime) -> bool:
audit = store.export_audit_log(cutoff)
counts["audit"] = self._write_jsonl(audit, "audit/log.jsonl", root, files)

self._write_checksums(files, root)
checksums = self._write_checksums(files, root)
capabilities = [
"bitemporal",
"tombstones",
Expand All @@ -185,6 +187,9 @@ def keep(ts: datetime) -> bool:
counts=counts,
files=files,
capabilities=capabilities,
coverage=_coverage_of(episodes),
scopes=_scopes_of(episodes, contexts),
bundle_digest=sha256_hex(checksums),
)
self._write_manifest(manifest, root, signing_key)
return manifest
Expand Down Expand Up @@ -214,7 +219,7 @@ def export_evidence_pack(
counts["provenanceEdge"] = self._write_jsonl(
store.export_edges(), "provenance/edges.jsonl", root, files
)
self._write_checksums(files, root)
checksums = self._write_checksums(files, root)
capabilities = ["evidence-pack", "proof-of-deletion", "audit", "provenance"]
if signing_key is not None:
capabilities.append("signed")
Expand All @@ -226,6 +231,7 @@ def export_evidence_pack(
counts=counts,
files=files,
capabilities=capabilities,
bundle_digest=sha256_hex(checksums),
)
self._write_manifest(manifest, root, signing_key)
return manifest
Expand Down Expand Up @@ -272,9 +278,16 @@ def _make_manifest(
counts: dict[str, int],
files: list[MemFileEntry],
capabilities: list[str],
coverage: MemCoverage | None = None,
scopes: list[str] | None = None,
bundle_digest: str | None = None,
) -> MemManifest:
return MemManifest(
format=MemFormat.VERSION,
spec_url=MemFormat.SPEC_URL,
coverage=coverage,
scopes=scopes,
bundle_digest=bundle_digest,
generator=info.generator,
conformance_level=level,
# Aware UTC — a naive datetime.now() is LOCAL time, and the codec would
Expand All @@ -292,13 +305,15 @@ def _make_manifest(
files=sorted(files, key=lambda f: f.path),
)

def _write_checksums(self, files: list[MemFileEntry], root: str) -> None:
def _write_checksums(self, files: list[MemFileEntry], root: str) -> bytes:
"""Write ``CHECKSUMS`` in ``sha256sum`` format (``<hex> <path>``), sorted by path
with a trailing newline. Two spaces separate hash and path per the tool's format."""
with a trailing newline. Two spaces separate hash and path per the tool's format.
Returns the exact bytes written — the manifest's ``bundleDigest`` hashes them."""
ordered = sorted(files, key=lambda f: f.path)
lines = [f"{f.sha256} {f.path}" for f in ordered]
data = ("\n".join(lines) + "\n").encode("utf-8")
self._write_bytes(data, "CHECKSUMS", root)
return data

def _write_episodes(
self,
Expand Down Expand Up @@ -358,3 +373,22 @@ def _rmtree(path: str) -> None:
else:
os.remove(child)
os.rmdir(path)


# ── Format 1.1 manifest summaries (spec §3.1) ─────────────────────────────────────────

def _coverage_of(episodes: list) -> MemCoverage | None:
"""Earliest and latest ``eventTime`` among the episodes *in this bundle*; None when
the bundle carries no episodes."""
if not episodes:
return None
times = [e.event_time for e in episodes]
return MemCoverage(from_=min(times), to=max(times))


def _scopes_of(episodes: list, contexts: list) -> list[str] | None:
"""Sorted (by code point), de-duplicated scope ids: every episode ``contextID`` plus
the id of every exported ``context`` record; None when empty."""
ids = {e.context_id for e in episodes if e.context_id}
ids.update(c.id for c in contexts)
return sorted(ids) or None
25 changes: 24 additions & 1 deletion portable_memory/format.py
Original file line number Diff line number Diff line change
Expand Up @@ -15,7 +15,11 @@

class MemFormat:
#: Semantic version of the on-disk format. Importers negotiate by this + capabilities.
VERSION = "1.0.0"
#: 1.1.0 added the optional manifest fields ``specURL``, ``coverage``, ``scopes`` and
#: ``bundleDigest``; 1.0 bundles remain valid (same major).
VERSION = "1.1.0"
#: Where the specification this bundle follows lives (``manifest.specURL``).
SPEC_URL = "https://github.com/MacPaw/portable-memory/blob/main/Spec/portable-memory-spec.md"
#: Conventional bundle directory suffix.
BUNDLE_SUFFIX = "mem"

Expand Down Expand Up @@ -67,6 +71,15 @@ class MemFileEntry:
bytes: int = field(metadata={"json": "bytes"})


@dataclass(kw_only=True)
class MemCoverage:
"""The time span of the memories in a bundle — the earliest and latest episode
``eventTime`` (format 1.1). Lets a reader answer "what period does this archive
cover?" without opening a stream."""
from_: datetime = field(metadata={"json": "from"})
to: datetime = field(metadata={"json": "to"})


@dataclass(kw_only=True)
class MemManifest:
"""Declares everything an importer needs to negotiate capabilities and verify integrity."""
Expand All @@ -83,6 +96,16 @@ class MemManifest:
counts: dict[str, int] = field(metadata={"json": "counts"})
files: list[MemFileEntry] = field(metadata={"json": "files"})
since: datetime | None = field(default=None, metadata={"json": "since"})
# ── Format 1.1 additions. Optional on read: a 1.0 bundle has none of them. ──
#: URL of the specification the bundle follows.
spec_url: str | None = field(default=None, metadata={"json": "specURL"})
#: Earliest/latest episode ``eventTime`` in the bundle; absent when it has no episodes.
coverage: MemCoverage | None = field(default=None, metadata={"json": "coverage"})
#: Sorted, unique scope (context) ids the exported records reference; absent when none.
scopes: list[str] | None = field(default=None, metadata={"json": "scopes"})
#: Lowercase-hex SHA-256 of the exact ``CHECKSUMS`` bytes — one hash for the whole
#: archive. Validators recompute it when present.
bundle_digest: str | None = field(default=None, metadata={"json": "bundleDigest"})


class MemLimits:
Expand Down
16 changes: 16 additions & 0 deletions portable_memory/validator.py
Original file line number Diff line number Diff line change
Expand Up @@ -178,6 +178,22 @@ def validate(
issues.append(f"malformed {kind.value} record")
break

# ── Archive-level digest (format 1.1, spec §3.1): bundleDigest = sha256(CHECKSUMS). ──
if manifest.bundle_digest is not None:
cpath = BundlePath.safe_path("CHECKSUMS", bundle)
if cpath is None or not os.path.isfile(cpath):
issues.append("bundleDigest declared but CHECKSUMS is missing")
else:
size = MemLimits.file_size(cpath)
if size is None or size > MemLimits.max_file_bytes:
issues.append("CHECKSUMS exceeds size limit")
else:
with open(cpath, "rb") as fh:
if sha256_hex(fh.read()) != manifest.bundle_digest:
issues.append(
"bundleDigest mismatch: CHECKSUMS does not hash to the manifest's bundleDigest"
)

return ValidationResult(manifest=manifest, issues=issues)

@staticmethod
Expand Down
Loading
Loading