From a1f07eaccb3f8d45ac89fd33c84b7994a2996393 Mon Sep 17 00:00:00 2001 From: Krivoblotsky Date: Mon, 14 Sep 2026 10:31:53 +0300 Subject: [PATCH 1/2] =?UTF-8?q?docs(spec):=20RFC-0001=20Scopes=20&=20visib?= =?UTF-8?q?ility=20=E2=80=94=20who=20may=20see=20a=20memory?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Opens the permissions layer for public design. Spec §2 now states plainly that the format does not yet express who may see a memory and points to Spec/rfcs/0001-scopes-and-visibility.md: an optional visibility object (private/group/shared/public + opaque principals), context defaults with per-record overrides, most-restrictive-wins on merge, receiver obligations, adapter mappings, compatibility (target format 1.2.0), conformance additions, alternatives and open questions. Discussion: MacPaw/portable-memory#11 (comment period to 2026-10-05). Spec byte-identical in both repositories. Co-Authored-By: Claude Fable 5.1 --- Spec/portable-memory-spec.md | 13 ++ Spec/rfcs/0001-scopes-and-visibility.md | 195 ++++++++++++++++++++++++ 2 files changed, 208 insertions(+) create mode 100644 Spec/rfcs/0001-scopes-and-visibility.md diff --git a/Spec/portable-memory-spec.md b/Spec/portable-memory-spec.md index 14b0e9f..a887691 100644 --- a/Spec/portable-memory-spec.md +++ b/Spec/portable-memory-spec.md @@ -215,6 +215,19 @@ is. Embeddings are an optional accelerator. (`embeddingsIncluded: false`); the receiver re-embeds on import, restoring vector search without any cross-model risk. +### Scopes & visibility *(status: RFC — not yet normative)* + +The format says *where* a memory belongs (`contextID` → `context` records, which nest via +`parentID`) and *how sensitive* it is (`sensitivity`), but not yet *who may see it* — a +channel, one person, or a group. Vendor visibility settings survive a round-trip only as +opaque `ext` / metadata: lossless, but not meaningful to a receiver. A shared `visibility` +model — four levels (`private` / `group` / `shared` / `public`) plus opaque principals, +context-level defaults with per-record overrides, most-restrictive-wins on merge — is +proposed for format **1.2** in [RFC-0001](rfcs/0001-scopes-and-visibility.md) (discussion: +[issue #11](https://github.com/MacPaw/portable-memory/issues/11)). Until it is adopted, +a receiver SHOULD treat imported memories as private to the importing user unless the +source says otherwise. + --- ## 3. The manifest diff --git a/Spec/rfcs/0001-scopes-and-visibility.md b/Spec/rfcs/0001-scopes-and-visibility.md new file mode 100644 index 0000000..b87d772 --- /dev/null +++ b/Spec/rfcs/0001-scopes-and-visibility.md @@ -0,0 +1,195 @@ +# RFC-0001: Scopes & visibility — who may see a memory + +| | | +|---|---| +| **Status** | Draft — open for comment until 2026-10-05 | +| **Discussion** | https://github.com/MacPaw/portable-memory/issues/11 | +| **Target** | format **1.2.0** (minor, backward-compatible) | +| **Author** | MacPaw (Sergii Kryvoblotskyi) · created 2026-09-14 | +| **Affects** | spec §2 (data model), §3 (manifest), §4 (merge), §11 (conformance); `Schemas/`; both reference SDKs | + +## 1. Problem + +Portable Memory says **where** a memory belongs — `contextID` → `context` records, which nest +via `parentID` — and **how sensitive** it is (`sensitivity`). It does not yet say **who may +see it**. Real memory is scoped: some of it is visible to everyone in a channel or workspace, +some is private to one person, some is restricted to a group. Every source we ingest carries +such a setting — PLUR engrams have `visibility: private | public | template` and +`scope: agent:X | space:X | global`; team memory tools bind memories to a channel; personal +assistants keep everything private to the account owner. + +Today those settings survive a round-trip only as opaque `ext` / namespaced metadata. That is +lossless, but not *meaningful*: a receiving system cannot honor an audience restriction it +cannot read, so a cross-vendor import can silently **widen** who sees a private memory. This +is the one layer of the format that still has "no shared meaning" — the failure the format +was built to remove everywhere else. + +## 2. Goals and non-goals + +**Goals** + +- **G1** Express the three audiences that occur in practice: *visible to a shared scope* + (channel, space, workspace), *private to one person*, *restricted to a group*. +- **G2** Portable **without a shared identity provider**: principals are labels the sender + chose, not accounts the receiver must know. +- **G3** Honorable without an ACL engine: a receiver that can only follow a simple + default-deny rule still does the right thing. +- **G4** Backward-compatible minor bump: 1.0/1.1 readers ignore the new field; 1.2 readers + accept its absence. +- **G5** Lossless: vendor-specific ACLs beyond this model still ride along in `ext`. + +**Non-goals** + +- Enforcement. The format *labels*; the host *enforces*. Conformance can test that a + receiver honors labels (§7), not mandate an access-control architecture. +- Confidentiality of the bundle itself (encryption per scope). Complementary; a separate + RFC. `secretRef` already keeps secret material out of bundles. +- Identity federation or resolving principals across vendors. +- Per-field redaction (see §5 of the spec for whole-record redaction). + +## 3. Proposal + +### 3.1 The `visibility` object + +```json +"visibility": { + "level": "group", + "principals": ["group:platform-team", "user:alice@example.com"] +} +``` + +- **`level`** (required when the object is present) — one of: + - `private` — one person. `principals` SHOULD name exactly one `user:` principal; when + omitted it means *the bundle's owner*. + - `group` — a closed set: exactly the principals listed. + - `shared` — everyone with access to the containing scope (the `context` the record + belongs to, or the scope principal named in `principals`, e.g. `channel:` / `space:`). + - `public` — no restriction. +- **`principals`** (optional, `string[]`, de-duplicated, sorted by Unicode code point on + export like every other list the canonical form sorts) — opaque identifiers of the form + `:`. Recommended types: `user`, `group`, `channel`, `space`, `agent`, `org`. + The format never resolves them (G2). Unknown types MUST be preserved. + +### 3.2 Where it lives + +- **On `context` records** — a *scope default*: applies to every record whose `contextID` + resolves to that context or to a descendant (via `parentID`) unless the record overrides. + This is how "everything in #ops is channel-visible" is said once. +- **On records** — an optional `visibility` on `episode`, `fact`, `entity`, `resource` + (its `chunk`s inherit), `core`, `procedure`, `preference`. It overrides the context + default. One private note inside a shared channel is a per-record override. +- **Derived records** — an `edge` or `fact` whose evidence (`evidenceEpisodeIDs`, + `episodeID`) is restricted inherits the **most restrictive** visibility among its evidence + unless it carries its own. + +### 3.3 Precedence + +``` +record.visibility > nearest context.visibility (walking parentID) > unspecified +``` + +**Unspecified** (no `visibility` anywhere, which is every 1.0/1.1 bundle) means the sender +said nothing. A receiver applies its own default and **MUST NOT widen beyond +private-to-the-importing-user without an explicit user decision.** Default-deny is the only +safe reading of silence. + +### 3.4 Receiver obligations (normative in 1.2) + +1. **Preserve** `visibility` verbatim on re-export, including principals it cannot resolve + (this is L1 losslessness, spec §4). +2. **Honor** it when surfacing memories: show a record to a viewer only if the viewer maps + to a listed principal (`private`, `group`), has access to the scope (`shared`), or the + level is `public`. A principal the receiver cannot map is treated as *no one* — so an + unresolvable `group` is visible only to the importing owner. +3. **Never downgrade on merge.** The order is `private < group < shared < public`; when two + versions of the same record disagree, the merge result carries the lower level and the + intersection of principals. A stale bundle can therefore never re-widen a memory the + owner has since restricted (the mirror of the tombstone no-resurrection rule). +4. Deletion and redaction (§5) are unchanged: a tombstone removes the record for every + audience. + +### 3.5 Manifest summary (optional, 1.2) + +Alongside 1.1's `scopes`, an optional distribution so an auditor sees the audience mix +without opening a stream: + +```json +"visibility": { "private": 12, "group": 3, "shared": 40, "public": 0, "unspecified": 5 } +``` + +### 3.6 Audience-scoped export (SDK behavior, not format) + +Exporters SHOULD offer `export(..., audience=...)`: emit only records the given principal +may see, so a user can hand a teammate "the shared memories, not my private ones". The +manifest's `scopes`/`visibility` summaries describe the *result*. This is a filter, not a +format feature, and needs no spec change beyond this section. + +## 4. Adapter mappings (informative) + +| Source | `visibility` on import | +|---|---| +| Engram / PLUR — `visibility: private`, `scope: agent:X` | `private`, principals `["agent:X"]` | +| Engram / PLUR — `visibility: public`, `scope: space:X` / `global` | `shared` + `["space:X"]` / `public` | +| Engram / PLUR — `visibility: template` | `public` | +| Channel-bound team memory (Slack / Discord tools) | `shared` + `["channel:/"]` | +| ChatGPT / Claude / Gemini personal memory, pasted transfer text | `private` (owner) | +| Cursor / repo-scoped rules | `shared` + `["space:"]` | +| mem0 — `user_id` / `agent_id` / `run_id` | `private` + `["user:"]` / `["agent:"]`; org-scoped → `shared` | + +Adapters keep the source's raw setting in namespaced metadata as today; `visibility` is the +shared meaning layered on top. + +## 5. Compatibility + +- **Format:** minor bump **1.1.0 → 1.2.0**. All new fields optional. A 1.0/1.1 reader + ignores them; a 1.2 reader accepts their absence (§3.3). +- **Losslessness prerequisite.** Unknown-field carriage (`ext`) exists today only on + `episode` (spec §10). For `visibility` on other kinds to survive a re-export by a 1.0/1.1 + reader, `ext` must extend to **every** record kind. That is a separate, prerequisite + change (candidate **RFC-0002**); until it lands, a 1.0/1.1 reader would drop `visibility` + from non-episode kinds on re-export — a losslessness violation we would rather fix than + document. +- **Schemas:** one `$defs/visibility` reused by every kind that may carry it; `context` + gains the same property. +- **Identity & merge (§4.1):** `visibility` is **not** part of any record's identity; the + merge rule in §3.4(3) applies. +- **Both reference SDKs** land the change together with a shared fixture (GOVERNANCE), as + every format change has so far. + +## 6. Conformance + +- **L1** gains: *visibility preserved verbatim on round-trip* (including unknown principal + types). +- **L3** gains an optional probe: *visibility honored* — import a bundle with `private` and + `group` needle memories, then assert a viewer outside the principals cannot retrieve them + through any route the host exposes (the deletion probe, pointed at audience). + +## 7. Alternatives considered + +- **A. Context-only ACL, no per-record field.** Fewer bytes, but cannot express one private + memory inside a shared channel — the most common real case. Rejected; kept as the default + layer (§3.2). +- **B. Full ACL model (allow/deny lists, roles, inheritance flags).** Hosts differ too much + and principals are unresolvable across vendors anyway; the extra expressiveness would not + be honored. Rejected in favor of four levels + labels. +- **C. Reuse `sensitivity`.** Sensitivity is *classification* (how harmful if leaked); + visibility is *audience*. A public-but-sensitive memory and a private-but-mundane one are + both real. Orthogonal; keep both. +- **D. Encrypt per scope instead.** Protects confidentiality in transit and at rest, not + "who may see" inside a receiving host. Complementary; future RFC. + +## 8. Open questions + +1. Principal identifiers: opaque prefixed strings only, or recommend W3C DIDs + (`did:…`) for the `user` type as the concurrent PAM proposal does? +2. Should `shared` **require** a scope principal, or may it be bare (= the record's own + context)? +3. Is *unspecified → receiver default, never widen* the right reading of 1.0/1.1 bundles, + or should absence mean `private`? +4. Should `principals` participate in the manifest `scopes` enumeration (1.1), or stay a + separate summary (§3.5)? + +## 9. Timeline + +Comment period until **2026-10-05** on the discussion issue; decision by lazy consensus +(GOVERNANCE); then implementation in both SDKs + fixture; format **1.2.0**. From 098156b130c1f13eb4b2ec3d3c3befcbd0f1be0c Mon Sep 17 00:00:00 2001 From: Krivoblotsky Date: Mon, 14 Sep 2026 11:03:02 +0300 Subject: [PATCH 2/2] docs(spec): RFC-0002 ext for all kinds, RFC-0003 provenance typing; precise losslessness wording MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit - Spec/rfcs/0002-ext-for-all-kinds.md — foreign-field carriage for every record kind (no wire change; generalized ext sidecar; two generic seam methods; L1 gains the all-kinds round-trip). Prerequisite for RFC-0001. Discussion: MacPaw/portable-memory#13. - Spec/rfcs/0003-provenance-typing.md — claimClass (asserted/observed/inferred/merged/ imported), assertedBy (RFC-0001 principal syntax), mergedFrom; Evidence Pack rows carry them; PROV alignment. Discussion: MacPaw/portable-memory#14. - Spec §10 scope note: ext is specified for episodes only today (other kinds' unknown fields are dropped by the reference readers) → RFC-0002. §2 provenance bullet → RFC-0003. - README: the losslessness claim now says exactly what holds — unknown kinds verbatim, unknown fields on episodes via ext — instead of 'foreign fields round-trip verbatim'. Spec byte-identical in both repositories. Co-Authored-By: Claude Fable 5.1 --- README.md | 4 +- Spec/portable-memory-spec.md | 9 +- Spec/rfcs/0001-scopes-and-visibility.md | 2 +- Spec/rfcs/0002-ext-for-all-kinds.md | 123 +++++++++++++++++++ Spec/rfcs/0003-provenance-typing.md | 155 ++++++++++++++++++++++++ 5 files changed, 288 insertions(+), 5 deletions(-) create mode 100644 Spec/rfcs/0002-ext-for-all-kinds.md create mode 100644 Spec/rfcs/0003-provenance-typing.md diff --git a/README.md b/README.md index a0d6a53..2c79dc4 100644 --- a/README.md +++ b/README.md @@ -51,14 +51,14 @@ This is the data-portability gap of the AI era — the same gap the web closed f |---|---|---| | **Own your memory.** Export it, keep a local copy, move it between assistants and devices. | **Adopt, don't reinvent.** Implement one small protocol and exchange memory with any other adopter. | **No lock-in.** A neutral, local-first alternative to N proprietary silos. | | **Real deletion.** A delete propagates to *every* derived copy, with a recorded proof-of-reach you can verify (supports GDPR / EU AI Act erasure). | **Trust & compliance for free.** Inherit verifiable deletion, an audit trail, and an Evidence Pack you can hand to a compliance team. | **An ecosystem.** Interop is the substrate for portable, composable AI memory. | -| **Inspect it.** Plain text + checksums — open it in any editor. | **Lossless on-ramp.** Foreign fields and record kinds round-trip verbatim, so adopting never costs you data. | **A conformance bar.** A badge that *means* something, gated on the hardest guarantee. | +| **Inspect it.** Plain text + checksums — open it in any editor. | **Lossless on-ramp.** Unknown record kinds round-trip verbatim and unknown fields on episodes ride along as `ext`, so adopting never costs you data (every kind: [RFC-0002](Spec/rfcs/0002-ext-for-all-kinds.md)). | **A conformance bar.** A badge that *means* something, gated on the hardest guarantee. | ## What makes it trustworthy Three properties set this apart from "just another export schema": 1. **Verifiable deletion is the headline, not a footnote.** A deletion is a first-class, portable **tombstone** that propagates to every artifact derived from the content — indexes, vectors, caches, graph edges, replicas — and records *proof of what was removed*. The conformance badge is **gated on this** (level L2): you must prove a deleted item is unreachable via every retrieval route, not merely "removed from a table." -2. **Lossless superset, not a lowest common denominator.** A memory can move *into* the standard and back out — or between two vendors — without losing anything. Fields the standard doesn't model are preserved as `ext`; entire vendor-specific record kinds round-trip **verbatim**. +2. **Lossless superset, not a lowest common denominator.** A memory can move *into* the standard and back out — or between two vendors — without losing what the format doesn't model: entire vendor-specific record kinds round-trip **verbatim**, and fields the standard doesn't model on an episode — the atomic memory — are preserved as `ext`. Extending `ext` carriage to every record kind is proposed in [RFC-0002](Spec/rfcs/0002-ext-for-all-kinds.md). 3. **Local-first and inspectable.** A bundle is a self-contained directory of JSONL + a manifest + a `CHECKSUMS` file. No server is needed to read, verify, or transfer it; source text is the portable truth and embeddings are an optional, model-tagged accelerator the receiver re-derives — so a bundle is never tied to one embedding model. ### Design principles diff --git a/Spec/portable-memory-spec.md b/Spec/portable-memory-spec.md index a887691..f030b8d 100644 --- a/Spec/portable-memory-spec.md +++ b/Spec/portable-memory-spec.md @@ -198,7 +198,9 @@ Vendors **may** add kinds; unknown kinds round-trip verbatim. information loss. - **Provenance** — edges carry `evidenceEpisodeIDs`; facts carry `episodeID`. Every derived assertion ties back to the episodes that justify it. Required for the - Evidence Pack (§7). + Evidence Pack (§7). *Typed* provenance — how a claim came to be (asserted / observed / + inferred / merged / imported), who asserted it, and merge lineage — is proposed for + format 1.2 in [RFC-0003](rfcs/0003-provenance-typing.md). ### Embeddings & engine-agnostic representation @@ -426,7 +428,10 @@ and back out without loss. Two mechanisms make this concrete: - **`ext` (foreign fields).** On import, any key on an `episode` record that is not in the native schema is captured and re-emitted on export. Nothing a vendor attached to - a memory is dropped. + a memory is dropped. *Scope note:* `ext` is specified for `episode` records only; the + reference readers decode the known fields of other kinds and drop the rest. Extending + `ext` to every kind is proposed in [RFC-0002](rfcs/0002-ext-for-all-kinds.md) (target + format 1.2). - **Passthrough (foreign kinds).** An `items/.jsonl` whose kind this engine doesn't recognize is stored raw and re-emitted byte-for-byte. diff --git a/Spec/rfcs/0001-scopes-and-visibility.md b/Spec/rfcs/0001-scopes-and-visibility.md index b87d772..dca704d 100644 --- a/Spec/rfcs/0001-scopes-and-visibility.md +++ b/Spec/rfcs/0001-scopes-and-visibility.md @@ -146,7 +146,7 @@ shared meaning layered on top. - **Losslessness prerequisite.** Unknown-field carriage (`ext`) exists today only on `episode` (spec §10). For `visibility` on other kinds to survive a re-export by a 1.0/1.1 reader, `ext` must extend to **every** record kind. That is a separate, prerequisite - change (candidate **RFC-0002**); until it lands, a 1.0/1.1 reader would drop `visibility` + change (**RFC-0002**); until it lands, a 1.0/1.1 reader would drop `visibility` from non-episode kinds on re-export — a losslessness violation we would rather fix than document. - **Schemas:** one `$defs/visibility` reused by every kind that may carry it; `context` diff --git a/Spec/rfcs/0002-ext-for-all-kinds.md b/Spec/rfcs/0002-ext-for-all-kinds.md new file mode 100644 index 0000000..e85c357 --- /dev/null +++ b/Spec/rfcs/0002-ext-for-all-kinds.md @@ -0,0 +1,123 @@ +# RFC-0002: Foreign-field carriage (`ext`) for every record kind + +| | | +|---|---| +| **Status** | Draft — open for comment until 2026-10-05 | +| **Discussion** | https://github.com/MacPaw/portable-memory/issues/13 | +| **Target** | format **1.2.0** (minor); **prerequisite for RFC-0001** | +| **Author** | MacPaw (Sergii Kryvoblotskyi) · created 2026-09-14 | +| **Affects** | spec §4 (import/merge), §10 (vendor interoperability), §11 (conformance); the store seam in both reference SDKs | + +## 1. Problem + +The format promises a *lossless superset*: whatever a receiver does not model rides along +and comes back out unchanged. Today that promise is kept in two places and broken in a +third: + +1. **Unknown record kinds** — `items/.jsonl` is stored raw and re-emitted + byte-for-byte (§1.2, §10). Kept. +2. **Unknown fields on `episode` records** — captured into the `ext` sidecar on import and + merged back onto the native JSON on export (§10). Kept. +3. **Unknown fields on every other kind** — `entity`, `edge`, `fact`, `context`, … — are + **silently dropped** by both reference readers: the Python codec decodes only the + declared fields (`from_wire`) and Swift's synthesized `Decodable` ignores extra keys. + A vendor field on a fact, or a format-1.2 `visibility` on an entity read by a 1.1 + reader, disappears on re-export. + +The spec is precise about this — §10 says "any key on an `episode` record" — but the +project's READMEs and launch text have said "foreign fields round-trip verbatim" without +the qualifier. This RFC closes the gap in the format; the wording is corrected alongside. + +## 2. Goals and non-goals + +**Goals** + +- **G1** Unknown fields on **any** record kind survive import → export byte-for-byte, as + they already do for episodes. +- **G2** No wire-format change: foreign keys stay where vendors put them — at the top level + of the record line — so a bundle stays readable with `cat`. +- **G3** Minimal store-seam churn: adopters who implement the seam today should not have + to add fourteen method pairs. +- **G4** Make RFC-0001 (and any future optional field) safe to deploy against 1.1 readers. + +**Non-goals** + +- Interpreting foreign fields. They are opaque bytes with a stable home. +- Carrying foreign fields *across* kinds or merging two vendors' foreign fields for the + same record (last-writer-wins at the record level, as today). + +## 3. Proposal + +### 3.1 Wire format — unchanged + +A record line MAY carry keys outside its kind's schema. Canonical ordering (§1.1) sorts +them with everything else. Every kind's JSON Schema sets `additionalProperties: true` +(today `episode` does; the others are checked and aligned in the implementation PR). + +### 3.2 Reader behavior — `ext` for every kind (normative in 1.2) + +On import a reader MUST, for **every** native kind, split each record line into the +native fields it decodes and a foreign remainder, and hand the remainder to the host +keyed by the record's identity. On export it MUST merge the remainder back onto the +native JSON before canonicalization — exactly the `ext` mechanism §10 specifies for +episodes, generalized. A foreign key never overrides a native key. + +**Record identity for the sidecar:** the record `id` for kinds that have one; for id-less +kinds (`factLink`, `episodeLink`, `preference`) the lowercase-hex SHA-256 of the record's +canonical *native* line — stable across implementations because the canonical form is. + +### 3.3 Store seam — two generic methods instead of fourteen pairs + +``` +export_ext(kind: str) -> dict[str, str] # record key → foreign-field JSON object +import_ext(kind: str, key: str, ext: str) -> None +``` + +(Swift: `exportExt(kind:)` / `importExt(kind:key:ext:)`.) Defaults: empty / no-op, so a +store that ignores foreign fields for a kind keeps working and simply is not L1-lossless +for that kind. The existing `export_episode_ext` / `import_episode(e, ext)` remain as the +episode specialization (they are what adopters implement today); the generic methods +default to delegating to them for `kind == "episode"`. + +### 3.4 Conformance + +- **L1 (Import / Merge — lossless)** gains: *foreign fields on every kind round-trip + verbatim*. The shared conformance fixture gains an entity, an edge, a fact and a context + each carrying a foreign field; both SDKs must reproduce the bundle byte-for-byte. +- The validator's "known-kind stream decodes" check is unchanged — foreign keys never make + a record undecodable. + +## 4. Compatibility + +- **Wire:** no change. Every 1.0/1.1 bundle is unaffected. +- **Readers:** 1.0/1.1 readers keep dropping foreign fields on non-episode kinds (the + status quo); 1.2 readers preserve them. Because RFC-0001 places `visibility` on + non-episode kinds, RFC-0001 SHOULD NOT be adopted before this RFC. +- **Format version:** bundled into **1.2.0** with RFC-0001 (a normative reader + requirement is a spec change even though bytes don't move). +- **SDKs:** additive; the two generic seam methods have defaults; no adopter code breaks. + +## 5. Alternatives considered + +- **A. A nested `ext` object inside each record on the wire.** Explicit, but it changes + the wire form of every existing vendor field, breaks the "open it and read it" + property, and would make today's bundles non-canonical. Rejected. +- **B. Document the limitation and leave it.** Cheapest, but it makes "lossless superset" + a claim about one of fifteen kinds — and blocks RFC-0001. Rejected. +- **C. Per-kind method pairs on the seam** (`export_entity_ext`, …). Fourteen more + methods for adopters to learn; the generic pair covers it. Rejected. + +## 6. Open questions + +1. Should the sidecar key for id-less kinds be the canonical-line hash (proposed) or + should those kinds simply gain an `id`? (Giving `factLink` / `episodeLink` / + `preference` ids is a larger change; the hash needs no wire change.) +2. Size bound per record for foreign remainders (`MemLimits`) — one bound for the whole + file today; is a per-record cap needed against pathological input? +3. Should the manifest advertise the capability (`capabilities: ["ext:all-kinds"]`) so a + sender can tell whether a receiver is L1-lossless for every kind? + +## 7. Timeline + +Comment period until **2026-10-05** alongside RFC-0001; decision by lazy consensus +(GOVERNANCE); implementation in both SDKs with the extended fixture; format **1.2.0**. diff --git a/Spec/rfcs/0003-provenance-typing.md b/Spec/rfcs/0003-provenance-typing.md new file mode 100644 index 0000000..8f88186 --- /dev/null +++ b/Spec/rfcs/0003-provenance-typing.md @@ -0,0 +1,155 @@ +# RFC-0003: Provenance typing — how a memory came to be + +| | | +|---|---| +| **Status** | Draft — open for comment until 2026-10-05 | +| **Discussion** | https://github.com/MacPaw/portable-memory/issues/14 | +| **Target** | format **1.2.0** (minor, backward-compatible); depends on RFC-0002 for lossless carriage on non-episode kinds | +| **Author** | MacPaw (Sergii Kryvoblotskyi) · created 2026-09-14 | +| **Affects** | spec §2 (data model — identity, time & provenance), §6 (Evidence Pack), §10 (adapters), §11 (conformance); `Schemas/`; both reference SDKs | + +## 1. Problem + +Provenance in the format today answers *what justifies this?* — an edge carries +`evidenceEpisodeIDs`, a fact carries `episodeID`, every record has `confidence`, updates +supersede rather than overwrite (`supersededBy`), and every mutation lands in the audit +log. It does not answer two questions a reader — or an auditor — asks first: + +- **How did this memory come to be?** Was it *asserted* by a person ("I prefer metric + units"), *observed* by the system (a message that happened), *inferred* by a model + ("the user seems to work in fintech"), *merged* from several earlier records, or + *imported* from another system through an adapter? +- **Who asserted it?** A person, an agent, a model (which one, which version)? + +Those distinctions are exactly what separates a fact from a guess when memory crosses a +vendor boundary, and exactly what an erasure or accuracy review needs ("show me every +*inferred* claim about this person"). Sources carry them — the Engram spec has +`claim_class` (`observed | documented | structural | asserted | inferred | revised`) and +`attribution.asserted_by` / `.model`; conversation exports know the speaker's role — and +our adapters preserve them only as opaque, vendor-namespaced metadata. + +## 2. Goals and non-goals + +**Goals** + +- **G1** A small, shared vocabulary for *how* a memory came to be, usable by every adapter + and understood by every receiver. +- **G2** A typed *who* — the asserting principal — reusing RFC-0001's principal syntax so + attribution and visibility speak the same language. +- **G3** Merge lineage: when records are consolidated, the result names what it replaced. +- **G4** Optional everywhere; minor bump; readers accept absence. + +**Non-goals** + +- A full provenance graph (W3C PROV). We align names with PROV (§6) but carry three + fields, not a graph. +- Trust scoring or truth adjudication. `confidence` stays what it is; this RFC only makes + the *origin* of a claim explicit. + +## 3. Proposal + +Three optional fields on `episode`, `fact`, `edge` and `entity` (facts and edges are +where derived knowledge lives; episodes and entities are where imported and observed +knowledge lands): + +### 3.1 `claimClass` + +One of: + +| value | meaning | typical producer | +|---|---|---| +| `asserted` | stated directly by a person or an authoritative source, kept verbatim in spirit | user says "I use metric units"; a document states a fact | +| `observed` | recorded by the system as it happened | a conversation turn; a captured event | +| `inferred` | derived by a model or rule from other memories | "works in fintech" from several episodes | +| `merged` | the consolidation of two or more earlier records | reconciliation, dedup | +| `imported` | brought in from another system through an adapter, class unknown at source | pasted memory-transfer text; a vendor export with no origin data | + +Receivers MUST preserve unknown values (the enumeration may grow) and SHOULD treat a +record without `claimClass` as `imported` when it arrived through an adapter and as +unspecified otherwise. + +### 3.2 `assertedBy` + +A single principal string in RFC-0001's `:` form naming who made the claim — +`user:alice@example.com`, `agent:claude-code`, `model:gpt-5@2026-06`, `system:mnemos`. +Opaque; never resolved by the format. For `inferred` records this is the model or agent +that inferred; for `merged` records the process that merged. + +### 3.3 `mergedFrom` + +`string[]` — the ids of the records consolidated into this one. Used with +`claimClass: merged`. The merged-away records are expected to be **superseded, not +deleted** (§2 bi-temporal rule), so the lineage stays reconstructible; if a host does +delete them, the ids remain as opaque history. + +### 3.4 Where existing fields stop and these start + +| question | today | with RFC-0003 | +|---|---|---| +| what justifies it? | `episodeID`, `evidenceEpisodeIDs` | unchanged | +| how sure? | `confidence` | unchanged | +| what replaced what? | `supersededBy` (one-to-one, in time) | + `mergedFrom` (many-to-one, by consolidation) | +| how did it come to be? | — | `claimClass` | +| who said so? | `speaker` (the utterer of an episode), `actors` | + `assertedBy` (the *claimant* of a fact/edge/entity — often a model, not a speaker) | + +`speaker` is not renamed: it answers "who was talking" on an episode; `assertedBy` +answers "who stands behind this claim" on any record. + +### 3.5 Evidence Pack (§6) + +The Evidence Pack SHOULD include `claimClass` and `assertedBy` on its provenance rows so a +reviewer can partition a person's memory into asserted / observed / inferred and see +which models produced the inferences — the single most useful cut in an erasure or +accuracy review. + +## 4. Adapter mappings (informative) + +| Source | `claimClass` | `assertedBy` | +|---|---|---| +| Engram / PLUR `claim_class` | `observed`→`observed`, `documented`/`structural`/`asserted`→`asserted`, `inferred`→`inferred`, `revised`→`merged` (with `mergedFrom` if `previous_version_ref` present) | `attribution.asserted_by` / `attribution.model.name` → `user:`/`agent:`/`model:` | +| ChatGPT `conversations.json` turns | `observed` | `user:` for user turns, `agent:chatgpt` (+ `model:`) for assistant turns | +| Claude memory files | `asserted` | `agent:claude` | +| Pasted memory-transfer text | `imported` | `agent:` (the assistant that produced the export) | +| mem0 memories (LLM-extracted) | `inferred` | `agent:mem0` | + +Adapters keep the raw source values in namespaced metadata as today. + +## 5. Compatibility + +- **Format:** minor bump, bundled into **1.2.0** with RFC-0001/0002. All fields optional. +- **Old readers:** ignore the fields; for non-episode kinds they would drop them on + re-export until RFC-0002 lands — hence the dependency. +- **Identity & merge (§4.1):** none of the three fields is part of record identity. On + merge, `claimClass` follows the surviving record; `mergedFrom` lists are unioned. +- **Schemas:** a shared `$defs/claimClass` enumeration (open — `string` with documented + values, so unknown future values validate) and `$defs/principal` reused from RFC-0001. + +## 6. Alignment with W3C PROV (informative) + +`assertedBy` ≈ `prov:wasAttributedTo`; `mergedFrom` ≈ `prov:wasDerivedFrom`; +`evidenceEpisodeIDs` / `episodeID` ≈ `prov:used`; `claimClass` has no direct PROV +counterpart (PROV types the *activity*, not the epistemic status of the claim) — it is the +field that makes the vocabulary useful to a reviewer without a graph query. + +## 7. Conformance + +- **L1** gains: the three fields round-trip verbatim (via RFC-0002 for non-episode kinds). +- **L3 / Evidence Pack** gains: provenance rows carry `claimClass` and `assertedBy` when + the host has them. + +## 8. Open questions + +1. Is five classes the right granularity? (`documented` — from a file — vs `asserted` — + from a person — is a distinction the Engram spec keeps and this RFC collapses.) +2. Should `assertedBy` allow a list (co-asserted claims), or stay singular with + `mergedFrom` covering multi-origin cases? +3. Should `episode` carry `claimClass` at all, or is `sourceType` (`chat` / `note` / + `event` / …) already that field for episodes? (Proposed: allow both; `sourceType` is + the *channel*, `claimClass` the *epistemic status*.) +4. Model identifiers: recommend `model:@` or leave the id fully opaque? + +## 9. Timeline + +Comment period until **2026-10-05** alongside RFC-0001 and RFC-0002; decision by lazy +consensus (GOVERNANCE); implementation in both SDKs with fixture coverage; format +**1.2.0**.