diff --git a/changelog.d/tool-result-blocks.added.md b/changelog.d/tool-result-blocks.added.md new file mode 100644 index 000000000..60d208237 --- /dev/null +++ b/changelog.d/tool-result-blocks.added.md @@ -0,0 +1 @@ +- `ToolResult.dataIsContent` says a result's array `data` is a protocol result's content, in which every element is a block, of a type the host can show or not. The model then reads each element as itself, as a reference's stub, or as a marker saying what it was, never as the array's JSON. agent-framework sets it on an MCPL tool's result, and a module that returns a content array can set it too. diff --git a/changelog.d/tool-result-blocks.changed.md b/changelog.d/tool-result-blocks.changed.md new file mode 100644 index 000000000..4a41a4cf6 --- /dev/null +++ b/changelog.d/tool-result-blocks.changed.md @@ -0,0 +1,5 @@ +- In an MCPL tool's result, or a module's, a link (MCP's `resource_link`) the host can't fetch whose uri may carry a credential now shows a stub that says why it can't be fetched, with no uri, where the result's JSON showed the uri. So an agent no longer sees that uri, and has no other handle on the link here. Such a uri has userinfo, or a query or fragment parameter that has a value and is named like a signed URL's or a token, such as `X-Amz-Signature`, `sig`, `hm`, `token`, `access_token` or `key`, compared by letters and digits alone, so `accessToken` and `token[]` count too. A parameter's value that is itself a uri is read the same way, whether absolute (an image proxy's `?url=` holding a percent-encoded signed URL), protocol-relative, or relative with a query or fragment (a login's `?next=%2Freset%3Ftoken%3D…`), and also once decoded again where a redirector encoded it twice, each read and decode a level, up to four; a uri reached deeper counts too, and one in another encoding, such as base64, isn't read. RFC-005 §5 gives the reason: signed URLs and query capabilities are bearer credentials that look like locations. So does a link to a uri that any reference in the same result marks `disposition: never`, or to a uri that shows as the same text (a label's cleaning drops controls and bidi marks and trims surrounding space, and a lone surrogate shows as U+FFFD), or, where no reference in the result names the uri (a link names it without claiming anything), whose reference from an earlier result, as the host's record of the uri holds it, did (the latest reference until its bytes are saved, then the one they were verified against), since RFC-005 §5 removes such a uri unconditionally. An ordinary link with a query, such as `watch?v=…`, and an anchor such as `#token`, which has no value, still show their uri. A standard MCP server's result, a modern server's or an MCP-only one's, isn't read this way: its links show their uri as the server sent it. +- In a script's structured result, from an MCPL peer or a module, a reference arrives as its object without its uri, as before, except a link (MCP's `resource_link`), which carries `uri` where the model's line for it shows the uri. One the fetcher would refuse carries `unfetchable`, the words a stub gives in place of advising `fetch_reference`, as one that counts as expired carries `expired`. Both give why a `fetch_reference` with the object's `refId` would be refused, as the result's references claim it, so a link whose line shows its uri carries them too. An MCPL peer's link reached a script as the unknown-block notice, and a module's as its fields, its uri among them. A script's view reads the claims of every reference it shows, those nested in a block's fields included, so a `disposition: never` nested in one block withholds that uri from a link elsewhere in the same result, where the model's line, which shows a block and not its fields, can still show it. A link's object carries its uri only where the model's line for the result shows it, by the claims that line reads and the uri's record as that line leaves the registry, before the script lane's own registrations, so a script never sees, through a result's blocks, a uri the model's line withholds, within 2,000 references in one result: past that, either can lose an earlier result's `never`, and a script reading a module's content array can lose it where the line, rendering in order, hasn't yet. A result's structured content reaches a script as sent, as before. A script's demand fetch of an MCPL peer's references reads the same claims as its view, as a module's does, so a uri a reference nested in a block calls expired isn't fetched. +- A uri inline data marks `disposition: never` beside its data, which is withheld, is withheld from every other block of the result too, such as a link naming it, as a reference's `never` is (RFC-005 §5). +- Inline data beside an empty `uri` reaches a script as sent, from an MCPL peer as from a module, where 0.26.1 and 0.26.2 gave a script an MCPL peer's as the withheld notice. An empty uri carries nothing a credential could ride on. Inline data beside one that also claims a disposition is still withheld. +- A tool result's data that can't be read through to apply RFC-005's rules, or shown once it has been, shows `[Result not shown: the tool ran, but its result couldn't be read through, checked for references and shown, so this host withheld it.]` on the model's lane, in the stored and sent copies alike, with a log line naming the tool and the error. It was shown with none of its references' claims where the walk reading them stopped and the render's didn't, or, for data no JSON can hold, such as a cycle or a BigInt, a getter that throws, or data too deep to write, the copies' serializer threw. So a host passing such a result to the direct Agent API gets the notice where it got an exception, with the tool-result guard off; with it on, its archive of the result, and the fields it keeps of a result it withholds, still throw, as before, as does an error result whose own error field can't be read. A script meets it as a failure naming the tool, on its text route as on its structured view, and from an MCPL peer's structured view as a result it can't convert. diff --git a/changelog.d/tool-result-blocks.fixed.md b/changelog.d/tool-result-blocks.fixed.md new file mode 100644 index 000000000..2427e6063 --- /dev/null +++ b/changelog.d/tool-result-blocks.fixed.md @@ -0,0 +1,7 @@ +- A tool result's content array with a block agent-framework couldn't show, such as inline audio, an image with no media type or no data, or a block of a type it doesn't know, was stored, and sent to the model, as the whole array's JSON. Inline data's base64 went into the agent's context and history, and so did a block's uri, and an image beside it went as base64 text instead of an image. Each block now shows on its own, in the stored copy and the one the model reads alike: as itself, as a reference's stub, or as a marker saying what it was and why it isn't shown. Beside an image, the model reads them as blocks while the result's links and markers fit its inline cap together, and past that it reads the stored text, cut at the cap. Inline audio reads `[Audio not shown: this host's tool results carry only text and images (, ).]`, and the other blocks read the markers the push lane shows for them. An MCPL tool's result is content whatever its blocks' types. Any other array, such as a module's that doesn't set `dataIsContent`, is content only when every element is a block of one of MCP's types (text, image, audio, resource, resource_link). Any other array is data, written as JSON as before. Results stored before the upgrade keep the text they were stored with. +- MCP's `resource_link` is a reference (RFC-005 §8.1): its `uri`, `name`, `mimeType` and `size` (as `sizeBytes`) make its record, with no disposition. On every lane, a link `fetch_reference` can fetch shows as a reference's stub, with its name, type and size. In a tool result, whose JSON showed every link's uri, one it can't fetch shows as `[resource link "": , ; not fetched]`, keeping the uri, the agent's only handle on it, as a standard MCP server's link shows. That holds at any depth in a tool result's data too, where a link's fields were written as data, and a link whose uri is empty, null or not a string is the marker that says so. A link that names no uri at all is the marker as a block of a content array the model reads, and at depth, or in a script's view of a module's result, stays data, as before, since its fields hold no uri this reads. That covers a uri that isn't http(s), one off its server's origin, one from a server with no address, and a link no server delivered. Where the uri may carry a credential, the link shows a stub instead (under Changed), and so does a link whose uri is a `data:` uri, which holds its content rather than naming where it lies. On every other lane, a link that can't be fetched shows the stub that says why it can't be fetched, with no uri, where the push lane showed an unknown-block marker. A link is registered with the server that sent it, but it's never fetched eagerly or on a script's demand: a uri a link names is fetched only when a block of the result that isn't a link names it too. A link's claims never replace a reference's in the registry, so a later link to a uri can't lift its expiry or its digest check. +- A reference's stub now offers `fetch_reference` only where a fetch can reach the reference through a server that delivered it and the reference hasn't expired, as the servers' configs have it. Where two frameworks in one process configure one server id at different urls, a stub reads the address recorded most recently by a framework that hasn't since removed the server or stopped. A framework records its servers' addresses as it configures them and again as it starts. Otherwise it ends with why `fetch_reference` would refuse, without what calling again can do. The stub reads an expiry from the references in its result where they name its uri, and the fetch from the reference as the host last recorded it, so where one result names a uri both expired and fresh, the stub says it's expired while a fetch can succeed. A fetch's own limits, a spent budget or a claim over its ceiling, still apply when it's made. That covers a server with no address, such as a stdio one, a server the host doesn't know, one whose address doesn't parse, a uri off its server's origin, one with a user name or password, or a scheme the fetcher doesn't fetch. A third-party origin and a refused scheme are described there without naming the origins or the scheme. Before, the stub offered the fetch for any reference a server delivered. A server's id in a stub, or in `fetch_reference`'s refusal, is bounded and stripped of controls, as a label is, and so is a scheme in that refusal. +- A label in a stub (a name, a type, a server's id, a provenance or a saved path) loses any line or paragraph separator (U+2028, U+2029), as it loses control characters. +- A script's demand fetch of a reference not yet fetched reads that reference's own digest and expiry, where it read whatever an earlier reference to the same uri had claimed. +- Inline data beside an empty `uri` is shown as any inline data is, on every lane, where it was withheld as data that also claimed a link. +- On the tool-result lane, an image whose data or media type is an empty string shows its marker, as on the push lane, where it was sent to the model as an image with that field empty. diff --git a/src/agent.ts b/src/agent.ts index c939288b8..db257c9a4 100644 --- a/src/agent.ts +++ b/src/agent.ts @@ -2096,7 +2096,7 @@ export class Agent { toolUseId: r.id, content: isError ? truncateForHistory(typeof error === 'string' ? error.toWellFormed() : error as string, DEFAULT_TOOL_RESULT_INLINE_MAX_CHARS) - : nonErrorResultText(r.result, DEFAULT_TOOL_RESULT_INLINE_MAX_CHARS), + : nonErrorResultText(r.result, DEFAULT_TOOL_RESULT_INLINE_MAX_CHARS, { toolName: r.name }), isError, }; }); diff --git a/src/framework.ts b/src/framework.ts index 1e62c4120..203b72478 100644 --- a/src/framework.ts +++ b/src/framework.ts @@ -1,7 +1,7 @@ import { ToolPresentation, presentationTools, isPresentationTool, renderCatalogue, type PresentationSnapshot } from "./tool-presentation.js"; import { constants as bufferConstants } from 'node:buffer'; import { dirname, join } from 'node:path'; -import { INLINE_WITHHELD_TEXT, REFERENCE_LIMITS, classifyBlock, isInlineContradiction, isReferenceExpired, referenceBlockForScript, referenceRegistry, referenceStubOrNull, sanitizeLabel } from './mcpl/references.js'; +import { EXPIRY_REASON, REFERENCE_LIMITS, claimedExpiry, classifyBlock, isReferenceExpired, referenceBlockForScript, referenceRegistry, sanitizeLabel, type RegistryJournal, type ResultClaims, type ScriptLine, type ServerAddress } from './mcpl/references.js'; import { ReferenceFetcher, DEFAULT_FETCH_MAX_BYTES, EAGER_FETCH_TIMEOUT_MS } from './mcpl/reference-fetcher.js'; import { appendFileSync, closeSync, existsSync, fsyncSync, mkdirSync, openSync, readFileSync, renameSync, unlinkSync, @@ -115,13 +115,19 @@ import { ProviderWaits, EVERY_MODEL, isReleaseModel, withoutControls, type Provi import type { WorkspaceModule } from './modules/workspace/index.js'; import type { ToolResultGuard, WithheldSpill } from './tool-result-guard.js'; import { + goesNative, nonErrorResultText, + nonErrorResultTextOrThrow, + isContentArray, + resultClaimsOrThrow, + scriptViewClaims, shownToScript, shownToScriptFromServer, toolResultDataToHistoryString, truncateForHistory, DEFAULT_TOOL_RESULT_INLINE_MAX_CHARS, type HistoryBlockSpan, + type ToolResultBlock, } from './tool-result-history.js'; import { TOOL_IMAGE_REF_RE, @@ -523,7 +529,7 @@ const SEND_ENGAGEMENT_TOOLS = new Set([ function extractChannelIdFromToolResult( content: Array<{ type: string; text?: string }> | undefined, ): string | undefined { - const text = content?.find((c) => c.type === 'text' && c.text)?.text; + const text = content?.find((c) => c?.type === 'text' && c.text)?.text; if (!text) return undefined; try { const parsed = JSON.parse(text) as { channelId?: unknown }; @@ -535,6 +541,33 @@ function extractChannelIdFromToolResult( } } +/** A tool result's stored copy as the history path wrote it: its text, + * spilled or cut at the cap, the spill file, and, where its live copy goes + * as native blocks (goesNative), the blocks the text was made from, which + * the live copy is built from (toMembraneToolResult). */ +interface StoredToolResult { + text: string; + filePath: string | null; + live: ToolResultBlock[] | null; +} + +/** + * How the host reaches an MCPL server, for fetching its references: the url + * it dials the server at, as the transport decides. None for a server the + * host spawns, as a stdio server, nor for a WebSocket server whose url the + * transport won't dial, which buildWebSocketUrl refuses: one that isn't a + * ws:// or wss:// url, or has a fragment. + */ +function serverAddressOf(cfg: McplServerConfig): ServerAddress { + if (!isWebSocketTransport(cfg) || !cfg.url) return 'no-address'; + try { + buildWebSocketUrl(cfg, null); + } catch { + return 'no-address'; + } + return { url: cfg.url }; +} + /** * The text of an MCPL tools/call result: its text blocks' text, joined by * line breaks, with empty ones left out. A text block whose text isn't a @@ -543,8 +576,10 @@ function extractChannelIdFromToolResult( * too, has no text. */ function mcplResultText(content: McpToolResultContent[] | undefined): string { + // An element that isn't a block, null among them, has no text: it shows as + // its marker where the content is shown block by block. return (Array.isArray(content) ? content : []) - .filter((c) => c.type === 'text') + .filter((c) => c?.type === 'text') .map((c) => (typeof c.text === 'string' ? c.text : NON_STRING_TEXT_PLACEHOLDER)) .filter((text) => text !== '') .join('\n'); @@ -1084,6 +1119,15 @@ const SCRIPT_CALL_NO_RESULT = new WeakSet(); */ const SCRIPT_RESULT_UNCONVERTIBLE = new WeakMap(); +/** + * The claims of an MCPL peer's blocks as a script's view reads them + * (scriptViewClaims), or what stopped their walk, read once at dispatch, by + * the content array the result carries: its demand fetch reads them, and the + * script lane's reading of the same blocks takes them from here, so the two + * read the same claims (Ada-1017, room-203 #201013). + */ +const SCRIPT_VIEW_CLAIMS = new WeakMap(); + /** * One logical turn's routing state and receipt tallies * (AgentFramework.turnRouting): everything the turn has decided about where @@ -4060,6 +4104,11 @@ export class AgentFramework { // and withheld spills, closed for the generation a stop ended, are open // for this one. this.lifecycleGeneration++; + // Its servers' addresses, which an earlier stop() took out of the + // reference registry, for its stubs. + for (const config of this.mcplServerConfigs.values()) { + referenceRegistry.setServerAddress(config.id, serverAddressOf(config), this); + } // A sleep kept through a stop() wakes this lifecycle at its end, or now // if it ended while stopped. Before the loop's first pass, so that a wake // owed now joins the requests already pending in one turn. @@ -4578,6 +4627,10 @@ export class AgentFramework { shutdownPromises.push(this.mcplServerRegistry.closeAll(teardownTimeoutMs)); } shutdownPromises.push(this.closeModernMcpConnections(teardownTimeoutMs)); + // A stopped framework's servers' addresses leave the reference registry, + // so no other framework's stub reads them and the registry doesn't keep + // this one reachable. start() records them again. + step(() => { for (const id of this.mcplServerConfigs.keys()) referenceRegistry.forgetServerAddress(id, this); }); // Timed-out or failed teardown retains its context and may still write. // Sync available state best-effort, but leave its store open for a retry. for (const result of await Promise.allSettled(shutdownPromises)) { @@ -6162,18 +6215,8 @@ export class AgentFramework { (serverId) => { const cfg = this.mcplServerConfigs.get(serverId); if (!cfg) return undefined; - // A server's address is the url the host dials it at, as the - // transport decides: none for a server the host spawns, as a stdio - // server, nor for a WebSocket server whose url the transport won't - // dial, which buildWebSocketUrl refuses: one that isn't a ws:// or - // wss:// url, or has a fragment. - if (!isWebSocketTransport(cfg) || !cfg.url) return 'no-address'; - try { - buildWebSocketUrl(cfg, null); - } catch { - return 'no-address'; - } - return { url: cfg.url, token: cfg.token, autofetch: cfg.autofetch }; + const address = serverAddressOf(cfg); + return address === 'no-address' ? address : { url: address.url, token: cfg.token, autofetch: cfg.autofetch }; }, // A null tells the agent only that the payload couldn't be stored; // why, which can name the host's paths, goes to the log. @@ -6205,6 +6248,14 @@ export class AgentFramework { return this.referenceFetcherInstance; } + /** Configure an MCPL server: its config, for tool routing and the fetcher, + * which reads a server's address and credential from it, and its address + * in the reference registry, where a reference's stub reads it. */ + private setMcplServerConfig(config: McplServerConfig): void { + this.mcplServerConfigs.set(config.id, config); + referenceRegistry.setServerAddress(config.id, serverAddressOf(config), this); + } + /** Register every RFC-005 reference in an MCPL peer's tool-result content * against its server, then fetch by `mode`: * - `register`: nothing. An in-process caller (executeMcplToolCall) may @@ -6216,12 +6267,15 @@ export class AgentFramework { * the eager ceiling, connection origin — never worse than the server * having inlined the bytes). Failures are non-fatal: the stub then says * fetch_reference instead of a path. - * - `script`: as `model`, then fetch every reference ("eagerly lazy"): a + * - `script`: as `model`, then fetch every reference but a link ("eagerly lazy"): a * script receiving a result is the strongest available signal that the * bytes are wanted (scripts process, they do not browse), so the stub it * receives carries a workspace path it can open directly. - * A uri the result calls expired is fetched in no mode. */ - private async autoFetchReferences(content: unknown, serverId: string, mode: 'register' | 'model' | 'script'): Promise { + * A uri the result calls expired is fetched in no mode: by its blocks, or + * by `claims` where given, those of everything a script's view of it + * shows (scriptViewClaims), nested fields included, as the module + * route's demand fetch reads them (Ada-1017, room-203 #201013). */ + private async autoFetchReferences(content: unknown, serverId: string, mode: 'register' | 'model' | 'script', claims?: ResultClaims): Promise { if (!Array.isArray(content)) return; // Register every block first, then judge each uri once, by its latest // record. The registry keys records by uri and evicts the one registered @@ -6231,17 +6285,30 @@ export class AgentFramework { // record here, hence the key by uri. A uri that any of this result's // blocks calls expired takes no slot, whichever block registered last. A // later result's fresh expiry still supersedes, through register(). - const byUri = new Map(); + // A link (MCP's resource_link) is registered, so the agent can fetch it, + // but never fetched here: a tool's links are what it offers to follow, + // and fetching them would spend the server's fetch budget on bytes + // nobody asked for. A uri a link names is fetched only when a block of + // the result that isn't a link names it too. + const byUri = new Map(); for (const block of content) { const c = classifyBlock(block); if (c.kind !== 'reference') continue; const record = referenceRegistry.register(c.testimony, serverId); - const expiredHere = byUri.get(c.testimony.uri)?.expiredHere === true || isReferenceExpired(c.testimony); - byUri.set(c.testimony.uri, { record, expiredHere }); + const prior = byUri.get(c.testimony.uri); + const expiredHere = prior?.expiredHere === true || isReferenceExpired(c.testimony); + const linkOnly = (prior?.linkOnly ?? true) && c.testimony.fromLink === true; + byUri.set(c.testimony.uri, { record, expiredHere, linkOnly }); } if (mode === 'register') return; const fetcher = this.getReferenceFetcher(); - const live = [...byUri.values()].filter(({ expiredHere }) => !expiredHere).map(({ record }) => record); + const nowMs = Date.now(); + const live = [...byUri.values()] + .filter(({ record, expiredHere, linkOnly }) => !expiredHere && !linkOnly + && (claims === undefined || Boolean(record.fetchedPath) || claimedExpiry(record, claims, nowMs) === null)) + .map(({ record }) => record); // Parallel with a count cap: N small fetches cost one round-trip of // wall-clock, and anything past the cap stays lazy (stub says so). const MAX_EAGER_PER_RESULT = 8; @@ -12309,7 +12376,7 @@ export class AgentFramework { this.pendingAssistantBlocks.delete(agent.name); } const membraneResults = currentState.toolResults.map(tc => - this.toMembraneToolResult(tc.id, tc.result, maxChars, spilled.get(tc.id)) + this.toMembraneToolResult(tc.id, tc.result, maxChars, spilled.get(tc.id), tc.name) ); agent.toolResultGuard.storeResults(toolResultContent, membraneResults, currentState.toolResults, turnRowMetadata, keptCap); @@ -20125,7 +20192,7 @@ export class AgentFramework { bound, ); agent.toolResultGuard.storeResults(toolResultContent, readyState.toolResults.map((tc) => - this.toMembraneToolResult(tc.id, tc.result, cap, spilled.get(tc.id))), readyState.toolResults, + this.toMembraneToolResult(tc.id, tc.result, cap, spilled.get(tc.id), tc.name)), readyState.toolResults, this.silentTurnRowMetadata(agent.name), keptCap); // The response is already complete: this batch is never // submitted in this turn, so it cannot be refused. Admit it. @@ -21668,10 +21735,12 @@ export class AgentFramework { // agent's own fetch_reference needs, on bytes nobody asked for. A // failure keeps only its text, so no stub is made of its references and // they aren't registered. Only an MCPL peer's blocks are references; a - // modern server's and an MCP-only peer's are standard. + // modern server's and an MCP-only peer's are standard. An MCPL peer's + // content is marked as content (dataIsContent), as on the model's path. if (server && server.capabilities !== null) { await this.autoFetchReferences(result.content, serverId, 'register') .catch((e) => console.error('[rfc005] reference registration error:', (e as Error).message)); + return Array.isArray(direct.data) ? { ...direct, dataIsContent: true } : direct; } return direct; } catch (err) { @@ -22063,12 +22132,12 @@ export class AgentFramework { bound: number | undefined = undefined, ): Promise<{ blocks: ContentBlock[]; - spilled: Map; + spilled: Map; /** Where a withheld stub's kept fields are cut: the share's keptCap * (toolResultShare), which is the cap where there's no bound. */ keptCap: number | undefined; }> { - const spilled = new Map(); + const spilled = new Map(); const dateLabel = new Date().toISOString().slice(0, 10); // The results are stored as one message, whose bound a reader's // strategy spends across them in order. Each is stored as it would be @@ -22077,20 +22146,23 @@ export class AgentFramework { // stored under its share (toolResultShare). A result carrying an image // keeps its live copy cut at the cap; any other's live copy is its stored // text. - const alone = new Map(); + const alone = new Map(); for (const tc of toolResults) { - const { text, shown } = this.storedToolResultText(agentName, tc, maxChars); - alone.set(tc.id, { text, shown, stored: await this.spillOrTruncate(text, maxChars, `${dateLabel}-${tc.id}`, shown, bound) }); + const { text, shown, live } = this.storedToolResultText(agentName, tc, maxChars); + alone.set(tc.id, { text, shown, live, stored: await this.spillOrTruncate(text, maxChars, `${dateLabel}-${tc.id}`, shown, bound) }); } const share = maxChars === undefined ? undefined : toolResultShare(maxChars, bound, toolResults.length); const total = [...alone.values()].reduce((sum, { stored }) => sum + stored.text.length, 0); const byShare = share?.bound !== undefined && toolResults.length > 1 && (this.toolRoundsByShare.has(toolResults) || total > bound!); for (const tc of toolResults) { - const { text, shown, stored } = alone.get(tc.id)!; - spilled.set(tc.id, byShare + const { text, shown, live, stored } = alone.get(tc.id)!; + const kept = byShare ? await this.spillOrTruncate(text, share!.cap, `${dateLabel}-${tc.id}`, shown, share!.bound, stored.written, { readCap: maxChars! }) - : stored); + : stored; + // The live copy is built from the blocks the stored text was made + // from, never classified again (StoredToolResult.live). + spilled.set(tc.id, { text: kept.text, filePath: kept.filePath, live }); } const blocks: ContentBlock[] = toolResults.map(tc => ({ type: 'tool_result' as const, @@ -22116,21 +22188,24 @@ export class AgentFramework { * the image stays reachable by provenance after the bytes leave the wire. * * The live copy is the stored text itself unless the result is a content - * array carrying an inline image, which tryNativeToolResultContent sends - * as native blocks. It takes the same arrays the serializer takes block by - * block, and shows each image and each reference block's stub whole and - * each text block up to where it cuts that block at the cap (`maxChars`, - * toolResultCutAt), so those are the ranges of the text it showed. + * array carrying an inline image whose links and markers fit the cap + * together (goesNative). Then `live` holds the blocks this text was made + * from, which toMembraneToolResult sends as native blocks (nativeContent), + * showing each image, stub and marker whole and each text block up to where + * it cuts that block at the cap (`maxChars`, toolResultCutAt), so those are + * the ranges of the text it showed. */ private storedToolResultText( agentName: string, tc: CompletedToolCall, maxChars: number | undefined, - ): { text: string; shown: ShownRanges | null } { - if (tc.result.isError) return { text: (tc.result.error ?? 'Unknown error').toWellFormed(), shown: null }; + ): { text: string; shown: ShownRanges | null; live: ToolResultBlock[] | null } { + if (tc.result.isError) return { text: (tc.result.error ?? 'Unknown error').toWellFormed(), shown: null, live: null }; const ledger = this.toolImageLedgerFor(agentName); let blocks = null as HistoryBlockSpan[] | null; + let classified = null as ToolResultBlock[] | null; const text = nonErrorResultText(tc.result, undefined, { + toolName: tc.name, imageRef: (blockIndex, image) => ledger.retain({ toolCallId: tc.id, toolName: tc.name, @@ -22138,13 +22213,16 @@ export class AgentFramework { data: image.data, mediaType: image.mimeType, }).ref, - onBlocks: (spans) => { blocks = spans; }, + onBlocks: (spans, result) => { blocks = spans; classified = result; }, }); - if (!blocks?.some((block) => block.kind === 'image')) return { text, shown: null }; + if (!blocks || !classified || !goesNative(blocks.map(({ kind, start, end }) => ({ kind, length: end - start })), maxChars)) { + return { text, shown: null, live: null }; + } return { text, shown: blocks.map(({ kind, start, end }): [number, number] => [start, kind === 'text' && maxChars ? start + toolResultCutAt(text.slice(start, end), maxChars) : end]), + live: classified, }; } @@ -22457,7 +22535,9 @@ export class AgentFramework { * what the model would read: a structured result's blocks as the dispatch * gave them (the direct reading, with RFC-005's treatment of an MCPL peer's * blocks there, bound to the server that sent them), a module's under - * history's reading of RFC-005 at every depth (shownToScript), and + * history's reading of RFC-005 at every depth (shownToScript), each + * reading the claims of everything the script's view shows, nested fields + * included (scriptViewClaims), and * otherwise one text block of the model's text, which for a failure is its * error. A module's is any result whose call the dispatch didn't send to * a server (`toServer`, from the call's waiter). `structuredContent` is the @@ -22475,31 +22555,26 @@ export class AgentFramework { // RFC-005 "eagerly lazy": a script receiving a result is the strongest // available signal that the bytes are wanted (scripts process, they do // not browse), so the stub the script receives carries a workspace path - // it can open directly. An MCPL peer's references were fetched for the - // script at dispatch (autoFetchReferences). Any other tool's content + // it can open directly. An MCPL peer's references, but its links, were + // fetched for the script at dispatch (autoFetchReferences). Any other tool's content // array, a module's, is materialized here. Only a reference registered // against its server can be fetched: a module's result can carry content // it got from a call of its own (ctx.callTool), whose references were // registered on arrival (executeMcplToolCall). A reference a module makes // itself has no server, so its fetch is refused (and logged here) and the // script gets its stub, as it does for a reference nested in other data, - // which isn't fetched. + // which isn't fetched. A link (MCP's resource_link) isn't fetched either: + // a tool's links are what it offers to follow, and the script can fetch + // the ones it wants with fetch_reference. // Whenever the result's blocks reach the script: a structured failure's // included, as a module's error can carry content. // Not a server's result: a module's, whose blocks no dispatch has given // RFC-005's treatment. const fromModule = !toServer; - if ((!isError || structured) && Array.isArray(result.data) && fromModule) { - const fetcher = this.getReferenceFetcher(); - for (const block of result.data) { - const c = classifyBlock(block); - if (c.kind !== 'reference') continue; - const record = referenceRegistry.findByUri(c.testimony.uri) - ?? referenceRegistry.register(c.testimony); - const outcome = await fetcher.fetch(record); - if (!outcome.ok) console.error(`[rfc005] script-demand fetch of ${record.refId} failed: ${outcome.error}`); - } - } + // A structured result's blocks, as #538 gives them: none when its content + // isn't a list, the structured view carrying the result. A failure with + // no blocks reads as its error text. + const showsBlocks = structured && (!isError || (Array.isArray(result.data) && result.data.length > 0)); const failed = (err: unknown): ToolAnswer => { const error = unconvertibleScriptResult(toolName, err, isError ? result.error : undefined); return { @@ -22509,55 +22584,145 @@ export class AgentFramework { brief: error, }; }; - // An MCPL peer's result whose blocks JSON couldn't hold, so that RFC-005's - // treatment couldn't be given them at dispatch. - if (SCRIPT_RESULT_UNCONVERTIBLE.has(result)) return failed(SCRIPT_RESULT_UNCONVERTIBLE.get(result)); - let content: unknown[]; - let text: string | undefined; - // A structured result's blocks, as #538 gives them: none when its content - // isn't a list, the structured view carrying the result. A failure with - // no blocks reads as its error text. - if (structured && (!isError || (Array.isArray(result.data) && result.data.length > 0))) { - content = Array.isArray(result.data) ? result.data : []; - // A module's blocks, such as an MCPL peer's it got from ctx.callTool - // and returns whole, are read as history reads a module's data: what - // RFC-005 governs, at any depth, as the script lane shows it, and - // anything else as sent. What a script prints comes back to the model, - // so no uri the model's lanes withhold reaches it. What this gives is - // a stub, a notice or the value itself, so a module that returns - // blocks it already gave has them left as they are. - if (fromModule) { + // The claims a module's result is shown by, read once, so the fetches + // below and what the script gets read the same: a structured result's + // view's (scriptViewClaims), nested fields included, or the model's text's + // (resultClaimsOrThrow, which throws where resultClaimsOf answers null). + // Where the walk stops, the script's answer is a + // failure naming the tool, before any fetch or treatment, so neither runs + // without them (Ada-1017, room-203 #200250, #200374): on both of a + // script's routes, `isError` says nothing could be shown, where the + // model's lane shows UNCHECKED_RESULT_TEXT. + let claims: ResultClaims | undefined; + // The model's line for the result, its claims and its records as it reads + // them, which decide where a link's object shows its uri + // (referenceBlockForScript's line). Its records are the registry's as + // the model's line leaves it, before the lane's own registrations + // (openJournal): where the line renders the result's blocks, the demand + // loop below registers them in their order, as the line's render does; + // the treatment then registers, rebinds and evicts as it goes, which the + // line never sees (Ada-1017, room-203 #202683, #203011, #203172). + let line: ScriptLine | undefined; + let journal: RegistryJournal | undefined; + try { + if ((!isError || structured) && Array.isArray(result.data) && fromModule) { + let lineClaims: ResultClaims; try { - content = JSON.parse(JSON.stringify(content, (_key, value: unknown) => shownToScript(value))) as unknown[]; + // The flag as the render reads it, so a value other than a + // boolean decides the same way here (isContentArray reads `true`). + lineClaims = resultClaimsOrThrow(result.data, Boolean(result.dataIsContent)); + claims = showsBlocks ? scriptViewClaims(result.data, false) : lineClaims; } catch (err) { return failed(err); } + // The loop's registrations are the line's own only where the line + // renders these blocks: a content array's, block by block at the top + // level, and not a failure's, whose line is its error. There the loop + // registers links too, in the result's order, and the journal opens + // after it. Elsewhere the line registers nothing (a failure) or + // registers in its own walk's order (data at depth), which the loop + // can't follow, so the journal opens before the loop, which leaves + // links to the treatment: the line's records are then the registry + // before the line's own walk, which can only be stricter, by an + // eviction (Ada-1017, room-203 #203172). Past 2,000 references in one + // result, a content array's loop can evict an earlier never before + // the line, rendering in order, would have (#203184). + const blocksAreLines = !isError && isContentArray(result.data, Boolean(result.dataIsContent)); + if (showsBlocks && !blocksAreLines) { + journal = referenceRegistry.openJournal(); + line = { claims: lineClaims, recordBefore: journal.before }; + } + const fetcher = this.getReferenceFetcher(); + for (const block of result.data) { + const c = classifyBlock(block); + if (c.kind !== 'reference' || (c.testimony.fromLink && !blocksAreLines)) continue; + // Registered, not only found, so a re-issued reference's claims + // replace a link's, or a stale reference's, before the fetch reads + // them, as on any other path, while the record isn't fetched yet. A + // record its uri already has keeps the server it's bound to. + const record = referenceRegistry.register(c.testimony); + // A link of blocks the line renders is registered too, in the + // result's order, and not fetched: the registry then holds its + // records in the order the model's own render makes, so a + // registration here evicts what that render's would, short of more + // references in one result than the registry holds, and the text + // route, which reads the live registry, decides each link's uri as + // the model's line does (Ada-1017, room-203 #203000). + if (c.testimony.fromLink) continue; + // A uri any of the result's references gives an expiry that counts + // isn't fetched, as autoFetchReferences passes over one that any block + // of an MCPL result marks expired. + const expired = record.fetchedPath ? null : claimedExpiry(record, claims, Date.now()); + if (expired !== null) { + console.error(`[rfc005] script-demand fetch of ${record.refId} not made: by the result's claims, the reference is ${EXPIRY_REASON[expired]}`); + continue; + } + const outcome = await fetcher.fetch(record); + if (!outcome.ok) console.error(`[rfc005] script-demand fetch of ${record.refId} failed: ${outcome.error}`); + } + // Where the loop's registrations are the ones the model's line makes, + // the treatment's, below, are the script lane's own, which the + // journal keeps the line's records from (Ada-1017, room-203 #203011). + if (showsBlocks && blocksAreLines) { + journal = referenceRegistry.openJournal(); + line = { claims: lineClaims, recordBefore: journal.before }; + } } - } else { - // As in history, a reference is its stub at any depth and inline data - // claiming a uri or a disposition is withheld (RFC-005). + // An MCPL peer's result whose blocks JSON couldn't hold, so that RFC-005's + // treatment couldn't be given them at dispatch. + if (SCRIPT_RESULT_UNCONVERTIBLE.has(result)) return failed(SCRIPT_RESULT_UNCONVERTIBLE.get(result)); + let content: unknown[]; + let text: string | undefined; + if (showsBlocks) { + content = Array.isArray(result.data) ? result.data : []; + // A module's blocks, such as an MCPL peer's it got from ctx.callTool + // and returns whole, are read as history reads a module's data: what + // RFC-005 governs, at any depth, as the script lane shows it, and + // anything else as sent. What a script prints comes back to the model, + // so no uri the model's lanes withhold reaches it through the blocks + // (structured content goes as sent, as on main). What this gives is + // a stub, a notice or the value itself, so a module that returns + // blocks it already gave has them left as they are. + if (fromModule) { + try { + content = JSON.parse(JSON.stringify(content, (_key, value: unknown) => shownToScript(value, claims, line))) as unknown[]; + } catch (err) { + return failed(err); + } + } + } else { + // As in history, a reference is its stub at any depth and inline data + // claiming a uri or a disposition is withheld (RFC-005), by the claims + // the fetches read, or else the serializer's own reading of them, + // whose walk throws where it stops, failing as a stopped render does. + try { + text = isError + ? result.error ?? 'tool call failed' + : result.data === undefined || result.data === null + ? nonErrorResultTextOrThrow(result) + : toolResultDataToHistoryString(result.data, undefined, { + dataIsContent: Boolean(result.dataIsContent), + ...(claims ? { claims } : {}), + }); + } catch (err) { + return failed(err); + } + content = [{ type: 'text', text }]; + } + let json: string; try { - text = isError - ? result.error ?? 'tool call failed' - : result.data === undefined || result.data === null - ? nonErrorResultText(result) - : toolResultDataToHistoryString(result.data); + json = JSON.stringify({ + content, + structuredContent: structured && result.structured !== undefined ? result.structured : null, + isError, + }); } catch (err) { return failed(err); } - content = [{ type: 'text', text }]; - } - let json: string; - try { - json = JSON.stringify({ - content, - structuredContent: structured && result.structured !== undefined ? result.structured : null, - isError, - }); - } catch (err) { - return failed(err); + return { ok: true, json, isError, brief: text ?? scriptResultBrief(content) }; + } finally { + journal?.close(); } - return { ok: true, json, isError, brief: text ?? scriptResultBrief(content) }; } /** @@ -22688,9 +22853,13 @@ export class AgentFramework { callId: string, afResult: ToolResult, maxChars?: number, - /** Pre-spilled outcome from the history path — used for the non-image - * path so the live wire copy byte-matches what the window stored. */ - precomputed?: { text: string; filePath: string | null }, + /** The history path's outcome: its stored text, spilled or cut, and the + * blocks it was made from where the live copy goes natively. The live + * copy is built from these, never classified again, so the two copies + * agree on every block and on whether it goes natively (goesNative). */ + precomputed?: StoredToolResult, + /** The tool's name, for the log line of a result shown as the notice. */ + toolName?: string, ): MembraneToolResult { if (afResult.isError) { // Same spill policy as successes: reuse the history copy so a giant @@ -22706,79 +22875,63 @@ export class AgentFramework { // blocks natively rather than JSON-stringifying them away. Anything else // goes as text. The error path was handled above, so isError is always // false on these return paths. - const blocks = this.tryNativeToolResultContent(afResult.data, maxChars, precomputed?.filePath ?? null); - if (blocks) { - return { toolUseId: callId, content: blocks, isError: false }; - } if (precomputed !== undefined) { - return { toolUseId: callId, content: precomputed.text, isError: false }; + return { + toolUseId: callId, + content: precomputed.live ? this.nativeContent(precomputed.live, maxChars, precomputed.filePath) : precomputed.text, + isError: false, + }; } - // Without the history path's copy, the text is made as that copy is - // (nonErrorResultText), so a reference's uri stays out of it at any depth - // (RFC-005), a result with no data reads its error text, or no text, and - // never the value undefined. - return { toolUseId: callId, content: nonErrorResultText(afResult, maxChars), isError: false }; - } - - /** - * If `data` is an MCP tool-result content array carrying at least one image, - * convert to Membrane's native ToolResultContentBlock[]. Returns null when - * the array is text-only (let JSON path handle it; saves a code path). - * `maxChars`, when provided, caps each accompanying text block so an image - * inlined alongside an enormous text payload can't blow the context. A - * block is cut at the cap as the stored copy's cut at that cap is - * (toolResultCutAt), never inside a slot, so the stored copy can keep - * exactly the slots this showed (#403); under a readers' bound the stored - * copy's cut can come earlier (cutStoredToolResultToFit). - */ - private tryNativeToolResultContent( - data: unknown, - maxChars?: number, + // Without the history path's copy, one classification makes both the + // decision and the copy, as on that path: the text is made as that copy + // is (nonErrorResultText), so a reference shows at any depth as its stub, + // or a link as its line (RFC-005), a result with no data reads its error + // text, or no text, and never the value undefined; and where it's a + // content array whose links and markers fit the cap beside an image + // (goesNative), the very blocks that text was made from go natively. + let spans = null as HistoryBlockSpan[] | null; + let classified = null as ToolResultBlock[] | null; + const text = nonErrorResultText(afResult, maxChars, { toolName, onBlocks: (s, b) => { spans = s; classified = b; } }); + if (spans && classified && goesNative((spans as HistoryBlockSpan[]).map(({ kind, start, end }) => ({ kind, length: end - start })), maxChars)) { + return { toolUseId: callId, content: this.nativeContent(classified, maxChars, null), isError: false }; + } + return { toolUseId: callId, content: text, isError: false }; + } + + /** + * A content array's blocks (toolResultBlocks) as Membrane's native + * ToolResultContentBlock[], for a caller that found they go natively: at + * least one image, with links and markers that fit the cap together + * (goesNative), which this doesn't check. Each image as itself, each text block + * cut at the cap, and every other block whole, as the text the stored copy + * shows for it. `maxChars`, when provided, caps each text block so an image + * inlined beside an enormous text payload can't blow the context. A block + * is cut as the stored copy is at that cap (toolResultCutAt), never inside a + * slot, so the stored copy can keep exactly the slots this showed (#403); + * under a readers' bound the stored copy's cut can come earlier + * (cutStoredToolResultToFit). Each text is already well formed, as the + * stored copy's is (toolResultBlocks). + */ + private nativeContent( + blocks: ToolResultBlock[], + maxChars: number | undefined, /** Spill file written by the history path — referenced from truncation * notices so an image-adjacent giant text block stays recoverable. */ - spillPath: string | null = null, - ): ToolResultContentBlock[] | null { - if (!Array.isArray(data)) return null; - let hasImage = false; - const blocks: ToolResultContentBlock[] = []; - for (const raw of data) { - if (!raw || typeof raw !== 'object') return null; - const b = raw as { type?: unknown; text?: unknown; data?: unknown; mimeType?: unknown }; - if (isInlineContradiction(b)) { - // RFC-005 vector 2: withhold inline data claiming bulk disposition. - blocks.push({ type: 'text', text: INLINE_WITHHELD_TEXT }); - continue; - } - if (b.type === 'text') { - // A text that isn't a string shows as the converters' placeholder, - // rather than sending the whole array to the JSON fallback. - // Well formed, as the stored copy's text is (nonErrorResultText). - let text = typeof b.text === 'string' ? b.text.toWellFormed() : NON_STRING_TEXT_PLACEHOLDER; - if (maxChars && text.length > maxChars) { - text = text.slice(0, toolResultCutAt(text, maxChars)) - + '\n\n[truncated — original was ' + text.length + ' chars' - + (spillPath ? `; full serialized result: workspace file ${spillPath}` : '') - + ']'; - } - blocks.push({ type: 'text', text }); - } else if (b.type === 'image' && typeof b.data === 'string' && typeof b.mimeType === 'string') { - hasImage = true; - blocks.push({ - type: 'image', - source: { type: 'base64', data: b.data, mediaType: b.mimeType }, - }); - } else if (b.type === 'resource' || ((b.type === 'image' || b.type === 'audio') && typeof (b as { uri?: unknown }).uri === 'string')) { - // RFC-005 reference block → bounded stub. Previously this bailed the - // whole array to JSON, which both inlined the reference verbatim and - // demoted any adjacent image block to stringified base64. - const stub = referenceStubOrNull(b, 'from tool result'); - if (stub === null) return null; - blocks.push({ type: 'text', text: stub }); - } else { - return null; // unknown shape: the stored copy's text, which shows any reference in it as its stub - } - } - return hasImage ? blocks : null; + spillPath: string | null, + ): ToolResultContentBlock[] { + return blocks.map((block): ToolResultContentBlock => { + if (block.kind === 'image') { + return { type: 'image', source: { type: 'base64', data: block.data, mediaType: block.mimeType } }; + } + let text = block.text; + if (block.kind === 'text' && maxChars && text.length > maxChars) { + text = text.slice(0, toolResultCutAt(text, maxChars)) + + '\n\n[truncated — original was ' + text.length + ' chars' + + (spillPath ? `; full serialized result: workspace file ${spillPath}` : '') + + ']'; + } + return { type: 'text', text }; + }); } private toolImageLedgerFor(agentName: string): ToolImageLedger { @@ -25670,7 +25823,7 @@ export class AgentFramework { for (const config of serverConfigs) { const prefix = config.toolPrefix ?? `mcpl--${config.id}`; this.mcplPrefixMap.set(prefix, config.id); - this.mcplServerConfigs.set(config.id, config); + this.setMcplServerConfig(config); } // Find shouldTriggerInference callback: @@ -26619,7 +26772,7 @@ export class AgentFramework { if (owner === config.id && other !== prefix) this.mcplPrefixMap.delete(other); } this.mcplPrefixMap.set(prefix, config.id); - this.mcplServerConfigs.set(config.id, config); + this.setMcplServerConfig(config); if (family === 'modern') { // Registered and wired before its first connect, so the connect in @@ -27484,6 +27637,7 @@ export class AgentFramework { if (owner === id) this.mcplPrefixMap.delete(prefix); } this.mcplServerConfigs.delete(id); + referenceRegistry.forgetServerAddress(id, this); await this.refreshMcplTools(); this.emitMcplToolDiff(oldToolNames, id); @@ -28955,7 +29109,9 @@ export class AgentFramework { if (mcpl) { if (!Array.isArray(direct.data)) return direct; // Each block by the classifier, then what's nested in each, at any - // depth. The blocks came from the wire as JSON, so the round trip + // depth, each reading the claims of everything the script's view + // shows, nested fields included (scriptViewClaims; Ada-1017, room-203 + // #200094). The blocks came from the wire as JSON, so the round trip // changes nothing else. But a value that parsed can still fail to go // back: past some depth JSON.stringify overflows the stack where // JSON.parse didn't (on Node 24.19 an array 5,000 deep does, on Bun @@ -28972,15 +29128,47 @@ export class AgentFramework { // dropped, since they are untreated and the result travels on, to the // waiter and to the listeners of its event's trace. Its structured // content is kept, as every result keeps it, since RFC-005's - // treatment doesn't reach it. - const blocks = direct.data.map((block: unknown) => referenceBlockForScript(block, serverId)); + // treatment doesn't reach it. The view's claims are read first, by + // a walk of their own: where it stops, the treatment doesn't run + // (Ada-1017, room-203 #200250), a RangeError meaning a result it + // can't convert, as below, and anything else a fault. + const unconvertible = (err: unknown): ToolResult => { + if (!(err instanceof RangeError)) throw err; + const dropped: ToolResult = { ...direct, data: [] }; + SCRIPT_RESULT_UNCONVERTIBLE.set(dropped, err); + return dropped; + }; + let claims: ResultClaims; + // The model's line for a peer's content reads its blocks' claims + // alone, and each record as it stood before the treatment registered + // anything (openJournal), which decide where a link's object shows + // its uri (Ada-1017, room-203 #202683). + let lineClaims: ResultClaims; try { - return { ...direct, data: JSON.parse(JSON.stringify(blocks, (_key, value: unknown) => shownToScriptFromServer(value, serverId))) as unknown[] }; + // As the dispatch read them for its demand fetch, where it did. + const read = SCRIPT_VIEW_CLAIMS.get(direct.data); + if (read && 'error' in read) throw read.error; + claims = read ? read.claims : scriptViewClaims(direct.data, true); + lineClaims = resultClaimsOrThrow(direct.data, true); } catch (err) { - if (!(err instanceof RangeError)) throw err; - const unconvertible: ToolResult = { ...direct, data: [] }; - SCRIPT_RESULT_UNCONVERTIBLE.set(unconvertible, err); - return unconvertible; + return unconvertible(err); + } + const journal = referenceRegistry.openJournal(); + try { + const line: ScriptLine = { claims: lineClaims, recordBefore: journal.before }; + const blocks = direct.data.map((block: unknown) => referenceBlockForScript(block, serverId, claims, line)); + // The top-level blocks were treated just above; the pass treats + // what's nested in them. Reading a treated block again would take + // a link object's uri, as shownUri shows it, for a new link, + // registered apart from the one the result named. + const nested = function (this: unknown, _key: string, value: unknown): unknown { + return this === blocks ? value : shownToScriptFromServer(value, serverId, claims, line); + }; + return { ...direct, data: JSON.parse(JSON.stringify(blocks, nested)) as unknown[] }; + } catch (err) { + return unconvertible(err); + } finally { + journal.close(); } } // A standard peer's content that isn't a list of blocks is a malformed @@ -29066,7 +29254,26 @@ export class AgentFramework { if (mcplPeer) { const scriptReads = forScript && (!result.isError || Object.prototype.hasOwnProperty.call(result, 'structuredContent')); - await this.autoFetchReferences(result.content, serverId, scriptReads ? 'script' : 'model') + // A script's demand fetch reads the claims of everything its view + // shows, as the module route's does; where that walk stops, the view + // isn't written (reading()), so nothing is fetched for it, the + // references only registered (Ada-1017, room-203 #201013). + let mode: 'register' | 'model' | 'script' = scriptReads ? 'script' : 'model'; + let claims: ResultClaims | undefined; + // Only where the script reads a view of the blocks (reading()'s own + // condition): without structured content it reads the model lane's + // text, and its fetch passes over what each top-level block's own + // expiry marks, as a model's call does. No fetch reads a `never`. + if (scriptReads && Object.prototype.hasOwnProperty.call(result, 'structuredContent') && Array.isArray(result.content)) { + try { + claims = scriptViewClaims(result.content, true); + SCRIPT_VIEW_CLAIMS.set(result.content, { claims }); + } catch (error) { + SCRIPT_VIEW_CLAIMS.set(result.content, { error }); + mode = 'register'; + } + } + await this.autoFetchReferences(result.content, serverId, mode, claims) .catch((e) => console.error('[rfc005] autofetch error:', (e as Error).message)); } return { result, toolResult: await reading(result, mcplPeer) }; @@ -29257,11 +29464,14 @@ export class AgentFramework { // for callers that expect data to be string-ish (mcplResultText). let textContent = mcplResultText(blocks); if (blocks.length === 0 && hasStructured) textContent = JSON.stringify(result.structuredContent) ?? 'null'; - const hasNonText = blocks.some((c) => c.type !== 'text'); + const hasNonText = blocks.some((c) => c?.type !== 'text'); const structured = hasStructured ? { structured: result.structuredContent } : {}; + // Blocks passed through are marked as content (dataIsContent), so each + // shows as itself, its stub or its marker, never as the array's JSON. return { success: !result.isError, data: result.isError ? undefined : hasNonText ? blocks : (textContent || undefined), + ...(!result.isError && hasNonText ? { dataIsContent: true } : {}), error: result.isError ? (textContent.trim() ? textContent : mcplToolNoWordsText(serverId, callName)) : undefined, isError: result.isError ?? false, ...structured, @@ -29934,8 +30144,8 @@ export class AgentFramework { * Handle the synthesized `read_image` tool: read image bytes from a * workspace mount and return them as a native MCP-shaped content array * (`[{type:'text'…},{type:'image', data, mimeType}]`). The live-turn - * converter (tryNativeToolResultContent → membrane tool_result image - * blocks) delivers the actual image to the model; history storage keeps + * converter (toMembraneToolResult's nativeContent → membrane tool_result + * image blocks) delivers the actual image to the model; history storage keeps * the standard compact `[image: …]` placeholder. */ /** RFC-005 on-demand dereference: [ref id] → workspace file, via the diff --git a/src/mcpl/convert-block.ts b/src/mcpl/convert-block.ts index 13cf18616..50a300ccf 100644 --- a/src/mcpl/convert-block.ts +++ b/src/mcpl/convert-block.ts @@ -91,6 +91,11 @@ export function convertBlock(block: McplContentBlock, provenance: string, server return { type: 'text', text: referenceStubOrNull(block, provenance, serverId) ?? '[reference]' }; default: { + // MCP's link to a resource isn't one of MCPL's own blocks, but a server + // can send one, and RFC-005 §8.1 makes it a reference, shown as one. + if ((block as { type?: unknown }).type === 'resource_link') { + return { type: 'text', text: referenceStubOrNull(block, provenance, serverId) ?? '[reference]' }; + } // Unknown wire block types previously fell off the exhaustive switch // and propagated `undefined` into ContentBlock[]. Fail visibly. A type // that isn't a string is no type, and is never converted to one: an diff --git a/src/mcpl/push-coalescer.ts b/src/mcpl/push-coalescer.ts index 9e1f93100..668404db7 100644 --- a/src/mcpl/push-coalescer.ts +++ b/src/mcpl/push-coalescer.ts @@ -446,7 +446,7 @@ export function validateCoalescedContent( */ function showsAsItself(b: Record): boolean { if (b.type === 'text') return typeof b.text === 'string'; - if (b.type === 'resource') return classifyBlock(b).kind === 'reference'; + if (b.type === 'resource' || b.type === 'resource_link') return classifyBlock(b).kind === 'reference'; if (b.type !== 'image' && b.type !== 'audio') return false; if (isInlineContradiction(b)) return false; if (isFilledString(b.uri) && b.disposition !== undefined) return classifyBlock(b).kind === 'reference'; diff --git a/src/mcpl/reference-fetcher.ts b/src/mcpl/reference-fetcher.ts index 46252db79..455e9792c 100644 --- a/src/mcpl/reference-fetcher.ts +++ b/src/mcpl/reference-fetcher.ts @@ -31,9 +31,11 @@ * that the bytes are wanted). Per-server budgets cap cumulative appetite. */ import { createHash } from 'node:crypto'; -import type { ReferenceRecord } from './references.js'; +import type { ReferenceRecord, Refusal, ServerAddress } from './references.js'; import { boundedForLog, describeCause } from './errors.js'; -import { EXPIRY_REASON, noAddressReason, referenceExpiry, REFERENCE_LIMITS, sanitizeLabel, UNBOUND_REASON } from './references.js'; +import { dialCheck, dialServer, FRESH_REFERENCE, NOTHING_TO_DO, REFERENCE_LIMITS, sanitizeLabel, shownServerId } from './references.js'; + +export { normalizedOrigin, type Refusal } from './references.js'; /** Per-fetch hard ceiling on actual bytes. A call can lower it for itself * (fetch_reference's max_bytes, which it clamps to this); nothing raises it. */ @@ -50,11 +52,9 @@ const EAGER_FETCH_TIMEOUT_MS = 10_000; // What calling again can do, as each failure's remedy says it. The fetcher // doesn't know where a reference came from (a tool result, a push, a channel // message), so none names a tool, and fetch_reference's max_bytes is named -// only where a larger one can fetch it. -const NOTHING_TO_DO = "Calling again won't change that."; -/** The registry keys a record by its uri, so a fresh reference to the same - * uri arrives as this very id and refreshes its record. */ -const FRESH_REFERENCE = "Once a fresh reference to it arrives, fetching may work; until then, fetching it again won't."; +// only where a larger one can fetch it. The refusals before dialing that a +// stub shares come from references.ts (dialServer, dialCheck) with their +// remedies (NOTHING_TO_DO, FRESH_REFERENCE, SERVER_BACK). const TRY_LATER = 'Trying again later may work.'; /** A request that failed before an answer: a refused or reset connection * can clear, and a cause that repeats on every try (a port fetch refuses, a @@ -71,8 +71,6 @@ const BUDGET = "The budget doesn't renew on its own, so calling again won't chan /** The host's own ceiling: fetch_reference clamps max_bytes to it, and no * config sets another. */ const HOST_CEILING = "No max_bytes or setting lifts that limit, so calling again won't change it."; -/** The resolver reads the host's server configs as they are at each fetch. */ -const SERVER_BACK = "Calling again won't change that unless whoever runs this host adds that server back."; export interface ResolvedServerBinding { /** The url the host dials the server at. The framework's resolver gives only @@ -90,18 +88,6 @@ export interface ResolvedServerBinding { * know resolves to undefined. */ export type ServerResolution = ResolvedServerBinding | 'no-address'; -/** Why a fetch failed, or would be refused before dialing. */ -export interface Refusal { - /** What happened, as a clause that completes "couldn't fetch : ", - * whatever it's about: fetch_reference's sentence, save_recent_image's, - * and the eager and script-demand fetches' log lines read it. */ - error: string; - /** What calling again can do, as a sentence. fetch_reference and - * save_recent_image show it; the eager and script-demand fetches log the - * error alone. */ - remedy: string; -} - /** A fetch's result: the saved copy, or why it failed, which every failure * says with both a clause and a remedy. The fields the other side lacks are * declared undefined, so a caller can read them before checking ok. */ @@ -116,7 +102,7 @@ export type FetchOutcome = error?: undefined; remedy?: undefined; } - | ({ ok: false; path?: undefined; bytes?: undefined; mimeType?: undefined; digestVerified?: undefined } & Refusal); + | ({ ok: false; path?: undefined; bytes?: undefined; mimeType?: undefined; digestVerified?: undefined } & Pick); const refused = (error: string, remedy: string): { refusal: Refusal } => ({ refusal: { error, remedy } }); const fail = (error: string, remedy: string): FetchOutcome => ({ ok: false, error, remedy }); @@ -191,28 +177,6 @@ const EXT_BY_MIME = new Map(Object.entries({ 'application/pdf': 'pdf', 'application/zip': 'zip', })); -/** A server's address as it's configured, its scheme and host, for a - * clause: never its userinfo, path or query. URL's own origin is 'null' for - * a scheme that isn't special, which normalizedOrigin accepts. The caller - * has already parsed it. */ -function configuredOrigin(url: string): string { - const u = new URL(url); - return `${u.protocol}//${u.host}`; -} - -/** ws→http / wss→https with default ports elided, for origin comparison. */ -export function normalizedOrigin(url: string): string | null { - try { - const u = new URL(url); - const scheme = u.protocol === 'ws:' ? 'http:' : u.protocol === 'wss:' ? 'https:' : u.protocol; - const defaultPort = scheme === 'https:' ? '443' : '80'; - const port = u.port === '' || u.port === defaultPort ? '' : `:${u.port}`; - return `${scheme}//${u.hostname.toLowerCase()}${port}`; - } catch { - return null; - } -} - export class ReferenceFetcher { /** Bytes read through each server in this process, charged to the server a * fetch dials through as each chunk is read, kept or not. Concurrent @@ -222,12 +186,42 @@ export class ReferenceFetcher { private inFlight = new Map>(); constructor( + /** A server's address, credential and limits, by id, from the + * framework's own config: the origin a fetch is checked against and the + * token it carries come from one answer. */ private readonly resolveServer: (serverId: string) => ServerResolution | undefined, /** Persist the payload; returns a mount-prefixed path or null when no * writable mount without an include exists. File name is host-generated (§7.3). */ private readonly saveFile: (fileName: string, data: Buffer, mimeType: string) => Promise, ) {} + /** The server a check (dialServer or dialCheck) chose, with its credential + * and limits from the same resolver answer whose address the check read: + * each server's resolution is read once here, so the origin the check + * passes and the token the fetch sends come from one answer, whatever a + * resolver answers the next time it's asked. */ + private chosen( + check: (addressOf: (serverId: string) => ServerAddress | undefined) => { serverId: string } | { refusal: Refusal }, + ): { serverId: string; binding: ResolvedServerBinding } | { refusal: Refusal } { + const resolutions = new Map(); + const addressOf = (serverId: string): ServerAddress | undefined => { + if (!resolutions.has(serverId)) resolutions.set(serverId, this.resolveServer(serverId)); + const resolution = resolutions.get(serverId); + return resolution === undefined || resolution === 'no-address' ? resolution : { url: resolution.url }; + }; + const server = check(addressOf); + if ('refusal' in server) return server; + // The check chose only a server whose address it read as a url, so that + // server's resolution is a binding. + return { serverId: server.serverId, binding: resolutions.get(server.serverId) as ResolvedServerBinding }; + } + + /** The server a fetch of a record dials (§6.2), with its credential and + * limits, or why none can be. */ + private serverFor(record: ReferenceRecord): { serverId: string; binding: ResolvedServerBinding } | { refusal: Refusal } { + return this.chosen((addressOf) => dialServer(record, addressOf)); + } + /** Would the eager (autofetch) default fetch this record? Not one the fetch * would refuse before dialing, asked as the eager fetch asks it, with the * default ceiling, so such a record takes no eager slot; and only one whose @@ -243,49 +237,22 @@ export class ReferenceFetcher { return record.testimony.sizeBytes !== undefined && record.testimony.sizeBytes <= cap; } - /** The server to dial for a record (§6.2): the first, in the order they - * delivered it, whose address is on the uri's origin. When none is, the - * first server's refusal (unknown to this host, with no address to dial, - * or the origin check's), and for a record no server delivered, that it's - * unbound. */ - private serverFor(record: ReferenceRecord): { serverId: string; binding: ResolvedServerBinding } | { refusal: Refusal } { - let refusal: { refusal: Refusal } | undefined; - for (const serverId of record.serverIds) { - const binding = this.resolveServer(serverId); - if (!binding) { - refusal ??= refused(`this host no longer has the "${serverId}" server it came from`, SERVER_BACK); - continue; - } - if (binding === 'no-address') { - refusal ??= refused(noAddressReason(serverId), NOTHING_TO_DO); - continue; - } - const offOrigin = this.originCheck(record, binding, serverId); - if (offOrigin === null) return { serverId, binding }; - refusal ??= offOrigin; - } - // A later delivery of the same uri by a server binds this record. - return refusal ?? refused(`it's ${UNBOUND_REASON}`, FRESH_REFERENCE); - } - /** Every check the fetch makes before it dials, in its order: the server * (none, unknown, no address, an address of its own or the reference's * that doesn't parse, a reference address carrying a user name or - * password, a scheme other than http: or https:, off the uri's origin), - * the expiry, a claimed size over the host's own ceiling, the server's - * spent budget, a claimed size over a call's ceiling. For a record with no - * saved copy and no fetch under way, the fetch, `refusal()` and the eager - * check all ask this, so what save_recent_image says is the reason the - * fetch would refuse with, and no eager slot goes to a fetch that would be - * refused. */ + * password, a scheme other than http: or https:, off the uri's origin) + * and the expiry, which dialCheck makes as a reference's stub does; then + * a claimed size over the host's own ceiling, the server's spent budget, + * a claimed size over a call's ceiling. For a record with no saved copy + * and no fetch under way, the fetch, `refusal()` and the eager check all + * ask this, so what save_recent_image says is the reason the fetch would + * refuse with, and no eager slot goes to a fetch that would be refused. */ private beforeDial(record: ReferenceRecord, opts?: { maxBytes?: number }): { refusal: Refusal } | { serverId: string; binding: ResolvedServerBinding; ceiling: number; budget: number } { - const server = this.serverFor(record); + const server = this.chosen((addressOf) => dialCheck(record, addressOf)); if ('refusal' in server) return server; const { serverId, binding } = server; const t = record.testimony; - const expiry = referenceExpiry(t); - if (expiry) return refused(`the reference is ${EXPIRY_REASON[expiry]}`, FRESH_REFERENCE); const ceiling = opts?.maxBytes ?? DEFAULT_FETCH_MAX_BYTES; // A claim past the host's own ceiling is refused first: nothing lifts @@ -299,7 +266,7 @@ export class ReferenceFetcher { const budget = binding.autofetch?.maxTotalBytes ?? DEFAULT_MAX_TOTAL_BYTES; const used = this.totals.get(serverId) ?? 0; if (used >= budget) { - return refused(`this host's fetch budget for the "${serverId}" server is used up (${used} of ${budget} bytes)`, BUDGET); + return refused(`this host's fetch budget for the "${shownServerId(serverId)}" server is used up (${used} of ${budget} bytes)`, BUDGET); } // Vector 4: a claim already over the ceiling refuses without dialing. One // that's also over what's left of the budget is told about the budget, @@ -308,7 +275,7 @@ export class ReferenceFetcher { if (t.sizeBytes !== undefined && t.sizeBytes > ceiling) { if (t.sizeBytes > budget - used) { return refused( - `it claims ${t.sizeBytes} bytes, more than the ${budget - used} bytes left of this host's fetch budget for the "${serverId}" server`, + `it claims ${t.sizeBytes} bytes, more than the ${budget - used} bytes left of this host's fetch budget for the "${shownServerId(serverId)}" server`, BUDGET, ); } @@ -331,43 +298,6 @@ export class ReferenceFetcher { return 'refusal' in check ? check.refusal : null; } - /** §6.1/§7.1 policy check; returns a refusal or null. */ - private originCheck(record: ReferenceRecord, binding: ResolvedServerBinding, serverId: string): { refusal: Refusal } | null { - const refOrigin = normalizedOrigin(record.testimony.uri); - if (!refOrigin) return refused("its address doesn't parse as a URL", NOTHING_TO_DO); - // A server's address that doesn't parse leaves nothing to dial. The - // framework's resolver gives only an address its transport dials, so this - // guards other resolvers. - const dialedOrigin = normalizedOrigin(binding.url); - if (!dialedOrigin) { - return refused(`the "${serverId}" server's address doesn't parse as a URL, so there's nowhere to fetch it from`, - "Calling again won't change that unless whoever runs this host corrects the server's address."); - } - const refUrl = new URL(record.testimony.uri); - // A user name or password in the uri is a credential embedded in it, - // which RFC-005 §6.1 forbids a server to mint and lets a host refuse - // (vector 6). This host neither sends nor repeats it: fetch refuses such - // a url with a message that holds it whole. - if (refUrl.username !== '' || refUrl.password !== '') { - return refused("its address carries a user name or password, which this host doesn't send", NOTHING_TO_DO); - } - // The uri's own scheme, not its origin's: an origin reads ws: as http: - // and wss: as https:, and fetch refuses a ws: or wss: uri ('unknown - // scheme') before connecting, which would read as a failed request a - // later try can mend. - const scheme = refUrl.protocol; - if (scheme !== 'https:' && scheme !== 'http:') { - return refused(`its address uses ${scheme}, and this host fetches only http: and https: addresses`, NOTHING_TO_DO); - } - // Origins compare with their schemes, so this also refuses an http: - // reference from a server dialed over TLS (wss://). - if (refOrigin !== dialedOrigin) { - return refused(`it points to ${refOrigin}, not to the "${serverId}" server's own address (${configuredOrigin(binding.url)}), ` + - 'and this host fetches only from the server itself', NOTHING_TO_DO); - } - return null; - } - /** Fetch a reference to workspace storage. Idempotent per record; concurrent * callers share one in-flight fetch. */ fetch(record: ReferenceRecord, opts?: { maxBytes?: number; timeoutMs?: number }): Promise { @@ -385,9 +315,9 @@ export class ReferenceFetcher { private async doFetch(record: ReferenceRecord, opts?: { maxBytes?: number; timeoutMs?: number }): Promise { const t = record.testimony; const check = this.beforeDial(record, opts); - if ('refusal' in check) return { ok: false, ...check.refusal }; + if ('refusal' in check) return fail(check.refusal.error, check.refusal.remedy); const { serverId, binding, ceiling, budget } = check; - const server = `the "${serverId}" server`; + const server = `the "${shownServerId(serverId)}" server`; const timeoutMs = opts?.timeoutMs ?? FETCH_TIMEOUT_MS; const controller = new AbortController(); diff --git a/src/mcpl/references.ts b/src/mcpl/references.ts index 0aa4ea55c..c03f439f5 100644 --- a/src/mcpl/references.ts +++ b/src/mcpl/references.ts @@ -20,7 +20,8 @@ * retires). * * The ReferenceRegistry is the host-private reference record (§6.2): raw - * URIs and testimony live here and never in model-visible content. Fetching + * URIs and testimony live here, and model-visible content shows a uri only + * as a tool result's link line shows one (shownLinkUri). Fetching * (§6/§7 origin binding, ceilings, redirects) is deliberately not here yet — * the registry is what makes a future host-mediated fetcher and the * code-execution materialization hook possible. @@ -49,6 +50,10 @@ export interface ReferenceTestimony { disposition?: ReferenceDisposition; rejectedFields: string[]; truncatedFields: string[]; + /** Set when the testimony is an MCP resource_link's, which claims no + * digest, expiry or disposition. A link is registered but never fetched + * on the side, and its testimony never replaces a record's (register). */ + fromLink?: true; } export type BlockClassification = @@ -92,29 +97,39 @@ export function classifyBlock(block: unknown): BlockClassification { : { kind: 'invalid', shown: NON_STRING_TEXT_PLACEHOLDER }; case 'image': case 'audio': { - const hasData = typeof block.data === 'string'; - const hasUri = typeof block.uri === 'string'; - if (hasData) { - // Inline form claiming bulk disposition (or smuggling a uri beside + // An empty uri is no link, as the push lane's converter reads it. + const hasLink = typeof block.uri === 'string' && block.uri !== ''; + if (typeof block.data === 'string') { + // Inline form claiming bulk disposition (or smuggling a link beside // the data) is the vector-2 contradiction: fail closed, withhold. - // A uri of '' counts here too, though it's no link: until the - // tool-result lane gives every inline shape a marker of its own, - // showing that data would put some of it into history as raw JSON. - // Once that lane does, an empty link beside data reads as no link. - return { kind: 'inline', contradiction: block.disposition !== undefined || hasUri }; + return { kind: 'inline', contradiction: block.disposition !== undefined || hasLink }; } - // With no data, an empty uri is no link, as the push lane's converter - // reads it. - if (hasUri && block.uri !== '') return parseReferenceFields(block); + if (hasLink) return parseReferenceFields(block); return { kind: 'invalid', shown: block.type === 'image' ? IMAGE_NO_DATA_TEXT : AUDIO_NO_DATA_TEXT }; } case 'resource': return parseReferenceFields(block); + case 'resource_link': + // MCP's link to a resource is a reference too. RFC-005 §8.1 maps it + // onto the reference record field for field, its `size` as `sizeBytes`, + // with no disposition: the host's default policy, as for any reference + // that claims none. Its title, description and annotations aren't + // testimony. + // A link's size is MCP's `size`; the script lane's object for a link + // names it `sizeBytes`, so given again it reads the same, where + // sanitizeLabel left its uri as it was. Read on every lane, so a + // peer's link that names a `sizeBytes` is labelled with it too. + return linkTestimony(parseReferenceFields({ uri: block.uri, name: block.name, mimeType: block.mimeType, sizeBytes: block.size ?? block.sizeBytes })); default: return { kind: 'invalid', shown: unknownBlockText(block) }; } } +/** A link's classification, its testimony marked as a link's (fromLink). */ +function linkTestimony(c: BlockClassification): BlockClassification { + return c.kind === 'reference' ? { kind: 'reference', testimony: { ...c.testimony, fromLink: true } } : c; +} + function parseReferenceFields(block: Record): BlockClassification { if (typeof block.uri !== 'string' || block.uri.length === 0) { return { kind: 'invalid', shown: REFERENCE_NO_LINK_TEXT }; @@ -222,22 +237,349 @@ export const EXPIRY_REASON = { * only through a server that delivered the reference. */ export const UNBOUND_REASON = "not tied to any server, so there's nowhere to fetch it from"; -/** The fetcher's refusal for a server it has no address to dial, one the - * host doesn't connect to over a network, such as a stdio server, as a - * clause that completes "couldn't fetch : " and says so of the server; - * the fetcher dials only through a server's address. save_recent_image gives - * it too. A stub can't know it, since only the framework holds its servers' - * configs. */ +/** The refusal for a server the host has no address to dial, one it doesn't + * connect to over a network, such as a stdio server, as a clause that + * completes "couldn't fetch : " and says so of the server: the fetcher + * dials only through a server's address. A reference's stub, fetch_reference + * and save_recent_image all give it (dialCheck). */ export function noAddressReason(serverId: string): string { return `this host doesn't connect to the "${serverId}" server over a network, so there's nowhere to fetch it from`; } -/** Strip control characters (C0, DEL, C1) and every bidi control (marks, - * embeddings, overrides, isolates, and the Arabic letter mark), trim the - * whitespace (a BOM included) left at either end, and bound the length. - * Labels are labels — storage naming is host-generated elsewhere (§7.3). */ +/** A server's id as the agent reads it in a stub or a refusal: as a label + * is, bounded and with its controls removed. */ +export function shownServerId(serverId: string): string { + return sanitizeLabel(serverId, STUB_FIELD_CHARS); +} + +/** + * Query and fragment parameters that carry a credential, by name, compared + * by their letters and digits alone, without letter case, so `access_token`, + * `accessToken`, `access-token` and `access_token[]` are one name. RFC-005 §5: "Signed URLs and query capabilities are + * bearer credentials that look like locations", so a uri is shown only where + * the host's policy classifies it as non-capability-bearing, which here is + * one with no userinfo and none of these parameters, nor any uri in a + * parameter's value that has them (uriMayCarryCredential). + * For example: + * - AWS's signed URLs: `https://b.s3.amazonaws.com/k?X-Amz-Algorithm=…&X-Amz-Credential=…&X-Amz-Security-Token=…&X-Amz-Signature=…` + * - Google Cloud Storage's: `https://storage.googleapis.com/b/k?X-Goog-Algorithm=…&X-Goog-Credential=…&X-Goog-Signature=…` + * - Azure's shared access signatures: `https://a.blob.core.windows.net/c/b?sv=…&se=…&sp=r&sig=…` + * - CloudFront's: `https://d1.cloudfront.net/k?Expires=…&Signature=…&Key-Pair-Id=…`, or `?Policy=…&Signature=…&Key-Pair-Id=…` + * - Discord's CDN: `https://cdn.discordapp.com/attachments/1/2/f.png?ex=…&is=…&hm=…` + * - Backblaze B2's: `https://f000.backblazeb2.com/file/b/k?Authorization=…` + * - OAuth's: `#access_token=…&token_type=bearer`, `#id_token=…`, `?refresh_token=…`, `?client_secret=…`, + * and OAuth 1.0a's `?oauth_token=…&oauth_signature=…` + * - the generic: `?token=…`, `?auth=…`, `?auth_token=…`, `?key=…`, `?apikey=…`, `?api_key=…`, `?api_token=…`, + * `?x-api-key=…`, `?private_token=…`, `?secret=…`, `?password=…`, `?signature=…` + * - a session's id, a bearer credential too: `?session_id=…`, `?jsessionid=…` (in a path, as + * `;jsessionid=…` usually rides, it isn't read) + * A credential named otherwise, or carried in a path, such as a webhook's + * `https://discord.com/api/webhooks//`, isn't caught. + */ +const CREDENTIAL_PARAMETERS: ReadonlySet = new Set([ + 'x-amz-signature', 'x-amz-credential', 'x-amz-security-token', + 'x-goog-signature', 'x-goog-credential', + 'sig', + 'signature', 'key-pair-id', 'policy', + 'hm', + 'authorization', + 'access_token', 'id_token', 'refresh_token', 'client_secret', + 'oauth_token', 'oauth_signature', + 'token', 'auth', 'auth_token', 'key', 'api_key', 'api_token', 'x-api-key', 'private_token', 'secret', 'password', + 'session_id', 'jsessionid', +].map(credentialName)); + +/** A parameter's name as CREDENTIAL_PARAMETERS compares it: its letters and + * digits alone, in lower case. */ +function credentialName(name: string): string { + return name.toLowerCase().replace(/[^a-z0-9]/g, ''); +} + +/** + * Whether a uri may carry a credential: userinfo, or a query or fragment + * parameter named on CREDENTIAL_PARAMETERS with a value, a fragment read as + * `name=value&…` as a query is. A name with no value carries nothing, as an + * anchor (`#token`) or `?token=` doesn't. A `?` or `;` inside either may + * separate parameters too, as in a route kept in a fragment + * (`#/reset?token=…`), so the parameters are read both with and without them + * as separators, and one with a value in either reading counts. A + * parameter's value that is itself a uri is read the same way + * (nestedMayCarry): an absolute one, as an image proxy's `?url=` is once decoded; a protocol-relative one, or + * a relative one with a query or fragment, as a login's + * `?next=%2Freset%3Ftoken%3D…` is; and one encoded again inside the value, + * as a redirector that encodes an already encoded url leaves it, each read + * and each decode a level of its own. That goes up to CREDENTIAL_NESTING + * levels deep, and a uri reached deeper may carry one, since a bound that + * let the next level through would be the same gap one level down. A uri + * carried in another encoding, such as base64, isn't read. A uri that + * doesn't parse, even as a relative reference, may, since its parts can't be + * read. + */ +export function uriMayCarryCredential(uri: string): boolean { + let u: URL; + try { + // Absolute first: against an http base, `http:user:pass@host` would read + // as a relative path and hide its userinfo. + u = new URL(uri); + } catch { + try { + u = new URL(uri, RELATIVE_BASE); + } catch { + return true; + } + } + return urlMayCarryCredential(u, 0); +} + +/** The base a relative uri is read against: it supplies only a scheme and a + * path, and a credential lives in the uri's own authority, query or + * fragment. */ +const RELATIVE_BASE = 'http://relative.invalid/'; + +/** How deep uriMayCarryCredential reads uris nested in parameters' values, + * each one a parameter's value in the uri before it. */ +const CREDENTIAL_NESTING = 4; + +function urlMayCarryCredential(u: URL, depth: number): boolean { + if (u.username !== '' || u.password !== '') return true; + // Read both ways: with '&' alone separating parameters, as a server + // reading the query as WHATWG does sees them, and with '?' and ';' too, so + // a value that opens with one (`token=;…`) still counts in the first. + const named = (params: string) => [params, params.replace(/[?;]/g, '&')].some((read) => + [...new URLSearchParams(read).entries()].some(([name, value]) => value !== '' + && (CREDENTIAL_PARAMETERS.has(credentialName(name)) || nestedMayCarry(value, depth)))); + return named(u.search) || named(u.hash.slice(1)); +} + +/** Whether a parameter's value is a uri that may carry a credential, read in + * this order: as an absolute uri; else, where its form is a relative + * reference's (it starts with `/` or `\`, or holds `?` or `#`), resolved + * against RELATIVE_BASE, never against the uri it sits in, which may have + * no path to resolve against, as a `mailto:` hasn't; else, where decoding + * it once more changes it, decoded and read again from the start. Each read + * or decode is a level, and a decode is lenient, as the parameter's own + * was, so a stray `%` doesn't stop it. A value that is none of these, such + * as a word, answers false; within the bound a decode alone never + * withholds, and past it a value that would still decode does. */ +function nestedMayCarry(value: string, depth: number): boolean { + let inner: URL | null = null; + try { + inner = new URL(value); + } catch { + // Its form as the URL parser reads it, which first trims C0 controls and + // spaces, so `\t//user:pass@…` is the protocol-relative uri it resolves + // as. The tabs and newlines it also drops can't change this form once the + // lead is trimmed: none is a `/`, `\\`, `?` or `#`. + if (/^[\u0000-\u0020]*[/\\]|[?#]/.test(value)) { + try { + inner = new URL(value, RELATIVE_BASE); + } catch { + // Against a hierarchical base only a value that is protocol-relative + // as the parser reads it (`//`, or `\\` as a special scheme reads it) + // fails to resolve, on its authority, and it may carry one, as a uri + // that doesn't parse may: its authority can't be read. So an + // ordinary value such as `q=// note` is withheld too. + return true; + } + } + } + if (inner !== null) return depth >= CREDENTIAL_NESTING || urlMayCarryCredential(inner, depth + 1); + // Decoded as the parameter itself was, leniently: a stray `%` stays as it + // is rather than stopping the reading, as it doesn't stop a server's. + // `+` and `&` are escaped first, so they come back as themselves. + const decoded = new URLSearchParams(`v=${value.replace(/[+&]/g, encodeURIComponent)}`).get('v') ?? value; + if (decoded === value) return false; + // Past the bound, a value that would still decode is withheld: the check's + // work stops where its reading does, and a uri found deeper would be. + return depth >= CREDENTIAL_NESTING || nestedMayCarry(decoded, depth + 1); +} + +/** How the host reaches a server, as far as fetching its references goes: + * the WebSocket url it dials an MCPL server at, or none: for a server it + * spawns, such as a stdio one, or a modern MCP server over HTTP, whose + * results carry no references (RFC-005 governs MCPL peers' alone), or one + * whose url the transport won't dial. A framework records each of its + * servers' in the registry (ReferenceRegistry.setServerAddress), where a + * stub reads it; its fetcher reads the same from the framework's own + * configs, beside each server's credential. */ +export type ServerAddress = { url: string } | 'no-address'; + +/** Why a fetch of a reference failed, or would be refused before it dials. */ +export interface Refusal { + /** What happened, as a clause that completes "couldn't fetch : ", + * whatever it's about: fetch_reference's sentence, save_recent_image's, + * and the eager and script-demand fetches' log lines read it. */ + error: string; + /** What calling again can do, as a sentence. fetch_reference and + * save_recent_image show it; the eager and script-demand fetches log the + * error alone, and a reference's stub never shows it. */ + remedy: string; + /** How the reference's stub words it, where the clause names parts of the + * uri the stub never shows. A stub shows this or the clause. */ + inStub?: string; +} + +// What calling again can do, for the refusals a reference's stub shares with +// a fetch; the fetcher has these and its own. None names a tool: a fetch +// doesn't know where its reference came from. +export const NOTHING_TO_DO = "Calling again won't change that."; +/** The registry keys a record by its uri, so a fresh reference to the same + * uri arrives as this very id and refreshes its record. */ +export const FRESH_REFERENCE = "Once a fresh reference to it arrives, fetching may work; until then, fetching it again won't."; +/** A server's address is read from the host's configs as they are at each + * fetch. */ +export const SERVER_BACK = "Calling again won't change that unless whoever runs this host adds that server back."; +const SERVER_ADDRESS = "Calling again won't change that unless whoever runs this host corrects the server's address."; + +/** ws→http / wss→https with default ports elided, for origin comparison. */ +export function normalizedOrigin(url: string): string | null { + try { + const u = new URL(url); + const scheme = u.protocol === 'ws:' ? 'http:' : u.protocol === 'wss:' ? 'https:' : u.protocol; + const defaultPort = scheme === 'https:' ? '443' : '80'; + const port = u.port === '' || u.port === defaultPort ? '' : `:${u.port}`; + return `${scheme}//${u.hostname.toLowerCase()}${port}`; + } catch { + return null; + } +} + +/** A server's address as it's configured, its scheme and host, for a + * clause: never its userinfo, path or query. URL's own origin is 'null' for + * a scheme that isn't special, which normalizedOrigin accepts. The caller + * has already parsed it. */ +function configuredOrigin(url: string): string { + const u = new URL(url); + return `${u.protocol}//${u.host}`; +} + +/** The §6.1/§7.1 policy check of a uri against the url a server is dialed + * at: null where a fetch through that server may fetch it. */ +function originRefusal(uri: string, serverUrl: string, serverId: string): Refusal | null { + const refOrigin = normalizedOrigin(uri); + if (!refOrigin) return { error: "its address doesn't parse as a URL", remedy: NOTHING_TO_DO }; + // A server's address that doesn't parse leaves nothing to dial. The + // framework's resolver gives only an address its transport dials, so this + // guards other resolvers. + const dialedOrigin = normalizedOrigin(serverUrl); + if (!dialedOrigin) { + return { + error: `the "${shownServerId(serverId)}" server's address doesn't parse as a URL, so there's nowhere to fetch it from`, + remedy: SERVER_ADDRESS, + }; + } + const refUrl = new URL(uri); + // A user name or password in the uri is a credential embedded in it, + // which RFC-005 §6.1 forbids a server to mint and lets a host refuse + // (vector 6). This host neither sends nor repeats it: fetch refuses such + // a url with a message that holds it whole. + if (refUrl.username !== '' || refUrl.password !== '') { + return { error: "its address carries a user name or password, which this host doesn't send", remedy: NOTHING_TO_DO }; + } + // The uri's own scheme, not its origin's: an origin reads ws: as http: + // and wss: as https:, and fetch refuses a ws: or wss: uri ('unknown + // scheme') before connecting, which would read as a failed request a + // later try can mend. + const scheme = refUrl.protocol; + if (scheme !== 'https:' && scheme !== 'http:') { + return { + error: `its address uses ${sanitizeLabel(scheme, STUB_FIELD_CHARS)}, and this host fetches only http: and https: addresses`, + remedy: NOTHING_TO_DO, + inStub: 'scheme not allowed: this host fetches only http and https uris', + }; + } + // Origins compare with their schemes, so this also refuses an http: + // reference from a server dialed over TLS (wss://). + if (refOrigin !== dialedOrigin) { + return { + error: `it points to ${refOrigin}, not to the "${shownServerId(serverId)}" server's own address (${configuredOrigin(serverUrl)}), ` + + 'and this host fetches only from the server itself', + remedy: NOTHING_TO_DO, + inStub: "third-party origin: it isn't on its server's own address, so this host doesn't fetch it", + }; + } + return null; +} + +/** + * The server a fetch of a reference would dial: its servers, in the order + * they delivered it, the first whose address is on the uri's origin (§6.2). + * When none is, the first server's refusal (unknown to this host, with no + * address, or the origin check's), and for a record no server delivered, + * that it's unbound, which a later delivery of the uri by a server mends. + */ +export function dialServer( + record: ReferenceRecord, + addressOf: (serverId: string) => ServerAddress | undefined, +): { serverId: string } | { refusal: Refusal } { + let refusal: Refusal | undefined; + let dialed: string | undefined; + for (const serverId of record.serverIds) { + const address = addressOf(serverId); + if (address === undefined) { + refusal ??= { error: `this host no longer has the "${shownServerId(serverId)}" server it came from`, remedy: SERVER_BACK }; + } else if (address === 'no-address') { + refusal ??= { error: noAddressReason(shownServerId(serverId)), remedy: NOTHING_TO_DO }; + } else { + const offOrigin = originRefusal(record.testimony.uri, address.url, serverId); + if (offOrigin === null) { + dialed = serverId; + break; + } + refusal ??= offOrigin; + } + } + return dialed === undefined + ? { refusal: refusal ?? { error: `it's ${UNBOUND_REASON}`, remedy: FRESH_REFERENCE } } + : { serverId: dialed }; +} + +/** + * Where a fetch of a reference would dial, or why it would be refused before + * dialing, whatever the fetch: dialServer's server, then the reference's + * expiry. A pure function of the record and the servers' addresses: a + * reference's stub asks it of the registry's (unfetchableReason), and the + * fetcher's pre-dial check of its own framework's configs, whose credential + * goes with the address it checks. In a process with one running framework, + * or with running frameworks that agree on each id they share, the two + * sources hold the same addresses, so a stub promises the server the fetch + * dials. A stub reads an expiry with its result's claims, and the fetch with + * the record's latest testimony, so where one result names a uri both + * expired and fresh, its stub says expired while a fetch can succeed. A + * stopped framework's addresses are out of the registry, so its stubs read + * another running framework's entry for the same id where there is one, and + * otherwise say the server is unknown, while its configs still resolve it. + * Where two + * frameworks configure one id differently, only a stub's words can be wrong, + * never where a token goes. A fetch's own limits, its budget and ceiling, + * come after it. + */ +export function dialCheck( + record: ReferenceRecord, + addressOf: (serverId: string) => ServerAddress | undefined, + nowMs = Date.now(), + /** For a stub or a script's object, the claims of the result it's shown + * in, which decide where its references name the record's uri, the + * record's own testimony deciding otherwise (claimedExpiry). A fetch + * reads the record alone. */ + claims?: ResultClaims, +): { serverId: string } | { refusal: Refusal } { + const server = dialServer(record, addressOf); + if ('refusal' in server) return server; + const expiry = claimedExpiry(record, claims, nowMs); + return expiry ? { refusal: { error: `the reference is ${EXPIRY_REASON[expiry]}`, remedy: FRESH_REFERENCE } } : server; +} + +/** Strip control characters (C0, DEL, C1), every bidi control (marks, + * embeddings, overrides, isolates, and the Arabic letter mark) and the line + * and paragraph separators (U+2028, U+2029), which a regex reads as line + * ends, so a label can't start a line of its own, such as one the image + * ledger would read as a slot. Then trim the whitespace (a BOM included) + * left at either end, and bound the length. Labels are labels — storage + * naming is host-generated elsewhere (§7.3). */ export function sanitizeLabel(s: string, maxChars: number): string { - const cleaned = s.replace(/[\p{Cc}\p{Bidi_Control}]/gu, '').trim(); + const cleaned = s.replace(/[\p{Cc}\p{Bidi_Control}\u2028\u2029]/gu, '').trim(); if (cleaned.length <= maxChars) return cleaned; let cut = cleaned.slice(0, Math.max(0, maxChars - 1)); // never split a surrogate pair at the truncation boundary @@ -290,7 +632,7 @@ const MAX_RECORDS = 2000; export class ReferenceRegistry { private records = new Map(); /** uri → refId. The uri ALONE is the identity key: a tool result's stubs - * (tool-result-history.ts's, and framework.ts's tryNativeToolResultContent) + * (tool-result-history.ts's, which both copies of a tool result take) * and a script's demand fetch register with no server context, * while the registrations that know the server carry its id: a tool * result's, before it's shown, whether the agent's dispatch or an @@ -307,8 +649,51 @@ export class ReferenceRegistry { * its own origin, for a uri it delivered. */ private byKey = new Map(); private counter = 0; + /** Each open journal's copies (openJournal), by uri. */ + private journals = new Set>(); + + /** + * Keep, until `close`, each record as it stood when this opened: a copy + * taken the first time a registration names its uri, whatever it changes, + * or evicts its record, or null where the uri had none then. `before(uri)` + * gives that copy, or, where no registration has named the uri since, the + * record as it is, which is the same. A fetch that saves a copy while it's + * open sets its record's saved path without registering: a record no + * registration has named since is read with it, and one a registration + * named first is read as it stood then, saved or not. For a record a fetch + * saved meanwhile, both readings are of one the host can fetch, since a + * fetch saves only what passed the checks a link's line reads too, except + * where two frameworks configure one server id differently, which a stub's + * words can't tell apart either (setServerAddress). Each + * open has its own copies, so lanes that overlap keep theirs apart, and + * closing one lets its copies go. + */ + openJournal(): RegistryJournal { + const copies = new Map(); + this.journals.add(copies); + return { + before: (uri) => (copies.has(uri) ? copies.get(uri) ?? undefined : this.findByUri(uri)), + close: () => { this.journals.delete(copies); }, + }; + } + + /** Copy a uri's record into each open journal that has none for it yet, + * a copy of its own for each. */ + private keepFirst(uri: string): void { + if (this.journals.size === 0) return; + const record = this.findByUri(uri); + for (const copies of this.journals) { + if (!copies.has(uri)) copies.set(uri, record ? { ...record, serverIds: [...record.serverIds] } : null); + } + } + + /** How many journals are open: none once every lane has closed its own. */ + get journalCount(): number { + return this.journals.size; + } register(testimony: ReferenceTestimony, serverId?: string): ReferenceRecord { + this.keepFirst(testimony.uri); const existingId = this.byKey.get(testimony.uri); if (existingId !== undefined) { const existing = this.records.get(existingId); @@ -319,8 +704,9 @@ export class ReferenceRegistry { // Refresh testimony while unfetched: a re-issued reference with a new // digest/expiry supersedes the stale claims. Once fetched, the record // describes verified bytes on disk and keeps the testimony they were - // verified against. - if (!existing.fetchedPath) existing.testimony = testimony; + // verified against. A link claims no digest, expiry or disposition, + // so its testimony never replaces a record's claims. + if (!existing.fetchedPath && !testimony.fromLink) existing.testimony = testimony; // Newest again, so a uri just mentioned is the last to be evicted. this.records.delete(existingId); this.records.set(existingId, existing); @@ -335,6 +721,7 @@ export class ReferenceRegistry { const oldest = this.records.keys().next().value; if (oldest !== undefined) { const old = this.records.get(oldest); + if (old) this.keepFirst(old.testimony.uri); this.records.delete(oldest); if (old) this.byKey.delete(old.testimony.uri); } @@ -346,6 +733,38 @@ export class ReferenceRegistry { return this.records.get(refId); } + /** How the host reaches each server, by id, as each framework in the process + * configured it, which a reference's stub reads (unfetchableReason). A + * fetch reads its own framework's configs instead, so a server's token only + * ever goes to the origin its own config names. Keyed by id as a record's + * servers are: where two frameworks configure one id, the latest entry is + * in force, and a stub's words can be wrong for the other one's fetch. */ + private addresses = new Map>(); + + /** Record how the host reaches a server, as a framework (`owner`) does for + * each server it configures. Its entry for the id is then in force. */ + setServerAddress(serverId: string, address: ServerAddress, owner: object): void { + const entries = (this.addresses.get(serverId) ?? []).filter((e) => e.owner !== owner); + entries.push({ owner, address }); + this.addresses.set(serverId, entries); + } + + /** Take back a framework's entry for a server, as it does for a server it + * removes, and for each of its servers when it stops. Another framework's + * entry for the id, if any, is in force again; with none, a stub reads the + * id's references as an unknown server's. After a removal, that framework's + * fetch refuses them too. A stopped framework keeps its configs, so its + * fetcher, called before it starts again, can still dial them. */ + forgetServerAddress(serverId: string, owner: object): void { + const entries = (this.addresses.get(serverId) ?? []).filter((e) => e.owner !== owner); + if (entries.length > 0) this.addresses.set(serverId, entries); + else this.addresses.delete(serverId); + } + + serverAddress(serverId: string): ServerAddress | undefined { + return this.addresses.get(serverId)?.at(-1)?.address; + } + /** Find the record for a uri. */ findByUri(uri: string): ReferenceRecord | undefined { const id = this.byKey.get(uri); @@ -355,6 +774,23 @@ export class ReferenceRegistry { export const referenceRegistry = new ReferenceRegistry(); +/** A registry's records as they stood when it opened (openJournal). */ +export interface RegistryJournal { + before(uri: string): ReferenceRecord | undefined; + close(): void; +} + +/** + * The model's line for a result as a script's link object reads it: the + * claims that line reads, and each record as it stood when the script lane + * began to treat the result (a journal's `before`), whatever the treatment + * has registered, rebound or evicted since, which the line never sees. + */ +export interface ScriptLine { + claims: ResultClaims; + recordBefore(uri: string): ReferenceRecord | undefined; +} + // ============================================================================ // Stub building (§5) // ============================================================================ @@ -366,13 +802,15 @@ export function referenceTypeLabel(mimeType: string | undefined): string { } /** - * The longest line buildReferenceStub writes, in UTF-16 code units, from its + * The longest stub buildReferenceStub writes, in UTF-16 code units, from its * own bounds: the longest id the registry mints, then the name, the type with * the largest size it shows, and the origin, each at its bound, and the * longest ending, a saved copy's path, with their separators. The other * endings, the fetch hint and the reasons a fetch would be refused, are * shorter. A line shaped like a stub but longer than this is quoted text the - * writer can't have written, and no image slot (tool-image-ledger.ts). + * writer can't have written, and no image slot (tool-image-ledger.ts). A + * tool result's link line (StubOptions.unfetchedLinkUri) isn't shaped like a + * stub, so it's no slot whatever its length. */ export const REFERENCE_STUB_MAX_CHARS = `[ref_${MAX_SAFE.toString(36)}_] `.length + REF_ID_SUFFIX_CHARS @@ -381,15 +819,206 @@ export const REFERENCE_STUB_MAX_CHARS = + ' — '.length + STUB_FIELD_CHARS + ' — '.length + 'saved: '.length + SAVED_PATH_CHARS; +/** How a lane shows a reference beyond its stub. */ +export interface StubOptions { + /** + * Show a link (MCP's resource_link) the host can't fetch as its name, uri + * and type, unless its uri may carry a credential (uriMayCarryCredential). + * For a tool result's links, whose JSON showed every link's uri before + * this host read links as references: there the uri is the agent's only + * handle on a link the host can't fetch, and showing it is §5's MAY for a + * reference the host's policy classifies as non-capability-bearing. No + * other lane showed a link's uri, so none shows one now. + */ + unfetchedLinkUri?: boolean; + /** The claims of the result this block is shown in (resultClaims), which + * decide what the stub says where its references name the record's uri, + * the record's own testimony deciding otherwise. */ + claims?: ResultClaims; +} + +/** + * What one tool result's references claim of each uri they name, each uri's + * testimonies read as one (UriClaims): the strictest claim wins, so a uri any + * of them marks `disposition: never` is withheld, and one any of them gives + * an expiry that counts is expired (claimedExpiry), whatever the order of the + * blocks and whatever the registry evicted while it was read (Ada-1017's + * call). A decision about a record reads its result's claims where the + * result's references name its uri, a plain one included, and otherwise the + * record's own testimony, from an earlier result. A reference naming the uri + * is that result's word on it, the latest, as the registry keeps it + * (register() replaces a record's testimony with it until the record is + * fetched), so an earlier result's `never` or expiry gives way to it + * (Ada-1017's call). A link claims no disposition or expiry, so it adds + * nothing here, and a result that names a uri only in links leaves its + * record's testimony in force. + */ +export type ResultClaims = ReadonlyMap; + +/** + * A uri as a link shows it: cleaned as sanitizeLabel cleans a label (controls + * and bidi marks dropped, surrounding space trimmed; a uri past + * REFERENCE_LIMITS.uri is no reference, so none is shortened), and well + * formed, as every copy is before it's shown, a lone surrogate showing as + * U+FFFD. A `never` is about what a link shows, so it's found by this text + * (neverShown): a claim of `never` on a uri withholds every link that shows + * the same text, two uris that clean alike sharing it. Every other claim is about the record, + * which the fetch reads by its raw uri, so it's kept by the raw uri (Ada-1017, + * room-203 #202239). + */ +export function shownUri(uri: string): string { + return sanitizeLabel(uri, REFERENCE_LIMITS.uri).toWellFormed(); +} + +const NEVER_SHOWN = new WeakMap>(); + +/** The texts a link shows (shownUri) for the uris a result's claims mark + * `never`, read once per claims. A ResultClaims isn't changed once made. */ +function neverShown(claims: ResultClaims): ReadonlySet { + let shown = NEVER_SHOWN.get(claims); + if (!shown) { + shown = new Set([...claims].filter(([, c]) => c.never).map(([uri]) => shownUri(uri))); + NEVER_SHOWN.set(claims, shown); + } + return shown; +} + +/** What a result's references say of one uri, read as one. */ +export interface UriClaims { + /** Whether any of them marks it `disposition: never`. */ + never: boolean; + /** Whether any of them gives an expiry rejected as a field. */ + rejected: boolean; + /** Whether any of them gives an expiry that doesn't parse. */ + unparseable: boolean; + /** The earliest expiry any of them gives, in ms since the epoch, or null. */ + earliestMs: number | null; +} + +/** A result's claims (ResultClaims), from the testimonies of its references, + * each read once, as referenceExpiry reads an expiry. */ +export function resultClaims(testimonies: Iterable, nevers: Iterable = []): ResultClaims { + const claims = new Map(); + // A uri inline data marks `disposition: never` beside its data, the + // classifier's contradiction (inlineNeverUri): withheld unconditionally + // (RFC-005 §5), though the block is no reference. Only its never counts, + // since its other claims could loosen (Ada-1017, room-203 #201013). + const entry = (uri: string): UriClaims => { + let c = claims.get(uri); + if (!c) claims.set(uri, c = { never: false, rejected: false, unparseable: false, earliestMs: null }); + return c; + }; + for (const uri of nevers) entry(uri).never = true; + for (const t of testimonies) { + if (t.fromLink) continue; + const c = entry(t.uri); + if (t.disposition === 'never') c.never = true; + if (t.rejectedFields.includes('expiresAt')) { + c.rejected = true; + } else if (t.expiresAt !== undefined) { + const ms = Date.parse(t.expiresAt); + if (Number.isNaN(ms)) c.unparseable = true; + else if (c.earliestMs === null || ms < c.earliestMs) c.earliestMs = ms; + } + } + return claims; +} + +/** The claims of a content array's references, read from its elements as + * the array shows them: an element's own fields, such as an image's + * `_meta`, aren't shown, so a reference inside them doesn't count. */ +export function contentClaims(blocks: readonly unknown[]): ResultClaims { + const testimonies: ReferenceTestimony[] = []; + const nevers: string[] = []; + for (const block of blocks) { + const c = classifyBlock(block); + if (c.kind === 'reference') testimonies.push(c.testimony); + const never = inlineNeverUri(block, c); + if (never !== null) nevers.push(never); + } + return resultClaims(testimonies, nevers); +} + +/** The uri inline data marks `disposition: never` beside its data, the + * classifier's contradiction, which counts for that uri's claims + * (resultClaims); null for any other block. A uri that isn't a non-empty + * string names nothing a claim is kept for. */ +export function inlineNeverUri(block: unknown, c: BlockClassification = classifyBlock(block)): string | null { + if (c.kind !== 'inline' || !c.contradiction) return null; + const b = block as { uri?: unknown; disposition?: unknown }; + return b.disposition === 'never' && typeof b.uri === 'string' && b.uri !== '' ? b.uri : null; +} + +/** The claims a decision about a record reads: its result's for its uri, + * where the result's references name it, and otherwise the record's own + * testimony. */ +function uriClaims(record: ReferenceRecord, claims?: ResultClaims): UriClaims | undefined { + return claims?.get(record.testimony.uri) ?? resultClaims([record.testimony]).get(record.testimony.uri); +} + +/** Whether a record's uri is withheld: marked `disposition: never` by its + * result's references, for any uri a link shows as the same text + * (shownUri), or by its own testimony where none of them names it. */ +export function claimedNever(record: ReferenceRecord, claims?: ResultClaims): boolean { + if (claims && neverShown(claims).has(shownUri(record.testimony.uri))) return true; + return uriClaims(record, claims)?.never === true; +} + +/** The expiry a record counts, as of `nowMs`: the first of EXPIRY_REASON's + * cases its claims give (uriClaims), so the order of the blocks doesn't + * choose the words. Null when none does. */ +export function claimedExpiry(record: ReferenceRecord, claims: ResultClaims | undefined, nowMs: number): keyof typeof EXPIRY_REASON | null { + const c = uriClaims(record, claims); + if (!c) return null; + const given: Record = { + passed: c.earliestMs !== null && c.earliestMs <= nowMs, + unparseable: c.unparseable, + rejected: c.rejected, + }; + return (Object.keys(EXPIRY_REASON) as Array).find((reason) => given[reason]) ?? null; +} + +/** + * The uri a tool result's link shows, or null where it shows none: a link + * (fromLink) the host can't fetch, whose uri can't carry a credential, and + * that no reference withholds with `disposition: never`, keeps its uri, the + * agent's only handle on it. A `data:` uri holds its content rather than + * naming where it lies, so it's never shown: a link's line isn't a way for + * inline data to reach context. A tool result's link line (buildReferenceStub, + * unfetchedLinkUri) and a script's object for the link + * (referenceBlockForScript) both ask this, so the model and a script, whose + * output comes back to the model, see a link's uri alike. + * + * The credential check reads the uri as it would be shown, with what + * sanitizeLabel strips gone, so a uri split by a control character is read + * as it would read: `ht\u0000tps://user:pass@…` parses as a path, and shows + * as userinfo. A `disposition: "never"` for the uri keeps it out whatever + * the link says: any reference's in the same result (claims), for this uri + * or one shown as the same text, or, where no reference in the result names + * the uri, the record's. + */ +export function shownLinkUri(t: ReferenceTestimony, record: ReferenceRecord, nowMs = Date.now(), claims?: ResultClaims): string | null { + if (!t.fromLink || claimedNever(record, claims)) return null; + if (unfetchableReason(record, nowMs, claims) === null) return null; + const uri = shownUri(t.uri); + return uriMayCarryCredential(uri) || /^data:/i.test(uri) ? null : uri; +} + /** - * Build the model-visible stub for a reference. Output length is bounded by - * host constants and independent of every server-supplied field length. - * Under `disposition:"never"` the uri is omitted unconditionally; under - * `ref`/absent it is omitted too — this host's default policy is the opaque - * id everywhere (the safe default of §5), with retrieval via the record. + * Build the model-visible stub for a reference. Its length is bounded by + * host constants and independent of every server-supplied field's length: + * REFERENCE_STUB_MAX_CHARS. Under `disposition:"never"` the uri is omitted + * unconditionally; under `ref`/absent it is omitted too — this host's default + * policy is the opaque id everywhere (the safe default of §5), with retrieval + * via the record. The one exception is a tool result's link the host can't + * fetch (StubOptions.unfetchedLinkUri), which is no stub but a line of its + * own, its uri bounded by REFERENCE_LIMITS.uri. */ -export function buildReferenceStub(t: ReferenceTestimony, provenance?: string, serverId?: string): string { +export function buildReferenceStub(t: ReferenceTestimony, provenance?: string, serverId?: string, options?: StubOptions): string { const record = referenceRegistry.register(t, serverId); + // One clock reading for the whole stub, so whether it shows a link's uri + // and the reason it gives can't disagree across an expiry. + const nowMs = Date.now(); const parts: string[] = []; // A label is shown only when something is left of it once sanitized. const name = t.name ? sanitizeLabel(t.name, STUB_FIELD_CHARS) : ''; @@ -406,29 +1035,40 @@ export function buildReferenceStub(t: ReferenceTestimony, provenance?: string, s // workspace path (host-generated, §7.3) so existing tools just work. parts.push(`saved: ${sanitizeLabel(record.fetchedPath, SAVED_PATH_CHARS)}`); } else { - // A reference whose record shows the fetcher would refuse it says why, - // rather than inviting a fetch_reference call. - parts.push(unfetchableReason(record) ?? 'fetch with fetch_reference'); + // A reference the fetcher would refuse says why, rather than inviting a + // fetch_reference call. + if (options?.unfetchedLinkUri) { + const uri = shownLinkUri(t, record, nowMs, options.claims); + if (uri !== null) return unfetchedLinkText(t, uri); + } + parts.push(unfetchableReason(record, nowMs, options?.claims) ?? 'fetch with fetch_reference'); } return `[${record.refId}] ${parts.join(' — ') || 'referenced content'}`; } /** - * Why the fetcher would refuse a reference, as far as its record alone shows, - * for its stub: no server registered for it, which the fetcher checks first, - * or its expiry. Null for a saved copy, which the fetcher serves before any - * check, and when the record shows neither. The fetcher's other refusals rest - * on what a record doesn't hold: whether any of its servers is known to the - * host, has an address and is on the uri's origin, the dialed server's fetch - * budget spent, or a claimed size over the ceiling a fetch is given. The - * fetcher gives those in its own words, and save_recent_image, in the - * framework where it can ask the fetcher, says them too. + * What a tool result's link the host can't fetch shows, where its uri can't + * carry a credential: its name, its uri and its type, as a standard MCP + * server's link shows (StubOptions.unfetchedLinkUri). + */ +function unfetchedLinkText(t: ReferenceTestimony, uri: string): string { + const name = t.name ? sanitizeLabel(t.name, STUB_FIELD_CHARS) : ''; + const mimeType = referenceTypeLabel(t.mimeType); + return `[resource link${name ? ` "${name}"` : ''}: ${uri}${mimeType ? `, ${mimeType}` : ''}; not fetched]`; +} + +/** + * Why the fetcher would refuse a reference before it dials, whatever the + * fetch, in its stub's words: the registry's dialCheck, from its record and + * its servers' addresses. Null for a saved copy, which the fetcher serves + * before any check, and where a fetch would dial. A fetch's own limits, the + * server's spent budget or a claimed size over the ceiling it's given, the + * fetcher gives in its own words, and save_recent_image says them too. */ -export function unfetchableReason(record: ReferenceRecord, nowMs = Date.now()): string | null { +export function unfetchableReason(record: ReferenceRecord, nowMs = Date.now(), claims?: ResultClaims): string | null { if (record.fetchedPath) return null; - if (record.serverIds.length === 0) return UNBOUND_REASON; - const expiry = referenceExpiry(record.testimony, nowMs); - return expiry ? EXPIRY_REASON[expiry] : null; + const dial = dialCheck(record, (serverId) => referenceRegistry.serverAddress(serverId), nowMs, claims); + return 'refusal' in dial ? dial.refusal.inStub ?? dial.refusal.error : null; } /** @@ -439,13 +1079,13 @@ export function unfetchableReason(record: ReferenceRecord, nowMs = Date.now()): * surrogate becomes U+FFFD), as the context store would make it: a tool * result's copies then carry the same text (#695). */ -export function referenceStubOrNull(block: unknown, provenance?: string, serverId?: string): string | null { +export function referenceStubOrNull(block: unknown, provenance?: string, serverId?: string, options?: StubOptions): string | null { const c = classifyBlock(block); switch (c.kind) { case 'text': return null; case 'inline': return c.contradiction ? INLINE_WITHHELD_TEXT : null; - case 'reference': return buildReferenceStub(c.testimony, provenance, serverId).toWellFormed(); + case 'reference': return buildReferenceStub(c.testimony, provenance, serverId, options).toWellFormed(); case 'invalid': return c.shown; } } @@ -455,26 +1095,37 @@ export function referenceStubOrNull(block: unknown, provenance?: string, serverI * result, which carries blocks rather than text. It makes the decision * referenceStubOrNull makes for the model's lanes, since what a script prints * comes back to the model, except that a script reads every field, so inline - * data beside a uri of any type but null or absent is withheld, where the - * model's lanes, which show no uri, withhold it beside a string one. A reference becomes the stub's facts as data, - * without its URI: the host's reference id, what the server said about it in - * this block (its labels stripped and bounded as in the stub), and `savedTo`, - * the workspace path its payload was fetched to, or null when it wasn't - * (`fetch_reference` takes the id as `ref_id`). An unsaved reference that - * counts as expired also carries `expired`, the reason the stub gives in - * place of advising fetch_reference. The reference is registered bound to - * `serverId`, the server whose tool returned it, as the model's lanes bind - * a tool result's references, so the fetch its id offers can be made. With - * no `serverId`, as for a module's result, it's registered as the model's + * data beside a uri of any type but null, absent or empty is withheld, where + * the model's lanes, which show no uri, withhold it beside a string one that + * isn't empty. A reference becomes the stub's facts as data: the host's + * reference id, what the server said about it in this block (its labels + * stripped and bounded as in the stub), and `savedTo`, the workspace path + * its payload was fetched to, or null when it wasn't (`fetch_reference` + * takes the id as `ref_id`). Its uri is left out, except that a link (MCP's + * resource_link) carries `uri` where the model's line for it shows the uri + * (shownLinkUri). An unsaved reference that counts as expired also carries + * `expired`, and one the fetcher would refuse carries `unfetchable`, the + * words a stub gives in place of advising fetch_reference + * (unfetchableReason). Both give why a fetch_reference with the object's + * `refId` would be refused, as the result's references claim it, so a link + * whose line shows its uri, which the host can't fetch, carries them too: its + * line has no id to fetch with. The fetch itself reads the record, which + * keeps the latest testimony registered until it's fetched, and then the one + * its saved copy was verified against (Ada-1017's call). `claims` are those + * of everything the script's view shows (scriptViewClaims), so its other + * references count (ResultClaims). The reference is registered bound to + * `serverId`, the server whose tool returned it, as the model's lanes bind a + * tool result's references, so the fetch its id offers can be made. With no + * `serverId`, as for a module's result, it's registered as the model's * history registers a module's: the record its uri already has, bound where * it arrived (a module's ctx.callTool registers an MCPL peer's references), * or a new one bound to no server. A block the classifier * rejects becomes the notice the model's lanes show, and inline data that - * also claims a bulk disposition, or a uri of any type but null or absent - * (an empty string included), the withheld notice, each as a text block. + * also claims a bulk disposition, or a uri of any type but null, absent or + * empty, the withheld notice, each as a text block. * A text block, or inline data that claims nothing, is returned as sent. */ -export function referenceBlockForScript(block: unknown, serverId?: string): unknown { +export function referenceBlockForScript(block: unknown, serverId?: string, claims?: ResultClaims, line?: ScriptLine): unknown { const c = classifyBlock(block); switch (c.kind) { case 'text': return block; @@ -482,10 +1133,12 @@ export function referenceBlockForScript(block: unknown, serverId?: string): unkn // A script reads every field, as history's JSON writes it, so inline // data beside a uri of any type is withheld here, not only beside a // string one as in the model's lanes, which show no uri: an object can - // carry a credential. An absent or null uri carries none; a string one, - // an empty one included, is already the classifier's contradiction. + // carry a credential. An absent, null or empty uri carries none, as at + // any depth (hasUri), so a peer's and a module's agree (Ada-1017, + // room-203 #200094); one beside a disposition is the classifier's + // contradiction, a reference's claim. const uri = (block as { uri?: unknown }).uri; - return c.contradiction || (uri !== undefined && uri !== null) + return c.contradiction || (uri !== undefined && uri !== null && uri !== '') ? { type: 'text', text: INLINE_WITHHELD_TEXT } : block; } @@ -493,11 +1146,38 @@ export function referenceBlockForScript(block: unknown, serverId?: string): unkn case 'reference': { const t = c.testimony; const record = referenceRegistry.register(t, serverId); - // As the stub judges it: a saved copy is still served. - const expiry = record.fetchedPath ? null : referenceExpiry(record.testimony); + // One clock reading for the whole object, so its words can't disagree + // with each other across an expiry. + const nowMs = Date.now(); + // A link's uri where the model's line for the result shows it, by the + // claims the line reads and its record as the line reads it (`line`, + // where they differ from the view's), and never where the view's + // claims withhold it: reading more claims can loosen as well as + // tighten, since an expiry makes a link one the host can't fetch, whose + // uri the line shows (Ada-1017, room-203 #201013). The line's record is + // the registry's as the model's line leaves it, before the script + // lane's own registrations (#202683, #203011), since the treatment + // registers, rebinds and evicts as it goes, which the line never sees; + // where there was none, it's the one the link's own registration + // makes. A script never sees a uri the model's line + // withholds, through its blocks; structured content goes as sent, as + // on main. + const lineRecord = line + ? line.recordBefore(t.uri) ?? { refId: record.refId, serverIds: serverId === undefined ? [] : [serverId], testimony: t, receivedAt: record.receivedAt } + : record; + const lineShows = shownLinkUri(t, lineRecord, nowMs, line ? line.claims : claims); + const uri = lineShows !== null && line !== undefined && claimedNever(record, claims) ? null : lineShows; + // Why a fetch_reference with this object's refId would be refused, in + // the words a stub gives, whether or not the model's line for it is a + // stub: the line has no id to fetch with, and the object does. As the + // stub judges it, a saved copy is still served, and the result's own + // claims decide where its references name the uri (uriClaims). + const expiry = record.fetchedPath ? null : claimedExpiry(record, claims, nowMs); + const unfetchable = unfetchableReason(record, nowMs, claims); return { type: (block as { type: string }).type, refId: record.refId, + ...(uri !== null ? { uri } : {}), ...(t.name !== undefined ? { name: sanitizeLabel(t.name, STUB_FIELD_CHARS) } : {}), ...(t.mimeType !== undefined ? { mimeType: sanitizeLabel(t.mimeType, STUB_FIELD_CHARS) } : {}), ...(t.sizeBytes !== undefined ? { sizeBytes: t.sizeBytes } : {}), @@ -505,6 +1185,7 @@ export function referenceBlockForScript(block: unknown, serverId?: string): unkn ...(t.expiresAt !== undefined ? { expiresAt: sanitizeLabel(t.expiresAt, STUB_FIELD_CHARS) } : {}), savedTo: record.fetchedPath ?? null, ...(expiry ? { expired: EXPIRY_REASON[expiry] } : {}), + ...(unfetchable !== null ? { unfetchable } : {}), }; } } diff --git a/src/module-registry.ts b/src/module-registry.ts index fe13b7ae0..669a8559b 100644 --- a/src/module-registry.ts +++ b/src/module-registry.ts @@ -977,7 +977,7 @@ function withoutRefusal(call: ToolCall, result: ToolResult): ToolResult { } /** A result's flags, which consumers read for their truth: booleans here. */ -const RESULT_FLAGS = ['success', 'isError', 'endTurn'] as const; +const RESULT_FLAGS = ['success', 'isError', 'endTurn', 'dataIsContent'] as const; /** * A module's tool result as the framework can store, send and trace it. A @@ -1048,7 +1048,7 @@ export function checkedToolResult(call: ToolCall, result: unknown): { result: To endTurn = Boolean(fields.endTurn); read('error'); ownText = errorText(fields.error); - for (const key of ['success', 'isError', 'data']) read(key); + for (const key of ['success', 'isError', 'data', 'dataIsContent']) read(key); for (const key of Object.keys(candidate)) { if (key !== 'toJSON' && !(key in fields)) fields[key] = candidate[key]; } diff --git a/src/tool-image-ledger.ts b/src/tool-image-ledger.ts index c018bdf8b..98cad3951 100644 --- a/src/tool-image-ledger.ts +++ b/src/tool-image-ledger.ts @@ -327,10 +327,10 @@ export interface StoredCut { * Where a tool result's text longer than `cap` is cut: by its stored copy * (cutStoredToolResult), by history's own truncation (truncateForHistory), * and by its live copy, for each text block beside an image - * (tryNativeToolResultContent, framework.ts), so at one cap the stored copy - * and the live copy cut alike. Under a readers' bound the stored copy's cut - * can come earlier (cutStoredToolResultToFit). The text's length when it - * isn't longer than `cap`. + * (toMembraneToolResult's nativeContent, framework.ts), so at one cap the + * stored copy and the live copy cut alike. Under a readers' bound the stored + * copy's cut can come earlier (cutStoredToolResultToFit). The text's length + * when it isn't longer than `cap`. * * The cut is made at the cap, or before it when the cap would split a * surrogate pair (safeSlice's rule), and that is where the moves back are @@ -367,10 +367,11 @@ export function toolResultCutAt(text: string, cap: number): number { * * `shown` is what the live copy showed beside the stored text: null when the * live copy was the text itself, as for every result that carried no inline - * image. Then what the cut keeps is exactly what the resident saw, and - * nothing past it is kept (#403). A result that carried an inline image went - * to the model as native blocks (tryNativeToolResultContent, framework.ts), - * every image whatever the text's length, so its stored text is the only + * image, or whose links and markers came to more than the cap together, so + * goesNative said no. + * Then what the cut keeps is exactly what the resident saw, and nothing past + * it is kept (#403). A result whose live copy went to the model as native + * blocks (framework.ts) sent every image whatever the text's length, so its stored text is the only * place save_recent_image's inventory finds them, and a slot dropped by the * cut would be an image the resident saw that the index slides past (#104). * The image slots past the cut that lie wholly within what the live copy diff --git a/src/tool-result-guard.ts b/src/tool-result-guard.ts index 7990b1767..40584f949 100644 --- a/src/tool-result-guard.ts +++ b/src/tool-result-guard.ts @@ -4,7 +4,7 @@ import type { ContextManager, MessageId, MessageMetadata } from '@animalabs/cont import type { ContentBlock, NormalizedMessage, ToolResult } from '@animalabs/membrane'; import type { CompletedToolCall } from './types/index.js'; import { ensureStateRegistered } from './module-registry.js'; -import { rfc005TextOf, toolResultDataToHistoryString } from './tool-result-history.js'; +import { resultClaimsOf, rfc005TextOf, toolResultDataToHistoryString } from './tool-result-history.js'; import { safeSlice } from './safe-slice.js'; export const TOOL_RESULT_GUARD_NOTICE = 'Tool result withheld by the guard. The tool has already executed.'; @@ -90,9 +90,10 @@ export function withheldResultNotice(spill: WithheldSpill, auditDurable: boolean * inline cap or less (storeResults' keptCap), they're cut, with a note that * says so and where the whole result is, since a later field can then be cut * away entirely. Undefined when it keeps nothing, as for a failed result - * (agent-framework #277). A write's file - * list is evidence of what it did, rarely what drew a refusal, and keeping it - * is what shows a clobbered file when the receipt is withheld. + * (agent-framework #277), or one whose claims can't be read whole + * (resultClaimsOf). A write's file list is evidence of what it did, rarely + * what drew a refusal, and keeping it is what shows a clobbered file when the + * receipt is withheld. */ export function keptWhenWithheld(result: CompletedToolCall['result'], maxChars: number | undefined): string | undefined { const { data, keepWhenWithheld: keys } = result; @@ -105,7 +106,13 @@ export function keptWhenWithheld(result: CompletedToolCall['result'], maxChars: // Data that RFC-005 shows as text, such as a reference's stub, keeps that // text: picked out of it, a kept uri would lose the type that makes it a // reference, and be written out of the stub that hides it. - const whole = rfc005TextOf(fields) ?? toolResultDataToHistoryString(kept); + // The kept fields are shown as the whole result claims (resultClaimsOf), so + // a never elsewhere in it still withholds a kept link's uri. Where a value + // stops that walk, such as a getter that throws, which the audit's JSON + // doesn't read, what the rest claims is unknown, and nothing is kept. + const claims = resultClaimsOf(fields); + if (claims === null) return undefined; + const whole = rfc005TextOf(fields) ?? toolResultDataToHistoryString(kept, undefined, { claims }); if (!maxChars || whole.length <= maxChars) return whole; // Never half a surrogate pair: the stub goes to the provider as text, so // the cut can be one character short of the cap. diff --git a/src/tool-result-history.ts b/src/tool-result-history.ts index 9f91485bc..c231cf426 100644 --- a/src/tool-result-history.ts +++ b/src/tool-result-history.ts @@ -9,30 +9,194 @@ * `maxChars` slice and re-introduces, one turn deferred, exactly the silent * hallucination this feature exists to kill. * - * For MCP-shaped content arrays, this helper keeps text blocks verbatim, - * apart from a lone UTF-16 surrogate, which becomes U+FFFD (#695), and - * replaces image blocks with a short `[image: mimeType, ~NKB]` placeholder. - * A string is already text and is kept as it is. For anything else it falls - * back to JSON, written by RFC-005's rules at every depth (historyJson): a - * reference is shown as its stub wherever it sits in the data, so its uri - * never reaches history, and inline data that claims a uri or a disposition - * is withheld. + * A content array (toolResultBlocks) is shown block by block, each block as + * both copies show it: text verbatim, apart from a lone UTF-16 surrogate, + * which becomes U+FFFD (#695), an image as a short + * `[image: mimeType, ~NKB]` placeholder, a reference as its stub, and any + * other block as a marker saying what it was and why it isn't shown. Anything + * else falls back to JSON, written by RFC-005's rules at every depth + * (historyJson): a reference is shown as its stub wherever it sits in the + * data, so its uri reaches history only as a link's line shows it where the + * host can't fetch the link and its uri can't carry a credential + * (shownLinkUri), and inline data that claims a uri or a disposition is + * withheld. A string is already text and is kept as it is. */ -import { INLINE_WITHHELD_TEXT, isInlineContradiction, referenceBlockForScript, referenceStubOrNull } from './mcpl/references.js'; -import { NON_STRING_TEXT_PLACEHOLDER } from './mcpl/visible-content.js'; +import { + AUDIO_NO_DATA_TEXT, + AUDIO_NO_MEDIA_TYPE_TEXT, + IMAGE_NO_DATA_TEXT, + IMAGE_NO_MEDIA_TYPE_TEXT, + INLINE_WITHHELD_TEXT, + buildReferenceStub, + classifyBlock, + contentClaims, + inlineNeverUri, + referenceBlockForScript, + referenceStubOrNull, + referenceTypeLabel, + resultClaims, + type ReferenceTestimony, + type ResultClaims, + type ScriptLine, +} from './mcpl/references.js'; import { formatToolImagePlaceholder, toolResultCutAt } from './tool-image-ledger.js'; /** Where one block of a content array sits in its history text, and what * kind of block it was: text, an inline image (its placeholder), an RFC-005 - * reference (its stub), or inline data withheld for a contradiction. */ + * reference (its stub), inline data withheld for a contradiction, or a + * marker in place of a block the lane can't show. */ export interface HistoryBlockSpan { - kind: 'text' | 'image' | 'reference' | 'withheld'; + kind: ToolResultBlock['kind']; start: number; end: number; } +/** + * Whether a tool result's live copy goes as native blocks, an image among + * them, rather than as its stored text. It does when the result holds an + * image and its links' blocks and markers come to no more than `maxChars` + * together. Past that, the live copy is the stored text, cut at the cap, so + * no number of links and markers carries a result past it. A reference's + * stub and a withheld line go whole beside an image whatever their length, + * and past its cut the stored copy keeps the newest 64 image slots the live + * copy showed (#403). Text blocks don't count either: the native copy cuts + * each at the cap on its own. Both copies decide by this, from each block's + * kind and the length of its text. + */ +export function goesNative( + blocks: ReadonlyArray<{ kind: ToolResultBlock['kind']; length: number }>, + maxChars: number | undefined, +): boolean { + if (!blocks.some((block) => block.kind === 'image')) return false; + if (!maxChars) return true; + let bounded = 0; + for (const block of blocks) if (block.kind === 'link' || block.kind === 'marker') bounded += block.length; + return bounded <= maxChars; +} + +/** + * One block of a tool result's content array as both of its copies show it + * (toolResultBlocks): a text block's text, an inline image, which the stored + * copy shows as its placeholder and the live copy as itself, or the text that + * stands in its place, the same in both. That text is a reference's stub, a + * link's stub or line (MCP's resource_link, `link`), inline data withheld for + * a contradiction, or a marker saying what kind of block it was and why the + * lane can't show it. + */ +export type ToolResultBlock = + | { kind: 'text'; text: string } + | { kind: 'image'; data: string; mimeType: string } + | { kind: 'reference' | 'link' | 'withheld' | 'marker'; text: string }; + +/** The block types MCP defines for a tool result's content. */ +const MCP_BLOCK_TYPES = new Set(['text', 'image', 'audio', 'resource', 'resource_link']); + +/** + * Each element of `data` as both copies of a tool result show it, or null + * when `data` isn't a content array, so the result is data, written as JSON. + * An array is content when the result says so (`dataIsContent`, which the + * framework sets on an MCPL tool's result: every element of a protocol + * result's content is a block, of a type the host shows or not). Otherwise + * it's content only when every element is a block of one of MCP's own types, + * so a list of records of other types stays data. A module's array of + * records that all carry MCP's type names, such as `[{ type: 'image', url }]`, + * is read as content, and shows a marker for each record that isn't a block + * the lane can show. + */ +export function toolResultBlocks(data: unknown, dataIsContent?: boolean, claims?: ResultClaims): ToolResultBlock[] | null { + if (!isContentArray(data, dataIsContent)) return null; + // The result's claims, read as one (ResultClaims): what any of its + // references says of a uri (a `disposition: never`, an expiry) holds for + // each block that names it, whatever their order. + const read = claims ?? contentClaims(data); + // Each text well formed, as the context store makes it when it writes, so + // the copy sent and the copy stored carry the same text (#695): a stub's + // labels and a link's uri come from the block too. + return data.map((raw) => { + const block = toolResultBlock(raw, read); + return 'text' in block ? { ...block, text: block.text.toWellFormed() } : block; + }); +} + +/** Whether a tool result's data is a content array, which toolResultBlocks + * shows block by block. */ +export function isContentArray(data: unknown, dataIsContent?: boolean): data is unknown[] { + return Array.isArray(data) + && (dataIsContent === true || data.every((element) => isPlainRecord(element) && MCP_BLOCK_TYPES.has(element.type as string))); +} + +/** + * The claims of a tool result's data (ResultClaims), as showing it reads + * them: a content array's from its elements (contentClaims), and any other + * data's by the walk serialization takes (claimsInHistory). For a decision + * made about the result outside showing it, such as which of its references + * to fetch, or for showing part of it as the rest of it claims. Null where a + * value stops the walk, so what the data claims is unknown (claimsInHistory). + */ +export function resultClaimsOf(data: unknown, dataIsContent?: boolean): ResultClaims | null { + return isContentArray(data, dataIsContent) ? contentClaims(data) : claimsInHistory(data); +} + +/** resultClaimsOf, throwing what stops the walk, for a caller that answers + * a stop itself: a script's text route, which fails then, as its view does + * (Ada-1017, room-203 #200374). */ +export function resultClaimsOrThrow(data: unknown, dataIsContent?: boolean): ResultClaims { + return isContentArray(data, dataIsContent) ? contentClaims(data) : claimsInHistoryOrThrow(data); +} + +/** One element of a content array, as toolResultBlocks gives it. Only an + * image with data and a media type is shown as itself, as the push lane's + * converter shows one, and only text and images can be: this host's tool + * results carry nothing else, so inline audio is a marker here. */ +function toolResultBlock(raw: unknown, claims: ResultClaims): ToolResultBlock { + const c = classifyBlock(raw); + switch (c.kind) { + case 'text': + return { kind: 'text', text: (raw as { text: string }).text }; + case 'reference': + // A bounded stub, never the block's JSON, which would show its uri. A + // stub that shows an image type occupies a slot in save_recent_image's + // inventory (a failing one: its bytes are behind fetch_reference), so + // the index never slides past an image the resident saw. A link the + // host can't fetch, shown by its uri, is no slot, as no link was before + // links were references, when the result's JSON showed it. A link's + // block, stub or line, is `link`, counted toward the cap (goesNative). + return { + kind: c.testimony.fromLink ? 'link' : 'reference', + text: buildReferenceStub(c.testimony, 'from tool result', undefined, { unfetchedLinkUri: true, claims }), + }; + case 'invalid': + return { kind: 'marker', text: c.shown }; + case 'inline': { + if (c.contradiction) return { kind: 'withheld', text: INLINE_WITHHELD_TEXT }; + const b = raw as { type: 'image' | 'audio'; data: string; mimeType?: unknown }; + const image = b.type === 'image'; + if (b.data === '') return { kind: 'marker', text: image ? IMAGE_NO_DATA_TEXT : AUDIO_NO_DATA_TEXT }; + if (typeof b.mimeType !== 'string' || b.mimeType === '') { + return { kind: 'marker', text: image ? IMAGE_NO_MEDIA_TYPE_TEXT : AUDIO_NO_MEDIA_TYPE_TEXT }; + } + if (image) return { kind: 'image', data: b.data, mimeType: b.mimeType }; + return { kind: 'marker', text: toolResultAudioText(b.mimeType, b.data) }; + } + } +} + +/** What a tool result's inline audio shows as: its type and size, and that + * this host's tool results carry only text and images. */ +function toolResultAudioText(mimeType: string, data: string): string { + const what = [referenceTypeLabel(mimeType), formatSize(Math.floor(data.length * 3 / 4))].filter(Boolean).join(', '); + return `[Audio not shown: this host's tool results carry only text and images (${what}).]`; +} + export interface HistorySerializeOptions { + /** + * Whether the result's data is a protocol result's content + * (ToolResult.dataIsContent), so an array is shown block by block whatever + * its blocks' types (toolResultBlocks). nonErrorResultText takes it from + * the result instead, whatever this says. + */ + dataIsContent?: boolean; /** * Called for every image block the serializer replaces with a placeholder * (issue #104). Returns the retention ref to embed in the placeholder — @@ -43,10 +207,23 @@ export interface HistorySerializeOptions { /** * Called once a content array has been serialized block by block, rather * than given up on for the fallback, with where each block sits in the - * joined text, before any cut at `maxChars`. The framework reads from it - * what a result's live copy showed (#403). + * joined text, before any cut at `maxChars`, and the blocks themselves as + * this text was made from them. The framework reads from it what a + * result's live copy shows (#403), and builds that copy from these very + * blocks, so the two copies come from one classification. + */ + onBlocks?: (spans: HistoryBlockSpan[], blocks: ToolResultBlock[]) => void; + /** + * The claims to read (ResultClaims) in place of the data's own, where the + * data shown is part of a larger result whose claims these are + * (resultClaimsOf), such as the fields a withheld result keeps. + */ + claims?: ResultClaims; + /** + * Whose result this is, as nonErrorResultText's log line names it where it + * shows UNCHECKED_RESULT_TEXT for a result it couldn't read or show. */ - onBlocks?: (spans: HistoryBlockSpan[]) => void; + toolName?: string; } /** @@ -85,7 +262,7 @@ export function toolResultDataToHistoryString( // RFC-005 shows only objects otherwise (shownInHistory), so a string has // nothing for historyJson to stub or withhold. const str = fromArray - ?? (typeof data === 'string' ? data.toWellFormed() : historyJson(data)) + ?? (typeof data === 'string' ? data.toWellFormed() : historyJson(data, options?.claims)) ?? ''; return maxChars ? truncateForHistory(str, maxChars) : str; } @@ -98,6 +275,11 @@ export function toolResultDataToHistoryString( * * Each field is read once: the direct Agent API passes its caller's own * result here, unchecked, and a getter needn't answer the same way twice. + * Whatever stops reading or showing the data is UNCHECKED_RESULT_TEXT, with + * a log line naming the tool (`options.toolName`) and the error: the model's + * lane has no failure answer that doesn't wedge the round. + * Whether an array is content is the result's own `dataIsContent`, read for + * its truth as a result's flags are, whatever `options` says. * * Each text a tool result's copies carry is made well formed where it's * taken from the result: a lone UTF-16 surrogate becomes U+FFFD, as the @@ -108,7 +290,44 @@ export function toolResultDataToHistoryString( * as before; a cut at a lone high half no longer steps back over it. */ export function nonErrorResultText( - result: { data?: unknown; error?: string }, + result: { data?: unknown; error?: string; dataIsContent?: boolean }, + maxChars?: number, + options?: HistorySerializeOptions, +): string { + // The blocks go to the caller only once the text is made from them, so a + // result that shows the notice has no live copy of its own: the notice is + // both copies' text. + let blocks: [HistoryBlockSpan[], ToolResultBlock[]] | undefined; + const onBlocks = options?.onBlocks; + try { + // The result's own fields are read inside the try too: the direct Agent + // API's result is its caller's own object, whose `data` can be a getter. + const text = nonErrorResultTextOrThrow(result, maxChars, { + ...options, + ...(onBlocks ? { onBlocks: (spans: HistoryBlockSpan[], made: ToolResultBlock[]) => { blocks = [spans, made]; } } : {}), + }); + if (onBlocks && blocks) onBlocks(blocks[0], blocks[1]); + return text; + } catch (error) { + // The model's lane has no failure answer that doesn't wedge the round, so + // whatever stops reading or showing a result, such as a getter that + // throws in a content array's block or a render deeper than the claims + // walk got (a script's routes call the serializer directly, and fail), + // is the notice, with a line naming the tool and the error, since it can + // also be a fault of this host's (Ada-1017, room-203 #201013). + console.error(`[tool-results] ${options?.toolName ? `tool '${options.toolName}'` : 'a tool'}: its result couldn't be read or shown, so the model reads that it was withheld: ${thrownLine(error)}`); + return maxChars ? truncateForHistory(UNCHECKED_RESULT_TEXT, maxChars) : UNCHECKED_RESULT_TEXT; + } +} + +/** + * nonErrorResultText's text, throwing whatever stops reading or showing the + * result, such as a stopped claims walk (historyJson), for a caller that + * answers a stop itself: a script's routes, which fail naming the tool + * (Ada-1017, room-203 #200374), never the model lane's notice. + */ +export function nonErrorResultTextOrThrow( + result: { data?: unknown; error?: string; dataIsContent?: boolean }, maxChars?: number, options?: HistorySerializeOptions, ): string { @@ -120,7 +339,16 @@ export function nonErrorResultText( return maxChars ? truncateForHistory(text, maxChars) : text; } } - return toolResultDataToHistoryString(data, maxChars, options); + return toolResultDataToHistoryString(data, maxChars, { ...options, dataIsContent: Boolean(result.dataIsContent) }); +} + +/** The first line of what was thrown, for a log line. */ +function thrownLine(error: unknown): string { + try { + return (error instanceof Error ? error.message : String(error)).split('\n')[0] || 'no reason given'; + } catch { + return 'a value that can\'t be read'; + } } /** Bounded copy of an arbitrary string with the standard truncation notice, @@ -137,46 +365,20 @@ export function truncateForHistory(str: string, maxChars: number): string { } /** - * If `data` is an MCP content array whose every block has a shape we - * recognize, return a history-safe string (images → placeholders). Otherwise - * return `null` to defer to the fallback, historyJson, which shows a - * reference anywhere in the data as its stub. + * A content array's history text (toolResultBlocks): each block's text, an + * image as its placeholder, joined by newlines. Null when `data` isn't a + * content array, for the fallback, historyJson. */ function tryHistoryStringFromContentArray(data: unknown, options?: HistorySerializeOptions): string | null { - if (!Array.isArray(data)) return null; - const parts: Array<{ kind: HistoryBlockSpan['kind']; text: string }> = []; - for (const [blockIndex, raw] of data.entries()) { - if (!raw || typeof raw !== 'object') return null; - const b = raw as { type?: unknown; text?: unknown; data?: unknown; mimeType?: unknown }; - if (b.type === 'text') { - // A text that isn't a string shows as the converters' placeholder, - // rather than sending the whole array to the JSON fallback. - // Read once: a direct Agent API result is unchecked. - const text = b.text; - parts.push({ kind: 'text', text: typeof text === 'string' ? text.toWellFormed() : NON_STRING_TEXT_PLACEHOLDER }); - } else if (isInlineContradiction(b)) { - // RFC-005 vector 2: checked before the data branch so inline data - // claiming bulk disposition is withheld, not rendered. - parts.push({ kind: 'withheld', text: INLINE_WITHHELD_TEXT }); - } else if (b.type === 'image' && typeof b.data === 'string' && typeof b.mimeType === 'string') { - // Decoded byte estimate from base64 length (3/4 ratio, rounded). - const approxBytes = Math.floor(b.data.length * 3 / 4); - const ref = options?.imageRef?.(blockIndex, { data: b.data, mimeType: b.mimeType }) ?? null; - parts.push({ kind: 'image', text: formatToolImagePlaceholder(b.mimeType, formatSize(approxBytes), ref) }); - } else if (b.type === 'resource' || ((b.type === 'image' || b.type === 'audio') && typeof (b as { uri?: unknown }).uri === 'string')) { - // RFC-005 reference block: a bounded stub, never the raw block JSON - // (which used to inline the URI — and the whole array — into history). - // A stub that shows an image type occupies a slot in - // save_recent_image's inventory (a failing one — bytes live behind - // fetch_reference), so the index never slides past an image the - // resident saw. - const stub = referenceStubOrNull(b, 'from tool result'); - if (stub === null) return null; - parts.push({ kind: 'reference', text: stub }); - } else { - return null; - } - } + const blocks = toolResultBlocks(data, options?.dataIsContent, options?.claims); + if (blocks === null) return null; + const parts = blocks.map((block, blockIndex): { kind: HistoryBlockSpan['kind']; text: string } => { + if (block.kind !== 'image') return block; + // Decoded byte estimate from base64 length (3/4 ratio, rounded). + const approxBytes = Math.floor(block.data.length * 3 / 4); + const ref = options?.imageRef?.(blockIndex, { data: block.data, mimeType: block.mimeType }) ?? null; + return { kind: 'image', text: formatToolImagePlaceholder(block.mimeType, formatSize(approxBytes), ref) }; + }); if (options?.onBlocks) { const spans: HistoryBlockSpan[] = []; let start = 0; @@ -184,7 +386,7 @@ function tryHistoryStringFromContentArray(data: unknown, options?: HistorySerial spans.push({ kind: part.kind, start, end: start + part.text.length }); start += part.text.length + 1; // the newline that joins them } - options.onBlocks(spans); + options.onBlocks(spans, blocks); } return parts.map((part) => part.text).join('\n'); } @@ -194,13 +396,30 @@ function tryHistoryStringFromContentArray(data: unknown, options?: HistorySerial * A reference is a `resource` with a uri, at its top level or under * `resource` (where MCP embeds a resource's uri), or an `image` or `audio` * given by uri; inline data is an `image` or `audio` whose data is a string. - * A uri that is null, absent or empty counts as none, and data that isn't a - * string isn't inline data, so a record that only shares one of those type - * names is data. A uri of any other value, string or not, counts, so a - * credential fails closed. + * There a uri that is null, absent or empty counts as none, and data that + * isn't a string isn't inline data, so a record that only shares one of + * those type names is data. A uri of any other value, string or not, counts, + * so a credential fails closed. MCP's `resource_link` (RFC-005 §8.1) is + * governed whatever value its uri has, so one whose uri isn't usable is its + * marker; one that names no uri is data here, where as a block of a content + * array, which the classifier reads, it's the marker. */ function governedByRfc005(b: Record): boolean { if (b.type === 'resource') return hasUri(b.uri) || (isPlainRecord(b.resource) && hasUri(b.resource.uri)); + // A link that names a uri is governed whatever its value, so one without + // a usable uri (empty, null, not a string) is its marker here, as on the + // content lanes, never its fields. One that names none is data at depth, + // on the model's lanes and a script's, as main has every link at depth: + // its fields hold no uri RFC-005 governs, though a value in them, such as + // an untyped object's `uri`, is data as anywhere else. As a block of a + // content array on the model's line, the classifier reads it, and there + // it's the marker; a script's view of a module's array, which reads every + // block here, shows its fields, as module data reached a script on main + // (Ada-1017, room-203 #202138). + // That's the object the script lane makes of a link whose uri it leaves + // out (referenceBlockForScript), which a script's view, given it again, + // leaves as it is, as it leaves a reference's stub. + if (b.type === 'resource_link') return b.uri !== undefined; if (b.type === 'image' || b.type === 'audio') return typeof b.data === 'string' || hasUri(b.uri); return false; } @@ -218,13 +437,27 @@ function isPlainRecord(value: unknown): value is Record { * wherever it sits in the data. A value RFC-005 governs (governedByRfc005) is * shown as the reference lanes show it: a reference as its stub, an invalid * one (such as a uri that isn't a string, or MCP's embedded resource, whose - * uri isn't at its top level) as an unrecognized-block line, and inline data - * that also claims a uri or a disposition as withheld, so no uri is ever - * shown. Inline data that claims neither is itself, as is any other value: - * a block of another type keeps its fields, a `resource_link`'s uri among - * them, since RFC-005 governs references. + * uri isn't at its top level) as the marker saying why it isn't shown, and + * inline data that also claims a uri or a disposition as withheld. So of + * the uris RFC-005 governs, the only one shown is a link's, in its line, + * where the host can't fetch the link and its uri can't carry a credential + * (shownLinkUri). Inline data that claims neither is itself, as is any other + * value: a block of another type keeps its fields, a url among them, since + * RFC-005 governs references. + */ +function shownInHistory(value: unknown, claims?: ResultClaims): unknown { + return walkedInHistory(value, (b) => referenceStubOrNull(b, 'from tool result', undefined, { unfetchedLinkUri: true, claims })); +} + +/** + * The one walk of a tool result's data that showing it and reading its + * claims both take: a value RFC-005 governs is text, from `asText`, except + * inline data that claims no uri or disposition, which is itself, as is any + * other value, so serialization descends into the same values each time, so + * long as each value's toJSON and getters answer the same each time they're + * read. `asText` gives a governed value's text, or null where it's itself. */ -function shownInHistory(value: unknown): unknown { +function walkedInHistory(value: unknown, asText: (b: Record) => string | null): unknown { if (!value || typeof value !== 'object' || Array.isArray(value)) return value; const b = value as Record; if (!governedByRfc005(b)) return value; @@ -235,30 +468,113 @@ function shownInHistory(value: unknown): unknown { // carries none (hasUri). return hasUri(b.uri) || b.disposition !== undefined ? INLINE_WITHHELD_TEXT : value; } - return referenceStubOrNull(b, 'from tool result') ?? value; + return asText(b) ?? value; +} + +/** + * What a tool result's data claims of each uri its references name + * (ResultClaims), read by the walk serialization takes (walkedInHistory), + * through toJSON, the fields of inline data and every depth it shows: so + * every reference whose stub the result shows counts, and nothing it doesn't + * show does. A value the walk can't read or write, such as a cycle or a + * getter that throws, stops it, and then it's null, since what the rest of + * the data claims is unknown. It's a walk of its own before the one that + * shows the data, so a value's toJSON and getters run in both: one that + * answers differently the second time can claim what the shown text doesn't + * show, or show a reference the claims never read. Only data no check has + * read through JSON can do either: a module's result is parsed JSON from its + * dispatch, and an MCPL peer's blocks are parsed wire JSON, so this is the + * direct Agent API's, whose results a host gives unchecked. + */ +function claimsInHistory(data: unknown): ResultClaims | null { + try { + return claimsInHistoryOrThrow(data); + } catch { + // What the rest of the data claims is unknown, so a withheld result's + // kept fields keep nothing (keptWhenWithheld). + return null; + } +} + +/** claimsInHistory's walk, which throws what stops it. */ +function claimsInHistoryOrThrow(data: unknown): ResultClaims { + const testimonies: ReferenceTestimony[] = []; + // Inline data's never, which walkedInHistory withholds before it asks + // `read` (inlineNeverUri). + const nevers: string[] = []; + const noteInline = (value: unknown): void => { + if (!isPlainRecord(value) || (value.type !== 'image' && value.type !== 'audio')) return; + const never = inlineNeverUri(value); + if (never !== null) nevers.push(never); + }; + const read = (b: Record): string | null => { + const c = classifyBlock(b); + if (c.kind === 'reference') testimonies.push(c.testimony); + // Text where referenceStubOrNull gives text, so the walk descends where + // showing descends. Inline data never reaches here: walkedInHistory + // decides it first. + return c.kind === 'text' ? null : ''; + }; + noteInline(data); + const root = walkedInHistory(data, read); + // Its text is thrown away, so a string is written as 0, which keeps a + // long one from being copied for nothing; the walk descends as before. + if (root === data || typeof root !== 'string') { + JSON.stringify(data, (_key, value: unknown) => { + if (value !== data) noteInline(value); + const walked = walkedInHistory(value, read); + return typeof walked === 'string' ? 0 : walked; + }); + } + return resultClaims(testimonies, nevers); } /** * A value of a tool result's data as a script reads it, by history's rules * (shownInHistory), as a JSON replacer applies them at every depth: what * RFC-005 governs is shown as the script lane shows it, a reference as its - * stub's facts without its uri and an invalid one as the notice - * (referenceBlockForScript), and inline data that also claims a uri of any - * type or a disposition as the withheld notice, each as a text block. A - * null, absent or empty uri claims none (hasUri), so inline data beside one - * is itself. Any - * other value is itself. What it gives is either a stub or a notice, which + * stub's facts, a link's uri among them only where the model's line for it + * would show it, and an invalid one as the notice (referenceBlockForScript), + * and inline data that also claims a uri of any type or a disposition as the + * withheld notice, each as a text block. A null, absent or empty uri claims + * none (hasUri), so inline data beside one is itself. Any other value is + * itself. `claims` are those of everything the script's view shows + * (scriptViewClaims), and `line` the model's line for the result, its claims + * and its records as it reads them (ScriptLine), which decide where a link's + * uri is shown. What it gives is either a stub or a notice, which * RFC-005 doesn't govern, or the value itself, which it gives again, so * applying it twice gives what applying it once gave. */ -export function shownToScript(value: unknown): unknown { +export function shownToScript(value: unknown, claims?: ResultClaims, line?: ScriptLine): unknown { if (!value || typeof value !== 'object' || Array.isArray(value)) return value; const b = value as Record; if (!governedByRfc005(b)) return value; if ((b.type === 'image' || b.type === 'audio') && typeof b.data === 'string') { return hasUri(b.uri) || b.disposition !== undefined ? { type: 'text', text: INLINE_WITHHELD_TEXT } : value; } - return referenceBlockForScript(b); + return referenceBlockForScript(b, undefined, claims, line); +} + +/** + * The claims (ResultClaims) of everything a script's view of a content array + * shows, nested fields included (Ada-1017, room-203 #200094), by the walk + * history's JSON takes (claimsInHistory): a script is shown the references + * nested in a block's fields, each as its stub, so a `never` among them + * withholds that uri from every block in the view. A link's uri is still + * shown only where the model's line shows it, by the claims the line reads + * (contentClaims, which reads a block and not its fields): reading more + * claims can loosen as well as tighten, since an expiry makes a link one the + * host can't fetch (referenceBlockForScript's `line`). An MCPL peer's + * top-level block the classifier rejects is + * its notice, its fields unshown, so they don't count (`fromServer`). It + * throws what stops the walk, such as a value too deep to convert or a + * getter that throws, and the script's view then isn't written: the + * treatment never runs without the claims it reads (Ada-1017, room-203 + * #200250), whether or not its own walk, with other replacers and so other + * stack depths, would stop at the same value. + */ +export function scriptViewClaims(content: readonly unknown[], fromServer: boolean): ResultClaims { + return claimsInHistoryOrThrow(fromServer ? content.filter((block) => classifyBlock(block).kind !== 'invalid') : content); } /** @@ -270,32 +586,63 @@ export function shownToScript(value: unknown): unknown { * peer's block (referenceBlockForScript), its reference bound to the server * that sent it, and any other value is itself. A stub or a notice isn't * governed, so a block the top-level pass already treated is left as it is. + * `claims` are those of the script's whole view (scriptViewClaims). */ -export function shownToScriptFromServer(value: unknown, serverId: string): unknown { +export function shownToScriptFromServer(value: unknown, serverId: string, claims?: ResultClaims, line?: ScriptLine): unknown { if (!value || typeof value !== 'object' || Array.isArray(value)) return value; const b = value as Record; - return governedByRfc005(b) ? referenceBlockForScript(b, serverId) : value; + return governedByRfc005(b) ? referenceBlockForScript(b, serverId, claims, line) : value; } /** * The text RFC-005 shows a value of a tool result's data as, when it shows it * as text rather than as its fields: a reference's stub, an invalid one's - * unrecognized-block line, or inline data withheld. Null for any other value. + * marker, or inline data withheld. Null for any other value. */ export function rfc005TextOf(value: unknown): string | null { + // A single value claims only what it says itself, which its stub reads. const shown = shownInHistory(value); return shown !== value && typeof shown === 'string' ? shown : null; } +/** + * What the model's lane shows for a tool result it couldn't read through, + * check for references and show (nonErrorResultText, the lane's one catch, + * which logs the tool and the error): the tool ran, and its result isn't + * shown, rather than shown with fewer claims, where a link could show a uri a + * reference in the result withholds, and rather than a throw, which would + * leave the round unstored and unsent. Where the walk of its claims stops + * (historyJson throws then), the claims walk meets data no JSON can hold, + * such as a cycle or a BigInt, or data too deep to write, before the render + * does, so such data shows this, where the render threw (Desmond-2579's + * run); so does a render that stops after the walk, or a getter that throws + * (Ada-1017, room-203 #200344, #201013). The round completes, as a withheld + * result's kept fields keep nothing then (keptWhenWithheld). A script never + * gets it: its routes call the serializer, or nonErrorResultTextOrThrow, + * directly, whose walk throws where it stops, and fail naming the tool + * (#200374). An error result's own error field is read as before, and so + * are the fields the tool-result guard keeps of a result it withholds. + */ +export const UNCHECKED_RESULT_TEXT = + "[Result not shown: the tool ran, but its result couldn't be read through, checked for references and shown, so this host withheld it.]"; + /** * JSON of a tool result's data with each value in it shown as history shows - * it (shownInHistory), at any depth. Data that is itself shown as text, such - * as a bare reference, is that text. + * it (shownInHistory), at any depth, by the claims `given` or else its own + * (claimsInHistoryOrThrow), throwing where that walk stops. Data that is + * itself shown as text, such as a bare reference, is that text. */ -function historyJson(data: unknown): string | undefined { - const root = shownInHistory(data); +function historyJson(data: unknown, given?: ResultClaims): string | undefined { + // A walk that stopped can't say what the data claims, and showing it with + // no claims would let a link show a uri a reference withholds, should the + // render's own walk, with other replacers, get past the same value. + // Where the walk stops this throws: the model's lane answers with + // UNCHECKED_RESULT_TEXT and a log line (nonErrorResultText), a script's + // routes with a failure naming the tool. + const claims = given ?? claimsInHistoryOrThrow(data); + const root = shownInHistory(data, claims); if (root !== data && typeof root === 'string') return root; - return JSON.stringify(data, (_key, value: unknown) => shownInHistory(value)) as string | undefined; + return JSON.stringify(data, (_key, value: unknown) => shownInHistory(value, claims)) as string | undefined; } function formatSize(bytes: number): string { diff --git a/src/types/events.ts b/src/types/events.ts index 998b470a9..2c2b2a0a6 100644 --- a/src/types/events.ts +++ b/src/types/events.ts @@ -194,6 +194,18 @@ export interface ToolResult { * it reads the result's error text, if any. */ data?: unknown; + /** + * Whether `data` is a protocol result's content: an array in which every + * element is a content block, of a type this host can show or not. The + * framework sets it on an MCPL tool's result, and a module that returns a + * content array can set it too. When it's set, each element reaches the + * model as itself, as a reference's stub, or as a marker saying what it was + * and why it isn't shown, never as JSON. When it isn't, an array counts as + * content only when every element is a block of one of MCP's types (text, + * image, audio, resource, resource_link), and any other array is data, + * written as JSON. + */ + dataIsContent?: boolean; /** Error message (if failure) */ error?: string; /** Whether this was an error (for LLM) */ diff --git a/test/mcpl-agent-words.test.ts b/test/mcpl-agent-words.test.ts index 57b9f5a75..d676df25e 100644 --- a/test/mcpl-agent-words.test.ts +++ b/test/mcpl-agent-words.test.ts @@ -62,13 +62,15 @@ test("a reference block that can't be read says which part is missing, and what assert.equal(referenceStubOrNull(block), text); assert.deepEqual(convertBlock(block as never, 'test', 'words'), { type: 'text', text }); } - // Beside inline data, any string uri, an empty one too, stays withheld as - // on main: the tool-result lane can't show every inline shape yet, and - // showing this audio would put its data into history as raw JSON. + // Beside inline data, an empty link is no link either, now that the + // tool-result lane gives every inline shape its marker: this audio shows as + // a tool result's audio does, and its data stays out of history. const audio = { type: 'audio', data: 'aGVsbG8=', mimeType: 'audio/wav', uri: '' }; - assert.deepEqual(classifyBlock(audio), { kind: 'inline', contradiction: true }); + assert.deepEqual(classifyBlock(audio), { kind: 'inline', contradiction: false }); const stored = toolResultDataToHistoryString([{ type: 'text', text: 't' }, audio]); - assert.ok(stored.includes(INLINE_WITHHELD_TEXT) && !stored.includes('aGVsbG8='), stored); + assert.equal(stored, "t\n[Audio not shown: this host's tool results carry only text and images (audio/wav, 6B).]"); + // A link that isn't empty still withholds it. + assert.deepEqual(classifyBlock({ ...audio, uri: 'https://zz.example/a.wav' }), { kind: 'inline', contradiction: true }); }); // --------------------------------------------------------------------------- diff --git a/test/mcpl-reference-binding.test.ts b/test/mcpl-reference-binding.test.ts index 678a20060..4cf065ac8 100644 --- a/test/mcpl-reference-binding.test.ts +++ b/test/mcpl-reference-binding.test.ts @@ -150,19 +150,23 @@ test("fetch_reference reaches a push event's reference through its server's own dispatchFetchReferenceToolCall(agentName: string, call: { id: string; name: string; input: unknown }): void; }; Object.assign(framework, { - mcplServerConfigs: new Map([ - ['push-srv', { id: 'push-srv', url: wsOrigin }], - ['stdio-srv', { id: 'stdio-srv', command: 'stdio-server' }], - ['spawned-srv', { id: 'spawned-srv', command: 'stdio-server', url: wsOrigin }], - ['ws-command-srv', { id: 'ws-command-srv', transport: 'websocket', command: 'stdio-server', url: wsOrigin }], - ['stdio-url-srv', { id: 'stdio-url-srv', transport: 'stdio', url: wsOrigin }], - ['ws-http-srv', { id: 'ws-http-srv', transport: 'websocket', url: origin }], - ['ws-fragment-srv', { id: 'ws-fragment-srv', url: `${wsOrigin}/mcpl#frag` }], - ]), + mcplServerConfigs: new Map(), getWorkspaceModule: () => ({ writeBinary: async () => ({ success: true }) }), firstWritableMountName: () => 'scratch', emitTrace: () => {}, }); + // Configured as the framework configures a server: its config, which the + // fetcher reads, and its address in the reference registry, which the + // stub reads. + for (const config of [ + { id: 'push-srv', url: wsOrigin }, + { id: 'stdio-srv', command: 'stdio-server' }, + { id: 'spawned-srv', command: 'stdio-server', url: wsOrigin }, + { id: 'ws-command-srv', transport: 'websocket', command: 'stdio-server', url: wsOrigin }, + { id: 'stdio-url-srv', transport: 'stdio', url: wsOrigin }, + { id: 'ws-http-srv', transport: 'websocket', url: origin }, + { id: 'ws-fragment-srv', url: `${wsOrigin}/mcpl#frag` }, + ]) (framework as unknown as { setMcplServerConfig(config: unknown): void }).setMcplServerConfig(config); // Bounded, so a dispatch that never answers fails the test rather than hanging it. const fetchReference = (refId: string): Promise => { let timer: NodeJS.Timeout | undefined; @@ -249,6 +253,11 @@ async function puppetCall(answer: unknown, fetcher: unknown): Promise<{ result: }, getReferenceFetcher: () => fetcher, }); + // eido is dialed on its references' origin, so a stub of one can offer + // the fetch: the reference registry records its address, as the + // framework does for a server it configures. + (framework as unknown as { setMcplServerConfig(config: unknown): void }) + .setMcplServerConfig({ id: 'eido', url: 'wss://editor.example.invalid' }); const quiet = console.log; console.log = () => {}; const { result } = await framework.puppetToolCall('princess', 'mcpl--eido--look', {}) diff --git a/test/mcpl-reference-fetcher.test.ts b/test/mcpl-reference-fetcher.test.ts index 7199809dd..b0ac74dc3 100644 --- a/test/mcpl-reference-fetcher.test.ts +++ b/test/mcpl-reference-fetcher.test.ts @@ -9,7 +9,7 @@ import assert from 'node:assert/strict'; import { createServer } from 'node:http'; import { createHash } from 'node:crypto'; import { DEFAULT_FETCH_MAX_BYTES, ReferenceFetcher, normalizedOrigin, type ServerResolution } from '../src/mcpl/reference-fetcher.js'; -import { EXPIRY_REASON, noAddressReason, ReferenceRegistry, UNBOUND_REASON, classifyBlock, referenceExpiry, referenceRegistry } from '../src/mcpl/references.js'; +import { EXPIRY_REASON, noAddressReason, ReferenceRegistry, UNBOUND_REASON, classifyBlock, referenceExpiry, referenceRegistry, shownServerId, type ReferenceTestimony } from '../src/mcpl/references.js'; import { AgentFramework } from '../src/framework.js'; const PAYLOAD = Buffer.from('RIFF-not-really-a-wav-'.repeat(100)); @@ -103,6 +103,59 @@ test('happy path: origin-bound fetch, bearer auth applied host-side, digest veri assert.equal(again.path, out.path); }); +test("a registry journal keeps each record as it stood when it opened, through a change, a rebinding and an eviction, each open its own copies", () => { + const reg = new ReferenceRegistry(); + const t = (uri: string, extra: Record = {}): ReferenceTestimony => { + const c = classifyBlock({ type: 'resource', uri, name: 'x.csv', ...extra }); + assert.equal(c.kind, 'reference'); + return (c as { testimony: ReferenceTestimony }).testimony; + }; + const u = 'https://zz-journal.example/u.csv'; + const w = 'https://zz-journal.example/w.csv'; + reg.register(t(u, { disposition: 'never' }), 'first'); + reg.register(t(w)); + const one = reg.openJournal(); + // A later testimony and a second server, then a second journal, which + // takes the record as it stands then, and a third change neither sees. + reg.register(t(u), 'second'); + const two = reg.openJournal(); + reg.register(t(u, { expiresAt: '2000-01-01T00:00:00Z' }), 'third'); + assert.deepEqual([one.before(u)?.testimony.disposition, one.before(u)?.serverIds], ['never', ['first']]); + assert.deepEqual([two.before(u)?.testimony.disposition, two.before(u)?.testimony.expiresAt, two.before(u)?.serverIds], [undefined, undefined, ['first', 'second']]); + assert.deepEqual(reg.findByUri(u)?.serverIds, ['first', 'second', 'third']); + // A uri with no record when a journal opened has none before; one nothing + // has touched since reads as it is. + const v = 'https://zz-journal.example/v.csv'; + reg.register(t(v)); + assert.equal(one.before(v), undefined); + assert.equal(one.before(w), reg.findByUri(w)); + // A closed journal copies nothing more. + one.close(); + reg.register(t(w, { expiresAt: '2000-01-01T00:00:00Z' })); + assert.equal(one.before(w)?.testimony.expiresAt, '2000-01-01T00:00:00Z'); + // An eviction: the registry holds 2,000 records, so 2,001 more take u's, + // which the open journal still has as it stood. + for (let i = 0; i < 2001; i++) reg.register(t(`https://zz-journal.example/many/${i}.csv`)); + assert.equal(reg.findByUri(u), undefined); + assert.deepEqual([two.before(u)?.testimony.disposition, two.before(u)?.serverIds], [undefined, ['first', 'second']]); + two.close(); + // Two journals open at once, with no registration between their opens, + // each get a copy of their own. + const x = 'https://zz-journal.example/x.csv'; + reg.register(t(x), 'first'); + const three = reg.openJournal(); + const four = reg.openJournal(); + assert.equal(reg.journalCount, 2); + reg.register(t(x), 'second'); + assert.notEqual(three.before(x), four.before(x)); + assert.deepEqual(three.before(x)?.serverIds, ['first']); + assert.notEqual(three.before(x)?.serverIds, four.before(x)?.serverIds); + three.close(); + assert.equal(reg.journalCount, 1); + four.close(); + assert.equal(reg.journalCount, 0); +}); + test('vector 5: digest mismatch is never presented', async () => { const reg = new ReferenceRegistry(); const f = makeFetcher({ srv: { url: ORIGIN } }); @@ -850,10 +903,11 @@ test('refusal() answers what the fetch would refuse before dialing, case by case const refusal = f.refusal(record); assert.ok(refusal, `${what}: a refusal`); assert.equal(f.isEagerEligible(record), false, `${what}: no eager slot`); - assert.deepEqual(refusal, { error, remedy }, what); + // A stub's own words (inStub) ride along where the clause names the uri's parts. + assert.deepEqual({ error: refusal.error, remedy: refusal.remedy }, { error, remedy }, what); const out = await f.fetch(record); assert.equal(out.ok, false, what); - assert.deepEqual({ error: out.error, remedy: out.remedy }, refusal, `${what}: the fetch answers what refusal() said`); + assert.deepEqual({ error: out.error, remedy: out.remedy }, { error, remedy }, `${what}: the fetch answers what refusal() said`); } assert.equal([...hits.keys()].filter((k) => k.startsWith('/ok?case=')).length, 0, 'nothing dialed'); // Null where the fetch serves a saved copy or dials, and the one it would dial takes an eager slot. @@ -989,6 +1043,27 @@ test('vector 13: magic bytes beat a lying Content-Type header for storage type', assert.ok(rec.fetchedPath!.endsWith('.wav'), rec.fetchedPath); }); +test("the fetcher's own clauses name a server by its id as a stub does, bounded and with its controls removed", async () => { + const reg = new ReferenceRegistry(); + const id = `zz-\u202E${'x'.repeat(300)}`; + const shown = shownServerId(id); + // Bounded at 120 characters, its controls gone, cut with an ellipsis. + assert.equal(shown, `zz-${'x'.repeat(116)}…`); + // An answer's status, which every clause after dialing names the server in. + const answered = await makeFetcher({ [id]: { url: `ws://127.0.0.1:${port}` } }).fetch(makeRecord(reg, `${ORIGIN}/status/503`, {}, id)); + assert.equal(answered.error, `the "${shown}" server answered HTTP 503`); + // A spent budget, and a claim past what's left of one. + const spent = await makeFetcher({ [id]: { url: `ws://127.0.0.1:${port}`, autofetch: { maxTotalBytes: 0 } } }) + .fetch(makeRecord(reg, `${ORIGIN}/ok?shown-id-spent`, {}, id)); + assert.equal(spent.error, `this host's fetch budget for the "${shown}" server is used up (0 of 0 bytes)`); + const claimed = await makeFetcher({ [id]: { url: `ws://127.0.0.1:${port}`, autofetch: { maxTotalBytes: 1000 } } }) + .fetch(makeRecord(reg, `${ORIGIN}/ok?shown-id-claim`, { sizeBytes: 2000 }, id), { maxBytes: 1500 }); + assert.equal(claimed.error, `it claims 2000 bytes, more than the 1000 bytes left of this host's fetch budget for the "${shown}" server`); + for (const error of [answered.error, spent.error, claimed.error]) { + assert.ok(!error!.includes('\u202E') && !error!.includes('x'.repeat(200)), error); + } +}); + test("each HTTP status says what calling again can do: later, a fresh reference, the host's credentials, or nothing", async () => { const reg = new ReferenceRegistry(); const f = makeFetcher({ srv: { url: ORIGIN } }); diff --git a/test/mcpl-reference-fields.test.ts b/test/mcpl-reference-fields.test.ts index 73a11bcab..00e59340b 100644 --- a/test/mcpl-reference-fields.test.ts +++ b/test/mcpl-reference-fields.test.ts @@ -5,6 +5,11 @@ import { referenceBlockForScript, referenceRegistry, referenceStubOrNull, UNBOUND_REASON, unfetchableReason, } from '../src/mcpl/references.js'; +// The server these tests bind references to, dialed on their origin, so a +// bound reference's stub can offer the fetch; the framework records each +// server it configures the same way. +referenceRegistry.setServerAddress('srv', { url: 'wss://example.test' }, {}); + const fields = ['name', 'mimeType', 'expiresAt'] as const; const types = ['resource', 'image', 'audio'] as const; function reference(type: string, extra: Record = {}) { @@ -141,8 +146,8 @@ test("an expired reference reaches a script with the reason its stub gives, and test("an expired reference's stub says why, where a live one's advises fetch_reference", () => { // Each bound to its server, as a tool result's references are, so the expiry is what's left to say. - const cases = [['2001-01-01T00:00:00Z', /[\]—] expired$/], ['not-a-date', /[\]—] treated as expired, since its expiry time can't be read$/], - ['x'.repeat(REFERENCE_LIMITS.expiresAt + 1), /[\]—] treated as expired, since its expiry time isn't in a form this host accepts$/]] as const; + const cases = [['2001-01-01T00:00:00Z', /[\]—] the reference is expired$/], ['not-a-date', /[\]—] the reference is treated as expired, since its expiry time can't be read$/], + ['x'.repeat(REFERENCE_LIMITS.expiresAt + 1), /[\]—] the reference is treated as expired, since its expiry time isn't in a form this host accepts$/]] as const; for (const [i, [expiresAt, why]] of cases.entries()) { const { testimony } = reference('resource', { uri: `https://example.test/stub-expired-${i}`, expiresAt }); const stub = buildReferenceStub(testimony, undefined, 'srv'); @@ -164,11 +169,11 @@ test("an unbound reference's stub says it can't be fetched and why, where a boun // Registered with no server, as an in-process module's reference or a channel message's is. const { testimony } = reference('resource', { uri: 'https://example.test/stub-unbound' }); const unbound = buildReferenceStub(testimony, 'from a module'); - assert.ok(unbound.endsWith(`] from a module — ${UNBOUND_REASON}`), unbound); + assert.ok(unbound.endsWith(`] from a module — it's ${UNBOUND_REASON}`), unbound); assert.ok(!unbound.includes('fetch_reference'), unbound); // Expired as well: the fetcher refuses it as unbound first, so the stub says that. const { testimony: both } = reference('resource', { uri: 'https://example.test/stub-unbound-expired', expiresAt: '2001-01-01T00:00:00Z' }); - assert.ok(buildReferenceStub(both).endsWith(`] ${UNBOUND_REASON}`)); + assert.ok(buildReferenceStub(both).endsWith(`] it's ${UNBOUND_REASON}`)); // A later registration that knows its server binds the record, and the next stub advises fetch_reference. assert.ok(buildReferenceStub(testimony, 'from a server', 'srv').endsWith('] from a server — fetch with fetch_reference')); }); diff --git a/test/mcpl-runtime-lifecycle.test.ts b/test/mcpl-runtime-lifecycle.test.ts index 9f0f5cb6f..761a49249 100644 --- a/test/mcpl-runtime-lifecycle.test.ts +++ b/test/mcpl-runtime-lifecycle.test.ts @@ -14,6 +14,7 @@ import { WebSocketServer } from 'ws'; import type { AddressInfo } from 'node:net'; import { AgentFramework } from '../src/index.js'; +import { referenceRegistry } from '../src/mcpl/references.js'; import type { EventResponse, Module, ModuleContext, ToolDefinition, ToolResult } from '../src/index.js'; import { MockMembrane } from './helpers/mock-membrane.js'; @@ -102,6 +103,9 @@ test('connectMcplServer lazily initializes MCPL on a zero-server framework and e assert.deepEqual(framework.listMcplServers(), [], 'no servers configured at start'); await framework.connectMcplServer(serverConfig('dyn')); + // Its address is recorded where a reference's stub reads it: none, for a + // server the host spawns. + assert.equal(referenceRegistry.serverAddress('dyn'), 'no-address'); const servers = framework.listMcplServers(); assert.equal(servers.length, 1); diff --git a/test/save-recent-image.test.ts b/test/save-recent-image.test.ts index 840327cfe..5037507a8 100644 --- a/test/save-recent-image.test.ts +++ b/test/save-recent-image.test.ts @@ -73,8 +73,8 @@ import { createMockResponse, MockMembrane } from './helpers/mock-membrane.js'; * connecting to them: a ws:// or wss:// url for one the host connects to over * a network, a command alone for a stdio server. */ function configureServers(framework: AgentFramework, configs: Array<{ id: string; url?: string; command?: string; autofetch?: { maxTotalBytes?: number } }>): void { - const known = (framework as unknown as { mcplServerConfigs: Map }).mcplServerConfigs; - for (const config of configs) known.set(config.id, config); + const configure = (framework as unknown as { setMcplServerConfig(config: unknown): void }).setMcplServerConfig.bind(framework); + for (const config of configs) configure(config); } const PNG = Buffer.from( @@ -2041,11 +2041,46 @@ describe('ToolImageLedger', () => { ['bound', {}, 'srv'], ['unbound', {}, undefined], ]; + // And the reasons a fetch would be refused before it dials, each at its + // longest: a server's id is bounded as a label is, and a refused scheme + // isn't named. + const longServer = (kind: string) => `${kind}-${'s'.repeat(400)}`; + const host = {}; + // srv is dialed on the references' own origin, so its rows reach the + // expiry's reasons and the fetch the stub offers. + referenceRegistry.setServerAddress('srv', { url: 'wss://example.invalid' }, host); + referenceRegistry.setServerAddress(longServer('stdio'), 'no-address', host); + referenceRegistry.setServerAddress(longServer('elsewhere'), { url: 'wss://elsewhere.invalid' }, host); + referenceRegistry.setServerAddress(longServer('dialed'), { url: 'wss://example.invalid' }, host); + referenceRegistry.setServerAddress(longServer('garbled'), { url: 'not a url' }, host); + endings.push( + ['no address', {}, longServer('stdio')], + ['unknown server', {}, longServer('unknown')], + ['server address', {}, longServer('garbled')], + ['third-party origin', {}, longServer('elsewhere')], + ['scheme', { uri: `${'x'.repeat(400)}://example.invalid/sri-longest-scheme.png` }, longServer('dialed')], + ['user name or password', { uri: 'https://user:pass@example.invalid/sri-longest-userinfo.png' }, longServer('dialed')], + ); + const reasons: Record = { + passed: '— the reference is expired', + unparseable: '— the reference is treated as expired, since its expiry time can\'t be read', + rejected: '— the reference is treated as expired, since its expiry time isn\'t in a form this host accepts', + bound: '— fetch with fetch_reference', + unbound: '— it\'s not tied to any server', + 'no address': '— this host doesn\'t connect to the "stdio-s', + 'unknown server': '— this host no longer has the "unknown-s', + 'server address': '— the "garbled-s', + 'third-party origin': '— third-party origin:', + scheme: '— scheme not allowed: this host fetches only', + 'user name or password': '— its address carries a user name or password', + }; for (const [name, extra, serverId] of endings) { const stub = longestId(buildReferenceStub(testimony(name, extra), 'o'.repeat(400), serverId)); assert.ok(stub.length < REFERENCE_STUB_MAX_CHARS, `${name}: ${stub.length}`); assert.strictEqual(parseImagePlaceholders(stub).length, 1, name); + assert.ok(stub.includes(reasons[name]!), `${name}: ${stub}`); } + referenceRegistry.forgetServerAddress('srv', host); // A placeholder: RFC 6838 bounds a type or subtype name at 127 // characters, so normalizeMediaType keeps the longest essence and refuses // one character more, and the size and ref take a safe integer's digits. diff --git a/test/script-structured-references.test.ts b/test/script-structured-references.test.ts index 3c37b894e..d9b6d3c2a 100644 --- a/test/script-structured-references.test.ts +++ b/test/script-structured-references.test.ts @@ -2,7 +2,8 @@ * What a script receives from a tool whose result carries structuredContent, * through the real dispatch. An MCPL peer's content blocks get the treatment * RFC-005 gives them on every lane the model reads, since what a script prints - * comes back to the model: a reference arrives without its URI, as the host's + * comes back to the model: a reference arrives without its URI (a link + * carries one only where the model's line shows it), as the host's * reference id, what the server said about it in that block, and `savedTo`, * the workspace path the dispatch fetched its payload to for the script; a * block the classifier rejects arrives as the notice the model's lanes show, @@ -23,8 +24,9 @@ import { join } from 'node:path'; import { AgentFramework, WorkspaceModule } from '../src/index.js'; import type { EventResponse, Module, ModuleContext, ToolCall, ToolDefinition, ToolResult } from '../src/index.js'; -import { EXPIRY_REASON, INLINE_WITHHELD_TEXT, referenceRegistry } from '../src/mcpl/references.js'; -import type { ReferenceRecord } from '../src/mcpl/references.js'; +import { EXPIRY_REASON, INLINE_WITHHELD_TEXT, UNBOUND_REASON, classifyBlock, referenceRegistry } from '../src/mcpl/references.js'; +import type { ReferenceRecord, ReferenceTestimony } from '../src/mcpl/references.js'; +import { nonErrorResultText } from '../src/tool-result-history.js'; import { MockMembrane } from './helpers/mock-membrane.js'; const SAVED = 'board/references/report.pdf'; @@ -43,11 +45,11 @@ const blocks = (test: string) => [ // A reference whose payload the fetch saves. Its labels carry a bidi // override, a bidi mark and a control character. { type: 'resource', uri: uris(test).report, name: 'report\u202E.pdf\u0007', mimeType: 'application/pdf\u200F', sizeBytes: 1234, - digest: DIGEST, expiresAt: '2026-12-31T00:00:00Z' }, + digest: DIGEST, expiresAt: '2099-12-31T00:00:00Z' }, // The same reference again, saying something else about itself. { type: 'resource', uri: uris(test).report, name: 'report-copy.pdf' }, // A reference the fetch doesn't save. - { type: 'image', uri: uris(test).big, mimeType: 'image/png', disposition: 'ref', expiresAt: '2027-01-01T00:00:00Z' }, + { type: 'image', uri: uris(test).big, mimeType: 'image/png', disposition: 'ref', expiresAt: '2099-12-31T00:00:00Z' }, // Inline data that claims a bulk disposition, and inline data with a uri // beside it: withheld. { type: 'image', data: 'iVBORw0KGgo=', mimeType: 'image/png', disposition: 'ref' }, @@ -55,16 +57,17 @@ const blocks = (test: string) => [ // Inline data that claims nothing. { type: 'image', data: 'iVBORw0KGgo=', mimeType: 'image/png' }, // Blocks the classifier rejects: a uri longer than its limit (4096 UTF-16 - // units), an MCP embedded resource (its uri nested), and a type RFC-005 - // doesn't define. + // units), and an MCP embedded resource (its uri nested). { type: 'resource', uri: `https://files.example.invalid/long?sig=${'s'.repeat(4096)}`, name: 'long' }, { type: 'resource', resource: { uri: 'https://files.example.invalid/embedded?sig=embedded-token', mimeType: 'text/plain', text: 'embedded' } }, + // A link (MCP's resource_link), a reference (RFC-005 §8.1) the host can + // fetch here, so its uri, which also carries a signature, stays out. { type: 'resource_link', uri: 'https://files.example.invalid/link?sig=link-token', name: 'link' }, { type: 'text', text: 'two attachments' }, // A reference that counts as expired: with a bidi override after it, its // expiresAt doesn't parse, so it fails closed. It isn't fetched, and the // script is told why. - { type: 'image', uri: uris(test).old, mimeType: 'image/png', expiresAt: '2027-01-01T00:00:00Z\u202E' }, + { type: 'image', uri: uris(test).old, mimeType: 'image/png', expiresAt: '2099-12-31T00:00:00Z\u202E' }, ]; /** A `never` reference, which RFC-005 withholds, with its uri, from every lane the model reads. */ @@ -157,6 +160,9 @@ async function withPeers(fn: (h: Harness) => Promise, opts: { eager?: bool internals.mcplServerRegistry = { getServer: (id: string) => peers.get(id) ?? null, getAllServers: () => [], closeAll: async () => {} }; internals.mcplPrefixMap.set('mcpl--files', 'files'); internals.mcplPrefixMap.set('mcpl--plain', 'plain'); + // As configuring `files` records its address, which a reference's stub, + // and a script's object for it, read for what a fetch would do. + referenceRegistry.setServerAddress('files', { url: 'wss://files.example.invalid' }, framework); // Stands in for the host-mediated fetcher: every reference is eager or // none is, as the test asks, the report is saved and anything else // refused, and a saved record returns at once. @@ -211,24 +217,26 @@ async function withPeers(fn: (h: Harness) => Promise, opts: { eager?: bool }); } finally { await framework.stop(); + referenceRegistry.forgetServerAddress('files', framework); rmSync(dir, { recursive: true, force: true }); } } /** What a script reads for those blocks from an MCPL peer, given each reference's id. */ -const treated = (report: string, big: string, old: string) => [ +const treated = (report: string, big: string, old: string, link: string) => [ { type: 'resource', refId: report, name: 'report.pdf', mimeType: 'application/pdf', sizeBytes: 1234, digest: DIGEST, - expiresAt: '2026-12-31T00:00:00Z', savedTo: SAVED }, + expiresAt: '2099-12-31T00:00:00Z', savedTo: SAVED }, { type: 'resource', refId: report, name: 'report-copy.pdf', savedTo: SAVED }, - { type: 'image', refId: big, mimeType: 'image/png', expiresAt: '2027-01-01T00:00:00Z', savedTo: null }, + { type: 'image', refId: big, mimeType: 'image/png', expiresAt: '2099-12-31T00:00:00Z', savedTo: null }, { type: 'text', text: INLINE_WITHHELD_TEXT }, { type: 'text', text: INLINE_WITHHELD_TEXT }, { type: 'image', data: 'iVBORw0KGgo=', mimeType: 'image/png' }, { type: 'text', text: '[Reference not shown: its link is longer than this host accepts.]' }, { type: 'text', text: '[Reference not shown: it arrived without a link.]' }, - { type: 'text', text: `[Content not shown: this host can't display "resource_link" content.]` }, + { type: 'resource_link', refId: link, name: 'link', savedTo: null }, { type: 'text', text: 'two attachments' }, - { type: 'image', refId: old, mimeType: 'image/png', expiresAt: '2027-01-01T00:00:00Z', savedTo: null, expired: EXPIRY_REASON.unparseable }, + { type: 'image', refId: old, mimeType: 'image/png', expiresAt: '2099-12-31T00:00:00Z', savedTo: null, expired: EXPIRY_REASON.unparseable, + unfetchable: `the reference is ${EXPIRY_REASON.unparseable}` }, ]; test("an MCPL peer's structured result reaches a script with each block treated as the model's lanes treat it", async () => { @@ -237,15 +245,14 @@ test("an MCPL peer's structured result reaches a script with each block treated const out = await script('mcpl--files--attach'); assert.ok(!out.includes('example.invalid'), `no URI reaches the script: ${out.slice(0, 2000)}`); assert.deepEqual(fetched, [uris('treated').report, uris('treated').big], - 'every reference was fetched for the script before it read the result, except the one that counts as expired'); + 'every reference was fetched for the script before it read the result, except the one that counts as expired and the link, which is never fetched for a script'); const object = JSON.parse(out) as { content: Array<{ refId?: string }>; structuredContent: unknown; isError: boolean }; const [report, , big] = object.content; + const link = object.content[8]; const old = object.content[10]; - assert.match(String(report!.refId), /^ref_/); - assert.match(String(big!.refId), /^ref_/); - assert.match(String(old!.refId), /^ref_/); - assert.equal(new Set([report!.refId, big!.refId, old!.refId]).size, 3); - assert.deepEqual(object, { content: treated(report!.refId!, big!.refId!, old!.refId!), structuredContent: { attachments: 2 }, isError: false }); + for (const ref of [report, big, link, old]) assert.match(String(ref!.refId), /^ref_/); + assert.equal(new Set([report!.refId, big!.refId, link!.refId, old!.refId]).size, 4); + assert.deepEqual(object, { content: treated(report!.refId!, big!.refId!, old!.refId!, link!.refId!), structuredContent: { attachments: 2 }, isError: false }); }); }); @@ -308,7 +315,7 @@ test("an MCPL peer's large structured result reaches a script whole, its referen const object = JSON.parse(out) as { content: Array<{ refId?: string }>; structuredContent: { pad: string }; isError: boolean }; assert.equal(object.isError, false); assert.equal(object.structuredContent.pad.length, 5_000_000); - assert.deepEqual(object.content, treated(object.content[0]!.refId!, object.content[2]!.refId!, object.content[10]!.refId!)); + assert.deepEqual(object.content, treated(object.content[0]!.refId!, object.content[2]!.refId!, object.content[10]!.refId!, object.content[8]!.refId!)); }); }); @@ -344,7 +351,7 @@ test('an error with structured content is fetched for the script, and reaches it assert.ok(!out.includes('example.invalid'), out); const object = JSON.parse(out) as { content: Array<{ refId?: string }>; structuredContent: unknown; isError: boolean }; assert.deepEqual(object, { - content: [treated(object.content[0]!.refId!, '', '')[0], { type: 'text', text: 'the attach failed' }], + content: [treated(object.content[0]!.refId!, '', '', '')[0], { type: 'text', text: 'the attach failed' }], structuredContent: { partial: true }, isError: true, }); @@ -352,6 +359,54 @@ test('an error with structured content is fetched for the script, and reaches it }); }); +test("an MCPL peer's link reads every reference in its result, so a never anywhere in it withholds the link's uri from a script", { timeout: 30_000 }, async () => { + await withPeers(async (h) => { + // Off the server's origin, so the host can't fetch it and its line would + // show its uri, but a reference in the same result says never, and a + // later one, registered last, doesn't. + const uri = 'https://elsewhere.example.invalid/never-in-result.csv'; + h.answer({ + content: [ + { type: 'resource', uri, name: 'kept.csv', disposition: 'never' }, + { type: 'resource', uri, name: 'kept.csv' }, + { type: 'resource_link', uri, name: 'kept.csv' }, + ], + structuredContent: {}, + }); + const out = await h.script('mcpl--files--attach'); + assert.ok(!out.includes('elsewhere.example.invalid'), out); + // Each block reached the script, the link as a reference object. + const object = JSON.parse(out) as { content: Array<{ type: string; refId?: string; uri?: string }>; isError: boolean }; + assert.equal(object.isError, false); + assert.deepEqual(object.content.map((b) => b.type), ['resource', 'resource', 'resource_link']); + assert.match(String(object.content[2]!.refId), /^ref_/); + assert.equal(object.content[2]!.uri, undefined); + }); +}); + +test("a script's view reads the claims of the references nested in its blocks, so a nested never withholds a link's uri there", { timeout: 30_000 }, async () => { + await withPeers(async (h) => { + // Off the server's origin and with no credential in it, so the host can't + // fetch it and its line shows its uri: the control shows the script gets it. + const uri = 'https://elsewhere.example.invalid/nested-never/plain.csv'; + const link = { type: 'resource_link', uri, name: 'plain.csv' }; + h.answer({ content: [{ type: 'text', text: 'one link' }, link], structuredContent: {} }); + const control = JSON.parse(await h.script('mcpl--files--attach')) as { content: Array<{ uri?: string }> }; + assert.equal(control.content[1]!.uri, uri, 'with nothing claiming it, the link carries its uri'); + // A reference nested in another block's fields says never for that uri. + // The script is shown it, as its stub, so its claim counts for the view + // (Ada-1017, room-203 #200094), where the model's line, which shows the + // text block and not its fields, doesn't read it. + const sealed = { type: 'resource', uri, name: 'plain.csv', disposition: 'never' }; + h.answer({ content: [{ type: 'text', text: 'one link', attachment: sealed }, link], structuredContent: {} }); + const out = await h.script('mcpl--files--attach'); + assert.ok(!out.includes('elsewhere.example.invalid'), out); + const object = JSON.parse(out) as { content: Array> }; + assert.equal(object.content[1]!.uri, undefined); + assert.equal(typeof (object.content[0]!.attachment as Record).refId, 'string'); + }); +}); + test("an MCPL peer's structured failure reaches a script with what's nested in its blocks treated too", async () => { await withPeers(async ({ answer, script }) => { const sealed = never('nested-failure'); @@ -374,16 +429,19 @@ test("a module's structured result, an MCPL peer's it got through ctx.callTool, } const object = JSON.parse(out) as { content: Array<{ refId?: string }>; structuredContent: unknown; isError: boolean }; const [report, , big] = object.content; + const link = object.content[8]; const old = object.content[10]; const sealed = object.content[11]; - const expected: unknown[] = [...treated(report!.refId!, big!.refId!, old!.refId!)]; - // A resource_link isn't a reference RFC-005 governs: the model's history - // shows a module's as its fields, and so does the script. - expected[8] = blocks('module')[8]!; + // A resource_link is a reference (RFC-005 §8.1), a module's as an MCPL + // peer's: its object, its uri left out where the model's line for it + // leaves it out, as here, where it carries a signature. + const expected: unknown[] = [...treated(report!.refId!, big!.refId!, old!.refId!, link!.refId!)]; expected.push({ type: 'resource', refId: sealed!.refId!, name: 'sealed.pdf', savedTo: null }); assert.deepEqual(object, { content: expected, structuredContent: { attachments: 2 }, isError: false }); - assert.deepEqual(fetched, [uris('module').report, uris('module').big, uris('module').old, never('module').uri], - 'the module\'s references were fetched for the script, as before'); + // The one that counts as expired isn't fetched, as the model's lanes' + // fetch passes over it (autoFetchReferences). + assert.deepEqual(fetched, [uris('module').report, uris('module').big, never('module').uri], + 'the module\'s references were fetched for the script, but the expired one'); // Each came through ctx.callTool, whose arrival bound it to the server // that sent it, and the exit adds no other server. for (const uri of [uris('module').report, never('module').uri]) { @@ -415,7 +473,9 @@ test("a module's reference deeper in its blocks reaches a script as the model's assert.ok(!out.includes('never-token') && !out.includes('object-token'), out); const object = JSON.parse(out) as { content: [unknown, { value: { attachment: { refId: string } } }] }; const refId = object.content[1].value.attachment.refId; - const stub = { type: 'resource', refId, name: 'sealed.pdf', savedTo: null }; + // A reference the module made itself, bound to no server, says why its + // fetch would be refused. + const stub = { type: 'resource', refId, name: 'sealed.pdf', savedTo: null, unfetchable: `it's ${UNBOUND_REASON}` }; assert.deepEqual(object, { content: [ { type: 'text', text: 'one sealed report' }, @@ -470,7 +530,7 @@ test("a module tool's result is treated when a server with a prefix that fits it assert.equal(prefixes.get('fwd--deep'), 'files', 'the prefix arrived while the call ran'); assert.ok(!out.includes('never-token'), out); const object = JSON.parse(out) as { content: [{ refId: string }] }; - assert.deepEqual(object.content, [{ type: 'resource', refId: object.content[0].refId, name: 'sealed.pdf', savedTo: null }]); + assert.deepEqual(object.content, [{ type: 'resource', refId: object.content[0].refId, name: 'sealed.pdf', savedTo: null, unfetchable: `it's ${UNBOUND_REASON}` }]); assert.deepEqual(fetched, [never('connect').uri], 'the exit fetches for it as for any module\'s result'); }); }); @@ -482,7 +542,7 @@ test("a module's failed result with structured content reaches a script with its assert.ok(!out.includes('never-token'), out); const object = JSON.parse(out) as { content: [unknown, { refId: string }]; structuredContent: unknown; isError: boolean }; assert.deepEqual(object, { - content: [{ type: 'text', text: 'partly attached' }, { type: 'resource', refId: object.content[1].refId, name: 'sealed.pdf', savedTo: null }], + content: [{ type: 'text', text: 'partly attached' }, { type: 'resource', refId: object.content[1].refId, name: 'sealed.pdf', savedTo: null, unfetchable: `it's ${UNBOUND_REASON}` }], structuredContent: { partial: true }, isError: true, }); @@ -578,6 +638,569 @@ test("a RangeError from serializing an MCPL peer's blocks, as a result past the }); }); +test("a nested never decides over a plain reference to its uri at the top, for a link at the top or nested, from a peer or a module", { timeout: 30_000 }, async () => { + await withPeers(async ({ answer, script, exit }) => { + // The top-level blocks alone claim nothing that withholds the uri, so + // only a reading of the claims the whole view shows (scriptViewClaims) + // keeps it from the link. Each link comes before the references, so a + // registry that read them as it went wouldn't have them yet. + // A uri of its own for each route, so neither finds the other's + // references in the registry. + const content = (route: string, where: string) => { + const uri = `https://elsewhere.example.invalid/strictest-${route}-${where}/plain.csv`; + const link = { type: 'resource_link', uri, name: 'plain.csv' }; + return [ + where === 'top' ? link : { type: 'text', text: 'a nested link', link }, + { type: 'resource', uri, name: 'plain.csv' }, + { type: 'text', text: 'sealed', attachment: { type: 'resource', uri, name: 'plain.csv', disposition: 'never' } }, + ]; + }; + for (const where of ['top', 'nested']) { + answer({ content: content('peer', where), structuredContent: {} }); + const fromPeer = await script('mcpl--files--attach'); + const fromModule = await exit({ success: true, data: content('module', where), structured: {} }, 'fwd--whole', false); + for (const out of [fromPeer, fromModule]) { + assert.ok(!out.includes('elsewhere.example.invalid'), `${where}: ${out}`); + // The view was written, each block in it, the link as its object. + const object = JSON.parse(out) as { content: Array>; isError: boolean }; + assert.equal(object.isError, false, out); + assert.equal(object.content.length, 3, out); + const link = (where === 'top' ? object.content[0] : object.content[0]!.link) as Record; + assert.equal(link.type, 'resource_link', out); + assert.match(String(link.refId), /^ref_/, out); + assert.equal(link.uri, undefined, out); + } + } + }); +}); + +test("a link's uri reaches a script only where the model's line shows it: a nested expiry, which makes the link unfetchable, doesn't show it", { timeout: 30_000 }, async () => { + await withPeers(async ({ answer, script, modelCall }) => { + // On the server's origin and with no credential in it, so the host can + // fetch it, and the model's line, reading the top-level blocks, shows no + // uri. A reference nested in another block's field says the uri has + // expired, which a script's view reads: the host couldn't fetch it then, + // and a link it can't fetch shows its uri, so reading more claims would + // show the script what the line withholds (Ada-1017, room-203 #201013). + const uri = 'https://files.example.invalid/line-claims/report.csv'; + const content = () => [ + { type: 'resource_link', uri, name: 'report.csv' }, + { type: 'text', text: 'an old copy', attachment: { type: 'resource', uri, name: 'report.csv', expiresAt: '2000-01-01T00:00:00Z' } }, + ]; + answer({ content: content(), structuredContent: {} }); + // The model's line for the same result, as history stores it. + const line = nonErrorResultText(await modelCall('attach')); + assert.match(line, /report\.csv/); + assert.ok(!line.includes('line-claims'), line); + answer({ content: content(), structuredContent: {} }); + const out = await script('mcpl--files--attach'); + assert.ok(!out.includes('line-claims'), out); + const object = JSON.parse(out) as { content: Array>; isError: boolean }; + assert.equal(object.isError, false); + // Its words still read the view's claims: a fetch with its id would be refused. + assert.equal(object.content[0]!.expired, EXPIRY_REASON.passed); + }); +}); + +test("an MCPL peer's script-demand fetch reads the view's claims: a nested expiry isn't fetched, and a view that can't be read fetches nothing", async () => { + await withPeers(async ({ answer, script, fetched }) => { + // A reference the demand fetch would save, and a reference nested in + // another block that says its uri has expired, which only a reading of + // the whole view's claims sees (Ada-1017, room-203 #201013). + const expiredNested = blocks('nested-expiry')[0] as { uri: string }; + answer({ + content: [expiredNested, { type: 'text', text: 'an old copy', attachment: { type: 'resource', uri: expiredNested.uri, expiresAt: '2000-01-01T00:00:00Z' } }], + structuredContent: {}, + }); + const object = JSON.parse(await script('mcpl--files--attach')) as { content: Array>; isError: boolean }; + assert.equal(object.isError, false); + assert.deepEqual(fetched, [], 'the uri a nested reference calls expired was not fetched'); + assert.equal(object.content[0]!.expired, EXPIRY_REASON.passed); + // A view whose claims walk stops isn't written, and nothing is fetched for it first. + const stuck = { get type(): string { throw new RangeError('Maximum call stack size exceeded'); } }; + answer({ content: [blocks('stopped-walk')[0], { type: 'text', text: 'deep', deep: stuck }], structuredContent: {} }); + const failed = JSON.parse(await script('mcpl--files--attach')) as { content: [{ text: string }]; isError: boolean }; + assert.equal(failed.isError, true); + assert.match(failed.content[0].text, /can't be converted to JSON/); + assert.deepEqual(fetched, [], 'nothing was fetched for a view that was never written'); + }); +}); + +test("a nested link's uri doesn't turn on the order of the blocks: a reference nested before it doesn't change what its line reads", { timeout: 30_000 }, async () => { + await withPeers(async ({ answer, script, exit }) => { + // Fetchable, so the line, reading no claim of this result for the uri, + // shows no uri. A nested expired reference registered before the link, + // as the treatment's own pass registers what it shows, would make the + // record unfetchable for the link and show its uri, in one order only. + for (const [i, order] of ([['expired', 'link'], ['link', 'expired']] as const).entries()) { + const uri = `https://files.example.invalid/block-order-${i}/report.csv`; + const fields: Record = {}; + for (const name of order) { + fields[name] = name === 'link' + ? { type: 'resource_link', uri, name: 'report.csv' } + : { type: 'resource', uri, name: 'report.csv', expiresAt: '2000-01-01T00:00:00Z' }; + } + const content = [{ type: 'text', text: 'two attachments', ...fields }]; + answer({ content, structuredContent: {} }); + const fromPeer = await script('mcpl--files--attach'); + assert.ok(!fromPeer.includes(`block-order-${i}`), `${order.join(' then ')}: ${fromPeer}`); + const view = JSON.parse(fromPeer) as { content: Array>>; isError: boolean }; + assert.deepEqual([view.isError, typeof view.content[0]!.link!.refId, 'uri' in view.content[0]!.link!], [false, 'string', false], fromPeer); + } + // An earlier result's never, which a later result's plain reference, + // nested before the link, would clear on the record. + const sealedUri = 'https://elsewhere.example.invalid/order-never/report.csv'; + answer({ content: [{ type: 'resource', uri: sealedUri, name: 'report.csv', disposition: 'never' }], structuredContent: {} }); + await script('mcpl--files--attach'); + const later = [{ type: 'text', text: 'again', plain: { type: 'resource', uri: sealedUri, name: 'report.csv' }, link: { type: 'resource_link', uri: sealedUri, name: 'report.csv' } }]; + answer({ content: later, structuredContent: {} }); + const out = await script('mcpl--files--attach'); + // The view reads the result's plain reference, the latest word, so its + // never no longer holds there; the line, which shows neither, reads the + // record as it stood, and withholds the uri; so does the script. + assert.ok(!out.includes('order-never'), out); + const view = JSON.parse(out) as { content: Array>>; isError: boolean }; + assert.deepEqual([view.isError, typeof view.content[0]!.link!.refId], [false, 'string'], out); + // A module's result the same way, for a uri of its own, in both orders. + for (const [i, nestedFirst] of [true, false].entries()) { + const moduleUri = `https://elsewhere.example.invalid/order-never-module-${i}/report.csv`; + answer({ content: [{ type: 'resource', uri: moduleUri, name: 'report.csv', disposition: 'never' }], structuredContent: {} }); + await script('mcpl--files--attach'); + const plain = { type: 'resource', uri: moduleUri, name: 'report.csv' }; + const link = { type: 'resource_link', uri: moduleUri, name: 'report.csv' }; + const block = nestedFirst ? { type: 'text', text: 'again', plain, link } : { type: 'text', text: 'again', link, plain }; + const fromModule = await exit({ success: true, data: [block], structured: {} }, 'fwd--whole', false); + assert.ok(!fromModule.includes(`order-never-module-${i}`), fromModule); + const moduleView = JSON.parse(fromModule) as { content: Array>>; isError: boolean }; + assert.deepEqual([moduleView.isError, typeof moduleView.content[0]!.link!.refId], [false, 'string'], fromModule); + } + }); +}); + +test("a link's uri reads its record as it stood before the script lane registered anything, through an eviction the treatment's own registrations make", { timeout: 60_000 }, async () => { + await withPeers(async ({ answer, script, exit }) => { + for (const route of ['peer', 'module'] as const) { + // An earlier result marks the uri never; off the server's origin, so a + // link to it with no such claim shows its uri. + const uri = `https://elsewhere.example.invalid/evicted-${route}/report.csv`; + answer({ content: [{ type: 'resource', uri, name: 'report.csv', disposition: 'never' }], structuredContent: {} }); + await script('mcpl--files--attach'); + // More references nested before the link than the registry holds, so + // the treatment evicts the uri's record before it reaches the link. + const many = Array.from({ length: 2001 }, (_, i) => ({ type: 'resource', uri: `https://files.example.invalid/many-${route}/${i}.csv`, name: `${i}.csv` })); + const content = [{ type: 'text', text: 'many attachments', many, link: { type: 'resource_link', uri, name: 'report.csv' } }]; + let out: string; + if (route === 'peer') { + answer({ content, structuredContent: {} }); + out = await script('mcpl--files--attach'); + } else { + out = await exit({ success: true, data: content, structured: {} }, 'fwd--whole', false); + } + assert.ok(!out.includes(`evicted-${route}/report.csv`), `${route}: ${out.slice(-600)}`); + const view = JSON.parse(out) as { content: Array>>; isError: boolean }; + assert.deepEqual([view.isError, typeof view.content[0]!.link!.refId], [false, 'string'], route); + // The never record was evicted: the uri's record now is the link's own. + assert.equal(referenceRegistry.findByUri(uri)?.testimony.fromLink, true, route); + assert.equal(referenceRegistry.journalCount, 0, `${route}: the lane closed its journal`); + } + }); +}); + +/** A testimony, as the classifier reads a block. */ +const testimonyOf = (block: Record): ReferenceTestimony => (classifyBlock(block) as { testimony: ReferenceTestimony }).testimony; + +/** Make `uri`'s record the registry's oldest, with the registry full: it + * holds 2,000, so 1,999 new records after it take the place of every older + * one. */ +const fillAfter = (tag: string): void => { + for (let i = 0; i < 1999; i++) referenceRegistry.register(testimonyOf({ type: 'resource', uri: `https://files.example.invalid/fill-${tag}/${i}.csv`, name: `${i}.csv` })); +}; + +test("a module's result a script reads as text registers its links in the result's order, so a later reference keeps the model's line's never on a link before it", { timeout: 60_000 }, async () => { + await withPeers(async ({ answer, script, exit }) => { + const uri = 'https://elsewhere.example.invalid/text-order/report.csv'; + answer({ content: [{ type: 'resource', uri, name: 'report.csv', disposition: 'never' }], structuredContent: {} }); + await script('mcpl--files--attach'); + fillAfter('text-order'); + // The model's render reaches the link first, which keeps its record, and + // then registers the new reference, which evicts another. + const data = [ + { type: 'resource_link', uri, name: 'report.csv' }, + { type: 'resource', uri: 'https://files.example.invalid/text-order/new.csv', name: 'new.csv' }, + ]; + const out = await exit({ success: true, data }, 'fwd--whole', false); + assert.ok(!out.includes('text-order/report.csv'), out); + const reply = JSON.parse(out) as { content: [{ text: string }]; isError: boolean }; + assert.equal(reply.isError, false, out); + // The link's line rendered, as the withheld stub that names it. + assert.match(reply.content[0].text, /^\[ref_\w+\] report\.csv/m, out); + assert.equal(referenceRegistry.findByUri(uri)?.testimony.disposition, 'never'); + }); +}); + +test("a module's view reads the registry as the model's line leaves it: where the result's own reference evicts an earlier never before its link, script and line both show the uri", { timeout: 60_000 }, async () => { + await withPeers(async ({ answer, script, exit }) => { + // The demand loop registers the result's blocks in the order the model's + // render does, so the new reference evicts the never record before the + // link, for both: an earlier result's never lasts as long as the registry + // keeps it (Ada-1017, room-203 #203011). + const uri = 'https://elsewhere.example.invalid/view-evicted/report.csv'; + const data = () => [ + { type: 'resource', uri: 'https://files.example.invalid/view-evicted/new.csv', name: 'new.csv' }, + { type: 'resource_link', uri, name: 'report.csv' }, + ]; + const seed = async (): Promise => { + answer({ content: [{ type: 'resource', uri, name: 'report.csv', disposition: 'never' }], structuredContent: {} }); + await script('mcpl--files--attach'); + }; + await seed(); + fillAfter('view-evicted-line'); + const line = nonErrorResultText({ data: data() }); + assert.ok(line.includes('view-evicted/report.csv'), line); + await seed(); + fillAfter('view-evicted-script'); + const out = await exit({ success: true, data: data(), structured: {} }, 'fwd--whole', false); + const link = (JSON.parse(out) as { content: Array> }).content[1]!; + assert.equal(link.uri, uri, out); + assert.equal(referenceRegistry.journalCount, 0); + }); +}); + +test("a module's array the model's line walks at depth keeps its never on a nested link the walk reaches before a later top-level one, as a script reads it", { timeout: 90_000 }, async () => { + await withPeers(async ({ answer, script, exit }) => { + // Not a content array, so the model's line walks it depth-first: it + // reaches the nested link first, which keeps its record, and then the + // later top-level link, whose registration evicts another. + const uri = 'https://elsewhere.example.invalid/walk-order/report.csv'; + const data = () => [ + { wrap: { type: 'resource_link', uri, name: 'report.csv' } }, + { type: 'resource_link', uri: 'https://files.example.invalid/walk-order/new.csv', name: 'new.csv' }, + ]; + const seed = async (tag: string): Promise => { + answer({ content: [{ type: 'resource', uri, name: 'report.csv', disposition: 'never' }], structuredContent: {} }); + await script('mcpl--files--attach'); + fillAfter(`walk-order-${tag}`); + }; + await seed('line'); + const line = nonErrorResultText({ data: data() }); + assert.ok(!line.includes('walk-order/report.csv'), line); + for (const structured of [true, false]) { + await seed(structured ? 'view' : 'text'); + const out = await exit({ success: true, data: data(), ...(structured ? { structured: {} } : {}) }, 'fwd--whole', false); + assert.ok(!out.includes('walk-order/report.csv'), `${structured ? 'view' : 'text'}: ${out}`); + const reply = JSON.parse(out) as { content: Array> & { text?: string }>; isError: boolean }; + assert.equal(reply.isError, false, out); + // The nested link was shown, as its object or its stub, not dropped. + if (structured) assert.equal(typeof reply.content[0]!.wrap!.refId, 'string', out); + else assert.match(String(reply.content[0]!.text), /\[ref_\w+\] report\.csv/, out); + } + assert.equal(referenceRegistry.journalCount, 0); + }); +}); + +test("a module's array the model's line walks at depth reads its records from before the demand loop, so the loop's own reference can't evict a never the view's nested link keeps", { timeout: 60_000 }, async () => { + await withPeers(async ({ answer, script, exit }) => { + // The later top-level element is a reference, which the loop registers, + // in a full registry evicting the never record; the journal, open before + // the loop for data the line walks at depth, keeps it. + const uri = 'https://elsewhere.example.invalid/walk-resource/report.csv'; + answer({ content: [{ type: 'resource', uri, name: 'report.csv', disposition: 'never' }], structuredContent: {} }); + await script('mcpl--files--attach'); + fillAfter('walk-resource'); + const data = [ + { wrap: { type: 'resource_link', uri, name: 'report.csv' } }, + { type: 'resource', uri: 'https://files.example.invalid/walk-resource/new.csv', name: 'new.csv' }, + ]; + const out = await exit({ success: true, data, structured: {} }, 'fwd--whole', false); + assert.ok(!out.includes('walk-resource/report.csv'), out); + const view = JSON.parse(out) as { content: Array>>; isError: boolean }; + assert.deepEqual([view.isError, typeof view.content[0]!.wrap!.refId], [false, 'string'], out); + assert.equal(referenceRegistry.findByUri(uri)?.testimony.fromLink, true, 'the loop evicted the never record'); + assert.equal(referenceRegistry.journalCount, 0); + }); +}); + +test("a module's array its result says is content has its links registered in order, an unknown block among them", { timeout: 60_000 }, async () => { + await withPeers(async ({ answer, script, exit }) => { + // Content by the result's own flag, though one element is of no MCP + // type: the model's render shows it block by block, so the loop + // registers the link first, as the render does, keeping its record. + const uri = 'https://elsewhere.example.invalid/flag-order/report.csv'; + answer({ content: [{ type: 'resource', uri, name: 'report.csv', disposition: 'never' }], structuredContent: {} }); + await script('mcpl--files--attach'); + fillAfter('flag-order'); + const data = [ + { type: 'resource_link', uri, name: 'report.csv' }, + { type: 'resource', uri: 'https://files.example.invalid/flag-order/new.csv', name: 'new.csv' }, + { type: 'zz-unknown', value: 1 }, + ]; + const out = await exit({ success: true, data, dataIsContent: true } as ToolResult, 'fwd--whole', false); + assert.ok(!out.includes('flag-order/report.csv'), out); + assert.equal((JSON.parse(out) as { isError: boolean }).isError, false, out); + assert.equal(referenceRegistry.findByUri(uri)?.testimony.disposition, 'never'); + }); +}); + +test("a module's failed result, whose model's line is its error and registers nothing, keeps a never its blocks' registrations would evict, as a script reads it", { timeout: 60_000 }, async () => { + await withPeers(async ({ answer, script, exit }) => { + const uri = 'https://elsewhere.example.invalid/error-order/report.csv'; + answer({ content: [{ type: 'resource', uri, name: 'report.csv', disposition: 'never' }], structuredContent: {} }); + await script('mcpl--files--attach'); + fillAfter('error-order'); + const data = [ + { type: 'resource', uri: 'https://files.example.invalid/error-order/new.csv', name: 'new.csv' }, + { type: 'resource_link', uri, name: 'report.csv' }, + ]; + const out = await exit({ success: false, isError: true, error: 'the attach failed', data, structured: {} } as unknown as ToolResult, 'fwd--whole', false); + const view = JSON.parse(out) as { content: Array>; isError: boolean }; + assert.equal(view.isError, true, out); + assert.ok(!out.includes('error-order/report.csv'), out); + assert.equal(typeof view.content[1]!.refId, 'string', out); + assert.equal(referenceRegistry.journalCount, 0); + }); +}); + +test("a link's line reads its record's server binding as it stood before an eviction the treatment makes, so a link the host could fetch shows a script no uri", { timeout: 60_000 }, async () => { + await withPeers(async ({ answer, script, exit }) => { + // Bound to the files server, on its origin, by the peer's dispatch. + const uri = 'https://files.example.invalid/bound-evicted/report.csv'; + answer({ content: [{ type: 'resource', uri, name: 'report.csv' }], structuredContent: {} }); + await script('mcpl--files--attach'); + assert.deepEqual(referenceRegistry.findByUri(uri)?.serverIds, ['files']); + // A module's nested references, registered with no server, evict it + // before the link, whose own registration then makes it unbound. + const many = Array.from({ length: 2001 }, (_, i) => ({ type: 'resource', uri: `https://files.example.invalid/bound-many/${i}.csv`, name: `${i}.csv` })); + const out = await exit({ success: true, data: [{ type: 'text', text: 'many', many, link: { type: 'resource_link', uri, name: 'report.csv' } }], structured: {} }, 'fwd--whole', false); + assert.ok(!out.includes('bound-evicted/report.csv'), out.slice(-600)); + assert.deepEqual(referenceRegistry.findByUri(uri)?.serverIds, [], 'evicted, and made again unbound'); + const view = JSON.parse(out) as { content: Array>> }; + assert.equal(typeof view.content[0]!.link!.refId, 'string'); + }); +}); + +test("an earlier result's never on one uri doesn't withhold, from a script, a link to another that shows as the same text, where the model's line shows it", { timeout: 30_000 }, async () => { + await withPeers(async ({ answer, script, modelCall }) => { + // Each record speaks for its own uri: the earlier never is the marked + // uri's, and the link names the clean one, whose record has none. + const marked = 'https://elsewhere.example.invalid/cut-history/\u200Freport.csv'; + const clean = 'https://elsewhere.example.invalid/cut-history/report.csv'; + answer({ content: [{ type: 'resource', uri: marked, name: 'report.csv', disposition: 'never' }], structuredContent: {} }); + await script('mcpl--files--attach'); + const content = () => [ + { type: 'text', text: 'again', attachment: { type: 'resource', uri: marked, name: 'report.csv' } }, + { type: 'resource_link', uri: clean, name: 'report.csv' }, + ]; + answer({ content: content(), structuredContent: {} }); + const line = nonErrorResultText(await modelCall('attach')); + assert.ok(line.includes('cut-history/report.csv'), line); + answer({ content: content(), structuredContent: {} }); + const out = await script('mcpl--files--attach'); + const link = (JSON.parse(out) as { content: Array> }).content[1]!; + assert.equal(link.uri, clean, out); + }); +}); + +test("an MCPL peer's result a script reads as text has its demand fetch read each top-level block's own expiry, as a model's call does, not a view it never gets", async () => { + await withPeers(async ({ answer, script, fetched }) => { + // No structured content: the script reads the model lane's text, whose + // stubs read the top-level blocks' claims, so a reference nested in a + // block's field, which the text doesn't show, stops no fetch. + const top = blocks('text-route')[0] as { uri: string }; + answer({ content: [top, { type: 'text', text: 'an old copy', attachment: { type: 'resource', uri: top.uri, expiresAt: '2000-01-01T00:00:00Z' } }] }); + const out = await script('mcpl--files--attach'); + assert.deepEqual(fetched, [top.uri]); + assert.ok(out.includes(SAVED), out); + }); +}); + +test("an MCPL peer's view reads the claims its dispatch read for the demand fetch, not a second reading of its blocks", { timeout: 30_000 }, async () => { + await withPeers(async ({ answer, script, fetched }) => { + // A nested reference that says never until the demand fetch has run, as + // only a value read twice could: a view that read its claims again would + // see none, and show the link's uri. + const uri = 'https://elsewhere.example.invalid/read-once/report.csv'; + const sealed = { type: 'resource', uri, name: 'report.csv', get disposition(): string | undefined { return fetched.length === 0 ? 'never' : undefined; } }; + answer({ + content: [blocks('read-once')[0], { type: 'text', text: 'sealed', sealed }, { type: 'resource_link', uri, name: 'report.csv' }], + structuredContent: {}, + }); + const out = await script('mcpl--files--attach'); + assert.equal(fetched.length, 1, 'the demand fetch ran between the two readings'); + assert.ok(!out.includes('read-once/report.csv'), out); + const view = JSON.parse(out) as { content: Array>; isError: boolean }; + assert.deepEqual([view.isError, view.content.length, typeof view.content[2]!.refId], [false, 3, 'string'], out); + }); +}); + +test("inline data's expiry claims nothing for a link to its uri, which the host can fetch, so the link shows no uri", { timeout: 30_000 }, async () => { + await withPeers(async ({ answer, script, modelCall }) => { + // Fetchable, so only an expiry claimed for the uri would make it a link + // the host can't fetch, whose uri shows. Inline data's claims beside its + // data count for nothing but never (Ada-1017, room-203 #201013). + const uri = 'https://files.example.invalid/inline-expiry/report.csv'; + const content = () => [ + { type: 'image', data: 'iVBORw0KGgo=', mimeType: 'image/png', uri, disposition: 'ref', expiresAt: '2000-01-01T00:00:00Z' }, + { type: 'resource_link', uri, name: 'report.csv' }, + ]; + answer({ content: content(), structuredContent: {} }); + const line = nonErrorResultText(await modelCall('attach')); + assert.ok(!line.includes('inline-expiry'), line); + answer({ content: content(), structuredContent: {} }); + const out = await script('mcpl--files--attach'); + assert.ok(!out.includes('inline-expiry'), out); + const view = JSON.parse(out) as { content: Array>; isError: boolean }; + assert.deepEqual([view.isError, typeof view.content[1]!.refId], [false, 'string'], out); + }); +}); + +test("a never nested where only the script's view reads it withholds a link whose uri shows as the same text", { timeout: 30_000 }, async () => { + await withPeers(async ({ answer, script, modelCall }) => { + // Off the server's origin, so the line, which reads the top-level blocks + // alone, shows the link's uri; the view reads the nested never, found by + // the text the link shows. + const uri = 'https://elsewhere.example.invalid/view-cut/report.csv'; + const content = () => [ + { type: 'text', text: 'sealed', attachment: { type: 'resource', uri, name: 'report.csv', disposition: 'never' } }, + { type: 'resource_link', uri: 'https://elsewhere.example.invalid/view-cut/\u200Freport.csv', name: 'report.csv' }, + ]; + answer({ content: content(), structuredContent: {} }); + const line = nonErrorResultText(await modelCall('attach')); + assert.ok(line.includes('view-cut/report.csv'), line); + answer({ content: content(), structuredContent: {} }); + const out = await script('mcpl--files--attach'); + assert.ok(!out.includes('view-cut'), out); + const link = (JSON.parse(out) as { content: Array> }).content[1]!; + assert.deepEqual(['uri' in link, typeof link.refId], [false, 'string'], out); + }); +}); + +test("an expiry on one uri says nothing of a link to another that shows as the same text, which the host can fetch", { timeout: 30_000 }, async () => { + await withPeers(async ({ answer, script, modelCall }) => { + // Only a `never` is found by the text a link shows; an expiry is about the + // record, which the fetch reads by its raw uri (Ada-1017, room-203 + // #202239). Read by the text, this expiry would make the link one the + // host can't fetch, showing its uri and words saying so. + const uri = 'https://files.example.invalid/expiry-cut/report.csv'; + const content = () => [ + { type: 'resource', uri, name: 'old.csv', expiresAt: '2000-01-01T00:00:00Z' }, + { type: 'resource_link', uri: 'https://files.example.invalid/expiry-cut/\u200Freport.csv', name: 'report.csv' }, + ]; + answer({ content: content(), structuredContent: {} }); + const line = nonErrorResultText(await modelCall('attach')); + assert.ok(!line.includes('expiry-cut'), line); + answer({ content: content(), structuredContent: {} }); + const out = await script('mcpl--files--attach'); + assert.ok(!out.includes('expiry-cut'), out); + const link = (JSON.parse(out) as { content: Array> }).content[1]!; + assert.deepEqual(['uri' in link, 'unfetchable' in link, 'expired' in link, typeof link.refId], [false, false, false, 'string'], out); + }); +}); + +test("an MCPL peer's top-level link, treated once, keeps the record of the uri its result named, whatever sanitizeLabel cut from the uri it shows", async () => { + await withPeers(async ({ answer, script }) => { + // Off the server's origin, so its uri is shown, cut of its bidi override. + // The nested pass leaves the top-level blocks to the treatment above it, + // so the link's object isn't read again as a link to the cut uri. + const raw = 'https://elsewhere.example.invalid/re\u202Eport.csv'; + answer({ content: [{ type: 'resource_link', uri: raw, name: 'report.csv' }], structuredContent: {} }); + const object = JSON.parse(await script('mcpl--files--attach')) as { content: Array> }; + assert.equal(object.content[0]!.uri, 'https://elsewhere.example.invalid/report.csv'); + assert.equal(object.content[0]!.refId, referenceRegistry.findByUri(raw)?.refId); + assert.equal(referenceRegistry.findByUri('https://elsewhere.example.invalid/report.csv'), undefined); + }); +}); + +test("where the treatment fails after the claims walk read the view, a peer's result is one the script can't convert and a module's a failure naming the tool", async () => { + await withPeers(async ({ answer, script, exit }) => { + // The claims walk reads a reference and registers nothing; the treatment + // registers what it shows. A registry that throws a RangeError for one + // nested reference stands in for a treatment that runs out of stack where + // the claims walk, with lighter replacers, didn't. + const uri = 'https://files.example.invalid/treatment-stop/report.csv'; + const real = referenceRegistry.register; + referenceRegistry.register = function (this: typeof referenceRegistry, ...args: Parameters) { + if (args[0].uri === uri) throw new RangeError('Maximum call stack size exceeded'); + return real.apply(this, args); + } as typeof real; + try { + const content = () => [{ type: 'text', text: 'a report', attachment: { type: 'resource', uri, name: 'report.csv' } }]; + answer({ content: content(), structuredContent: {} }); + const fromPeer = JSON.parse(await script('mcpl--files--attach')) as { content: [{ text: string }]; isError: boolean }; + assert.equal(fromPeer.isError, true); + assert.match(fromPeer.content[0].text, /^tool 'mcpl--files--attach' returned a result that can't be converted to JSON/); + // Each route closed its journal on the way out of the failure too. + assert.equal(referenceRegistry.journalCount, 0, 'peer'); + const fromModule = JSON.parse(await exit({ success: true, data: content(), structured: {} }, 'fwd--whole', false)) as { content: [{ text: string }]; isError: boolean }; + assert.equal(fromModule.isError, true); + assert.match(fromModule.content[0].text, /^tool 'fwd--whole' returned a result that can't be converted to JSON/); + assert.equal(referenceRegistry.journalCount, 0, 'module'); + } finally { + referenceRegistry.register = real; + } + }); +}); + +test("a peer's top-level block the classifier rejects shows none of its fields, so a never among them claims nothing in a script's view", { timeout: 30_000 }, async () => { + await withPeers(async ({ answer, script }) => { + const uri = 'https://elsewhere.example.invalid/rejected-fields/plain.csv'; + const content = [ + { type: 'video', uri: 'https://elsewhere.example.invalid/clip.mp4', sealed: { type: 'resource', uri, disposition: 'never' } }, + { type: 'resource', uri, name: 'plain.csv' }, + { type: 'resource_link', uri, name: 'plain.csv' }, + ]; + answer({ content, structuredContent: {} }); + const object = JSON.parse(await script('mcpl--files--attach')) as { content: Array> }; + assert.equal(object.content[0]!.type, 'text', 'the video is its notice'); + assert.equal(object.content[2]!.uri, uri, 'the view shows no never for the uri, so the link carries it, as the model\'s line shows it'); + }); +}); + +test("where the walk of a script view's claims stops, the view isn't written, though the treatment's own walk would pass", async () => { + await withPeers(async ({ answer, script, exit }) => { + // A value that stops the claims walk and not the treatment's: its type + // throws a RangeError the first time it's read, as a stack can run out + // one frame sooner in one walk than in the other, and then reads as data. + const once = () => { + let read = false; + return { get type(): string { if (!read) { read = true; throw new RangeError('Maximum call stack size exceeded'); } return 'note'; } }; + }; + const uri = 'https://elsewhere.example.invalid/stopped-walk/plain.csv'; + const content = () => [{ type: 'text', text: 'one link', deep: once(), attachment: { type: 'resource', uri, disposition: 'never' } }, { type: 'resource_link', uri, name: 'plain.csv' }]; + answer({ content: content(), structuredContent: {} }); + const fromPeer = JSON.parse(await script('mcpl--files--attach')) as { content: [{ text: string }]; isError: boolean }; + assert.equal(fromPeer.isError, true); + assert.match(fromPeer.content[0].text, /^tool 'mcpl--files--attach' returned a result that can't be converted to JSON/); + // A module's result, as the exit receives it. + const fromModule = JSON.parse(await exit({ success: true, data: content(), structured: {} }, 'fwd--whole', false)) as { content: [{ text: string }]; isError: boolean }; + assert.equal(fromModule.isError, true); + assert.match(fromModule.content[0].text, /^tool 'fwd--whole' returned a result that can't be converted to JSON/); + for (const out of [fromPeer, fromModule]) assert.ok(!JSON.stringify(out).includes('elsewhere.example.invalid'), JSON.stringify(out)); + }); +}); + +test("where the walk of a result's claims stops, a script's text route fails naming the tool, as its structured view does, for an object or a list", async () => { + await withPeers(async ({ exit }) => { + // A type that throws a RangeError the first time it's read stops the + // claims walk only: the render's walk would read it as data. + const once = () => { + let read = false; + return { get type(): string { if (!read) { read = true; throw new RangeError('Maximum call stack size exceeded'); } return 'note'; } }; + }; + // No structured content, so the text route; data that isn't content. + // No data and an error that can't be read: still a failure, never the + // model lane's notice (nonErrorResultTextOrThrow). + const noError = { success: true, data: null, get error(): string { throw new Error('zz-unreadable error'); } }; + const failedRead = JSON.parse(await exit(noError as unknown as ToolResult, 'fwd--whole', false)) as { content: [{ text: string }]; isError: boolean }; + assert.equal(failedRead.isError, true, JSON.stringify(failedRead)); + assert.match(failedRead.content[0].text, /^tool 'fwd--whole' returned a result that can't be converted to JSON/); + for (const data of [{ deep: once() }, [{ deep: once() }]]) { + const out = JSON.parse(await exit({ success: true, data }, 'fwd--whole', false)) as { content: [{ text: string }]; isError: boolean }; + assert.equal(out.isError, true, JSON.stringify(out)); + assert.match(out.content[0].text, /^tool 'fwd--whole' returned a result that can't be converted to JSON, so the script can't receive it/); + } + }); +}); + test("a fault in an MCPL peer's round trip other than a RangeError fails the dispatch, as any fault does, with its own message", async () => { await withPeers(async ({ answer, script, observe }) => { const seen = observe(); @@ -630,13 +1253,27 @@ test("an MCPL peer's inline data beside a uri that isn't a string reaches a scri }); }); -test("inline data beside an empty uri reaches a script as sent from a module, as history reads it, and withheld from an MCPL peer", async () => { +test("inline data beside an empty uri reaches a script as sent, from a module and from an MCPL peer alike, and beside a disposition is withheld", async () => { await withPeers(async ({ answer, script, forward }) => { + // An empty uri carries nothing a credential could ride on, so it's no + // uri here, as at any depth (Ada-1017, room-203 #200094). const inline = { type: 'image', data: 'iVBORw0KGgo=', mimeType: 'image/png', uri: '' }; forward.nested = [inline]; assert.deepEqual((JSON.parse(await script('fwd--nested')) as { content: unknown }).content, [inline]); answer({ content: [inline], structuredContent: {} }); - assert.deepEqual((JSON.parse(await script('mcpl--files--attach')) as { content: unknown }).content, [{ type: 'text', text: INLINE_WITHHELD_TEXT }]); + assert.deepEqual((JSON.parse(await script('mcpl--files--attach')) as { content: unknown }).content, [inline]); + // A disposition is a reference's claim, so beside one it's withheld, + // from either, at the top and nested. + const claiming = { ...inline, disposition: 'never' }; + for (const content of [[claiming], [{ type: 'text', text: 'nested', inline: claiming }]]) { + answer({ content, structuredContent: {} }); + const fromPeer = JSON.stringify((JSON.parse(await script('mcpl--files--attach')) as { content: unknown }).content); + forward.nested = content; + const fromModule = JSON.stringify((JSON.parse(await script('fwd--nested')) as { content: unknown }).content); + for (const out of [fromPeer, fromModule]) { + assert.ok(out.includes(INLINE_WITHHELD_TEXT) && !out.includes('iVBORw0KGgo='), out); + } + } }); }); @@ -687,8 +1324,9 @@ test("each shape RFC-005 governs, nested in an MCPL peer's block, reaches a scri const shownByUri = block.byUri as Record; assert.deepEqual([typeof shownByUri.refId, 'uri' in shownByUri], ['string', false]); assert.deepEqual(referenceRegistry.findByUri(byUri.uri)?.serverIds, ['files']); - // An empty uri beside inline data is the classifier's contradiction on a peer's lane. - assert.deepEqual(block.emptyUri, { type: 'text', text: INLINE_WITHHELD_TEXT }); + // An empty uri beside inline data is no uri, on a peer's lane as on a + // module's (Ada-1017, room-203 #200094). + assert.deepEqual(block.emptyUri, emptyUri); // Inline data claiming only a disposition is withheld, and a uri that isn't a string is the classifier's notice. assert.deepEqual(block.sealedInline, { type: 'text', text: INLINE_WITHHELD_TEXT }); assert.equal((block.objectUri as Record).type, 'text'); diff --git a/test/tool-result-blocks.test.ts b/test/tool-result-blocks.test.ts new file mode 100644 index 000000000..b6d2b7c62 --- /dev/null +++ b/test/tool-result-blocks.test.ts @@ -0,0 +1,1742 @@ +/** + * A tool result's content array reaches the agent block by block, on both of + * its copies, the stored text history keeps and the live copy the model reads + * when the result is answered: each block as itself, as a reference's stub, + * or as a marker saying what kind of block it was and why it isn't shown, + * never as the array's JSON, which would carry inline data's base64 and a + * link's uri. A protocol result's content is content whatever its blocks' + * types (ToolResult.dataIsContent, which an MCPL tool's result carries). Any + * other array is content only when every element is a block of one of MCP's + * types, so a list of records of other types stays data, written as JSON. + * + * Run: node --import tsx --test test/tool-result-blocks.test.ts + */ +import { describe, it, test } from 'node:test'; +import assert from 'node:assert/strict'; +import { mkdtempSync, rmSync } from 'node:fs'; +import { createServer } from 'node:http'; +import type { AddressInfo } from 'node:net'; +import { tmpdir } from 'node:os'; +import { join } from 'node:path'; +import { JsStore } from '@animalabs/chronicle'; +import { AgentFramework } from '../src/index.js'; +import * as referencesModule from '../src/mcpl/references.js'; +import { keptWhenWithheld } from '../src/tool-result-guard.js'; +import type { Module, ModuleContext, ToolCall, ToolResult, ToolDefinition } from '../src/types/index.js'; +import { + AUDIO_NO_DATA_TEXT, + AUDIO_NO_MEDIA_TYPE_TEXT, + EXPIRY_REASON, + IMAGE_NO_DATA_TEXT, + IMAGE_NO_MEDIA_TYPE_TEXT, + INLINE_WITHHELD_TEXT, + REFERENCE_LIMITS, + REFERENCE_LINK_TOO_LONG_TEXT, + REFERENCE_NO_LINK_TEXT, + buildReferenceStub, + classifyBlock, + noAddressReason, + referenceBlockForScript, + referenceRegistry, + sanitizeLabel, + unknownBlockText, + type BlockClassification, +} from '../src/mcpl/references.js'; +import { convertBlock } from '../src/mcpl/convert-block.js'; +import { CoalesceError, validateCoalescedContent } from '../src/mcpl/push-coalescer.js'; +import { ReferenceFetcher } from '../src/mcpl/reference-fetcher.js'; +import { checkedToolResult } from '../src/module-registry.js'; +import { parseImagePlaceholders } from '../src/tool-image-ledger.js'; +import { UNCHECKED_RESULT_TEXT, nonErrorResultText, scriptViewClaims, shownToScript, toolResultDataToHistoryString, truncateForHistory } from '../src/tool-result-history.js'; +import { MockMembrane, createMockResponse } from './helpers/mock-membrane.js'; +import { fixture } from './helpers/coalescing-fixture.js'; + +/** What a script's call answers (ToolAnswer, the isolate runner's), as these tests read it. */ +interface ScriptAnswer { ok: boolean; json: string; isError?: boolean } + +/** The text a script reads for a result with no structured content: its one text block. */ +const answerText = (answer: ScriptAnswer): string => (JSON.parse(answer.json) as { content: Array<{ text: string }> }).content[0]!.text; + +const SECRET = 'zz-secret-signature'; +/** 3,000 bytes of audio or image data, base64. */ +const BASE64 = Buffer.alloc(3000, 7).toString('base64'); +const PNG = Buffer.from('89504e470d0a1a0a', 'hex').toString('base64'); +const TEXT = { type: 'text', text: 'zz-text' }; +const IMAGE = { type: 'image', data: PNG, mimeType: 'image/png' }; +const AUDIO = { type: 'audio', data: BASE64, mimeType: 'audio/wav' }; +const AUDIO_TEXT = "[Audio not shown: this host's tool results carry only text and images (audio/wav, ~3KB).]"; +const VIDEO = { type: 'video', uri: `https://zz-files.example/clip.mp4?sig=${SECRET}` }; +const LINK_URI = `https://zz-files.example/report.csv?sig=${SECRET}`; +const LINK = { + type: 'resource_link', uri: LINK_URI, name: 'report.csv', mimeType: 'text/csv', size: 1234, + title: 'zz-title', description: 'zz-description', annotations: { audience: ['user'] }, +}; +const STUB = /\[ref_[^\]]+\] report\.csv — text\/csv, ~1KB claimed — from tool result — /; +/** LINK's stub where the host can't fetch it, as where no server delivered it: its uri may carry a credential (sig). */ +const LINK_STUB = new RegExp(`${STUB.source}it's not tied to any server, so there's nowhere to fetch it from$`); +/** A link whose uri can't carry a credential. */ +const PLAIN_URI = 'https://zz-files.example/report.csv'; +const PLAIN_LINK = { ...LINK, uri: PLAIN_URI }; +/** What PLAIN_LINK shows in a tool result where the host can't fetch it, as where no server delivered it. */ +const PLAIN_LINK_TEXT = `[resource link "report.csv": ${PLAIN_URI}, text/csv; not fetched]`; +/** A content array's claims, by the module's own contentClaims, read off the + * module's namespace so the file still loads against main's src, which + * lacks it, for red-first. */ +const contentClaimsOf = (blocks: unknown[]) => + (referencesModule as unknown as { contentClaims(blocks: unknown[]): referencesModule.ResultClaims }).contentClaims(blocks); +/** Record a server dialed on the links' origin, as the framework records a server it configures. Each + * test that needs it asks, so the file loads against a registry that keeps no addresses. */ +const servingFiles = () => referenceRegistry.setServerAddress('zz-files', { url: 'wss://zz-files.example' }, TEST_HOST); +/** The framework these tests record servers' addresses for, as one does. */ +const TEST_HOST = {}; + +/** Each block the lane can't show as itself, and the marker that stands in its place. */ +const MARKED: Array<[string, unknown, string]> = [ + ['inline audio', AUDIO, AUDIO_TEXT], + ['audio data with no media type', { type: 'audio', data: BASE64 }, AUDIO_NO_MEDIA_TYPE_TEXT], + ['image data with no media type', { type: 'image', data: BASE64 }, IMAGE_NO_MEDIA_TYPE_TEXT], + ['image data with an empty media type', { type: 'image', data: BASE64, mimeType: '' }, IMAGE_NO_MEDIA_TYPE_TEXT], + ['an image whose data is empty', { type: 'image', data: '', mimeType: 'image/png' }, IMAGE_NO_DATA_TEXT], + ['audio whose data is empty', { type: 'audio', data: '', mimeType: 'audio/wav' }, AUDIO_NO_DATA_TEXT], + ['an image with neither data nor a link', { type: 'image' }, IMAGE_NO_DATA_TEXT], + ['a block of a type the host can\'t show', VIDEO, unknownBlockText(VIDEO)], + ['an element that isn\'t a block', 42, unknownBlockText(42)], + ['null', null, unknownBlockText(null)], +]; + +type Live = { content: unknown; isError: boolean }; +const framework = () => Object.create(AgentFramework.prototype) as { + toMembraneToolResult(callId: string, result: ToolResult, maxChars?: number): Live; +}; +/** Both copies of a result: the stored text, and the content the model is sent. */ +function copies(result: ToolResult, maxChars?: number): { stored: string; live: unknown } { + return { stored: nonErrorResultText(result, maxChars), live: framework().toMembraneToolResult('zz-call', result, maxChars).content }; +} + +describe("a protocol result's content (dataIsContent)", () => { + it('shows each block the lane can\'t show as its marker on both copies, never as JSON', () => { + assert.equal(AUDIO_TEXT, `[Audio not shown: this host's tool results carry only text and images (audio/wav, ~3KB).]`); + for (const [what, block, marker] of MARKED) { + const { stored, live } = copies({ success: true, data: [TEXT, block], dataIsContent: true }); + assert.equal(stored, `zz-text\n${marker}`, what); + // With no image, the live copy is the stored text. + assert.equal(live, stored, what); + } + const all = copies({ success: true, data: [TEXT, ...MARKED.map(([, block]) => block)], dataIsContent: true }).stored; + assert.ok(!all.includes(BASE64.slice(0, 64)) && !all.includes(SECRET), all); + }); + + it('keeps an image native in the live copy beside them, and its placeholder in the stored text', () => { + for (const [what, block, marker] of MARKED) { + const { stored, live } = copies({ success: true, data: [IMAGE, TEXT, block], dataIsContent: true }); + assert.equal(stored, `[image: image/png, 9B]\nzz-text\n${marker}`, what); + assert.deepEqual(live, [ + { type: 'image', source: { type: 'base64', data: PNG, mediaType: 'image/png' } }, + { type: 'text', text: 'zz-text' }, + { type: 'text', text: marker }, + ], what); + } + }); + + it('shows a reference as its stub, and inline data claiming a link as withheld, as before', () => { + const { stored, live } = copies({ + success: true, + data: [IMAGE, { type: 'resource', uri: `https://zz-files.example/a.pdf?sig=${SECRET}`, name: 'a.pdf' }, { ...AUDIO, uri: 'https://zz-files.example/a.wav' }], + dataIsContent: true, + }); + assert.match(stored, /^\[image: image\/png, 9B\]\n\[ref_[^\]]+\] a\.pdf — from tool result — [^\n]+\n/); + assert.ok(stored.endsWith(`\n${INLINE_WITHHELD_TEXT}`), stored); + assert.ok(!stored.includes(SECRET), stored); + assert.deepEqual((live as Array<{ type: string; text?: string }>).map((b) => b.text ?? b.type), stored.split('\n').map((line, i) => (i === 0 ? 'image' : line))); + }); + + it('cuts a text block at the cap in the live copy, and shows each marker whole', () => { + const long = { type: 'text', text: 'z'.repeat(500) }; + const { live } = copies({ success: true, data: [IMAGE, long, AUDIO], dataIsContent: true }, 100); + const [, text, marker] = live as Array<{ type: string; text: string }>; + assert.match(text.text, /^z{100}\n\n\[truncated — original was 500 chars\]$/); + assert.equal(marker.text, AUDIO_TEXT); + }); + + it('sends the stored text, cut at the cap, once its links pass the cap together', () => { + // Each link line carries its uri whole. Together past the cap, the live + // copy is the stored text cut at the cap, as main sent a result holding a + // link, so no number of them carries the result past it. + const links = Array.from({ length: 6 }, (_, i) => + ({ type: 'resource_link', uri: `https://zz-files.example/${'p'.repeat(1000)}/${i}.csv`, name: `${i}.csv` })); + const { stored, live } = copies({ success: true, data: [IMAGE, TEXT, ...links], dataIsContent: true }, 2000); + assert.equal(live, stored); + assert.match(stored, /^\[image: image\/png, 9B\]\nzz-text\n\[resource link "0\.csv": /); + assert.match(stored, /\[truncated — original was \d+ chars\]$/); + assert.ok(stored.length < 2100, String(stored.length)); + // Within the cap, they go beside the image as before. + const few = copies({ success: true, data: [IMAGE, TEXT, links[0]], dataIsContent: true }, 2000).live; + assert.ok(Array.isArray(few) && (few as Array<{ type: string }>)[0]!.type === 'image', JSON.stringify(few).slice(0, 200)); + }); + + it('counts markers toward that budget too', () => { + const { stored, live } = copies({ success: true, data: [IMAGE, ...Array.from({ length: 30 }, () => AUDIO)], dataIsContent: true }, 1000); + assert.ok(AUDIO_TEXT.length * 30 > 1000); + assert.equal(live, stored); + assert.match(stored, /\[truncated — original was \d+ chars\]$/); + const few = copies({ success: true, data: [IMAGE, AUDIO, AUDIO], dataIsContent: true }, 1000).live as Array<{ type: string; text?: string }>; + assert.deepEqual(few.map((b) => b.text ?? b.type), ['image', AUDIO_TEXT, AUDIO_TEXT]); + }); + + it("sends references' stubs and withheld lines whole beside an image, whatever their length together, as before", () => { + const refs = Array.from({ length: 20 }, (_, i) => ({ type: 'resource', uri: `https://zz-files.example/r${i}.pdf`, name: `r${i}.pdf` })); + const withheld = { ...AUDIO, uri: 'https://zz-files.example/a.wav' }; + for (const data of [[IMAGE, ...refs], [IMAGE, ...refs.slice(0, 5), ...Array.from({ length: 12 }, () => withheld)]]) { + const result: ToolResult = { success: true, data, dataIsContent: true }; + // The stored text before its cut: each block a line of it. + const whole = nonErrorResultText(result); + assert.ok(whole.length > 1000, String(whole.length)); + const blocks = copies(result, 1000).live as Array<{ type: string; text?: string }>; + assert.ok(Array.isArray(blocks), String(blocks).slice(0, 200)); + assert.deepEqual(blocks.map((b) => b.text ?? b.type), whole.split('\n').map((line, i) => (i === 0 ? 'image' : line))); + assert.match(blocks[1]!.text!, /^\[ref_[^\]]+\] r0\.pdf — from tool result — /); + if (data.includes(withheld)) assert.equal(blocks.at(-1)!.text, INLINE_WITHHELD_TEXT); + } + // At a cap shorter than one stub, the live copy shows the stub whole too: + // it cuts only text blocks (#403). + const stub = nonErrorResultText({ data: [IMAGE, refs[0]], dataIsContent: true }).split('\n')[1]!; + assert.ok(stub.length > 40, stub); + const tiny = copies({ success: true, data: [IMAGE, refs[0]], dataIsContent: true }, 40).live as Array<{ type: string; text?: string }>; + assert.deepEqual(tiny.map((b) => b.text ?? b.type), ['image', stub]); + }); +}); + +describe('an array without dataIsContent', () => { + it('is content when every element is a block of one of MCP\'s types, each block the lane can\'t show as its marker', () => { + assert.equal(toolResultDataToHistoryString([TEXT, AUDIO]), `zz-text\n${AUDIO_TEXT}`); + assert.equal(toolResultDataToHistoryString([TEXT, PLAIN_LINK]), `zz-text\n${PLAIN_LINK_TEXT}`); + assert.equal(toolResultDataToHistoryString([]), ''); + // A list of records that all carry MCP's type names is read as content + // too: this one, with its url, shows a marker where JSON showed it. + assert.equal(toolResultDataToHistoryString([{ type: 'image', url: 'https://zz-images.example/a.png' }]), IMAGE_NO_DATA_TEXT); + }); + + it('is data, written as JSON, when any element is of another type or isn\'t a record', () => { + for (const data of [ + [TEXT, VIDEO], + [TEXT, { type: 'user', id: 1 }], + [{ name: 'a' }, { name: 'b' }], + ['a', 'b'], + [TEXT, 42], + [TEXT, null], + [TEXT, { type: 42 }], + ]) { + assert.equal(toolResultDataToHistoryString(data), JSON.stringify(data)); + } + // So is data that isn't an array, whatever it holds. + assert.equal(toolResultDataToHistoryString({ content: [TEXT] }), JSON.stringify({ content: [TEXT] })); + }); + + it('is content whatever its elements when the result says it is, and only then', () => { + assert.equal(toolResultDataToHistoryString([TEXT, VIDEO], undefined, { dataIsContent: true }), `zz-text\n${unknownBlockText(VIDEO)}`); + assert.equal(nonErrorResultText({ data: [TEXT, VIDEO], dataIsContent: true }), `zz-text\n${unknownBlockText(VIDEO)}`); + assert.equal(nonErrorResultText({ data: [TEXT, VIDEO], dataIsContent: false }), JSON.stringify([TEXT, VIDEO])); + // Data that isn't an array is data, whatever the result says. + assert.equal(nonErrorResultText({ data: { a: 1 }, dataIsContent: true }), '{"a":1}'); + // The flag is read for its truth, as a result's other flags are, on both copies. + const loose = { success: true, data: [IMAGE, TEXT, VIDEO], dataIsContent: 1 as never }; + const { stored, live } = copies(loose); + assert.equal(stored, `[image: image/png, 9B]\nzz-text\n${unknownBlockText(VIDEO)}`); + assert.equal((live as unknown[]).length, 3); + }); +}); + +/** A block's reference testimony, as classifyBlock reads it. */ +function testimonyOf(block: unknown) { + const c = classifyBlock(block); + assert.equal(c.kind, 'reference'); + return (c as Extract).testimony; +} + +describe("MCP's resource_link, a reference (RFC-005 §8.1)", () => { + it("reads its size from MCP's size, and from sizeBytes only where size is absent or null, a size of 0 included", () => { + const sized = (fields: Record) => testimonyOf({ type: 'resource_link', uri: PLAIN_URI, name: 'report.csv', ...fields }).sizeBytes; + assert.equal(sized({ size: 10, sizeBytes: 20 }), 10, 'size decides where both are named'); + assert.equal(sized({ size: 0, sizeBytes: 20 }), 0, 'a size of 0 is a size'); + assert.equal(sized({ sizeBytes: 2048 }), 2048); + assert.equal(sized({ size: null, sizeBytes: 2048 }), 2048); + }); + + it('maps onto the reference record field for field, its size as sizeBytes, with no disposition', () => { + const t = testimonyOf({ ...LINK, disposition: 'never', digest: `sha256:${'A'.repeat(43)}`, expiresAt: '2020-01-01T00:00:00Z' }); + assert.deepEqual([t.uri, t.name, t.mimeType, t.sizeBytes], [LINK_URI, 'report.csv', 'text/csv', 1234]); + assert.deepEqual([t.disposition, t.digest, t.expiresAt], [undefined, undefined, undefined]); + assert.deepEqual(t.rejectedFields, []); + // A size that isn't a byte count is dropped as a field, as a sizeBytes is. + const bad = testimonyOf({ ...LINK, size: -1 }); + assert.equal(bad.sizeBytes, undefined); + assert.deepEqual(bad.rejectedFields, ['sizeBytes']); + assert.deepEqual(classifyBlock({ type: 'resource_link', name: 'zz' }), { kind: 'invalid', shown: REFERENCE_NO_LINK_TEXT }); + assert.deepEqual(classifyBlock({ type: 'resource_link', uri: 'u'.repeat(REFERENCE_LIMITS.uri + 1) }), { kind: 'invalid', shown: REFERENCE_LINK_TOO_LONG_TEXT }); + }); + + it('shows as a stub where the host can fetch it, never its uri, in a content array and at any depth in data', () => { + const uri = `https://zz-files.example/fetchable.csv?sig=${SECRET}`; + const fetchable = { ...LINK, uri }; + servingFiles(); + referenceRegistry.register(testimonyOf(fetchable), 'zz-files'); + const { stored, live } = copies({ success: true, data: [TEXT, fetchable], dataIsContent: true }); + assert.match(stored, new RegExp(`^zz-text\\n${STUB.source}fetch with fetch_reference$`)); + assert.equal(live, stored); + const nested = toolResultDataToHistoryString({ found: [fetchable], page: 2 }); + assert.ok(!nested.includes(SECRET), nested); + assert.match((JSON.parse(nested) as { found: string[] }).found[0]!, STUB); + }); + + it("in a tool result, shows its name, uri and type where the host can't fetch it and the uri can't carry a credential", () => { + // PLAIN_LINK has no server, as a module's link hasn't. + const { stored, live } = copies({ success: true, data: [TEXT, PLAIN_LINK], dataIsContent: true }); + assert.equal(stored, `zz-text\n${PLAIN_LINK_TEXT}`); + assert.equal(live, stored); + assert.deepEqual(JSON.parse(toolResultDataToHistoryString({ found: [PLAIN_LINK], page: 2 })), { found: [PLAIN_LINK_TEXT], page: 2 }); + assert.equal(toolResultDataToHistoryString(PLAIN_LINK), PLAIN_LINK_TEXT); + // So too where a server delivered it, with a scheme the fetcher doesn't + // fetch or off its server's origin. A resource there gets the stub that + // says why, with no uri, as a resource always has. + servingFiles(); + for (const [uri, why] of [ + ['file:///zz/report.csv', 'scheme not allowed: this host fetches only http and https uris'], + ['zz-server://reports/report.csv', 'scheme not allowed: this host fetches only http and https uris'], + ['https://zz-elsewhere.example/watch?v=zz', "third-party origin: it isn't on its server's own address, so this host doesn't fetch it"], + ['https://zz-elsewhere.example/d/zz/edit?usp=sharing#heading=h.zz', "third-party origin: it isn't on its server's own address, so this host doesn't fetch it"], + ['https://zz-elsewhere.example/cb?token_type=bearer&key_id=zz', "third-party origin: it isn't on its server's own address, so this host doesn't fetch it"], + // A credential's name with no value, such as an anchor, carries nothing. + ...['#token', '#policy', '#access_token=', '?token', '?token=&page=2'].map((tail) => + [`https://zz-elsewhere.example/docs/auth${tail}`, "third-party origin: it isn't on its server's own address, so this host doesn't fetch it"]), + // A relative one is read as a relative reference is. + ['zz-reports/report.csv?page=2', "its address doesn't parse as a URL"], + ]) { + const link = { type: 'resource_link', uri, name: 'report.csv' }; + referenceRegistry.register(testimonyOf(link), 'zz-files'); + assert.equal(toolResultDataToHistoryString([TEXT, link]), `zz-text\n[resource link "report.csv": ${uri}; not fetched]`); + const resource = { type: 'resource', uri, name: 'report.csv' }; + const stub = toolResultDataToHistoryString([TEXT, resource]); + assert.ok(stub.endsWith(`report.csv — from tool result — ${why}`) && !stub.includes(uri), stub); + } + }); + + it("gives a script's object the uri a tool result's line shows for the same link, and none where the line shows none", () => { + // One decision for both lanes (shownLinkUri): a script's output comes + // back to the model, so it holds what the model's own line holds. + servingFiles(); + const withheldUri = 'https://zz-elsewhere.example/withheld-for-script.csv'; + referenceRegistry.register(testimonyOf({ type: 'resource', uri: withheldUri, name: 'withheld.csv', disposition: 'never' })); + const thirdParty = "third-party origin: it isn't on its server's own address, so this host doesn't fetch it"; + for (const [what, link, shows, refused] of [ + ["off its server's origin, with no credential", { type: 'resource_link', uri: 'https://zz-elsewhere.example/script-plain.csv', name: 'a.csv' }, true, thirdParty], + ['with a scheme the fetcher refuses', { type: 'resource_link', uri: 'file:///zz/script-plain.csv', name: 'a.csv' }, true, 'scheme not allowed: this host fetches only http and https uris'], + ['that may carry a credential', { type: 'resource_link', uri: `https://zz-elsewhere.example/script-signed.csv?sig=${SECRET}`, name: 'a.csv' }, false, thirdParty], + ['withheld by a reference with disposition never', { type: 'resource_link', uri: withheldUri, name: 'a.csv' }, false, thirdParty], + ['the host can fetch', { type: 'resource_link', uri: 'https://zz-files.example/script-fetchable.csv', name: 'a.csv' }, false, undefined], + ] as const) { + // Delivered by the zz-files server, as a script's MCPL call registers it. + const object = referenceBlockForScript(link, 'zz-files') as { uri?: string; refId: string; unfetchable?: string }; + const line = toolResultDataToHistoryString([TEXT, link], undefined, { dataIsContent: true }).split('\n')[1]!; + assert.equal(line.startsWith('[resource link '), shows, `${what}: ${line}`); + assert.equal(object.uri, shows ? link.uri : undefined, what); + // The object says why a fetch_reference with its refId would be + // refused: where the line is a stub, the reason it gives in place of advising + // fetch_reference, in the same words, and where the line shows the + // uri, which it does only for a link the host can't fetch, the reason + // that fetch is refused, which the line, having no id, doesn't give. + if (!shows) { + const ending = line.slice(line.lastIndexOf(' — ') + 3); + assert.equal(ending, refused ?? 'fetch with fetch_reference', what); + } + assert.equal(object.unfetchable, refused, what); + if (!shows) assert.ok(!line.includes(link.uri) && !JSON.stringify(object).includes(link.uri), what); + assert.match(object.refId, /^ref_/, what); + } + }); + + it("reads a uri in a parameter's value as it reads the link's own, four levels deep, and withholds one nested deeper", () => { + const presigned = `https://zz-bucket.s3.amazonaws.com/k.png?X-Amz-Algorithm=AWS4-HMAC-SHA256&X-Amz-Credential=zz&X-Amz-Signature=${SECRET}`; + const line = (uri: string) => toolResultDataToHistoryString([TEXT, { type: 'resource_link', uri, name: 'report.csv' }]).split('\n')[1]!; + const stub = /^\[ref_[^\]]+\] report\.csv — from tool result — it's not tied to any server, so there's nowhere to fetch it from$/; + // An image proxy's shape: the signed URL percent-encoded as one + // parameter's value, one decode level down, and two. + const once = `https://zz-site.example/_next/image?url=${encodeURIComponent(presigned)}&w=640`; + const twice = `https://zz-camo.example/?u=${encodeURIComponent(once)}`; + // Userinfo in a nested uri, and a credential in a nested uri's fragment. + const userinfo = `https://zz-site.example/go?to=${encodeURIComponent('https://zz-user:zz-pass@zz-elsewhere.example/r.csv')}`; + const fragment = `https://zz-site.example/go#next=${encodeURIComponent(`https://zz-elsewhere.example/cb#access_token=${SECRET}`)}`; + for (const uri of [once, twice, userinfo, fragment]) assert.match(line(uri), stub, uri); + // An ordinary nested uri shows, four levels down too; a fifth level is + // past the bound, and withheld whatever it holds. + const nest = (levels: number) => { + let uri = 'https://zz-files.example/page?v=1'; + for (let i = 0; i < levels; i += 1) uri = `https://zz-site${i}.example/r?next=${encodeURIComponent(uri)}`; + return uri; + }; + for (const levels of [1, 4]) { + const uri = nest(levels); + assert.equal(line(uri), `[resource link "report.csv": ${uri}; not fetched]`, String(levels)); + } + assert.match(line(nest(5)), stub); + // A value that isn't an absolute uri is a value, though it reads as a + // path: the relative reading's fail-closed answer never meets it. + const plain = 'https://zz-files.example/search?q=zz%20elsewhere%20report&page=zz-reports/2'; + assert.equal(line(plain), `[resource link "report.csv": ${plain}; not fetched]`); + }); + + it("reads a relative uri in a parameter's value, and one encoded again inside it, each read and decode a level", () => { + const presigned = `https://zz-bucket.s3.amazonaws.com/k.png?X-Amz-Signature=${SECRET}`; + const line = (uri: string) => toolResultDataToHistoryString([TEXT, { type: 'resource_link', uri, name: 'report.csv' }]).split('\n')[1]!; + const stub = /^\[ref_[^\]]+\] report\.csv — from tool result — it's not tied to any server, so there's nowhere to fetch it from$/; + const twice = (value: string) => encodeURIComponent(encodeURIComponent(value)); + for (const uri of [ + // A relative one with a query, and a protocol-relative one with userinfo. + `https://zz-app.example/login?next=${encodeURIComponent(`/reset?token=${SECRET}`)}`, + `https://zz-proxy.example/img?url=${encodeURIComponent('//zz-user:zz-pass@zz-bucket.example/k')}`, + // Encoded twice inside one parameter, as a redirector leaves it: an + // absolute one, and a protocol-relative one read once decoded. + `https://zz-r.example/go?u=${twice(presigned)}`, + `https://zz-r.example/go?u=${twice('//zz-user:zz-pass@zz-bucket.example/k')}`, + // Decoded leniently, as the parameter itself was: a stray `%` doesn't + // stop the reading, and an `&` the first decode made stays in the value. + `https://zz-r.example/go?u=${twice('//zz-user:zz-pass@zz-bucket.example/k')}%`, + `https://zz-r.example/go?u=${encodeURIComponent(`%2Fpath&x%3Ftoken%3D${SECRET}`)}`, + // A relative value that doesn't resolve can't be read, so it may carry + // one, as a uri that doesn't parse may: here its authority holds a `%`. + `https://zz-r.example/go?u=${twice('//zz-user:zz-pass@zz-host%')}`, + // The backslash form a special scheme reads as `//`. + `https://zz-r.example/go?u=${encodeURIComponent('\\\\zz-user:zz-pass@zz-bucket.example\\k')}`, + // Its form read as the URL parser reads it: led by a tab, a space or a + // C0 control, a protocol-relative value is still one, and one whose + // authority doesn't parse is withheld through its failed resolve. + ...['\t', ' ', '\u0001'].map((lead) => `https://zz-r.example/go?u=${encodeURIComponent(`${lead}//zz-user:zz-pass@zz-bucket.example/k`)}`), + `https://zz-r.example/go?u=${encodeURIComponent('\t//zz-user:zz-pass@zz-host%?x')}`, + // Read against a fixed base, so a uri with no path to resolve against, + // such as a mailto:, still has its parameters' relative values read. + `mailto:zz-help@zz-files.example?body=${encodeURIComponent(`?token=${SECRET}`)}`, + ]) assert.match(line(uri), stub, uri); + // A decode is a level: three nestings below one value encoded twice put + // the fourth at level five, past the bound, where encoded once it's read. + const nest = (levels: number) => { + let uri = 'https://zz-files.example/page?v=1'; + for (let i = 0; i < levels; i += 1) uri = `https://zz-site${i}.example/r?next=${encodeURIComponent(uri)}`; + return uri; + }; + const once = `https://zz-r.example/go?u=${encodeURIComponent(nest(3))}`; + assert.equal(line(once), `[resource link "report.csv": ${once}; not fetched]`); + assert.match(line(`https://zz-r.example/go?u=${twice(nest(3))}`), stub); + // A value percent-encoded past the bound is withheld, uri or not, so the + // check's work stops where its reading does; one level less shows. + let encoded = 'zz plain words'; + for (let i = 0; i < 5; i += 1) encoded = encodeURIComponent(encoded); + const four = `https://zz-files.example/x?q=${encoded}`; + assert.equal(line(four), `[resource link "report.csv": ${four}; not fetched]`); + assert.match(line(`https://zz-files.example/x?q=${encodeURIComponent(encoded)}`), stub); + // Ordinary values still show: a path, values with a colon, a value with + // a space, and a colour, a fragment alone with no parameter in it. + for (const uri of [ + 'https://zz-files.example/x?next=/home', + 'https://zz-files.example/x?lang=en:US&tag=type:image', + 'https://zz-files.example/x?q=zz%20elsewhere%20report', + 'https://zz-files.example/x?q=100%25%20off%', + `https://zz-files.example/x?q=${encodeURIComponent('/a b?c d#e f')}`, + // A relative value with an empty query carries nothing, in a mailto: + // as anywhere. + 'mailto:zz-help@zz-files.example?subject=Question%3F', + 'https://zz-files.example/badge?color=%23ff0000&label=%23zz', + ]) assert.equal(line(uri), `[resource link "report.csv": ${uri}; not fetched]`, uri); + }); + + it("never shows a data: link's uri, which holds its content rather than naming where it lies", () => { + const payload = 'iVBORw0KGgo'.repeat(40); + for (const uri of [`data:image/png;base64,${payload}`, `DATA:text/plain,${payload}`]) { + const link = { type: 'resource_link', uri, name: 'inline.png' }; + const shown = toolResultDataToHistoryString([TEXT, link]); + assert.match(shown, /^zz-text\n\[ref_[^\]]+\] inline\.png — from tool result — it's not tied to any server, so there's nowhere to fetch it from$/, uri.slice(0, 20)); + assert.ok(!shown.includes(payload), shown.slice(0, 200)); + // A script's object for it carries no uri either, as the line shows none. + const object = referenceBlockForScript(link, undefined) as { uri?: string }; + assert.equal(object.uri, undefined); + } + // From a server, the stub says why the fetch is refused, without the scheme. + servingFiles(); + const link = { type: 'resource_link', uri: `data:image/png;base64,${payload}`, name: 'inline.png' }; + referenceRegistry.register(testimonyOf(link), 'zz-files'); + const shown = toolResultDataToHistoryString([TEXT, link]); + assert.ok(shown.endsWith('inline.png — from tool result — scheme not allowed: this host fetches only http and https uris') && !shown.includes(payload), shown.slice(0, 200)); + }); + + it("in a tool result, shows the stub that says why where the host can't fetch it and the uri may carry a credential", () => { + const carriers = [ + // userinfo, whatever the scheme + 'postgres://zz-user:zz-pass@zz-db.example/zz', + 'https://zz-user@zz-elsewhere.example/report.csv', + // AWS's and Google's signed URLs, names compared without letter case + 'https://zz-bucket.s3.amazonaws.com/report.csv?X-Amz-Algorithm=AWS4-HMAC-SHA256&X-Amz-Credential=zz&X-Amz-Signature=zz', + 'https://zz-bucket.s3.amazonaws.com/report.csv?x-amz-security-token=zz', + 'https://storage.googleapis.com/zz-bucket/report.csv?X-Goog-Credential=zz&X-Goog-Signature=zz', + // Azure's shared access signature, CloudFront's, Discord's CDN, B2's + 'https://zz.blob.core.windows.net/c/report.csv?sv=2024-01-01&se=2030-01-01&sp=r&sig=zz', + 'https://zz.cloudfront.net/report.csv?Expires=1&Signature=zz&Key-Pair-Id=zz', + 'https://zz.cloudfront.net/report.csv?Policy=zz', + 'https://cdn.discordapp.com/attachments/1/2/report.csv?ex=1&is=2&hm=zz', + 'https://f000.backblazeb2.com/file/zz/report.csv?Authorization=zz', + // OAuth's, in a fragment or a query, and the generic names + 'https://zz-elsewhere.example/cb#access_token=zz&token_type=bearer', + 'https://zz-elsewhere.example/cb#id_token=zz', + 'https://zz-elsewhere.example/#/reset?token=zz', + 'https://zz-elsewhere.example/cb?v=1;sig=zz', + 'https://zz-elsewhere.example/cb?v=1&token=zz', + // a value that opens with a character read as a separator + 'https://zz-elsewhere.example/cb?token=;zz', + 'https://zz-elsewhere.example/cb#access_token=?zz', + // names compared by their letters and digits alone, without case, and OAuth 1.0a's + ...['accessToken', 'access-token', 'Api-Key', 'x-api-key', 'private-token', 'oauth_token', 'oauth_signature', 'token[]', 'api.key'] + .map((name) => `https://zz-elsewhere.example/cb?${name}=zz`), + // userinfo after a scheme with no slashes, read as an absolute uri + 'http:zz-user:zz-pass@zz-elsewhere.example/report.csv', + // a credential's name, or a scheme before userinfo, split by a + // character the line doesn't show: the check reads the uri as shown + 'https://zz-elsewhere.example/cb?to\u0000ken=zz', + 'ht\u0000tps://zz-user:zz-pass@zz-elsewhere.example/report.csv', + 'https://zz-elsewhere.example/cb?refresh_token=zz', + 'https://zz-elsewhere.example/cb?client_secret=zz', + ...['token', 'auth', 'key', 'apikey', 'api_key', 'secret', 'password', 'signature'].map((name) => `zz-server://reports/report.csv?${name}=zz`), + // a relative one, read as a relative reference is, and one that doesn't + // parse even so, whose parts can't be read + 'zz-reports/report.csv?token=zz', + 'https://zz elsewhere.example/report.csv', + ]; + for (const uri of carriers) { + const link = { type: 'resource_link', uri, name: 'report.csv' }; + const shown = toolResultDataToHistoryString([TEXT, link]); + assert.match(shown, /^zz-text\n\[ref_[^\]]+\] report\.csv — from tool result — it's not tied to any server, so there's nowhere to fetch it from$/, uri); + assert.ok(!shown.includes(uri), shown); + } + const { stored, live } = copies({ success: true, data: [TEXT, LINK], dataIsContent: true }); + assert.match(stored, new RegExp(`^zz-text\\n${LINK_STUB.source}`)); + assert.equal(live, stored); + assert.ok(!stored.includes(SECRET), stored); + // So too where a server delivered it and the origin or the scheme refuses + // the fetch: the stub says why. + servingFiles(); + for (const [uri, why] of [ + [`https://zz-elsewhere.example/report.csv?sig=${SECRET}`, "third-party origin: it isn't on its server's own address, so this host doesn't fetch it"], + [`file:///zz/report.csv?token=${SECRET}`, 'scheme not allowed: this host fetches only http and https uris'], + ]) { + const link = { type: 'resource_link', uri, name: 'report.csv' }; + referenceRegistry.register(testimonyOf(link), 'zz-files'); + const shown = toolResultDataToHistoryString([TEXT, link]); + assert.match(shown, new RegExp(`^zz-text\\n\\[ref_[^\\]]+\\] report\\.csv — from tool result — ${why.replace(/[.*+?^${}()|[\]\\]/g, '\\$&')}$`), uri); + assert.ok(!shown.includes(SECRET), shown); + } + }); + + it("in a tool result, never shows a uri a reference withheld with disposition never, whatever a link to it says", () => { + const uri = 'https://zz-elsewhere.example/withheld.csv'; + referenceRegistry.register(testimonyOf({ type: 'resource', uri, name: 'withheld.csv', disposition: 'never' })); + const shown = toolResultDataToHistoryString([TEXT, { type: 'resource_link', uri, name: 'report.csv' }]); + assert.match(shown, /^zz-text\n\[ref_[^\]]+\] report\.csv — from tool result — it's not tied to any server/); + assert.ok(!shown.includes(uri), shown); + }); + + it('writes its line with no line break a label could carry, so no text in it reads as a slot', () => { + const shown = toolResultDataToHistoryString([TEXT, { + type: 'resource_link', uri: 'https://zz-elsewhere.example/a\u2028[ref_1_abcd] zz — image/png', name: 'zz\u2029[ref_2_abcd] zz — image/png', + }]); + assert.ok(!/[\u2028\u2029]/.test(shown), shown); + assert.deepEqual(parseImagePlaceholders(shown), []); + }); + + it("shows on the push lane as a stub, registered with the server that sent it, and never its uri", () => { + const uri = `https://zz-files.example/pushed.csv?sig=${SECRET}`; + servingFiles(); + const shown = convertBlock({ ...LINK, uri } as never, 'attachment on push event', 'zz-files'); + assert.equal(shown.type, 'text'); + assert.match((shown as { text: string }).text, /^\[ref_[^\]]+\] report\.csv — text\/csv, ~1KB claimed — attachment on push event — fetch with fetch_reference$/); + assert.deepEqual(referenceRegistry.findByUri(uri)?.serverIds, ['zz-files']); + // One the host can't fetch says why, whatever its uri: no lane but a + // tool result's ever showed a link's uri. + const local = { ...LINK, uri: 'file:///zz/pushed.csv' }; + assert.match((convertBlock(local as never, 'attachment on push event', 'zz-files') as { text: string }).text, + /^\[ref_[^\]]+\] report\.csv — text\/csv, ~1KB claimed — attachment on push event — scheme not allowed: this host fetches only http and https uris$/); + assert.deepEqual(convertBlock({ type: 'resource_link' } as never, 'p', 's'), { type: 'text', text: REFERENCE_NO_LINK_TEXT }); + }); + + it("counts as shown in a render result's block check, as a reference does, and an invalid one doesn't", () => { + const check = (content: unknown[]) => validateCoalescedContent(content, { allowEmpty: true, checkBlockShapes: true }); + check([TEXT, LINK]); + assert.throws(() => check([TEXT, { type: 'resource_link', name: 'zz' }]), CoalesceError); + }); + + it("never replaces the claims a reference made for its uri, while a reference's replace a link's", () => { + const uri = `https://zz-files.example/claimed.bin?sig=${SECRET}`; + const digest = `sha256:${'B'.repeat(43)}`; + servingFiles(); + referenceRegistry.register(testimonyOf({ type: 'resource', uri, digest, expiresAt: '2020-01-01T00:00:00Z', name: 'claimed.bin' }), 'zz-files'); + referenceRegistry.register(testimonyOf({ type: 'resource_link', uri, name: 'linked.bin' }), 'zz-files'); + const kept = referenceRegistry.findByUri(uri)!.testimony; + assert.deepEqual([kept.digest, kept.expiresAt, kept.name, kept.fromLink], [digest, '2020-01-01T00:00:00Z', 'claimed.bin', undefined]); + // A link's stub reads its record, so it says what the reference's expiry makes of it. + assert.match((convertBlock({ type: 'resource_link', uri, name: 'linked.bin' } as never, 'p', 'zz-files') as { text: string }).text, /^\[ref_[^\]]+\] linked\.bin — p — the reference is expired$/); + const later = 'https://zz-files.example/linked-first.bin'; + referenceRegistry.register(testimonyOf({ type: 'resource_link', uri: later, name: 'first' }), 'zz-files'); + referenceRegistry.register(testimonyOf({ type: 'resource', uri: later, digest, name: 'second' }), 'zz-files'); + const replaced = referenceRegistry.findByUri(later)!.testimony; + assert.deepEqual([replaced.digest, replaced.name, replaced.fromLink], [digest, 'second', undefined]); + }); + + it('is registered with its server and never fetched eagerly, unless another block names its uri', async () => { + const fw = Object.create(AgentFramework.prototype) as { + autoFetchReferences(content: unknown[], serverId: string, eager: boolean): Promise; + getReferenceFetcher(): unknown; + }; + const fetched: string[] = []; + fw.getReferenceFetcher = () => ({ + isEagerEligible: () => true, + fetch: async (record: { testimony: { uri: string } }) => { fetched.push(record.testimony.uri); return { ok: true }; }, + }); + const u = (name: string) => `https://zz-files.example/eager/${name}`; + await fw.autoFetchReferences([ + { type: 'resource_link', uri: u('linked') }, + { type: 'resource_link', uri: u('linked-then-referenced') }, + { type: 'resource', uri: u('linked-then-referenced') }, + { type: 'resource', uri: u('referenced-then-linked') }, + { type: 'resource_link', uri: u('referenced-then-linked') }, + { type: 'resource', uri: u('referenced') }, + ], 'zz-eager-server', true); + assert.deepEqual(fetched.sort(), [u('linked-then-referenced'), u('referenced'), u('referenced-then-linked')].sort()); + assert.deepEqual(referenceRegistry.findByUri(u('linked'))?.serverIds, ['zz-eager-server']); + }); + + it("isn't fetched for a script on demand, where a reference is", async () => { + const fw = Object.create(AgentFramework.prototype) as { + getReferenceFetcher(): unknown; + scriptResultAnswer(result: ToolResult, toolName: string, toServer: boolean): Promise; + }; + const reference = { + type: 'resource', uri: `https://zz-files.example/wanted.pdf?sig=${SECRET}`, name: 'wanted.pdf', + digest: `sha256:${'C'.repeat(43)}`, expiresAt: '2099-01-01T00:00:00Z', + }; + // A link to the same uri came first, so its record holds the link's + // claims until the reference's replace them. + referenceRegistry.register(testimonyOf({ type: 'resource_link', uri: reference.uri, name: 'linked.pdf' }), 'zz-files'); + const fetched: string[] = []; + const claims: unknown[] = []; + fw.getReferenceFetcher = () => ({ + fetch: async (record: { testimony: { uri: string; name?: string; digest?: string; expiresAt?: string } }) => { + fetched.push(record.testimony.uri); + claims.push([record.testimony.name, record.testimony.digest, record.testimony.expiresAt]); + return { ok: false, error: 'zz-refused' }; + }, + }); + const error = console.error; + console.error = () => {}; + let result: ScriptAnswer; + try { + // A module's result: no dispatch sent the call to a server. + result = await fw.scriptResultAnswer({ success: true, data: [TEXT, LINK, reference, VIDEO], dataIsContent: true }, 'zz_tool', false); + } finally { + console.error = error; + } + assert.equal(result.isError, false); + const text = answerText(result); + assert.deepEqual(fetched, [reference.uri]); + assert.deepEqual(claims, [['wanted.pdf', reference.digest, reference.expiresAt]]); + // The script reads the text history stores for a protocol result's + // content: the link, which no server delivered, as the stub that says so. + assert.match(text, new RegExp(`^zz-text\\n${LINK_STUB.source.slice(0, -1)}\\n\\[ref_[^\\]]+\\] wanted\\.pdf — from tool result[^\\n]+\\n`)); + assert.ok(text.endsWith(`\n${unknownBlockText(VIDEO)}`), text); + assert.ok(!text.includes(SECRET), text); + }); +}); + +describe('what a stub says of a fetch, which the fetch then does', () => { + // Servers as a framework records them: one dialed on the references' + // origin, one with no address, as a stdio server; zz-gone is unknown. + const serving = () => { + referenceRegistry.setServerAddress('zz-dialed', { url: 'wss://zz-refs.example' }, TEST_HOST); + referenceRegistry.setServerAddress('zz-stdio', 'no-address', TEST_HOST); + }; + /** The fetcher of a framework configured as `serving` records it. */ + const servedFetcher = () => new ReferenceFetcher( + (serverId) => (serverId === 'zz-dialed' ? { url: 'wss://zz-refs.example' } : serverId === 'zz-stdio' ? 'no-address' : undefined), + async () => null, + ); + const record = (name: string, serverIds: string[], extra: Record = {}) => { + const t = testimonyOf({ type: 'resource', uri: `https://zz-refs.example/${name}?sig=${SECRET}`, name, ...extra }); + let r = referenceRegistry.register(t); + for (const id of serverIds) r = referenceRegistry.register(t, id); + return r; + }; + // Made when a test first asks, so the file loads against a registry that + // keeps no addresses. + let made: Array<[string, ReturnType, string | null]> | undefined; + const cases = () => made ??= (serving(), [ + ['dialed on its origin', record('dialed.bin', ['zz-dialed']), null], + ['from no server', record('unbound.bin', []), "it's not tied to any server, so there's nowhere to fetch it from"], + ['from an unknown server', record('gone.bin', ['zz-gone']), 'this host no longer has the "zz-gone" server it came from'], + ['from a server with no address', record('stdio.bin', ['zz-stdio']), noAddressReason('zz-stdio')], + ['from a server with no address, and one dialed on its origin', record('both.bin', ['zz-stdio', 'zz-dialed']), null], + ['dialed on its origin, but expired', record('expired.bin', ['zz-dialed'], { expiresAt: '2020-01-01T00:00:00Z' }), 'the reference is expired'], + ]); + + it("ends a resource's stub with why a fetch would be refused, or the fetch it offers", () => { + for (const [what, r, why] of cases()) { + const stub = buildReferenceStub(r.testimony, 'zz'); + assert.ok(stub.endsWith(`— zz — ${why ?? 'fetch with fetch_reference'}`) && !stub.includes(SECRET), `${what}: ${stub}`); + } + }); + + it("agrees with the fetcher's own check, which reads its framework's configs", () => { + const fetcher = servedFetcher(); + for (const [what, r, why] of cases()) { + const refusal = fetcher.refusal(r); + assert.equal(refusal === null, why === null, `${what}: ${refusal?.error}`); + if (refusal && why) assert.equal(refusal.error, why, what); + } + }); + + it("is the framework's fetcher's check too, and the framework records each server's address for the stub", () => { + const fw = Object.create(AgentFramework.prototype) as { + mcplServerConfigs: Map; + setMcplServerConfig(config: unknown): void; + getReferenceFetcher(): ReferenceFetcher; + }; + fw.mcplServerConfigs = new Map(); + fw.setMcplServerConfig({ id: 'zz-wired', url: 'wss://zz-refs.example' }); + assert.deepEqual(referenceRegistry.serverAddress('zz-wired'), { url: 'wss://zz-refs.example' }); + // A server the host spawns has no address, whatever url its config holds. + fw.setMcplServerConfig({ id: 'zz-spawned', command: 'zz-server', url: 'wss://zz-refs.example' }); + assert.equal(referenceRegistry.serverAddress('zz-spawned'), 'no-address'); + const wired = record('wired.bin', ['zz-wired']); + const spawned = record('spawned.bin', ['zz-spawned']); + const fetcher = fw.getReferenceFetcher(); + assert.equal(fetcher.refusal(wired), null); + assert.ok(buildReferenceStub(wired.testimony, 'zz').endsWith('— zz — fetch with fetch_reference')); + assert.equal(fetcher.refusal(spawned)?.error, noAddressReason('zz-spawned')); + assert.ok(buildReferenceStub(spawned.testimony, 'zz').endsWith(`— zz — ${noAddressReason('zz-spawned')}`)); + }); + + it("never sends a framework's token to another framework's origin for one id, and a removal takes back only its own address", () => { + const framework = () => { + const fw = Object.create(AgentFramework.prototype) as { + mcplServerConfigs: Map; + setMcplServerConfig(config: unknown): void; + getReferenceFetcher(): ReferenceFetcher; + }; + fw.mcplServerConfigs = new Map(); + return fw; + }; + const earlier = framework(); + const later = framework(); + earlier.setMcplServerConfig({ id: 'zz-shared', url: 'wss://zz-earlier.example', token: 'zz-earlier-token' }); + later.setMcplServerConfig({ id: 'zz-shared', url: 'wss://zz-later.example', token: 'zz-later-token' }); + // The later framework's entry is in force for the stub. + assert.deepEqual(referenceRegistry.serverAddress('zz-shared'), { url: 'wss://zz-later.example' }); + const onLater = referenceRegistry.register(testimonyOf({ type: 'resource', uri: 'https://zz-later.example/shared.bin', name: 'shared.bin' }), 'zz-shared'); + const onEarlier = referenceRegistry.register(testimonyOf({ type: 'resource', uri: 'https://zz-earlier.example/shared.bin', name: 'shared.bin' }), 'zz-shared'); + // Each fetcher checks a uri against its own config's address, whose token + // it carries: the earlier framework's never dials the later one's origin. + assert.equal(earlier.getReferenceFetcher().refusal(onLater)!.error, + 'it points to https://zz-later.example, not to the "zz-shared" server\'s own address (wss://zz-earlier.example), and this host fetches only from the server itself'); + assert.equal(earlier.getReferenceFetcher().refusal(onEarlier), null); + assert.equal(later.getReferenceFetcher().refusal(onLater), null); + assert.match(later.getReferenceFetcher().refusal(onEarlier)!.error, /^it points to https:\/\/zz-earlier\.example, not to the "zz-shared" server's own address \(wss:\/\/zz-later\.example\)/); + // The later framework's removal takes back only its own entry, and the + // earlier one's is in force again. + referenceRegistry.forgetServerAddress('zz-shared', later); + assert.deepEqual(referenceRegistry.serverAddress('zz-shared'), { url: 'wss://zz-earlier.example' }); + assert.ok(buildReferenceStub(onEarlier.testimony, 'zz').endsWith('— zz — fetch with fetch_reference')); + referenceRegistry.forgetServerAddress('zz-shared', earlier); + assert.equal(referenceRegistry.serverAddress('zz-shared'), undefined); + }); + + it("carries each framework's own token to its own server's origin, and to no other, when it fetches", async (t) => { + const heard: Array<[string, string | undefined]> = []; + const serve = async (name: string) => { + const server = createServer((req, res) => { + heard.push([name, req.headers.authorization]); + res.writeHead(200, { 'content-type': 'application/octet-stream' }); + res.end(Buffer.from('zz-bytes')); + }); + await new Promise((resolve) => server.listen(0, '127.0.0.1', resolve)); + t.after(() => new Promise((resolve) => server.close(() => resolve()))); + return `127.0.0.1:${(server.address() as AddressInfo).port}`; + }; + const [hostA, hostB] = [await serve('a'), await serve('b')]; + const framework = (url: string, token: string) => { + const fw = Object.create(AgentFramework.prototype) as { + setMcplServerConfig(config: unknown): void; + getReferenceFetcher(): ReferenceFetcher; + }; + Object.assign(fw, { + mcplServerConfigs: new Map(), + getWorkspaceModule: () => ({ writeBinary: async () => ({ success: true }) }), + firstWritableMountName: () => 'scratch', + emitTrace: () => {}, + }); + fw.setMcplServerConfig({ id: 'zz-shared-fetch', url, token }); + return fw; + }; + const a = framework(`ws://${hostA}`, 'zz-token-a'); + const b = framework(`ws://${hostB}`, 'zz-token-b'); + const onA = referenceRegistry.register(testimonyOf({ type: 'resource', uri: `http://${hostA}/a.bin`, name: 'a.bin' }), 'zz-shared-fetch'); + const onB = referenceRegistry.register(testimonyOf({ type: 'resource', uri: `http://${hostB}/b.bin`, name: 'b.bin' }), 'zz-shared-fetch'); + // Each framework refuses the other's origin without dialing it, and + // dials its own with its own token. + assert.equal((await b.getReferenceFetcher().fetch(onA)).ok, false); + assert.equal((await a.getReferenceFetcher().fetch(onB)).ok, false); + assert.equal((await a.getReferenceFetcher().fetch(onA)).ok, true); + assert.equal((await b.getReferenceFetcher().fetch(onB)).ok, true); + assert.deepEqual(heard, [['a', 'Bearer zz-token-a'], ['b', 'Bearer zz-token-b']]); + referenceRegistry.forgetServerAddress('zz-shared-fetch', a); + referenceRegistry.forgetServerAddress('zz-shared-fetch', b); + }); + + it("words a server's id as a label in a stub, and a third-party origin and a refused scheme without the uri's parts", () => { + serving(); + const weird = `zz-\u202E${'x'.repeat(300)}`; + const r = record('weird.bin', [weird]); + const stub = buildReferenceStub(r.testimony, 'zz'); + assert.ok(stub.includes('this host no longer has the "zz-xxx') && !stub.includes('\u202E') && stub.length < 400, stub); + referenceRegistry.setServerAddress('zz-other', { url: 'wss://zz-other.example' }, TEST_HOST); + const offOrigin = record('elsewhere.bin', ['zz-other']); + assert.ok(buildReferenceStub(offOrigin.testimony, 'zz').endsWith("— zz — third-party origin: it isn't on its server's own address, so this host doesn't fetch it")); + const fetcher = new ReferenceFetcher(() => ({ url: 'wss://zz-other.example' }), async () => null); + assert.equal(fetcher.refusal(offOrigin)!.error, + 'it points to https://zz-refs.example, not to the "zz-other" server\'s own address (wss://zz-other.example), and this host fetches only from the server itself'); + // A scheme the fetcher doesn't fetch: the stub doesn't name it, and the + // fetch's refusal names it bounded as a label is. + const scheme = `zz${'s'.repeat(300)}`; + const t = testimonyOf({ type: 'resource', uri: `${scheme}://zz-refs.example/long.bin`, name: 'long.bin' }); + const long = referenceRegistry.register(t, 'zz-dialed'); + assert.ok(buildReferenceStub(t, 'zz').endsWith('— zz — scheme not allowed: this host fetches only http and https uris')); + assert.equal(new ReferenceFetcher(() => ({ url: 'wss://zz-refs.example' }), async () => null).refusal(long)?.error, + `its address uses ${sanitizeLabel(`${scheme}:`, 120)}, and this host fetches only http: and https: addresses`); + }); + + it('refuses a reference through a server whose url does not parse, saying so', () => { + referenceRegistry.setServerAddress('zz-garbled', { url: 'zz not a url' }, TEST_HOST); + const garbled = referenceRegistry.register(testimonyOf({ type: 'resource', uri: 'https://zz-refs.example/garbled.bin', name: 'garbled.bin' }), 'zz-garbled'); + assert.ok(buildReferenceStub(garbled.testimony, 'zz').endsWith('— zz — the "zz-garbled" server\'s address doesn\'t parse as a URL, so there\'s nowhere to fetch it from')); + }); +}); + +describe('an empty link beside inline data is no link', () => { + const image = { ...IMAGE, uri: '' }; + const audio = { ...AUDIO, uri: '' }; + + it('so the data is shown as any inline data is, on the push lane and the tool-result lane', () => { + assert.deepEqual(classifyBlock(image), { kind: 'inline', contradiction: false }); + assert.deepEqual(classifyBlock(audio), { kind: 'inline', contradiction: false }); + assert.deepEqual(convertBlock(image as never, 'p', 's'), { type: 'image', source: { type: 'base64', data: PNG, mediaType: 'image/png' } }); + assert.deepEqual(convertBlock(audio as never, 'p', 's'), { type: 'audio', source: { type: 'base64', data: BASE64, mediaType: 'audio/wav' } }); + const { stored, live } = copies({ success: true, data: [image, audio], dataIsContent: true }); + assert.equal(stored, `[image: image/png, 9B]\n${AUDIO_TEXT}`); + assert.deepEqual(live, [{ type: 'image', source: { type: 'base64', data: PNG, mediaType: 'image/png' } }, { type: 'text', text: AUDIO_TEXT }]); + }); + + it('while a link that isn\'t empty still withholds it, on both lanes', () => { + const linked = { ...AUDIO, uri: 'https://zz-files.example/a.wav' }; + assert.deepEqual(classifyBlock(linked), { kind: 'inline', contradiction: true }); + assert.deepEqual(convertBlock(linked as never, 'p', 's'), { type: 'text', text: INLINE_WITHHELD_TEXT }); + assert.equal(nonErrorResultText({ data: [TEXT, linked], dataIsContent: true }), `zz-text\n${INLINE_WITHHELD_TEXT}`); + }); +}); + +describe('dataIsContent travels with the result', () => { + it('is kept as its truth when a module\'s result is checked, read once', () => { + const call = { id: 'zz', name: 'zz_tool', input: {} }; + assert.equal(checkedToolResult(call, { success: true, data: [TEXT], dataIsContent: 1 }).result.dataIsContent, true); + assert.equal(checkedToolResult(call, { success: true, data: [TEXT], dataIsContent: 0 }).result.dataIsContent, false); + assert.equal('dataIsContent' in checkedToolResult(call, { success: true, data: [TEXT] }).result, false); + let reads = 0; + class Answer { + success = true; + data = [TEXT, VIDEO]; + get dataIsContent() { reads += 1; return true; } + } + assert.equal(checkedToolResult(call, new Answer()).result.dataIsContent, true); + assert.equal(reads, 1); + }); + + it("through a model's round, an image past the stored copy's cut keeps no slot there when the live copy is the stored text", async () => { + // With the links before the image passing the cap, the model reads the + // stored text, cut before the image's placeholder: no slot of it is kept + // past the cut, since the model never saw it. + const links = Array.from({ length: 30 }, (_, i) => + ({ type: 'resource_link', uri: `https://zz-files.example/${'p'.repeat(1000)}/${i}.csv`, name: `${i}.csv` })); + const { wire, stored } = await answerModelCall({ success: true, data: [TEXT, ...links, IMAGE], dataIsContent: true }); + assert.equal(wire, stored); + const [text] = JSON.parse(stored) as string[]; + assert.ok(!text!.includes('fell past the cut'), text!.slice(-400)); + assert.ok(!/\[image: image\/png/.test(text!), text!.slice(-400)); + }); + + it("through a model's round, sends the stored copy when its links pass the cap together", async () => { + const links = Array.from({ length: 30 }, (_, i) => + ({ type: 'resource_link', uri: `https://zz-files.example/${'p'.repeat(1000)}/${i}.csv`, name: `${i}.csv` })); + const { wire, stored } = await answerModelCall({ success: true, data: [IMAGE, TEXT, ...links], dataIsContent: true }); + assert.equal(wire, stored); + const [text] = JSON.parse(stored) as string[]; + // The image's placeholder keeps its ledger ref, and the text is cut at the cap. + assert.match(text!, /^\[image: image\/png, 9B, ref img_[a-z0-9]+_\d+\]\nzz-text\n/); + assert.ok(text!.length < 24_000 + 500, String(text!.length)); + }); + + it("through a model's round, from a module's result: both copies show its blocks", async () => { + for (const dataIsContent of [true, undefined]) { + const { wire, stored } = await answerModelCall({ success: true, data: [IMAGE, TEXT, AUDIO, VIDEO], ...(dataIsContent ? { dataIsContent } : {}) }); + if (dataIsContent) { + // The image's placeholder carries the ref the ledger retained it under. + assert.equal(stored.replace(/ref img_[A-Za-z0-9]+_\d+/, 'ref IMG'), JSON.stringify([`[image: image/png, 9B, ref IMG]\nzz-text\n${AUDIO_TEXT}\n${unknownBlockText(VIDEO)}`])); + assert.equal(wire, JSON.stringify([[ + { type: 'image', source: { type: 'base64', data: PNG, mediaType: 'image/png' } }, + { type: 'text', text: 'zz-text' }, + { type: 'text', text: AUDIO_TEXT }, + { type: 'text', text: unknownBlockText(VIDEO) }, + ]])); + } else { + // Without it, the block of a type MCP doesn't define makes the array data. + assert.equal(stored, JSON.stringify([JSON.stringify([IMAGE, TEXT, AUDIO, VIDEO])])); + assert.equal(wire, stored); + } + } + }); + + test("an MCPL tool's result carries it, to the model's call, an in-process caller and the puppet's stored pair", { timeout: 30_000 }, async (t) => { + // The fixture's server is dialed at ws://127.0.0.1:, so a link on + // that origin can be fetched, and one elsewhere can't. + let BLOCKS: unknown[] = []; + const f = await fixture({ + tools: [{ name: 'lookup', description: 'Look something up.', inputSchema: { type: 'object' } }], + toolsCall: (params) => ((params.arguments as { case: string }).case === 'text' + ? { content: [TEXT, { type: 'text', text: 'zz-more' }] } + : { content: BLOCKS }), + }); + t.after(f.close); + const elsewhere = `https://zz-files.example/mcpl.csv?sig=${SECRET}`; + const plain = 'https://zz-files.example/mcpl.csv'; + const own = `http://127.0.0.1:${f.port}/mcpl.csv?sig=${SECRET}`; + BLOCKS = [TEXT, VIDEO, AUDIO, { ...LINK, uri: own }, { ...LINK, uri: elsewhere, name: 'elsewhere.csv' }, { ...LINK, uri: plain, name: 'plain.csv' }]; + assert.deepEqual(referenceRegistry.serverAddress('editor'), { url: `ws://127.0.0.1:${f.port}` }); + const internals = f.framework as unknown as { pushEvent(event: unknown): void; dispatchToolCall(agent: string, call: unknown): void }; + const viaModel = (id: string, c: string) => new Promise((resolve) => { + const original = internals.pushEvent; + internals.pushEvent = (event: unknown) => { + const e = event as { type: string; callId?: string; result?: ToolResult }; + if (e.type === 'tool-result' && e.callId === id) { internals.pushEvent = original; resolve(e.result!); } + else original.call(internals, event); + }; + internals.dispatchToolCall('agent', { id, name: 'mcpl--editor--lookup', input: { case: c } }); + }); + // The puppet stores the pair from a checked copy of the in-process + // result. It runs first, while the agent is idle. + const { toolUseId } = await f.framework.puppetToolCall('agent', 'mcpl--editor--lookup', { case: 'blocks' }); + const stored = f.framework.getAgent('agent')!.getContextManager().getAllMessages() + .flatMap((m) => m.content as unknown as Array>) + .find((b) => b.type === 'tool_result' && b.toolUseId === toolUseId); + const model = await viaModel('zz-model', 'blocks'); + assert.equal(model.dataIsContent, true); + assert.deepEqual(model.data, BLOCKS); + const expected = nonErrorResultText(model); + // The server's own link fetches. Another origin's, which may carry a + // credential, says why it can't be fetched, and the plain one shows its uri. + assert.match(expected, new RegExp( + `^zz-text\\n\\[Content not shown: this host can't display "video" content\\.\\]\\n\\[Audio not shown: [^\\n]+\\n${STUB.source}fetch with fetch_reference\\n` + + `\\[ref_[^\\]]+\\] elsewhere\\.csv — text/csv, ~1KB claimed — from tool result — third-party origin: it isn't on its server's own address, so this host doesn't fetch it\\n` + + `\\[resource link "plain\\.csv": https://zz-files\\.example/mcpl\\.csv, text/csv; not fetched\\]$`)); + assert.ok(!expected.includes(SECRET), expected); + assert.deepEqual(referenceRegistry.findByUri(own)?.serverIds, ['editor']); + // The puppet's stored pair holds what a model's call stores. + assert.equal(stored?.content, expected); + // A text-only result is still its joined text, and carries nothing more. + const text = await viaModel('zz-text', 'text'); + assert.deepEqual([text.data, text.dataIsContent], ['zz-text\nzz-more', undefined]); + const inProcess = await f.framework.executeToolCall({ id: 'zz-x', name: 'mcpl--editor--lookup', input: { case: 'blocks' } }); + assert.deepEqual([inProcess.data, inProcess.dataIsContent], [BLOCKS, true]); + // A server the framework removes is forgotten, and a link of its then + // can't be fetched. + await f.framework.disconnectMcplServer('editor'); + assert.equal(referenceRegistry.serverAddress('editor'), undefined); + assert.match(nonErrorResultText(model), new RegExp(`\\n${STUB.source}this host no longer has the "editor" server it came from\\n`)); + }); +}); + +test("a framework's stop takes its servers' addresses out of the registry, and its start records them again", async () => { + // On a store of its own, so it can start again after it stops. + const dir = mkdtempSync(join(tmpdir(), 'tool-result-blocks-lifecycle-')); + const store = JsStore.openOrCreate({ path: join(dir, 'store') }); + const framework = await AgentFramework.create({ + store, + membrane: new MockMembrane().asMembrane(), + agents: [{ name: 'agent', model: 'test-model', systemPrompt: 'test', allowedTools: 'all' }], + modules: [], + syncIntervalMs: 0, + maintenanceIntervalMs: 0, + }); + try { + (framework as unknown as { setMcplServerConfig(config: unknown): void }) + .setMcplServerConfig({ id: 'zz-lifecycle', url: 'wss://zz-lifecycle.example' }); + framework.start(); + assert.deepEqual(referenceRegistry.serverAddress('zz-lifecycle'), { url: 'wss://zz-lifecycle.example' }); + await framework.stop(); + assert.equal(referenceRegistry.serverAddress('zz-lifecycle'), undefined); + framework.start(); + assert.deepEqual(referenceRegistry.serverAddress('zz-lifecycle'), { url: 'wss://zz-lifecycle.example' }); + } finally { + await framework.stop(); + store.close(); + rmSync(dir, { recursive: true, force: true }); + } +}); + +/** A module whose one tool answers with the result it was given. */ +class AnswerModule implements Module { + readonly name = 'content'; + constructor(private readonly answer: ToolResult) {} + async start(_ctx: ModuleContext): Promise {} + async stop(): Promise {} + getTools(): ToolDefinition[] { + return [{ name: 'get', description: 'Answer with content.', inputSchema: { type: 'object', properties: {} } }]; + } + async handleToolCall(_call: ToolCall): Promise { + return this.answer; + } + async onProcess(): Promise> { + return {}; + } +} + +/** The content of the tool result the model received for one call, and of what the agent's history stored. */ +async function answerModelCall(answer: ToolResult): Promise<{ wire: string; stored: string }> { + const dir = mkdtempSync(join(tmpdir(), 'tool-result-blocks-')); + const membrane = new MockMembrane(); + const framework = await AgentFramework.create({ + storePath: join(dir, 'store'), + membrane: membrane.asMembrane(), + agents: [{ name: 'agent', model: 'test-model', systemPrompt: 'test', allowedTools: 'all' }], + modules: [new AnswerModule(answer)], + syncIntervalMs: 0, + maintenanceIntervalMs: 0, + }); + try { + membrane.pushResponse(createMockResponse([{ type: 'tool_use', id: 'zz-call', name: 'content--get', input: {} }], 'tool_use')); + membrane.pushResponse(createMockResponse([{ type: 'text', text: 'zz-done' }])); + framework.start(); + (framework as unknown as { pendingRequests: unknown[] }).pendingRequests.push({ + agentName: 'agent', reason: 'external-message', source: 'test', timestamp: Date.now(), + }); + const stream = () => membrane.lastStream as unknown as { receivedToolResults: unknown[][] } | undefined; + const deadline = Date.now() + 5_000; + while ((stream()?.receivedToolResults.length ?? 0) < 1 && Date.now() < deadline) await new Promise((r) => setTimeout(r, 25)); + const received = stream()?.receivedToolResults[0]; + assert.ok(received, 'the model was answered'); + const stored = framework.getAgent('agent')!.getContextManager().getAllMessages() + .flatMap((message) => message.content) + .filter((block) => (block as { type?: string }).type === 'tool_result'); + assert.equal(stored.length, 1, 'the history holds the tool result'); + const contentOf = (results: unknown[]) => JSON.stringify(results.map((r) => (r as { content?: unknown }).content)); + return { wire: contentOf(received), stored: contentOf(stored) }; + } finally { + await framework.stop(); + rmSync(dir, { recursive: true, force: true }); + } +} + +describe('one answer for every block of a result', () => { + it("withholds a link's uri wherever a reference in the same result marks it never, whatever the blocks' order", () => { + const uri = 'https://zz-elsewhere.example/order-never.csv'; + const link = { type: 'resource_link', uri, name: 'order.csv' }; + const never = { type: 'resource', uri, name: 'order.csv', disposition: 'never' }; + const plain = { type: 'resource', uri, name: 'order.csv' }; + const STUB = /^\[ref_[^\]]+\] order\.csv — from tool result — /; + for (const order of [[link, never, plain], [never, link, plain], [never, plain, link], [plain, never, link], [plain, link, never]]) { + const what = order.map((b) => (b === link ? 'link' : b === never ? 'never' : 'plain')).join(', '); + const lines = toolResultDataToHistoryString([TEXT, ...order]).split('\n'); + // Each block is there, the link as a stub with no uri. + assert.equal(lines.length, 4, `${what}: ${lines.join('\n')}`); + assert.ok(lines.slice(1).every((line) => STUB.test(line)), `${what}: ${lines.join('\n')}`); + // At any depth in data too. + const deep = JSON.parse(toolResultDataToHistoryString({ first: order.slice(0, 1), then: { rest: order.slice(1) } })) as + { first: string[]; then: { rest: string[] } }; + assert.ok([...deep.first, ...deep.then.rest].every((stub) => STUB.test(stub)), `${what}: ${JSON.stringify(deep)}`); + // And in a script's object, with the array's claims. + const claims = contentClaimsOf(order); + for (const block of order) { + const object = referenceBlockForScript(block, 'zz-files', claims) as { type: string; uri?: string; refId?: string }; + assert.equal(object.type, block.type, what); + assert.match(String(object.refId), /^ref_/, what); + assert.equal(object.uri, undefined, what); + } + } + }); + + it("reads an expiry any reference in the same result gives a uri for every block that names it, whatever their order", () => { + servingFiles(); + const uri = 'https://zz-files.example/order-expired.csv'; + const link = { type: 'resource_link', uri, name: 'order.csv' }; + const expired = { type: 'resource', uri, name: 'order.csv', expiresAt: '2001-01-01T00:00:00Z' }; + const fresh = { type: 'resource', uri, name: 'order.csv', expiresAt: '2099-01-01T00:00:00Z' }; + referenceRegistry.register(testimonyOf(link), 'zz-files'); + const texts = new Set(); + for (const order of [[fresh, link, expired], [expired, link, fresh], [link, fresh, expired]]) { + const lines = toolResultDataToHistoryString(order).split('\n'); + // The link, which the expiry makes unfetchable, shows its line, and + // each reference's stub says the reference is expired. + assert.equal(lines.filter((line) => line === `[resource link "order.csv": ${uri}; not fetched]`).length, 1, lines.join('\n')); + assert.equal(lines.filter((line) => line.endsWith('— the reference is expired')).length, 2, lines.join('\n')); + texts.add([...lines].sort().join('\n').replace(/ref_[^\]]+/g, 'ref')); + // And a script's object for each reference, with the array's claims. + const claims = contentClaimsOf(order); + for (const block of [fresh, expired]) { + const object = referenceBlockForScript(block, 'zz-files', claims) as { expired?: string; unfetchable?: string }; + assert.equal(object.expired, EXPIRY_REASON.passed); + assert.equal(object.unfetchable, `the reference is ${EXPIRY_REASON.passed}`); + } + } + assert.equal(texts.size, 1); + // At any depth in data too. + const deep = JSON.parse(toolResultDataToHistoryString({ first: [link], then: { ref: expired } })) as { first: string[]; then: { ref: string } }; + assert.deepEqual(deep.first, [`[resource link "order.csv": ${uri}; not fetched]`]); + assert.ok(deep.then.ref.endsWith('— the reference is expired'), deep.then.ref); + }); + + it("words competing expiries in a result by EXPIRY_REASON's order, whatever the order of its blocks", () => { + servingFiles(); + const permutations = (items: T[]): T[][] => items.length <= 1 ? [items] + : items.flatMap((item, i) => permutations([...items.slice(0, i), ...items.slice(i + 1)]).map((rest) => [item, ...rest])); + const cases: Array<[string, Array]> = [ + ['competing-all.csv', ['passed', 'unparseable', 'rejected']], + ['competing-unread.csv', ['unparseable', 'rejected']], + ]; + for (const [name, kinds] of cases) { + const uri = `https://zz-files.example/${name}`; + const expiresAt = { passed: '2001-01-01T00:00:00Z', unparseable: 'zz-not-a-time', rejected: 'x'.repeat(REFERENCE_LIMITS.expiresAt + 1) }; + const blocks = kinds.map((kind) => ({ type: 'resource', uri, name, expiresAt: expiresAt[kind] })); + referenceRegistry.register(testimonyOf(blocks[0]), 'zz-files'); + // The first of EXPIRY_REASON's cases the blocks give. + for (const order of permutations(blocks)) { + const lines = toolResultDataToHistoryString(order).split('\n'); + assert.ok(lines.every((line) => line.endsWith(`— the reference is ${EXPIRY_REASON[kinds[0]!]}`)), lines.join('\n')); + } + } + }); + + it("reads the result's own references for a uri before what an earlier result left in the registry, whatever the order", () => { + servingFiles(); + const uri = 'https://zz-files.example/earlier-expired.csv'; + const link = { type: 'resource_link', uri, name: 'earlier.csv' }; + referenceRegistry.register(testimonyOf(link), 'zz-files'); + // An earlier result left the uri expired. + toolResultDataToHistoryString([{ type: 'resource', uri, name: 'earlier.csv', expiresAt: '2001-01-01T00:00:00Z' }]); + const fresh = { type: 'resource', uri, name: 'earlier.csv', expiresAt: '2099-01-01T00:00:00Z' }; + const before = toolResultDataToHistoryString([link, fresh]).split('\n')[0]; + const after = toolResultDataToHistoryString([fresh, link]).split('\n')[1]; + // This result's fresh reference decides: the link is a stub that fetches. + assert.match(before!, /^\[ref_[^\]]+\] earlier\.csv — from tool result — fetch with fetch_reference$/); + assert.equal(after, before); + }); + + it("lets a later result's plain reference to a uri lift an earlier result's never, whatever the order of its link and reference", () => { + const orders: Array<[string, (link: unknown, plain: unknown) => unknown[]]> = [ + ['link-first', (link, plain) => [link, plain]], + ['reference-first', (link, plain) => [plain, link]], + ]; + for (const [order, blocks] of orders) { + const uri = `https://zz-elsewhere.example/lifted-${order}.csv`; + // An earlier result marks the uri never. + toolResultDataToHistoryString([{ type: 'resource', uri, name: 'lifted.csv', disposition: 'never' }]); + // The later result's reference is its server's latest word on the + // uri, so the link shows it, as if no never had been given. + const lines = toolResultDataToHistoryString(blocks( + { type: 'resource_link', uri, name: 'lifted.csv' }, { type: 'resource', uri, name: 'lifted.csv' })).split('\n'); + assert.ok(lines.includes(`[resource link "lifted.csv": ${uri}; not fetched]`), `${order}: ${lines.join('\n')}`); + } + // A result naming the uri only in a link leaves the earlier never in force. + const uri = 'https://zz-elsewhere.example/lifted-link-only.csv'; + toolResultDataToHistoryString([{ type: 'resource', uri, name: 'kept.csv', disposition: 'never' }]); + assert.match(toolResultDataToHistoryString([{ type: 'resource_link', uri, name: 'kept.csv' }]), /^\[ref_[^\]]+\] kept\.csv — from tool result — /); + }); + + it("finds a never inside inline data's other fields, which showing it descends into, before any block is shown", () => { + const uri = 'https://zz-elsewhere.example/meta-never.csv'; + const data = { + items: [ + { type: 'resource_link', uri, name: 'meta.csv' }, + { type: 'image', data: PNG, mimeType: 'image/png', _meta: { ref: { type: 'resource', uri, name: 'meta.csv', disposition: 'never' } } }, + ], + }; + const shown = JSON.parse(toolResultDataToHistoryString(data)) as { items: [string, { _meta: { ref: string } }] }; + assert.match(shown.items[0], /^\[ref_[^\]]+\] meta\.csv — from tool result — /); + assert.match(shown.items[1]._meta.ref, /^\[ref_[^\]]+\] meta\.csv — from tool result — /); + // A content array shows an image, not its fields, so there a reference + // inside them doesn't count, and the link shows its uri. + const shownUri = 'https://zz-elsewhere.example/meta-in-array.csv'; + const array = toolResultDataToHistoryString([ + { type: 'resource_link', uri: shownUri, name: 'meta.csv' }, + { type: 'image', data: PNG, mimeType: 'image/png', _meta: { ref: { type: 'resource', uri: shownUri, name: 'meta.csv', disposition: 'never' } } }, + ], undefined, { dataIsContent: true }); + assert.equal(array.split('\n')[0], `[resource link "meta.csv": ${shownUri}; not fetched]`); + }); + + it("keeps a never for its result when more references than the registry holds come between it and the link", () => { + const uri = 'https://zz-elsewhere.example/evicted-never.csv'; + const many = Array.from({ length: 2001 }, (_, i) => ({ type: 'resource', uri: `https://zz-elsewhere.example/many/${i}.bin`, name: `${i}.bin` })); + const lines = toolResultDataToHistoryString([ + { type: 'resource', uri, name: 'evicted.csv', disposition: 'never' }, ...many, { type: 'resource_link', uri, name: 'evicted.csv' }, + ]).split('\n'); + assert.equal(lines.length, 2003); + assert.match(lines[2002]!, /^\[ref_[^\]]+\] evicted\.csv — from tool result — /); + }); + + it("reads claims through toJSON, as serialization does, so a never toJSON gives counts and a getter it leaves out isn't read", () => { + const uri = 'https://zz-elsewhere.example/to-json-never.csv'; + let reads = 0; + const value = { + toJSON: () => ({ ref: { type: 'resource', uri, name: 'to-json.csv', disposition: 'never' } }), + get unread(): string { reads += 1; return 'zz-unread'; }, + }; + const shown = JSON.parse(toolResultDataToHistoryString({ link: { type: 'resource_link', uri, name: 'to-json.csv' }, value })) as + { link: string; value: { ref: string } }; + assert.match(shown.link, /^\[ref_[^\]]+\] to-json\.csv — from tool result — /); + assert.match(shown.value.ref, /^\[ref_[^\]]+\] to-json\.csv — from tool result — /); + assert.equal(reads, 0); + }); + + it("registers nothing a result's serialization never reaches, such as what lies past a cycle, which shows as the unchecked notice", () => { + const uri = 'https://zz-elsewhere.example/past-cycle.csv'; + const loop: Record = {}; + loop.self = loop; + // The claims walk meets the cycle before the render does, so the result + // is the notice where main's serializer threw (Ada-1017, room-203 + // #200374), and nothing past the cycle is registered either way. + const data = { loop, later: { type: 'resource', uri, name: 'past.csv', disposition: 'never' } }; + const error = console.error; + console.error = () => {}; + try { + assert.equal(nonErrorResultText({ data }), UNCHECKED_RESULT_TEXT); + } finally { + console.error = error; + } + // A script's routes take the serializer's own answer: it throws. + assert.throws(() => toolResultDataToHistoryString(data), TypeError); + assert.equal(referenceRegistry.findByUri(uri), undefined); + }); + + it("keeps a withheld result's fields as the whole result claims them, so a never elsewhere withholds a kept link's uri", () => { + const uri = 'https://zz-elsewhere.example/kept-never.csv'; + const kept = keptWhenWithheld({ + success: true, + data: { files: [{ type: 'resource_link', uri, name: 'kept.csv' }], other: { type: 'resource', uri, name: 'kept.csv', disposition: 'never' } }, + keepWhenWithheld: ['files'], + }, undefined); + assert.ok(kept !== undefined && /^\{"files":\["\[ref_[^\]]+\] kept\.csv — from tool result — /.test(kept), kept); + }); + + it("keeps nothing of a withheld result whose claims a value stops the walk of, since what the rest claims is unknown", () => { + const uri = 'https://zz-elsewhere.example/kept-unread.csv'; + // A getter on the prototype, which the audit's JSON doesn't read, and + // the claims walk does. + class Status { get type(): string { throw new Error('zz-unreadable'); } } + const result = { + success: true, + data: { status: new Status(), withheld: { type: 'resource', uri, name: 'kept.csv', disposition: 'never' }, files: [{ type: 'resource_link', uri, name: 'kept.csv' }] }, + keepWhenWithheld: ['files'], + }; + assert.doesNotThrow(() => JSON.stringify(result)); + assert.equal(keptWhenWithheld(result, undefined), undefined); + }); + + it("reads a result's data once without a stored copy, and sends what that one reading made", () => { + // Natively, beside an image, and as text, with none. + for (const [data, expected] of [ + [[IMAGE, TEXT], ['image', 'text']], + [[TEXT, AUDIO], `zz-text\n${AUDIO_TEXT}`], + ] as const) { + let reads = 0; + const result = { success: true, dataIsContent: true, get data() { reads += 1; return data; } } as unknown as ToolResult; + const live = framework().toMembraneToolResult('zz-once', result, 1000).content; + assert.deepEqual(Array.isArray(live) ? (live as Array<{ type: string }>).map((b) => b.type) : live, expected); + assert.equal(reads, 1, JSON.stringify(expected)); + } + }); + + /** A module's result for a script, its data `data`: what the demand fetch + * tried, what it logged, and what the script got. */ + const demandFetch = async (data: unknown[], dataIsContent?: boolean) => { + const fw = Object.create(AgentFramework.prototype) as { + getReferenceFetcher(): unknown; + scriptResultAnswer(result: ToolResult, toolName: string, toServer: boolean): Promise; + }; + const fetched: string[] = []; + fw.getReferenceFetcher = () => ({ fetch: async (record: { testimony: { uri: string } }) => { fetched.push(record.testimony.uri); return { ok: false, error: 'zz-refused' }; } }); + const logged: string[] = []; + const error = console.error; + console.error = (line: string) => { logged.push(line); }; + try { + const answer = await fw.scriptResultAnswer({ success: true, data, ...(dataIsContent ? { dataIsContent } : {}) }, 'zz_tool', false); + return { fetched, logged, result: { isError: answer.isError === true, text: answerText(answer) } }; + } finally { + console.error = error; + } + }; + const NOT_MADE_EXPIRED = /script-demand fetch of ref_\S+ not made: by the result's claims, the reference is expired/; + + it("isn't fetched for a script on demand where a reference at depth in the module's data gives its uri an expiry, reading the claims as the text does", async () => { + // Data, not content: its second element isn't a block, so its text reads + // the result's claims at depth. + const at = (uri: string, nestedExpiry: string) => [ + { type: 'resource', uri, name: 'deep.pdf', expiresAt: '2099-01-01T00:00:00Z' }, + { nested: { type: 'resource', uri, name: 'deep.pdf', expiresAt: nestedExpiry } }, + ]; + // Where the nested reference is fresh too, the top one is fetched. + const control = 'https://zz-files.example/demand-deep-fresh.pdf'; + const fresh = await demandFetch(at(control, '2099-06-01T00:00:00Z')); + assert.equal(fresh.result.isError, false); + assert.deepEqual(fresh.fetched, [control]); + // Where it has expired, it isn't, for the expiry only a reading at depth sees. + const uri = 'https://zz-files.example/demand-deep-expired.pdf'; + const expired = await demandFetch(at(uri, '2001-01-01T00:00:00Z')); + assert.equal(expired.result.isError, false); + assert.deepEqual(expired.fetched, []); + assert.ok(expired.logged.some((line) => NOT_MADE_EXPIRED.test(line)), expired.logged.join('\n')); + }); + + it("is fetched for a script on demand where only a content array's unshown block names its uri expired, as the text reads it", async () => { + // Content from an MCPL call a module forwards: its elements are blocks, + // and one the lane doesn't know is a marker, its fields unshown. + const uri = 'https://zz-files.example/demand-content-flag.pdf'; + const { fetched, result } = await demandFetch([ + { type: 'resource', uri, name: 'flag.pdf', expiresAt: '2099-01-01T00:00:00Z' }, + { type: 'zz-unknown', inner: { type: 'resource', uri, name: 'flag.pdf', expiresAt: '2001-01-01T00:00:00Z' } }, + ], true); + assert.equal(result.isError, false); + assert.deepEqual(fetched, [uri]); + }); + + it("isn't fetched for a script on demand where any reference in the module's array gives its uri an expiry that counts", async () => { + const uri = 'https://zz-files.example/demand-expired.pdf'; + const fresh = { type: 'resource', uri, name: 'demand.pdf', expiresAt: '2099-01-01T00:00:00Z' }; + const expired = { type: 'resource', uri, name: 'demand.pdf', expiresAt: '2001-01-01T00:00:00Z' }; + const { fetched, logged, result } = await demandFetch([fresh, expired], true); + assert.equal(result.isError, false); + assert.deepEqual(fetched, []); + // The fetch not made is logged, as a refused one is. + assert.ok(logged.some((line) => NOT_MADE_EXPIRED.test(line)), logged.join('\n')); + }); + + it('shows a link that names no uri at depth as its fields, which hold none, and the script lane leaves its own link object as it is', () => { + // No uri to withhold, as for a reference's stub. + const named = { type: 'resource_link', name: 'zz-no-uri', size: 10 }; + assert.deepEqual(JSON.parse(toolResultDataToHistoryString({ found: [named] })), { found: [named] }); + // The object a script gets for a link, given to the script lane again, + // reads the same, its uri left out or shown, and its size kept. + for (const uri of [LINK_URI, PLAIN_URI]) { + const object = referenceBlockForScript({ type: 'resource_link', uri, name: 'report.csv', mimeType: 'text/csv', size: 1024 }) as Record; + assert.equal(object.sizeBytes, 1024, uri); + const again = JSON.parse(JSON.stringify([object], (_key, value: unknown) => shownToScript(value))) as unknown[]; + assert.deepEqual(again, [object], uri); + } + }); + + it('shows a link with no usable uri at any depth as its marker, never its fields', () => { + for (const uri of ['', 42, null]) { + const shown = JSON.parse(toolResultDataToHistoryString({ found: [{ type: 'resource_link', uri, name: 'x', description: 'zz-link-description' }] })); + assert.equal(typeof shown.found[0], 'string', String(uri)); + assert.ok(!JSON.stringify(shown).includes('zz-link-description'), String(uri)); + } + }); + + it("builds the live copy from the stored pass's own blocks, never classifying them again", () => { + const fw = framework() as unknown as { + toMembraneToolResult(callId: string, result: ToolResult, maxChars: number | undefined, precomputed: unknown): Live; + }; + const result: ToolResult = { success: true, data: [IMAGE, PLAIN_LINK], dataIsContent: true }; + // What the stored pass decided, whatever the registry says now. + const live = fw.toMembraneToolResult('zz', result, 1000, { + text: 'zz-stored', filePath: null, + live: [{ kind: 'image', data: PNG, mimeType: 'image/png' }, { kind: 'link', text: 'zz-as-the-stored-pass-wrote-it' }], + }).content as Array<{ type: string; text?: string }>; + assert.deepEqual(live.map((b) => b.text ?? b.type), ['image', 'zz-as-the-stored-pass-wrote-it']); + // And where it decided the stored text, that's the live copy. + assert.equal(fw.toMembraneToolResult('zz', result, 1000, { text: 'zz-stored', filePath: null, live: null }).content, 'zz-stored'); + }); + + it('goes natively with links and markers exactly at the cap, and as the stored text one character past it', () => { + const cap = AUDIO_TEXT.length * 2; + const at = copies({ success: true, data: [IMAGE, AUDIO, AUDIO], dataIsContent: true }, cap).live; + assert.ok(Array.isArray(at), String(at)); + const past = copies({ success: true, data: [IMAGE, AUDIO, AUDIO], dataIsContent: true }, cap - 1); + assert.equal(past.live, past.stored); + }); + + it("reads a result's own dataIsContent, whatever the options say", () => { + const data = [TEXT, { type: 'zz-other' }]; + // The result says data, so the array is JSON, though the options say content. + assert.equal(nonErrorResultText({ data, dataIsContent: false }, undefined, { dataIsContent: true }), JSON.stringify(data)); + // The result says content, so each block shows on its own. + assert.equal(nonErrorResultText({ data, dataIsContent: true }, undefined, { dataIsContent: false }), `zz-text\n${unknownBlockText({ type: 'zz-other' })}`); + }); + + it('hides a link uri with any one credential parameter, each on its own', () => { + for (const name of ['X-Amz-Signature', 'X-Amz-Credential', 'X-Amz-Security-Token', 'X-Goog-Signature', 'X-Goog-Credential', + 'sig', 'signature', 'Key-Pair-Id', 'Policy', 'hm', 'Authorization', 'access_token', 'id_token', 'refresh_token', + 'client_secret', 'oauth_token', 'oauth_signature', 'token', 'auth', 'auth_token', 'key', 'api_key', 'api_token', + 'x-api-key', 'private_token', 'secret', 'password', 'session_id', 'jsessionid']) { + const uri = `https://zz-elsewhere.example/one-${name}.csv?v=1&${name}=zz`; + const line = toolResultDataToHistoryString([TEXT, { type: 'resource_link', uri, name: 'one.csv' }]).split('\n')[1]!; + assert.ok(!line.includes('zz-elsewhere.example'), `${name}: ${line}`); + } + }); + + it("bounds an audio marker's type and strips its controls", () => { + const type = `audio/\u202Ewav\u0007${'x'.repeat(300)}`; + // The array's one block, so the text is its marker alone. + const marker = toolResultDataToHistoryString([{ type: 'audio', data: BASE64, mimeType: type }], undefined, { dataIsContent: true }); + assert.match(marker, /^\[Audio not shown: /); + assert.ok(!marker.includes('\u202E') && !marker.includes('\u0007'), marker); + assert.ok(marker.length < 250, String(marker.length)); + }); + + it("gives a script's object for a link whose line shows its uri why its refId would be refused, which the line, having no id, doesn't say", () => { + // On its server's own origin, so the expiry is the one refusal. + servingFiles(); + const uri = 'https://zz-files.example/expired-link.csv'; + referenceRegistry.register(testimonyOf({ type: 'resource', uri, name: 'expired.csv', expiresAt: '2001-01-01T00:00:00Z' })); + const link = { type: 'resource_link', uri, name: 'expired.csv' }; + const object = referenceBlockForScript(link, 'zz-files') as { uri?: string; refId: string; expired?: string; unfetchable?: string }; + assert.equal(object.uri, uri); + assert.match(object.refId, /^ref_/); + assert.equal(object.expired, EXPIRY_REASON.passed); + assert.equal(object.unfetchable, `the reference is ${EXPIRY_REASON.passed}`); + assert.ok(toolResultDataToHistoryString([TEXT, link]).endsWith(`[resource link "expired.csv": ${uri}; not fetched]`)); + }); + + it("reads the clock once for a script's object and once for a stub, so neither disagrees with itself across an expiry", () => { + servingFiles(); + const expiresAt = '2030-01-01T00:00:00Z'; + const at = Date.parse(expiresAt); + for (const [what, block] of [ + ['a reference', { type: 'resource', uri: 'https://zz-files.example/clock.csv', name: 'clock.csv', expiresAt }], + ['a link to a reference', { type: 'resource_link', uri: 'https://zz-files.example/clock-link.csv', name: 'clock.csv' }], + ] as const) { + // Its record holds the expiry, as a reference delivered for the link's + // uri leaves it, before the clock moves. + referenceRegistry.register(testimonyOf({ ...block, type: 'resource', expiresAt }), 'zz-files'); + const now = Date.now; + // A clock that reads just before the expiry the first time it's read, + // and past it every time after. + let reads = 0; + Date.now = () => (reads++ === 0 ? at - 1000 : at + 1000); + let object: { uri?: string; expired?: string; unfetchable?: string }; + let stub: string; + try { + object = referenceBlockForScript(block, 'zz-files') as typeof object; + reads = 0; + stub = buildReferenceStub(testimonyOf(block), 'from tool result', undefined, { unfetchedLinkUri: true }); + } finally { + Date.now = now; + } + // Each says, as of its one reading, that a fetch would dial. + assert.deepEqual([object.uri, object.expired, object.unfetchable], [undefined, undefined, undefined], what); + assert.ok(stub.endsWith('— fetch with fetch_reference'), `${what}: ${stub}`); + } + }); + + it("asks the resolver once for each server a fetch's check reads, so its address and its credential are one answer", () => { + const record = referenceRegistry.register(testimonyOf({ type: 'resource', uri: 'https://zz-refs.example/once.bin', name: 'once.bin' }), 'zz-once'); + let asked = 0; + const fetcher = new ReferenceFetcher(() => { + asked += 1; + // A resolver that would answer elsewhere the second time it's asked. + return asked === 1 ? { url: 'wss://zz-refs.example' } : { url: 'wss://zz-other.example' }; + }, async () => null); + assert.equal(fetcher.refusal(record), null); + assert.equal(asked, 1); + }); + + it("pairs each check's origin with the same answer's token and eager limit, on a fetch and on the eager check that joins it", async (t) => { + const heard: Array = []; + let arrived!: () => void; + const arrival = new Promise((resolve) => { arrived = resolve; }); + let release!: () => void; + const held = new Promise((resolve) => { release = resolve; }); + const server = createServer((req, res) => { + heard.push(req.headers.authorization); + arrived(); + void held.then(() => { + res.writeHead(200, { 'content-type': 'application/octet-stream' }); + res.end(Buffer.from('zz-bytes')); + }); + }); + await new Promise((resolve) => server.listen(0, '127.0.0.1', resolve)); + t.after(() => new Promise((resolve) => server.close(() => resolve()))); + const host = `127.0.0.1:${(server.address() as AddressInfo).port}`; + const record = referenceRegistry.register( + testimonyOf({ type: 'resource', uri: `http://${host}/answer.bin`, name: 'answer.bin', sizeBytes: 8 }), 'zz-answer'); + // Every answer is on the server's own origin, each with its own token + // and eager limit, so a second read for one check would mix answers. + const answers = [ + { url: `ws://${host}`, token: 'zz-first' }, + { url: `ws://${host}`, token: 'zz-second', autofetch: { maxBytes: 1 } }, + { url: `ws://${host}`, token: 'zz-third', autofetch: { maxBytes: 100 } }, + ]; + let asked = 0; + const fetcher = new ReferenceFetcher(() => answers[Math.min(asked++, answers.length - 1)], async () => 'scratch/answer.bin'); + const fetching = fetcher.fetch(record); + await arrival; + assert.equal(asked, 1); + // The eager check of a record in flight joins it through dialServer, + // with one answer: the second's eager limit of 1 byte, which 8 bytes pass. + assert.equal(fetcher.isEagerEligible(record), false); + assert.equal(asked, 2); + release(); + assert.equal((await fetching).ok, true); + assert.deepEqual(heard, ['Bearer zz-first']); + }); + + it("hands the live copy the stored pass's own blocks, so a change between the two leaves the live copy as stored", async () => { + servingFiles(); + const link = { type: 'resource_link', uri: 'https://zz-files.example/handed.csv', name: 'handed.csv' }; + // Delivered by the zz-files server, as an MCPL dispatch registers it. + referenceRegistry.register(testimonyOf(link), 'zz-files'); + const result: ToolResult = { success: true, data: [IMAGE, link], dataIsContent: true }; + const fw = Object.create(AgentFramework.prototype) as { + buildStoredToolResultContent(agent: string, calls: unknown[], maxChars: number | undefined): Promise<{ spilled: Map }>; + toMembraneToolResult(callId: string, result: ToolResult, maxChars: number | undefined, precomputed: unknown): Live; + }; + Object.assign(fw, { toolImageLedgers: new Map() }); + const call = { id: 'zz-handed', name: 'zz_tool', input: {}, result }; + const { spilled } = await fw.buildStoredToolResultContent('zz-agent', [call], 1000); + const stored = spilled.get('zz-handed') as { text: string; live: unknown[] | null }; + const [, line] = stored.text.split('\n'); + // Fetchable when stored: the link is a stub. + assert.match(line!, /^\[ref_[^\]]+\] handed\.csv — from tool result — fetch with fetch_reference$/); + // Then its server's address goes, so classified again it would be a line. + referenceRegistry.forgetServerAddress('zz-files', TEST_HOST); + try { + const live = fw.toMembraneToolResult('zz-handed', result, 1000, stored).content as Array<{ type: string; text?: string }>; + assert.deepEqual(live.map((b) => b.text ?? b.type), ['image', line]); + } finally { + servingFiles(); + } + }); + + it("withholds a uri inline data marks never from a link naming it, in a content array, at depth and in a script's view", () => { + // Off the server's origin, so a link to it shows its uri where nothing + // withholds it; the inline block's never does, though it's no reference + // (Ada-1017, room-203 #201013). + const uri = 'https://zz-elsewhere.example/inline-never/report.csv'; + const sealed = { type: 'image', data: 'iVBORw0KGgo=', mimeType: 'image/png', uri, disposition: 'never' }; + const link = { type: 'resource_link', uri, name: 'report.csv' }; + const control = toolResultDataToHistoryString([link], undefined, { dataIsContent: true }); + assert.ok(control.includes('inline-never'), control); + for (const shown of [ + toolResultDataToHistoryString([sealed, link], undefined, { dataIsContent: true }), + toolResultDataToHistoryString({ sealed, link }), + JSON.stringify(shownToScript(link, scriptViewClaims([sealed, link], false))), + ]) { + assert.ok(!shown.includes('inline-never'), shown); + } + // Only its never counts: an expiry beside its data claims nothing. + const dated = { ...sealed, disposition: 'ref', expiresAt: '2000-01-01T00:00:00Z' }; + assert.ok(toolResultDataToHistoryString([dated, link], undefined, { dataIsContent: true }).includes('inline-never')); + }); + + it("withholds a never uri from a link whose uri shows as the same text once well formed, a lone surrogate showing as U+FFFD", () => { + // Each side as the other shows it: a never with the replacement character + // and a link with a lone surrogate, and the other way round. + // Each pair at a path of its own, since the registry keeps each never. + for (const [i, [sealedEnd, linkEnd]] of [['\uFFFD', '\uD800'], ['\uD800', '\uFFFD']].entries()) { + const at = `https://zz-elsewhere.example/sur-never-${i}/`; + const sealed = { type: 'resource', uri: `${at}${sealedEnd}.csv`, name: 'report.csv', disposition: 'never' }; + const link = { type: 'resource_link', uri: `${at}${linkEnd}.csv`, name: 'report.csv' }; + const control = toolResultDataToHistoryString([link], undefined, { dataIsContent: true }); + assert.ok(control.includes(`sur-never-${i}`), control); + for (const shown of [ + toolResultDataToHistoryString([sealed, link], undefined, { dataIsContent: true }), + toolResultDataToHistoryString({ sealed, link }), + JSON.stringify(shownToScript(link, scriptViewClaims([sealed, link], false))), + ]) { + assert.ok(!shown.includes(`sur-never-${i}`), shown); + } + } + }); + + it("withholds a never uri from a link whose uri shows as the same text, its bidi control cut", () => { + // The same uri with a right-to-left mark, which sanitizeLabel cuts, so + // the link would show exactly the text the never withholds: the mark on + // the link's side, and on the never's, each pair at a path of its own, + // since the registry keeps each never. + for (const [i, markOnLink] of [true, false].entries()) { + const clean = `https://zz-elsewhere.example/cut-never-${i}/report.csv`; + const marked = `https://zz-elsewhere.example/cut-never-${i}/\u200Freport.csv`; + const sealed = { type: 'resource', uri: markOnLink ? clean : marked, name: 'report.csv', disposition: 'never' }; + const link = { type: 'resource_link', uri: markOnLink ? marked : clean, name: 'report.csv' }; + const control = toolResultDataToHistoryString([link], undefined, { dataIsContent: true }); + assert.ok(control.includes(`cut-never-${i}/report.csv`), control); + for (const shown of [ + toolResultDataToHistoryString([sealed, link], undefined, { dataIsContent: true }), + toolResultDataToHistoryString({ sealed, link }), + JSON.stringify(shownToScript(link, scriptViewClaims([sealed, link], false))), + ]) { + assert.ok(!shown.includes(`cut-never-${i}`), shown); + } + } + }); + + it("shows the model the unchecked notice, with a line naming the tool, for whatever stops reading or showing a result, where the serializer throws for a script", () => { + const logged: string[] = []; + const error = console.error; + console.error = (...args: unknown[]) => { logged.push(args.map(String).join(' ')); }; + try { + // A block of a content array whose text can't be read. + const unreadable = [{ type: 'text', get text(): string { throw new Error('zz-unreadable text'); } }]; + let blocks = false; + const text = nonErrorResultText({ data: unreadable, dataIsContent: true }, undefined, { toolName: 'zz_tool', onBlocks: () => { blocks = true; } }); + assert.equal(text, UNCHECKED_RESULT_TEXT); + assert.equal(blocks, false, 'no live copy of its own: the notice is both copies'); + assert.throws(() => toolResultDataToHistoryString(unreadable, undefined, { dataIsContent: true }), /zz-unreadable text/); + // A result whose data itself can't be read, as a host's object to the + // direct Agent API can be. + const result = { get data(): unknown { throw new Error('zz-unreadable data'); } }; + assert.equal(nonErrorResultText(result, undefined, { toolName: 'zz_data' }), UNCHECKED_RESULT_TEXT); + const noError = { data: null, get error(): string { throw new Error('zz-unreadable error'); } }; + assert.equal(nonErrorResultText(noError, undefined, { toolName: 'zz_error' }), UNCHECKED_RESULT_TEXT); + const noFlag = { data: [{ type: 'text', text: 'x' }], get dataIsContent(): boolean { throw new Error('zz-unreadable flag'); } }; + assert.equal(nonErrorResultText(noFlag, undefined, { toolName: 'zz_flag' }), UNCHECKED_RESULT_TEXT); + // What's thrown can be a value String() can't read itself; the line + // still goes out, saying so. + const unprintable = { get data(): unknown { throw Object.create(null); } }; + // The words themselves, which the agent reads, pinned here once. + assert.equal(UNCHECKED_RESULT_TEXT, "[Result not shown: the tool ran, but its result couldn't be read through, checked for references and shown, so this host withheld it.]"); + assert.equal(nonErrorResultText(unprintable, undefined, { toolName: 'zz_unprintable' }), UNCHECKED_RESULT_TEXT); + // A render that stops where the claims walk got through: the render + // registers what it shows, and the claims walk registers nothing. + const uri = 'https://zz-elsewhere.example/render-stop/report.csv'; + const real = referencesModule.referenceRegistry.register; + referencesModule.referenceRegistry.register = function (this: unknown, ...args: Parameters) { + if (args[0].uri === uri) throw new RangeError('Maximum call stack size exceeded'); + return real.apply(referencesModule.referenceRegistry, args); + } as typeof real; + try { + const data = { found: { type: 'resource', uri, name: 'report.csv' } }; + assert.equal(nonErrorResultText({ data }, 100, { toolName: 'zz_deep' }), truncateForHistory(UNCHECKED_RESULT_TEXT, 100)); + assert.throws(() => toolResultDataToHistoryString(data), RangeError); + } finally { + referencesModule.referenceRegistry.register = real; + } + } finally { + console.error = error; + } + assert.equal(logged.length, 6, logged.join('\n')); + assert.match(logged[0]!, /tool 'zz_tool'.*zz-unreadable text/); + assert.match(logged[1]!, /tool 'zz_data'.*zz-unreadable data/); + assert.match(logged[2]!, /tool 'zz_error'.*zz-unreadable error/); + assert.match(logged[3]!, /tool 'zz_flag'.*zz-unreadable flag/); + assert.match(logged[4]!, /tool 'zz_unprintable'.*a value that can't be read/); + assert.match(logged[5]!, /tool 'zz_deep'.*Maximum call stack size exceeded/); + }); + + it('shows data whose claims walk stops as the unchecked notice, never with no claims, and a round with it completes, both copies alike', async () => { + // A value that stops the claims walk and not the render's: its type + // throws a RangeError the first time it's read, as a stack can run out + // one frame sooner in one walk than in the other, and then reads as data. + const once = () => { + let read = false; + return { get type(): string { if (!read) { read = true; throw new RangeError('Maximum call stack size exceeded'); } return 'note'; } }; + }; + // A link beside a never to its uri, which a render with no claims would show. + const data = () => ({ link: { type: 'resource_link', uri: PLAIN_URI, name: 'report.csv' }, sealed: { type: 'resource', uri: PLAIN_URI, disposition: 'never' }, deep: once() }); + assert.throws(() => toolResultDataToHistoryString(data()), RangeError, 'a script\'s routes take the serializer\'s own answer'); + const fw = Object.create(AgentFramework.prototype) as { + buildStoredToolResultContent(agent: string, calls: unknown[], maxChars: number | undefined): Promise<{ blocks: Array<{ content: unknown }>; spilled: Map }>; + toMembraneToolResult(callId: string, result: ToolResult, maxChars: number | undefined, precomputed: unknown): Live; + }; + Object.assign(fw, { toolImageLedgers: new Map(), writeToolResultFile: async () => null, moduleRegistry: { getAllModules: () => [] } }); + const call = { id: 'zz-unchecked', name: 'zz_tool', input: {}, result: { success: true, data: data() } }; + const logged: string[] = []; + const error = console.error; + console.error = (...args: unknown[]) => { logged.push(args.map(String).join(' ')); }; + try { + const { blocks, spilled } = await fw.buildStoredToolResultContent('zz-agent', [call], 24_000); + assert.deepEqual(blocks.map((block) => block.content), [UNCHECKED_RESULT_TEXT]); + assert.equal(fw.toMembraneToolResult(call.id, call.result, 24_000, spilled.get(call.id)).content, UNCHECKED_RESULT_TEXT); + } finally { + console.error = error; + } + // The stored copy's builder names the tool in the model lane's log line. + assert.ok(logged.some((line) => /tool 'zz_tool'.*Maximum call stack size exceeded/.test(line)), logged.join('\n')); + }); + + it("keeps a result's native live copy when its turn's results are stored by their shares of the readers' bound", async () => { + const fw = Object.create(AgentFramework.prototype) as { + buildStoredToolResultContent(agent: string, calls: unknown[], maxChars: number | undefined, bound?: number): Promise<{ spilled: Map }>; + }; + Object.assign(fw, { + toolImageLedgers: new Map(), toolRoundsByShare: new WeakSet(), + writeToolResultFile: async () => null, moduleRegistry: { getAllModules: () => [] }, + }); + // Two results that pass a bound of 600 together, so each is stored by + // its share; the first holds an image, which goes natively at the cap. + const withImage = { id: 'zz-share-image', name: 'zz_tool', input: {}, result: { success: true, data: [IMAGE, { type: 'text', text: 'i'.repeat(400) }], dataIsContent: true } }; + const plain = { id: 'zz-share-text', name: 'zz_tool', input: {}, result: { success: true, data: 't'.repeat(400) } }; + const { spilled } = await fw.buildStoredToolResultContent('zz-agent', [withImage, plain], 24_000, 600); + const stored = spilled.get(withImage.id) as { text: string; live: Array<{ kind: string }> | null }; + assert.ok(stored.text.length < 500, `stored by its share: ${stored.text.length}`); + assert.deepEqual(stored.live?.map((block) => block.kind), ['image', 'text']); + }); + + it("goes natively from the stored pass with links and markers exactly at the cap, and as its stored text one past it", async () => { + const fw = Object.create(AgentFramework.prototype) as { + buildStoredToolResultContent(agent: string, calls: unknown[], maxChars: number | undefined): Promise<{ spilled: Map }>; + toMembraneToolResult(callId: string, result: ToolResult, maxChars: number | undefined, precomputed: unknown): Live; + }; + // A spill that can't be written names where the agent would read it, so + // the notice asks the module registry for a workspace: there's none. + Object.assign(fw, { toolImageLedgers: new Map(), writeToolResultFile: async () => null, moduleRegistry: { getAllModules: () => [] } }); + const result: ToolResult = { success: true, data: [IMAGE, AUDIO, AUDIO], dataIsContent: true }; + const cap = AUDIO_TEXT.length * 2; + for (const [maxChars, native] of [[cap, true], [cap - 1, false]] as const) { + const call = { id: `zz-cap-${maxChars}`, name: 'zz_tool', input: {}, result }; + const { spilled } = await fw.buildStoredToolResultContent('zz-agent', [call], maxChars); + const stored = spilled.get(call.id) as { text: string; live: unknown[] | null }; + assert.equal(stored.live !== null, native, String(maxChars)); + const live = fw.toMembraneToolResult(call.id, result, maxChars, stored).content; + if (native) assert.ok(Array.isArray(live), String(maxChars)); + else assert.equal(live, stored.text, String(maxChars)); + } + }); + + test("an MCPL send whose result has an element that isn't a block before its channel's id still succeeds", { timeout: 30_000 }, async (t) => { + const f = await fixture({ + tools: [{ name: 'send_message', description: 'Send a message.', inputSchema: { type: 'object' } }], + toolsCall: () => ({ content: [null, { type: 'text', text: JSON.stringify({ channelId: 'zz-chan' }) }] }), + }); + t.after(f.close); + const internals = f.framework as unknown as { pushEvent(event: unknown): void; dispatchToolCall(agent: string, call: unknown): void }; + const result = await new Promise((resolve) => { + const original = internals.pushEvent; + internals.pushEvent = (event: unknown) => { + const e = event as { type: string; callId?: string; result?: ToolResult }; + if (e.type === 'tool-result' && e.callId === 'zz-send') { internals.pushEvent = original; resolve(e.result!); } + else original.call(internals, event); + }; + internals.dispatchToolCall('agent', { id: 'zz-send', name: 'mcpl--editor--send_message', input: { channelId: 'zz-chan' } }); + }); + assert.equal(result.success, true, JSON.stringify(result)); + assert.equal(nonErrorResultText(result), `${unknownBlockText(null)}\n{"channelId":"zz-chan"}`); + }); + + it("makes an MCPL result with an element that isn't a block its marker, not a failure", async () => { + const fw = framework() as unknown as { + mcpToolResult(result: unknown, callId: string, mcplPeer: boolean, serverId: string, callName: string): Promise; + }; + const result = await fw.mcpToolResult({ content: [null, TEXT] }, 'zz-null', true, 'zz-server', 'zz-tool'); + assert.equal(result.success, true); + assert.equal(result.dataIsContent, true); + assert.equal(nonErrorResultText(result), `${unknownBlockText(null)}\nzz-text`); + }); +}); diff --git a/test/tool-result-history.test.ts b/test/tool-result-history.test.ts index 149bae2c5..d641f68fc 100644 --- a/test/tool-result-history.test.ts +++ b/test/tool-result-history.test.ts @@ -45,11 +45,19 @@ test('unknown block shape falls back to JSON', () => { assert.equal(toolResultDataToHistoryString(data), JSON.stringify(data)); }); -test('mixed known + unknown block shape falls back to JSON (all-or-nothing)', () => { +test('blocks of MCP types the lane cannot show are markers, block by block, never the JSON with their data', () => { const data = [ { type: 'text', text: 'fine' }, { type: 'audio', data: 'XX', mimeType: 'audio/mpeg' }, ]; + assert.equal(toolResultDataToHistoryString(data), "fine\n[Audio not shown: this host's tool results carry only text and images (audio/mpeg, 1B).]"); +}); + +test('a block of a type MCP does not define makes the array data, written as JSON (all-or-nothing)', () => { + const data = [ + { type: 'text', text: 'fine' }, + { type: 'mystery', payload: 42 }, + ]; assert.equal(toolResultDataToHistoryString(data), JSON.stringify(data)); }); diff --git a/test/tool-result-reference-uri.test.ts b/test/tool-result-reference-uri.test.ts index 66b1ff182..44a715fb5 100644 --- a/test/tool-result-reference-uri.test.ts +++ b/test/tool-result-reference-uri.test.ts @@ -4,7 +4,9 @@ * string or a content array of blocks the serializer recognizes is written * as JSON with each reference as its stub and inline data claiming a uri or * a disposition withheld. Everything else in it is written as JSON writes - * it, as before. + * it, as before. MCP's resource_link is a reference too, and where the host + * can't fetch one, as with no server, it shows its uri, the agent's only + * handle on it, unless the uri may carry a credential. */ import { describe, it } from 'node:test'; import assert from 'node:assert/strict'; @@ -122,11 +124,27 @@ describe('data the serializer doesn\'t read as content blocks', () => { }); it('writes data with nothing RFC-005 governs exactly as JSON does', () => { - const plain = [{ id: 1, url: URI, type: 'zz-record' }, { type: 'resource_link', uri: URI, name: 'zz' }, 'zz', 7, null]; + const plain = [{ id: 1, url: URI, type: 'zz-record' }, 'zz', 7, null]; assert.equal(toolResultDataToHistoryString(plain), JSON.stringify(plain)); assert.equal(toolResultDataToHistoryString({ a: 1 }), '{"a":1}'); }); + it("governs MCP's resource_link at any depth: one the host can't fetch shows its uri only where it can't carry a credential", () => { + // References (RFC-005 §8.1) with no server, as a module's are. A link's + // uri is the agent's only handle on it, so it stays where it can't carry + // a credential, as the JSON showed it; a resource's never shows. + const signed = { type: 'resource_link', uri: URI, name: 'zz.pdf', mimeType: 'application/pdf' }; + const plainUri = 'https://zz-files.example/plain.pdf'; + const plainLink = { ...signed, uri: plainUri }; + const text = toolResultDataToHistoryString([{ id: 1, type: 'zz-record' }, signed, 'zz', RESOURCE, plainLink]); + assert.ok(!text.includes(SIGNATURE), text); + const [record, signedShown, plain, stub, plainShown] = parsed(text) as [unknown, string, string, string, string]; + assert.deepEqual([record, plain], [{ id: 1, type: 'zz-record' }, 'zz']); + assert.match(signedShown, /^\[ref_[^\]]+\] zz\.pdf — application\/pdf — from tool result — it's not tied to any server/); + assert.match(stub, STUB); + assert.equal(plainShown, `[resource link "zz.pdf": ${plainUri}, application/pdf; not fetched]`); + }); + it('writes a record that only shares a type name, with no uri and no string data, as JSON does', () => { // A uri that is null, absent or empty is none, and data that isn't a // string isn't inline data: nothing here is a reference or inline bytes.