Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
4 changes: 2 additions & 2 deletions kernel/package-lock.json

Some generated files are not rendered by default. Learn more about how customized files appear on GitHub.

2 changes: 1 addition & 1 deletion kernel/package.json
Original file line number Diff line number Diff line change
@@ -1,6 +1,6 @@
{
"name": "acp-kernel",
"version": "0.0.151",
"version": "0.0.153",
"description": "Framework-agnostic context-compression engine (model-driven, 3-tier LSM). Pure core: no host dependency.",
"license": "MIT",
"author": "ranxianglei",
Expand Down
55 changes: 20 additions & 35 deletions kernel/src/compress-tools.ts
Original file line number Diff line number Diff line change
Expand Up @@ -69,21 +69,29 @@ const COMPRESS_RANGE_OBJECT = {

/** Shared parameter schema for every compress wire shape (anthropic
* `input_schema`, openai `function.parameters`, responses flat
* `parameters`). Contract (#374): a deliberate SUPERSET of what
* parseCompressArgs accepts — array content (line strings / range objects),
* string content (bare line form or JSON-encoded array), and the flat
* single-range form ({startId|startRef, endId|endRef, summary}, no content).
* Pre-validating hosts must never kill a call the kernel would honor; do not
* narrow this below the parser without re-checking parse-compress-input.ts.
* `parameters`).
*
* ADVERTISED FORM (#4252 decision): exactly one — content = ONE plain
* string holding ALL ranges (line form). The parser keeps honoring the
* legacy shapes (array content of line strings / range objects, the flat
* single-range call, and summary-as-plain-text in content with top-level
* refs), but the schema no longer teaches them: advertising several forms
* is what led models to blend them into invalid hybrids.
*
* Contract (#374, still holds): the schema must never kill a call the
* kernel would honor. It stays an OPEN object schema — no top-level
* `required`, no `additionalProperties: false` — so a flat single-range
* call {startId, endId, summary} still validates even though those
* properties are no longer declared. Do not add strictness here without
* re-checking parse-compress-input.ts.
*
* Wire-legality constraint (bili #1299, fixed after the #374 follow-up
* regressed it): the TOP LEVEL must stay a plain object schema — no
* `oneOf`/`allOf`/`anyOf`/`not`. Anthropic rejects tool input_schema with
* top-level combinators (400 on every request carrying the tool), so the
* content-vs-flat alternation lives in the descriptions and is enforced by
* parseCompressInput, never by a top-level combinator or `required` (a
* top-level `required: ["content"]` would let strict hosts kill the flat
* form). Copilot Gemini also needs scalar types and self-contained typed
* content string-vs-array alternation lives in the `anyOf` under `content`
* and is enforced by parseCompressInput, never by a top-level combinator.
* Copilot Gemini also needs scalar types and self-contained typed
* alternatives; keep each nested object alternative's properties with it.
*/
export const COMPRESS_PARAMETERS = {
Expand All @@ -95,7 +103,7 @@ export const COMPRESS_PARAMETERS = {
},
content: {
description:
"One or more ranges to compress into separate summary blocks. PREFERRED (multi-range batch): ONE plain string holding ALL ranges — each block starts with its 'm00150–m00220 optional topic' header line followed by that block's summary (a valid JSON string: internal quotes \\\" and newlines \\n; a JSON-encoded array as that string is also accepted). Array form: one entry per range — a line-form string or an object {startId,endId,summary,topic?}. Batch multiple ranges into ONE call — do not split into one call per range. REQUIRED unless the flat single-range form is used.",
"ONE plain string holding ALL ranges to compress — each block starts with its 'm00150–m00220 optional topic' header line followed by that block's summary markdown (a valid JSON string: internal quotes \\\" and newlines \\n; a JSON-encoded array as that string is also accepted). Batch multiple ranges into ONE call — do not split into one call per range.",
anyOf: [
{
type: "array",
Expand All @@ -120,36 +128,13 @@ export const COMPRESS_PARAMETERS = {
{ type: "string" },
],
},
startId: {
type: "string",
description:
"Flat single-range form (no content): mNNNNN ref at the start of the range",
},
endId: {
type: "string",
description:
"Flat single-range form (no content): mNNNNN ref at the end of the range",
},
startRef: {
type: "string",
description: "Flat single-range form: alternate spelling of startId",
},
endRef: {
type: "string",
description: "Flat single-range form: alternate spelling of endId",
},
summary: {
type: "string",
description:
"Flat single-range form (no content): self-contained summary replacing the range",
},
},
};

export const COMPRESS_TOOL = {
name: COMPRESS_TOOL_NAME,
description:
"Replace consumed conversation ranges with self-contained summaries you write, identified by their refs. PREFERRED form: content = ONE plain string holding ALL ranges — each block starts with its 'm00150–m00220 optional topic' header line followed by that block's summary (a valid JSON string — internal quotes \\\", newlines \\n; a JSON-encoded array as that string is also accepted for gateways that stringify arrays). Also accepted: object entries {startId,endId,summary,topic?} in the content array, and a flat single-range call {startId,endId,summary,topic?} without content. Batch multiple ranges into ONE call — do not split into one call per range. Use when content is genuinely consumed. REQUIRED — compress without content or flat range fields is invalid.",
"Replace consumed conversation ranges with self-contained summaries you write, identified by their refs. content = ONE plain string holding ALL ranges — each block starts with its 'm00150–m00220 optional topic' header line followed by that block's summary (a valid JSON string — internal quotes \\\", newlines \\n; a JSON-encoded array as that string is also accepted for gateways that stringify arrays). Batch multiple ranges into ONE call — do not split into one call per range. Use when content is genuinely consumed.",
input_schema: COMPRESS_PARAMETERS,
};

Expand Down
101 changes: 98 additions & 3 deletions kernel/src/parse-compress-input.ts
Original file line number Diff line number Diff line change
Expand Up @@ -44,6 +44,10 @@ export interface CompressParseDiagnostics {
kind: CompressParseKind;
/** True when the winning parse came from single→double quote repair. */
quoteSalvage?: boolean;
/** True when a flat single-range call carried its summary text in the
* `content` STRING alongside top-level refs ({content, startId, endId})
* — the two schema-advertised shapes merged — and was recovered. */
flatSummarySalvage?: boolean;
/** True when a non-array `content` value (single entry object or nested
* `ranges` array) was recovered into the canonical array form (#470). */
contentSalvage?: boolean;
Expand Down Expand Up @@ -211,7 +215,28 @@ function parseObjectValue(
diag.kind = "ok";
return finish(ranges, diag);
}
diag.kind = "content-not-array";
if (entries.length === 0 || entries.every(isHeaderlessLineEntry)) {
// No refs headers inside the string: the text may be the SUMMARY of
// a single flat range whose bounds sit at the TOP level — the two
// schema-advertised shapes ("content = ONE plain string" + the
// {startId, endId} single-range form) merged into one call.
// Observed live as bursts of identical `content-not-array`
// rejections while the model retried the same valid-intent call.
const flat = flatSingleRangeFromContent(value, content, callId);
if (flat !== undefined) {
diag.flatSummarySalvage = true;
diag.kind = "ok";
return finish([flat], diag);
}
}
// Refs-bearing entries that failed validation are a real line-form
// attempt (report no-valid-ranges + reasons); all-headerless text was
// never line-form (keep the generic content-not-array classification).
diag.invalidItems = invalid;
if (reasons.length > 0) diag.invalidReasons = reasons;
diag.kind = entries.some((e) => !isHeaderlessLineEntry(e))
? "no-valid-ranges"
: "content-not-array";
return finish([], diag);
}
entries = parsed.entries;
Expand Down Expand Up @@ -258,6 +283,62 @@ function parseObjectValue(
return finish(ranges, diag);
}

// A split line entry whose first line carries NO refs at all — pure prose.
// Only headerless content is eligible for the flat single-range recovery
// below (text with refs headers must keep its line-form reading, whatever
// its validation outcome).
function isHeaderlessLineEntry(e: unknown): boolean {
if (typeof e !== "string") return false;
const nl = e.indexOf("\n");
const head = (nl === -1 ? e : e.slice(0, nl)).trim();
return !REF_PAIR_IN_LINE.test(head) && !SINGLE_REF_IN_LINE.test(head);
}

// Flat single-range recovery: summary text in the `content` STRING with
// bounds at the top level ({content, startId, endId, topic?}). Applies only
// when the content string carries NO refs headers of its own — a line-form
// string with headers was already parsed (or reported) above. A JSON-encoded
// {summary, topic} object inside the string (gateway double-encode) is
// unwrapped first. Returns undefined when the shape does not fit.
function flatSingleRangeFromContent(
value: Record<string, unknown>,
content: string,
callId: string | undefined,
): CompressRangeSpec | undefined {
const start =
stringOr(value["startRef"]) ??
stringOr(value["startId"]) ??
stringOr(value["messageId"]);
const end =
stringOr(value["endRef"]) ??
stringOr(value["endId"]) ??
stringOr(value["messageId"]);
if (start === undefined || end === undefined) return undefined;
let summary = stringOr(value["summary"]);
let topic = stringOr(value["topic"]);
if (summary === undefined) {
const inner = tryParseLenient(stripFence(content.trim()));
if (
inner !== null &&
typeof inner === "object" &&
!Array.isArray(inner)
) {
const o = inner as Record<string, unknown>;
summary = stringOr(o["summary"]);
if (topic === undefined) topic = stringOr(o["topic"]);
}
if (summary === undefined) {
const text = content.trim();
if (text.length > 0) summary = text;
}
}
if (summary === undefined) return undefined;
const entry: Record<string, unknown> = { startRef: start, endRef: end, summary };
if (topic !== undefined) entry["topic"] = topic;
const outcome = validateEntry(entry, callId);
return "range" in outcome ? outcome.range : undefined;
}

function parseContentArray(
s: string,
): { entries: unknown[]; salvaged: boolean; quoteRepaired?: boolean } | null {
Expand Down Expand Up @@ -375,8 +456,13 @@ type EntryOutcome = { range: CompressRangeSpec } | { reason: string };
// anchored (^): summaries cite mNNNNN/bN refs routinely, so an unanchored
// scan would let a citation inside a headerless entry supply fake bounds.
const REF_PAIR_IN_LINE =
/^([mb]\d{1,7})\s*(?:[-\u2013\u2014\u2026~]|\.\.\.|to)\s*([mb]\d{1,7})\b/i;
/^([mb]\d{1,7})\s*(?:[-\u2013\u2014\u2026~\u2212\uFF0D]|\.\.\.|to)\s*([mb]\d{1,7})\b/i;
const SINGLE_REF_IN_LINE = /^([mb]\d{1,7})\b/i;
// An unrecognized divider between two refs (punctuation run + a ref right
// after) — the pair regex missed a separator variant. Folding as a single
// ref would silently compress ONE message of a many-message range.
const UNRECOGNIZED_REF_SEP_RE =
/^\s*[^\p{L}\p{N}\s]{1,3}\s*[mb]\d{1,7}\b/iu;

function normalizeLineRef(raw: string): string {
const lower = raw.toLowerCase();
Expand All @@ -403,6 +489,15 @@ function parseLineEntry(
const single = SINGLE_REF_IN_LINE.exec(head);
if (single === null)
return { reason: "line entry: no mNNNNN/bN refs in header" };
// A second ref behind an unrecognized separator means this was meant to
// be a PAIR — accept it as a single ref and the fold would silently
// cover ONE message of a many-message range. Reject with a precise
// reason so the caller restates the range with a known separator.
const rest = head.slice(single.index + single[0].length);
if (UNRECOGNIZED_REF_SEP_RE.test(rest))
return {
reason: "line entry: refs separator not recognized — refusing to narrow a ref pair to its first ref (use -, –, —, …, ~, .., or to)",
};
startRef = normalizeLineRef(single[1]!);
endRef = startRef;
afterRefs = single.index + single[0].length;
Expand Down Expand Up @@ -439,7 +534,7 @@ export function deriveTopicFromSummary(summary: string): string | undefined {
}

const LINE_ENTRY_REFS_RE =
/[mb]\d{1,7}\s*(?:[-\u2013\u2014\u2026~]|\.\.\.|to)\s*[mb]\d{1,7}\b/iy;
/[mb]\d{1,7}\s*(?:[-\u2013\u2014\u2026~\u2212\uFF0D]|\.\.\.|to)\s*[mb]\d{1,7}\b/iy;
const WHITESPACE_RE = /\s/;

function skipWhitespace(s: string, i: number): number {
Expand Down
124 changes: 124 additions & 0 deletions kernel/tests/parse-compress-input.test.ts
Original file line number Diff line number Diff line change
Expand Up @@ -1236,3 +1236,127 @@ test("line-form split still breaks on headers after blank lines and element resi
],
);
});

// ---------------------------------------------------------------------------
// Flat single-range: summary text in the content STRING + top-level bounds
// ({content, startId, endId, topic?}) — the two schema-advertised shapes
// merged into one call. Observed live as repeated kind=content-not-array
// rejections while the model retried the same valid-intent compress.
// ---------------------------------------------------------------------------

test("parseCompressArgs recovers a flat single-range call with summary text in content", () => {
const { ranges, diagnostics } = parseCompressArgs({
content: "Summary of the folded range.\nSecond line of the summary.",
startId: "m00150",
endId: "m00168",
});
assert.equal(diagnostics.ok, true);
assert.equal(diagnostics.flatSummarySalvage, true);
assert.equal(diagnostics.kind, "ok");
assert.equal(ranges.length, 1);
assert.equal(ranges[0]?.startRef, "m00150");
assert.equal(ranges[0]?.endRef, "m00168");
assert.equal(ranges[0]?.summary, "Summary of the folded range.\nSecond line of the summary.");
});

test("parseCompressArgs flat single-range keeps top-level topic and ref variants", () => {
const { ranges } = parseCompressArgs({
content: "The summary text.",
startRef: "m00002",
endRef: "m00009",
topic: "repro loop",
});
assert.equal(ranges.length, 1);
assert.equal(ranges[0]?.startRef, "m00002");
assert.equal(ranges[0]?.endRef, "m00009");
assert.equal(ranges[0]?.topic, "repro loop");
});

test("parseCompressArgs flat single-range unwraps a JSON-encoded {summary,topic} in content", () => {
const { ranges, diagnostics } = parseCompressArgs({
content: JSON.stringify({ summary: "encoded summary", topic: "inner topic" }),
startId: "m00001",
endId: "m00004",
});
assert.equal(diagnostics.flatSummarySalvage, true);
assert.equal(ranges[0]?.summary, "encoded summary");
assert.equal(ranges[0]?.topic, "inner topic");
});

test("parseCompressArgs flat single-range prefers an explicit top-level summary over content text", () => {
const { ranges } = parseCompressArgs({
content: "ignored prose",
summary: "explicit summary wins",
startId: "m00001",
endId: "m00002",
});
assert.equal(ranges[0]?.summary, "explicit summary wins");
});

test("parseCompressArgs does NOT apply flat single-range when content carries refs headers", () => {
// Line-form material keeps its line-form reading even with top-level refs:
// valid headers parse as ranges...
const ok = parseCompressArgs({
content: "m00010–m00020 topic\nsummary text.",
startId: "m00001",
endId: "m00099",
});
assert.equal(ok.ranges.length, 1);
assert.equal(ok.ranges[0]?.startRef, "m00010");
assert.equal(ok.diagnostics.flatSummarySalvage, undefined);
// ...and header-only entries report the missing summary, not a salvage.
const bad = parseCompressArgs({
content: "m00010–m00020 topic only",
startId: "m00001",
endId: "m00099",
});
assert.equal(bad.diagnostics.ok, false);
assert.equal(bad.diagnostics.flatSummarySalvage, undefined);
assert.equal(bad.diagnostics.kind, "no-valid-ranges");
assert.match(bad.diagnostics.invalidReasons?.[0] ?? "", /missing summary/);
});

test("parseCompressArgs keeps headerless prose without top-level refs as content-not-array", () => {
const { diagnostics } = parseCompressArgs({ content: "just prose, no refs, no bounds" });
assert.equal(diagnostics.kind, "content-not-array");
assert.equal(diagnostics.flatSummarySalvage, undefined);
});

test("parseCompressArgs reports header-only line entries as no-valid-ranges with reasons", () => {
const { diagnostics } = parseCompressArgs({
content: "m00010–m00020 topic one\nsummary ok.\nm00030–m00040 topic two",
});
assert.equal(diagnostics.kind, "ok"); // 1 valid + 1 invalid
assert.equal(diagnostics.invalidItems, 1);
assert.match(diagnostics.invalidReasons?.[0] ?? "", /missing summary/);
const none = parseCompressArgs({ content: "m00010–m00020 topic only" });
assert.equal(none.diagnostics.kind, "no-valid-ranges");
assert.match(none.diagnostics.invalidReasons?.[0] ?? "", /missing summary/);
});

// ---------------------------------------------------------------------------
// Refs separator variants: a pair divider outside the recognized dash class
// must never silently narrow the range to its first ref.
// ---------------------------------------------------------------------------

test("parseCompressArgs accepts U+2212 minus and fullwidth hyphen as range separators", () => {
const minus = parseCompressArgs({ content: "m00150−m00168 topic\nsummary." });
assert.equal(minus.ranges[0]?.startRef, "m00150");
assert.equal(minus.ranges[0]?.endRef, "m00168");
const fw = parseCompressArgs({ content: "m00150-m00168 topic\nsummary." });
assert.equal(fw.ranges[0]?.startRef, "m00150");
assert.equal(fw.ranges[0]?.endRef, "m00168");
});

test("parseCompressArgs refuses to narrow a ref pair with an unrecognized separator", () => {
const { ranges, diagnostics } = parseCompressArgs({
content: "m00150 => m00168 topic\nsummary.",
});
assert.equal(ranges.length, 0);
assert.equal(diagnostics.kind, "no-valid-ranges");
assert.match(diagnostics.invalidReasons?.[0] ?? "", /separator not recognized/);
// A legit single-ref header with a punctuation topic still parses.
const single = parseCompressArgs({ content: "m00150 — intro notes\nsummary." });
assert.equal(single.ranges[0]?.startRef, "m00150");
assert.equal(single.ranges[0]?.endRef, "m00150");
});
2 changes: 1 addition & 1 deletion package-lock.json

Some generated files are not rendered by default. Learn more about how customized files appear on GitHub.

Loading
Loading