Skip to content

Commit 1b8fb49

Browse files
committed
fix(compiler): cap short-doc source text via configurable max_doc_chars
The short-doc compile path sends the whole markdown source file as a single LLM message, so an oversized doc overflows the model context and fails (issue #73). Add max_doc_chars (default 500k chars, configurable per-KB/ global and via the API PATCH) that truncates the payload with an explicit marker so the model knows the tail is missing. - config.py: add max_doc_chars to DEFAULT_CONFIG + GLOBAL_SCALAR_KEYS - api_models.py: expose max_doc_chars in the writable config schema - compiler.py: _maybe_truncate_doc helper + use in compile_short_doc - docs: config.yaml.example, README, examples/configuration - tests: unit + integration (RED->GREEN proven); 6 new tests
1 parent ff54396 commit 1b8fb49

7 files changed

Lines changed: 205 additions & 0 deletions

File tree

‎README.md‎

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -255,6 +255,7 @@ OpenKB settings are initialized by `openkb init` and stored in `.openkb/config.y
255255
model: gpt-5.4 # LLM model (any LiteLLM-supported provider)
256256
language: en # Wiki output language
257257
pageindex_threshold: 20 # PDF pages threshold for PageIndex
258+
max_doc_chars: 500000 # Hard cap (chars) on source text per short-doc compile prompt (see issue #73)
258259
```
259260
260261
The full settings reference — `entity_types`, OAuth providers (`chatgpt/*`, `github_copilot/*`), and LiteLLM tuning (timeouts for slow local runtimes like Ollama / LM Studio, `drop_params`, GitHub Copilot headers, install notes) — is in **[`examples/configuration/`](examples/configuration/)**.

‎config.yaml.example‎

Lines changed: 4 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -1,6 +1,10 @@
11
model: gpt-5.4 # LLM model (any LiteLLM-supported provider)
22
language: en # Wiki output language
33
pageindex_threshold: 20 # PDF pages threshold for PageIndex
4+
# max_doc_chars: 500000 # Hard cap (chars) on the source text sent to a
5+
# # single short-doc compile prompt. Prevents a
6+
# # large markdown doc overflowing the model
7+
# # context (issue #73). Lower for local models.
48

59
# Optional: cap concurrent LLM calls during ingest (PageIndex indexing and
610
# concept/entity compilation — they never overlap, so one setting covers

‎examples/configuration/README.md‎

Lines changed: 5 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -69,6 +69,10 @@ The file `init` writes is small; everything else is optional. This is the shippe
6969
model: gpt-5.4 # LLM model (any LiteLLM-supported provider)
7070
language: en # Wiki output language
7171
pageindex_threshold: 20 # PDF pages threshold for PageIndex
72+
# max_doc_chars: 500000 # Hard cap (chars) on the source text sent to a
73+
# # single short-doc compile prompt. Prevents a
74+
# # large markdown doc overflowing the model
75+
# # context (issue #73). Lower for local models.
7276

7377
# Optional: cap concurrent LLM calls during ingest (PageIndex indexing and
7478
# concept/entity compilation — they never overlap, so one setting covers
@@ -111,6 +115,7 @@ pageindex_threshold: 20 # PDF pages threshold for PageIndex
111115
| `model` | `gpt-5.4` | LLM used for all compile/query/chat work. |
112116
| `language` | `en` | Language the wiki is written in. |
113117
| `pageindex_threshold` | `20` | PDFs with this many pages **or more** take the long-doc (PageIndex) path; shorter ones go through the short-doc path. See [`pageindex-cloud/`](../pageindex-cloud/). |
118+
| `max_doc_chars` | `500000` | Hard cap (characters) on the source text sent to a single short-doc compile prompt. Prevents a large markdown / text doc from overflowing the model context (which would otherwise fail or produce a truncated wiki page). Set it below your model's max input if you run local models. See issue #73. |
114119
| `concurrency` | `null` | Caps concurrent LLM calls OpenKB makes during ingest — both PageIndex's indexing of a long document and OpenKB's own concept/entity compilation. The two never run at once for the same document, so one setting covers both. Lower it if you hit provider rate limits or "too many open files" on large PDFs. `null` lets each stage apply its own default. |
115120
| `parallel_tool_calls` | unset | Whether the LLM agents (query, chat, lint, skill) may call tools in parallel. Unset keeps OpenKB's per-agent defaults; `true`/`false` force allow/sequential for every agent; `null` omits the setting (provider default). **Amazon Bedrock needs `null`** (see below). |
116121
| `entity_types` | 7 defaults | Custom vocabulary for entity pages. `other` is always kept. |

‎openkb/agent/compiler.py‎

Lines changed: 37 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -260,6 +260,33 @@
260260
# ---------------------------------------------------------------------------
261261

262262

263+
def _maybe_truncate_doc(content: str, max_doc_chars: int, doc_name: str = "") -> str:
264+
"""Cap ``content`` at ``max_doc_chars`` characters when it overflows.
265+
266+
The short-doc compile path sends the whole source document as one LLM
267+
message (no chunking — see issue #73). If the text is larger than the
268+
budget the prompt would overflow the model context and fail, or the
269+
provider would silently drop the tail. When that happens this truncates
270+
the payload and appends an explicit marker so the model knows the tail is
271+
missing. Documents within the budget are returned untouched (fast path).
272+
"""
273+
if len(content) <= max_doc_chars:
274+
return content
275+
logger.warning(
276+
"Markdown doc %s is %d chars (over max_doc_chars=%d); "
277+
"truncating the tail with an explicit marker",
278+
doc_name,
279+
len(content),
280+
max_doc_chars,
281+
)
282+
trunc_marker = (
283+
f"\n\n> **NOTE:** this document was truncated at "
284+
f"{max_doc_chars} characters (config `max_doc_chars`). "
285+
f"The following sections were not included."
286+
)
287+
return content[:max_doc_chars] + trunc_marker
288+
289+
263290
def _cached_text(text: str) -> list[dict]:
264291
"""Wrap a text payload into a content-block list with an Anthropic
265292
ephemeral cache_control marker.
@@ -2225,6 +2252,16 @@ async def compile_short_doc(
22252252
schema_md = get_agents_md(wiki_dir)
22262253
content = source_path.read_text(encoding="utf-8")
22272254

2255+
# Issue #73 — the short-doc path sends the whole document as a single LLM
2256+
# message, so an oversized markdown file overflows the model context and
2257+
# fails (or the provider silently drops the tail). Cap the payload at the
2258+
# configured `max_doc_chars` budget and mark the truncation explicitly so
2259+
# the model knows the tail is missing and the user can tell it happened.
2260+
# Default 500k is a pure safety cap for normal docs; local-model users set
2261+
# `max_doc_chars` to fit their context window.
2262+
max_doc_chars = int(config.get("max_doc_chars", 500_000))
2263+
content = _maybe_truncate_doc(content, max_doc_chars, doc_name)
2264+
22282265
# Base context A: system + document. cache_control marker on the doc
22292266
# message creates a cache breakpoint that covers (system + doc) for
22302267
# every downstream call (summary, concepts-plan, every concept page).

‎openkb/api_models.py‎

Lines changed: 4 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -419,6 +419,10 @@ class _KbConfigWritable(BaseModel):
419419
model: str | None = None
420420
language: str | None = None
421421
pageindex_threshold: int | None = None
422+
# Upper bound (chars) on source text fed to a single short-doc compile
423+
# prompt (see config.DEFAULT_CONFIG) — mirrors pageindex_threshold's scalar
424+
# layering, so a sane default lives in code and users override per-KB/global.
425+
max_doc_chars: int | None = None
422426
# Entity-type vocabulary for extraction. A list REPLACES the layer below; an
423427
# explicit null reverts to inherited. Values are cleaned/deduped and "other"
424428
# is always ensured at read time (config.resolve_entity_types).

‎openkb/config.py‎

Lines changed: 8 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -31,6 +31,13 @@
3131
"model": "gpt-5.4",
3232
"language": "en",
3333
"pageindex_threshold": 20,
34+
# Upper bound (characters) on the source text fed to a *single* short-doc
35+
# compile prompt. The short-doc path sends the whole document as one LLM
36+
# message (no chunking — see issue #73); without a cap, a large markdown
37+
# file overflows the model context and fails or silently truncates. This is
38+
# a pure safety cap, set high enough to never affect normal docs; local-model
39+
# users should lower it to fit their context window. 500k chars ≈ 125k tokens.
40+
"max_doc_chars": 500_000,
3441
# A GLOBAL_SCALAR_KEY like the three above, so the merged `effective` dict
3542
# always carries it (the layering/`sources` logic is type-agnostic). A
3643
# global/KB list overrides it wholesale; resolve_entity_types cleans the
@@ -502,6 +509,7 @@ def load_global_config() -> dict[str, Any]:
502509
"model",
503510
"language",
504511
"pageindex_threshold",
512+
"max_doc_chars",
505513
"entity_types",
506514
)
507515

‎tests/test_compiler.py‎

Lines changed: 146 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -3,6 +3,7 @@
33
from __future__ import annotations
44

55
import json
6+
from pathlib import Path
67
from unittest.mock import AsyncMock, MagicMock, patch
78

89
import pytest
@@ -16,6 +17,7 @@
1617
_backlink_summary_entities,
1718
_compile_concepts,
1819
_filter_entity_items,
20+
_maybe_truncate_doc,
1921
_parse_entities_plan,
2022
_parse_json,
2123
_prepend_source_to_frontmatter,
@@ -2830,3 +2832,147 @@ def test_concept_update_malformed_frontmatter_rebuilds(self, tmp_path):
28302832
assert 'type: "Concept"' in text
28312833
# Must have a properly closed frontmatter block (two '---' occurrences).
28322834
assert text.count("---") >= 2
2835+
2836+
2837+
class TestMaybeTruncateDoc:
2838+
"""``_maybe_truncate_doc`` caps oversized source docs for the short-doc path.
2839+
2840+
Issue #73: the short-doc compiler sends the whole markdown file as one LLM
2841+
message, so an oversized doc overflows the model context. The helper must
2842+
truncate over-budget docs with an explicit marker and pass through docs
2843+
within the budget untouched.
2844+
"""
2845+
2846+
def test_under_budget_passthrough_unchanged(self):
2847+
content = "short markdown body"
2848+
assert _maybe_truncate_doc(content, 1000, "doc") == content
2849+
2850+
def test_exactly_at_budget_passthrough(self):
2851+
content = "x" * 100
2852+
assert _maybe_truncate_doc(content, 100, "doc") == content
2853+
2854+
def test_over_budget_truncates_with_marker(self):
2855+
content = "A" * 200
2856+
out = _maybe_truncate_doc(content, 50, "doc")
2857+
# Body is capped at the budget...
2858+
assert out.startswith("A" * 50)
2859+
# ...and carries an explicit marker so the model knows the tail is gone.
2860+
assert "truncated at 50 characters" in out
2861+
assert "not included" in out
2862+
# The original tail text must NOT appear in the output.
2863+
assert "A" * 100 not in out
2864+
2865+
def test_marker_reports_configured_budget(self):
2866+
out = _maybe_truncate_doc("B" * 500, 123, "big-doc")
2867+
assert "truncated at 123 characters" in out
2868+
assert "config `max_doc_chars`" in out
2869+
2870+
2871+
class TestShortDocContentBudget:
2872+
"""Integration: ``compile_short_doc`` respects ``max_doc_chars`` from config.
2873+
2874+
An oversized markdown source must be truncated before it reaches the LLM
2875+
summary call, with the explicit marker visible in the user message.
2876+
"""
2877+
2878+
def _write_kb(self, tmp_path, max_doc_chars: int) -> Path:
2879+
wiki = tmp_path / "wiki"
2880+
(wiki / "sources").mkdir(parents=True)
2881+
(wiki / "summaries").mkdir(parents=True)
2882+
(wiki / "concepts").mkdir(parents=True)
2883+
(wiki / "index.md").write_text(
2884+
"# Index\n\n## Documents\n\n## Concepts\n",
2885+
encoding="utf-8",
2886+
)
2887+
(tmp_path / ".openkb").mkdir()
2888+
(tmp_path / ".openkb" / "config.yaml").write_text(
2889+
f"max_doc_chars: {max_doc_chars}\n",
2890+
encoding="utf-8",
2891+
)
2892+
src = wiki / "sources" / "big.md"
2893+
return src
2894+
2895+
@pytest.mark.asyncio
2896+
async def test_large_markdown_is_truncated_with_marker(self, tmp_path):
2897+
src = self._write_kb(tmp_path, max_doc_chars=100)
2898+
# A document comfortably over the 100-char budget, with a distinctive tail.
2899+
src.write_text("Start of document.\n" + ("T" * 500) + "\nEND_TAIL_MARKER", encoding="utf-8")
2900+
2901+
captured: list[list[dict]] = []
2902+
2903+
def sync_side_effect(*args, **kwargs):
2904+
captured.append(kwargs["messages"])
2905+
mock_resp = MagicMock()
2906+
mock_resp.choices = [MagicMock()]
2907+
mock_resp.choices[0].message.content = json.dumps(
2908+
{"description": "d", "content": "summary"}
2909+
)
2910+
mock_resp.usage = MagicMock(prompt_tokens=1, completion_tokens=1)
2911+
mock_resp.usage.prompt_tokens_details = None
2912+
return mock_resp
2913+
2914+
async def async_side_effect(*args, **kwargs):
2915+
mock_resp = MagicMock()
2916+
mock_resp.choices = [MagicMock()]
2917+
mock_resp.choices[0].message.content = json.dumps({"brief": "c", "content": "page"})
2918+
mock_resp.usage = MagicMock(prompt_tokens=1, completion_tokens=1)
2919+
mock_resp.usage.prompt_tokens_details = None
2920+
return mock_resp
2921+
2922+
with patch("openkb.agent.compiler.litellm") as mock_litellm:
2923+
mock_litellm.completion = MagicMock(side_effect=sync_side_effect)
2924+
mock_litellm.acompletion = AsyncMock(side_effect=async_side_effect)
2925+
await compile_short_doc("big", src, tmp_path, "gpt-4o-mini")
2926+
2927+
# The first message sent to the LLM is the document-summary request; its
2928+
# user payload must be truncated and carry the marker.
2929+
summary_call = captured[0]
2930+
doc_user = summary_call[1]
2931+
sent_text = doc_user["content"]
2932+
# Content may be a list-of-blocks (from _cached_text) or a plain string;
2933+
# handle both the way the pipeline emits it.
2934+
if isinstance(sent_text, list):
2935+
sent_text = "".join(b.get("text", "") for b in sent_text)
2936+
assert "truncated at 100 characters" in sent_text, sent_text
2937+
# The distinctive tail is gone from the payload.
2938+
assert "END_TAIL_MARKER" not in sent_text
2939+
2940+
@pytest.mark.asyncio
2941+
async def test_doc_within_budget_passes_through(self, tmp_path):
2942+
src = self._write_kb(tmp_path, max_doc_chars=100_000)
2943+
body = "A modest document.\n" + ("P" * 400)
2944+
src.write_text(body, encoding="utf-8")
2945+
2946+
captured: list[list[dict]] = []
2947+
2948+
def sync_side_effect(*args, **kwargs):
2949+
captured.append(kwargs["messages"])
2950+
mock_resp = MagicMock()
2951+
mock_resp.choices = [MagicMock()]
2952+
mock_resp.choices[0].message.content = json.dumps(
2953+
{"description": "d", "content": "summary"}
2954+
)
2955+
mock_resp.usage = MagicMock(prompt_tokens=1, completion_tokens=1)
2956+
mock_resp.usage.prompt_tokens_details = None
2957+
return mock_resp
2958+
2959+
async def async_side_effect(*args, **kwargs):
2960+
mock_resp = MagicMock()
2961+
mock_resp.choices = [MagicMock()]
2962+
mock_resp.choices[0].message.content = json.dumps({"brief": "c", "content": "page"})
2963+
mock_resp.usage = MagicMock(prompt_tokens=1, completion_tokens=1)
2964+
mock_resp.usage.prompt_tokens_details = None
2965+
return mock_resp
2966+
2967+
with patch("openkb.agent.compiler.litellm") as mock_litellm:
2968+
mock_litellm.completion = MagicMock(side_effect=sync_side_effect)
2969+
mock_litellm.acompletion = AsyncMock(side_effect=async_side_effect)
2970+
await compile_short_doc("big", src, tmp_path, "gpt-4o-mini")
2971+
2972+
summary_call = captured[0]
2973+
sent_text = summary_call[1]["content"]
2974+
if isinstance(sent_text, list):
2975+
sent_text = "".join(b.get("text", "") for b in sent_text)
2976+
# No truncation marker when the doc fits the budget.
2977+
assert "truncated at" not in sent_text
2978+
assert "P" * 400 in sent_text

0 commit comments

Comments
 (0)