Repository navigation
feat: Plan the Turn Context at Send Time and Freeze the System Prompt per Segment - #17146
Open
Shabonasar wants to merge 75 commits into
Open
Shabonasar wants to merge 75 commits into
Shabonasar wants to merge 75 commits into
Conversation
The per-request web search context (current date/time with milliseconds) was joined before additional_instructions, which carry mostly static text such as the artifacts prompt. Provider-side prefix caches stopped at the timestamp, so the static text after it was recomputed on every request. Move the dynamic context to the end via buildAdditionalInstructions and cover the ordering with unit tests.
…che-order # Conflicts: # packages/api/src/agents/initialize.ts
…e date changes in history The web search runtime context and temporal special variables were anchored to the turn start with millisecond precision. The system message precedes the whole history, so every turn invalidated the cached history and new conversations could not reuse the system prompt (LibreChat-AI#16877). - Web search runtime context states the conversation start in the user's timezone with minute precision; temporal variables resolve against the same persisted anchor. - A user message sent on a later local date than the model last saw gets a <context_update> block in its prompt copy only, derived from stored createdAt values so it renders identically on later turns; compaction boundaries follow the agents SDK formatter. - Optional endpoints.agents.promptMessageTimestamps adds the send time to every user message; the turn's user message is stored with createdAt = turnStartedAt on every save path. - Provider payload stability tests through createRun for OpenAI, an OpenAI-compatible endpoint, Anthropic, OpenRouter, Bedrock and Google.
… routes The browser resolved promptPrefix with its own clock on every send, so turn 1 of a spec-backed conversation carried the raw template while turn 2 carried resolved text in a different system-prompt position. Only Assistants routes resolve on the client now; the server resolves against the conversation start.
… of a conversation
Adds `endpoints.agents.promptCache.stableContext` (default off). Off, the prompt is
built as before. On:
- prompt time (`{{current_datetime}}` and friends, the web search runtime context) is
anchored to the conversation start, and a date change is announced by a
`<context_update>` block at the start of the first user message sent after it;
- the conversation-stable part of `additional_instructions` is merged into the system
instructions for Anthropic and OpenRouter with prompt caching, while memory, RAG and
one-turn queued file context stay in `additional_instructions`, where the SDK moves
them next to the newest user message;
- dynamic tool context goes after the additional instructions;
- the user message is inserted with the turn start as `createdAt`, so later turns render
the same stamp.
`promptMessageTimestamps` (default off) additionally prefixes each user message with the
time it was sent.
The history notes, the instruction split and the lever live in `packages/api`; the
controller only calls them. `applyContextToAgent` accepts the run context as parts and
records which are volatile, so no text is split by matching.
…assified parts Producers (project, memory guard, artifacts, skills catalog, RAG, memory, agent context files, always-apply skills of isolated subagents) contribute keyed parts through contributeContext instead of concatenating strings into additional_instructions. One registry fixes the canonical order and classifies every part, tool context map entries included, as static, conversation or turn. A runtime key the registry does not know counts as turn, so a new producer cannot break the stable prefix by default. assembleSystemPrompt builds the strings handed to the SDK. Once parts exist they are the only source of the tail; a direct write to additional_instructions is ignored with a warning. Without promptCache.stableContext it reproduces the previous request; with it, createRun moves only the turn parts with the prompt-cache tail and merges the rest into the instructions. The assembled text is byte-identical to string concatenation for both values of the lever, guarded by a snapshot recorded on the previous implementation. Isolated subagents with an always-apply skill place the skill ahead of the tool runtime context when stableContext is on.
…m prompt per segment Behind `endpoints.agents.promptCache.stableContext`; off, nothing here runs and the prompt is built as before. A single send-time step (turnContext) decides what the primary agent's model sees. Context sources (project, artifacts mode, memory tool guard, skills catalog, tool runtime context, memory, date, send time, retrieved context, queued upload files) are sampled once when the user sends a message; only the net difference from what the model already knows along the parent chain is stored on that user message (message.contextUpdates) and rendered as one <context_update> block at the start of its prompt copy. Toggles flipped back and forth before sending leave no trace, and stored values render the same bytes on every later turn, whatever the timezone. A regeneration sends no message: it renders what the message stored and writes nothing. The system prompt is rendered from the segment's anchor instead of the live parts, so it stays byte-identical until the next compaction. The anchor of the first segment is stored on the conversation, the anchor of a segment opened by a compaction on the message carrying the summary (contextAnchor); a conversation without one is anchored to its first turn after the upgrade. A re-anchor keeps the previous order, drops what is off and appends what is new, and dates itself, so the first message after the boundary announces nothing. Memory is a state source: an update lists the entries added, changed and removed (by key where the memory has keys) instead of the whole memory, and no longer rewrites the system prompt; memory switched off is withheld from the whole request. Retrieved context is stored with the user message it was retrieved for, like the file context bound to the message that carried the files; isolated subagents still take it in their system prompt. Queued upload files describe their own message only. A resumed turn (tool approval or an answered question) renders its system prompt from the paused turn's anchor, not from the live state. The planning, the anchor storage and the token accounting live in packages/api; the controller calls them. The conversation anchor is read with the conversation the ingress already loads, and a resume reads the messages once with a projection. contextUpdates and contextAnchor are server-written only, select: false, and stripped from client input, so they stay out of reads, shares and forks.
This was referenced Oct 15, 2026
codegraph-librechat
Bot
added
🗺️ Backend Infra
Shabonasar
marked this pull request as draft
October 16, 2026 06:21
…assified parts Producers (project, memory guard, artifacts, skills catalog, RAG, memory, agent context files, always-apply skills of isolated subagents) contribute keyed parts through contributeContext instead of concatenating strings into additional_instructions. One registry fixes the canonical order and classifies every part, tool context map entries included, as static, conversation or turn. A runtime key the registry does not know counts as turn, so a new producer cannot break the stable prefix by default. assembleSystemPrompt builds the strings handed to the SDK. Once parts exist they are the only source of the tail; a direct write to additional_instructions is ignored with a warning. Without promptCache.stableContext it reproduces the previous request; with it, createRun moves only the turn parts with the prompt-cache tail and merges the rest into the instructions. The assembled text is byte-identical to string concatenation for both values of the lever, guarded by a snapshot recorded on the previous implementation. Isolated subagents with an always-apply skill place the skill ahead of the tool runtime context when stableContext is on.
…m prompt per segment Behind `endpoints.agents.promptCache.stableContext`; off, nothing here runs and the prompt is built as before. A single send-time step (turnContext) decides what the primary agent's model sees. Context sources (project, artifacts mode, memory tool guard, skills catalog, tool runtime context, memory, date, send time, retrieved context, queued upload files) are sampled once when the user sends a message; only the net difference from what the model already knows along the parent chain is stored on that user message (message.contextUpdates) and rendered as one <context_update> block at the start of its prompt copy. Toggles flipped back and forth before sending leave no trace, and stored values render the same bytes on every later turn, whatever the timezone. A regeneration sends no message: it renders what the message stored and writes nothing. The system prompt is rendered from the segment's anchor instead of the live parts, so it stays byte-identical until the next compaction. The anchor of the first segment is stored on the conversation, the anchor of a segment opened by a compaction on the message carrying the summary (contextAnchor); a conversation without one is anchored to its first turn after the upgrade. A re-anchor keeps the previous order, drops what is off and appends what is new, and dates itself, so the first message after the boundary announces nothing. Memory is a state source: an update lists the entries added, changed and removed (by key where the memory has keys) instead of the whole memory, and no longer rewrites the system prompt; memory switched off is withheld from the whole request. Retrieved context is stored with the user message it was retrieved for, like the file context bound to the message that carried the files; isolated subagents still take it in their system prompt. Queued upload files describe their own message only. A resumed turn (tool approval or an answered question) renders its system prompt from the paused turn's anchor, not from the live state. The planning, the anchor storage and the token accounting live in packages/api; the controller calls them. The conversation anchor is read with the conversation the ingress already loads, and a resume reads the messages once with a projection. contextUpdates and contextAnchor are server-written only, select: false, and stripped from client input, so they stay out of reads, shares and forks.
Shabonasar
marked this pull request as ready for review
October 16, 2026 17:20
The per-request web search context (current date/time with milliseconds) was joined before additional_instructions, which carry mostly static text such as the artifacts prompt. Provider-side prefix caches stopped at the timestamp, so the static text after it was recomputed on every request. Move the dynamic context to the end via buildAdditionalInstructions and cover the ordering with unit tests.
…e date changes in history The web search runtime context and temporal special variables were anchored to the turn start with millisecond precision. The system message precedes the whole history, so every turn invalidated the cached history and new conversations could not reuse the system prompt (LibreChat-AI#16877). - Web search runtime context states the conversation start in the user's timezone with minute precision; temporal variables resolve against the same persisted anchor. - A user message sent on a later local date than the model last saw gets a <context_update> block in its prompt copy only, derived from stored createdAt values so it renders identically on later turns; compaction boundaries follow the agents SDK formatter. - Optional endpoints.agents.promptMessageTimestamps adds the send time to every user message; the turn's user message is stored with createdAt = turnStartedAt on every save path. - Provider payload stability tests through createRun for OpenAI, an OpenAI-compatible endpoint, Anthropic, OpenRouter, Bedrock and Google.
… routes The browser resolved promptPrefix with its own clock on every send, so turn 1 of a spec-backed conversation carried the raw template while turn 2 carried resolved text in a different system-prompt position. Only Assistants routes resolve on the client now; the server resolves against the conversation start.
… of a conversation
Adds `endpoints.agents.promptCache.stableContext` (default off). Off, the prompt is
built as before. On:
- prompt time (`{{current_datetime}}` and friends, the web search runtime context) is
anchored to the conversation start, and a date change is announced by a
`<context_update>` block at the start of the first user message sent after it;
- the conversation-stable part of `additional_instructions` is merged into the system
instructions for Anthropic and OpenRouter with prompt caching, while memory, RAG and
one-turn queued file context stay in `additional_instructions`, where the SDK moves
them next to the newest user message;
- dynamic tool context goes after the additional instructions;
- the user message is inserted with the turn start as `createdAt`, so later turns render
the same stamp.
`promptMessageTimestamps` (default off) additionally prefixes each user message with the
time it was sent.
The history notes, the instruction split and the lever live in `packages/api`; the
controller only calls them. `applyContextToAgent` accepts the run context as parts and
records which are volatile, so no text is split by matching.
Contributor
Author
|
Rebased on
Live check (Anthropic, Haiku 5.5): tools scenario 87.8% cached, 0 misses; multi-turn G2 85.3%, 0 misses. |
…memory Since LibreChat-AI#16301 formatMemoryContext('') returns the bare memory instructions sentence, and prepareRuntimeAgent applied it to the primary agent whenever the user had no saved memories. An agent with no memory tools and no automatic memory extraction can neither read nor write memory, so the sentence told the model nothing it could act on, yet it reached the model as per-turn context. The empty-store notice is now added only when the agent has memory tools or automatic extraction is on. Saved memories are still injected as before.
…m prompt per segment Behind `endpoints.agents.promptCache.stableContext`; off, nothing here runs and the prompt is built as before. A single send-time step (turnContext) decides what the primary agent's model sees. Context sources (project, artifacts mode, memory tool guard, skills catalog, tool runtime context, memory, date, send time, retrieved context, queued upload files) are sampled once when the user sends a message; only the net difference from what the model already knows along the parent chain is stored on that user message (message.contextUpdates) and rendered as one <context_update> block at the start of its prompt copy. Toggles flipped back and forth before sending leave no trace, and stored values render the same bytes on every later turn, whatever the timezone. A regeneration sends no message: it renders what the message stored and writes nothing. The system prompt is rendered from the segment's anchor instead of the live parts, so it stays byte-identical until the next compaction. The anchor of the first segment is stored on the conversation, the anchor of a segment opened by a compaction on the message carrying the summary (contextAnchor); a conversation without one is anchored to its first turn after the upgrade. A re-anchor keeps the previous order, drops what is off and appends what is new, and dates itself, so the first message after the boundary announces nothing. Memory is a state source: an update lists the entries added, changed and removed (by key where the memory has keys) instead of the whole memory, and no longer rewrites the system prompt; memory switched off is withheld from the whole request. Retrieved context is stored with the user message it was retrieved for, like the file context bound to the message that carried the files; isolated subagents still take it in their system prompt. Queued upload files describe their own message only. A resumed turn (tool approval or an answered question) renders its system prompt from the paused turn's anchor, not from the live state. The planning, the anchor storage and the token accounting live in packages/api; the controller calls them. The conversation anchor is read with the conversation the ingress already loads, and a resume reads the messages once with a projection. contextUpdates and contextAnchor are server-written only, select: false, and stripped from client input, so they stay out of reads, shares and forks.
A handoff target that shares the primary agent's memory partition kept a live copy of the memory in its per-turn tail. With prompt caching the SDK relocates that tail next to the newest user message, so the memory sat at a different place on every turn: the target's earlier requests no longer matched its next one, which missed the cache and, for models that bind thinking blocks to the prefix, made the API drop the target's own thinking blocks. The memory the turn context anchors for the segment now goes into the target's conversation block, and its changes reach it as the <context_update> blocks already stored on the history messages, as they reach the primary agent. Targets on another memory partition keep their live copy.
…he runtime note stays byte-stable The code-file note of an attached workspace is runtime context that the turn context anchors and updates. Priming returns files by last update, which moves with usage, so the same files could be listed in another order on a later turn and the note was announced again without having changed.
…se memory The send snapshot formatted an empty memory store as the bare memory instructions sentence whatever the agent could do with it, so the turn context anchored a sentence that tells the model nothing it can act on. The snapshot now applies the rule of prepareRuntimeAgent: an empty store is announced only to an agent with memory tools or when automatic extraction is on. Saved memories are anchored as before.
Contributor
Author
|
Added (from #17416): the turn-context anchor no longer pins the bare empty-memory sentence for an agent that cannot use memory (no memory tools, automatic extraction off). Same rule as the standalone fix for the base code; the two merge in any order. Delta: Shabonasar/LibreChat@8ad002a...fdafbef — new case in |
1 task done
No file changes (tree of dev c81e244). Joins earlier snapshots of the series that the branches share, so any two of them have a single merge base and merge cleanly.
With endpoints.agents.promptCache.stableContext, prompt time is anchored to the conversation start, a later date is announced in the history, the attached workspace's attachment note lists files in upload order, and the Agents route receives promptPrefix as a template that the server resolves against the conversation start instead of the browser clock. endpoints.agents.promptMessageTimestamps (default false) states the send time on each user message. Co-authored-by: Lia <lia@librechat.ai>
No file changes beyond LibreChat-AI#16369. Gives LibreChat-AI#17146 and LibreChat-AI#17189 one merge base.
With endpoints.agents.promptCache.stableContext, memory, project, artifacts, skills and the date are sampled at Send; changes reach the model as <context_update> on the user message and the system prompt is rendered from a per-segment anchor. Includes LibreChat-AI#17145.
This was referenced Oct 28, 2026
Contributor
Author
|
Reduced to minimize dependencies and the review size: includes #17145 (closed), +7551 -> +4833 lines against |
Shabonasar
added a commit
to Shabonasar/LibreChat
that referenced
this pull request
Nov 2, 2026
Convenience build for trying the prompt-cache series together; not for review. Tree = dev c81e244 + the following PRs: - LibreChat-AI#16369 fix/prompt-prefix-cache-order cd044f1 - LibreChat-AI#17146 feat/turn-context 4b52745 - LibreChat-AI#17147 feat/stable-tool-groups 9dcea9a - feat/conversation-block 38caf37 - LibreChat-AI#17149 feat/provider-output-items 2e8e145 - LibreChat-AI#17189 feat/reasoning-replay 4621693 Resolved: the history select in BaseClient.loadHistory (LibreChat-AI#17146 and LibreChat-AI#17149 both extend it) and its test. Needs @librechat/agents built from the agents branch integration/prompt-cache (LibreChat-AI/agents main + LibreChat-AI#613, LibreChat-AI#614, LibreChat-AI#615, LibreChat-AI#619, LibreChat-AI#629). Enable with endpoints.agents.promptCache.stableContext: true in librechat.yaml. See LibreChat-AI#17102.
Shabonasar
added a commit
to Shabonasar/LibreChat
that referenced
this pull request
Nov 2, 2026
Convenience build for trying the prompt-cache series together; not for review. Tree = dev 28d8780 + the following PRs (branches based on dev 28d8780): - LibreChat-AI#16369 fix/prompt-prefix-cache-order a719740 - LibreChat-AI#17146 feat/turn-context 9d1b4dd - LibreChat-AI#17147 feat/stable-tool-groups 42be726 - LibreChat-AI#17514 feat/conversation-block 1058c92 - LibreChat-AI#17149 feat/provider-output-items 79ad89a - LibreChat-AI#17189 feat/reasoning-replay 8c1ea69 - fix/file-context-drift ac8cdcc Resolved: the history select in BaseClient.loadHistory (LibreChat-AI#17146 and LibreChat-AI#17149 both extend it) and its test. Needs @librechat/agents built from the agents branch integration/prompt-cache (LibreChat-AI/agents main + LibreChat-AI#613, LibreChat-AI#614, LibreChat-AI#615, LibreChat-AI#619, LibreChat-AI#629). Enable with endpoints.agents.promptCache.stableContext: true in librechat.yaml. See LibreChat-AI#17102.
Shabonasar
added a commit
to Shabonasar/LibreChat
that referenced
this pull request
Nov 2, 2026
… dev, not main) Convenience build for trying the prompt-cache series together; not for review. The integration branch now tracks dev instead of main; the main-based variant is no longer maintained. Tree = dev 28d8780 + the following PRs (branches based on dev): - LibreChat-AI#16369 fix/prompt-prefix-cache-order a719740 - LibreChat-AI#17146 feat/turn-context 9d1b4dd - LibreChat-AI#17514 feat/conversation-block 1058c92 - LibreChat-AI#17147 feat/stable-tool-groups 42be726 - LibreChat-AI#17149 feat/provider-output-items 79ad89a - LibreChat-AI#17189 feat/reasoning-replay 8c1ea69 - fix/file-context-drift ac8cdcc Resolved: the history select in BaseClient.loadHistory (LibreChat-AI#17146 and LibreChat-AI#17149 both extend it) and its test. Needs @librechat/agents built from the agents branch integration/prompt-cache (LibreChat-AI/agents main + LibreChat-AI#613, LibreChat-AI#614, LibreChat-AI#615, LibreChat-AI#619, LibreChat-AI#629). Enable with endpoints.agents.promptCache.stableContext: true in librechat.yaml. See LibreChat-AI#17102.
Shabonasar
added a commit
to Shabonasar/LibreChat
that referenced
this pull request
Nov 2, 2026
… dev, not main) Convenience build for trying the prompt-cache series together; not for review. The integration branch now tracks dev instead of main; the main-based variant is no longer maintained. Tree = dev 28d8780 + the following PRs (branches based on dev): - LibreChat-AI#16369 fix/prompt-prefix-cache-order a719740 - LibreChat-AI#17146 feat/turn-context 9d1b4dd - LibreChat-AI#17514 feat/conversation-block 1058c92 - LibreChat-AI#17147 feat/stable-tool-groups 42be726 - LibreChat-AI#17149 feat/provider-output-items 79ad89a - LibreChat-AI#17189 feat/reasoning-replay 8c1ea69 - fix/file-context-drift aeaf487 Resolved: the history select in BaseClient.loadHistory (LibreChat-AI#17146 and LibreChat-AI#17149 both extend it) and its test. Needs @librechat/agents built from the agents branch integration/prompt-cache (LibreChat-AI/agents main + LibreChat-AI#613, LibreChat-AI#614, LibreChat-AI#615, LibreChat-AI#619, LibreChat-AI#629). Enable with endpoints.agents.promptCache.stableContext: true in librechat.yaml. See LibreChat-AI#17102.
8 tasks done
This branch has not been deployed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Part of #17102. Design discussion: #17481.
Summary
Memory, project, artifacts, skills and other context sources are re-read on every request and edit the system prompt, so a change anywhere in a conversation rewrites the early bytes of every later request. With
endpoints.agents.promptCache.stableContext(#16369; off, none of this runs) the sources are sampled once at Send, only the difference from what the model already saw along the parent chain is stored on the user message (contextUpdates) and rendered as one<context_update>block, and the system prompt is rendered from a per-segment anchor that stays byte-identical until the next compaction. Includes #17145 (system prompt assembled from keyed, classified parts).Related to #17102. Based on #16369. No SDK dependency.
How it works
systemParts.ts); memory, RAG and this turn's queued files stay per-turn inadditional_instructions, the rest joinsinstructionson providers where the SDK movesadditional_instructions(Anthropic, OpenRouter, Bedrock withpromptCache).contextUpdatesandcontextAnchorare server-written,select: false, stripped from client input; the anchor is read with the conversation the request already loads.Results
Live, Claude Haiku 5.5, thinking medium, cache TTL 1h, all PRs of #17102 together:
devNo API errors. Without #17514 G3 measured 35.4-41.0% and 13 misses: the SDK moves a summary from an earlier run with the per-turn part.
Changes from the previous version
dev+7551 -236 -> +4833 -179, of which fix: Keep the Prompt Prefix Stable Across Turns and Conversations #16369 is +919 -32.conversation_instructions) to feat: Pass the Stable Context as the SDK Conversation Block #17514; the attachment upload order to fix: Keep the Prompt Prefix Stable Across Turns and Conversations #16369; always-apply skills of isolated subagents as a separate part; the end-to-end capture tests and helper to the measurement kit.instructions(the SDK moves its tail since fix: Keep Per-Conversation Context and the Summary at a Fixed System Position agents#613).Type of change
Testing
Enable the setting, toggle memory or artifacts between messages:
toolsand the system prompt of consecutive requests stay identical, the change arrives as<context_update>in the user message.Automated tests:
BaseClient.test.js,client.test.js,request.resumeMetadata.spec.js,openai.spec.js,responses.unit.spec.js,convoAccess.spec.js,initialize.test.ts,promptPrefixStability.test.ts,context.spec.ts,run.spec.ts,systemParts.spec.ts,anchor.spec.ts,prepare.spec.ts,turnContext.spec.ts,conversation.spec.ts,message.spec.ts,share.test.ts. Guards (setting off -> as ondev):initialize.test.ts"builds the system prompt fields exactly as upstream without stableContext" (inline snapshot recorded ondev),client.test.js"keeps the prompt live and reads no anchor without stableContext". Passes with the npm SDK and with the SDK PRs.tsc --noEmitin packages/api, packages/data-schemas, api; eslint and prettier on the touched files.Risk / compatibility
Default off writes nothing and reads no anchor. New non-selected fields
contextUpdates(message) andcontextAnchor(conversation, summary message); their size is not capped (very large MCP tool schemas could make a document large). A conversation without an anchor is anchored at its first turn after the upgrade.Checklist