Order getContext sections stable to volatile so provider prompt caching can take effect - #341
Open
Leul-Negash wants to merge 3 commits into
Open
Order getContext sections stable to volatile so provider prompt caching can take effect#341Leul-Negash wants to merge 3 commits into
Leul-Negash wants to merge 3 commits into
Conversation
getContext placed LAST_SKILL_USE_RESULTS, which is rewritten every cycle and is one of the largest sections, ahead of HISTORY. Providers that cache prompts only reuse a prefix that is byte-identical to the previous request, so the reusable part ended right after the small static head and almost nothing was served from cache on consecutive calls. Add a stableContextOrder flag (default off) that moves HISTORY ahead of the volatile results section and keeps TIME last. With the flag off the assembled prompt is byte-for-byte identical to before, so behaviour is unchanged unless the flag is turned on. Refs #300
The first version of tests/src_loop.metta drove contextVolatileTail through the global stableContextOrder flag and the &lastresults state cell, toggling the flag with add-atom/remove-atom. That errored out under the PeTTa version CI runs and stopped the suite before any assertion ran. Split the ordering into a pure orderContextTail helper and test that directly with literal section strings, so no loop state, state cell or flag mutation is involved. getContext behaviour is unchanged: with the flag off the tail is still byte-for-byte identical to the original layout.
Building the final string with py-str inside the test (and importing lib_llm to get it) hung under the PeTTa version CI runs and never reached an assertion. Return the ordered sections as a plain tuple from orderedContextSections and let contextVolatileTail wrap that in py-str, then assert on the tuple. The test no longer imports lib_llm or calls py-str. getContext output is unchanged.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Description
getContextplaces LAST_SKILL_USE_RESULTS — which is rewritten every cycle and is one of the largest sections — before HISTORY. Providers that cache prompts only reuse a prefix that is byte-identical to the previous request, so the reusable prefix ends right after the small static head and almost none of the prompt is served from cache on consecutive calls. This is #300.Adds a
stableContextOrderflag (default off). When it is on, HISTORY moves ahead of the volatile results section and TIME stays last, so the stable part of the prompt sits in the reusable prefix. When it is off, the assembled prompt is byte-for-byte identical to before, so nothing changes unless the flag is set. The reorder is isolated in a smallcontextVolatileTailhelper so the rest of getContext is untouched.Note on #284 (Context Frames): that PR also reworks getContext, and #300 calls out the same interaction. This change is default-off and scoped to the current layout, so it can land independently and the same ordering can be carried over when #284 is rebased. This is part 1 (section ordering); the fixed-block HISTORY cut suggested in #300 can follow as a separate change.
How Has This Been Tested?
Added
tests/src_loop.metta, which asserts the flag-off tail is byte-for-byte identical to the previous inline layout and that the flag-on tail places HISTORY ahead of the volatile results section. The full metta test suite passes locally.Checklist