Skip to content

feat: Optimize LLM request layout for provider caching and add opt-in Anthropic prompt caching with CLI session usage summary. - #44

Merged
vinodvx merged 1 commit into
mainfrom
feat/prompt-caching
Jul 14, 2026
Merged

feat: Optimize LLM request layout for provider caching and add opt-in Anthropic prompt caching with CLI session usage summary.#44
vinodvx merged 1 commit into
mainfrom
feat/prompt-caching

Conversation

@vinodvx

@vinodvx vinodvx commented Jul 14, 2026

Copy link
Copy Markdown
Contributor

Description

  • Structure LLM requests for provider caching: stable system/tools prefix, memory then RAG as labeled messages, sanitized history at call time, and tool resolve order native → MCP → A2A → sub-agents → memory → RAG.
  • Add Anthropic-only llm.WithPromptCaching (default off) so cache breakpoints stay optional; OpenAI/Gemini ignore it.
  • CLI: show_llm_usage / AGENT_SHOW_LLM_USAGE accumulates session token usage and prints a summary on exit; docs updated for caching and CLI usage.

Test plan

  • Unit tests: go test ./internal/runtime/base/... ./pkg/llm/... ./pkg/agent/... ./cmd/...
  • Anthropic: enable WithPromptCaching(true), run multi-turn; confirm cached_prompt > 0 when eligible
  • CLI: set show_llm_usage: true, run a few turns, exit; confirm session usage summary
  • Memory/RAG prefetch still appear as labeled messages (not mutated into system prompt)

Type of change

  • Bug fix (non-breaking change which fixes an issue)
  • New feature (non-breaking change which adds functionality)
  • Breaking change (fix or feature that would cause existing functionality to change)
  • Documentation update
  • Refactor / chore

Related issues

Closes #43

Checklist

  • I have run make check
  • I have run task examples:all
  • I have run make tidy if I added or removed dependencies
  • Commit messages follow conventional commits (e.g. feat:, fix:, docs:)
  • I have added/updated tests for my changes
  • Documentation is updated if needed

… Anthropic prompt caching with CLI session usage summary.
@mintlify

mintlify Bot commented Jul 14, 2026

Copy link
Copy Markdown

Preview deployment for your docs. Learn more about Mintlify Previews.

Project Status Preview Updated (UTC)
agent-sdk-go 🟢 Ready View Preview Jul 14, 2026, 6:06 PM

💡 Tip: Enable Workflows to automatically generate PRs for you.

@codecov

codecov Bot commented Jul 14, 2026

Copy link
Copy Markdown

Codecov Report

❌ Patch coverage is 77.27273% with 20 lines in your changes missing coverage. Please review.

Files with missing lines Patch % Lines
pkg/llm/anthropic/client.go 68.62% 13 Missing and 3 partials ⚠️
internal/runtime/base/sanitize.go 77.77% 2 Missing and 2 partials ⚠️

📢 Thoughts on this report? Let us know!

@mintlify

mintlify Bot commented Jul 14, 2026

Copy link
Copy Markdown

Preview deployment for your docs. Learn more about Mintlify Previews.

Project Status Preview Updated (UTC)
agent-sdk-go 🟡 Building Jul 14, 2026, 6:06 PM

💡 Tip: Enable Workflows to automatically generate PRs for you.

@vinodvx
vinodvx merged commit 1950dc2 into main Jul 14, 2026
5 checks passed
@vinodvx
vinodvx deleted the feat/prompt-caching branch July 14, 2026 18:18
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

[Feature]: Optimize Token Usage with Provider-Native Prompt Caching

1 participant