diff --git a/CHANGELOG.md b/CHANGELOG.md index 181a727..9bbd615 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -6,6 +6,8 @@ All notable changes to this project are documented here. The format follows [Kee - `openresponses:` backend (`OpenResponses`) for the [Open Responses](https://www.openresponses.org) spec (OpenAI Responses API), replaying encrypted reasoning between turns. `openai-responses:` is kept as an alias. - `llms.txt` with a short usage guide for coding agents. +- `speed=` on `Session`, `wsp.read/fix/write/improve` and backends (CLI `--speed` / `--fast`) for provider fast mode: Anthropic `speed`, OpenAI `service_tier`. Saved in `result.json`; the Exp. 8 script takes `--speed fast`. +- Anthropic `thinking=False` falls back to `effort="low"` on models that can't disable thinking (Opus 5.5); `usage.thinking_tokens` is now recorded. ### Added - Exp. 8 replication harness for Qwen3.5-0.8B in `experiments/exp8`. diff --git a/README.md b/README.md index a9e3f3e..36e2dde 100644 --- a/README.md +++ b/README.md @@ -94,10 +94,12 @@ With thinking off, the provider's reasoning is disabled and the LLM is told to g | Open Responses | `reasoning.effort="none"` | `reasoning.effort=` | | OpenRouter | `reasoning.enabled=false` | `reasoning.effort=` | | vLLM / SGLang (Qwen3 etc.) | `OpenAIChat(..., thinking_style="chat_template")` → `enable_thinking=false` | `enable_thinking=true` | -| Anthropic | `thinking.type="disabled"` | adaptive thinking with `effort=` on newer models such as Opus 5.5; older models fall back to a token budget | +| Anthropic | `thinking.type="disabled"`; on Opus 5.5, where thinking is always on, `effort="low"` instead | adaptive thinking with `effort=` on newer models such as Opus 5.5; older models fall back to a token budget | CLI: `--no-thinking` or `--thinking low|medium|high|xhigh|max`. The setting is saved in `result.json`, so you can compare runs with and without thinking. +`speed="fast"` (CLI `--fast`) turns on the provider's fast mode for the editing LLM: same model, faster output, higher price. See [providers](docs/providers.md#fast-mode). + ## Probes A probe is any function. It runs after the current weights are synced into the model, and whatever it returns is what the LLM sees. Its arguments become the tool's schema. diff --git a/docs/providers.md b/docs/providers.md index d23ef8c..096516d 100644 --- a/docs/providers.md +++ b/docs/providers.md @@ -22,7 +22,18 @@ For vLLM, SGLang, Ollama or any OpenAI-compatible server, use `OpenAIChat(model, - OpenAI: `reasoning_effort` (Chat Completions) or `reasoning.effort` (Open Responses). Some models reject `"none"`, so set `OpenAIChat(..., off_effort="minimal")`. - OpenRouter: `reasoning.enabled` / `reasoning.effort`. - vLLM / SGLang chat templates: `OpenAIChat(..., thinking_style="chat_template")` sends `enable_thinking`. -- Anthropic: adaptive thinking with `output_config.effort`, falling back to a token budget on models that don't support it. Prompt caching is on by default (`AnthropicChat(..., cache=False)` to turn it off). +- Anthropic: adaptive thinking with `output_config.effort`, falling back to a token budget on models that don't support it. Models whose thinking is always on (Opus 5.5) reject `disabled`; there `thinking=False` falls back to `effort="low"`, where the model usually skips thinking, plus the act-directly prompt. `usage.thinking_tokens` in `result.json` shows how much it still thought. Prompt caching is on by default (`AnthropicChat(..., cache=False)` to turn it off). + +## Fast mode + +`speed="fast"` (CLI `--fast`) runs the same editing model with faster inference at a higher price. `None` keeps the provider's standard speed. The value is saved in `result.json`. + +- Anthropic: `speed="fast"` on the request, up to 2.5x faster output on Opus 5.5 for about 2x the price. +- OpenAI / Open Responses: sent as `service_tier`. `"fast"` costs about 2x; `speed="ultrafast"` also passes through for models that support it. + +```python +wsp.improve(model, "...", llm="anthropic:claude-opus-5-5", speed="fast") +``` ## Custom LLMs diff --git a/experiments/exp8/run_edit.py b/experiments/exp8/run_edit.py index 35b65d7..7687c4c 100644 --- a/experiments/exp8/run_edit.py +++ b/experiments/exp8/run_edit.py @@ -80,6 +80,7 @@ def main(): ap.add_argument("--thinking", default="high") ap.add_argument("--work", default="/root/exp/work") ap.add_argument("--practice", default="/root/exp/practice") + ap.add_argument("--speed", choices=["fast"], help="speed of the editing LLM (provider fast mode)") ap.add_argument("--gpu-mem", type=float, default=0.8, help="vLLM memory share per probe call (split in two for paired runs)") a = ap.parse_args() work = a.work @@ -319,6 +320,7 @@ def read_tf(k): max_tool_chars=8000, sandbox_timeout=900, thinking=a.thinking, + speed=a.speed, log_path=f"{work}/transcript.jsonl", verbose=True, ) diff --git a/llms.txt b/llms.txt index a1edbb4..412ad0b 100644 --- a/llms.txt +++ b/llms.txt @@ -15,6 +15,7 @@ Things to know: - `llm` is a `provider:model` string (`openai:`, `openresponses:`, `openrouter:`, `anthropic:`) or an `LLM` instance. Use `openresponses:` (Open Responses / OpenAI Responses API) for models that need reasoning together with tools, such as gpt-6 models. - `thinking=False` (CLI `--no-thinking`) makes the editing LLM act with no reasoning; `"low"`...`"max"` set the level; `None` keeps the provider default. +- `speed="fast"` (CLI `--fast`) uses the provider's fast mode (Anthropic `speed`, OpenAI `service_tier`): same model, faster, about 2x the price. - The LLM never gets the model object. It sees tensors through tools (list, describe, read slices, write) and a Python sandbox with `R` (read-only) and `W` (writable) dicts of numpy arrays, plus checkpoint / restore / revert. - The only way it can run the network is through probes: functions decorated with `@wsp.probe(cost=1)` whose docstring and arguments become the tool schema. `forward_budget` caps the total cost. - Lower level: `wsp.WeightSpace.from_torch / from_hf / from_file / from_numpy`, `ws.diff()`, `ws.save_patch()`, `ws.apply_patch()`, and `wsp.Session(ws, llm, mode=..., probes=..., namespace=..., max_steps=...).run(task)`. diff --git a/src/weightspace/agent.py b/src/weightspace/agent.py index 27fc477..9a20239 100644 --- a/src/weightspace/agent.py +++ b/src/weightspace/agent.py @@ -13,7 +13,7 @@ import numpy as np from . import prompts -from .llm import LLM, get_llm, normalize_thinking +from .llm import LLM, get_llm, normalize_speed, normalize_thinking from .probes import Probe from .render import describe, extract_blocks, fmt_array, parse_index, parse_weights, render_space from .sandbox import Sandbox @@ -65,6 +65,7 @@ class Result: transcript: list[dict] usage: dict ws: WeightSpace = field(repr=False) + speed: str | None = None @property def forward_used(self) -> int: @@ -73,6 +74,7 @@ def forward_used(self) -> int: def __str__(self) -> str: conf = f" (confidence {self.confidence:.2f})" if self.confidence is not None else "" think = f" thinking={self.thinking}" if self.thinking else "" + think += f" speed={self.speed}" if self.speed else "" return ( f"[{self.mode}] {self.answer}{conf}\n" f"steps={self.steps}{think} forward passes={self.forward_used}/{self.forward_budget}\n{self.diff}" @@ -86,6 +88,7 @@ def save(self, directory: str) -> None: summary = { "mode": self.mode, "thinking": self.thinking, + "speed": self.speed, "answer": self.answer, "confidence": self.confidence, "steps": self.steps, @@ -116,6 +119,8 @@ class Session: python: give the LLM a persistent Python sandbox over the weights thinking: reasoning of the editing LLM: None = provider default, False/"off" = no thinking (provider reasoning disabled and the LLM told to act directly), True or "low"/"medium"/"high" + speed: inference speed of the editing LLM: None = provider default, "fast" (or True) = fast mode + (same model, faster output, higher price), "standard", or a provider tier like "ultrafast" """ def __init__( @@ -136,6 +141,7 @@ def __init__( sandbox_timeout: float | None = 300, system_prompt: str | None = None, thinking: bool | str | None = None, + speed: bool | str | None = None, log_path: str | None = None, verbose: bool = False, ): @@ -153,6 +159,9 @@ def __init__( self.thinking = normalize_thinking(thinking) if self.thinking is not None: self.llm.thinking = self.thinking + self.speed = normalize_speed(speed) + if self.speed is not None: + self.llm.speed = self.speed self.system_prompt = system_prompt or prompts.SYSTEM if self.thinking == "off": self.system_prompt += prompts.NO_THINKING @@ -420,6 +429,7 @@ def _result(self, messages, steps) -> Result: transcript=messages, usage=self.usage, ws=self.ws, + speed=self.speed, ) diff --git a/src/weightspace/cli.py b/src/weightspace/cli.py index 4067176..9834580 100644 --- a/src/weightspace/cli.py +++ b/src/weightspace/cli.py @@ -31,6 +31,8 @@ def main(argv=None) -> int: "--thinking", choices=["off", "low", "medium", "high", "xhigh", "max"], help="reasoning of the editing LLM" ) b.add_argument("--no-thinking", dest="thinking", action="store_const", const="off", help="same as --thinking off") + b.add_argument("--speed", help='inference speed of the editing LLM, e.g. "fast" (provider fast mode)') + b.add_argument("--fast", dest="speed", action="store_const", const="fast", help="same as --speed fast") b.add_argument("--out", help="directory for transcript, result.json, patch.npz") b.add_argument("--save-weights", help="write edited weights to .safetensors/.npz") b.add_argument("-v", "--verbose", action="store_true") @@ -58,6 +60,7 @@ def main(argv=None) -> int: context=context, max_steps=args.max_steps, thinking=args.thinking, + speed=args.speed, verbose=args.verbose, ) res = s.run_text(args.task) if args.text_mode else s.run(args.task) diff --git a/src/weightspace/llm/__init__.py b/src/weightspace/llm/__init__.py index d41f31e..fc6c61a 100644 --- a/src/weightspace/llm/__init__.py +++ b/src/weightspace/llm/__init__.py @@ -1,6 +1,6 @@ from __future__ import annotations -from .base import LLM, LLMResponse, ToolCall, normalize_thinking +from .base import LLM, LLMResponse, ToolCall, normalize_speed, normalize_thinking from .mock import ScriptedLLM, call, reply @@ -36,4 +36,4 @@ def get_llm(spec: str | LLM, **kw) -> LLM: raise ValueError(f"unknown provider {provider!r}") -__all__ = ["normalize_thinking", "LLM", "LLMResponse", "ToolCall", "ScriptedLLM", "call", "reply", "get_llm"] +__all__ = ["normalize_thinking", "normalize_speed", "LLM", "LLMResponse", "ToolCall", "ScriptedLLM", "call", "reply", "get_llm"] diff --git a/src/weightspace/llm/anthropic.py b/src/weightspace/llm/anthropic.py index 3621a72..e89f4a7 100644 --- a/src/weightspace/llm/anthropic.py +++ b/src/weightspace/llm/anthropic.py @@ -3,7 +3,7 @@ import os from typing import Any -from .base import LLM, LLMResponse, ToolCall, normalize_thinking +from .base import LLM, LLMResponse, ToolCall, normalize_speed, normalize_thinking THINKING_BUDGET = {"low": 2048, "medium": 8192, "high": 16384, "xhigh": 24576, "max": 32000} @@ -59,6 +59,7 @@ def __init__( client=None, thinking=None, thinking_style: str = "auto", + speed=None, cache: bool = True, **params, ): @@ -72,16 +73,21 @@ def __init__( self.max_tokens = max_tokens self.params = params self.thinking = normalize_thinking(thinking) + self.speed = normalize_speed(speed) if thinking_style not in ("auto", "adaptive", "budget"): raise ValueError("thinking_style must be 'auto', 'adaptive' or 'budget'") self.thinking_style = thinking_style self.cache = cache + # models with always-on thinking (e.g. Opus 5.5) reject "disabled"; "off" then means lowest effort + self.off_style = "disabled" def _thinking_kwargs(self, style: str) -> tuple[dict, int]: t = normalize_thinking(self.thinking) if t is None: return {}, self.max_tokens if t == "off": + if self.off_style == "low_effort": + return {"thinking": {"type": "adaptive"}, "output_config": {"effort": "low"}}, self.max_tokens return {"thinking": {"type": "disabled"}}, self.max_tokens if style == "budget": budget = THINKING_BUDGET[t] @@ -94,6 +100,8 @@ def complete(self, messages, tools=None): # prompt caching: everything up to the newest block is reused on the next step msgs[-1]["content"][-1] = {**msgs[-1]["content"][-1], "cache_control": {"type": "ephemeral"}} kw = dict(self.params) + if self.speed: + kw["speed"] = self.speed if system: kw["system"] = system if tools: @@ -103,10 +111,14 @@ def complete(self, messages, tools=None): try: r = self.client.messages.create(model=self.model, max_tokens=max_tokens, messages=msgs, **kw, **tk) except Exception as e: - if self.thinking_style != "auto" or "thinking" not in tk or "adaptive" not in str(e): + if tk.get("thinking") == {"type": "disabled"} and "disabled" in str(e): + self.off_style = "low_effort" + elif self.thinking_style == "auto" and "thinking" in tk and "adaptive" in str(e): + self.thinking_style = "budget" + style = "budget" + else: raise - self.thinking_style = "budget" - tk, max_tokens = self._thinking_kwargs("budget") + tk, max_tokens = self._thinking_kwargs(style) r = self.client.messages.create(model=self.model, max_tokens=max_tokens, messages=msgs, **kw, **tk) if self.thinking_style == "auto": self.thinking_style = style @@ -118,6 +130,9 @@ def complete(self, messages, tools=None): elif b.type == "tool_use": calls.append(ToolCall(b.id, b.name, dict(b.input))) usage = r.usage.model_dump() if getattr(r, "usage", None) is not None else {} + thinking_tokens = (usage.get("output_tokens_details") or {}).get("thinking_tokens") + if thinking_tokens is not None: + usage["thinking_tokens"] = thinking_tokens return LLMResponse( text="\n".join(text) or None, tool_calls=calls, diff --git a/src/weightspace/llm/base.py b/src/weightspace/llm/base.py index c684159..af2e4bb 100644 --- a/src/weightspace/llm/base.py +++ b/src/weightspace/llm/base.py @@ -58,14 +58,27 @@ def normalize_thinking(value) -> str | None: return v +def normalize_speed(value) -> str | None: + """None = provider default; True/"fast" = fast mode; False/"standard"; other tiers pass through (e.g. "ultrafast").""" + if value is None: + return None + if value is True: + return "fast" + if value is False: + return "standard" + return str(value).lower() + + class LLM: """Implement `complete` to plug in any provider. - `thinking` is set by `Session(thinking=...)`; backends map it to their own reasoning switch. + `thinking` and `speed` are set by `Session(thinking=..., speed=...)`; backends map them to their own + reasoning switch and fast-inference option. """ name = "llm" thinking: str | None = None + speed: str | None = None def complete(self, messages: list[dict], tools: list[dict] | None = None) -> LLMResponse: raise NotImplementedError diff --git a/src/weightspace/llm/openai.py b/src/weightspace/llm/openai.py index 01be812..00be17b 100644 --- a/src/weightspace/llm/openai.py +++ b/src/weightspace/llm/openai.py @@ -3,7 +3,7 @@ import json import os -from .base import LLM, LLMResponse, ToolCall, normalize_thinking +from .base import LLM, LLMResponse, ToolCall, normalize_speed, normalize_thinking def to_openai_messages(messages: list[dict]) -> list[dict]: @@ -43,6 +43,8 @@ class OpenAIChat(LLM): "openrouter" OpenRouter's unified `reasoning` field "chat_template" vLLM / SGLang hybrid models (Qwen3...): `chat_template_kwargs.enable_thinking` None never send anything + + speed is sent as `service_tier` ("fast", "ultrafast", ...; "standard" maps to OpenAI's "default"). """ def __init__( @@ -55,6 +57,7 @@ def __init__( thinking=None, thinking_style: str | None = "reasoning_effort", off_effort: str = "none", + speed=None, **params, ): if client is None: @@ -71,6 +74,11 @@ def __init__( self.thinking = normalize_thinking(thinking) self.thinking_style = thinking_style self.off_effort = off_effort + self.speed = normalize_speed(speed) + + def _speed_params(self, kw: dict) -> None: + if self.speed: + kw["service_tier"] = "default" if self.speed == "standard" else self.speed def _thinking_params(self, kw: dict) -> None: t, style = normalize_thinking(self.thinking), self.thinking_style @@ -93,6 +101,7 @@ def _thinking_params(self, kw: dict) -> None: def complete(self, messages, tools=None): kw = dict(self.params) self._thinking_params(kw) + self._speed_params(kw) if tools: kw["tools"] = to_openai_tools(tools) r = self.client.chat.completions.create(model=self.model, messages=to_openai_messages(messages), **kw) @@ -149,6 +158,7 @@ def complete(self, messages, tools=None): t = normalize_thinking(self.thinking) if t is not None and self.thinking_style is not None: kw["reasoning"] = {"effort": self.off_effort if t == "off" else t} + self._speed_params(kw) if instructions: kw["instructions"] = instructions if tools: diff --git a/tests/test_agent.py b/tests/test_agent.py index 56e65f7..6e7e034 100644 --- a/tests/test_agent.py +++ b/tests/test_agent.py @@ -134,3 +134,9 @@ def test_thinking_off_reaches_llm_and_prompt(arrays): llm = ScriptedLLM([reply(None, call("submit", answer="ok"))]) wsp.read(dict(arrays), "?", llm) assert llm.thinking_seen == [None] + + +def test_speed_reaches_llm_and_result(arrays): + llm = ScriptedLLM([reply(None, call("submit", answer="ok"))]) + res = wsp.fix(dict(arrays), "fix", llm, speed=True) + assert llm.speed == "fast" and res.speed == "fast" and "speed=fast" in str(res) diff --git a/tests/test_llm.py b/tests/test_llm.py index 6e57822..e63a896 100644 --- a/tests/test_llm.py +++ b/tests/test_llm.py @@ -98,6 +98,12 @@ def test_openai_thinking_styles(): assert "reasoning_effort" not in _capture_openai(thinking="off", thinking_style=None) +def test_openai_speed(): + assert "service_tier" not in _capture_openai() + assert _capture_openai(speed="fast")["service_tier"] == "fast" + assert _capture_openai(speed=False)["service_tier"] == "default" + + def test_anthropic_thinking(): from anthropic.types import Message @@ -123,7 +129,10 @@ def create(**kw): llm = AnthropicChat("x", client=SimpleNamespace(messages=SimpleNamespace(create=create)), max_tokens=4000) llm.complete([{"role": "user", "content": "hi"}]) - assert "thinking" not in seen + assert "thinking" not in seen and "speed" not in seen + llm.speed = "fast" + llm.complete([{"role": "user", "content": "hi"}]) + assert seen["speed"] == "fast" llm.thinking = "off" llm.complete([{"role": "user", "content": "hi"}]) assert seen["thinking"] == {"type": "disabled"} @@ -147,6 +156,20 @@ def create_old(**kw): old.complete([{"role": "user", "content": "hi"}]) assert len(calls) == 3 + sent = [] + + def create_always_on(**kw): + sent.append(kw) + if kw.get("thinking") == {"type": "disabled"}: + raise ValueError('"thinking.type.disabled" is not supported for this model.') + return msg + + on = AnthropicChat("x", client=SimpleNamespace(messages=SimpleNamespace(create=create_always_on)), thinking="off") + on.complete([{"role": "user", "content": "hi"}]) + on.complete([{"role": "user", "content": "hi"}]) + assert len(sent) == 3 and on.off_style == "low_effort" + assert sent[-1]["thinking"] == {"type": "adaptive"} and sent[-1]["output_config"] == {"effort": "low"} + def test_openresponses_replays_items(): from weightspace.llm import get_llm