Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 2 additions & 0 deletions CHANGELOG.md
Original file line number Diff line number Diff line change
Expand Up @@ -6,6 +6,8 @@ All notable changes to this project are documented here. The format follows [Kee

- `openresponses:<model>` backend (`OpenResponses`) for the [Open Responses](https://www.openresponses.org) spec (OpenAI Responses API), replaying encrypted reasoning between turns. `openai-responses:` is kept as an alias.
- `llms.txt` with a short usage guide for coding agents.
- `speed=` on `Session`, `wsp.read/fix/write/improve` and backends (CLI `--speed` / `--fast`) for provider fast mode: Anthropic `speed`, OpenAI `service_tier`. Saved in `result.json`; the Exp. 8 script takes `--speed fast`.
- Anthropic `thinking=False` falls back to `effort="low"` on models that can't disable thinking (Opus 5.5); `usage.thinking_tokens` is now recorded.

### Added
- Exp. 8 replication harness for Qwen3.5-0.8B in `experiments/exp8`.
Expand Down
4 changes: 3 additions & 1 deletion README.md
Original file line number Diff line number Diff line change
Expand Up @@ -94,10 +94,12 @@ With thinking off, the provider's reasoning is disabled and the LLM is told to g
| Open Responses | `reasoning.effort="none"` | `reasoning.effort=<level>` |
| OpenRouter | `reasoning.enabled=false` | `reasoning.effort=<level>` |
| vLLM / SGLang (Qwen3 etc.) | `OpenAIChat(..., thinking_style="chat_template")` → `enable_thinking=false` | `enable_thinking=true` |
| Anthropic | `thinking.type="disabled"` | adaptive thinking with `effort=<level>` on newer models such as Opus 5.5; older models fall back to a token budget |
| Anthropic | `thinking.type="disabled"`; on Opus 5.5, where thinking is always on, `effort="low"` instead | adaptive thinking with `effort=<level>` on newer models such as Opus 5.5; older models fall back to a token budget |

CLI: `--no-thinking` or `--thinking low|medium|high|xhigh|max`. The setting is saved in `result.json`, so you can compare runs with and without thinking.

`speed="fast"` (CLI `--fast`) turns on the provider's fast mode for the editing LLM: same model, faster output, higher price. See [providers](docs/providers.md#fast-mode).

## Probes

A probe is any function. It runs after the current weights are synced into the model, and whatever it returns is what the LLM sees. Its arguments become the tool's schema.
Expand Down
13 changes: 12 additions & 1 deletion docs/providers.md
Original file line number Diff line number Diff line change
Expand Up @@ -22,7 +22,18 @@ For vLLM, SGLang, Ollama or any OpenAI-compatible server, use `OpenAIChat(model,
- OpenAI: `reasoning_effort` (Chat Completions) or `reasoning.effort` (Open Responses). Some models reject `"none"`, so set `OpenAIChat(..., off_effort="minimal")`.
- OpenRouter: `reasoning.enabled` / `reasoning.effort`.
- vLLM / SGLang chat templates: `OpenAIChat(..., thinking_style="chat_template")` sends `enable_thinking`.
- Anthropic: adaptive thinking with `output_config.effort`, falling back to a token budget on models that don't support it. Prompt caching is on by default (`AnthropicChat(..., cache=False)` to turn it off).
- Anthropic: adaptive thinking with `output_config.effort`, falling back to a token budget on models that don't support it. Models whose thinking is always on (Opus 5.5) reject `disabled`; there `thinking=False` falls back to `effort="low"`, where the model usually skips thinking, plus the act-directly prompt. `usage.thinking_tokens` in `result.json` shows how much it still thought. Prompt caching is on by default (`AnthropicChat(..., cache=False)` to turn it off).

## Fast mode

`speed="fast"` (CLI `--fast`) runs the same editing model with faster inference at a higher price. `None` keeps the provider's standard speed. The value is saved in `result.json`.

- Anthropic: `speed="fast"` on the request, up to 2.5x faster output on Opus 5.5 for about 2x the price.
- OpenAI / Open Responses: sent as `service_tier`. `"fast"` costs about 2x; `speed="ultrafast"` also passes through for models that support it.

```python
wsp.improve(model, "...", llm="anthropic:claude-opus-5-5", speed="fast")
```

## Custom LLMs

Expand Down
2 changes: 2 additions & 0 deletions experiments/exp8/run_edit.py
Original file line number Diff line number Diff line change
Expand Up @@ -80,6 +80,7 @@ def main():
ap.add_argument("--thinking", default="high")
ap.add_argument("--work", default="/root/exp/work")
ap.add_argument("--practice", default="/root/exp/practice")
ap.add_argument("--speed", choices=["fast"], help="speed of the editing LLM (provider fast mode)")
ap.add_argument("--gpu-mem", type=float, default=0.8, help="vLLM memory share per probe call (split in two for paired runs)")
a = ap.parse_args()
work = a.work
Expand Down Expand Up @@ -319,6 +320,7 @@ def read_tf(k):
max_tool_chars=8000,
sandbox_timeout=900,
thinking=a.thinking,
speed=a.speed,
log_path=f"{work}/transcript.jsonl",
verbose=True,
)
Expand Down
1 change: 1 addition & 0 deletions llms.txt
Original file line number Diff line number Diff line change
Expand Up @@ -15,6 +15,7 @@ Things to know:

- `llm` is a `provider:model` string (`openai:`, `openresponses:`, `openrouter:`, `anthropic:`) or an `LLM` instance. Use `openresponses:` (Open Responses / OpenAI Responses API) for models that need reasoning together with tools, such as gpt-6 models.
- `thinking=False` (CLI `--no-thinking`) makes the editing LLM act with no reasoning; `"low"`...`"max"` set the level; `None` keeps the provider default.
- `speed="fast"` (CLI `--fast`) uses the provider's fast mode (Anthropic `speed`, OpenAI `service_tier`): same model, faster, about 2x the price.
- The LLM never gets the model object. It sees tensors through tools (list, describe, read slices, write) and a Python sandbox with `R` (read-only) and `W` (writable) dicts of numpy arrays, plus checkpoint / restore / revert.
- The only way it can run the network is through probes: functions decorated with `@wsp.probe(cost=1)` whose docstring and arguments become the tool schema. `forward_budget` caps the total cost.
- Lower level: `wsp.WeightSpace.from_torch / from_hf / from_file / from_numpy`, `ws.diff()`, `ws.save_patch()`, `ws.apply_patch()`, and `wsp.Session(ws, llm, mode=..., probes=..., namespace=..., max_steps=...).run(task)`.
Expand Down
12 changes: 11 additions & 1 deletion src/weightspace/agent.py
Original file line number Diff line number Diff line change
Expand Up @@ -13,7 +13,7 @@
import numpy as np

from . import prompts
from .llm import LLM, get_llm, normalize_thinking
from .llm import LLM, get_llm, normalize_speed, normalize_thinking
from .probes import Probe
from .render import describe, extract_blocks, fmt_array, parse_index, parse_weights, render_space
from .sandbox import Sandbox
Expand Down Expand Up @@ -65,6 +65,7 @@ class Result:
transcript: list[dict]
usage: dict
ws: WeightSpace = field(repr=False)
speed: str | None = None

@property
def forward_used(self) -> int:
Expand All @@ -73,6 +74,7 @@ def forward_used(self) -> int:
def __str__(self) -> str:
conf = f" (confidence {self.confidence:.2f})" if self.confidence is not None else ""
think = f" thinking={self.thinking}" if self.thinking else ""
think += f" speed={self.speed}" if self.speed else ""
return (
f"[{self.mode}] {self.answer}{conf}\n"
f"steps={self.steps}{think} forward passes={self.forward_used}/{self.forward_budget}\n{self.diff}"
Expand All @@ -86,6 +88,7 @@ def save(self, directory: str) -> None:
summary = {
"mode": self.mode,
"thinking": self.thinking,
"speed": self.speed,
"answer": self.answer,
"confidence": self.confidence,
"steps": self.steps,
Expand Down Expand Up @@ -116,6 +119,8 @@ class Session:
python: give the LLM a persistent Python sandbox over the weights
thinking: reasoning of the editing LLM: None = provider default, False/"off" = no thinking
(provider reasoning disabled and the LLM told to act directly), True or "low"/"medium"/"high"
speed: inference speed of the editing LLM: None = provider default, "fast" (or True) = fast mode
(same model, faster output, higher price), "standard", or a provider tier like "ultrafast"
"""

def __init__(
Expand All @@ -136,6 +141,7 @@ def __init__(
sandbox_timeout: float | None = 300,
system_prompt: str | None = None,
thinking: bool | str | None = None,
speed: bool | str | None = None,
log_path: str | None = None,
verbose: bool = False,
):
Expand All @@ -153,6 +159,9 @@ def __init__(
self.thinking = normalize_thinking(thinking)
if self.thinking is not None:
self.llm.thinking = self.thinking
self.speed = normalize_speed(speed)
if self.speed is not None:
self.llm.speed = self.speed
self.system_prompt = system_prompt or prompts.SYSTEM
if self.thinking == "off":
self.system_prompt += prompts.NO_THINKING
Expand Down Expand Up @@ -420,6 +429,7 @@ def _result(self, messages, steps) -> Result:
transcript=messages,
usage=self.usage,
ws=self.ws,
speed=self.speed,
)


Expand Down
3 changes: 3 additions & 0 deletions src/weightspace/cli.py
Original file line number Diff line number Diff line change
Expand Up @@ -31,6 +31,8 @@ def main(argv=None) -> int:
"--thinking", choices=["off", "low", "medium", "high", "xhigh", "max"], help="reasoning of the editing LLM"
)
b.add_argument("--no-thinking", dest="thinking", action="store_const", const="off", help="same as --thinking off")
b.add_argument("--speed", help='inference speed of the editing LLM, e.g. "fast" (provider fast mode)')
b.add_argument("--fast", dest="speed", action="store_const", const="fast", help="same as --speed fast")
b.add_argument("--out", help="directory for transcript, result.json, patch.npz")
b.add_argument("--save-weights", help="write edited weights to .safetensors/.npz")
b.add_argument("-v", "--verbose", action="store_true")
Expand Down Expand Up @@ -58,6 +60,7 @@ def main(argv=None) -> int:
context=context,
max_steps=args.max_steps,
thinking=args.thinking,
speed=args.speed,
verbose=args.verbose,
)
res = s.run_text(args.task) if args.text_mode else s.run(args.task)
Expand Down
4 changes: 2 additions & 2 deletions src/weightspace/llm/__init__.py
Original file line number Diff line number Diff line change
@@ -1,6 +1,6 @@
from __future__ import annotations

from .base import LLM, LLMResponse, ToolCall, normalize_thinking
from .base import LLM, LLMResponse, ToolCall, normalize_speed, normalize_thinking
from .mock import ScriptedLLM, call, reply


Expand Down Expand Up @@ -36,4 +36,4 @@ def get_llm(spec: str | LLM, **kw) -> LLM:
raise ValueError(f"unknown provider {provider!r}")


__all__ = ["normalize_thinking", "LLM", "LLMResponse", "ToolCall", "ScriptedLLM", "call", "reply", "get_llm"]
__all__ = ["normalize_thinking", "normalize_speed", "LLM", "LLMResponse", "ToolCall", "ScriptedLLM", "call", "reply", "get_llm"]
23 changes: 19 additions & 4 deletions src/weightspace/llm/anthropic.py
Original file line number Diff line number Diff line change
Expand Up @@ -3,7 +3,7 @@
import os
from typing import Any

from .base import LLM, LLMResponse, ToolCall, normalize_thinking
from .base import LLM, LLMResponse, ToolCall, normalize_speed, normalize_thinking

THINKING_BUDGET = {"low": 2048, "medium": 8192, "high": 16384, "xhigh": 24576, "max": 32000}

Expand Down Expand Up @@ -59,6 +59,7 @@ def __init__(
client=None,
thinking=None,
thinking_style: str = "auto",
speed=None,
cache: bool = True,
**params,
):
Expand All @@ -72,16 +73,21 @@ def __init__(
self.max_tokens = max_tokens
self.params = params
self.thinking = normalize_thinking(thinking)
self.speed = normalize_speed(speed)
if thinking_style not in ("auto", "adaptive", "budget"):
raise ValueError("thinking_style must be 'auto', 'adaptive' or 'budget'")
self.thinking_style = thinking_style
self.cache = cache
# models with always-on thinking (e.g. Opus 5.5) reject "disabled"; "off" then means lowest effort
self.off_style = "disabled"

def _thinking_kwargs(self, style: str) -> tuple[dict, int]:
t = normalize_thinking(self.thinking)
if t is None:
return {}, self.max_tokens
if t == "off":
if self.off_style == "low_effort":
return {"thinking": {"type": "adaptive"}, "output_config": {"effort": "low"}}, self.max_tokens
return {"thinking": {"type": "disabled"}}, self.max_tokens
if style == "budget":
budget = THINKING_BUDGET[t]
Expand All @@ -94,6 +100,8 @@ def complete(self, messages, tools=None):
# prompt caching: everything up to the newest block is reused on the next step
msgs[-1]["content"][-1] = {**msgs[-1]["content"][-1], "cache_control": {"type": "ephemeral"}}
kw = dict(self.params)
if self.speed:
kw["speed"] = self.speed
if system:
kw["system"] = system
if tools:
Expand All @@ -103,10 +111,14 @@ def complete(self, messages, tools=None):
try:
r = self.client.messages.create(model=self.model, max_tokens=max_tokens, messages=msgs, **kw, **tk)
except Exception as e:
if self.thinking_style != "auto" or "thinking" not in tk or "adaptive" not in str(e):
if tk.get("thinking") == {"type": "disabled"} and "disabled" in str(e):
self.off_style = "low_effort"
elif self.thinking_style == "auto" and "thinking" in tk and "adaptive" in str(e):
self.thinking_style = "budget"
style = "budget"
else:
raise
self.thinking_style = "budget"
tk, max_tokens = self._thinking_kwargs("budget")
tk, max_tokens = self._thinking_kwargs(style)
r = self.client.messages.create(model=self.model, max_tokens=max_tokens, messages=msgs, **kw, **tk)
if self.thinking_style == "auto":
self.thinking_style = style
Expand All @@ -118,6 +130,9 @@ def complete(self, messages, tools=None):
elif b.type == "tool_use":
calls.append(ToolCall(b.id, b.name, dict(b.input)))
usage = r.usage.model_dump() if getattr(r, "usage", None) is not None else {}
thinking_tokens = (usage.get("output_tokens_details") or {}).get("thinking_tokens")
if thinking_tokens is not None:
usage["thinking_tokens"] = thinking_tokens
return LLMResponse(
text="\n".join(text) or None,
tool_calls=calls,
Expand Down
15 changes: 14 additions & 1 deletion src/weightspace/llm/base.py
Original file line number Diff line number Diff line change
Expand Up @@ -58,14 +58,27 @@ def normalize_thinking(value) -> str | None:
return v


def normalize_speed(value) -> str | None:
"""None = provider default; True/"fast" = fast mode; False/"standard"; other tiers pass through (e.g. "ultrafast")."""
if value is None:
return None
if value is True:
return "fast"
if value is False:
return "standard"
return str(value).lower()


class LLM:
"""Implement `complete` to plug in any provider.

`thinking` is set by `Session(thinking=...)`; backends map it to their own reasoning switch.
`thinking` and `speed` are set by `Session(thinking=..., speed=...)`; backends map them to their own
reasoning switch and fast-inference option.
"""

name = "llm"
thinking: str | None = None
speed: str | None = None

def complete(self, messages: list[dict], tools: list[dict] | None = None) -> LLMResponse:
raise NotImplementedError
12 changes: 11 additions & 1 deletion src/weightspace/llm/openai.py
Original file line number Diff line number Diff line change
Expand Up @@ -3,7 +3,7 @@
import json
import os

from .base import LLM, LLMResponse, ToolCall, normalize_thinking
from .base import LLM, LLMResponse, ToolCall, normalize_speed, normalize_thinking


def to_openai_messages(messages: list[dict]) -> list[dict]:
Expand Down Expand Up @@ -43,6 +43,8 @@ class OpenAIChat(LLM):
"openrouter" OpenRouter's unified `reasoning` field
"chat_template" vLLM / SGLang hybrid models (Qwen3...): `chat_template_kwargs.enable_thinking`
None never send anything

speed is sent as `service_tier` ("fast", "ultrafast", ...; "standard" maps to OpenAI's "default").
"""

def __init__(
Expand All @@ -55,6 +57,7 @@ def __init__(
thinking=None,
thinking_style: str | None = "reasoning_effort",
off_effort: str = "none",
speed=None,
**params,
):
if client is None:
Expand All @@ -71,6 +74,11 @@ def __init__(
self.thinking = normalize_thinking(thinking)
self.thinking_style = thinking_style
self.off_effort = off_effort
self.speed = normalize_speed(speed)

def _speed_params(self, kw: dict) -> None:
if self.speed:
kw["service_tier"] = "default" if self.speed == "standard" else self.speed

def _thinking_params(self, kw: dict) -> None:
t, style = normalize_thinking(self.thinking), self.thinking_style
Expand All @@ -93,6 +101,7 @@ def _thinking_params(self, kw: dict) -> None:
def complete(self, messages, tools=None):
kw = dict(self.params)
self._thinking_params(kw)
self._speed_params(kw)
if tools:
kw["tools"] = to_openai_tools(tools)
r = self.client.chat.completions.create(model=self.model, messages=to_openai_messages(messages), **kw)
Expand Down Expand Up @@ -149,6 +158,7 @@ def complete(self, messages, tools=None):
t = normalize_thinking(self.thinking)
if t is not None and self.thinking_style is not None:
kw["reasoning"] = {"effort": self.off_effort if t == "off" else t}
self._speed_params(kw)
if instructions:
kw["instructions"] = instructions
if tools:
Expand Down
6 changes: 6 additions & 0 deletions tests/test_agent.py
Original file line number Diff line number Diff line change
Expand Up @@ -134,3 +134,9 @@ def test_thinking_off_reaches_llm_and_prompt(arrays):
llm = ScriptedLLM([reply(None, call("submit", answer="ok"))])
wsp.read(dict(arrays), "?", llm)
assert llm.thinking_seen == [None]


def test_speed_reaches_llm_and_result(arrays):
llm = ScriptedLLM([reply(None, call("submit", answer="ok"))])
res = wsp.fix(dict(arrays), "fix", llm, speed=True)
assert llm.speed == "fast" and res.speed == "fast" and "speed=fast" in str(res)
25 changes: 24 additions & 1 deletion tests/test_llm.py
Original file line number Diff line number Diff line change
Expand Up @@ -98,6 +98,12 @@ def test_openai_thinking_styles():
assert "reasoning_effort" not in _capture_openai(thinking="off", thinking_style=None)


def test_openai_speed():
assert "service_tier" not in _capture_openai()
assert _capture_openai(speed="fast")["service_tier"] == "fast"
assert _capture_openai(speed=False)["service_tier"] == "default"


def test_anthropic_thinking():
from anthropic.types import Message

Expand All @@ -123,7 +129,10 @@ def create(**kw):

llm = AnthropicChat("x", client=SimpleNamespace(messages=SimpleNamespace(create=create)), max_tokens=4000)
llm.complete([{"role": "user", "content": "hi"}])
assert "thinking" not in seen
assert "thinking" not in seen and "speed" not in seen
llm.speed = "fast"
llm.complete([{"role": "user", "content": "hi"}])
assert seen["speed"] == "fast"
llm.thinking = "off"
llm.complete([{"role": "user", "content": "hi"}])
assert seen["thinking"] == {"type": "disabled"}
Expand All @@ -147,6 +156,20 @@ def create_old(**kw):
old.complete([{"role": "user", "content": "hi"}])
assert len(calls) == 3

sent = []

def create_always_on(**kw):
sent.append(kw)
if kw.get("thinking") == {"type": "disabled"}:
raise ValueError('"thinking.type.disabled" is not supported for this model.')
return msg

on = AnthropicChat("x", client=SimpleNamespace(messages=SimpleNamespace(create=create_always_on)), thinking="off")
on.complete([{"role": "user", "content": "hi"}])
on.complete([{"role": "user", "content": "hi"}])
assert len(sent) == 3 and on.off_style == "low_effort"
assert sent[-1]["thinking"] == {"type": "adaptive"} and sent[-1]["output_config"] == {"effort": "low"}


def test_openresponses_replays_items():
from weightspace.llm import get_llm
Expand Down
Loading