diff --git a/AGENTS.md b/AGENTS.md index 2ae0cea..8c678b6 100644 --- a/AGENTS.md +++ b/AGENTS.md @@ -122,8 +122,8 @@ that makes them stale. This regression has happened twice (#33 and again returns a real verdict with NO `CORA_*` exports in the environment — verified across PRs #164/#166). The 2026-09-18 export requirement is obsolete. -- **2026-10-08:** both `tole-core` and `tole-cli` are on crates.io at - **0.7.2** (released via the GitHub release `v0.7.2`; `publish.yml` +- **2026-10-09:** both `tole-core` and `tole-cli` are on crates.io at + **0.8.0** (released via the GitHub release `v0.8.0`; `publish.yml` published core then cli; checked via the crates.io API). Registry parity with the checkout is current as of this date; live missions still run the freshly built `./target/release/tole` (see CONTRIBUTING → diff --git a/CHANGELOG.md b/CHANGELOG.md index 0dd6f8f..be1ad21 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -7,6 +7,18 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0 ## [Unreleased] +## [0.8.1] — 2026-10-09 + +### Fixed +- The usage report no longer renders an absent per-step `messages` count as a measured `0`; it is `null` in the status JSON when unknown (found by the pre-tag scan). +- `read_file` rejects an empty `path` with a clear message instead of a confusing "is a directory" error. + +### Added +- `tole status` and the serve status JSON now report token usage, cache-hit rate and a per-step request-size split (system+tools vs history, stored per step under `tole_wire` in the usage record); `fact/mission` gains `cached_tokens`. Instrumentation only: the wire, budgets and durable entry shapes are unchanged (#211, part 1 of 3). + +### Changed +- `read_file` now returns at most 20,000 characters per call and accepts `offset`/`limit` (characters); the result carries `truncated`, `total_chars` and `next_offset` so large files are read in chunks instead of being re-sent whole on every later step (#211, part 3 of 3). + ## [0.8.0] — 2026-10-09 ### Added diff --git a/Cargo.lock b/Cargo.lock index be66b07..8ef56df 100644 --- a/Cargo.lock +++ b/Cargo.lock @@ -1427,7 +1427,7 @@ dependencies = [ [[package]] name = "tole-cli" -version = "0.8.0" +version = "0.8.1" dependencies = [ "anyhow", "clap", @@ -1447,7 +1447,7 @@ dependencies = [ [[package]] name = "tole-core" -version = "0.8.0" +version = "0.8.1" dependencies = [ "libc", "rmcp", diff --git a/Cargo.toml b/Cargo.toml index 9ba17b6..954392a 100644 --- a/Cargo.toml +++ b/Cargo.toml @@ -3,7 +3,7 @@ resolver = "2" members = ["crates/tole-core", "crates/tole-cli"] [workspace.package] -version = "0.8.0" +version = "0.8.1" edition = "2021" license = "MIT" repository = "https://github.com/codecoradev/tole" diff --git a/README.md b/README.md index 5fba490..dfe0ce3 100644 --- a/README.md +++ b/README.md @@ -9,10 +9,10 @@ Durable Rust agent harness: a conversational agent with risk-tiered approval gates, a write-once JSONL session log, and a register state machine — resumable after crashes, replayable forever. -**Status:** v0.7.2 released; v0.8.0 is release-candidate on `develop` — it adds -the per-project config `.tole/config.toml` (trusted before it applies; see the -Configuration section) on top of v0.7.x (rescan-2 hardening, one tool-call -authorization gate; see the CHANGELOG for the behavior changes). Four faces on one durable core: the CLI (run / chat / +**Status:** v0.8.1 released — it adds token-usage reporting and a capped, resumable `read_file` (#211) on top of v0.8.0, which added the per-project config `.tole/config.toml` +(trusted before it applies; see the Configuration section) on top of v0.7.x +(rescan-2 hardening, one tool-call authorization gate; see the CHANGELOG for +the behavior changes). Four faces on one durable core: the CLI (run / chat / resume / sessions / jobs / **mission**), `tole mcp` (tool server), `tole acp` (editor agent), and `tole serve` (REST + multi-session MCP-over-HTTP daemon). Identity (owner-approved): a chat-first @@ -93,7 +93,7 @@ tole chat --memory uteke # …with cross-session memory tole chat --resume # pick the thread back up (also: --last) tole resume "next instruction" # continue a settled session tole sessions # list durable sessions -tole status # pc / seq / turns / token usage +tole status # pc / seq / turns / token + cache usage / request-size split ``` Sessions live in `.tole/sessions/.jsonl` (override with @@ -219,7 +219,7 @@ hard error. | Tool | Risk | Notes | |------|------|-------| -| `read_file`, `write_file`, `edit_file` | RO / Write | jailed to `--workspace` (TOCTOU-safe, symlink-refusing) | +| `read_file`, `write_file`, `edit_file` | RO / Write | jailed to `--workspace` (TOCTOU-safe, symlink-refusing); `read_file` returns at most 20,000 chars per call (`offset`/`limit` in chars, `next_offset` to continue) | | `delete_file` | Destructive | always prompts; never allowlistable, even with `--yes` | | `git` | Write | `status` / `diff` / `add` / `commit` only — **push stays human** | | `gh` | Write | read-only ops, argv-validated per op; `repo` defaults to the checkout's GitHub remote, optional per-call override (validated) | @@ -278,7 +278,13 @@ cap; expiry cancels at a checkpoint — resumable, never dead). Budget tiers (#201): conservative defaults (48 steps / 15 min / 200k tokens) with headroom under `--trust internal` (96 / 30 / 500k) — explicit flags always win; a durable cost report (turns, steps, tokens, -per-risk-tier tool calls) lands on the session and `tole status` renders it. +per-risk-tier tool calls, cached tokens) lands on the session and `tole status` renders it. +`tole status` (and `GET /sessions/{id}/status`, as `usage_report`) also shows +the usage derived from the ledger for any session: prompt / completion / +reasoning / cached tokens, the cache-hit rate, and the request size in +characters split into system+tools vs history at the first and last step +(#211; recorded per step under `tole_wire`; sessions recorded before it read +`n/a`). Reported only — `--max-tokens` accounting is unchanged. ## Approval gates diff --git a/crates/tole-cli/Cargo.toml b/crates/tole-cli/Cargo.toml index 83c3d19..6202b37 100644 --- a/crates/tole-cli/Cargo.toml +++ b/crates/tole-cli/Cargo.toml @@ -33,7 +33,7 @@ mcp-http = ["mcp", "shell-tools", "dep:hyper", "dep:hyper-util", "dep:http-body- replay = [] [dependencies] -tole-core = { version = "0.8.0", path = "../tole-core" } +tole-core = { version = "0.8.1", path = "../tole-core" } tokio = { workspace = true } serde_json = { workspace = true } anyhow = { workspace = true } diff --git a/crates/tole-cli/src/lib.rs b/crates/tole-cli/src/lib.rs index 8a8339f..ab0b494 100644 --- a/crates/tole-cli/src/lib.rs +++ b/crates/tole-cli/src/lib.rs @@ -11,3 +11,4 @@ pub mod session_host; pub mod session_tools; pub mod tools; pub mod trust; +pub mod usage_report; diff --git a/crates/tole-cli/src/main.rs b/crates/tole-cli/src/main.rs index 7cca2e8..0fddf30 100644 --- a/crates/tole-cli/src/main.rs +++ b/crates/tole-cli/src/main.rs @@ -2120,6 +2120,11 @@ fn status_command(sessions_dir: &Path, id: &str) -> Result<()> { if let Some(mission) = storage.get_register("fact", "mission") { println!("mission: {}", mission); } + // Issue #211: usage report derived from the ledger (additive block, + // rendered after the pre-existing lines; older sessions read `n/a`). + for line in tole_cli::usage_report::usage_report(usage).render_lines() { + println!("{line}"); + } Ok(()) } diff --git a/crates/tole-cli/src/mission.rs b/crates/tole-cli/src/mission.rs index ba96f3d..893adff 100644 --- a/crates/tole-cli/src/mission.rs +++ b/crates/tole-cli/src/mission.rs @@ -152,7 +152,9 @@ fn run_verify(cmd: &str, timeout: std::time::Duration) -> Result<(), String> { /// Snapshot the durable usage ledger (provider steps so far). fn steps_used(storage: &JsonlStorage) -> u64 { - storage.usages().len() as u64 + // Wire-only rows (#211: request sizes without provider usage) are not + // provider steps — budget semantics are exactly as before. + tole_cli::usage_report::provider_steps(storage.usages()) } /// Tokens (prompt + completion) recorded in the durable ledger. @@ -406,6 +408,10 @@ pub fn run_mission( "turns": turns, "steps": steps_used(&storage), "tokens_in_out": tokens_used(&storage), + // Issue #211: reported only — never counted against any budget. + // `null` when no step reported cached tokens (unknown, not 0). + "cached_tokens": tole_cli::usage_report::usage_report(storage.usages()) + .to_json()["cached_tokens"], "verify_failures": verify_failures, "wall_seconds": wall_secs, "budget": {"max_steps": cfg.max_steps, "max_minutes": cfg.max_minutes, @@ -445,7 +451,75 @@ pub fn run_mission( #[cfg(test)] mod budget_tests { - use super::BudgetTier; + use super::{steps_used, tokens_used, BudgetTier}; + use serde_json::{json, Value}; + use tole_core::entry::Entry; + use tole_core::provider::{Provider, ProviderError, ProviderOutput}; + use tole_core::storage::{JsonlStorage, Storage}; + use tole_core::tool::ToolRegistry; + use tole_core::turn::run_turn; + + /// Reports request sizes (#211) and, optionally, provider usage. + struct Wired { + usage: Option, + } + impl Provider for Wired { + fn complete(&mut self, _t: &[Entry]) -> Result { + Ok(ProviderOutput::Final { text: "ok".into() }) + } + fn last_usage(&self) -> Option { + self.usage.clone() + } + fn last_wire_stats(&self) -> Option { + Some(json!({"system_chars": 1, "tools_chars": 2, "history_chars": 3, "messages": 1})) + } + } + + fn store(tag: &str) -> JsonlStorage { + let d = std::env::temp_dir().join(format!( + "tole-budget-{tag}-{}-{:x}", + std::process::id(), + std::time::SystemTime::now() + .duration_since(std::time::UNIX_EPOCH) + .unwrap() + .as_nanos() + )); + std::fs::create_dir_all(&d).unwrap(); + JsonlStorage::create(&d, "b", None).unwrap() + } + + /// #211 invariant: a gateway that reports NO usage leaves wire-only + /// ledger rows, which must not count as steps or tokens — the step + /// and token budgets behave exactly as before. + #[test] + fn wire_only_rows_do_not_move_step_or_token_budgets() { + let mut s = store("wire-only"); + let reg = ToolRegistry::new(); + let mut p = Wired { usage: None }; + for t in ["a", "b", "c"] { + run_turn(&mut s, &mut p, ®, t).unwrap(); + } + assert_eq!(s.usages().len(), 3, "rows are stored (PR 2 needs them)"); + assert_eq!(steps_used(&s), 0); + assert_eq!(tokens_used(&s), 0); + } + + /// Mixed ledger: only rows with provider usage are steps. + #[test] + fn mixed_ledger_counts_only_provider_usage_steps() { + let mut s = store("mixed"); + let reg = ToolRegistry::new(); + run_turn(&mut s, &mut Wired { usage: None }, ®, "a").unwrap(); + let mut with = Wired { + usage: Some(json!({"prompt_tokens": 10, "completion_tokens": 4})), + }; + run_turn(&mut s, &mut with, ®, "b").unwrap(); + run_turn(&mut s, &mut Wired { usage: None }, ®, "c").unwrap(); + run_turn(&mut s, &mut with, ®, "d").unwrap(); + assert_eq!(s.usages().len(), 4); + assert_eq!(steps_used(&s), 2); + assert_eq!(tokens_used(&s), 28); + } /// Issue #201: explicit flags win per-field; the trusted tier only /// fills where the operator gave nothing. diff --git a/crates/tole-cli/src/serve.rs b/crates/tole-cli/src/serve.rs index 58ab004..d2323a7 100644 --- a/crates/tole-cli/src/serve.rs +++ b/crates/tole-cli/src/serve.rs @@ -544,17 +544,22 @@ fn route(state: &State, method: &str, path: &str, body: &str) -> (u16, serde_jso // blocking status on it would pin the endpoint (and // the map) for the entire turn (CodeCora scan // 2026-09-28). try_lock keeps this non-blocking. - let (busy, entries) = { + let (busy, entries, report) = { let sessions = lock_sessions(&state.sessions); let Some(st) = sessions.map.get(id) else { return (404, json!({"error": "unknown session"})); }; let busy = *st.busy.lock().expect("busy lock"); - let entries = match st.storage.try_lock() { - Ok(guard) => Some(guard.entries().len()), - Err(_) => None, // turn in flight — skip the count + let (entries, report) = match st.storage.try_lock() { + Ok(guard) => ( + Some(guard.entries().len()), + // Issue #211: additive usage report. + tole_cli::usage_report::usage_report(guard.usages()).to_json(), + ), + // turn in flight — skip the count and report + Err(_) => (None, json!(null)), }; - (busy, entries) + (busy, entries, report) }; let entries = match entries { Some(n) => json!(n), @@ -566,6 +571,7 @@ fn route(state: &State, method: &str, path: &str, body: &str) -> (u16, serde_jso "id": id, "entries": entries, "busy": busy, + "usage_report": report, }), ) } diff --git a/crates/tole-cli/src/session_tools.rs b/crates/tole-cli/src/session_tools.rs index 2363d39..f8a9aa0 100644 --- a/crates/tole-cli/src/session_tools.rs +++ b/crates/tole-cli/src/session_tools.rs @@ -381,11 +381,16 @@ impl Tool for SessionStatusTool { use tole_core::storage::Storage; let st = self.0.resolve(&input)?; let busy = *st.busy.lock().expect("busy lock"); - let entries = match st.storage.try_lock() { - Ok(guard) => json!(guard.entries().len()), - Err(_) => json!(null), // turn in flight + let (entries, report) = match st.storage.try_lock() { + Ok(guard) => ( + json!(guard.entries().len()), + // Issue #211: additive usage report (same object as the + // serve status JSON). + crate::usage_report::usage_report(guard.usages()).to_json(), + ), + Err(_) => (json!(null), json!(null)), // turn in flight }; - Ok(json!({ "busy": busy, "entries": entries })) + Ok(json!({ "busy": busy, "entries": entries, "usage_report": report })) } } diff --git a/crates/tole-cli/src/usage_report.rs b/crates/tole-cli/src/usage_report.rs new file mode 100644 index 0000000..509c25a --- /dev/null +++ b/crates/tole-cli/src/usage_report.rs @@ -0,0 +1,432 @@ +//! Usage report derived from the durable usage ledger (issue #211). +//! +//! A pure fold over `UsageRecord`s: shared by `tole status`, the serve +//! status JSON, the session-status tool and the mission `fact/mission` +//! register, so every face reports the same numbers. Read-time only — +//! nothing here writes or changes any budget decision. +//! +//! Tolerant by construction: older sessions have no `tole_wire`, some +//! gateways report no cached tokens, mocks report nothing. Unknown is +//! modelled as `None`/`*_known: false` and rendered `n/a`, never as a +//! measured zero. + +use serde_json::{json, Value}; +use tole_core::storage::UsageRecord; + +/// One step's request-size split, in characters (see +/// `tole_core::openai::WireStats`). +#[derive(Debug, Clone, Copy, PartialEq, Eq)] +pub struct WirePoint { + /// `system_chars + tools_chars`: the stable prefix of the request. + pub prefix_chars: u64, + pub system_chars: u64, + pub tools_chars: u64, + pub history_chars: u64, + /// Message count of the request; `None` when the record does not carry it + /// (unknown is never rendered as a measured zero). + pub messages: Option, +} + +#[derive(Debug, Clone, PartialEq)] +pub struct UsageReport { + /// Provider steps with reported usage (`provider_steps`, same as the mission + /// step budget counts). + pub steps: u64, + pub prompt_tokens: u64, + pub completion_tokens: u64, + /// `completion_tokens_details.reasoning_tokens`; 0 with + /// `reasoning_known: false` when no step reported it. + pub reasoning_tokens: u64, + pub reasoning_known: bool, + /// Cached prompt tokens; 0 with `cached_known: false` when no step + /// reported them. + pub cached_tokens: u64, + pub cached_known: bool, + /// cached / prompt over the steps that reported cached tokens; None + /// when unknown or when those steps' prompt total is 0. + pub cache_hit_rate: Option, + /// First / last ledger step carrying `tole_wire` (None for sessions + /// recorded before it existed). + pub wire_first: Option, + pub wire_last: Option, + /// History growth in characters per step between `wire_first` and + /// `wire_last` (None unless they are different steps). + pub history_growth_per_step: Option, +} + +/// True when the ledger row carries a provider-reported usage object, +/// i.e. it is NOT a wire-only row (`{"tole_wire": ...}` and nothing +/// else, stored when a provider reports request sizes but no usage). +/// The single definition of "provider step" for the mission step budget, +/// `fact/mission.steps` and the report, so wire-only rows never change +/// step accounting (#211). +pub fn reports_provider_usage(rec: &UsageRecord) -> bool { + match rec.usage.as_object() { + Some(o) => !(o.len() == 1 && o.contains_key("tole_wire")), + None => true, + } +} + +/// Number of provider steps in a ledger (see `reports_provider_usage`). +pub fn provider_steps(usages: &[UsageRecord]) -> u64 { + usages.iter().filter(|u| reports_provider_usage(u)).count() as u64 +} + +fn u64_at(v: &Value, path: &[&str]) -> Option { + let mut cur = v; + for k in path { + cur = cur.get(k)?; + } + cur.as_u64() +} + +/// Cached prompt tokens of one step. Preference order: the OpenAI +/// `prompt_tokens_details.cached_tokens`, then the gateway variant +/// `prompt_tokens_details.cached_read_tokens` (both seen in +/// evals/traces/glm), then top-level `cached_tokens`, Anthropic-style +/// `cache_read_input_tokens` and DeepSeek-style `prompt_cache_hit_tokens`. +fn cached_of(u: &Value) -> Option { + u64_at(u, &["prompt_tokens_details", "cached_tokens"]) + .or_else(|| u64_at(u, &["prompt_tokens_details", "cached_read_tokens"])) + .or_else(|| u64_at(u, &["cached_tokens"])) + .or_else(|| u64_at(u, &["cache_read_input_tokens"])) + .or_else(|| u64_at(u, &["prompt_cache_hit_tokens"])) +} + +fn wire_of(u: &Value) -> Option { + let w = u.get("tole_wire")?; + let system_chars = w.get("system_chars")?.as_u64()?; + let tools_chars = w.get("tools_chars")?.as_u64()?; + let history_chars = w.get("history_chars")?.as_u64()?; + Some(WirePoint { + prefix_chars: system_chars.saturating_add(tools_chars), + system_chars, + tools_chars, + history_chars, + messages: w.get("messages").and_then(Value::as_u64), + }) +} + +/// Fold a session's usage ledger into a report. Never panics; missing or +/// malformed fields are treated as unknown. +pub fn usage_report(usages: &[UsageRecord]) -> UsageReport { + let mut r = UsageReport { + steps: provider_steps(usages), + prompt_tokens: 0, + completion_tokens: 0, + reasoning_tokens: 0, + reasoning_known: false, + cached_tokens: 0, + cached_known: false, + cache_hit_rate: None, + wire_first: None, + wire_last: None, + history_growth_per_step: None, + }; + // Prompt tokens of the steps that reported cached tokens (rate + // denominator: a step with no cache info must not dilute the rate). + let mut prompt_of_cached_steps: u64 = 0; + let mut first_idx: Option = None; + let mut last_idx: Option = None; + for (i, rec) in usages.iter().enumerate() { + let u = &rec.usage; + let prompt = u64_at(u, &["prompt_tokens"]).unwrap_or(0); + r.prompt_tokens = r.prompt_tokens.saturating_add(prompt); + r.completion_tokens = r + .completion_tokens + .saturating_add(u64_at(u, &["completion_tokens"]).unwrap_or(0)); + if let Some(x) = u64_at(u, &["completion_tokens_details", "reasoning_tokens"]) { + r.reasoning_known = true; + r.reasoning_tokens = r.reasoning_tokens.saturating_add(x); + } + if let Some(c) = cached_of(u) { + r.cached_known = true; + r.cached_tokens = r.cached_tokens.saturating_add(c); + prompt_of_cached_steps = prompt_of_cached_steps.saturating_add(prompt); + } + if let Some(w) = wire_of(u) { + if r.wire_first.is_none() { + r.wire_first = Some(w); + first_idx = Some(i); + } + r.wire_last = Some(w); + last_idx = Some(i); + } + } + if r.cached_known && prompt_of_cached_steps > 0 { + r.cache_hit_rate = Some(r.cached_tokens as f64 / prompt_of_cached_steps as f64); + } + if let (Some(f), Some(l), Some(fi), Some(li)) = (r.wire_first, r.wire_last, first_idx, last_idx) + { + if li > fi { + r.history_growth_per_step = + Some((l.history_chars as f64 - f.history_chars as f64) / (li - fi) as f64); + } + } + r +} + +fn wire_json(w: &WirePoint) -> Value { + json!({ + "prefix_chars": w.prefix_chars, + "system_chars": w.system_chars, + "tools_chars": w.tools_chars, + "history_chars": w.history_chars, + "messages": w.messages, + }) +} + +impl UsageReport { + /// The additive JSON object served by the status faces. Unknown + /// values are `null` (never a fabricated zero). + pub fn to_json(&self) -> Value { + let wire = match (&self.wire_first, &self.wire_last) { + (Some(f), Some(l)) => json!({ + "first": wire_json(f), + "last": wire_json(l), + "history_growth_per_step": self.history_growth_per_step, + }), + _ => Value::Null, + }; + json!({ + "steps": self.steps, + "prompt_tokens": self.prompt_tokens, + "completion_tokens": self.completion_tokens, + "reasoning_tokens": if self.reasoning_known { json!(self.reasoning_tokens) } else { Value::Null }, + "cached_tokens": if self.cached_known { json!(self.cached_tokens) } else { Value::Null }, + "cache_hit_rate": self.cache_hit_rate, + "wire": wire, + }) + } + + /// Human lines for `tole status`, rendered AFTER the pre-existing + /// lines. Unknown values read `n/a`. + pub fn render_lines(&self) -> Vec { + let reasoning = if self.reasoning_known { + self.reasoning_tokens.to_string() + } else { + "n/a".to_string() + }; + let cached = if self.cached_known { + self.cached_tokens.to_string() + } else { + "n/a".to_string() + }; + let rate = match self.cache_hit_rate { + Some(x) => format!("{:.1}%", x * 100.0), + None => "n/a".to_string(), + }; + let mut lines = vec![ + format!("steps: {}", self.steps), + format!( + "tokens: {} prompt / {} completion / {reasoning} reasoning / {cached} cached (hit rate {rate})", + self.prompt_tokens, self.completion_tokens + ), + ]; + match (&self.wire_first, &self.wire_last) { + (Some(f), Some(l)) => { + let point = |w: &WirePoint| { + format!( + "{} prefix (system+tools) + {} history chars", + w.prefix_chars, w.history_chars + ) + }; + lines.push(format!("wire: first step: {}", point(f))); + lines.push(format!(" last step: {}", point(l))); + lines.push(match self.history_growth_per_step { + Some(g) => format!(" history growth: {g:+.1} chars/step"), + None => " history growth: n/a".to_string(), + }); + } + _ => lines.push("wire: n/a (no per-step request size recorded)".to_string()), + } + lines + } +} + +#[cfg(test)] +mod tests { + use super::*; + + fn rec(usage: Value) -> UsageRecord { + UsageRecord { + id: String::new(), + entry_id: "e".into(), + usage, + cost_usd: None, + } + } + + fn wire(sys: u64, tools: u64, hist: u64) -> Value { + json!({"system_chars": sys, "tools_chars": tools, "history_chars": hist, "messages": 3}) + } + + #[test] + fn no_usages_is_all_unknown_and_does_not_panic() { + let r = usage_report(&[]); + assert_eq!(r.steps, 0); + assert!(!r.cached_known && !r.reasoning_known); + assert_eq!(r.cache_hit_rate, None); + assert!(r.wire_first.is_none() && r.wire_last.is_none()); + let text = r.render_lines().join("\n"); + assert!(text.contains("steps: 0")); + assert!(text.contains("hit rate n/a")); + assert!(text.contains("wire: n/a")); + let j = r.to_json(); + assert_eq!(j["steps"], json!(0)); + assert_eq!(j["cached_tokens"], Value::Null); + assert_eq!(j["wire"], Value::Null); + } + + #[test] + fn usages_without_tole_wire_render_na_not_zero() { + let r = usage_report(&[ + rec(json!({"prompt_tokens": 100, "completion_tokens": 10})), + rec(json!({"prompt_tokens": 50, "completion_tokens": 5})), + ]); + assert_eq!(r.steps, 2); + assert_eq!((r.prompt_tokens, r.completion_tokens), (150, 15)); + assert!(r.wire_first.is_none()); + let text = r.render_lines().join("\n"); + assert!(text.contains("wire: n/a"), "{text}"); + assert!(text.contains("n/a reasoning / n/a cached"), "{text}"); + assert_eq!(r.to_json()["wire"], Value::Null); + } + + #[test] + fn wire_first_last_and_growth() { + let r = usage_report(&[ + rec(json!({"prompt_tokens": 10, "tole_wire": wire(100, 900, 50)})), + rec(json!({"prompt_tokens": 10, "tole_wire": wire(100, 900, 150)})), + rec(json!({"prompt_tokens": 10, "tole_wire": wire(100, 900, 250)})), + ]); + let (f, l) = (r.wire_first.unwrap(), r.wire_last.unwrap()); + assert_eq!((f.prefix_chars, f.history_chars), (1000, 50)); + assert_eq!((l.prefix_chars, l.history_chars), (1000, 250)); + assert_eq!(r.history_growth_per_step, Some(100.0)); + let j = r.to_json(); + assert_eq!(j["wire"]["first"]["history_chars"], json!(50)); + assert_eq!(j["wire"]["last"]["prefix_chars"], json!(1000)); + assert_eq!(j["wire"]["history_growth_per_step"], json!(100.0)); + assert!(r.render_lines().join("\n").contains("+100.0 chars/step")); + } + + #[test] + fn wire_only_rows_do_not_count_as_provider_steps() { + let wire_only = rec(json!({"tole_wire": wire(1, 2, 3)})); + assert!(!reports_provider_usage(&wire_only)); + assert!(reports_provider_usage(&rec(json!({"prompt_tokens": 1})))); + assert!( + reports_provider_usage(&rec(json!({}))), + "empty object counted before #211" + ); + assert!(reports_provider_usage(&rec( + json!({"prompt_tokens": 1, "tole_wire": wire(1, 2, 3)}) + ))); + let mixed = [ + rec(json!({"prompt_tokens": 5, "tole_wire": wire(1, 2, 3)})), + wire_only.clone(), + rec(json!({"prompt_tokens": 7})), + ]; + assert_eq!(provider_steps(&mixed), 2); + let r = usage_report(&mixed); + assert_eq!(r.steps, 2); + assert_eq!(r.prompt_tokens, 12); + // Wire points still span ALL rows that carry tole_wire. + assert!(r.wire_first.is_some() && r.wire_last.is_some()); + } + + #[test] + fn single_wire_step_has_no_growth() { + let r = usage_report(&[rec(json!({"tole_wire": wire(1, 2, 3)}))]); + assert!(r.wire_first.is_some()); + assert_eq!(r.history_growth_per_step, None); + assert!(r.render_lines().join("\n").contains("growth: n/a")); + // A wire-only record carries no tokens: they read 0 / unknown. + assert_eq!((r.prompt_tokens, r.completion_tokens), (0, 0)); + assert!(!r.cached_known); + } + + #[test] + fn records_without_wire_between_wired_ones_use_real_step_distance() { + // Wire on steps 0 and 3 (the middle ones predate/lack it). + let r = usage_report(&[ + rec(json!({"tole_wire": wire(0, 0, 10)})), + rec(json!({"prompt_tokens": 1})), + rec(json!({"prompt_tokens": 1})), + rec(json!({"tole_wire": wire(0, 0, 40)})), + ]); + assert_eq!(r.history_growth_per_step, Some(10.0)); + } + + #[test] + fn cached_tokens_under_each_key_name() { + let cases = [ + json!({"prompt_tokens": 100, "prompt_tokens_details": {"cached_tokens": 40}}), + json!({"prompt_tokens": 100, "prompt_tokens_details": {"cached_read_tokens": 40}}), + json!({"prompt_tokens": 100, "cached_tokens": 40}), + json!({"prompt_tokens": 100, "cache_read_input_tokens": 40}), + json!({"prompt_tokens": 100, "prompt_cache_hit_tokens": 40}), + ]; + for c in cases { + let r = usage_report(&[rec(c.clone())]); + assert!(r.cached_known, "{c}"); + assert_eq!(r.cached_tokens, 40, "{c}"); + assert_eq!(r.cache_hit_rate, Some(0.4), "{c}"); + } + } + + #[test] + fn cached_key_preference_and_real_trace_shape() { + // Shape archived in evals/traces/glm: both keys, same value; + // cached_tokens wins when they disagree. + let r = usage_report(&[rec(json!({ + "prompt_tokens": 3638, + "completion_tokens": 36, + "completion_tokens_details": {"reasoning_tokens": 22}, + "prompt_tokens_details": {"cache_write_tokens": 0, + "cached_read_tokens": 3584, "cached_tokens": 3500}, + }))]); + assert_eq!(r.cached_tokens, 3500); + assert_eq!(r.reasoning_tokens, 22); + assert!(r.reasoning_known); + } + + #[test] + fn rate_ignores_steps_without_cache_info_and_missing_prompt() { + let r = usage_report(&[ + rec(json!({"prompt_tokens": 1000})), // no cache info + rec(json!({"prompt_tokens": 100, "cached_tokens": 50})), + rec(json!({"cached_tokens": 5})), // no prompt tokens + ]); + assert_eq!(r.prompt_tokens, 1100); + assert_eq!(r.cached_tokens, 55); + // 55 cached over the 100 prompt tokens of the steps that reported. + assert_eq!(r.cache_hit_rate, Some(0.55)); + // Zero prompt among reporting steps: no rate (no division by 0). + let z = usage_report(&[rec(json!({"cached_tokens": 5}))]); + assert!(z.cached_known); + assert_eq!(z.cache_hit_rate, None); + } + + #[test] + fn malformed_fields_are_ignored() { + let r = usage_report(&[ + rec(json!({"prompt_tokens": "many", "tole_wire": {"system_chars": 1}})), + rec(json!("not an object")), + ]); + assert_eq!(r.steps, 2); + assert_eq!(r.prompt_tokens, 0); + assert!(r.wire_first.is_none()); + } + + #[test] + fn missing_messages_is_unknown_not_zero() { + let r = usage_report(&[rec( + json!({"tole_wire": {"system_chars": 1, "tools_chars": 2, "history_chars": 3}}), + )]); + let w = r.wire_first.expect("wire point without messages is kept"); + assert_eq!(w.messages, None); + assert_eq!(wire_json(&w)["messages"], Value::Null); + } +} diff --git a/crates/tole-cli/tests/mission_e2e.rs b/crates/tole-cli/tests/mission_e2e.rs index 9f2f3ff..7133cdf 100644 --- a/crates/tole-cli/tests/mission_e2e.rs +++ b/crates/tole-cli/tests/mission_e2e.rs @@ -132,7 +132,8 @@ fn spawn_mock(require_file: std::path::PathBuf) -> String { out.push_str(&format!( "data: {}\n\n", json!({"choices": [], "usage": {"prompt_tokens": 1, - "completion_tokens": 1, "total_tokens": 2}}) + "completion_tokens": 1, "total_tokens": 2, + "prompt_tokens_details": {"cached_tokens": 1}}}) )); out.push_str("data: [DONE]\n\n"); format!( @@ -392,3 +393,63 @@ fn token_budget_and_cost_report_visible_in_status() { ); let _ = std::fs::remove_dir_all(&dir); } + +/// #211: the settle-time `fact/mission` register gains an additive +/// `cached_tokens`, and the budget-facing fields are unchanged: every +/// mock step reports prompt 1 + completion 1 (cached 1), so +/// `tokens_in_out == 2 * steps` and `cached_tokens == steps`. +#[test] +fn mission_register_reports_cached_tokens_without_touching_budget_fields() { + let base = spawn_mock("NONE".into()); + let dir = temp_dir("cached"); + let env = [ + ("TOLE_BASE_URL".to_string(), base), + ("TOLE_MODEL".to_string(), "mock-model".to_string()), + ("TOLE_API_KEY".to_string(), "sk-test".to_string()), + ]; + let env_ref: Vec<(&str, &str)> = env.iter().map(|(k, v)| (k.as_str(), v.as_str())).collect(); + let sessions = dir.join("sessions"); + let (out, err, code) = run_tole( + &env_ref, + &[ + "mission", + "-s", + sessions.to_str().unwrap(), + "--yes", + "--max-steps", + "12", + "accomplish the thing", + ], + ); + assert_eq!(code, 0, "mission failed: {err}"); + let session_id = out + .lines() + .find_map(|l| l.strip_prefix("mission: ")) + .expect("session id") + .to_string(); + let (status_out, status_err, status_code) = run_tole( + &env_ref, + &["status", "-s", sessions.to_str().unwrap(), &session_id], + ); + assert_eq!(status_code, 0, "{status_err}"); + let register: Value = serde_json::from_str( + status_out + .lines() + .find_map(|l| l.strip_prefix("mission: ")) + .expect("mission register line"), + ) + .expect("register is JSON"); + let steps = register["steps"].as_u64().unwrap(); + assert!(steps >= 2, "{register}"); + assert_eq!(register["tokens_in_out"], json!(2 * steps), "{register}"); + assert_eq!(register["cached_tokens"], json!(steps), "{register}"); + // The real binary also stores the wire split: the status block + // renders it (first/last step, not n/a). + assert!( + status_out.contains(&format!("steps: {steps}")), + "{status_out}" + ); + assert!(status_out.contains("first step:"), "{status_out}"); + assert!(status_out.contains("(hit rate 100.0%)"), "{status_out}"); + let _ = std::fs::remove_dir_all(&dir); +} diff --git a/crates/tole-cli/tests/read_file_chunks_e2e.rs b/crates/tole-cli/tests/read_file_chunks_e2e.rs new file mode 100644 index 0000000..a9d06e1 --- /dev/null +++ b/crates/tole-cli/tests/read_file_chunks_e2e.rs @@ -0,0 +1,182 @@ +//! #211 part 3 — the model-facing `read_file` continuation contract, +//! end to end with the REAL binary and a scripted mock provider: a +//! 50,000-char file is read in 3 chunks by following `next_offset`, and +//! the mission concludes. No network. + +use serde_json::{json, Value}; +use std::io::{Read, Write}; +use std::net::TcpListener; +use std::process::{Command, Stdio}; +use std::sync::{Arc, Mutex}; + +/// Scripted mock: while the latest tool result carries `"next_offset":N` +/// (or there is no tool result yet) it asks for `read_file` at that +/// offset; once the result is not truncated it gives the final answer. +/// Every requested offset is recorded in `offsets`. +fn spawn_mock(offsets: Arc>>) -> String { + let listener = TcpListener::bind("127.0.0.1:0").unwrap(); + let addr = listener.local_addr().unwrap(); + std::thread::spawn(move || { + for stream in listener.incoming().flatten() { + let offsets = offsets.clone(); + std::thread::spawn(move || { + let mut s = stream; + let mut buf = Vec::new(); + let mut chunk = [0u8; 16384]; + let mut body = Value::Null; + let deadline = std::time::Instant::now() + std::time::Duration::from_secs(10); + while std::time::Instant::now() < deadline { + let n = s.read(&mut chunk).unwrap_or(0); + if n > 0 { + buf.extend_from_slice(&chunk[..n]); + } + if let Some(h) = buf.windows(4).position(|w| w == b"\r\n\r\n") { + if let Ok(v) = serde_json::from_slice::(&buf[h + 4..]) { + body = v; + break; + } + } + if n == 0 { + break; + } + } + let streamed = body.get("stream").and_then(Value::as_bool) == Some(true); + let msgs = body["messages"].as_array().cloned().unwrap_or_default(); + let last_tool = msgs + .iter() + .rev() + .take_while(|m| m["role"] != "user") + .find(|m| m["role"] == "tool") + .and_then(|m| m["content"].as_str()) + .map(str::to_string); + // Next offset to request, or None when the read is done. + let next: Option = match &last_tool { + None => Some(0), + Some(t) => t.split("\"next_offset\":").nth(1).map(|r| { + r.trim_start() + .chars() + .take_while(char::is_ascii_digit) + .collect::() + .parse() + .expect("next_offset digits") + }), + }; + let msg = match next { + Some(off) => { + offsets.lock().unwrap().push(off); + let args = if off == 0 { + json!({ "path": "big.txt" }) + } else { + json!({ "path": "big.txt", "offset": off }) + }; + json!({"role": "assistant", "content": null, + "tool_calls": [{"id": format!("call_{off}"), "type": "function", + "function": {"name": "read_file", "arguments": args.to_string()}}]}) + } + None => { + let done = last_tool + .unwrap_or_default() + .contains("\"truncated\":false"); + json!({"role": "assistant", "content": format!("CHUNKS_DONE untruncated={done}")}) + } + }; + let finish = if next.is_some() { "tool_calls" } else { "stop" }; + let usage = json!({"prompt_tokens": 1, "completion_tokens": 1, "total_tokens": 2}); + let resp = if streamed { + let mut out = String::new(); + let delta = if let Some(tcs) = msg["tool_calls"].as_array() { + json!({"tool_calls": [{"index": 0, "id": tcs[0]["id"], + "function": {"name": tcs[0]["function"]["name"], + "arguments": tcs[0]["function"]["arguments"]}}]}) + } else { + json!({"content": msg["content"]}) + }; + out.push_str(&format!( + "data: {}\n\n", + json!({"choices": [{"index": 0, "delta": delta, "finish_reason": null}]}) + )); + out.push_str(&format!( + "data: {}\n\n", + json!({"choices": [], "usage": usage}) + )); + out.push_str("data: [DONE]\n\n"); + format!( + "HTTP/1.1 200 OK\r\nContent-Type: text/event-stream\r\nContent-Length: {}\r\nConnection: close\r\n\r\n{}", + out.len(), out + ) + } else { + let data = json!({ + "id": "c", "object": "chat.completion", "created": 1, "model": "mock", + "choices": [{"index": 0, "message": msg, "finish_reason": finish}], + "usage": usage + }) + .to_string(); + format!( + "HTTP/1.1 200 OK\r\nContent-Type: application/json\r\nContent-Length: {}\r\nConnection: close\r\n\r\n{}", + data.len(), data + ) + }; + let _ = s.write_all(resp.as_bytes()); + }); + } + }); + format!("http://{addr}/v1") +} + +#[test] +fn mission_reads_a_50k_char_file_in_three_chunks() { + let offsets = Arc::new(Mutex::new(Vec::new())); + let base = spawn_mock(offsets.clone()); + let dir = std::env::temp_dir().join(format!( + "tole-readfile-e2e-{}-{:x}", + std::process::id(), + std::time::SystemTime::now() + .duration_since(std::time::UNIX_EPOCH) + .unwrap() + .as_nanos() + )); + let ws = dir.join("ws"); + std::fs::create_dir_all(&ws).unwrap(); + // 50,000 chars including multibyte ones (so bytes != chars). + let text: String = "abcdé日😀".chars().cycle().take(50_000).collect(); + assert_eq!(text.chars().count(), 50_000); + std::fs::write(ws.join("big.txt"), &text).unwrap(); + + let out = Command::new(env!("CARGO_BIN_EXE_tole")) + .args([ + "--workspace", + ws.to_str().unwrap(), + "run", + "-s", + dir.join("sessions").to_str().unwrap(), + "--yes", + "--", + "read big.txt completely", + ]) + .stdin(Stdio::null()) + .stdout(Stdio::piped()) + .stderr(Stdio::piped()) + .env_remove("TOLE_MEMORY") + .env_remove("TOLE_MEMORY_NAMESPACE") + .env_remove("TOLE_SYSTEM_PROMPT") + .env_remove("TOLE_TRUST") + .env_remove("TOLE_AGENT_DEPTH") + .env("TOLE_BASE_URL", &base) + .env("TOLE_MODEL", "mock-model") + .env("TOLE_API_KEY", "sk-test") + .output() + .expect("run tole"); + let stdout = String::from_utf8_lossy(&out.stdout); + let stderr = String::from_utf8_lossy(&out.stderr); + assert_eq!(out.status.code(), Some(0), "run failed: {stderr}"); + assert!( + stdout.contains("CHUNKS_DONE untruncated=true"), + "mission must conclude after the last chunk: {stdout}" + ); + assert_eq!( + *offsets.lock().unwrap(), + vec![0, 20_000, 40_000], + "the model follows next_offset: 3 chunks, 20k each" + ); + let _ = std::fs::remove_dir_all(&dir); +} diff --git a/crates/tole-cli/tests/serve_session_cancel.rs b/crates/tole-cli/tests/serve_session_cancel.rs index 0dda8e6..bb8260c 100644 --- a/crates/tole-cli/tests/serve_session_cancel.rs +++ b/crates/tole-cli/tests/serve_session_cancel.rs @@ -298,3 +298,46 @@ fn serve_cancel_mid_provider_settles_cancelled_and_resumable() { assert_eq!(code, 200, "{resp}"); assert_eq!(resp["stopReason"].as_str(), Some("end_turn"), "{resp}"); } + +/// #211: `GET /sessions/{id}/status` gains an ADDITIVE `usage_report` +/// object; the pre-existing keys stay. A fresh session reports zero +/// steps and `null` for what was never measured; after a turn the real +/// provider's per-step wire split is there. +#[test] +fn serve_status_json_has_additive_usage_report() { + let base = spawn_mock(); + let serve = ServeProcess::spawn(&base); + let (_, created) = serve.request("POST", "/sessions", "{}").unwrap(); + let sid = created["sessionId"].as_str().unwrap().to_string(); + + let (code, st) = serve + .request("GET", &format!("/sessions/{sid}/status"), "") + .unwrap(); + assert_eq!(code, 200, "{st}"); + assert_eq!(st["id"], json!(sid)); + assert!(st.get("entries").is_some() && st.get("busy").is_some()); + assert_eq!(st["usage_report"]["steps"], json!(0), "{st}"); + assert_eq!(st["usage_report"]["cached_tokens"], Value::Null, "{st}"); + assert_eq!(st["usage_report"]["wire"], Value::Null, "{st}"); + + let (code, resp) = serve + .request( + "POST", + &format!("/sessions/{sid}/prompt"), + r#"{"text": "quick"}"#, + ) + .unwrap(); + assert_eq!(code, 200, "{resp}"); + + let (_, st) = serve + .request("GET", &format!("/sessions/{sid}/status"), "") + .unwrap(); + let r = &st["usage_report"]; + assert_eq!(r["steps"], json!(1), "{st}"); + assert_eq!(r["prompt_tokens"], json!(1), "{st}"); + assert_eq!(r["completion_tokens"], json!(1), "{st}"); + let first = &r["wire"]["first"]; + assert!(first["history_chars"].as_u64().unwrap() > 0, "{st}"); + assert!(first["prefix_chars"].as_u64().unwrap() > 0, "{st}"); + assert_eq!(r["wire"]["history_growth_per_step"], Value::Null, "{st}"); +} diff --git a/crates/tole-cli/tests/session_tools.rs b/crates/tole-cli/tests/session_tools.rs index 5707d7f..5700806 100644 --- a/crates/tole-cli/tests/session_tools.rs +++ b/crates/tole-cli/tests/session_tools.rs @@ -59,6 +59,9 @@ fn new_list_status_roundtrip() { .execute(json!({"session_id": sid_a})) .unwrap(); assert_eq!(st["busy"], json!(false)); + // #211: additive usage report object (fresh session: no steps). + assert_eq!(st["usage_report"]["steps"], json!(0), "{st}"); + assert_eq!(st["usage_report"]["wire"], json!(null), "{st}"); } #[test] diff --git a/crates/tole-cli/tests/tool_specs.rs b/crates/tole-cli/tests/tool_specs.rs index d7eb842..b0d4ad9 100644 --- a/crates/tole-cli/tests/tool_specs.rs +++ b/crates/tole-cli/tests/tool_specs.rs @@ -136,3 +136,30 @@ fn git_and_systemone_summaries_are_static() { assert!(git.describe(&Value::Null).contains("'op' must be one of")); assert!(git.describe(&json!({"op": "status"})).contains("status")); } + +/// Issue #211 (PR 2): loose CI bound on the size of the FULL default +/// registry's `tools` array as it goes over the wire (characters of its +/// compact JSON, counted by the same `wire_stats` the usage ledger uses). +/// Measured 2026-10-09: 13,879 chars for 27 tools (no `web_search`: its +/// key is not set in CI). Bound = ceil(measured * 1.25) rounded up. A +/// bump needs a written justification in the PR, like the mission bounds +/// in `crates/tole-core/tests/wire_size_missions.rs`. +const FULL_REGISTRY_TOOLS_CHARS_BOUND: u64 = 18_000; + +#[test] +fn full_registry_tools_chars_within_bound() { + let reg = full_registry(); + let specs = reg.specs(); + let w = tole_core::openai::wire_stats(&json!({ "tools": specs, "messages": [] })); + println!( + "MEASURED full_registry: tools={} tools_chars={}", + specs.len(), + w.tools_chars + ); + assert!( + w.tools_chars <= FULL_REGISTRY_TOOLS_CHARS_BOUND, + "full-registry tool specs are {} chars, bound {FULL_REGISTRY_TOOLS_CHARS_BOUND}; \ + re-baseline only with a justification (see the constant's doc)", + w.tools_chars + ); +} diff --git a/crates/tole-cli/tests/usage_report_status.rs b/crates/tole-cli/tests/usage_report_status.rs new file mode 100644 index 0000000..ee19571 --- /dev/null +++ b/crates/tole-cli/tests/usage_report_status.rs @@ -0,0 +1,131 @@ +//! #211 `tole status` usage block, end-to-end with the REAL binary on a +//! fixture session: the pre-existing lines are unchanged and the new +//! block appears after them (`n/a` for sessions recorded before +//! `tole_wire` existed). + +use serde_json::{json, Value}; +use std::process::{Command, Stdio}; +use tole_core::entry::Entry; +use tole_core::provider::{Provider, ProviderError, ProviderOutput}; +use tole_core::storage::JsonlStorage; +use tole_core::tool::ToolRegistry; +use tole_core::turn::run_turn; + +/// Scripted provider reporting fixed usage and (optionally) a growing +/// wire breakdown. +struct Fixture { + step: usize, + usage: Value, + wire: bool, +} + +impl Provider for Fixture { + fn complete(&mut self, _t: &[Entry]) -> Result { + self.step += 1; + Ok(ProviderOutput::Final { + text: "ok".to_string(), + }) + } + fn last_usage(&self) -> Option { + Some(self.usage.clone()) + } + fn last_wire_stats(&self) -> Option { + self.wire.then(|| { + json!({"system_chars": 100, "tools_chars": 900, + "history_chars": 40 * self.step as u64, "messages": 2}) + }) + } +} + +fn tmp(tag: &str) -> std::path::PathBuf { + let d = std::env::temp_dir().join(format!( + "tole-usage-report-{tag}-{}-{:x}", + std::process::id(), + std::time::SystemTime::now() + .duration_since(std::time::UNIX_EPOCH) + .unwrap() + .as_nanos() + )); + let _ = std::fs::remove_dir_all(&d); + std::fs::create_dir_all(&d).unwrap(); + d +} + +fn status(dir: &std::path::Path, id: &str) -> String { + let out = Command::new(env!("CARGO_BIN_EXE_tole")) + .args(["status", "-s", dir.to_str().unwrap(), id]) + .stdin(Stdio::null()) + .output() + .expect("run tole status"); + assert!( + out.status.success(), + "{}", + String::from_utf8_lossy(&out.stderr) + ); + String::from_utf8_lossy(&out.stdout).to_string() +} + +fn fixture(dir: &std::path::Path, id: &str, wire: bool, usage: Value) { + let mut s = JsonlStorage::create(dir, id, None).unwrap(); + let mut p = Fixture { + step: 0, + usage, + wire, + }; + let reg = ToolRegistry::new(); + run_turn(&mut s, &mut p, ®, "hello").unwrap(); + run_turn(&mut s, &mut p, ®, "again").unwrap(); +} + +#[test] +fn old_style_session_keeps_old_lines_and_shows_na_block() { + let dir = tmp("old"); + fixture( + &dir, + "old", + false, + json!({"prompt_tokens": 100, "completion_tokens": 10}), + ); + let out = status(&dir, "old"); + let (head, _) = out.split_once("usage:").expect("usage line"); + // The pre-existing lines, byte for byte, in their original order. + assert!(head.starts_with("session: old\nentries: "), "{out}"); + assert!(head.contains("\npc: Final\n"), "{out}"); + assert!( + head.ends_with("turns: 2 (user messages)\nsystem: -\n"), + "{out}" + ); + let legacy_usage = "usage: 200 in / 20 out tokens, $0.0000 USD\n"; + let block = out.split_once(legacy_usage).expect("legacy usage line").1; + assert!(block.contains("steps: 2"), "{block}"); + assert!(block.contains("200 prompt / 20 completion"), "{block}"); + assert!(block.contains("n/a cached (hit rate n/a)"), "{block}"); + assert!(block.contains("wire: n/a"), "{block}"); + assert!(!block.contains("0 prefix"), "no fabricated zeros: {block}"); + let _ = std::fs::remove_dir_all(&dir); +} + +#[test] +fn session_with_tole_wire_shows_split_cache_rate_and_growth() { + let dir = tmp("new"); + fixture( + &dir, + "new", + true, + json!({"prompt_tokens": 100, "completion_tokens": 10, + "prompt_tokens_details": {"cached_tokens": 50}}), + ); + let out = status(&dir, "new"); + assert!(out.contains("usage: 200 in / 20 out tokens"), "{out}"); + assert!(out.contains("100 cached (hit rate 50.0%)"), "{out}"); + assert!( + out.contains("first step: 1000 prefix (system+tools) + 40 history chars"), + "{out}" + ); + assert!( + out.contains("last step: 1000 prefix (system+tools) + 80 history chars"), + "{out}" + ); + assert!(out.contains("history growth: +40.0 chars/step"), "{out}"); + let _ = std::fs::remove_dir_all(&dir); +} diff --git a/crates/tole-core/src/openai.rs b/crates/tole-core/src/openai.rs index b851bfe..db2dd48 100644 --- a/crates/tole-core/src/openai.rs +++ b/crates/tole-core/src/openai.rs @@ -12,6 +12,69 @@ use crate::provider::{Provider, ProviderError, ProviderOutput}; use serde_json::{json, Value}; use std::time::Duration; +/// Serialized size of one provider request, in CHARACTERS (Unicode +/// scalar values, `chars().count()`, not bytes: tokenizer-free and +/// independent of the script the transcript is written in). Issue #211. +/// +/// - `system_chars`: the text of the leading `role:"system"` message's +/// `content` (0 when there is none). +/// - `tools_chars`: the compact JSON serialization of the `tools` array +/// (0 when the body has none). +/// - `history_chars`: the compact JSON serialization of every other +/// message (role/envelope included), summed. +/// - `messages`: the number of messages in the body, system included. +#[derive(Debug, Clone, Copy, PartialEq, Eq)] +pub struct WireStats { + pub system_chars: u64, + pub tools_chars: u64, + pub history_chars: u64, + pub messages: u64, +} + +impl WireStats { + /// The stored shape, merged into the usage record under `tole_wire`. + pub fn to_value(&self) -> Value { + json!({ + "system_chars": self.system_chars, + "tools_chars": self.tools_chars, + "history_chars": self.history_chars, + "messages": self.messages, + }) + } +} + +/// Measure a request body as built by `OpenAiProvider::request_body` +/// (before the `stream`/`stream_options` envelope). Pure: the body is +/// only read. +pub fn wire_stats(body: &Value) -> WireStats { + let chars = |v: &Value| v.to_string().chars().count() as u64; + let msgs = body + .get("messages") + .and_then(Value::as_array) + .map(Vec::as_slice) + .unwrap_or(&[]); + let mut system_chars = 0; + let mut history_chars = 0; + for (i, m) in msgs.iter().enumerate() { + if i == 0 && m.get("role").and_then(Value::as_str) == Some("system") { + system_chars = m + .get("content") + .and_then(Value::as_str) + .map(|s| s.chars().count() as u64) + .unwrap_or(0); + } else { + history_chars += chars(m); + } + } + let tools_chars = body.get("tools").map(chars).unwrap_or(0); + WireStats { + system_chars, + tools_chars, + history_chars, + messages: msgs.len() as u64, + } +} + /// A live-token callback (issue #196 phase 3): invoked per reasoning/ /// content burst while a streamed completion is in flight. The turn /// loop never sees partial data — `complete()` still returns one @@ -132,6 +195,9 @@ pub struct OpenAiProvider { /// The model's reasoning from the last step (issue #196 phase 2): /// captured for the turn-loop observer; never durable. last_reasoning_obj: Option, + /// Request-size breakdown of the last call (issue #211), reset per + /// call like usage/reasoning. + last_wire_stats_obj: Option, /// Live-token sinks (issue #196 phase 3): present on host faces /// that render progress (ACP). When set, reasoning arrives via /// deltas and `last_reasoning` stays None — no double emission. @@ -169,6 +235,7 @@ impl OpenAiProvider { tool_specs: Vec::new(), last_usage_obj: None, last_reasoning_obj: None, + last_wire_stats_obj: None, on_text_delta: None, on_reasoning_delta: None, streaming: std::env::var("TOLE_STREAM") @@ -563,7 +630,11 @@ impl Provider for OpenAiProvider { // then sets what it actually has. self.last_usage_obj = None; self.last_reasoning_obj = None; + self.last_wire_stats_obj = None; let mut body = self.request_body(transcript); + // Issue #211: measure BEFORE the streaming envelope is layered + // on — the same body the Tier-1 golden pins. Read-only. + self.last_wire_stats_obj = Some(wire_stats(&body).to_value()); // Issue #196 phase 3: the streaming envelope rides on top of the // request body — the Tier-1 golden contract keeps asserting the // MESSAGE payload, which is what cache-friendliness depends on. @@ -628,6 +699,10 @@ impl Provider for OpenAiProvider { fn last_reasoning(&self) -> Option { self.last_reasoning_obj.clone() } + + fn last_wire_stats(&self) -> Option { + self.last_wire_stats_obj.clone() + } } /// Delimit untrusted tool output on the wire (threat-model: prompt @@ -1177,6 +1252,7 @@ mod models_probe_tests { #[cfg(test)] mod streaming_tests { use super::*; + use crate::entry::{Entry, EntryType}; use std::sync::{Arc, Mutex}; /// One-shot SSE mock: serves the given pre-built stream body and @@ -1451,4 +1527,169 @@ mod streaming_tests { "the escape hatch must be able to turn streaming off" ); } + + // ---- Issue #211: wire stats -------------------------------------- + + fn user_entry(id: &str, text: &str) -> Entry { + Entry { + id: id.into(), + parent_id: None, + seq: 0, + kind: EntryType::new(EntryType::MESSAGE), + payload: json!({ "role": "user", "text": text }), + timestamp: 0, + } + } + + fn two_tool_specs() -> Vec { + vec![ + json!({"type":"function","function":{"name":"a","description":"d","parameters":{}}}), + json!({"type":"function","function":{"name":"b","description":"d","parameters":{}}}), + ] + } + + #[test] + fn wire_stats_empty_tools_counts_zero_and_no_system() { + let body = json!({"model": "m", "messages": [{"role": "user", "content": "hi"}]}); + let w = wire_stats(&body); + assert_eq!(w.tools_chars, 0); + assert_eq!(w.system_chars, 0); + assert_eq!(w.messages, 1); + assert_eq!( + w.history_chars, + r#"{"content":"hi","role":"user"}"#.chars().count() as u64 + ); + // No messages key at all: all zeros, no panic. + assert_eq!( + wire_stats(&json!({})), + WireStats { + system_chars: 0, + tools_chars: 0, + history_chars: 0, + messages: 0 + } + ); + } + + #[test] + fn wire_stats_counts_characters_not_bytes() { + let body = json!({"messages": [ + {"role": "system", "content": "héllo世界"}, + {"role": "user", "content": "日本語"}, + ]}); + let w = wire_stats(&body); + assert_eq!(w.system_chars, 7, "chars, not the 11 UTF-8 bytes"); + assert_eq!( + w.history_chars, + r#"{"content":"日本語","role":"user"}"#.chars().count() as u64 + ); + assert!(w.history_chars < r#"{"content":"日本語","role":"user"}"#.len() as u64); + } + + #[test] + fn wire_stats_known_transcript_has_exact_numbers() { + let p = OpenAiProvider::new(OpenAiConfig::new("https://x/v1", "m", "k")) + .with_system_prompt("You are tole.") + .with_tool_specs(two_tool_specs()); + let body = p.request_body(&[user_entry("e1", "hi")]); + let w = wire_stats(&body); + let tools_literal = r#"[{"function":{"description":"d","name":"a","parameters":{}},"type":"function"},{"function":{"description":"d","name":"b","parameters":{}},"type":"function"}]"#; + assert_eq!(w.system_chars, 13); + assert_eq!(w.tools_chars, tools_literal.chars().count() as u64); + assert_eq!(w.history_chars, 30); + assert_eq!(w.messages, 2); + assert_eq!( + w.to_value(), + json!({"system_chars": 13, "tools_chars": tools_literal.chars().count(), + "history_chars": 30, "messages": 2}) + ); + } + + #[test] + fn wire_stats_includes_tool_call_and_tool_result_messages() { + let body = json!({"messages": [ + {"role": "user", "content": "go"}, + {"role": "assistant", "content": null, "tool_calls": [{"id": "i1"}]}, + {"role": "tool", "tool_call_id": "i1", "content": "output"}, + ]}); + let w = wire_stats(&body); + let expected: u64 = body["messages"] + .as_array() + .unwrap() + .iter() + .map(|m| m.to_string().chars().count() as u64) + .sum(); + assert_eq!(w.history_chars, expected); + assert_eq!(w.messages, 3); + } + + #[test] + fn wire_stats_does_not_touch_the_body() { + let p = OpenAiProvider::new(OpenAiConfig::new("https://x/v1", "m", "k")) + .with_system_prompt("s") + .with_tool_specs(two_tool_specs()); + let body = p.request_body(&[user_entry("e1", "hi")]); + let before = body.to_string(); + let _ = wire_stats(&body); + assert_eq!(body.to_string(), before); + } + + /// Wire unchanged (#211): the body that really goes over the socket, + /// minus the streaming envelope, is byte-identical to `request_body`; + /// and the stored stats equal `wire_stats` of that same body. + #[test] + fn complete_sends_request_body_unchanged_and_reports_its_stats() { + let capture = Arc::new(Mutex::new(String::new())); + let url = serve_stream( + concat!( + "data: {\"choices\":[{\"delta\":{\"content\":\"ok\"}}]}\n\n", + "data: [DONE]\n\n", + ) + .to_string(), + Arc::clone(&capture), + ); + let mut p = OpenAiProvider::new(OpenAiConfig::new(url, "m", "sk-test")) + .with_streaming(true) + .with_system_prompt("You are tole.") + .with_tool_specs(two_tool_specs()); + let transcript = vec![user_entry("e1", "hi")]; + let expected = p.request_body(&transcript); + p.complete(&transcript).unwrap(); + let raw = capture.lock().unwrap().clone(); + let sent = raw.split_once("\r\n\r\n").unwrap().1; + let mut sent: Value = serde_json::from_str(sent).unwrap(); + let obj = sent.as_object_mut().unwrap(); + assert_eq!(obj.remove("stream"), Some(json!(true))); + assert!(obj.remove("stream_options").is_some()); + assert_eq!( + sent.to_string(), + expected.to_string(), + "wire body must be unchanged" + ); + assert_eq!( + p.last_wire_stats().unwrap(), + wire_stats(&expected).to_value() + ); + } + + /// Per-call semantics: each call reports ITS request's numbers, and + /// a call that fails leaves no stale numbers from the previous one + /// reachable as this step's. + #[test] + fn wire_stats_are_per_call_not_carried_over() { + let url = serve_stream_seq(vec![ + "data: {\"choices\":[{\"delta\":{\"content\":\"a\"}}]}\n\ndata: [DONE]\n\n".to_string(), + "data: {\"choices\":[{\"delta\":{\"content\":\"b\"}}]}\n\ndata: [DONE]\n\n".to_string(), + ]); + let mut p = OpenAiProvider::new(OpenAiConfig::new(url, "m", "sk-test")); + assert!(p.last_wire_stats().is_none(), "nothing before any call"); + p.complete(&[user_entry("e1", "hi")]).unwrap(); + let first = p.last_wire_stats().unwrap(); + p.complete(&[user_entry("e1", "hi"), user_entry("e2", "more text here")]) + .unwrap(); + let second = p.last_wire_stats().unwrap(); + assert_eq!(first["messages"], json!(1)); + assert_eq!(second["messages"], json!(2)); + assert!(second["history_chars"].as_u64() > first["history_chars"].as_u64()); + } } diff --git a/crates/tole-core/src/provider.rs b/crates/tole-core/src/provider.rs index 352dc1a..85a77d5 100644 --- a/crates/tole-core/src/provider.rs +++ b/crates/tole-core/src/provider.rs @@ -67,4 +67,15 @@ pub trait Provider { fn last_reasoning(&self) -> Option { None } + + /// Request-size breakdown for the LAST `complete()` call (issue + /// #211): `{system_chars, tools_chars, history_chars, messages}`, + /// measured on the request body before any streaming envelope (see + /// `openai::wire_stats`). Like `last_usage`, it describes THIS call + /// only — providers reset it per call. The turn loop stores it under + /// the namespaced `tole_wire` key of the usage record. Default None: + /// mocks and providers without a wire notion add nothing. + fn last_wire_stats(&self) -> Option { + None + } } diff --git a/crates/tole-core/src/read_file.rs b/crates/tole-core/src/read_file.rs index 495efc5..ba410a0 100644 --- a/crates/tole-core/src/read_file.rs +++ b/crates/tole-core/src/read_file.rs @@ -11,6 +11,22 @@ use std::path::{Path, PathBuf}; /// dataset) must not blow up the durable log / provider context. pub const MAX_READ_BYTES: u64 = 1_048_576; // 1 MiB +/// Head cap (characters) on the content returned per call — the same +/// 20,000-char bound git/web apply. Larger files are read in chunks via +/// `offset`/`limit` and the `next_offset` continuation (#211). +pub(crate) const MAX_READ_CHARS: usize = 20_000; + +/// Optional non-negative integer parameter (absent/null = `None`). +fn opt_uint(input: &Value, key: &str) -> Result, String> { + match input.get(key) { + None | Some(Value::Null) => Ok(None), + Some(v) => v + .as_u64() + .map(Some) + .ok_or_else(|| format!("read_file: '{key}' must be a non-negative integer")), + } +} + /// Read a file under a fixed root directory. Input: `{ "path": "rel" }`. #[derive(Debug, Clone)] pub struct ReadFileTool { @@ -28,6 +44,9 @@ impl ReadFileTool { /// also an escape. Errors distinguish escape from plain not-found so /// the model gets actionable feedback. fn jailed(&self, rel: &str) -> Result { + if rel.is_empty() { + return Err("read_file: path is empty".to_string()); + } let rel_path = Path::new(rel); // Component-based validation (uniform with the other file tools; // CodeCora scan #6/#12): drive-prefixed/rooted Windows relatives @@ -69,7 +88,9 @@ impl Tool for ReadFileTool { } fn summary(&self) -> String { - "Read a text file inside the workspace.".into() + "Read a text file inside the workspace. Returns at most 20000 chars; if `truncated`, \ + continue with offset=`next_offset` (offset/limit are in chars)." + .into() } fn describe(&self, input: &Value) -> String { @@ -84,7 +105,9 @@ impl Tool for ReadFileTool { Some(json!({ "type": "object", "properties": { - "path": { "type": "string", "description": "Workspace-relative file path (no .., no absolute paths)" } + "path": { "type": "string", "description": "Workspace-relative file path (no .., no absolute paths)" }, + "offset": { "type": "integer", "minimum": 0, "description": "Start char (default 0)" }, + "limit": { "type": "integer", "minimum": 1, "description": "Max chars (default/max 20000)" } }, "required": ["path"] })) @@ -94,6 +117,11 @@ impl Tool for ReadFileTool { let Some(path) = input.get("path").and_then(Value::as_str) else { return Err("read_file: missing 'path'".into()); }; + let offset = opt_uint(&input, "offset")?; + let limit = opt_uint(&input, "limit")?; + if limit == Some(0) { + return Err("read_file: 'limit' must be at least 1".into()); + } let target = self.jailed(path)?; let meta = std::fs::metadata(&target) .map_err(|e| format!("read_file: stat {}: {e}", target.display()))?; @@ -125,7 +153,38 @@ impl Tool for ReadFileTool { // Content hash (E12): the model quotes this in `edit_file.old_hash` // so stale anchors are refused before they corrupt a file. let hash = crate::file_tools::content_hash(&content); - Ok(json!({ "path": path, "content": content, "bytes": bytes.len(), "hash": hash })) + // Slice in CHARACTERS on the decoded text (boundary-safe: byte + // positions come from `char_indices`, never from arithmetic). + // `hash`/`bytes` always describe the WHOLE file. + let total_chars = content.chars().count(); + let offset = usize::try_from(offset.unwrap_or(0)).unwrap_or(usize::MAX); + let limit = usize::try_from(limit.unwrap_or(MAX_READ_CHARS as u64)) + .unwrap_or(MAX_READ_CHARS) + .min(MAX_READ_CHARS); + if offset > 0 && offset >= total_chars { + return Ok(json!({ + "path": path, "content": "", "bytes": bytes.len(), "hash": hash, + "truncated": false, "total_chars": total_chars, + "note": format!("offset {offset} is at or beyond the end ({total_chars} chars)") + })); + } + let byte_at = |n: usize| { + content + .char_indices() + .nth(n) + .map_or(content.len(), |(b, _)| b) + }; + let (start, end) = (byte_at(offset), byte_at(offset + limit)); + let end_chars = (offset + limit).min(total_chars); + let truncated = end_chars < total_chars; + let mut out = json!({ + "path": path, "content": &content[start..end], "bytes": bytes.len(), "hash": hash, + "truncated": truncated, "total_chars": total_chars + }); + if truncated { + out["next_offset"] = json!(end_chars); + } + Ok(out) } } @@ -151,6 +210,13 @@ mod tests { assert_eq!(out["bytes"], json!(5)); } + #[test] + fn rejects_empty_path() { + let t = ReadFileTool::new(tmpdir("empty")); + let err = t.execute(json!({ "path": "" })).unwrap_err(); + assert!(err.to_string().contains("path is empty"), "{err}"); + } + #[test] fn rejects_absolute_and_traversal() { let dir = tmpdir("jail"); @@ -186,6 +252,164 @@ mod tests { assert!(!err.contains("escape"), "missing ≠ escape: {err}"); } + fn read(t: &ReadFileTool, input: Value) -> Value { + t.execute(input).unwrap() + } + + fn file_tool(name: &str, content: &str) -> ReadFileTool { + let dir = tmpdir(name); + std::fs::write(dir.join("f.txt"), content).unwrap(); + ReadFileTool::new(dir) + } + + #[test] + fn small_file_reports_untruncated_and_total() { + let t = file_tool("small", "hello"); + let out = read(&t, json!({ "path": "f.txt" })); + assert_eq!(out["content"], json!("hello")); + assert_eq!(out["truncated"], json!(false)); + assert_eq!(out["total_chars"], json!(5)); + assert!(out.get("next_offset").is_none()); + assert_eq!( + out["hash"], + json!(crate::file_tools::content_hash("hello")), + "hash stays the whole-file hash" + ); + } + + #[test] + fn exactly_cap_is_not_truncated() { + let t = file_tool("exact", &"a".repeat(MAX_READ_CHARS)); + let out = read(&t, json!({ "path": "f.txt" })); + assert_eq!(out["truncated"], json!(false)); + assert_eq!(out["total_chars"], json!(MAX_READ_CHARS)); + assert_eq!(out["content"].as_str().unwrap().len(), MAX_READ_CHARS); + assert!(out.get("next_offset").is_none()); + } + + #[test] + fn one_over_cap_is_truncated_with_next_offset() { + let t = file_tool("over", &"a".repeat(MAX_READ_CHARS + 1)); + let out = read(&t, json!({ "path": "f.txt" })); + assert_eq!(out["truncated"], json!(true)); + assert_eq!(out["total_chars"], json!(MAX_READ_CHARS + 1)); + assert_eq!(out["next_offset"], json!(MAX_READ_CHARS)); + assert_eq!( + out["content"].as_str().unwrap().chars().count(), + MAX_READ_CHARS + ); + let rest = read(&t, json!({ "path": "f.txt", "offset": MAX_READ_CHARS })); + assert_eq!(rest["content"], json!("a")); + assert_eq!(rest["truncated"], json!(false)); + } + + #[test] + fn multibyte_chunks_reassemble_exactly() { + // 1-, 2-, 3- and 4-byte chars, mixed so every boundary kind occurs. + let unit = "aé日本語😀x"; + let text: String = unit.repeat(40); + let t = file_tool("multibyte", &text); + for limit in [1usize, 2, 3, 7, 13, 1000] { + let mut out = String::new(); + let mut offset = 0u64; + loop { + let r = read( + &t, + json!({ "path": "f.txt", "offset": offset, "limit": limit }), + ); + let chunk = r["content"].as_str().unwrap(); + assert!(chunk.chars().count() <= limit); + out.push_str(chunk); + if r["truncated"] == json!(true) { + offset = r["next_offset"].as_u64().unwrap(); + } else { + break; + } + } + assert_eq!(out, text, "limit {limit}"); + } + } + + #[test] + fn offset_is_in_chars_not_bytes() { + let t = file_tool("charoff", "é日😀z"); + let r = read(&t, json!({ "path": "f.txt", "offset": 2, "limit": 2 })); + assert_eq!(r["content"], json!("😀z")); + assert_eq!(r["bytes"], json!("é日😀z".len())); + assert_eq!(r["total_chars"], json!(4)); + } + + #[test] + fn offset_beyond_end_is_empty_not_error() { + let t = file_tool("beyond", "abc"); + for off in [3u64, 4, 1_000_000] { + let r = read(&t, json!({ "path": "f.txt", "offset": off })); + assert_eq!(r["content"], json!("")); + assert_eq!(r["truncated"], json!(false)); + assert_eq!(r["total_chars"], json!(3)); + assert!(r["note"].as_str().unwrap().contains("beyond")); + } + } + + #[test] + fn limit_is_clamped_to_default_max() { + let t = file_tool("clamp", &"b".repeat(MAX_READ_CHARS + 500)); + let r = read(&t, json!({ "path": "f.txt", "limit": 1_000_000 })); + assert_eq!(r["content"].as_str().unwrap().len(), MAX_READ_CHARS); + assert_eq!(r["truncated"], json!(true)); + assert_eq!(r["next_offset"], json!(MAX_READ_CHARS)); + let r = read(&t, json!({ "path": "f.txt", "limit": 10 })); + assert_eq!(r["content"], json!("b".repeat(10))); + assert_eq!(r["next_offset"], json!(10)); + } + + #[test] + fn invalid_offset_or_limit_is_a_tool_error() { + let t = file_tool("badargs", "abc"); + for bad in [json!(-1), json!(1.5), json!("2"), json!(true), json!([1])] { + for key in ["offset", "limit"] { + let err = t + .execute(json!({ "path": "f.txt", key: bad.clone() })) + .unwrap_err(); + assert!(err.contains(key) && err.contains("non-negative"), "{err}"); + } + } + let err = t + .execute(json!({ "path": "f.txt", "limit": 0 })) + .unwrap_err(); + assert!(err.contains("limit"), "{err}"); + } + + #[test] + fn crlf_is_preserved_and_empty_file_is_fine() { + let t = file_tool("crlf", "a\r\nb\r\n"); + let r = read(&t, json!({ "path": "f.txt" })); + assert_eq!(r["content"], json!("a\r\nb\r\n")); + assert_eq!(r["total_chars"], json!(6)); + let r = read(&t, json!({ "path": "f.txt", "offset": 1, "limit": 2 })); + assert_eq!(r["content"], json!("\r\n")); + let e = file_tool("empty", ""); + for input in [ + json!({ "path": "f.txt" }), + json!({ "path": "f.txt", "offset": 0 }), + ] { + let r = read(&e, input); + assert_eq!(r["content"], json!("")); + assert_eq!(r["total_chars"], json!(0)); + assert_eq!(r["truncated"], json!(false)); + } + } + + #[test] + fn spec_declares_offset_and_limit() { + let t = ReadFileTool::new(PathBuf::from(".")); + let spec = t.spec().unwrap(); + assert!(spec["properties"]["offset"].is_object()); + assert!(spec["properties"]["limit"].is_object()); + assert_eq!(spec["required"], json!(["path"])); + assert!(t.summary().contains("next_offset")); + } + #[test] fn rejects_directory_read() { let dir = tmpdir("dir"); diff --git a/crates/tole-core/src/turn.rs b/crates/tole-core/src/turn.rs index d4185c5..1306634 100644 --- a/crates/tole-core/src/turn.rs +++ b/crates/tole-core/src/turn.rs @@ -517,7 +517,25 @@ fn drive( // itself is not part of its own request). Usage-only commits // carry no transition, so they are legal mid-Planning and // replay-safe (Record::Usage). - if let Some(u) = p.last_usage() { + // + // Issue #211: a provider that also reports the request-size + // breakdown has it merged under the namespaced `tole_wire` key; + // the provider's own keys stay untouched. A provider that + // reported wire stats but no usage still leaves a record holding + // only `tole_wire` (readers sum `prompt_tokens` etc. with a 0 + // default). Neither reported: no record, as before. + let usage = match (p.last_usage(), p.last_wire_stats()) { + (Some(mut u), Some(w)) => { + if let Some(obj) = u.as_object_mut() { + obj.insert("tole_wire".to_string(), w); + } + Some(u) + } + (Some(u), None) => Some(u), + (None, Some(w)) => Some(json!({ "tole_wire": w })), + (None, None) => None, + }; + if let Some(u) = usage { if let Some(last) = s.entries().last() { s.commit(Commit::new().usage(UsageRecord { id: String::new(), diff --git a/crates/tole-core/tests/turn_loop.rs b/crates/tole-core/tests/turn_loop.rs index 7ab18c7..5bd3b4a 100644 --- a/crates/tole-core/tests/turn_loop.rs +++ b/crates/tole-core/tests/turn_loop.rs @@ -123,6 +123,153 @@ fn usage_rows_are_written_durably_per_step() { assert_eq!(total_in, 200); } +/// Mock reporting a per-step wire breakdown (issue #211) next to an +/// optional provider usage object. `stats` is consumed one per step. +struct WireMock { + step: usize, + outputs: Vec, + usage: Option, + stats: Vec, +} +impl Provider for WireMock { + fn complete(&mut self, _t: &[Entry]) -> Result { + let out = self + .outputs + .get(self.step) + .cloned() + .ok_or_else(|| ProviderError("exhausted".into())); + self.step += 1; + out + } + fn last_usage(&self) -> Option { + self.usage.clone() + } + fn last_wire_stats(&self) -> Option { + self.stats.get(self.step - 1).cloned() + } +} + +fn wire_json(n: u64) -> Value { + json!({"system_chars": n, "tools_chars": n * 2, "history_chars": n * 3, "messages": n}) +} + +#[test] +fn wire_stats_are_merged_into_usage_under_tole_wire_and_provider_keys_survive() { + let dir = tmpdir("usage-wire"); + let mut s = JsonlStorage::create(&dir, "uw", None).unwrap(); + let mut p = WireMock { + step: 0, + outputs: vec![ + ProviderOutput::ToolCall { + tool: "echo".into(), + input: json!({"n": 1}), + }, + ProviderOutput::Final { + text: "done".into(), + }, + ], + usage: Some(json!({ + "prompt_tokens": 100, + "completion_tokens": 7, + "prompt_tokens_details": {"cached_tokens": 64}, + })), + stats: vec![wire_json(1), wire_json(2)], + }; + let mut reg = ToolRegistry::new(); + reg.register(Box::new(EchoTool)).unwrap(); + run_turn(&mut s, &mut p, ®, "hi").unwrap(); + let usages = s.usages(); + assert_eq!(usages.len(), 2); + for (i, u) in usages.iter().enumerate() { + assert_eq!(u.usage["tole_wire"], wire_json(i as u64 + 1), "step {i}"); + // Provider keys are untouched. + assert_eq!(u.usage["prompt_tokens"], json!(100)); + assert_eq!(u.usage["completion_tokens"], json!(7)); + assert_eq!(u.usage["prompt_tokens_details"]["cached_tokens"], json!(64)); + } + // Ledger consumers are unaffected: same step count, same token sums. + let total_in: u64 = usages + .iter() + .filter_map(|u| u.usage.get("prompt_tokens").and_then(|v| v.as_u64())) + .sum(); + assert_eq!(total_in, 200); +} + +#[test] +fn wire_stats_without_provider_usage_store_a_tole_wire_only_record() { + let dir = tmpdir("usage-wire-only"); + let mut s = JsonlStorage::create(&dir, "uwo", None).unwrap(); + let mut p = WireMock { + step: 0, + outputs: vec![ProviderOutput::Final { text: "f".into() }], + usage: None, + stats: vec![wire_json(5)], + }; + let mut reg = ToolRegistry::new(); + reg.register(Box::new(EchoTool)).unwrap(); + run_turn(&mut s, &mut p, ®, "hi").unwrap(); + let usages = s.usages(); + assert_eq!(usages.len(), 1, "still one ledger row = one step"); + assert_eq!(usages[0].usage, json!({"tole_wire": wire_json(5)})); + // A consumer summing prompt/completion tokens reads 0 from it. + let tokens: u64 = usages + .iter() + .map(|u| { + u.usage + .get("prompt_tokens") + .and_then(|v| v.as_u64()) + .unwrap_or(0) + + u.usage + .get("completion_tokens") + .and_then(|v| v.as_u64()) + .unwrap_or(0) + }) + .sum(); + assert_eq!(tokens, 0); +} + +/// Stored numbers equal `wire_stats` of the REAL request body for a +/// known transcript (fixed system prompt, 2 tools, one user message). +#[test] +fn stored_wire_numbers_equal_wire_stats_of_the_request_body() { + use tole_core::openai::{wire_stats, OpenAiConfig, OpenAiProvider}; + struct BodyMeasuring { + inner: OpenAiProvider, + last: Option, + } + impl Provider for BodyMeasuring { + fn complete(&mut self, t: &[Entry]) -> Result { + self.last = Some(wire_stats(&self.inner.request_body(t)).to_value()); + Ok(ProviderOutput::Final { text: "ok".into() }) + } + fn last_wire_stats(&self) -> Option { + self.last.clone() + } + } + let specs = vec![ + json!({"type":"function","function":{"name":"a","description":"d","parameters":{}}}), + json!({"type":"function","function":{"name":"b","description":"d","parameters":{}}}), + ]; + let inner = OpenAiProvider::new(OpenAiConfig::new("https://x/v1", "m", "k")) + .with_system_prompt("You are tole.") + .with_tool_specs(specs); + let mut p = BodyMeasuring { inner, last: None }; + let dir = tmpdir("usage-wire-golden"); + let mut s = JsonlStorage::create(&dir, "uwg", Some("You are tole.".into())).unwrap(); + let mut reg = ToolRegistry::new(); + reg.register(Box::new(EchoTool)).unwrap(); + run_turn(&mut s, &mut p, ®, "hi").unwrap(); + let usages = s.usages(); + assert_eq!(usages.len(), 1); + let tools_len = r#"[{"function":{"description":"d","name":"a","parameters":{}},"type":"function"},{"function":{"description":"d","name":"b","parameters":{}},"type":"function"}]"# + .chars() + .count(); + assert_eq!( + usages[0].usage["tole_wire"], + json!({"system_chars": 13, "tools_chars": tools_len, "history_chars": 30, "messages": 2}) + ); +} + #[test] fn no_usage_rows_without_provider_usage() { let dir = tmpdir("usage-none"); diff --git a/crates/tole-core/tests/wire_size_missions.rs b/crates/tole-core/tests/wire_size_missions.rs new file mode 100644 index 0000000..eebf709 --- /dev/null +++ b/crates/tole-core/tests/wire_size_missions.rs @@ -0,0 +1,809 @@ +//! Issue #211 (PR 2): reference missions with a loose CI bound on request +//! SIZE. Tokenizer-free, deterministic, offline. +//! +//! The REAL `OpenAiProvider` is driven through the real turn loop +//! (`run_turn` / `resume_turn`) against a local mock OpenAI-compatible +//! HTTP server that answers a scripted sequence (tool calls with fixed +//! arguments, then a final text) and reports NO `usage`. The stored +//! usage rows are therefore wire-only: just the `tole_wire` breakdown +//! `{system_chars, tools_chars, history_chars, messages}` written by the +//! PR 1 path. The mock also captures every request body so the tests can +//! assert on the exact bytes that went over the wire. +//! +//! What the guard fails on (see the self-tests at the bottom, which prove +//! each failure is actually detected): +//! +//! * system prompt bloat -> prefix bound + total bound +//! * tool-spec bloat -> prefix bound (+ total bound on small-result missions) +//! * history re-send growth -> linear-history check + total bound +//! * reshuffled / non-append-only history, or an unstable prefix across +//! steps or across a crash-resume -> prefix-stability / append-only checks +//! +//! ## Re-baselining +//! +//! The bounds below are `ceil(measured * 1.25)` rounded up to a round +//! number. Run +//! +//! ```text +//! cargo test -p tole-core --test wire_size_missions -- --nocapture +//! ``` +//! +//! to print the measured values (lines starting with `MEASURED`). A bound +//! bump is a deliberate statement that the request got bigger: it needs a +//! written justification in the PR that makes it (what grew, and why that +//! is worth the extra characters on EVERY request). Never bump a bound to +//! make a red run green without that justification. +//! +//! ## Registry used by the missions +//! +//! Profile-independent on purpose (one set of bounds for every feature +//! profile; the shell-tools tools are covered by the full-registry bound +//! in `crates/tole-cli/tests/tool_specs.rs`). 22 specs: +//! +//! * 8 real `tole-core` tools: `read_file`, `edit_file`, `delete_file`, +//! `web_fetch`, `load_skill`, `todo_write`, `todo_read`, `verify_package`; +//! * 14 synthetic ReadOnly fixture tools with realistic one-sentence +//! descriptions and 2-4 described parameters (see `SYNTHETIC`). The +//! missions only ever CALL synthetic tools, so results have a fixed, +//! configurable size. + +use serde_json::{json, Value}; +use std::io::{Read, Write}; +use std::net::{SocketAddr, TcpListener, TcpStream}; +use std::path::{Path, PathBuf}; +use std::sync::atomic::{AtomicBool, Ordering}; +use std::sync::{Arc, Mutex}; +use std::time::Duration; +use tole_core::approval::{Approver, ToolRequest, Verdict}; +use tole_core::entry::Entry; +use tole_core::openai::{wire_stats, OpenAiConfig, OpenAiProvider, WireStats}; +use tole_core::provider::{Provider, ProviderError, ProviderOutput}; +use tole_core::storage::{JsonlStorage, Storage}; +use tole_core::tool::{Risk, Tool, ToolRegistry}; +use tole_core::turn::{resume_turn, run_turn, TurnOutcome}; + +// --------------------------------------------------------------------------- +// Bounds (see "Re-baselining" above) +// --------------------------------------------------------------------------- + +/// Upper bound on `system_chars + tools_chars` of any single request +/// (the cacheable prefix every step re-sends). Measured 9,890 +/// (system 1,436 + tools 8,454); bound = ceil(9,890 * 1.25) rounded up. +const PREFIX_BOUND: u64 = 13_000; + +/// Upper bound on the SUM over all steps of +/// `system_chars + tools_chars + history_chars` for `many_small_steps`. +/// Measured 291,485; bound = ceil(measured * 1.25) rounded up. +const MANY_SMALL_TOTAL_BOUND: u64 = 370_000; +/// Same, for `read_heavy`. Measured 647,498. +const READ_HEAVY_TOTAL_BOUND: u64 = 810_000; +/// Same, for `mixed_with_resume` (pre-crash + post-resume requests). +/// Measured 277,380. +const MIXED_TOTAL_BOUND: u64 = 350_000; + +/// Per-step envelope allowed on top of a tool result's own characters in +/// the linear-history check: the assistant `tool_calls` message, the tool +/// message wrapper, the result fence and the JSON escaping of the result. +const HISTORY_SLACK_PER_STEP: u64 = 700; + +/// Characters of regression headroom the self-tests inject. +const SYSTEM_BLOAT: usize = 10_000; +const TOOL_BLOAT: usize = 5_000; + +// --------------------------------------------------------------------------- +// Local mock OpenAI-compatible server +// --------------------------------------------------------------------------- + +#[derive(Clone)] +enum Reply { + Tool { name: String, args: Value }, + Final(String), +} + +/// Scripted `/chat/completions` server on `127.0.0.1:0` (never a fixed +/// port). One thread per connection; every response carries +/// `Connection: close` so the client never reuses a socket and request +/// `n` is always answered with `replies[n]`. The request is read until +/// the Content-Length body is complete (a single `read` may return the +/// headers only, which is the classic source of flaky mock readers), and +/// every socket has a read timeout so a bug fails the test instead of +/// hanging it. Responses carry no `usage` object. +struct MockServer { + addr: SocketAddr, + bodies: Arc>>, + stop: Arc, +} + +impl MockServer { + fn start(replies: Vec) -> Self { + let listener = TcpListener::bind("127.0.0.1:0").unwrap(); + let addr = listener.local_addr().unwrap(); + let bodies: Arc>> = Arc::new(Mutex::new(Vec::new())); + let stop = Arc::new(AtomicBool::new(false)); + let replies = Arc::new(replies); + let (b, st) = (Arc::clone(&bodies), Arc::clone(&stop)); + std::thread::spawn(move || { + for conn in listener.incoming() { + if st.load(Ordering::SeqCst) { + break; + } + let Ok(stream) = conn else { continue }; + let (b, replies) = (Arc::clone(&b), Arc::clone(&replies)); + std::thread::spawn(move || serve_one(stream, &b, &replies)); + } + }); + MockServer { addr, bodies, stop } + } + + fn url(&self) -> String { + format!("http://{}/v1", self.addr) + } + + fn bodies(&self) -> Vec { + self.bodies.lock().unwrap().clone() + } +} + +impl Drop for MockServer { + fn drop(&mut self) { + // Unblock `accept` so the listener thread exits. + self.stop.store(true, Ordering::SeqCst); + let _ = TcpStream::connect(self.addr); + } +} + +fn read_request_body(stream: &mut TcpStream) -> Option> { + let mut buf: Vec = Vec::new(); + let mut chunk = [0u8; 16 * 1024]; + loop { + if let Some(h) = buf.windows(4).position(|w| w == b"\r\n\r\n") { + let head = String::from_utf8_lossy(&buf[..h]).to_string(); + let clen: usize = head + .lines() + .find_map(|l| { + let (k, v) = l.split_once(':')?; + k.eq_ignore_ascii_case("content-length") + .then(|| v.trim().parse().ok()) + .flatten() + }) + .unwrap_or(0); + if buf.len() >= h + 4 + clen { + return Some(buf[h + 4..h + 4 + clen].to_vec()); + } + } + let n = stream.read(&mut chunk).ok()?; + if n == 0 { + return None; + } + buf.extend_from_slice(&chunk[..n]); + } +} + +fn serve_one(mut stream: TcpStream, bodies: &Mutex>, replies: &[Reply]) { + let _ = stream.set_read_timeout(Some(Duration::from_secs(15))); + let Some(raw) = read_request_body(&mut stream) else { + return; + }; + let Ok(body) = serde_json::from_slice::(&raw) else { + return; + }; + // Index assignment and capture are one critical section. + let idx = { + let mut g = bodies.lock().unwrap(); + g.push(body); + g.len() - 1 + }; + let (status, payload) = match replies.get(idx) { + Some(Reply::Tool { name, args }) => ( + "200 OK", + json!({"choices":[{"message":{ + "role":"assistant","content":null, + "tool_calls":[{"id":format!("call_{idx}"),"type":"function", + "function":{"name":name,"arguments":args.to_string()}}] + }}]}), + ), + Some(Reply::Final(text)) => ( + "200 OK", + json!({"choices":[{"message":{"role":"assistant","content":text}}]}), + ), + None => ( + "500 Internal Server Error", + json!({"error":"mock script exhausted"}), + ), + }; + let payload = payload.to_string(); + let resp = format!( + "HTTP/1.1 {status}\r\nContent-Type: application/json\r\nContent-Length: {}\r\nConnection: close\r\n\r\n{payload}", + payload.len() + ); + let _ = stream.write_all(resp.as_bytes()); +} + +// --------------------------------------------------------------------------- +// Registry +// --------------------------------------------------------------------------- + +struct DenyAllInteractive; +impl Approver for DenyAllInteractive { + fn decide(&self, _req: &ToolRequest<'_>) -> Verdict { + Verdict::Deny + } + fn interactive(&self) -> bool { + true + } +} + +/// (name, description, [(param, type, param description)]) +type SyntheticSpec = ( + &'static str, + &'static str, + &'static [(&'static str, &'static str, &'static str)], +); + +const SYNTHETIC: &[SyntheticSpec] = &[ + ("search_docs", "Search the project documentation for a phrase and return the best matching sections with their file paths.", &[("query","string","Phrase or keywords to look for."),("limit","integer","Maximum number of sections to return (default 5)."),("path_prefix","string","Only search docs under this path prefix.")]), + ("list_dir", "List the entries of a directory inside the workspace, directories first, with sizes.", &[("path","string","Directory to list, relative to the workspace root."),("recursive","boolean","Descend into subdirectories."),("max_entries","integer","Cap on returned entries.")]), + ("grep_repo", "Search tracked files for a regular expression and return matching lines with file and line number.", &[("pattern","string","Regular expression to search for."),("glob","string","Restrict the search to files matching this glob."),("context","integer","Lines of context around each match."),("max_matches","integer","Cap on returned matches.")]), + ("read_symbol", "Return the source of a named function, type or constant from the code index.", &[("symbol","string","Fully qualified or bare symbol name."),("with_callers","boolean","Also list direct callers.")]), + ("fetch_issue", "Fetch one issue from the tracker: title, state, labels and the comment thread.", &[("number","integer","Issue number."),("comments","boolean","Include the comment thread.")]), + ("list_prs", "List pull requests of the repository filtered by state and author.", &[("state","string","One of open, closed, merged."),("author","string","Only PRs by this login."),("limit","integer","Maximum number of PRs.")]), + ("get_ci_status", "Report the CI check runs for a commit or branch with their conclusions.", &[("ref","string","Commit SHA or branch name."),("only_failed","boolean","Hide passing checks.")]), + ("query_metrics", "Query a recorded metric series over a time window and return summarized points.", &[("metric","string","Metric name."),("window_minutes","integer","Size of the window ending now."),("step_seconds","integer","Resolution of the returned points.")]), + ("lookup_dependency", "Resolve a dependency to its locked version, license and the crates that use it.", &[("name","string","Dependency name."),("ecosystem","string","Package ecosystem, for example cargo or npm.")]), + ("inspect_schema", "Describe the shape of a JSON or SQL schema file: tables, columns and types.", &[("path","string","Schema file to inspect."),("table","string","Only describe this table or definition.")]), + ("tail_log", "Return the last lines of a log file, optionally filtered by severity.", &[("path","string","Log file path."),("lines","integer","Number of trailing lines."),("level","string","Minimum severity to keep.")]), + ("diff_summary", "Summarize the diff between two revisions per file: additions, deletions and a short note.", &[("base","string","Base revision."),("head","string","Head revision."),("paths","array","Restrict to these paths.")]), + ("resolve_path", "Resolve a symbolic location such as a module name or a test name to a file path.", &[("name","string","Module, test or fixture name."),("kind","string","One of module, test, fixture.")]), + ("count_lines", "Count lines of code per language under a path, excluding vendored directories.", &[("path","string","Root to count under."),("exclude","array","Extra directory names to skip.")]), +]; + +/// ReadOnly fixture tool: returns `out_chars` deterministic characters. +struct FixtureTool { + spec: SyntheticSpec, + out_chars: usize, +} + +impl Tool for FixtureTool { + fn name(&self) -> &str { + self.spec.0 + } + fn risk(&self) -> Risk { + Risk::ReadOnly + } + fn summary(&self) -> String { + self.spec.1.to_string() + } + fn spec(&self) -> Option { + let props: serde_json::Map = self + .spec + .2 + .iter() + .map(|(n, t, d)| ((*n).to_string(), json!({"type": t, "description": d}))) + .collect(); + Some(json!({"type": "object", "properties": props})) + } + fn execute(&self, input: Value) -> Result { + let tag = input["n"].as_u64().unwrap_or(0); + Ok(json!({ "content": filler(self.out_chars, tag) })) + } +} + +/// Exactly `n` alphanumeric characters, a pure function of `(n, tag)` +/// (no JSON escaping, so one char on the wire is one char of content). +fn filler(n: usize, tag: u64) -> String { + const ALPHA: &[u8] = b"abcdefghijklmnopqrstuvwxyz0123456789"; + (0..n) + .map(|i| ALPHA[(i * 7 + tag as usize * 13) % ALPHA.len()] as char) + .collect() +} + +/// An extra, deliberately oversized spec for the self-test. +struct BloatedTool(usize); +impl Tool for BloatedTool { + fn name(&self) -> &str { + "bloated_tool" + } + fn risk(&self) -> Risk { + Risk::ReadOnly + } + fn summary(&self) -> String { + filler(self.0, 99) + } + fn execute(&self, _input: Value) -> Result { + Ok(json!({})) + } +} + +/// `size_of(i)` = result characters of the i-th synthetic tool. +fn build_registry(size_of: &dyn Fn(usize) -> usize, extra: Option>) -> ToolRegistry { + let root = PathBuf::from("."); + let mut reg = ToolRegistry::with_approver(DenyAllInteractive); + let real: Vec> = { + let todo = tole_core::todo::TodoState::new(); + vec![ + Box::new(tole_core::read_file::ReadFileTool::new(root.clone())), + Box::new(tole_core::file_tools::EditFileTool::new(root.clone())), + Box::new(tole_core::file_tools::DeleteFileTool::new(root)), + Box::new(tole_core::web::WebFetchTool), + Box::new(tole_core::skills::LoadSkillTool::new(None)), + Box::new(tole_core::todo::TodoWriteTool::new(todo.clone())), + Box::new(tole_core::todo::TodoReadTool::new(todo)), + Box::new(tole_core::verify_package::VerifyPackageTool::new()), + ] + }; + for t in real { + let n = t.name().to_string(); + reg.register(t).unwrap_or_else(|e| panic!("{n}: {e}")); + } + for (i, spec) in SYNTHETIC.iter().enumerate() { + reg.register(Box::new(FixtureTool { + spec: *spec, + out_chars: size_of(i), + })) + .unwrap(); + } + if let Some(t) = extra { + reg.register(t).unwrap(); + } + reg +} + +/// A fixed, realistic-sized system prompt (about 1.5k characters). +fn system_prompt() -> String { + let para = "You are tole, a careful coding agent working inside the user's repository. \ +Prefer reading before editing, keep changes small and reviewable, never invent file contents, \ +and explain failures plainly. Every write goes through the approval gate; never try to bypass it. \ +When a task is finished, answer with a short summary of what changed and what was verified. "; + para.repeat(4) +} + +// --------------------------------------------------------------------------- +// Mission runner +// --------------------------------------------------------------------------- + +/// One request as measured from the stored usage row. +#[derive(Debug, Clone, Copy, PartialEq, Eq)] +struct Step { + system: u64, + tools: u64, + history: u64, +} + +impl Step { + fn total(&self) -> u64 { + self.system + self.tools + self.history + } + fn prefix(&self) -> u64 { + self.system + self.tools + } +} + +/// The measured outcome of a (possibly resumed) mission. +struct MissionRun { + /// Captured request bodies, in request order (pre-crash first). + bodies: Vec, + /// Per-step stats from the stored `tole_wire`, aligned with `bodies`. + steps: Vec, + /// The largest single tool result the mission produces. + max_result_chars: u64, +} + +fn tmpdir(name: &str) -> PathBuf { + let d = std::env::temp_dir().join(format!("tole-wsm-{}-{}", name, std::process::id())); + let _ = std::fs::remove_dir_all(&d); + std::fs::create_dir_all(&d).unwrap(); + d +} + +fn provider(url: String, system: &str, reg: &ToolRegistry) -> OpenAiProvider { + OpenAiProvider::new(OpenAiConfig::new(url, "mock-model", "sk-test")) + .with_streaming(false) + .with_system_prompt(system) + .with_tool_specs(reg.specs()) +} + +/// Steps from the stored usage rows. Asserts the rows are wire-only (the +/// mock reports no `usage`) and that stored stats equal `wire_stats` of +/// the captured body — the PR 1 path and the mock agree. +fn steps_from_storage(s: &dyn Storage, bodies: &[Value]) -> Vec { + let rows = s.usages(); + assert_eq!(rows.len(), bodies.len(), "one usage row per request"); + rows.iter() + .zip(bodies) + .map(|(r, b)| { + let obj = r.usage.as_object().expect("usage is an object"); + assert_eq!( + obj.keys().collect::>(), + vec!["tole_wire"], + "wire-only" + ); + let w: WireStats = wire_stats(b); + assert_eq!(r.usage["tole_wire"], w.to_value(), "stored == measured"); + Step { + system: r.usage["tole_wire"]["system_chars"].as_u64().unwrap(), + tools: r.usage["tole_wire"]["tools_chars"].as_u64().unwrap(), + history: r.usage["tole_wire"]["history_chars"].as_u64().unwrap(), + } + }) + .collect() +} + +/// Script: `tool_steps` tool calls (cycling the synthetic tools, args +/// varying per step so the loop guard never trips) then a final text. +/// Total requests = `tool_steps + 1`. +fn script(tool_steps: usize) -> Vec { + let mut v: Vec = (0..tool_steps) + .map(|i| Reply::Tool { + name: SYNTHETIC[i % SYNTHETIC.len()].0.to_string(), + args: json!({"query": format!("step-{i:02}"), "n": i}), + }) + .collect(); + v.push(Reply::Final("done".to_string())); + v +} + +fn expect_final(out: Result) { + match out.expect("turn ran") { + TurnOutcome::Final { text, .. } => assert_eq!(text, "done"), + other => panic!("expected Final, got {other:?}"), + } +} + +/// Run a mission uninterrupted. `wrap` lets a self-test interpose a +/// misbehaving provider around the real one. +fn run_with( + dir: &Path, + id: &str, + steps: usize, + result_chars: usize, + system: &str, + extra: Option>, + doubling: bool, +) -> (MissionRun, JsonlStorage) { + let reg = build_registry(&|_| result_chars, extra); + let server = MockServer::start(script(steps - 1)); + let mut s = JsonlStorage::create(dir, id, None).unwrap(); + let inner = provider(server.url(), system, ®); + if doubling { + let mut p = DoublingProvider(inner); + expect_final(run_turn(&mut s, &mut p, ®, "start the mission")); + } else { + let mut p = inner; + expect_final(run_turn(&mut s, &mut p, ®, "start the mission")); + } + let bodies = server.bodies(); + let run = MissionRun { + steps: steps_from_storage(&s, &bodies), + bodies, + max_result_chars: result_chars as u64, + }; + (run, s) +} + +/// Misbehaving provider for the self-test: re-sends the whole history +/// twice (what a bug that appends the transcript to itself would do). +struct DoublingProvider(OpenAiProvider); +impl Provider for DoublingProvider { + fn complete(&mut self, transcript: &[Entry]) -> Result { + let doubled: Vec = transcript + .iter() + .chain(transcript.iter()) + .cloned() + .collect(); + self.0.complete(&doubled) + } + fn last_usage(&self) -> Option { + self.0.last_usage() + } + fn last_reasoning(&self) -> Option { + self.0.last_reasoning() + } + fn last_wire_stats(&self) -> Option { + self.0.last_wire_stats() + } +} + +// --------------------------------------------------------------------------- +// Checks (Result-returning so self-tests can observe a failure) +// --------------------------------------------------------------------------- + +fn check_bound(what: &str, measured: u64, bound: u64) -> Result<(), String> { + if measured <= bound { + Ok(()) + } else { + Err(format!( + "{what}: measured {measured} chars exceeds bound {bound} \ + (re-baseline only with a justification, see the file header)" + )) + } +} + +/// History is only the append-only sum of results: the last step's +/// history may exceed the first step's by at most +/// `steps * (max single result + slack)`. +fn check_linear_history(run: &MissionRun) -> Result<(), String> { + let first = run.steps.first().unwrap().history; + let last = run.steps.last().unwrap().history; + let allowed = first + run.steps.len() as u64 * (run.max_result_chars + HISTORY_SLACK_PER_STEP); + if last <= allowed { + Ok(()) + } else { + Err(format!( + "history grew super-linearly: last-step history {last} > allowed {allowed} \ + (first {first}, {} steps, max result {})", + run.steps.len(), + run.max_result_chars + )) + } +} + +/// `system_chars` and `tools_chars` identical on every step. +fn check_prefix_stable(steps: &[Step]) -> Result<(), String> { + let (s0, t0) = (steps[0].system, steps[0].tools); + for (i, st) in steps.iter().enumerate() { + if st.system != s0 || st.tools != t0 { + return Err(format!( + "prefix unstable at step {i}: system {}/{s0}, tools {}/{t0}", + st.system, st.tools + )); + } + } + Ok(()) +} + +/// Body n+1 starts with exactly the messages and the same tools array of +/// body n: append-only, no reshuffling, byte-identical prefix. +fn check_append_only(bodies: &[Value]) -> Result<(), String> { + for (n, pair) in bodies.windows(2).enumerate() { + let (a, b) = ( + pair[0]["messages"].as_array().unwrap(), + pair[1]["messages"].as_array().unwrap(), + ); + if b.len() <= a.len() || b[..a.len()] != a[..] { + return Err(format!("request {} is not an append of request {n}", n + 1)); + } + if pair[0]["tools"] != pair[1]["tools"] { + return Err(format!( + "tools array changed between request {n} and {}", + n + 1 + )); + } + } + Ok(()) +} + +/// All checks for one mission; empty = clean. +fn violations(run: &MissionRun, total_bound: u64) -> Vec { + let total: u64 = run.steps.iter().map(Step::total).sum(); + let prefix = run.steps[0].prefix(); + [ + check_bound("total request chars", total, total_bound), + check_bound("first-step prefix (system+tools)", prefix, PREFIX_BOUND), + check_linear_history(run), + check_prefix_stable(&run.steps), + check_append_only(&run.bodies), + ] + .into_iter() + .filter_map(Result::err) + .collect() +} + +fn report(name: &str, run: &MissionRun) { + let total: u64 = run.steps.iter().map(Step::total).sum(); + let first = run.steps[0]; + let last = run.steps.last().unwrap(); + println!( + "MEASURED {name}: steps={} total={total} prefix={} (system={} tools={}) \ + first_history={} last_history={}", + run.steps.len(), + first.prefix(), + first.system, + first.tools, + first.history, + last.history + ); +} + +// --------------------------------------------------------------------------- +// Missions +// --------------------------------------------------------------------------- + +fn many_small_steps(dir: &Path, id: &str) -> MissionRun { + run_with(dir, id, 20, 200, &system_prompt(), None, false).0 +} + +fn read_heavy(dir: &Path, id: &str) -> MissionRun { + run_with(dir, id, 8, 20_000, &system_prompt(), None, false).0 +} + +#[test] +fn many_small_steps_stays_within_bound() { + let dir = tmpdir("many-small"); + let run = many_small_steps(&dir, "ms"); + report("many_small_steps", &run); + assert_eq!(run.steps.len(), 20); + let v = violations(&run, MANY_SMALL_TOTAL_BOUND); + assert!(v.is_empty(), "{v:#?}"); +} + +#[test] +fn read_heavy_stays_within_bound() { + let dir = tmpdir("read-heavy"); + let run = read_heavy(&dir, "rh"); + report("read_heavy", &run); + assert_eq!(run.steps.len(), 8); + let v = violations(&run, READ_HEAVY_TOTAL_BOUND); + assert!(v.is_empty(), "{v:#?}"); +} + +/// 12 requests (11 tool calls + final) with varied result sizes, killed +/// after the 5th tool settlement and resumed from the on-disk file. +/// Result sizes cycle 200 / 2,000 / 5,000. +#[test] +fn mixed_with_resume_keeps_prefix_and_stays_within_bound() { + const STEPS: usize = 12; + const CRASH_AFTER_TOOLS: usize = 5; + let dir = tmpdir("mixed"); + let system = system_prompt(); + let sizes = [200usize, 2_000, 5_000]; + + // Tool i returns sizes[i % 3] characters. + let mixed_registry = || build_registry(&|i| sizes[i % sizes.len()], None); + + // Reference (uninterrupted) run. + let reg = mixed_registry(); + let server = MockServer::start(script(STEPS - 1)); + let mut s = JsonlStorage::create(&dir, "ref", None).unwrap(); + let mut p = provider(server.url(), &system, ®); + expect_final(run_turn(&mut s, &mut p, ®, "start the mission")); + let ref_bodies = server.bodies(); + assert_eq!(ref_bodies.len(), STEPS); + let ref_steps = steps_from_storage(&s, &ref_bodies); + + // Crash right after the 5th tool settled and the machine returned to + // planning: the file is cut after that `state: planning` line. Each + // tool step is 5 lines (state tool_call, intent, tool_result, state + // planning, usage row of the next request); the file starts with the + // header, the user message, the first usage row and the first + // tool_call state. A cut at a line boundary is byte-for-byte what a + // process killed there leaves behind (one fsynced line per commit). + let lines: Vec = std::fs::read_to_string(dir.join("ref.jsonl")) + .unwrap() + .lines() + .map(str::to_string) + .collect(); + let k = 6 + 5 * (CRASH_AFTER_TOOLS - 1); + let cut_line: Value = serde_json::from_str(&lines[k]).unwrap(); + assert_eq!(cut_line["kind"], json!("state")); + assert_eq!(cut_line["pc"], json!("planning")); + let cut = dir.join("cut.jsonl"); + std::fs::write(&cut, format!("{}\n", lines[..=k].join("\n"))).unwrap(); + + // Resume with a brand-new process-equivalent: new provider, new + // registry, new mock answering the remaining script. + let reg2 = mixed_registry(); + let server2 = MockServer::start(script(STEPS - 1)[CRASH_AFTER_TOOLS..].to_vec()); + let mut s2 = JsonlStorage::open(&cut).unwrap(); + let mut p2 = provider(server2.url(), &system, ®2); + expect_final(resume_turn(&mut s2, &mut p2, ®2)); + let post_bodies = server2.bodies(); + assert_eq!(post_bodies.len(), STEPS - CRASH_AFTER_TOOLS); + + // Prefix stability across the crash boundary: identical system and + // tools characters before and after, and the resumed requests are + // BYTE-IDENTICAL to the uninterrupted run's (same cache prefix). + for (i, b) in post_bodies.iter().enumerate() { + let (got, want) = (b.to_string(), ref_bodies[CRASH_AFTER_TOOLS + i].to_string()); + if got != want { + let at = got + .chars() + .zip(want.chars()) + .take_while(|(a, b)| a == b) + .count(); + let ctx = |t: &str| { + t.chars() + .skip(at.saturating_sub(80)) + .take(200) + .collect::() + }; + panic!( + "post-resume request {i} differs at char {at} (len {} vs {}):\n got: {}\nwant: {}", + got.len(), + want.len(), + ctx(&got), + ctx(&want) + ); + } + } + // The resumed session's rows only cover post-crash requests, but the + // file kept the pre-crash rows: pre + post == the whole mission. + let all_steps = steps_from_storage(&s2, &ref_bodies); + assert_eq!(all_steps, ref_steps); + let post0 = wire_stats(&post_bodies[0]); + let pre_last = wire_stats(&ref_bodies[CRASH_AFTER_TOOLS - 1]); + assert_eq!(post0.system_chars, pre_last.system_chars); + assert_eq!(post0.tools_chars, pre_last.tools_chars); + + let run = MissionRun { + bodies: ref_bodies[..CRASH_AFTER_TOOLS] + .iter() + .chain(post_bodies.iter()) + .cloned() + .collect(), + steps: all_steps, + max_result_chars: *sizes.iter().max().unwrap() as u64, + }; + report("mixed_with_resume", &run); + assert_eq!(run.steps.len(), STEPS); + let v = violations(&run, MIXED_TOTAL_BOUND); + assert!(v.is_empty(), "{v:#?}"); +} + +// --------------------------------------------------------------------------- +// Self-tests: the guard has teeth +// --------------------------------------------------------------------------- + +fn has(v: &[String], needle: &str) -> bool { + v.iter().any(|m| m.contains(needle)) +} + +#[test] +fn check_bound_reports_instead_of_panicking() { + assert!(check_bound("x", 10, 10).is_ok()); + assert!(check_bound("x", 11, 10).is_err()); +} + +#[test] +fn bloated_system_prompt_is_a_violation() { + let dir = tmpdir("bloat-system"); + let bloated = format!("{}{}", system_prompt(), filler(SYSTEM_BLOAT, 1)); + let (run, _s) = run_with(&dir, "bs", 20, 200, &bloated, None, false); + let v = violations(&run, MANY_SMALL_TOTAL_BOUND); + println!("bloated system -> {v:#?}"); + assert!(has(&v, "first-step prefix"), "{v:#?}"); + assert!(has(&v, "total request chars"), "{v:#?}"); +} + +#[test] +fn one_extra_bloated_tool_spec_is_a_violation() { + let dir = tmpdir("bloat-tool"); + let (run, _s) = run_with( + &dir, + "bt", + 20, + 200, + &system_prompt(), + Some(Box::new(BloatedTool(TOOL_BLOAT))), + false, + ); + let v = violations(&run, MANY_SMALL_TOTAL_BOUND); + println!("bloated tool -> {v:#?}"); + assert!(has(&v, "first-step prefix"), "{v:#?}"); + assert!(has(&v, "total request chars"), "{v:#?}"); +} + +#[test] +fn history_sent_twice_is_a_violation() { + let dir = tmpdir("history-twice"); + let (run, _s) = run_with(&dir, "ht", 8, 20_000, &system_prompt(), None, true); + let v = violations(&run, READ_HEAVY_TOTAL_BOUND); + println!("history twice -> {v:#?}"); + assert!(has(&v, "super-linearly"), "{v:#?}"); + assert!(has(&v, "total request chars"), "{v:#?}"); + assert!(has(&v, "not an append"), "{v:#?}"); +} + +#[test] +fn unstable_prefix_and_reshuffled_history_are_violations() { + let dir = tmpdir("unstable"); + let mut run = many_small_steps(&dir, "us"); + assert!(check_prefix_stable(&run.steps).is_ok()); + assert!(check_append_only(&run.bodies).is_ok()); + // A system prompt that drifts by one character on one step. + run.steps[3].system += 1; + assert!(check_prefix_stable(&run.steps).is_err()); + // Two messages swapped in a later request. + let msgs = run.bodies[5]["messages"].as_array_mut().unwrap(); + msgs.swap(1, 2); + assert!(check_append_only(&run.bodies).is_err()); +} diff --git a/docs/architecture.md b/docs/architecture.md index ac90510..fafe5da 100644 --- a/docs/architecture.md +++ b/docs/architecture.md @@ -82,7 +82,7 @@ graph TD |---|---| | `entries` | **Append-only, immutable** conversation tree: `id`, `parentId`, `seq`, `kind`, `payload`, `timestamp` | | `registers` | Namespaced mutable cells (`lane` / `op` / `pending` / `fact`), overwritten per key on `set`, removed on `delete` | -| `usage` ledger | Append-only cost rows per provider attempt | +| `usage` ledger | Append-only cost rows per provider attempt; the provider-reported `usage` object plus, when the provider reports it, a namespaced `tole_wire` = `{system_chars, tools_chars, history_chars, messages}` request-size split (#211) | | `state` | Singleton register: `pc` (program counter), `seq` — transition via CAS on `seq` | Invariants (all backends): diff --git a/docs/epics.md b/docs/epics.md index 23714bb..b2fd00e 100644 --- a/docs/epics.md +++ b/docs/epics.md @@ -205,8 +205,7 @@ Everything below is merged; CHANGELOG entries carry the detail. Dependabot PRs no longer blocked by naming/description/CLA checks (#322 plus a ruleset context change); dependency bumps (rmcp 3.5.0, tokio 1.53.2, libc 0.2.190, uuid 1.27.0). No behavior changes. -- **0.8.0 Project config `.tole/config.toml` (#208) (release-candidate on - develop 2026-10-09):** merged to develop 2026-10-09 (parts #342/#343/#345/#346). Strict TOML schema, +- **0.8.0 Project config `.tole/config.toml` (#208) (released 2026-10-09):** merged to develop 2026-10-09 (parts #342/#343/#345/#346). Strict TOML schema, content-bound trust store outside the repo, fail-closed gate on the protocol faces, `tole config check|trust|untrust`, precedence flag > env > config > default for every key (lists replaced wholesale, booleans only turned on by diff --git a/docs/mobile-control.md b/docs/mobile-control.md index 0d2261b..4dbc683 100644 --- a/docs/mobile-control.md +++ b/docs/mobile-control.md @@ -16,7 +16,7 @@ rate-limited per source IP). |---|---|---| | List sessions | `GET /sessions` | id + busy per session | | Start a mission | `POST /sessions` `{cwd}` → `POST /sessions/{id}/prompt` `{text}` | the prompt is the mission goal (`tole mission` semantics chain turns; a serve prompt runs ONE turn — mission chaining over serve is a follow-up) | -| Mission status | `GET /sessions/{id}/status` | busy flag, entry count, usage totals, mission cost report (`fact/mission` register, #201) | +| Mission status | `GET /sessions/{id}/status` | busy flag, entry count, `usage_report` (tokens, cache-hit rate, request-size split; #211), mission cost report (`fact/mission` register, #201) | | **Pending approvals** | `GET /approvals` | queue entries: id, session, tool, status; expires to denied after 15 min | | **Approve / deny** | `POST /approvals/{id}/decision` `{"decision":"allow"\|"deny"}` | allow = one-shot for exactly that (session, tool+input) + automatic resume; deny = recorded verdict | | Cancel a turn | `POST /sessions/{id}/cancel` (#178) | the turn settles `cancelled`, durably | @@ -32,6 +32,28 @@ The round trip the app must support (the cross-repo acceptance): 4. `202` (allow) means the resume is running; `GET /sessions/{id}/status` until `busy` clears. +## Status JSON fields + +`GET /sessions/{id}/status` returns `{id, entries, busy, usage_report}`. +`entries` and `usage_report` are `null` while a turn is in flight (the +session storage is busy). `usage_report` was added by #211 and is purely +additive: clients must ignore unknown fields (the existing three keep their +meaning). Its shape — every unknown value is `null`, never a fabricated 0: + +| Field | Meaning | +|---|---| +| `steps` | provider steps that reported usage (the same count the mission step budget uses; rows holding only `tole_wire` are not counted) | +| `prompt_tokens`, `completion_tokens` | sums over the usage ledger | +| `reasoning_tokens` | sum of `completion_tokens_details.reasoning_tokens`; `null` if no step reported it | +| `cached_tokens` | sum of cached prompt tokens (`prompt_tokens_details.cached_tokens`, falling back to `cached_read_tokens` and the other gateway spellings); `null` if no step reported it | +| `cache_hit_rate` | cached / prompt over the steps that reported cached tokens; `null` if unknown | +| `wire` | `null` for sessions recorded before #211, else `{first, last, history_growth_per_step}`; `first`/`last` = `{prefix_chars, system_chars, tools_chars, history_chars, messages}` of the first/last step that has it (characters, not bytes; `prefix_chars` = system + tools) | + +The mission cost report (`fact/mission`) is rendered by `tole status`; it +gained an additive `cached_tokens` (`null` when no step reported it). The +MCP/ACP `tole_session_status` tool returns the same `usage_report` next to +`busy` and `entries`. + ## Auth model The serve token **is approval authority** (see the threat model): a diff --git a/docs/threat-model.md b/docs/threat-model.md index f568976..5ec3641 100644 --- a/docs/threat-model.md +++ b/docs/threat-model.md @@ -37,7 +37,7 @@ lives) / **gap** (tracked) / **N/A** (with rationale). | OWASP item | Status | Where / rationale | |---|---|---| | Prompt injection (direct/indirect) | **control + documented residual** | Tool results on the wire are wrapped in `TOOL_RESULT_BEGIN/END` fences with inner fence markers neutralized (this fix); secrets scrubbed (E11). Residual: an injected instruction inside data can still steer the model — no framework fully solves this; delimiters + user-visible tool audit lines (`what: …`) limit blast radius. | -| Insecure output handling | **control** | Same fence treatment; result sizes capped (read_file 1 MiB cap, run_command 4000/2000 chars, job_poll 8 KiB window). | +| Insecure output handling | **control** | Same fence treatment; result sizes capped (read_file 1 MiB file cap and 20,000-char head cap per call, run_command 4000/2000 chars, job_poll 8 KiB window). | | Excessive agency / tool misuse | **control (defense-in-depth) + documented residual** | Primary control: the per-call approval gate (every non-ReadOnly spawn needs consent). Secondary: `check_destructive_argv` blocklist refuses headline catastrophic patterns (teardown tools, recursive rm escaping the workspace, wrapper-nested variants) in BOTH `run_command` and `job_start`. The blocklist is best-effort by nature — wrapper/interpreter bypasses (`sudo apt ...`, `timeout 5 dd ...`, `find / -delete`, `python -c rmtree`) are expected and accepted residual risk; a real argv sandbox is a different product and explicitly out of scope. | | Sensitive data disclosure (env) | **gap → fixed here (ENV-1)** | Children now inherit a scrubbed env: any var whose name contains `API_KEY`, `_SECRET`, `_TOKEN`, or `PASSWORD` is removed (`GITHUB_TOKEN` kept for gh/git auth — documented trade-off). | | Supply chain (deps) | **control** | Cargo Audit + Trivy FS Scan required by rulesets on every PR and push. |