Local engine model management: core, gateway, and protocol

CHANGELOG.md

@@ -9,6 +9,37 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0
 
 ### Added
 
+- **Configurable model-call deadline.** The gateway's per-turn model call
+  used a hard-coded 180s cap, which local engines on a loaded machine can
+  exceed while prefilling a large prompt (a single 2k-token prompt on a
+  busy CPU box took ~15 minutes here). `RUSTYCLAW_MODEL_TIMEOUT_SECS`
+  now overrides it (default stays 180), so local-model setups can give
+  turns the time they need instead of dying mid-prefill.
+
+- **Local model management in the UI: on-disk models in every picker, and
+  engine parameters.** The chat composer's model dropdown (desktop) and the
+  provider/model selectors (TUI) now show what is actually available locally
+  for the local engine providers — Joshua, llama.cpp, LM Studio, exo, Ollama
+  — even when the engine server is not running. The gateway's
+  `ProviderModelList` handler (and the TUI's beside-the-vault fetches) merge
+  the live API list with the engine registry's own model list; for the
+  file-based engines (Joshua, llama.cpp) that list is a scan of the models
+  directory, so a model only has to be on disk to be pickable. llama.cpp's
+  `list_models` gained the same on-disk scan. `EngineInfoDto` now carries the
+  engine's full `EngineConfig`, and the "Local Engines & Models" dialog
+  (desktop) gained a parameters editor per engine tab: context window
+  (`--n-ctx`/`--ctx-size`/`--num-ctx`), default model (picked from the local
+  model list), auto-start, and — for Joshua — device (`--device`), huge pages
+  (`--huge-pages`), `--mmap`, `--lazy-weights`, `--max-output-tokens` and
+  `--max-concurrency`. The TUI `/engines` panel got the same editor as a
+  keyboard-driven mode (p to edit, +/- adjust, x clear, Enter saves via
+  `EngineConfigSet`, which persists to config.toml). Parameters apply on the
+  next engine start or model load; per-load context overrides for Joshua now
+  map to `--n-ctx`. New `EngineConfig` fields are optional and defaulted, so
+  existing configs parse unchanged.
+
+### Added
+
 - **Messenger-kind registry (plugin architecture, `docs/PLUGIN_ARCHITECTURE.md`
   §14).** Messenger backends had the pre-phase-0 tool shape: a static schema
   table beside a hardcoded factory `match` in the gateway, plus a `cfg!`

config.example.toml

@@ -78,16 +78,26 @@ use_secrets = true
 # models = ["qwen3-coder-30b", "llama-3.3-70b"] # optional; /v1/models is also queried
 
 # ── Local inference engines ───────────────────────────────────────────────
-# Engines are managed from the TUI /engines panel (install/start/stop/pull).
-# Joshua (https://github.com/rexlunae/joshua) is a pure-Rust GGUF server:
-# one process serves one model, picked from models_dir at startup.
+# Engines are managed from the TUI /engines panel (install/start/stop/pull)
+# and the desktop "Local Engines & Models" dialog (which also edits the
+# parameters below).  Joshua (https://github.com/rexlunae/joshua) is a
+# pure-Rust GGUF server: one process serves one model, picked from models_dir
+# at startup.
 #
 # [engines.joshua]
 # enabled = true
 # port = 8331                                   # RustyClaw's default for joshua
 # models_dir = "/home/user/.rustyclaw/models/joshua"
 # default_model = "qwen3-4b-Q4_K_M"             # GGUF file stem to serve
 # auto_start = true                             # start with the gateway
+# context_length = 8192                         # --n-ctx (joshua/llama.cpp/ollama)
+# device = "auto"                               # joshua --device: auto|cpu|metal|cuda
+# huge_pages = "off"                            # joshua --huge-pages: off|transparent|2mb|1gb|huge
+# mmap = false                                  # joshua --mmap: require a memory-mappable model
+# lazy_weights = false                          # joshua --lazy-weights: optimise for huge models
+# max_output_tokens = 4096                      # joshua --max-output-tokens
+# max_concurrency = 8                           # joshua --max-concurrency
+# extra_args = []                               # any other flags, passed verbatim
 #
 # [engines.ollama]
 # enabled = true

crates/rustyclaw-core/src/engines/downloaders.rs

@@ -149,7 +149,7 @@ impl LocalEngine for HuggingFaceDownloader {
         anyhow::bail!("The Hugging Face CLI is a downloader tool, not a server — nothing to start.")
     }
 
-    async fn stop(&self) -> Result<String> {
+    async fn stop(&self, _cfg: &EngineConfig) -> Result<String> {
         anyhow::bail!("The Hugging Face CLI is a downloader tool, not a server — nothing to stop.")
     }
 

crates/rustyclaw-core/src/engines/exo.rs

@@ -156,7 +156,7 @@ impl LocalEngine for ExoEngine {
         }
     }
 
-    async fn stop(&self) -> Result<String> {
+    async fn stop(&self, _cfg: &EngineConfig) -> Result<String> {
         Self::sh("pkill -f 'exo' 2>/dev/null; echo 'stopped'").await
     }
 

crates/rustyclaw-core/src/engines/joshua.rs

@@ -86,6 +86,32 @@ impl JoshuaEngine {
             .unwrap_or_default()
     }
 
+    /// `(model name, port)` for every `joshua serve` process running on the
+    /// host — including servers started manually outside RustyClaw — so the
+    /// UI can show which models are actually running.  Best-effort (Linux);
+    /// empty elsewhere.
+    async fn running_servers() -> Vec<(String, Option<u16>)> {
+        let lines = crate::engines::running_server_cmdlines("joshua serve").await;
+        crate::engines::parse_server_cmdlines(&lines, &["--model", "-m"], &["--addr", "-a"], 8080)
+    }
+
+    /// The endpoint the engine should actually talk to: the configured one
+    /// when it answers, otherwise the first `joshua serve` detected on the
+    /// host — so the engine genuinely reaches a joshua that was started
+    /// outside RustyClaw instead of pretending it is unreachable.
+    async fn effective_endpoint(cfg: &EngineConfig) -> String {
+        let configured = Self::endpoint(cfg);
+        if Self::is_running(&configured).await {
+            return configured;
+        }
+        Self::running_servers()
+            .await
+            .first()
+            .and_then(|(_, port)| *port)
+            .map(|port| format!("http://127.0.0.1:{}", port))
+            .unwrap_or(configured)
+    }
+
     async fn sh(script: &str) -> Result<String> {
         let output = tokio::process::Command::new("sh")
             .arg("-c")
@@ -109,22 +135,37 @@ impl JoshuaEngine {
 
     /// Start `joshua serve` for the given GGUF file.
     async fn spawn_server(cfg: &EngineConfig, model_path: &Path) -> Result<String> {
+        if !Self::is_installed().await {
+            anyhow::bail!(
+                "joshua is not installed. Install it from the engines dialog, or \
+                 `cargo install --git https://github.com/rexlunae/joshua joshua`."
+            );
+        }
         let port = cfg.port.unwrap_or(DEFAULT_PORT);
         let mut cmd = format!(
-            "nohup joshua serve --model '{}' --addr 127.0.0.1:{}",
-            model_path.display(),
+            "nohup joshua serve --model {} --addr 127.0.0.1:{}",
+            sh_quote(&model_path.display().to_string()),
             port
         );
+        // Typed parameters (context window, device, huge pages, …) become
+        // flags first; raw extra_args come after so an explicit flag in
+        // extra_args still wins (clap takes the last occurrence).  Every
+        // extra_arg is sh_quoted: they are client-supplied strings that
+        // reach a `sh -c` command line, so metacharacters must stay inert.
+        for arg in joshua_serve_flags(cfg) {
+            cmd.push(' ');
+            cmd.push_str(&arg);
+        }
         for arg in &cfg.extra_args {
             cmd.push(' ');
-            cmd.push_str(arg);
+            cmd.push_str(&sh_quote(arg));
         }
         cmd.push_str(" > /dev/null 2>&1 &");
         Self::sh(&cmd).await.ignore();
 
         // GGUF loading is mmap-based and fast, but give it a moment.
         let endpoint = Self::endpoint(cfg);
-        for _ in 0..10 {
+        for _ in 0..20 {
             tokio::time::sleep(std::time::Duration::from_millis(500)).await;
             if Self::is_running(&endpoint).await {
                 return Ok(format!(
@@ -137,7 +178,55 @@ impl JoshuaEngine {
                 ));
             }
         }
-        Ok("joshua start command issued; the model may still be loading.".into())
+        // The server has not answered yet.  Distinguish a model that is
+        // still loading (process alive) from one that failed outright
+        // (process gone): the former is not an error — large models can
+        // take minutes to come up on a busy machine — while the latter
+        // deserves a real error with a way forward.
+        #[cfg(target_os = "linux")]
+        {
+            if Self::server_process_alive(port).await {
+                return Ok(format!(
+                    "joshua start command issued; the model may still be loading on {}.",
+                    endpoint
+                ));
+            }
+            anyhow::bail!(
+                "joshua serve did not answer on {} and its process exited — the model file may \
+                 be invalid or joshua failed at load. Run `joshua serve --model '{}'` manually \
+                 to see the error.",
+                endpoint,
+                model_path.display()
+            )
+        }
+        // No process inspection on this platform: a slow load must not be
+        // reported as a failure just because we cannot tell it apart from a
+        // crashed process.
+        #[cfg(not(target_os = "linux"))]
+        return Ok(format!(
+            "joshua start command issued; the model may still be loading on {}.",
+            endpoint
+        ));
+    }
+
+    /// Whether a `joshua serve` process for the given port is still running
+    /// (best-effort; Linux only).  Used to tell "still loading" apart from
+    /// "crashed" when the health probe has not answered yet.  The port match
+    /// must terminate the digit run, so 808 does not match a server on 8080.
+    #[cfg(target_os = "linux")]
+    async fn server_process_alive(port: u16) -> bool {
+        let needle = format!("127.0.0.1:{}", port);
+        crate::engines::running_server_cmdlines("joshua serve")
+            .await
+            .iter()
+            .any(|line| {
+                line.match_indices(&needle).any(|(end, _)| {
+                    line[end..]
+                        .chars()
+                        .next()
+                        .is_none_or(|c| !c.is_ascii_digit())
+                })
+            })
     }
 }
 
@@ -150,32 +239,6 @@ pub fn default_models_dir() -> PathBuf {
         .join("joshua")
 }
 
-/// Scan a directory (recursively, one level of subdirectories) for GGUF files.
-pub fn scan_gguf_models(dir: &Path) -> Vec<PathBuf> {
-    let mut found = Vec::new();
-    let Ok(entries) = std::fs::read_dir(dir) else {
-        return found;
-    };
-    for entry in entries.flatten() {
-        let path = entry.path();
-        if path.is_dir() {
-            // HF downloads land in per-repo subdirectories.
-            if let Ok(sub) = std::fs::read_dir(&path) {
-                for sub_entry in sub.flatten() {
-                    let sub_path = sub_entry.path();
-                    if sub_path.extension().is_some_and(|e| e == "gguf") {
-                        found.push(sub_path);
-                    }
-                }
-            }
-        } else if path.extension().is_some_and(|e| e == "gguf") {
-            found.push(path);
-        }
-    }
-    found.sort();
-    found
-}
-
 /// Resolve which GGUF file to serve, in priority order:
 /// 1. an explicit `--model <path>` in `extra_args`,
 /// 2. the configured `default_model` (matched against file stems),
@@ -287,6 +350,8 @@ impl LocalEngine for JoshuaEngine {
         let presence = self.detect().await;
         let endpoint = Self::endpoint(cfg);
         let available = scan_gguf_models(&Self::models_dir(cfg)).len() as u32;
+        let configured_port = cfg.port.unwrap_or(DEFAULT_PORT);
+        let detected = Self::running_servers().await;
 
         let run_status = if Self::is_running(&endpoint).await {
             let loaded = Self::loaded_model_ids(&endpoint).await.len() as u32;
@@ -295,7 +360,26 @@ impl LocalEngine for JoshuaEngine {
                 loaded_models: loaded,
                 available_models: available.max(loaded),
             }
+        } else if let Some((_, port)) = detected
+            .iter()
+            .find(|(_, port)| *port == Some(configured_port))
+        {
+            // A joshua server on the engine's own configured port is running
+            // (e.g. started manually outside RustyClaw): report it so the
+            // UI's lifecycle gating stays tied to this engine's server.
+            let detected_endpoint = port
+                .map(|port| format!("http://127.0.0.1:{}", port))
+                .unwrap_or(endpoint);
+            let loaded = detected.len() as u32;
+            EngineRunStatus::Running {
+                endpoint: detected_endpoint,
+                loaded_models: loaded,
+                available_models: available.max(loaded),
+            }
         } else {
+            // Joshua servers on *other* ports belong to someone else: the
+            // engine itself is not running, and reporting Running would hide
+            // the Start button while Stop could not touch that server.
             EngineRunStatus::Stopped
         };
 
@@ -331,21 +415,55 @@ impl LocalEngine for JoshuaEngine {
         if Self::is_running(&endpoint).await {
             return Ok("joshua is already running.".into());
         }
+        // Only a server on this engine's own port counts as "already
+        // running": a joshua started outside RustyClaw on another port must
+        // not prevent the configured server from starting.
+        let configured_port = cfg.port.unwrap_or(DEFAULT_PORT);
+        if let Some((_, port)) = Self::running_servers()
+            .await
+            .iter()
+            .find(|(_, port)| *port == Some(configured_port))
+        {
+            return Ok(format!(
+                "joshua is already running on port {} (detected outside RustyClaw). \
+                 Stop it manually or choose another port for this engine.",
+                port.unwrap_or(configured_port)
+            ));
+        }
         let model_path = resolve_model_path(cfg)?;
         Self::spawn_server(cfg, &model_path).await
     }
 
-    async fn stop(&self) -> Result<String> {
-        Self::sh("pkill -f 'joshua serve' 2>/dev/null; echo 'stopped'").await
+    async fn stop(&self, cfg: &EngineConfig) -> Result<String> {
+        // Scoped to the configured port: `pkill -f 'joshua serve'` would
+        // also kill servers started manually on other ports.  The pattern
+        // terminates the digit run (`( |$)`), so a short port such as 808
+        // cannot match a server running on 8080.
+        let port = cfg.port.unwrap_or(DEFAULT_PORT);
+        Self::sh(&format!(
+            "pkill -f 'joshua serve .*127.0.0.1:{}( |$)' 2>/dev/null; echo 'stopped'",
+            port
+        ))
+        .await
     }
 
     async fn list_models(&self, cfg: &EngineConfig) -> Result<Vec<LocalModel>> {
-        let endpoint = Self::endpoint(cfg);
-        let loaded_ids = if Self::is_running(&endpoint).await {
+        // Talk to the configured endpoint when it answers, otherwise to the
+        // first server detected on the host — the loaded set should reflect
+        // the joshua the engine can actually reach.
+        let endpoint = Self::effective_endpoint(cfg).await;
+        let mut loaded_ids = if Self::is_running(&endpoint).await {
             Self::loaded_model_ids(&endpoint).await
         } else {
             Vec::new()
         };
+        // Mark models served by running joshua processes (even ones started
+        // outside RustyClaw) as loaded, so the UI shows what is running.
+        for (name, _) in Self::running_servers().await {
+            if !loaded_ids.iter().any(|id| id == &name) {
+                loaded_ids.push(name);
+            }
+        }
 
         let dir = Self::models_dir(cfg);
         let mut models: Vec<LocalModel> = scan_gguf_models(&dir)
@@ -477,21 +595,40 @@ impl LocalEngine for JoshuaEngine {
         let path = find_model_file(&dir, model)
             .ok_or_else(|| anyhow::anyhow!("Model '{}' not found in {}", model, dir.display()))?;
 
+        // "Already loaded" means the server the engine actually talks to
+        // (the configured endpoint) serves this model — a joshua started
+        // outside RustyClaw on another port must not make Load claim
+        // success while the configured server has nothing loaded.
         let endpoint = Self::endpoint(cfg);
         if Self::is_running(&endpoint).await {
             let already = Self::loaded_model_ids(&endpoint).await;
             if already.iter().any(|id| id == model) {
                 return Ok(format!("Model '{}' is already loaded", model));
             }
-            self.stop().await.ignore();
+            self.stop(cfg).await.ignore();
             tokio::time::sleep(std::time::Duration::from_millis(500)).await;
         }
         Self::spawn_server(cfg, &path).await
     }
 
-    async fn unload(&self, model: &str, _cfg: &EngineConfig) -> Result<String> {
+    async fn unload(&self, model: &str, cfg: &EngineConfig) -> Result<String> {
+        // A model served by a joshua started outside RustyClaw is not ours
+        // to kill — stopping it would surprise the person who started it.
+        let configured_port = cfg.port.unwrap_or(DEFAULT_PORT);
+        if let Some((_, port)) = Self::running_servers()
+            .await
+            .iter()
+            .find(|(n, p)| n == model && *p != Some(configured_port))
+        {
+            anyhow::bail!(
+                "Model '{}' is served by a joshua started outside RustyClaw (port {}). \
+                 Stop it manually to unload it.",
+                model,
+                port.unwrap_or(8080)
+            );
+        }
         // One process serves one model: unloading stops the server.
-        self.stop().await.ignore();
+        self.stop(cfg).await.ignore();
         Ok(format!("Model '{}' unloaded (joshua stopped)", model))
     }