A small, hackable project for running a local LLM on Apple Silicon (via mlx-lm) and giving it a set of callable tools. The model decides when to search the web, read a page, look something up on Wikipedia, or check the weather — and answers from the results.
FastAPI backend · Next.js + TypeScript frontend · mlx-lm · fully local, no API keys.
| Tool | What it does | Backed by |
|---|---|---|
web_search |
Web search → list of titles, URLs, snippets | DuckDuckGo (ddgs, keyless) |
web_fetch |
Fetch a URL and extract readable content as Markdown | httpx + trafilatura (markdownify fallback) |
wikipedia |
Look up a topic and return a summary + article URL | Wikipedia public API |
weather |
Current conditions + multi-day forecast for a place | Open-Meteo (keyless) |
Each tool can be toggled on/off per request from the UI's Tools panel; only enabled tools are exposed to the model.
1. Ask your question goes to the LLM with the enabled tool schemas
2. Decide the model (e.g. Qwen3) decides whether to call a tool, and which
3. Dispatch the backend runs the tool and feeds the result back to the model
4. Answer the model writes a grounded answer, streamed token-by-token,
with a collapsible Reasoning panel and cited source URLs
The model is handed tool schemas via the tokenizer's native chat template
(tools=), so models trained on tool-calling use them reliably. The tool
registry lives in backend/src/tools/ — adding a tool is a single new module
exposing a TOOL object (schema + run(args, on_event)), registered in
tools/__init__.py. The tool-calling loop and streaming are in
backend/src/llm.py.
- macOS on Apple Silicon (M1/M2/M3/M4) — for MLX Metal GPU inference
- uv (Python ≥ 3.11) and Node 20+
make install # uv sync (backend) + npm install (frontend)
make dev # backend :8000 + frontend :3000 (background)
# open http://localhost:3000 — pick/download a model in the Model panel,
# toggle tools, and chat. Run `make stop` to shut down.Run pieces individually with make backend / make frontend. Configuration is
optional via backend/.env (see backend/.env.example) — MODEL sets the
default model; no keys are required for the tools.
The first chat loads the model into memory (downloads it first if needed — progress shows in the Model panel). Any mlx-community model works; Qwen3 models are recommended for reliable tool-calling.
| Endpoint | Method | Purpose |
|---|---|---|
/api/chat |
POST | Chat with tool-calling (SSE stream). Body: {message, history, tools} |
/api/tools |
GET | List available tools |
/api/status |
GET | Model + tools status |
/api/models/suggested |
GET | Suggested mlx-community models |
/api/models/cached |
GET | Locally downloaded models |
/api/models/activate |
POST | Switch active model |
/api/models/load |
POST | Download + load a model (SSE progress) |
tools in the chat body is the list of enabled tool ids (omit / null = all
enabled, [] = none).