Skip to content
Open
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
6 changes: 4 additions & 2 deletions .claude/rules/file-map.md
Original file line number Diff line number Diff line change
Expand Up @@ -6,7 +6,7 @@ Full module index for LocalChat. **Keep this current** — update in the same co
|------|------|
| `app.py` | Entry point; `main()` + `create_uvicorn_app()` for dev/prod (FastAPI + Uvicorn) |
| `src/app_fastapi.py` | `create_app()` — FastAPI factory, pure wiring only (no I/O); safe to call in tests |
| `src/app_bootstrap.py` | `bootstrap_app(app)` — all startup I/O (Ollama, DB, caching, plugins, connectors, reranker) plus `_clear_upload_staging()`, which removes staged uploads an interrupted ingest left behind; called from `app.py` only |
| `src/app_bootstrap.py` | `bootstrap_app(app)` — all startup I/O (Ollama, DB, caching, plugins, connectors, the core scheduler with the weekly reranker fine-tune as its first subscriber) plus `_clear_upload_staging()`, which removes staged uploads an interrupted ingest left behind; called from `app.py` only |
| `src/config.py` | All configuration constants, loads `.env` |
| `src/models.py` | Pydantic request/response models |
| `src/security_fastapi.py` | JWT (`PyJWT`), rate limiting (`slowapi`), CORS (Starlette middleware), `setup_security_headers()` (CSP + nosniff + referrer + framing + HSTS); `resolve_principal()` — the one place a caller is identified: decode, fail-closed revocation, and the role read from the database |
Expand Down Expand Up @@ -174,11 +174,13 @@ Full module index for LocalChat. **Keep this current** — update in the same co
| `src/tools/executor.py` | Ollama tool-call loop (multi-turn until final response) |
| `src/tools/registry.py` | Tool registration with JSON schemas |
| `src/tools/builtin.py` | Built-in tools: document search, calculator, datetime |
| `src/tools/plugin_loader.py` | Loads `.py` plugins from `plugins/` at startup; one with `setup(services)` gets its manifest validated and its services built, one without loads as legacy. A failing plugin is disabled alone, with any tool it registered removed |
| `src/tools/plugin_loader.py` | Loads `.py` plugins from `plugins/` at startup; one with `setup(services)` gets its manifest validated, its services built and its hooks subscribed, one without loads as legacy. A failing plugin is disabled alone, with any tool it registered removed; unloading also ends its subscriptions |
| `src/plugins/manifest.py` | PC-1 — `read_manifest()`: `PLUGIN_SCOPE`, `PLUGIN_MIN_ROLE`, `PLUGIN_HOOKS` (checked against the catalogue), `PLUGIN_CONFIG`; `PLUGIN_CONTRIBUTES` must be False. Raises `ManifestError` |
| `src/plugins/services.py` | PC-1 — the handles a plugin's `setup()` receives: `retrieval` (the request's workspace only, tier capped by the manifest), `llm.embed`, `config`, `identity` (read only), `tools.register` (refuses a taken name) |
| `src/plugins/hooks.py` | PC-2 — `HookBus` (`subscribe`/`unsubscribe` per owner, `emit` skipping a subscriber that raises), `Scheduler` (one daemon thread emitting `on_scheduler_tick`), the `HOOKS` catalogue and the `hook_bus` singleton |
| `src/plugins/__init__.py` | `build_services()` — types the plugin's settings from `PLUGIN_<PLUGIN>_<KEY>` and assembles its handles |
| `plugins/example_plugin.py` | Example plugin on the PC-1 contract — `setup(services)`, a manifest and a setting (`word_count`, `reverse_text` tools) |
| `tests/unit/test_hook_bus.py` | PC-2 — a raising subscriber skipped while the next runs, a copy of the payload each, unknown hooks refused, numbered ticks that stop; the loader subscribing a plugin's declared hooks, once across a reload, never for a failed setup, and unsubscribing on unload; the exact payloads of an ingest (new and replaced, none on failure) and of a permitted tool, none for a refused one |
| `tests/unit/test_plugin_contract.py` | PC-1 — the manifest (every malformed field refused by name), each handle (retrieval reads the request's workspace and refuses a tier beyond its reach or a key's), the executor hiding and refusing a tool below its role, the loader disabling one plugin alone and removing its half-registered tools, and the guard recording who it admitted |
| `plugins/README.md` | Plugin authoring guide |
| **Graph / Memory / Performance** | |
Expand Down
31 changes: 20 additions & 11 deletions .claude/rules/plugins.md
Original file line number Diff line number Diff line change
Expand Up @@ -3,14 +3,14 @@
How plugins extend LocalChat without destabilising the core. Read alongside the
"Plugin Contract" section in `CLAUDE.md` — that states the principle; this is the rule.

> ## ⚠ Status: partly built (PC-1 built 2026-11-11; PC-4 2026-10-23)
> ## ⚠ Status: partly built (PC-1 built 2026-11-11; PC-2 2026-11-13; PC-4 2026-10-23)
>
> | Described here | Reality |
> |---|---|
> | The manifest (`PLUGIN_SCOPE`, `PLUGIN_MIN_ROLE`, `PLUGIN_HOOKS`, `PLUGIN_CONFIG`) | **Built** (PC-1) in `src/plugins/manifest.py`; a malformed one disables that plugin only |
> | The services catalogue | **Built** (PC-1) in `src/plugins/services.py`: `retrieval`, `llm` (`embed` only), `config`, `identity`, and `tools` for registering. `storage` and `llm.complete` are deferred until a consumer needs them |
> | `PLUGIN_MIN_ROLE` | **Enforced by the core** (PC-1): the tool executor hides and refuses a tool below its minimum role |
> | The hooks catalogue (`on_document_ingested`, `on_scheduler_tick`, …) | No hook bus (PC-2). `PLUGIN_HOOKS` is validated against the names below and fires nothing |
> | The hooks catalogue | **Built** (PC-2) in `src/plugins/hooks.py`: `on_document_ingested`, `on_scheduler_tick`, `on_tool_invocation`. `on_route_mount` is deferred until a consumer needs it |
> | `plugins/_echo/` reference plugin | Absent (PC-3). `plugins/` holds `example_plugin.py`, on the PC-1 contract |
> | The `core-without-plugins` CI job | **Built** (2026-10-23) in `.github/workflows/tests.yml`; a required check since 2026-10-30 |
>
Expand Down Expand Up @@ -61,18 +61,27 @@ describes the capability generically. A `pricing_lookup` service is a leak; a

## Hooks catalogue

A plugin subscribes via its manifest. The core calls all subscribers and is unaffected
if there are none.
A plugin subscribes via its manifest: each hook named in `PLUGIN_HOOKS` is a module
function of the same name taking the payload, `def on_document_ingested(payload): ...`. A
hook named without its function is a malformed manifest. The core calls all subscribers
and is unaffected if there are none.

| Hook | Fires when | Typical use |
|------|-----------|-------------|
| `on_document_ingested` | A document finishes ingest in `SyncWorker._handle_event()` or `processor.py` | Post-process, extract structured data |
| `on_scheduler_tick` | The core scheduler emits a periodic tick | Background evaluation, batch jobs |
| `on_tool_invocation` | An LLM tool call is dispatched (`src/agent/tool_router.py`) | Register LLM-callable tools |
| `on_route_mount` | App startup, router assembly (`src/app_fastapi.py`) | Contribute a FastAPI `APIRouter` |
| Hook | Fires when | Payload |
|------|-----------|---------|
| `on_document_ingested` | A document finishes ingest in `DocumentProcessor.ingest_document()`, the one path uploads and connectors share | `document_id`, `filename`, `workspace_id`, `chunks`, `replaced`, `source_id` |
| `on_scheduler_tick` | The core scheduler ticks, every `SCHEDULER_TICK_SECONDS` | `tick` (1, 2, …), `at` (ISO 8601, UTC) |
| `on_tool_invocation` | An LLM tool call the caller was permitted has run (`src/tools/executor.py`) | `tool`, `source` (`builtin` or the plugin) |
| `on_route_mount` | *Deferred (2026-11-13).* Plugins load after the app is assembled, so a router they contribute needs a later mount | — |

A subscriber that raises is logged and skipped; the core operation completes normally.
Hooks are best-effort from the core's side.
Hooks are best-effort from the core's side. `emit()` is synchronous and runs subscribers in
the emitting thread, so a slow one slows that operation (an ingest, the scheduler); the tool
loop emits through `asyncio.to_thread`, so a subscriber never stalls the event loop. The
payloads carry identifiers, not content: a tool's arguments hold the user's question, and no
consumer needs them yet.

The weekly reranker fine-tune is the scheduler's first subscriber, a core one: the
scheduler was proven against an existing consumer, not a plugin.

## The manifest

Expand Down
13 changes: 13 additions & 0 deletions CHANGELOG.md
Original file line number Diff line number Diff line change
Expand Up @@ -19,6 +19,19 @@ reasoning attached, in [docs/history/LESSONS_LEARNED.md](docs/history/LESSONS_LE
it, and is refused if the model names it anyway. A malformed manifest disables that plugin
alone. **Upgrading:** nothing to do. A plugin without `setup()` still loads as before, now
listed as legacy in `/api/plugins` with a warning in the log.
- **Plugins can react to the core through hooks (PC-2).** A plugin names a hook in
`PLUGIN_HOOKS` and defines a function of that name: `on_document_ingested` after every
ingest, by upload or connector; `on_scheduler_tick` from a new core scheduler, every
`SCHEDULER_TICK_SECONDS` (default 3600); `on_tool_invocation` after a permitted tool ran. A
subscriber that raises is logged and skipped. **Upgrading:** a manifest naming
`on_route_mount`, or a hook without its function, now disables that plugin;
`on_route_mount` is not offered yet.

### Changed

- **The weekly reranker fine-tune runs on the core scheduler.** It is the scheduler's first
subscriber and runs on the first tick a week after its last attempt, so up to
`SCHEDULER_TICK_SECONDS` later than the timer it replaces.

### Fixed

Expand Down
2 changes: 1 addition & 1 deletion CLAUDE.md
Original file line number Diff line number Diff line change
Expand Up @@ -156,7 +156,7 @@ LocalChat supports plugins (LLM tools, connectors, background work) that extend

The enforcing procedure is a CI job, `core-without-plugins`, that verifies architectural integrity: the core builds, lints, and passes its full test suite **with the `plugins/` directory and all private plugin code absent.** If that job goes red, a plugin dependency has leaked into the core.

> **It exists since 2026-10-23** (PC-4, built with QP-3): it deletes everything in `plugins/` but its README, then runs `ruff` and the fast suite. It has been a required check since 2026-10-30 (see Pull Requests). The manifest and the services catalogue were built on 2026-11-11 (PC-1, `src/plugins/`), with `PLUGIN_MIN_ROLE` enforced by the core; `storage` and `llm.complete` wait for a consumer. The hook bus (PC-2) and the `_echo` reference plugin (PC-3) are designed and unbuilt. What ships is described in [`plugins/README.md`](plugins/README.md). See the status banner in [.claude/rules/plugins.md](.claude/rules/plugins.md).
> **It exists since 2026-10-23** (PC-4, built with QP-3): it deletes everything in `plugins/` but its README, then runs `ruff` and the fast suite. It has been a required check since 2026-10-30 (see Pull Requests). The manifest and the services catalogue were built on 2026-11-11 (PC-1, `src/plugins/`), with `PLUGIN_MIN_ROLE` enforced by the core; `storage` and `llm.complete` wait for a consumer. The hook bus and the core scheduler were built on 2026-11-13 (PC-2, `src/plugins/hooks.py`), with `on_route_mount` deferred until a consumer needs it. The `_echo` reference plugin (PC-3) is designed and unbuilt. What ships is described in [`plugins/README.md`](plugins/README.md). See the status banner in [.claude/rules/plugins.md](.claude/rules/plugins.md).

Full rules -> [.claude/rules/plugins.md](.claude/rules/plugins.md)

Expand Down
1 change: 1 addition & 0 deletions docs/CONFIGURATION.md
Original file line number Diff line number Diff line change
Expand Up @@ -463,6 +463,7 @@ to empty, in which case the router falls back to the active model.
> fixes this product at one node and one process; scale with a second node instead.
| `LOG_FILE` | `logs/app.log` | Path for the file sink |
| `PLUGINS_DIR` | `plugins` | Directory scanned for plugins |
| `SCHEDULER_TICK_SECONDS` | `3600` | Interval of the core scheduler's `on_scheduler_tick`; the weekly reranker fine-tune is one of its subscribers, so it runs at most this late |
| `PLUGIN_<PLUGIN>_<KEY>` | the plugin's declared default | A setting a plugin declares in `PLUGIN_CONFIG`. `<PLUGIN>` is its file name upper-cased, with anything but letters and digits made `_`. Read by `plugin_setting()` in `config.py`; see [plugins/README.md](../plugins/README.md) |
| `PRESENCE_TTL_SECONDS` | `30` | How long a workspace presence entry stays live |
| `CONNECTOR_LOCAL_ROOTS` | *(empty)* | Comma-separated absolute directories the `local_folder` connector may watch. **Empty disables that connector type** — see below |
Expand Down
23 changes: 21 additions & 2 deletions docs/ROADMAP.md
Original file line number Diff line number Diff line change
Expand Up @@ -639,7 +639,26 @@ PC-1 **extends** this foundation: it adds the manifest contract, service injecti

---

### PC-2 — Hook bus and core scheduler
### PC-2 — Hook bus and core scheduler ✅ (built 2026-11-13)

> **As built.** Three points were decided by the maintainer on 2026-11-13, and
> `.claude/rules/plugins.md` was amended to match:
> - `emit()` is synchronous. The tool loop calls it through `asyncio.to_thread`, so a slow
> subscriber cannot stall the event loop.
> - `on_route_mount` is deferred until a consumer needs it: plugins load in `bootstrap_app()`,
> after `create_app()` has assembled the routers, so a contributed router needs a later mount.
> - One daemon thread ticks every `SCHEDULER_TICK_SECONDS` (default 3600). The reranker's
> fine-tune subscribes and runs on the first tick a week after its last attempt, on a
> monotonic clock; the `threading.Timer` it replaces is gone.
>
> Two departures from the spec below. `on_document_ingested` fires from
> `DocumentProcessor.ingest_document()` only: `SyncWorker` ingests through it, so emitting in
> both would announce one document twice. And `on_tool_invocation` fires from the LLM tool
> loop (`src/tools/executor.py`), not `src/agent/tool_router.py`, which dispatches the
> aggregator's retrieval rather than tool calls; only a tool the caller was permitted is
> announced, with its name and source and not its arguments. A plugin subscribes with a
> module function named after the hook.


**What it builds:** the core-to-plugin event surface, and the generic scheduler that replaces ad-hoc timers.

Expand Down Expand Up @@ -2056,7 +2075,7 @@ the generation path is a deterministic pipeline unless an answer-level eval says
| PG-0..PG-8 | **Production-grade hardening** ✅ — see [PRODUCTION_PLAN.md](history/PRODUCTION_PLAN.md). Gated everything below; **gate lifted 2026-08-31**. | done |
| 8 | GKB-1 (schema + two-tier retrieval) ✅ 2026-10-09 — read-only global tier under RLS (0019) | 1 week |
| 9 | GKB-2 (contribution workflow) ✓ backend 2026-10-19 (admin-only contribute, migration 0020), UI 2026-10-20 | 1 week |
| 10 | PC-1 ✅ 2026-11-11 (manifest, services, the core's role gate) + PC-2 (hooks, scheduler) | 1 week |
| 10 | PC-1 ✅ 2026-11-11 (manifest, services, the core's role gate) + PC-2 ✅ 2026-11-13 (hooks, scheduler) | 1 week |
| 11 | PC-3 + PC-4 (echo plugin, CI gate) | 1 week |
| 12 | PR-1 (pricing plugin — private repo) | 1–2 weeks |
| 13 | CONN-1 (connector authorisation model — decision, no code) | 2–3 days |
Expand Down
33 changes: 30 additions & 3 deletions plugins/README.md
Original file line number Diff line number Diff line change
@@ -1,8 +1,8 @@
# LocalChat Plugin System

> Verified against `src/tools/`, `src/plugins/` and `src/app_bootstrap.py` on 2026-11-11.
> PC-1 of the contract in `.claude/rules/plugins.md` ships: the manifest and the services
> below. The hook bus (PC-2) and the `_echo` reference plugin (PC-3) do not exist yet.
> PC-1 and PC-2 of the contract in `.claude/rules/plugins.md` ship: the manifest, the
> services and the hooks below. The `_echo` reference plugin (PC-3) does not exist yet.

Drop a `.py` file here — LocalChat loads it automatically at startup.

Expand All @@ -28,7 +28,7 @@ PLUGIN_META = {"name": "My Plugin", "version": "1.0.0", "description": "Shown in
# The manifest. Every name is optional; these are the defaults.
PLUGIN_SCOPE = "local" # retrieval reach: local | global | hybrid
PLUGIN_MIN_ROLE = "viewer" # who may use its tools: viewer | editor | owner | admin
PLUGIN_HOOKS = [] # validated against the catalogue; nothing fires them until PC-2
PLUGIN_HOOKS = [] # each hook named here is a function of that name; see Hooks
PLUGIN_CONFIG = { # KEY: (type, default); type is str | int | float | bool
"TOP_K": ("int", 5),
}
Expand Down Expand Up @@ -73,6 +73,32 @@ API key's) and then admin (a global administrator).
for a plugin that needs it. Tools run synchronously inside the chat stream, so a completion
from inside one needs that changed first. `PLUGIN_CONTRIBUTES` must be `False`.

### Hooks

A plugin reacts to the core through hooks. Name each one in `PLUGIN_HOOKS` and define a
function of the same name taking one argument, the payload (a fresh `dict` per subscriber):

```python
PLUGIN_HOOKS = ["on_document_ingested", "on_scheduler_tick"]

def on_document_ingested(payload): # document_id, filename, workspace_id, chunks, replaced, source_id
...

def on_scheduler_tick(payload): # tick, at — every SCHEDULER_TICK_SECONDS (default 3600)
...
```

| Hook | Fires when |
|---|---|
| `on_document_ingested` | A document finished ingesting, by upload or connector |
| `on_scheduler_tick` | The core scheduler ticks |
| `on_tool_invocation` | A tool the caller was permitted has run; payload `tool`, `source` |

A hook named without its function disables the plugin. A subscriber that raises is logged
and skipped, and the core carries on. Subscribers run in the thread that emitted, so keep
them short: a slow `on_document_ingested` slows every ingest. A plugin's subscriptions end
when it is unloaded and are made afresh on reload. `on_route_mount` is not offered yet.

### Legacy plugins

A file without `setup(services)` loads the way every plugin did before PC-1: it imports
Expand All @@ -97,6 +123,7 @@ role. Move it to `setup(services)` — `example_plugin.py` shows the result.
|---|---|---|
| `PLUGINS_ENABLED` | `true` | Set to `false` to disable plugin loading |
| `PLUGINS_DIR` | `plugins` | Directory to scan (relative to repo root) |
| `SCHEDULER_TICK_SECONDS` | `3600` | Interval of `on_scheduler_tick` |
| `PLUGIN_<PLUGIN>_<KEY>` | the manifest's default | A plugin's declared setting |

---
Expand Down
1 change: 1 addition & 0 deletions pyproject.toml
Original file line number Diff line number Diff line change
Expand Up @@ -135,6 +135,7 @@ module = [
"src.services.workspaces",
"src.routes_fastapi._services",
"src.plugins",
"src.plugins.hooks",
"src.plugins.manifest",
"src.plugins.services",
"src.utils.caller",
Expand Down
Loading
Loading