diff --git a/docs/dashboard.md b/docs/dashboard.md new file mode 100644 index 000000000..5eff49252 --- /dev/null +++ b/docs/dashboard.md @@ -0,0 +1,444 @@ +# Offline optimization dashboard + +The dashboard renders a recorded InferRSI run or an explicitly assembled +campaign as a single HTML file. It needs no web service or external assets. +It is a read-only view of recorded data; generating it does not run an +experiment, capture an agent, validate performance, or accept a campaign. + +## Generate and open + +Install InferRSI as described in [quickstart](quickstart.md), then run: + +```sh +inferrsi dashboard --input --output +``` + +Omit `--output` to write `dashboard.html` in the input directory. Open the file +in a browser. Filters and review export work locally. A generated snapshot does +not update as an experiment runs; regenerate it to show new records. + +### Live observation + +To observe a run or campaign while its records are being written, start the +optional read-only server on that machine: + +```sh +inferrsi dashboard --input --live --port 8765 +``` + +Open the printed URL in a browser. The page checks for recorded changes about +every three seconds without reloading and keeps the language, filters, selected +candidate, expanded details and review text where possible. The header shows +the last successful check and warns when refresh fails; the last view remains +visible while it retries. Press Ctrl-C to stop the server. `--translations` +also works in live mode. Use the original command without `--live` whenever an +offline HTML snapshot is needed. + +The server always binds to `127.0.0.1`, with no non-loopback binding option, +and serves only the dashboard and its view data, not the experiment directory. +For a remote machine, use an SSH local port +forward, for example `ssh -N -L 127.0.0.1:8765:127.0.0.1:8765 `. +The server has no authentication: other users on the same host can access its +loopback port. Review recorded context before exposing a forwarded port or +sharing a snapshot. + +Live mode reads the same files as the offline export; it does not start an +experiment, capture an agent's reasoning, or create records. A running result +can be incomplete, and concurrently written files do not form a transactional +snapshot. Benchmark verdicts and metrics appear only once recorded. Context, +search space and narrative must be captured and updated by the caller during +the experiment. + +### Language + +The header switches the offline interface, chart labels and review export between +English and Chinese. The language choice is saved in browser storage when +available. Filter and candidate selections stay in place when switching. + +Recorded narratives are caller-provided text; the dashboard does not send them +to a translation service or infer translations. To include reviewed translations, +pass `--translations ` when exporting. The UTF-8 file maps the exact +recorded text to its translation in each target language: + +```json +{ + "schema_version": 1, + "zh": {"Reduce launch overhead": "减少启动开销"}, + "en": {"减少启动开销": "Reduce launch overhead"} +} +``` + +Both language maps are optional. Each key must match the complete original +field, including whitespace; the file is limited to 2 MiB and duplicate keys +are rejected. The overlay applies only to explicitly marked caller-authored +narrative fields; verdicts, quality, evidence states and other fixed UI labels +use code-owned translations even if the overlay contains the same text. +Untranslated text remains in the original language. Commands, identifiers, +structured event fields and chart labels other than the recorded chart title are kept as recorded. When a knowledge +context has a supplied translation, the page labels it as a translation and +offers the complete original separately. The overlay is supplied and reviewed +by the caller, not verified by InferRSI; review it for accuracy and sensitive +content before exporting or sharing. It is embedded in the resulting HTML. + +The input can be one existing run directory containing `run.lock.json`, or a +campaign containing `campaign.json`. Existing runs need no migration. Missing +events, validation, trace and other evidence are shown explicitly; no missing +metric is replaced with zero. + +## Campaign integration + +The following optional layout associates multiple runs with candidate and round +records. Create the manifest and supply your own campaign ID and optional target: + +```sh +inferrsi campaign init --directory --id \ + --target-pct +``` + +Use `inferrsi run --output-dir /runs` to produce the existing +run directories. After a run ID exists, associate it with a candidate and round: + +```sh +inferrsi campaign record --directory --candidate \ + --run-id --round --title '' +``` + +Optional `--agent-trace` and `--profile` arguments point to existing, +campaign-relative files; the recorder hashes them and does not copy them. The +caller assigns IDs and must serialize campaign record updates. Neither command +starts an agent or validates the experiment. + +### Record the optimization story (campaign trace) + +The **Optimization story** section is the campaign-level trace a reviewer reads +to understand the whole optimization: what knowledge was retrieved and why, the +declared search space and how it was narrowed, each round's hypothesis and +reason, the decision, and the success and failure summaries. It has two parts. + +**1. Capture the knowledge context while working.** Run the same retrieval the +agent uses through `campaign capture-context`. It calls `collect` with a context +budget and keeps the complete envelope: the request, everything search selected, +everything left out and why, and every resource body with its SHA-256. The +dashboard re-derives the assembled text from those resources and withholds a +capture whose bytes or hashes no longer match. + +```sh +inferrsi campaign capture-context --directory --id r1-diagnosis \ + --relevance '' --backend --context-bytes 65536 +``` + +Each call writes a new `knowledge/.json` and prints its SHA-256; an +existing id is never overwritten. The capture records what InferRSI prepared, +not what a model read. Hash checks detect changes relative to the cited bytes +and internal inconsistencies; they do not authenticate who produced a capture. +The campaign root and its `knowledge` directory must remain trusted and stable +while reading or writing. Captured text comes from the selected knowledge +sources; review it before sharing if a store or checkout source was added. + +**2. Record the optimization log.** Write a JSON log (`schema_version: 2`) and +record it with `campaign log`. Every call adds a numbered version +(`optimization-log-vN.json`) and an entry in `optimization-log.jsonl`; earlier +versions remain. The latest version is shown only when its hash matches; +otherwise the dashboard reports the problem instead of silently falling back. + +```sh +inferrsi campaign log --directory --file +``` + +| Field | Content | +| --- | --- | +| `goal` | `statement`, `metric`, optional `baseline`, `target`, `constraints` | +| `knowledge[]` | `context_id` and `sha256` of a capture, `stage`, `purpose` (why it was retrieved), `used_refs`, `takeaways`, optional `rejected_refs[]` with `ref` and `reason` | +| `search_space` | `state`, `missing_reason`, `dimensions[]` with `id`, `name`, optional `rationale`, and `options[]` (`id`, `description`, `status` = `tried`/`pruned`/`untried`, `reason`, optional `source_refs`) | +| `iterations[]` | `round` (increasing), `hypothesis`, `rationale` (why this optimization), `knowledge_refs`, optional `context_ids`, `observations`, `next_step`; `candidates[]` with `candidate_id`, `option_ids` (`.