Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
215 changes: 215 additions & 0 deletions src/docs/agent-knowledge-and-memory.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,215 @@
# How Hive injects knowledge and memory into agents

> Written to answer the 2026-11-05 community-meeting question "how does
> Hive's memory/knowledge injection work today?". It describes `v5` as of
> this page's merge; the [Known gaps](#known-gaps--in-progress) section links
> the work that will change the memory half.

Hive agents receive context from **three distinct layers**, and only one of
them is Hive's own:

| Layer | Who owns it | Where it lives | Hive's role |
|---|---|---|---|
| **Knowledge** β€” curated facts about the repos and the fleet | Hive (`pkg/knowledge`) | File vaults under `/data/knowledge/…`, optional HTTP wiki layers | Selects a few facts per kick and **pushes** them into the kick prompt; also serves them for **pull** via API/MCP/`hivectl` |
| **Conversation** β€” the current chat in the CLI pane | The CLI | In the CLI process | Resets it before every kick (`/clear`) |
| **CLI memory** β€” the CLI's own persistent, cross-session notes (Claude Code auto-memory, Codex memories, Gemini `GEMINI.md`, …) | The CLI | Dot-directories under the agent's `HOME` | **None.** Hive neither feeds, reads, filters nor clears it; it only decides which agents share the directory |

The confusion at the meeting was between layers 2 and 3: "every agent clears
its context before it runs" is true of the conversation and false of CLI
memory.

## 1. Knowledge: the kick primer (push)

```
knowledge.vaults / git_sources / connectors / bead_synthesizer vault
(FileStore per source; HTTP wiki layers for personal/project/org/community)
β”‚
registerKnowledgeStores (cmd/hive/boot_knowledge_primer.go)
β–Ό
β”Œβ”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€ Scheduler.primeKnowledge (pkg/scheduler/kick_priming.go) ───┐
β”‚ 1. extractKeywords ← titles + labels of the agent's first β”‚
β”‚ maxIssuesToPrime (=5) queued issues β”‚
β”‚ 2. scope ← knowledge.agent_scopes[<agent>] (or base agent for a replica) β”‚
β”‚ 3. Primer.PrimeScoped (pkg/knowledge/primer.go) β”‚
β”‚ search every layer/store β†’ merge by layer precedence β†’ β”‚
β”‚ FilterByLifecycle (approved only) β†’ scope.Filter β†’ β”‚
β”‚ Fisher-Rao rerank β†’ type priority β†’ cap at max_facts (25) β†’ β”‚
β”‚ expandWithGraph (never past the scope) β†’ Context7 hints β”‚
β”‚ 4. PrimedKnowledge.Render (primer_render.go): "# Relevant Knowledge", β”‚
β”‚ grouped by type, whole facts dropped past max_bytes (4096), β”‚
β”‚ "_… truncated: N facts / M bytes omitted (size cap)._" marker β”‚
β””β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”˜
β”‚
templates.go: skills section + AGENTS.md section + knowledge section
substituted into the agent's kick template; lane kicks (lane_messages.go)
append the knowledge section for scanner / quality / architect / <agent>
β–Ό
tmux: [/clear ⏎] then the kick text ⏎ (crux/agent/manager_kick.go)
```

**Where the facts come from.** Everything the primer searches is registered
in `registerKnowledgeStores` (`src/cmd/hive/boot_knowledge_primer.go`):
configured `knowledge.vaults` (personal layer), `knowledge.git_sources`
(layer from config), connector output, the **bead synthesizer vault**
(closed beads from every agent, classified into fact types β€” the main
organic source on a running hive), plus any HTTP wiki layers in
`knowledge.layers`. The repo's own docs are *not* a source unless you index
them through a git source, connector or code map
([knowledge-curator.md](knowledge-curator.md)).

**How it is scoped.**

- *By the work, not the repo:* the query is keywords from the titles and
labels of the agent's first five queued issues (`extractKeywords`,
`isNoiseLabel`). There is no automatic per-repo or per-issue filter; a fact
tagged with another repo can surface if the keywords match.
- *By agent:* `knowledge.agent_scopes.<agent>` is an allow-list of `layers`,
`repos`, `types`, `tags` (`KnowledgeAgentScope` in
`src/pkg/config/knowledge_config.go`). It is applied inside
`Primer.PrimeScoped` *after* the lifecycle filter and can only narrow; graph
expansion never pulls in an out-of-scope fact. A replica inherits its base
agent's entry (`Config.KnowledgeAgentScope`). No entry means unrestricted.
- *By lifecycle:* kicks carry **approved facts only**. The kick primer is built
with no `IncludeStates` (`newKnowledgePrimer` in `src/cmd/hive/main.go`), so
`draft`, `deprecated` and `superseded` never reach a kick; `agent_scopes.
include_states` only widens API/TOC reads that already asked for those
states ([knowledge-lifecycle.md](knowledge-lifecycle.md)).

**Size caps** (all drop whole units, never clip, and leave a visible marker):

| Block | Cap | Config |
|---|---|---|
| `# Relevant Knowledge` | 25 facts, 4096 bytes (floor 512) | `knowledge.primer.max_facts`, `knowledge.primer.max_bytes` |
| Repo `AGENTS.md` | 8192 bytes (floor 2048), whole sections | `project.agents_md_max_bytes` (`agentsmd.DefaultMaxInjectionBytes`) |
| Declared skills | 8192 bytes, whole skills | `maxSkillsInjectionBytes` in `kick_priming.go` (fixed) |

**Switches.** `knowledge.enabled` gates the primer only β€” the bead synthesizer
still writes facts when it is off, they just reach no kick. The dashboard
knowledge toggle registers/unregisters the primer live
(`knowledgePrimerControl`). `knowledge.enabled: false` or an empty keyword set
means the kick simply has no knowledge section.

## 2. Knowledge: pull access

Agents (and external tools) can fetch what the primer did not push:
`GET /api/knowledge/toc` and `/api/knowledge/entry/{id}` (pass `agent=` to
apply that agent's scope), `GET /api/knowledge/effective?agent=` to see what
a given agent is allowed to see and why entries are excluded, the public
`POST /mcp/knowledge` MCP (`knowledge_toc`, `knowledge_get`), `hivectl
knowledge search|list`, and `bd kb search` inside the pod. See
[knowledge-toc.md](knowledge-toc.md) and
[public-knowledge-mcp.md](public-knowledge-mcp.md).

## 3. AGENTS.md / CLAUDE.md: injected by Hive, not read natively

Each agent's CLI starts in `/data/agents/<name>` (`crux.DefaultWorkDir`),
not inside a repository checkout, so the CLIs' native instruction-file
discovery (`AGENTS.md` for Codex/Copilot, `CLAUDE.md` for Claude Code,
`GEMINI.md`) finds **nothing from the monitored repo** at launch. Repo
instructions reach agents only through the scheduler: `primeAgentsMd` parses
`<project.checkouts_dir>/<repo>/AGENTS.md` for the agent's *primary* repo and
prepends the rendered block to the knowledge section (`templates.go`). Without
`project.checkouts_dir` (or a matching `policies.local_dir`) the block is
empty β€” see [agents-md.md](agents-md.md). Skills declared on the agent are
prepended the same way ([skills.md](skills.md)); the hive-wide house style
precedes both ([house-style.md](house-style.md)).

Two instruction files *are* read natively: a user-level
`~/.claude/CLAUDE.md` / `~/.gemini/GEMINI.md` if one exists β€” and because
those directories are fleet-shared today (next section), such a file applies
to every Claude or Gemini agent on the hive.

## 4. CLI memory and conversation: what Hive does and does not touch

**Conversation.** `clear_on_kick` defaults to `true` (`ClearOnKick`,
`src/pkg/config/config.go`). `deliverKickLocked` in
`src/crux/agent/manager_kick.go` types `/clear` ⏎ before the kick text for
every TUI backend (agy headless starts a new conversation instead). The one
exception is a PR follow-up delivered with `SendResumeKick`
(`pr_session_resume.go`), which deliberately continues the authoring
conversation. `/clear` empties the chat; it does not touch anything on disk.

**Home layout** (`interactive_home.go`, `manager_env.go`, `manager_homes.go`):

- Per-UID agents get `HOME=/data/home/agents/<name>` (`AgentHome`,
`interactiveHomePath`) with per-agent `XDG_DATA_HOME`/`XDG_STATE_HOME` under
`~/.local` (`setupAgentXDGDirs`). `/data/home` is the persistent volume.
- `interactiveHomeBridgeDirs` symlinks `.claude`, `.config`, `.codex`, `.bob`,
`.gemini` and `.cache` from every per-agent home to the **same** directories
under `/data/home`, so one login authenticates the fleet. Anything a CLI
keeps *inside* those directories is therefore fleet-shared and persistent.
- Real per-agent directories with one narrow bridge: `~/.copilot` (only
`config.json` shared β€” `setupCopilotHome`, #9444), `~/.local`,
`.claude.json`, `.npm`, `.gitconfig` (#9478). Codex gets
`CODEX_HOME=/data/home/.codex-<name>` with only `auth.json` symlinked to the
shared login (`setupCodexHome`, `codexHomePath`).
- `HIVE_SHARED_AGENT_HOME=1` forces the legacy single shared home for all.
- Inference (API-routed) agents use throwaway homes under
`/tmp/.claude-inference-home-<name>` that vanish with the pod.
- Contributor containers stage only `.credentials.json` and `settings.json`
into the container (`CLAUDE_STAGE_ALLOWLIST` in the `justfile`, #7836);
transcripts, history and memory stay on the host. This is the one place
memory is isolated by construction.

**No memory handling.** Nothing in Hive sets a CLI's "disable auto-memory"
option (`inferenceSettingsSeed` seeds permissions/onboarding/remote-control
keys only), prunes a `memory/` directory, or clears memory on kick, `/clear`
or restart. What each CLI remembers, it remembers across kicks and pod
restarts, and β€” for the bridged directories β€” across agents.

### What persists across what

Columns: a cadence **kick**; an explicit **`/clear`**; a **pod restart**; a
**new agent** added to the same hive. βœ“ = survives / is visible, βœ— = gone /
not visible.

| | Kick | `/clear` | Pod restart | New agent in same hive |
|---|---|---|---|---|
| **Knowledge** (`# Relevant Knowledge`, AGENTS.md, skills) | re-selected for *this* kick's issues | n/a β€” re-injected next kick | βœ“ vaults on `/data` | βœ“ same stores, filtered by its own `agent_scopes` |
| **Conversation** β€” all CLIs | βœ— (`/clear` first, unless `clear_on_kick: false` or a resume kick) | βœ— | βœ— (CLI relaunched) | βœ— |
| **Claude Code memory** (`~/.claude/projects/<cwd-slug>/memory/`, `~/.claude/CLAUDE.md`, `history.jsonl`) | βœ“ | βœ“ | βœ“ (`/data/home/.claude` is shared & persistent) | βœ“ readable: the tree is fleet-shared; auto-memory is keyed by cwd `/data/agents/<name>` so files are per-agent by path, but `CLAUDE.md`/history are one copy |
| **Codex memory** (`$CODEX_HOME`) | βœ“ | βœ“ | βœ“ (`/data/home/.codex-<name>`) | βœ— per-agent dir; only `auth.json` shared |
| **Copilot CLI memory/sessions** (`~/.copilot`) | βœ“ | βœ“ | βœ“ | βœ— per-agent dir (#9444); only `config.json` shared |
| **Gemini memory** (`~/.gemini/GEMINI.md`, state) | βœ“ | βœ“ | βœ“ | βœ“ whole directory shared |
| **Bob memory/settings** (`~/.bob`) | βœ“ | βœ“ | βœ“ | βœ“ whole directory shared |

The dashboard's cost accounting reads the shared Claude tree
(`data.claude_sessions_dir`, default `/data/home/.claude/projects`, and
attributes sessions by the `-data-agents-<name>` project slug β€”
`src/pkg/tokens/claude_scanner.go`), which is why that tree is shared on
purpose today.

## Known gaps / in progress

Do not treat the following as current behaviour; follow the links.

- [#17031](https://github.com/hivecommons/hive/issues/17031) β€” Claude/Gemini/Bob
persistent memory is shared across every agent on a hive. Fix in flight:
[PR #17039](https://github.com/hivecommons/hive/pull/17039) makes `.claude`,
`.gemini` and `.bob` real per-agent directories with only credential/settings
entries bridged (the `setupCopilotHome` pattern) and teaches the token
collector to scan `/data/home/agents/*/.claude/projects`. Until it merges,
the table above stands.
- [#17032](https://github.com/hivecommons/hive/issues/17032) β€” disable CLI
auto-memory by default and add a `clear_memory_on_kick` option alongside
`clear_on_kick`.
- [#17033](https://github.com/hivecommons/hive/issues/17033) β€” drain CLI
auto-memory into beads/knowledge at session end instead of a hidden
`MEMORY.md` (revives open question 5 of
[design/knowledge-system.md](design/knowledge-system.md)).
- Repo `AGENTS.md` injection still requires `project.checkouts_dir`; nearest-
file (`ParseNearest`) targeting is deferred ([agents-md.md](agents-md.md)).

Closed history worth knowing: #7836 (contributor `~/.claude` staging), #9444
(Copilot session-state isolation), #9478 (per-agent `.gitconfig`), #11201
(`knowledge.agent_scopes`), #12124/#12129 (primer and AGENTS.md size caps),
#10524 (knowledge TOC, lifecycle, suggest).

## Related

[design/knowledge-system.md](design/knowledge-system.md) Β·
[knowledge-curator.md](knowledge-curator.md) Β·
[knowledge-lifecycle.md](knowledge-lifecycle.md) Β·
[knowledge-toc.md](knowledge-toc.md) Β· [agents-md.md](agents-md.md) Β·
[skills.md](skills.md) Β· [agent-configuration.md](agent-configuration.md) Β·
[contributor-relay.md](contributor-relay.md)
4 changes: 4 additions & 0 deletions src/docs/agents-md.md
Original file line number Diff line number Diff line change
Expand Up @@ -236,6 +236,10 @@ every lane honours that statement as a written policy and skips the class.

## What to read next

- [agent-knowledge-and-memory.md](agent-knowledge-and-memory.md) β€” where this
block sits in the whole kick (skills, AGENTS.md, `# Relevant Knowledge`), why
the CLI does not read the repo's `AGENTS.md` natively, and what `/clear`
does and does not reset.
- [house-style.md](house-style.md) β€” the short, hive-wide *how to change things*
section that precedes this block in every kick; a repo overrides it with a
root `HIVE_STYLE.md`, read from the same checkout root.
Expand Down
1 change: 1 addition & 0 deletions src/docs/documentation-map.md
Original file line number Diff line number Diff line change
Expand Up @@ -167,6 +167,7 @@ Documentation for the current Hive line (branch `v5`; the code and docs live und
- [Custom dashboard stylesheets](custom-stylesheets.md) β€” operator-supplied CSS for the dashboard and public snapshot.
- [Branding a hive](branding.md) β€” persistent per-deployment name, mark, and colours via `<data>/branding/branding.json` and `custom.css` (`HIVE_BRANDING_JSON`/`HIVE_BRANDING_CSS`); distinct from the per-request `?style=` stylesheets above.
- [Portable AgentDefinition format](../AGENT-DEFINITION.md) β€” standalone YAML schema for importing/exporting agent definitions.
- [How Hive injects knowledge and memory into agents](agent-knowledge-and-memory.md) β€” the three context layers (Hive knowledge primer, CLI conversation, CLI persistent memory), the kick-primer path and its scoping/caps, why repo `AGENTS.md` is injected rather than read natively, which per-agent home directories are fleet-shared, and what survives a kick, `/clear`, pod restart or a new agent.
- [Knowledge connectors](knowledge-connectors.md) β€” **Owner configuration:** connect, validate, and sync external document/wiki sources; includes scope and API reference.
- [Knowledge curator](knowledge-curator.md) β€” configure fact extraction, sources, vaults, document imports, code maps, and retention.
- [Knowledge confidence scoring](knowledge-confidence.md) β€” interpret confidence badges and score explanations.
Expand Down
3 changes: 2 additions & 1 deletion src/docs/knowledge-curator.md
Original file line number Diff line number Diff line change
Expand Up @@ -627,7 +627,8 @@ Maps an agent name to the `layers`, `repos`, `types`, `tags` and
are restricted to. A missing agent or an empty field is unrestricted (the
default). Unknown layer or lifecycle state names fail config load. See
[Per-agent scopes](knowledge-toc.md#per-agent-scopes) for the intersection
rules.
rules, and [agent-knowledge-and-memory.md](agent-knowledge-and-memory.md) for
how the scope is applied on the kick path end to end.

```yaml
knowledge:
Expand Down
2 changes: 1 addition & 1 deletion src/docs/operator-reference/README.md
Original file line number Diff line number Diff line change
Expand Up @@ -301,7 +301,7 @@ Go callers keep reading `cfg.Agents`, `cfg.Governor.Gateways` and so on through
| `knowledge.bead_synthesizer.publish` | Off (`enabled: false`). | Publishes synthesized facts into a repository through one PR on `hive/bead-synth-knowledge`: `enabled`, `repo` (empty = `project.primary_repo`), `path` (default `docs/knowledge/`; `.hive/knowledge/` is the alternative; relative, no `..`), `base_branch` (empty = repo default branch), `mode` (`pr` default or `commit`), `cadence` (`per_run` default or `daily`). ACMM L<=4 hives publish via PR only; `commit` is warned about and still opens a PR. See [knowledge-curator.md](../knowledge-curator.md#repository-publish-knowledgebead_synthesizerpublish). |
| `variables.security.*` | Deny by default. | `allow_exec`, `allow_http`, and GitHub prompt-source allowlists are honored only from the trusted seed, not dashboard overlays. |
| `agents.<name>.prompt_source` (size cap) | 96 KiB (`promptsrc.DefaultMaxPromptBytes`); `HIVE_PROMPT_SOURCE_MAX_BYTES` overrides it. | A fetched kick template over the cap is **rejected whole, not truncated** ([#12130](https://github.com/hivecommons/hive/issues/12130)): the kick uses the last successfully fetched copy if there is one, otherwise the ordinary `kick_template` β†’ pack β†’ convention β†’ embedded-default chain, and the scheduler logs `prompt from GitHub exceeds size cap; REJECTED whole (not truncated), falling back` with `repo`, `path`, `size_bytes` and `cap_bytes`. Saving an agent whose source is over the cap fails in the dashboard with the same sizes rather than baking a cut-off policy. A half-policy is a policy with its closing rules missing; no remote policy is the safer failure. Every shipped template fits with >2Γ— headroom (`TestShippedTemplatesFitUnderCap`). |
| `data.claude_sessions_dir` | `/data/home/.claude/projects` | Where the dashboard reads Claude Code session JSONL for per-agent token/cost accounting. Point it at the agents' real session directory if you relocate `HOME`. |
| `data.claude_sessions_dir` | `/data/home/.claude/projects` | Where the dashboard reads Claude Code session JSONL for per-agent token/cost accounting. Point it at the agents' real session directory if you relocate `HOME`. This shared tree is also where Claude Code keeps its persistent auto-memory; see [agent-knowledge-and-memory.md](../agent-knowledge-and-memory.md) for which per-agent home directories are fleet-shared and what `clear_on_kick` does and does not clear. |
| `data.copilot_sessions_dir` | `/data/home/.copilot/session-state` | Same, for the Copilot CLI backend's session state. |

### Label-driven approval
Expand Down
Loading