Repository navigation
✨ knowledge: finish CLI memory drain phase 1 — dedup tags, spool, draft create path - #17123
Conversation
|
[APPROVALNOTIFIER] This PR is NOT APPROVED This pull-request has been approved by: The full list of commands accepted by this bot can be found here. DetailsNeeds approval from an approver in each of these files:Approvers can indicate their approval by writing |
|
@Karibhou this PR is back in your court ( Failing checks:
Reply here or push a new head to this branch (same sign-off); CI and the review swarm re-run automatically and the PR returns to the normal path. Queued in your relay fix queue (2 waiting): fixes are handed out before new work as soon as you have capacity. |
There was a problem hiding this comment.
Read the risk-bearing parts of this against v5 and they look correct to me. What I verified, with locations:
- Spool path safety —
climemory/spool.go:33only buildsSpoolDir/<agent>whenagentNameRe(guard.go:14, no/, no leading.) matches; the entry key is a 16-hex sha256 ofcli\0path(spool.go:55-56), so no untrusted string reaches a path component. Dirs are 0700, files 0600 via tmp+rename, andsource.jsonis written last (spool.go:67-69) so a half-written copy is never retried. - Spool lifecycle — the copy is taken inside
Drain(drain.go:143-151), i.e. before the caller'sclear_on_kickruns; a retried copy is never re-spooled (spool.go:45) and is removed only whendrainFilereports complete (drain.go:137-141). If the live file survives a kick, the second pass is allmemhashdupes, so no double-write. - Mutex actually serialises —
Drainer.muis per-instance, butboot_cli_memory_drain.go:76builds oneDrainerand captures it in theSetCLIMemoryDrainerclosure, so every agent's drain goes through the same lock as the comment ondrain.go:90claims. - Provenance guard is not bypassable from
bd—cmd/bd/create_knowledge.goPOSTs to/api/knowledge/create, andapi_knowledge.go:603callsclimemory.ValidateCreatebeforeCreateFact. The guard requiresdraftand whitelists onlycli-memory,agent:,cli:,memhash:,repo:(guard.go:50-68). - Draft never overwrites —
api.go:380LstatsfactPath(title)beforeWriteFacts, and the draft branch runs before the remote-client lookup (api.go:350-358), so a remote layer getsErrNoDraftChannel→ 400 rather than an approved-by-default entry.
One nit, not a defect: addAgentTag (spool.go:118) returns an error for block-style tags: lists. Both writers in this tree emit inline lists (renderEntry via list(), FileStore.WriteFacts at filestore.go:132), so this only bites a hand-edited entry; the error is surfaced rather than silently skipped, which seems the right call.
CI was still running when I read this; nothing above depends on it. I have not changed anything.
— hive: agent=reviewer-2 backend=copilot model=claude-fable-5.1 copilot=1.0.88
Open PR overlap checkBase branch: Conflicts are checked with every PR merged onto the current tip of
|
|
Hive status — updated in place; edits do not notify. Full history is in the Activity Monitor. Review findings from Read at head What I verified
Low — spool copy can be replaced before its remaining facts are drained ( The spool key is Not repeating the earlier note about the per- stage 095-pre-merge-soak → 060-reviewed: untrusted-author-role Review findings from Style review (read at head
Review findings from Security perspective (read at head 3cf889a; verdict: approve, advisory only) The create-path guard ( Spooled raw memory is a vault-shaped ( |
|
Ready for a human merge: verdict approved @3cf889a, 63/63 checks green, DCO yes, soak 10m0s elapsed. Blocked only by The hive never approves on your behalf: Updated 2026-11-08 05:21 UTC. |
…alse positive Review fixes for #17141: - TestKnowledgeDrainLegacyMemory: put the fixture under a .claude/projects/ directory so repoFromClaudeSlug (which keys off the "projects" path segment, as in production) derives repo:hivecommons/hive. - drain-legacy 409 message: stop spelling out the dotted hive.yaml key agents.cli_memory.drain_to_knowledge, which made TestSettingsDialogReferencesEveryHiveYAMLLeaf believe the Settings UI referenced a still-allowlisted leaf. The docs-only conflict with #17123 is not actionable yet: #17123 is still open and this branch is mergeable against v5; whichever lands second rebases and runs check-docs-citations.py --fix. Hive-Fix-Attempt: 1/6 Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Signed-off-by: scanner <280983584+hivecommons-hive[bot]@users.noreply.github.com>
…legacy drain (#17141) * ✨ knowledge: review flow and never-prime for CLI auto-memory drafts, legacy drain Drained CLI-memory facts are agent-written text, so while one is still a draft it never reaches an agent, whatever include_states admits: - FilterByLifecycleFor (kick primer, knowledge advisor) drops it even for a draft-admitting scope or primer config, or an author match. The operator filter FilterByLifecycle still lists it for review. - TOCScope.HideAutoMemoryDrafts, set by AgentScope and on every ?agent= TOC/entry/effective read, drops it as reason "lifecycle". - AutoPromoteCandidates skips pages tagged cli-memory. Review: PUT /api/knowledge/entry/{id}/state now accepts drained entries (source: cli-memory) in the reserved bead-synth-wiki vault. Before this change they could never be approved. The dashboard knowledge view gets an "Auto-memory drafts" filter that shows agent: tags, with owner-only Approve (approved) and Discard (deprecated, so the memhash blocks a re-draft). hivectl knowledge drain-memory --legacy [--truncate] calls the new owner-only, audited POST /api/knowledge/cli-memory/drain-legacy, which drains /data/home/.claude/projects/*/memory into drafts tagged agent:shared. Files are emptied only with --truncate, and only when every fact in them is stored. Docs: Auto-memory drafts section in knowledge-lifecycle.md, operator reference, hivectl, API reference/OpenAPI, the design doc's "Phase 3 as shipped" section and the design index status. Route citations re-anchored with check-docs-citations.py --fix. Hive-Run: #17058 Hive-Plan: cli-memory-drain.md#phase-3--review-flow-draft-exclusion-legacy-migration-operator-docs Hive-Spec: cli-memory-drain#phase-3 Signed-off-by: Karibhou <Karibhou@users.noreply.github.com> * 🐛 knowledge: fix legacy-drain test fixture path and settings-parity false positive Review fixes for #17141: - TestKnowledgeDrainLegacyMemory: put the fixture under a .claude/projects/ directory so repoFromClaudeSlug (which keys off the "projects" path segment, as in production) derives repo:hivecommons/hive. - drain-legacy 409 message: stop spelling out the dotted hive.yaml key agents.cli_memory.drain_to_knowledge, which made TestSettingsDialogReferencesEveryHiveYAMLLeaf believe the Settings UI referenced a still-allowlisted leaf. The docs-only conflict with #17123 is not actionable yet: #17123 is still open and this branch is mergeable against v5; whichever lands second rebases and runs check-docs-citations.py --fix. Hive-Fix-Attempt: 1/6 Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Signed-off-by: scanner <280983584+hivecommons-hive[bot]@users.noreply.github.com> --------- Signed-off-by: Karibhou <Karibhou@users.noreply.github.com> Signed-off-by: scanner <280983584+hivecommons-hive[bot]@users.noreply.github.com> Co-authored-by: Karibhou <Karibhou@users.noreply.github.com> Co-authored-by: scanner <280983584+hivecommons-hive[bot]@users.noreply.github.com> Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
|
scanner: this head (8b3147a) now conflicts with 🐝 Hive Agent: — hive: agent=scanner backend=copilot model=claude-fable-5.1 copilot=1.0.88 |
…ft create path Completes the phase-1 scope of hivecommons#17056 on top of the drain merged in - climemory: a duplicate fact (any lifecycle state) now gains the draining agent's agent:<name> tag instead of being skipped; its state is never touched, so a deprecated or superseded fact is not revived. - climemory: a memory file whose facts did not all reach the vault (vault unwritable, per-drain cap) is copied to /data/knowledge/auto-memory-spool/<agent>/ before clear_on_kick wipes the store, retried first on the next drain, and removed once complete. Drains are serialised so concurrent agents cannot race on a tag rewrite. - knowledge: CreateFactRequest gains state (approved default, or draft); a draft is written with state: draft into the local channel named by layer and never overwrites an existing entry. - dashboard: POST /api/knowledge/create accepts a cli-memory-tagged fact only as a draft carrying the drain's own tags (400 otherwise); maps missing-channel to 400 and existing-title to 409; audits the state. - bd: create --type knowledge [--state] [--tag]... [--layer] routes to that endpoint instead of the bead store. - Docs, design status, changelog fragment. Hive-Run: hivecommons#17056 Hive-Plan: src/docs/design/cli-memory-drain.md#phase-1 Hive-Spec: cli-memory-drain#phase-1 Signed-off-by: Karibhou <Karibhou@users.noreply.github.com>
The coverage gate failed climemory at 86.9% (floor 90%): ValidateCreate was exercised only from the dashboard package. Table tests now pin its accept and refuse cases in the package itself. Hive-Run: hivecommons#17056 Hive-Plan: cli-memory-drain.md#phase-1 Hive-Spec: cli-memory-drain#phase-1 Signed-off-by: Karibhou <Karibhou@users.noreply.github.com>
…to v5 Signed-off-by: Karibhou <Karibhou@users.noreply.github.com>
8b3147a to
5e38e30
Compare
|
hive-base-moved: rebased onto
🐝 Hive Agent: |
There was a problem hiding this comment.
Correctness — delta re-review at 5e38e30b (previous approve at 8b3147ab; 29 of the 32 compare commits are the v5 rebase, the PR's own delta is the allStored map-type fix in legacy.go + the previously reviewed phase-1 work).
Low / advisory — the drain mutex is per-Drainer, but the vault now has two Drainers. src/pkg/knowledge/climemory/drain.go:95-96 serialises Drain on d.mu, and the PR body says two agents reaching the same fact cannot race on the tag rewrite. That holds for kick drains (one Drainer from src/cmd/hive/boot_cli_memory_drain.go:76), but the upstream legacy handler src/pkg/dashboard/api_knowledge_cli_memory.go:59-60 builds a second Drainer{Dir: vault.RootDir()} on the same directory and calls Drain with Agent: "shared". Its lock is a different sync.Mutex, so a legacy drain concurrent with a kick drain can:
- both read-modify-write the same entry in
addAgentTag(spool.go:112-178,writeAtomic= temp+rename, last writer wins) → oneagent:tag silently dropped; or - both miss the hash in
existingHashesand write the sameEntryFilename(h)→ the loser'sagent:attribution is lost.
No fact is lost (same body/hash either way), only an attribution tag, and the legacy drain is a one-shot owner action — hence advisory, not blocking. Simplest fix is to make the lock package-level so every Drainer on the host shares it:
--- a/src/pkg/knowledge/climemory/drain.go
+++ b/src/pkg/knowledge/climemory/drain.go
@@
type Drainer struct {
Dir string
Store *knowledge.FileStore
Layer knowledge.LayerType
SpoolDir string
Now func() time.Time
-
- // mu serialises drains: two agents reaching the same fact at once must
- // not race on the entry's tag rewrite or its spool directory.
- mu sync.Mutex
}
+
+// drainMu serialises every drain in the process, whichever Drainer runs it
+// (the kick drainer and the dashboard's legacy drain share one vault), so
+// two drains reaching the same fact at once cannot race on the entry's tag
+// rewrite or its spool directory.
+var drainMu sync.Mutex
@@ func (d *Drainer) Drain(req Request) Result {
- d.mu.Lock()
- defer d.mu.Unlock()
+ drainMu.Lock()
+ defer drainMu.Unlock()Previous note (block-list tags: entry yields a per-kick warning, spool.go:143) is still present and unchanged; not re-filed here.
— hive: agent=adjudicator backend=copilot model=claude-fable-5.1 copilot=1.0.88
|
scanner: the red 🐝 Hive Agent: — hive: agent=scanner backend=copilot model=claude-fable-5.1 copilot=1.0.88 |
There was a problem hiding this comment.
Read at head 0bd2714 against v5, focusing on the two risk paths — the spool (the data-loss fix) and the cli-memory create guard (a trust boundary) — rather than all 27 files. Both look correct to me; one residual gap worth a sentence in the docs or a follow-up.
What I verified
- Spool ordering is right.
Drainer.Drain(climemory/drain.go:120-180) runs underd.mu, retriesr.spooled()first and removes an entry only whendrainFilereports every fact stored (:155-166); a live file that comes back incomplete is spooled with the content already in hand (:167-176).drainCLIMemoryForKick(crux/agent/cli_memory_drain.go:81) runs beforeclearCLIMemoryForKick, so the copy exists before the wipe. - Half-written spool entries can't be drained.
spoolwritesmemory.mdthensource.jsonlast via tmp+rename (spool.go:59-68), andspooled()skips any directory without readable, valid metadata (:96-107). Key = sha256(cli, path) so a repeat failure replaces rather than accumulates (:55-56). Dirs 0700, files 0600 (:59,:116). - Dedup tag never revives state.
addAgentTag(spool.go:126-178) edits only the inlinetags:line and returns without writing when the tag is present; it refuses a block-style list rather than guessing.existingHashesonly indexes entries carryingmemhash, so the entries it rewrites are onesrenderEntryproduced with the samelist()formatting. - Create guard.
ValidateCreate(guard.go:28-48) is called before the state default is applied (api_knowledge.go), so acli-memory-tagged request with nostateis rejected rather than silently approved — that's the right direction. Tag allowlist regexes boundagent:,repo:andmemhash:to safe shapes;agentNameRealso gates spool path components (spool.go:34).
Residual gap (not a regression — v5 already loses these): the spool is the only thing between a failed drain and the wipe, and when the spool write itself fails (unwritable /data/knowledge/auto-memory-spool, disk full) spool appends to res.Errors (spool.go:69-72) but drainCLIMemoryForKick still proceeds to the wipe with only a warn (cli_memory_drain.go:96-99). If "a failed drain never loses facts" is the intended contract, the wipe should be skipped for a store whose spool failed; if best-effort is intended, the design doc's new "Phase 1 as shipped" section should say the spool is also best-effort. Either is fine; I just wouldn't leave it implicit.
I did not read cmd/bd/create_knowledge.go or the docs beyond the design section, so this is not a full review of the PR.
— hive: agent=reviewer-2 backend=copilot model=claude-fable-5.1 copilot=1.0.88
There was a problem hiding this comment.
Read the diff against the v5 tree at 3cf889a, plus the callers (boot_cli_memory_drain.go, crux/agent/cli_memory_drain.go, cmd/bd/kb.go). Looks correct to me; nothing I could verify as a defect.
What I checked:
- Dedup never revives state.
drain.go:drainFileonly callsaddAgentTag(existing, agent)on a hash hit;spool.go:addAgentTagrewrites just the inlinetags:line and leavesstate:untouched. A fact written twice in one drain hits the earlyit == tagreturn, so the tag is added once. - Spool is safe against tampered metadata.
spooled()reads content from<spoolDir>/memory.md, andsource.json'ssource_pathis used only for provenance, never opened — so a crafted spool entry cannot make the drain read an arbitrary file. Agent name is gated byagentNameRebefore it becomes a path component; dirs/files are 0700/0600; metadata is written last so a half-written copy is never retried. - Vault failure path.
prepareVaulterror → every live file is spooled (spool(f, nil)reads the file itself); spooled entries are left where they are (f.spoolDir != ""guard). Next drain retries spool before the live store andRemoveAlls only whendrainFilereturns complete. - Draft create path.
api.go:createDraftFactrequires a local channel viavaultByName, refuses an existing slug withErrFactExists(mapped to 409 inapi_knowledge.go), andfilestore.gonow writesstate:which the existing parser atfilestore.go:583reads back.ValidateCreateguard is case-folded oncli-memoryand allow-lists exactly the tagsrenderEntryemits. - Single
Draineris created ininstallCLIMemoryDrainer, sod.mudoes serialise all agents' drains as the body claims. - Intent. #17056 named
auto-memory/src/crux/agent/memdrain; this follows thecli-memory/pkg/knowledge/climemoryarchitecture #17086 already merged and records the delta in the design doc's "Phase 1 as shipped" — consistent with the merged base, not a new divergence.
Two non-blocking observations, not defects: createTypeArg scans all flag values too, so a --body whose text is literally -type=knowledge would route to the knowledge path (unrealistic); and the Lstat-then-WriteFacts check in createDraftFact is a TOCTOU window under the serialised drainer only, not under concurrent API callers — worst case a draft overwrites a draft of the same title.
— hive: agent=reviewer backend=copilot model=claude-fable-5.1 copilot=1.0.88
Review follow-ups (below
|
|
Merged by hive contributor-lane auto-merge: CI green for ≥10m, reviewer verdict approve, @Karibhou holds the trusted tier in this hive's contributor registry. |
Closes #17056
#17086 shipped the manager-side drain (Claude, Codex, Gemini →
state: draftentries inbead-synth-wiki, deduplicated bymemhash). This PR adds the phase-1 items #17086 left out, which the triage comment on #17056 listed.What
agent:<name>in its inlinetags:list; state untouched, so a deprecated/superseded fact is not revivedclear_on_kickwiped the store even when the vault write failed or the 200-fact cap left facts behind, so those facts were lost/data/knowledge/auto-memory-spool/<agent>/<key>/(0700/0600, key = sha256(cli, path), so a second failure replaces the copy instead of adding one). The next drain retries the spool before the live store and deletes each copy once all its facts are stored. Log line:… (dupes K, spooled S)CreateFactRequest.stateapproved(default, same remote-ingest path as before) ordraft. A draft is written withstate: draftinto the local channel named bylayer. Remote wiki layers store only a status, and an unknown status loads as approved, so a draft there gets 400. A draft never overwrites an existing entry (409).deprecated/superseded→ 400cli-memory-tagged request (case-insensitive) must bedraftand carry onlycli-memory,agent:<name>,cli:<claude|codex|gemini>,memhash:<hex>,repo:<org>/<repo>→ otherwise 400bd create --type knowledge--title,--body/--body-file(-= stdin, 256 KiB cap),--state, repeatable--tag,--layer,--fact-type→POST /api/knowledge/create; never touches the bead storeDrains are now serialised on the
Drainer, so two agents reaching the same fact at the same time cannot race on the tag rewrite.Deliberate differences from the design doc
Recorded in a new Phase 1 as shipped subsection of
src/docs/design/cli-memory-drain.md. #17086 already set the architecture, and this PR builds on it rather than adding a second one:src/crux/agent/memdrainHTTP library. The drain ispkg/knowledge/climemory, installed throughcrux.CLIMemoryDrainHost. Crux still imports nothing fromsrc/pkg.cli-memoryand keydrain_to_knowledge(as shipped), notauto-memory/drain. No new config key, so no new settings-parity entry.clear_on_kick, rather than truncating the store itself. That keeps the existingclear_on_kick: false+ drain behaviour unchanged and needs no second su-exec remover.Tests (CI runs them; per
AGENTS.mdnot run locally,gofmt -lclean)climemory/spool_test.go: dedup adds the tag once across two passes and leaves a deprecated fact deprecated;addAgentTagcases (append, already tagged, no tags line, no frontmatter, block list → error, unsafe agent); a vault failure spools, a repeat failure keeps one copy, the retry keepsrepo:/source_pathand empties the spool; cap-partial file spooled and completed; no spool for an unsafe or empty agent name.dashboard/api_knowledge_create_state_test.go: state validation, guard rejections (incl. case-folded tag, foreign tag, path-like agent), draft without a channel → 400, draft lands asstate: draftand failsFilterByLifecycle, duplicate title → 409, default create unchanged.cmd/bd/create_knowledge_test.go:--typedetection, flag parsing and errors, payload posted to the dashboard with the bead store left untouched.boot_cli_memory_drain_test.gonow points the spool at a temp dir, so it never reads a live/data.Symlinked or shared stores are still skipped by the #17086 code paths (
memoryFiles,cliMemoryStorePaths) and their existing tests. The drain still never blocks a kick: errors are only logged indrainCLIMemoryForKick.— hive: backend=omp model=anthropic/claude-opus-5-5
🐝 Hive Agent:
contributor| SHA:3a6815095