Skip to content

✨ knowledge: seed hosted spokes with the primary repo's knowledge dir and an 8 KB primer default - #17146

Merged
clubanderson merged 2 commits into
v5from
feat/knowledge-seed-hosted
Nov 8, 2026
Merged

clubanderson merged 2 commits into
v5from
feat/knowledge-seed-hosted

Conversation

@clubanderson

Copy link
Copy Markdown
Member

Fixes #17145

Hosted spokes had knowledge.git_sources empty and a 4 KiB primer cap, so the repo's curated knowledge entries never reached a kick and 15 of 25 primed facts were dropped (verified on hive's own spoke).

Seam: spoke-side boot. bootKnowledge runs after bootGitHub, so seedKnowledgeGitSource (src/cmd/hive/boot_knowledge_seed.go) runs just before the existing git-source auto-connect loop, using the hive's own *github.Client for the probe (DefaultBranch + PathExistsAtRef, 404 → skip).

  • When knowledge.git_sources is absent, seed <repo>-knowledge (layer project, branch = repo default branch) at the first existing path among bead_synthesizer.publish.path, docs/knowledge/, src/docs/knowledge/. In-memory only; never writes hive.yaml.
  • Operator lists are never touched; an explicit git_sources: [] (what a dashboard disconnect saves) opts out. Non-GitHub forge, no client, lookup/probe error, or no dir → one DEBUG line. Seeding → one INFO line.
  • knowledge.primer.max_bytes default 4096 → 8192 (floor 512 unchanged); docs and parity test updated.
  • Table-driven tests with a fake probe cover the decision matrix and idempotency.

… and an 8 KB primer default

A hive with no knowledge.git_sources now gets one entry at boot pointing
at its primary repo's knowledge directory (bead_synthesizer.publish.path,
else docs/knowledge/ then src/docs/knowledge/, probed via the GitHub
contents API on the default branch), named <repo>-knowledge at the
project layer. In-memory, idempotent, operator lists untouched.

knowledge.primer.max_bytes default rises from 4096 to 8192.

Fixes #17145

Signed-off-by: Andrew Anderson <andy@clubanderson.com>
@kubestellar-prow kubestellar-prow Bot added the dco-signoff: yes Indicates the PR's author has signed the DCO. label Nov 8, 2026
@kubestellar-prow

Copy link
Copy Markdown
Contributor

[APPROVALNOTIFIER] This PR is NOT APPROVED

This pull-request has been approved by:
Once this PR has been reviewed and has the lgtm label, please assign danathar for approval. For more information see the Code Review Process.

The full list of commands accepted by this bot can be found here.

Details Needs approval from an approver in each of these files:

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

@kubestellar-prow kubestellar-prow Bot added the size/L Denotes a PR that changes 100-499 lines, ignoring generated files. label Nov 8, 2026
@github-actions

github-actions Bot commented Nov 8, 2026 •

Copy link
Copy Markdown
Contributor

Open PR overlap check

Base branch: v5

Conflicts are checked with every PR merged onto the current tip of v5, so a PR that only needs a rebase is not reported as conflicting with its siblings.

PR Title Relation Files
#17123 ✨ knowledge: finish CLI memory drain phase 1 — dedup tags, spool, draft create path same files src/docs/agent-knowledge-and-memory.md
#17141 ✨ knowledge: review flow and never-prime for CLI auto-memory drafts, legacy drain same files src/docs/agent-knowledge-and-memory.md

@hivecommons-hive

hivecommons-hive Bot commented Nov 8, 2026 •

Copy link
Copy Markdown
Contributor

Hive status — updated in place; edits do not notify. Full history is in the Activity Monitor.

@clubanderson this PR is held at stage/071-needs-author and the hive will not merge it yet.

Failing checks:

  • test (dashboard 2/4)

Next: the hive does not push to your branch. Push a new head (or reply here); CI and the review swarm re-run automatically and the PR returns to the normal path.

Review findings from adjudicator (folded here because this head already carries its hive review):

Correctness review at 747624a (read the diff plus main.go bootKnowledgeWith, pkg/github/client.go/defaultbranch.go, dashboard/api_knowledge.go, dashboard/api.go).

1. The updated parity test still asserts the old default — CI test (dashboard …) is red at this head.
src/pkg/dashboard/knowledge_settings_parity_test.go:29 was bumped to 8192, but line 194 still requires resp.Defaults["primer_max_bytes"] to equal float64(4096), and the GET now reports "primer_max_bytes":8192 (knowledge_config.go:131). TestKnowledgePipelineParityBatch8RoundTrip fails in both dashboard shards.

		resp.Primer["merge_strategy"] != "precedence" || resp.Defaults["primer_max_bytes"] != float64(config.DefaultKnowledgePrimerMaxBytes) {

2. Seeding is not gated on knowledge.enabled, so it switches knowledge on for hives that never opted in.
seedKnowledgeGitSource (boot_knowledge_seed.go:33-50) checks GitSources != nil, repo, forge and client, but never cfg.Knowledge.Enabled, and the call in main.go:3003-3006 is unconditional. The loop right after it (main.go:3009-3018) auto-enables a fallback file-engine KnowledgeAPI whenever GitSources is non-empty — a path that previously only fired when an operator had explicitly listed a git source. After this PR, every GitHub-forge hive with knowledge.enabled: false/absent whose primary repo has docs/knowledge/ or src/docs/knowledge/ will auto-enable knowledge, clone the repo at boot and log "auto-enabled knowledge API for git sources". #17145 asks for a default for spokes that use knowledge, not for turning it on. Gate the seed and add a table row for it:

--- a/src/cmd/hive/boot_knowledge_seed.go
+++ b/src/cmd/hive/boot_knowledge_seed.go
@@ func seedKnowledgeGitSource(
 	if cfg == nil {
 		return false
 	}
+	if !cfg.Knowledge.Enabled {
+		logger.Debug("knowledge: git source seed skipped", "reason", "knowledge is disabled")
+		return false
+	}
 	if cfg.Knowledge.GitSources != nil {

(and set c.Knowledge.Enabled = true in seedCfg in the test, plus a "knowledge disabled skips" case).

3. "In-memory only; never writes hive.yaml" holds only until the next dashboard save.
The seed appends to cfg.Knowledge.GitSources (boot_knowledge_seed.go:94), and every dashboard saveConfig() (dashboard/api.go:747-754, ~30 call sites) marshals the whole Config, so the seeded entry is persisted the first time an operator saves any setting. From then on it is an operator-set list: the probe no longer runs, and a later default-branch rename or dir move surfaces as a boot WARN from main.go:3027 instead of a re-probe. Low severity, but the PR body and agent-knowledge-and-memory.md:65-66 should say the entry can become persistent, or the seed should be kept out of cfg (e.g. connect it directly in bootKnowledgeWith without appending).

Review findings from reviewer (folded here because this head already carries its hive review):

Conclusion: the seed's "absent key" gate does not survive the first config save, so the feature is likely to fire once at most — or never on a spoke whose hive.yaml has already been written by hive. One correctness finding, one docs drift; I read the callers in the tree rather than only the diff.

1. git_sources: nil is not a stable "operator did not set this" signal (high)

seedKnowledgeGitSource (src/cmd/hive/boot_knowledge_seed.go:36-39) treats cfg.Knowledge.GitSources != nil as "operator-set" and nil as "seed me", and the body/docs say the seed is "in-memory only; never writes hive.yaml".

But the in-memory cfg is what gets written to hive.yaml:

  • Config.Save() → saveLocked() does yaml.Marshal(c.redactedForPersist()) of the whole struct (src/pkg/config/config.go:7345, :7396).
  • persistState calls cfg.ReconcilePausedAndSave(livePaused) (src/cmd/hive/main.go:6702), and the dashboard's PersistFunc (main.go:1274-1276) runs it on every pause/resume; dashboard settings saves also go through s.saveConfig() → Config.Save() (src/pkg/dashboard/api.go:747-754).
  • GitSources is tagged yaml:"git_sources" with no omitempty (src/pkg/config/knowledge_config.go:33). I verified with yaml.v3 (the version in go.mod): a nil slice marshals as git_sources: [], and git_sources: [] unmarshals to a non-nil empty slice.

Consequences:

  • Seed fired at boot: the first Save() persists <repo>-knowledge into hive.yaml with the branch and subpath pinned. From then on it is an "operator list" and the probe never runs again. Not harmful, but the "never writes hive.yaml" claim in the body, the code comment (boot_knowledge_seed.go:27-32) and src/docs/agent-knowledge-and-memory.md is not true.
  • Seed skipped at boot (App auth not yet available so b.ghClient == nil, a transient DefaultBranch/PathExistsAtRef error, non-GitHub forge): the first Save() writes git_sources: [], which this PR itself defines as the opt-out. One transient GitHub failure on first boot permanently disables seeding for that spoke.
  • Spokes that have already been running: any spoke whose hive.yaml hive has saved at least once already has git_sources: [] in it, and the seed will not fire there at all. The issue text says the motivating spoke's git_sources "is empty" — if that file was written by hive (rather than hand-authored without the key), this PR is a no-op on exactly the host it was verified against. Worth checking the actual file before merging.

Suggested fix: don't use the slice's nil-ness as the signal. Either (a) keep the seeded entry out of cfg.Knowledge.GitSources entirely — hold it on boot and feed it to the auto-connect loop (main.go:3000) and boot_knowledge_primer.go:37 alongside the configured list — or (b) record "key present in YAML" at load time (a yaml.Node check or a *[]GitSourceConfigYAML) and gate on that. Either way Save() stops converting a seed or a skipped probe into operator config. The table-driven test should then include a "config was saved and reloaded" case, since that is the path that breaks today.

2. Docs still say 4096 (low)

src/docs/knowledge-curator.md:514 ("default 4096, minimum 512") and :535 (max_bytes: 4096 # 0 = default) were not updated alongside agent-knowledge-and-memory.md; the config comment at src/pkg/config/config.go:1061 is about style_max_bytes and is unrelated, fine to leave.

What I checked and found fine

  • *github.Client has both probe methods (pkg/github/defaultbranch.go:40, pkg/github/client.go:4415); the interface slice is correct.
  • bootGitHub precedes bootKnowledge, so b.ghClient is normally populated — though per finding 1, "normally" is the problem.
  • knowledgeSeedPaths dedup/trim and the dashboard connect/disconnect handlers (api_knowledge.go:1076-1134) are consistent with the seeded entry's URL/subpath shape.
  • The 4096 → 8192 default change is applied consistently in knowledge_config.go, primer_render.go, and the dashboard parity test.

I have not approved or requested changes as a review event; this is a comment for clubanderson to weigh.

Review findings from reviewer-2 (folded here because this head already carries its hive review):

Read the seed against the tree at 747624a (boot, Config.Save, the dashboard disconnect path) before the body. The seeding logic itself is sound — probe signatures match *github.Client (pkg/github/defaultbranch.go:40, client.go:4415), the disconnect handler really does leave a non-nil empty slice (pkg/dashboard/api_knowledge.go:1127-1134), and the fake-probe matrix covers the decision table. Two things need attention, one of which is why CI is red.

1. Stale test assertion — the dashboard CI failure (fix needed)

src/pkg/dashboard/knowledge_settings_parity_test.go:193 still asserts resp.Defaults["primer_max_bytes"] != float64(4096), while api_config_knowledge_pipeline.go:324 now reports config.DefaultKnowledgePrimerMaxBytes = 8192. The PR updated line 28 of the same file but not line 193; that is test (dashboard 2/4) and dashboard shuffle 3/3 failing. Change the literal to 8192 (or compare against the constant).

2. The absent-vs-[] guard is narrower than the body and docs say (medium)

The seed's opt-out keys on cfg.Knowledge.GitSources != nil (boot_knowledge_seed.go:36). But Config.Save (pkg/config/config.go:7383) does yaml.Marshal(c.redactedForPersist()) of the whole struct, GitSources has no omitempty (knowledge_config.go:33), and yaml.v3 encodes a nil slice as git_sources: [] — I checked with a 5-line program against gopkg.in/yaml.v3 v3.0.1: Marshal of a nil slice gives git_sources: [], and decoding [] yields a non-nil slice. Two consequences:

  • Any hive whose hive.yaml has ever been saved by the dashboard (any settings PUT, or SetAgentPausedAndSave at config.go:7351) already carries git_sources: [], so on those hives the seed never fires. The feature reaches only hives whose config was never dashboard-saved — probably not the hosted spokes ✨ knowledge: seed hosted spokes with the primary repo's knowledge dir and an 8 KB primer default #17145 is about, if they have ever toggled a setting.
  • On a hive where it does fire, the in-memory entry is persisted by the very next Save(), so "never writes hive.yaml" (body, agent-knowledge-and-memory.md) holds only until the first dashboard save. After that the entry is operator-set from the seed's point of view and is never re-probed (e.g. a default-branch rename goes stale).

Neither is a safety problem, but the docs sentence is currently inaccurate and the reach is much smaller than intended. Options: distinguish "absent" from "empty" by having redactedForPersist omit a nil GitSources (or add omitempty and make disconnect keep a non-nil [], which it already does), or drop the absent/empty distinction and use a separate explicit knowledge.seed_primary_repo: false opt-out. Either way, please say which in the docs paragraph.

The 4096→8192 primer default change is reasonable and knowledge-curator.md:514,535 still say 4096 — worth updating in the same PR since it touches that doc's subject.

@hivecommons-hive hivecommons-hive Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Security perspective (read at head 747624a).

Seeded clone URL can point at a different host than the one probed (GHE hives with blank base_url)

src/cmd/hive/boot_knowledge_seed.go:82-85 builds the git-source URL from raw cfg.GitHub.BaseURL, falling back to config.DefaultGitHubBaseURL (https://github.com) when blank. But GitHubConfig.ResolvedBaseURL() (src/pkg/config/config.go:4047-4064) exists precisely because "pooled/placeholder GHE hives legitimately carry base_url: "" with api_url: https://github.ibm.com/api/v3", and nothing in config loading fills BaseURL from APIURL.

On such a hive the probe (b.ghClient, built from ResolvedAPIURL()) confirms owner/repo has docs/knowledge on the GHE instance, then the seed appends URL: https://github.com/owner/repo. The auto-connect loop (src/cmd/hive/main.go:3023-3029) hands that to ConnectGitSource, which does an unauthenticated git clone (src/pkg/knowledge/gitsource.go:200-212, GIT_TERMINAL_PROMPT=0, no token). Whoever holds the same owner/repo namespace on public github.com — internal GHE org names are frequently unregistered publicly — gets their docs/knowledge/ indexed at the project layer and injected into every kick's # Relevant Knowledge primer. The probe and the clone must agree on host.

	base := strings.TrimRight(cfg.GitHub.ResolvedBaseURL(), "/")

(replacing the base := strings.TrimRight(cfg.GitHub.BaseURL, "/") / if base == "" { base = config.DefaultGitHubBaseURL } block). Please add a table case with APIURL = "https://github.example.com/api/v3", BaseURL = "" expecting https://github.example.com/acme/widgets.

Advisory (not security): for a private primary repo the same unauthenticated clone will fail with a WARN at every boot since the seeded source carries no credentials; consider checking repo visibility in the probe or documenting it.

— hive: agent=ci-maintainer backend=copilot model=claude-fable-5.1 copilot=1.0.88

Signed-off-by: Andrew Anderson <andy@clubanderson.com>
@clubanderson
clubanderson merged commit 310f28e into v5 Nov 8, 2026
79 of 80 checks passed
@kubestellar-prow
kubestellar-prow Bot deleted the feat/knowledge-seed-hosted branch November 8, 2026 01:50
@github-actions

github-actions Bot commented Nov 8, 2026

Copy link
Copy Markdown
Contributor

Thank you for your contribution! Your PR has been merged.

We'd love to hear how your experience was: share feedback

@hivecommons-hive hivecommons-hive Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Read this against the tree at 8269ed9. The seed logic itself is sound and the referenced symbols all exist with the signatures the probe interface needs (pkg/github/defaultbranch.go:40, pkg/github/client.go:4415; ProjectConfig.ForgeKind at pkg/config/config.go:1179; DefaultGitHubBaseURL at config.go:3720). One finding that contradicts the stated design:

The seeded entry is not in-memory-only — it is persisted to hive.yaml by the next unrelated save (medium).

  • seedKnowledgeGitSource appends to cfg.Knowledge.GitSources on the live *config.Config (src/cmd/hive/boot_knowledge_seed.go, the append just before the INFO log; wired at main.go in bootKnowledgeWith).
  • Config.Save → saveLocked marshals the whole in-memory struct: data, err := yaml.Marshal(c.redactedForPersist()) at src/pkg/config/config.go:7396, and redactedForPersist does not touch Knowledge.GitSources.
  • Save() is reached from ~12 call sites that have nothing to do with knowledge — e.g. SetAgentPausedAndSave (config.go:7357), and every dashboard s.saveConfig() (src/pkg/dashboard/api.go:747) on any settings change.

Consequence: after the first pause/resume or dashboard save, hive.yaml gains knowledge.git_sources: [{name: <repo>-knowledge, …}]. From then on the key is non-nil, so the seed's "absent key" check treats it as operator-set: the probe never re-runs (a moved knowledge dir or changed default branch is never picked up), and the PR body / agent-knowledge-and-memory.md sentence "Nothing is written to hive.yaml" is false in practice. Not data-destroying — the persisted entry is the correct one today — but it silently converts a default into operator config, which is exactly what the body says it avoids.

Two ways to make the claim true, either is small:

  1. Don't mutate cfg: have seedKnowledgeGitSource return the GitSourceConfigYAML and have bootKnowledgeWith connect it alongside (not appended to) b.cfg.Knowledge.GitSources.
  2. Or keep the append but strip seeded entries in redactedForPersist (would need a marker on the entry).

Option 1 also keeps the dashboard's handleGitSourcesList/Disconnect semantics unchanged, since those read s.deps.Config.Knowledge.GitSources (api_knowledge.go:1076, :1125).

The max_bytes 4096→8192 half is consistent across knowledge_config.go, primer_render.go, the parity test and the docs table.

— hive: agent=reviewer-2 backend=copilot model=claude-fable-5.1 copilot=1.0.88

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

dco-signoff: yes Indicates the PR's author has signed the DCO. size/L Denotes a PR that changes 100-499 lines, ignoring generated files.

Projects

None yet

Development

Successfully merging this pull request may close these issues.

✨ knowledge: seed hosted spokes with the primary repo's knowledge dir and an 8 KB primer default

1 participant