Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
1 change: 1 addition & 0 deletions changelog.d/added-knowledge-advisor-llm-config.md
Original file line number Diff line number Diff line change
@@ -0,0 +1 @@
- `knowledge.advisor_llm` config block (mode off/shadow/rank, model, endpoint, key, candidates, pane_lines, timeout_s, holdout_percent, on_error) with hard call ceilings of 6 per agent-hour and 300 per day unless `budget_override: true`; shadow/rank without a model resolves to off with a warning, and `Config.KnowledgeAdvisorLLM()` resolves model/endpoint/key like the reviewer (explicit → `governor.litellm`). Config only; the ranking pass lands separately (hivecommons/hive#17154).
5 changes: 5 additions & 0 deletions src/docs/agent-knowledge-and-memory.md
Original file line number Diff line number Diff line change
Expand Up @@ -117,6 +117,11 @@ knowledge search|list`, and `bd kb search` inside the pod. See
[knowledge-toc.md](knowledge-toc.md) and
[public-knowledge-mcp.md](public-knowledge-mcp.md).

The mid-task advisor (`knowledge.advisor`) can later rank candidates with a
model; its `knowledge.advisor_llm` block carries hard budget ceilings (6 calls
per agent-hour, 300 per day) that only `budget_override: true` lifts. See
[knowledge-curator.md](knowledge-curator.md#model-ranked-advisor-knowledgeadvisor_llm).

## 3. AGENTS.md / CLAUDE.md: injected by Hive, not read natively

Each agent's CLI starts in `/data/agents/<name>` (`crux.DefaultWorkDir`),
Expand Down
34 changes: 34 additions & 0 deletions src/docs/knowledge-curator.md
Original file line number Diff line number Diff line change
Expand Up @@ -683,6 +683,40 @@ knowledge:
advisor_exclude_agents: [scanner]
```

### Model-ranked advisor (`knowledge.advisor_llm`)

`knowledge.advisor_llm` is the config surface for the model-ranked variant
(hivecommons/hive#17054). This release only parses, validates and resolves
it; nothing calls a model yet. It is ignored unless `knowledge.advisor` is on.

```yaml
knowledge:
advisor: true
advisor_llm:
mode: off # off | shadow | rank
model: "<your-small-model>" # required for shadow/rank
endpoint: "" # default: governor.litellm endpoint
api_key_env: "" # or api_key_file; default: governor.litellm key
candidates: 5
pane_lines: 80
timeout_s: 15
max_calls_per_agent_hour: 6 # hard ceiling without budget_override
max_calls_per_day: 300 # hard ceiling without budget_override
budget_override: false
holdout_percent: 0 # 0..100
on_error: silence # silence | matcher
```

- `model`, `endpoint` and the key resolve like the governor reviewer:
explicit value, then `governor.litellm`, then unset. With no endpoint or key
from either place the block is unavailable.
- `shadow` or `rank` with no model resolves to `off` and logs a warning; it
never stops the hive from loading.
- `max_calls_per_agent_hour` above 6 or `max_calls_per_day` above 300 is a load
error naming the key unless `budget_override: true`.
- `holdout_percent` outside 0..100, an unknown `mode` or `on_error`, and a
negative `candidates`, `pane_lines` or `timeout_s` are load errors.

Phase 1 is a pure matcher with no model in the loop; the LLM-ranked variant
(OMP's advisor agent as the model) is hivecommons/hive#17054. The advisor is
wired only when the agent runtime runs in-process; the `crux serve` sidecar
Expand Down
160 changes: 160 additions & 0 deletions src/pkg/config/knowledge_advisor_llm.go
Original file line number Diff line number Diff line change
@@ -0,0 +1,160 @@
package config

import (
"fmt"
"strings"
)

// Defaults and hard ceilings for knowledge.advisor_llm (hivecommons/hive#17054).
const (
AdvisorLLMModeOff = "off"
AdvisorLLMModeShadow = "shadow"
AdvisorLLMModeRank = "rank"

AdvisorLLMOnErrorSilence = "silence"
AdvisorLLMOnErrorMatcher = "matcher"

DefaultAdvisorLLMCandidates = 5
DefaultAdvisorLLMPaneLines = 80
DefaultAdvisorLLMTimeoutS = 15
AdvisorLLMMaxCallsPerAgentHourCap = 6
AdvisorLLMMaxCallsPerDayCap = 300
)

// AdvisorLLMConfig is the knowledge.advisor_llm block. Zero numeric values
// take the defaults above; negative values are load errors.
type AdvisorLLMConfig struct {
Mode string `yaml:"mode,omitempty"`
Model string `yaml:"model,omitempty"`
Endpoint string `yaml:"endpoint,omitempty"`
APIKeyEnv string `yaml:"api_key_env,omitempty"`
APIKeyFile string `yaml:"api_key_file,omitempty"`
Candidates int `yaml:"candidates,omitempty"`
PaneLines int `yaml:"pane_lines,omitempty"`
TimeoutS int `yaml:"timeout_s,omitempty"`
MaxCallsPerAgentHour int `yaml:"max_calls_per_agent_hour,omitempty"`
MaxCallsPerDay int `yaml:"max_calls_per_day,omitempty"`
BudgetOverride bool `yaml:"budget_override,omitempty"`
HoldoutPercent int `yaml:"holdout_percent,omitempty"`
OnError string `yaml:"on_error,omitempty"`
}

// ResolvedAdvisorLLM is the effective advisor_llm setting with defaults and
// the governor.litellm fallbacks applied. The key value is read from a file or
// env var and is never stored in hive.yaml.
type ResolvedAdvisorLLM struct {
Mode string
Model string
Endpoint string
APIKey string
Candidates int
PaneLines int
TimeoutS int
MaxCallsPerAgentHour int
MaxCallsPerDay int
HoldoutPercent int
OnError string
}

func orIntDefault(v, def int) int {
if v == 0 {
return def
}
return v
}

// advisorLLMModel is the explicit model, else governor.litellm's default model.
func (c *Config) advisorLLMModel() string {
if m := strings.TrimSpace(c.Knowledge.AdvisorLLM.Model); m != "" {
return m
}
return strings.TrimSpace(c.Governor.LiteLLM.DefaultModel)
}

// validateKnowledgeAdvisorLLM returns load errors for malformed values and
// warnings for settings that are tolerated but resolve to off.
func (c *Config) validateKnowledgeAdvisorLLM() (warnings []string, err error) {
a := c.Knowledge.AdvisorLLM
switch a.Mode {
case "", AdvisorLLMModeOff, AdvisorLLMModeShadow, AdvisorLLMModeRank:
default:
return nil, fmt.Errorf("knowledge.advisor_llm.mode: %q is not %q, %q or %q", a.Mode, AdvisorLLMModeOff, AdvisorLLMModeShadow, AdvisorLLMModeRank)
}
switch a.OnError {
case "", AdvisorLLMOnErrorSilence, AdvisorLLMOnErrorMatcher:
default:
return nil, fmt.Errorf("knowledge.advisor_llm.on_error: %q is not %q or %q", a.OnError, AdvisorLLMOnErrorSilence, AdvisorLLMOnErrorMatcher)
}
for _, f := range []struct {
key string
v int
}{
{"candidates", a.Candidates},
{"pane_lines", a.PaneLines},
{"timeout_s", a.TimeoutS},
{"max_calls_per_agent_hour", a.MaxCallsPerAgentHour},
{"max_calls_per_day", a.MaxCallsPerDay},
} {
if f.v < 0 {
return nil, fmt.Errorf("knowledge.advisor_llm.%s: %d must not be negative (omit or 0 for the default)", f.key, f.v)
}
}
if a.HoldoutPercent < 0 || a.HoldoutPercent > 100 {
return nil, fmt.Errorf("knowledge.advisor_llm.holdout_percent: %d must be between 0 and 100", a.HoldoutPercent)
}
if !a.BudgetOverride {
if a.MaxCallsPerAgentHour > AdvisorLLMMaxCallsPerAgentHourCap {
return nil, fmt.Errorf("knowledge.advisor_llm.max_calls_per_agent_hour: %d exceeds the hard ceiling of %d; set budget_override: true to raise it", a.MaxCallsPerAgentHour, AdvisorLLMMaxCallsPerAgentHourCap)
}
if a.MaxCallsPerDay > AdvisorLLMMaxCallsPerDayCap {
return nil, fmt.Errorf("knowledge.advisor_llm.max_calls_per_day: %d exceeds the hard ceiling of %d; set budget_override: true to raise it", a.MaxCallsPerDay, AdvisorLLMMaxCallsPerDayCap)
}
}
if (a.Mode == AdvisorLLMModeShadow || a.Mode == AdvisorLLMModeRank) && c.advisorLLMModel() == "" {
warnings = append(warnings, fmt.Sprintf("knowledge.advisor_llm.mode %q has no model (set knowledge.advisor_llm.model); resolving to %q", a.Mode, AdvisorLLMModeOff))
}
return warnings, nil
}

// KnowledgeAdvisorLLM returns the effective advisor_llm setting. ok is false
// when knowledge.advisor is off (whatever the mode), or when no endpoint or key
// resolves from advisor_llm or governor.litellm. Mode is "off" when shadow/rank
// has no model. Precedence per field is explicit → governor.litellm → unset,
// matching GovernorConfig.ResolveReviewer.
func (c *Config) KnowledgeAdvisorLLM() (ResolvedAdvisorLLM, bool) {
if c == nil || !c.Knowledge.Advisor {
return ResolvedAdvisorLLM{}, false
}
a := c.Knowledge.AdvisorLLM
endpoint := strings.TrimRight(strings.TrimSpace(a.Endpoint), "/")
if endpoint == "" {
endpoint = c.Governor.LiteLLM.ResolveEndpoint()
}
apiKey := resolveKeyFromFileThenEnv(a.APIKeyFile, a.APIKeyEnv)
if apiKey == "" {
apiKey = c.Governor.LiteLLM.ResolveAPIKey()
}
if endpoint == "" || apiKey == "" {
return ResolvedAdvisorLLM{}, false
}
r := ResolvedAdvisorLLM{
Mode: strings.TrimSpace(a.Mode),
Model: c.advisorLLMModel(),
Endpoint: endpoint,
APIKey: apiKey,
Candidates: orIntDefault(a.Candidates, DefaultAdvisorLLMCandidates),
PaneLines: orIntDefault(a.PaneLines, DefaultAdvisorLLMPaneLines),
TimeoutS: orIntDefault(a.TimeoutS, DefaultAdvisorLLMTimeoutS),
MaxCallsPerAgentHour: orIntDefault(a.MaxCallsPerAgentHour, AdvisorLLMMaxCallsPerAgentHourCap),
MaxCallsPerDay: orIntDefault(a.MaxCallsPerDay, AdvisorLLMMaxCallsPerDayCap),
HoldoutPercent: a.HoldoutPercent,
OnError: strings.TrimSpace(a.OnError),
}
if r.Mode == "" || r.Model == "" {
r.Mode = AdvisorLLMModeOff
}
if r.OnError == "" {
r.OnError = AdvisorLLMOnErrorSilence
}
return r, true
}
129 changes: 129 additions & 0 deletions src/pkg/config/knowledge_advisor_llm_test.go
Original file line number Diff line number Diff line change
@@ -0,0 +1,129 @@
package config

import (
"strings"
"testing"

"gopkg.in/yaml.v3"
)

func advisorLLMCfg(a AdvisorLLMConfig) *Config {
return &Config{Knowledge: KnowledgeConfig{Enabled: true, Advisor: true, AdvisorLLM: a}}
}

func TestKnowledgeAdvisorLLM_Validate(t *testing.T) {
tests := []struct {
name string
cfg AdvisorLLMConfig
wantErr string
wantWarn bool
}{
{"empty block ok", AdvisorLLMConfig{}, "", false},
{"rank with model ok", AdvisorLLMConfig{Mode: "rank", Model: "m"}, "", false},
{"bad mode", AdvisorLLMConfig{Mode: "loud"}, "knowledge.advisor_llm.mode", false},
{"shadow without model warns", AdvisorLLMConfig{Mode: "shadow"}, "", true},
{"rank without model warns", AdvisorLLMConfig{Mode: "rank"}, "", true},
{"off without model silent", AdvisorLLMConfig{Mode: "off"}, "", false},
{"hourly ceiling exceeded", AdvisorLLMConfig{MaxCallsPerAgentHour: 7}, "max_calls_per_agent_hour", false},
{"daily ceiling exceeded", AdvisorLLMConfig{MaxCallsPerDay: 301}, "max_calls_per_day", false},
{"ceilings at limit ok", AdvisorLLMConfig{MaxCallsPerAgentHour: 6, MaxCallsPerDay: 300}, "", false},
{"ceilings raised with override", AdvisorLLMConfig{MaxCallsPerAgentHour: 60, MaxCallsPerDay: 3000, BudgetOverride: true}, "", false},
{"holdout negative", AdvisorLLMConfig{HoldoutPercent: -1}, "holdout_percent", false},
{"holdout over 100", AdvisorLLMConfig{HoldoutPercent: 101}, "holdout_percent", false},
{"holdout 100 ok", AdvisorLLMConfig{HoldoutPercent: 100}, "", false},
{"bad on_error", AdvisorLLMConfig{OnError: "panic"}, "knowledge.advisor_llm.on_error", false},
{"on_error matcher ok", AdvisorLLMConfig{OnError: "matcher"}, "", false},
{"candidates negative", AdvisorLLMConfig{Candidates: -1}, "candidates", false},
{"pane_lines negative", AdvisorLLMConfig{PaneLines: -1}, "pane_lines", false},
{"timeout_s negative", AdvisorLLMConfig{TimeoutS: -1}, "timeout_s", false},
}
for _, tt := range tests {
t.Run(tt.name, func(t *testing.T) {
warns, err := advisorLLMCfg(tt.cfg).validateKnowledgeAdvisorLLM()
if tt.wantErr == "" && err != nil {
t.Fatalf("unexpected error: %v", err)
}
if tt.wantErr != "" && (err == nil || !strings.Contains(err.Error(), tt.wantErr)) {
t.Fatalf("error = %v, want it to name %q", err, tt.wantErr)
}
if tt.wantWarn != (len(warns) > 0) {
t.Fatalf("warnings = %v, wantWarn %v", warns, tt.wantWarn)
}
})
}
}

func TestKnowledgeAdvisorLLM_ShadowWithoutModelResolvesOff(t *testing.T) {
c := advisorLLMCfg(AdvisorLLMConfig{Mode: "shadow", Endpoint: "http://gw.local", APIKeyEnv: "ADV_LLM_TEST_KEY"})
t.Setenv("ADV_LLM_TEST_KEY", "k")
r, ok := c.KnowledgeAdvisorLLM()
if !ok || r.Mode != AdvisorLLMModeOff {
t.Fatalf("got %+v ok=%v, want ok with mode off", r, ok)
}
}

func TestKnowledgeAdvisorLLM_DefaultsAndParse(t *testing.T) {
var k KnowledgeConfig
src := "advisor: true\nadvisor_llm:\n mode: rank\n model: small\n endpoint: http://gw.local/\n api_key_env: ADV_LLM_TEST_KEY\n"
if err := yaml.Unmarshal([]byte(src), &k); err != nil {
t.Fatal(err)
}
t.Setenv("ADV_LLM_TEST_KEY", "k")
r, ok := (&Config{Knowledge: k}).KnowledgeAdvisorLLM()
if !ok {
t.Fatal("ok = false")
}
want := ResolvedAdvisorLLM{Mode: "rank", Model: "small", Endpoint: "http://gw.local", APIKey: "k",
Candidates: 5, PaneLines: 80, TimeoutS: 15, MaxCallsPerAgentHour: 6, MaxCallsPerDay: 300, OnError: "silence"}
if r != want {
t.Fatalf("got %+v, want %+v", r, want)
}
}

func TestKnowledgeAdvisorLLM_RequiresAdvisor(t *testing.T) {
t.Setenv("ADV_LLM_TEST_KEY", "k")
for _, mode := range []string{"off", "shadow", "rank"} {
c := advisorLLMCfg(AdvisorLLMConfig{Mode: mode, Model: "m", Endpoint: "http://gw.local", APIKeyEnv: "ADV_LLM_TEST_KEY"})
c.Knowledge.Advisor = false
if _, ok := c.KnowledgeAdvisorLLM(); ok {
t.Errorf("mode %q: ok = true with knowledge.advisor false", mode)
}
}
var nilCfg *Config
if _, ok := nilCfg.KnowledgeAdvisorLLM(); ok {
t.Error("nil config ok = true")
}
}

func TestKnowledgeAdvisorLLM_ResolutionPrecedence(t *testing.T) {
t.Setenv(LiteLLMEndpointEnv, "")
t.Setenv(DefaultLiteLLMAPIKeyEnv, "")
t.Setenv("ADV_LLM_TEST_KEY", "explicit-key")
t.Setenv("ADV_LLM_GOV_KEY", "gov-key")

gov := func(c *Config) {
c.Governor.LiteLLM = LiteLLMConfig{Endpoint: "http://gov.local", APIKeyEnv: "ADV_LLM_GOV_KEY", DefaultModel: "gov-model"}
}
t.Run("explicit wins", func(t *testing.T) {
c := advisorLLMCfg(AdvisorLLMConfig{Mode: "rank", Model: "own", Endpoint: "http://own.local", APIKeyEnv: "ADV_LLM_TEST_KEY"})
gov(c)
r, ok := c.KnowledgeAdvisorLLM()
if !ok || r.Endpoint != "http://own.local" || r.APIKey != "explicit-key" || r.Model != "own" {
t.Fatalf("got %+v ok=%v", r, ok)
}
})
t.Run("absent falls back to governor.litellm", func(t *testing.T) {
c := advisorLLMCfg(AdvisorLLMConfig{Mode: "rank"})
gov(c)
r, ok := c.KnowledgeAdvisorLLM()
if !ok || r.Endpoint != "http://gov.local" || r.APIKey != "gov-key" || r.Model != "gov-model" || r.Mode != "rank" {
t.Fatalf("got %+v ok=%v", r, ok)
}
})
t.Run("both absent", func(t *testing.T) {
c := advisorLLMCfg(AdvisorLLMConfig{Mode: "rank", Model: "own"})
if _, ok := c.KnowledgeAdvisorLLM(); ok {
t.Fatal("ok = true with no endpoint or key anywhere")
}
})
}
3 changes: 3 additions & 0 deletions src/pkg/config/knowledge_config.go
Original file line number Diff line number Diff line change
Expand Up @@ -56,6 +56,9 @@ type KnowledgeConfig struct {
// AdvisorExcludeAgents lists agents (by name or family base name) the
// advisor never types at, e.g. lanes whose work is reading, not editing.
AdvisorExcludeAgents []string `yaml:"advisor_exclude_agents,omitempty"`
// AdvisorLLM configures the optional model-ranked advisor pass
// (hivecommons/hive#17054). Ignored unless Advisor is true.
AdvisorLLM AdvisorLLMConfig `yaml:"advisor_llm,omitempty"`
}

// PublicKnowledgeConfig is the dashboard-persisted owner override for the
Expand Down
17 changes: 16 additions & 1 deletion src/pkg/config/testdata/agent_runtime_golden.json
Original file line number Diff line number Diff line change
Expand Up @@ -550,7 +550,22 @@
"AgentScopes": null,
"Advisor": false,
"AdvisorMaxPerKick": 0,
"AdvisorExcludeAgents": null
"AdvisorExcludeAgents": null,
"AdvisorLLM": {
"Mode": "",
"Model": "",
"Endpoint": "",
"APIKeyEnv": "",
"APIKeyFile": "",
"Candidates": 0,
"PaneLines": 0,
"TimeoutS": 0,
"MaxCallsPerAgentHour": 0,
"MaxCallsPerDay": 0,
"BudgetOverride": false,
"HoldoutPercent": 0,
"OnError": ""
}
},
"Hub": {
"Enabled": false,
Expand Down
Loading
Loading