From 8050f6d54f0779e0b5840878ff168f807a359af2 Mon Sep 17 00:00:00 2001 From: Danathar Date: Thu, 19 Nov 2026 01:41:20 +0000 Subject: [PATCH 1/2] =?UTF-8?q?=E2=9C=A8=20ask:=20hub=20backend=20for=20op?= =?UTF-8?q?tional=20Ask=20AI=20help=20(phase=201=20of=20#19177)?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Adds the hub backend for the optional Ask AI help, off by default and inert: - `ask:` config block and HIVE_ASK_* env overrides; if enabled but a value is missing or the key file is unusable it stays off and logs one warning without values. The key file must sit under the allowed secret directory. Project and integration ids are redacted from the config export and never appear in JSON. - pkg/ask: vendor Chat API client (non-streamed, key in X-API-KEY, only integration_id and query sent, response shape pinned by a contract test), source allowlist, limits (person, hive, hub-wide per minute, in flight, daily), circuit breaker, server-side thread handles bound to the person with a TTL, and audit lines with outcome and counts only. - Hub: per-person consent record (own file, atomic, 0600, text_version, grant and withdraw, deleted with the user), POST /api/ask behind a browser session that refuses credential-shaped input with FindSecret, and an admin counter. With the switch off every route answers 404. Tests use a fake vendor server and assert the exact request body, that seeded secrets, repo and hive names and logins never leave, and that consent, limits and the breaker hold. Closes #19180 Hive-Run: hivecommons/hive#19180 Hive-Plan: dashboard-ask-ai-help phase 1 (hub backend) Hive-Spec: dashboard-ask-ai-help#phase-1 Co-Authored-By: Claude Sonnet 5.5 Signed-off-by: Danathar --- changelog.d/added-19180-ask-ai-hub-backend.md | 1 + src/docs/design/dashboard-ask-ai-help.md | 10 +- src/docs/env-vars.md | 22 + src/hive.yaml.example | 19 + src/pkg/ask/client.go | 155 +++++ src/pkg/ask/client_test.go | 173 ++++++ src/pkg/ask/limits.go | 236 ++++++++ src/pkg/ask/section.go | 33 ++ src/pkg/ask/service.go | 312 ++++++++++ src/pkg/ask/service_test.go | 531 ++++++++++++++++++ src/pkg/ask/sources.go | 93 +++ src/pkg/ask/sources_test.go | 91 +++ src/pkg/ask/threads.go | 105 ++++ src/pkg/config/ask.go | 243 ++++++++ src/pkg/config/ask_test.go | 158 ++++++ src/pkg/config/config.go | 2 + src/pkg/dashboard/api_config_export.go | 8 + .../dashboard/api_config_export_ask_test.go | 41 ++ src/pkg/hub/ask.go | 349 ++++++++++++ src/pkg/hub/ask_test.go | 526 +++++++++++++++++ src/pkg/hub/saas.go | 1 + src/pkg/hub/saas_users_admin.go | 16 + src/pkg/hub/server.go | 7 + 23 files changed, 3129 insertions(+), 3 deletions(-) create mode 100644 changelog.d/added-19180-ask-ai-hub-backend.md create mode 100644 src/pkg/ask/client.go create mode 100644 src/pkg/ask/client_test.go create mode 100644 src/pkg/ask/limits.go create mode 100644 src/pkg/ask/section.go create mode 100644 src/pkg/ask/service.go create mode 100644 src/pkg/ask/service_test.go create mode 100644 src/pkg/ask/sources.go create mode 100644 src/pkg/ask/sources_test.go create mode 100644 src/pkg/ask/threads.go create mode 100644 src/pkg/config/ask.go create mode 100644 src/pkg/config/ask_test.go create mode 100644 src/pkg/dashboard/api_config_export_ask_test.go create mode 100644 src/pkg/hub/ask.go create mode 100644 src/pkg/hub/ask_test.go diff --git a/changelog.d/added-19180-ask-ai-hub-backend.md b/changelog.d/added-19180-ask-ai-hub-backend.md new file mode 100644 index 0000000000..e759766944 --- /dev/null +++ b/changelog.d/added-19180-ask-ai-hub-backend.md @@ -0,0 +1 @@ +- Hub operators can switch on an optional Ask AI help backend: a signed-in person who confirms a disclosure can ask a question and get a cited answer from the vendor's Chat API, called from the hub with a key held in a Secret. Off by default; with it off the routes answer 404 and nothing leaves the hub. Questions that look like credentials are refused, answers need an allowlisted docs source, and per-person, hub-wide and daily limits with a circuit breaker apply. No panel yet ([#19180](https://github.com/hivecommons/hive/issues/19180)). diff --git a/src/docs/design/dashboard-ask-ai-help.md b/src/docs/design/dashboard-ask-ai-help.md index 680c7d2a16..605dc60239 100644 --- a/src/docs/design/dashboard-ask-ai-help.md +++ b/src/docs/design/dashboard-ask-ai-help.md @@ -551,9 +551,13 @@ Not verified, and the reason: - **Which CSP applies to the hub's main dashboard document** and whether the Node proxy or ingress adds one. Not traced. The recommended design does not depend on it. -- **The vendor's non-streamed chat response shape** (source field names), the - streamed endpoint path and events, and whether the chat endpoint returns a - thread id in the body. The public pages read do not show them. +- **The vendor's streamed endpoint path and events.** Not needed for v1. + Resolved in phase 1 (#19180): the non-streamed `POST + /query/v1/projects/{project_id}/chat/` returns `answer`, `thread_id`, + `question_answer_id`, `is_uncertain` and `relevant_sources` (each with + `source_url`, `title` and `contains_internal_data`); a follow-up is `POST + /query/v1/threads/{thread_id}/chat/`. The client pins this shape with a + contract test (`src/pkg/ask/client_test.go`). - **Whether the vendor documents a hosted or frame mode.** None was found; the claim is an absence in the pages read. - **Vendor per-source include and exclude patterns, and deletion of a single diff --git a/src/docs/env-vars.md b/src/docs/env-vars.md index e28acd1b66..8b8e38292d 100644 --- a/src/docs/env-vars.md +++ b/src/docs/env-vars.md @@ -707,6 +707,28 @@ only when neither is configured. With two or more providers configured, `/login` renders a provider picker; with exactly one it redirects straight into it. A provider with a client id but missing/invalid issuer is **silently skipped** so it cannot take down login for the others — check the `hub login enabled providers=` startup log line when a button is missing. +## Ask AI help (hub backend) + +Optional and **off by default**. Nothing here opens a route, makes a request or changes a header until the switch is on and every required value is set; if one is missing or the key file cannot be read, Ask stays off and logs one warning that names the reason and no value. Env wins over the `ask:` block in `hive.yaml`. The key is read from a file under the allowed secret directory; there is deliberately no variable that holds the key itself. Design: [dashboard Ask AI help](design/dashboard-ask-ai-help.md). + +| Variable | Required | Default | Purpose | +|---|---:|---|---| +| `HIVE_ASK_ENABLED` | No | config `ask.enabled` (default off) | The switch. Off: every Ask route answers 404, no consent prompt, no request to the vendor. | +| `HIVE_ASK_MODE` | No | config `ask.mode` (the hub defaults to `direct`) | `off`, `direct` (this process calls the vendor with its own key) or `hub` (a hive calling the hub relay; not available yet). The hub is always `direct`. | +| `HIVE_ASK_KAPA_PROJECT_ID` | When enabled | config `ask.kapa.project_id` | The vendor project. Configuration only; never returned by an API, logged or exported. | +| `HIVE_ASK_KAPA_INTEGRATION_ID` | When enabled | config `ask.kapa.integration_id` | The vendor integration identifier. Same handling as the project id. | +| `HIVE_ASK_KAPA_API_KEY_FILE` | When enabled | config `ask.kapa.api_key_file` | Path to the vendor API key, which must be under the allowed secret directory (a mounted Secret). | +| `HIVE_ASK_SEND_SECTION_LABEL` | No | config `ask.send_section_label` (default on) | Whether a fixed-vocabulary label for the dashboard section the person is in is sent with the question. Off sends the question text alone. | +| `HIVE_ASK_LIMIT_PERSON_BURST` | No | `3` | Questions one person may ask per minute. | +| `HIVE_ASK_LIMIT_PERSON_PER_HOUR` | No | `30` | Questions one person may ask per hour. | +| `HIVE_ASK_LIMIT_HIVE_PER_HOUR` | No | `120` | Questions one hive may relay per hour (used by the hive relay in a later phase). | +| `HIVE_ASK_LIMIT_GLOBAL_PER_MINUTE` | No | `8` | Vendor calls per minute across the hub; keep it under the vendor's per-team chat limit, which the docs site's widget shares. | +| `HIVE_ASK_LIMIT_IN_FLIGHT` | No | `4` | Vendor calls in flight at once across the hub (one per person). | +| `HIVE_ASK_LIMIT_DAILY` | No | `500` | Vendor calls per UTC day; when spent, Ask answers with the docs-link fallback until it resets. | +| `HIVE_ASK_LIMIT_QUESTION_CHARS` | No | `1000` | Longest question accepted, in characters. | +| `HIVE_ASK_LIMIT_ANSWER_CHARS` | No | `4000` | Longest answer returned, in characters. | +| `HIVE_ASK_LIMIT_TIMEOUT_SECONDS` | No | `30` | Time allowed for one vendor call. | + ## Kubernetes downward API and platform variables | Variable | Required | Default | Purpose | diff --git a/src/hive.yaml.example b/src/hive.yaml.example index 409f501051..05a7c76baf 100644 --- a/src/hive.yaml.example +++ b/src/hive.yaml.example @@ -1433,6 +1433,25 @@ data: # max_requests_per_hour: 10 # search requests all accepts may spend together per hour (default 10, at most 60); each accept spends at most 2 +# Ask AI help (docs/design/dashboard-ask-ai-help.md, #19177): an optional help +# panel backend that sends a person's question, after they consent, to the +# vendor's Chat API from the hub. Off by default and inert. The key lives in a +# mounted Secret, never here. If enabled but a value is missing or the key file +# is unreadable it stays off and logs one warning without values. +# ask: +# enabled: false # true (or env HIVE_ASK_ENABLED=true) switches it on; default off +# mode: direct # direct on the hub (default); env HIVE_ASK_MODE +# kapa: +# project_id: "" # env HIVE_ASK_KAPA_PROJECT_ID; never returned, logged or exported +# integration_id: "" # env HIVE_ASK_KAPA_INTEGRATION_ID; same handling +# api_key_file: /secrets/ask-kapa-api-key # env HIVE_ASK_KAPA_API_KEY_FILE; must be under the allowed secret directory +# send_section_label: true # false sends the question text alone +# limits: # optional; env HIVE_ASK_LIMIT_*; see docs/env-vars.md for the defaults +# person_burst: 3 +# global_per_minute: 8 +# daily: 500 + + # Sandbox Job GitHub read relay (docs/sandbox-isolation.md): lets sandbox Jobs # make authenticated, read-only GitHub reads without ever holding a credential. # Default off. There is NO default bind: with enabled: true, listen_addr is diff --git a/src/pkg/ask/client.go b/src/pkg/ask/client.go new file mode 100644 index 0000000000..304a808146 --- /dev/null +++ b/src/pkg/ask/client.go @@ -0,0 +1,155 @@ +// Package ask is the backend for the optional Ask AI help (#19177): a +// server-side call to the vendor's Chat API, source filtering, limits, a +// circuit breaker and thread bookkeeping. It holds no HTTP route and no +// consent record; the hub wires those around it. See +// src/docs/design/dashboard-ask-ai-help.md. +// +// Everything that leaves the deployment is built in one place, Client.Chat, +// from named inputs, so a test against a fake vendor server can assert the +// exact request body. +package ask + +import ( + "bytes" + "context" + "encoding/json" + "errors" + "io" + "net/http" + "net/url" + "strings" + "time" +) + +// DefaultBaseURL is the vendor's public API host. +const DefaultBaseURL = "https://api.kapa.ai" + +// maxResponseBytes bounds how much of a vendor response is read. +const maxResponseBytes = 1 << 20 + +// Errors the client returns. None carries a request value, a response body, the +// key or a vendor identifier. +var ( + // ErrRateLimited is the vendor's HTTP 429. + ErrRateLimited = errors.New("ask: vendor rate limited") + // ErrUnavailable is a timeout, a transport failure or a 5xx. + ErrUnavailable = errors.New("ask: vendor unavailable") + // ErrRejected is any other non-200 answer (a bad key, a bad request). + ErrRejected = errors.New("ask: vendor rejected the request") + // ErrBadResponse is a 200 whose body is not the pinned shape. + ErrBadResponse = errors.New("ask: vendor response not understood") +) + +// chatRequest is the complete set of fields sent to the vendor. Adding a field +// here is adding to "What leaves the deployment" in the design record: it needs +// a reviewer and a test. In particular there is no user object, no metadata and +// no origin URL. +type chatRequest struct { + IntegrationID string `json:"integration_id"` + Query string `json:"query"` +} + +// Source is one cited source of an answer. +type Source struct { + URL string `json:"url"` + Title string `json:"title"` +} + +// ChatResponse is the non-streamed response shape, pinned by a contract test +// against the vendor's API reference (POST /query/v1/projects/{project_id}/chat/ +// and POST /query/v1/threads/{thread_id}/chat/): answer, thread_id, +// question_answer_id, is_uncertain and relevant_sources[{source_url, title, +// contains_internal_data}]. +type ChatResponse struct { + Answer string `json:"answer"` + ThreadID string `json:"thread_id"` + QuestionAnswerID string `json:"question_answer_id"` + IsUncertain bool `json:"is_uncertain"` + RelevantSources []VendorSource `json:"relevant_sources"` +} + +// VendorSource is one entry of relevant_sources as the vendor returns it. +type VendorSource struct { + SourceURL string `json:"source_url"` + Title string `json:"title"` + ContainsInternalData bool `json:"contains_internal_data"` +} + +// Client calls the vendor Chat API. The zero value is not usable; build one +// with NewClient. The key, project id and integration id are held in +// unexported fields, are never part of a returned error, and are not written to +// any log by this package. +type Client struct { + baseURL string + projectID string + integrationID string + apiKey string + http *http.Client +} + +// NewClient builds a client. baseURL may be empty for the vendor default (tests +// pass a fake server). timeout bounds one request. +func NewClient(baseURL, projectID, integrationID, apiKey string, timeout time.Duration) *Client { + if baseURL == "" { + baseURL = DefaultBaseURL + } + if timeout <= 0 { + timeout = 30 * time.Second + } + return &Client{ + baseURL: strings.TrimRight(baseURL, "/"), + projectID: projectID, + integrationID: integrationID, + apiKey: apiKey, + http: &http.Client{ + Timeout: timeout, + // Never follow a redirect with the key attached. + CheckRedirect: func(*http.Request, []*http.Request) error { return http.ErrUseLastResponse }, + }, + } +} + +// Chat asks one question. An empty threadID starts a conversation; a non-empty +// one continues the vendor thread. Only query and the integration id are sent. +func (c *Client) Chat(ctx context.Context, query, threadID string) (ChatResponse, error) { + var endpoint string + if threadID == "" { + endpoint = c.baseURL + "/query/v1/projects/" + url.PathEscape(c.projectID) + "/chat/" + } else { + endpoint = c.baseURL + "/query/v1/threads/" + url.PathEscape(threadID) + "/chat/" + } + body, err := json.Marshal(chatRequest{IntegrationID: c.integrationID, Query: query}) + if err != nil { + return ChatResponse{}, ErrBadResponse + } + req, err := http.NewRequestWithContext(ctx, http.MethodPost, endpoint, bytes.NewReader(body)) + if err != nil { + return ChatResponse{}, ErrUnavailable + } + req.Header.Set("X-API-KEY", c.apiKey) + req.Header.Set("Content-Type", "application/json") + req.Header.Set("Accept", "application/json") + resp, err := c.http.Do(req) + if err != nil { + // err is a *url.Error carrying the full URL (project id, thread id); + // drop it. + return ChatResponse{}, ErrUnavailable + } + defer resp.Body.Close() + switch { + case resp.StatusCode == http.StatusTooManyRequests: + _, _ = io.Copy(io.Discard, io.LimitReader(resp.Body, maxResponseBytes)) + return ChatResponse{}, ErrRateLimited + case resp.StatusCode >= 500: + _, _ = io.Copy(io.Discard, io.LimitReader(resp.Body, maxResponseBytes)) + return ChatResponse{}, ErrUnavailable + case resp.StatusCode != http.StatusOK: + _, _ = io.Copy(io.Discard, io.LimitReader(resp.Body, maxResponseBytes)) + return ChatResponse{}, ErrRejected + } + var out ChatResponse + if err := json.NewDecoder(io.LimitReader(resp.Body, maxResponseBytes)).Decode(&out); err != nil { + return ChatResponse{}, ErrBadResponse + } + return out, nil +} diff --git a/src/pkg/ask/client_test.go b/src/pkg/ask/client_test.go new file mode 100644 index 0000000000..ecfe211343 --- /dev/null +++ b/src/pkg/ask/client_test.go @@ -0,0 +1,173 @@ +package ask + +import ( + "context" + "encoding/json" + "errors" + "io" + "net/http" + "net/http/httptest" + "strings" + "testing" + "time" +) + +const ( + testProjectID = "proj-0000-secret" + testIntegrationID = "integ-1111-secret" + testAPIKey = "kapa-key-2222-secret" +) + +// contractResponse is the vendor's documented non-streamed 200 body +// (POST /query/v1/projects/{project_id}/chat/): answer, thread_id, +// question_answer_id, is_uncertain and relevant_sources with source_url, title +// and contains_internal_data. It pins the shape the client decodes. +const contractResponse = `{ + "answer": "Open Settings, then Features.", + "thread_id": "vendor-thread-1", + "question_answer_id": "qa-1", + "is_uncertain": false, + "relevant_sources": [ + {"source_url": "https://docs.hivecommons.dev/docs/dashboard", "title": "Dashboard", "contains_internal_data": false} + ] +}` + +type recorded struct { + path string + method string + key string + ctype string + body []byte +} + +func fakeVendor(t *testing.T, status int, body string) (*httptest.Server, *[]recorded) { + t.Helper() + var got []recorded + srv := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) { + b, _ := io.ReadAll(r.Body) + got = append(got, recorded{r.URL.Path, r.Method, r.Header.Get("X-API-KEY"), r.Header.Get("Content-Type"), b}) + w.WriteHeader(status) + _, _ = io.WriteString(w, body) + })) + t.Cleanup(srv.Close) + return srv, &got +} + +func TestClientChatContractAndExactRequestBody(t *testing.T) { + srv, got := fakeVendor(t, 200, contractResponse) + c := NewClient(srv.URL, testProjectID, testIntegrationID, testAPIKey, 5*time.Second) + resp, err := c.Chat(context.Background(), "where is the Features toggle?", "") + if err != nil { + t.Fatal(err) + } + if resp.Answer != "Open Settings, then Features." || resp.ThreadID != "vendor-thread-1" || + resp.QuestionAnswerID != "qa-1" || resp.IsUncertain || len(resp.RelevantSources) != 1 || + resp.RelevantSources[0].SourceURL != "https://docs.hivecommons.dev/docs/dashboard" || + resp.RelevantSources[0].Title != "Dashboard" || resp.RelevantSources[0].ContainsInternalData { + t.Fatalf("response not decoded as the pinned shape: %+v", resp) + } + if len(*got) != 1 { + t.Fatalf("requests = %d", len(*got)) + } + r := (*got)[0] + if r.method != "POST" || r.path != "/query/v1/projects/"+testProjectID+"/chat/" { + t.Fatalf("unexpected request %s %s", r.method, r.path) + } + if r.key != testAPIKey || r.ctype != "application/json" { + t.Fatalf("headers: key=%q type=%q", r.key, r.ctype) + } + // The body holds exactly the two fields in the design's "What leaves the + // deployment" table that belong in the JSON body: no user object, no + // metadata, no origin URL. + var fields map[string]any + if err := json.Unmarshal(r.body, &fields); err != nil { + t.Fatal(err) + } + if len(fields) != 2 || fields["integration_id"] != testIntegrationID || fields["query"] != "where is the Features toggle?" { + t.Fatalf("request body = %s", r.body) + } + if strings.Contains(string(r.body), testAPIKey) || strings.Contains(string(r.body), testProjectID) { + t.Fatalf("body leaks the key or project id: %s", r.body) + } +} + +func TestClientContinuesVendorThread(t *testing.T) { + srv, got := fakeVendor(t, 200, contractResponse) + c := NewClient(srv.URL, testProjectID, testIntegrationID, testAPIKey, 5*time.Second) + if _, err := c.Chat(context.Background(), "and then?", "vendor-thread-1"); err != nil { + t.Fatal(err) + } + if p := (*got)[0].path; p != "/query/v1/threads/vendor-thread-1/chat/" { + t.Fatalf("follow-up path = %s", p) + } +} + +func TestClientErrorsCarryNoValues(t *testing.T) { + cases := []struct { + status int + body string + want error + }{ + {429, "slow down " + testAPIKey, ErrRateLimited}, + {500, testProjectID, ErrUnavailable}, + {503, "", ErrUnavailable}, + {401, "bad key " + testAPIKey, ErrRejected}, + {404, testProjectID, ErrRejected}, + {200, "not json " + testAPIKey, ErrBadResponse}, + } + for _, tc := range cases { + srv, _ := fakeVendor(t, tc.status, tc.body) + c := NewClient(srv.URL, testProjectID, testIntegrationID, testAPIKey, 5*time.Second) + _, err := c.Chat(context.Background(), "q", "") + if !errors.Is(err, tc.want) { + t.Fatalf("status %d: err = %v, want %v", tc.status, err, tc.want) + } + for _, secret := range []string{testAPIKey, testProjectID, testIntegrationID, srv.URL} { + if strings.Contains(err.Error(), secret) { + t.Fatalf("error %q carries %q", err, secret) + } + } + } +} + +func TestClientTransportFailureAndTimeoutCarryNoURL(t *testing.T) { + srv, _ := fakeVendor(t, 200, contractResponse) + url := srv.URL + srv.Close() // closed port + c := NewClient(url, testProjectID, testIntegrationID, testAPIKey, time.Second) + _, err := c.Chat(context.Background(), "q", "") + if !errors.Is(err, ErrUnavailable) || strings.Contains(err.Error(), testProjectID) || strings.Contains(err.Error(), url) { + t.Fatalf("transport error = %v", err) + } + + slow := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) { + select { + case <-r.Context().Done(): + case <-time.After(2 * time.Second): + } + })) + defer slow.Close() + c = NewClient(slow.URL, testProjectID, testIntegrationID, testAPIKey, 50*time.Millisecond) + if _, err := c.Chat(context.Background(), "q", ""); !errors.Is(err, ErrUnavailable) { + t.Fatalf("timeout error = %v", err) + } +} + +func TestClientDoesNotFollowRedirectsWithTheKey(t *testing.T) { + var leaked bool + other := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) { + leaked = r.Header.Get("X-API-KEY") != "" + })) + defer other.Close() + redir := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) { + http.Redirect(w, r, other.URL, http.StatusTemporaryRedirect) + })) + defer redir.Close() + c := NewClient(redir.URL, testProjectID, testIntegrationID, testAPIKey, time.Second) + if _, err := c.Chat(context.Background(), "q", ""); !errors.Is(err, ErrRejected) { + t.Fatalf("err = %v", err) + } + if leaked { + t.Fatal("the key followed a redirect") + } +} diff --git a/src/pkg/ask/limits.go b/src/pkg/ask/limits.go new file mode 100644 index 0000000000..c23636eb0b --- /dev/null +++ b/src/pkg/ask/limits.go @@ -0,0 +1,236 @@ +package ask + +import ( + "sync" + "time" + + "github.com/hivecommons/hive/pkg/config" +) + +// Denial says which limit stopped a request. +type Denial string + +const ( + DenyNone Denial = "" + DenyPerson Denial = "person" + DenyHive Denial = "hive" + DenyGlobal Denial = "global" + DenyBusy Denial = "busy" + DenyBudget Denial = "budget" +) + +// Limiter enforces the Ask limits: per person (a one-minute burst and an hourly +// cap, one request in flight), per hive (hourly), hub-wide (per minute and in +// flight) and a daily budget. In memory only; a restart resets it, which +// matches "no per-person cost data is kept beyond the hourly counter in +// memory" in the design record. +type Limiter struct { + mu sync.Mutex + now func() time.Time + lim config.AskLimits + + person map[string][]time.Time + hive map[string][]time.Time + global []time.Time + inFlight int + personInFlight map[string]bool + day string + dayCount int +} + +// NewLimiter builds a limiter. now may be nil for time.Now. +func NewLimiter(lim config.AskLimits, now func() time.Time) *Limiter { + if now == nil { + now = time.Now + } + return &Limiter{ + now: now, lim: lim, + person: map[string][]time.Time{}, hive: map[string][]time.Time{}, + personInFlight: map[string]bool{}, + } +} + +// prune drops timestamps at or before cutoff. +func prune(ts []time.Time, cutoff time.Time) []time.Time { + i := 0 + for i < len(ts) && !ts[i].After(cutoff) { + i++ + } + return ts[i:] +} + +// retryAfter is how long until the oldest kept timestamp leaves its window. +func retryAfter(ts []time.Time, window time.Duration, now time.Time) time.Duration { + if len(ts) == 0 { + return time.Second + } + d := ts[0].Add(window).Sub(now) + if d < time.Second { + d = time.Second + } + return d +} + +// Acquire reserves one request for person (and hive, which may be empty). On +// success it returns a release func the caller must call when the vendor call +// ends. On denial it returns the limit that stopped it and a retry hint; nothing +// is counted for a denied request. +func (l *Limiter) Acquire(person, hive string) (release func(), denial Denial, wait time.Duration) { + l.mu.Lock() + defer l.mu.Unlock() + now := l.now() + + day := now.UTC().Format("2006-01-02") + if day != l.day { + l.day, l.dayCount = day, 0 + } + if l.dayCount >= l.lim.Daily { + next := now.UTC().Truncate(24 * time.Hour).Add(24 * time.Hour) + return nil, DenyBudget, next.Sub(now) + } + + l.global = prune(l.global, now.Add(-time.Minute)) + if len(l.global) >= l.lim.GlobalPerMinute { + return nil, DenyGlobal, retryAfter(l.global, time.Minute, now) + } + if l.inFlight >= l.lim.InFlight || l.personInFlight[person] { + return nil, DenyBusy, 2 * time.Second + } + + pt := prune(l.person[person], now.Add(-time.Hour)) + l.person[person] = pt + if len(pt) >= l.lim.PersonPerHour { + return nil, DenyPerson, retryAfter(pt, time.Hour, now) + } + burst := 0 + for _, t := range pt { + if t.After(now.Add(-time.Minute)) { + burst++ + } + } + if burst >= l.lim.PersonBurst { + for _, t := range pt { + if t.After(now.Add(-time.Minute)) { + return nil, DenyPerson, retryAfter([]time.Time{t}, time.Minute, now) + } + } + } + if hive != "" { + ht := prune(l.hive[hive], now.Add(-time.Hour)) + l.hive[hive] = ht + if len(ht) >= l.lim.HivePerHour { + return nil, DenyHive, retryAfter(ht, time.Hour, now) + } + l.hive[hive] = append(ht, now) + } + + l.person[person] = append(pt, now) + l.global = append(l.global, now) + l.dayCount++ + l.inFlight++ + l.personInFlight[person] = true + var once sync.Once + return func() { + once.Do(func() { + l.mu.Lock() + defer l.mu.Unlock() + l.inFlight-- + delete(l.personInFlight, person) + }) + }, DenyNone, 0 +} + +// DailyCount is how many requests were admitted today (UTC). +func (l *Limiter) DailyCount() int { + l.mu.Lock() + defer l.mu.Unlock() + if l.now().UTC().Format("2006-01-02") != l.day { + return 0 + } + return l.dayCount +} + +// Breaker is a circuit breaker for the vendor call. After threshold +// consecutive failures it opens for cooldown; then one probe is let through, +// and its result closes or re-opens it. +type Breaker struct { + mu sync.Mutex + now func() time.Time + threshold int + cooldown time.Duration + + failures int + openedAt time.Time + open bool + probing bool + + // OnOpen and OnClose are called (without the lock held) when the breaker + // changes state, for the audit lines. + OnOpen func() + OnClose func() +} + +// NewBreaker builds a breaker. now may be nil for time.Now. +func NewBreaker(threshold int, cooldown time.Duration, now func() time.Time) *Breaker { + if now == nil { + now = time.Now + } + if threshold < 1 { + threshold = 5 + } + if cooldown <= 0 { + cooldown = time.Minute + } + return &Breaker{now: now, threshold: threshold, cooldown: cooldown} +} + +// Allow reports whether a vendor call may be made now. While open it refuses +// until the cool-down passes, then lets exactly one probe through. +func (b *Breaker) Allow() bool { + b.mu.Lock() + defer b.mu.Unlock() + if !b.open { + return true + } + if b.probing || b.now().Sub(b.openedAt) < b.cooldown { + return false + } + b.probing = true + return true +} + +// Success records a good vendor call. +func (b *Breaker) Success() { + b.mu.Lock() + wasOpen := b.open + b.failures, b.open, b.probing = 0, false, false + cb := b.OnClose + b.mu.Unlock() + if wasOpen && cb != nil { + cb() + } +} + +// Failure records a failed vendor call. +func (b *Breaker) Failure() { + b.mu.Lock() + b.failures++ + opened := false + if b.probing || (!b.open && b.failures >= b.threshold) { + opened = !b.open + b.open, b.probing = true, false + b.openedAt = b.now() + } + cb := b.OnOpen + b.mu.Unlock() + if opened && cb != nil { + cb() + } +} + +// Open reports whether the breaker is currently refusing calls. +func (b *Breaker) Open() bool { + b.mu.Lock() + defer b.mu.Unlock() + return b.open +} diff --git a/src/pkg/ask/section.go b/src/pkg/ask/section.go new file mode 100644 index 0000000000..ef71e04a69 --- /dev/null +++ b/src/pkg/ask/section.go @@ -0,0 +1,33 @@ +package ask + +// sectionLabels is the closed vocabulary of dashboard section labels that may +// be sent with a question. The keys are the dashboard's own section element +// ids; the values are our own fixed wording. A section id the browser submits +// that is not in this map is dropped, so nothing the person typed or the page +// shows can reach the vendor through this field. +var sectionLabels = map[string]string{ + "overview": "Overview", + "agents-section": "Agents", + "agent-activity-section": "Agent activity", + "activity-monitor-section": "Hive activity monitor", + "logs-section": "Agent Logs", + "knowledge-section": "Knowledge", + "meetings-section": "Meetings", + "inbox-section": "Inbox", + "calendar-section": "Calendar", + "customers-section": "Customers", + "contributors-section": "Contributors", + "debug-section": "Diagnostics", + "faq-section": "FAQ", + "settings": "Settings", + "settings-features": "Settings, Features", + "settings-agents": "Settings, Agents", + "settings-repos": "Settings, Repositories", + "hub-fleet": "Hub, Fleet", + "hub-users": "Hub, Users", + "hub-settings": "Hub, Settings", +} + +// SectionLabel returns the fixed label for a section id, or "" when the id is +// not in the closed vocabulary. +func SectionLabel(id string) string { return sectionLabels[id] } diff --git a/src/pkg/ask/service.go b/src/pkg/ask/service.go new file mode 100644 index 0000000000..f73f5a2b05 --- /dev/null +++ b/src/pkg/ask/service.go @@ -0,0 +1,312 @@ +package ask + +import ( + "context" + "errors" + "log/slog" + "strings" + "sync" + "time" + "unicode/utf8" + + "github.com/hivecommons/hive/crux/keystore" + "github.com/hivecommons/hive/crux/logscrub" + "github.com/hivecommons/hive/pkg/config" +) + +// Outcomes. These are the only values that appear in audit lines and counters. +const ( + OutcomeOK = "ok" + OutcomeNoCitedAnswer = "no_cited_answer" + OutcomeRefusedSecret = "refused_secret_shape" + OutcomeRefusedLength = "refused_length" + OutcomeRefusedEmpty = "refused_empty" + OutcomeRateLimited = "rate_limited" + OutcomeBudgetSpent = "budget_spent" + OutcomeBreakerOpen = "breaker_open" + OutcomeUpstreamError = "upstream_error" + OutcomeNotConsented = "not_consented" + breakerThreshold = 5 + breakerCooldown = time.Minute + rateLimitAuditWindow = 10 * time.Minute + maxThreadHandleChars = 64 + maxSectionIDChars = 64 + headerSectionLabelPrefix = "Dashboard section: " +) + +// Chatter is the vendor call. *Client implements it; tests pass a fake. +type Chatter interface { + Chat(ctx context.Context, query, threadID string) (ChatResponse, error) +} + +// Request is one question from one signed-in, consented person. +type Request struct { + // Person is the stem other audit lines use for the person. It is never + // sent to the vendor. + Person string + // Hive is the relaying hive's id for a hub relay request, "" otherwise. It + // is never sent to the vendor. + Hive string + // Question is what the person typed. + Question string + // Section is a dashboard section id; only ids in the closed vocabulary + // are used. + Section string + // Thread is the opaque handle returned by an earlier answer, or "". + Thread string +} + +// Result is the outcome of one question. It carries no vendor identifier. +type Result struct { + // Outcome is one of the Outcome constants. + Outcome string `json:"outcome"` + // Answer is plain text, set only when Outcome is OutcomeOK. + Answer string `json:"answer,omitempty"` + // Sources are the allowlisted citations, at least one when Outcome is OK. + Sources []Source `json:"sources,omitempty"` + // Uncertain is the vendor's own uncertainty flag. + Uncertain bool `json:"uncertain,omitempty"` + // Thread is the handle to send with the next question. + Thread string `json:"thread,omitempty"` + // Message is a short fixed explanation for a non-OK outcome. + Message string `json:"message,omitempty"` + // RetryAfterSeconds is set for rate_limited and budget_spent. + RetryAfterSeconds int `json:"retry_after_seconds,omitempty"` +} + +var outcomeMessages = map[string]string{ + OutcomeNoCitedAnswer: "No cited answer found in the docs.", + OutcomeRefusedSecret: "That looks like it contains a credential. Remove it and ask again.", + OutcomeRefusedLength: "That question is too long. Shorten it and ask again.", + OutcomeRefusedEmpty: "Type a question first.", + OutcomeRateLimited: "Ask is busy. Try again shortly.", + OutcomeBudgetSpent: "Ask has reached its daily limit. Use the docs link for now.", + OutcomeBreakerOpen: "Ask is unavailable right now. Use the docs link.", + OutcomeUpstreamError: "Ask is unavailable right now. Use the docs link.", + OutcomeNotConsented: "Confirm the disclosure before asking.", +} + +// Service answers questions: it validates, limits, calls the vendor and +// filters what comes back. +type Service struct { + settings config.AskSettings + chat Chatter + lim *Limiter + br *Breaker + threads *threads + logger *slog.Logger + + mu sync.Mutex + counts map[string]int64 + lastLimitLog map[string]time.Time + now func() time.Time +} + +// Options are test seams. The zero value is production behaviour. +type Options struct { + Now func() time.Time + // BaseURL overrides the vendor host (tests point it at a fake server). + BaseURL string + // Chatter replaces the vendor client entirely. + Chatter Chatter + // ThreadTTL overrides DefaultThreadTTL. + ThreadTTL time.Duration +} + +// ErrNotActive is returned by Open when the settings are not active. +var ErrNotActive = errors.New("ask: not active") + +// ErrKeyUnreadable is returned by Open when the key file cannot be used. It +// names no path and no value. +var ErrKeyUnreadable = errors.New("ask: api key file is unreadable or empty") + +// Open builds a Service from resolved settings, reading the key file. It +// returns ErrNotActive for inactive settings and ErrKeyUnreadable when the key +// cannot be loaded; in both cases the feature stays off and the caller logs one +// warning with the settings' Reason, never a value. +func Open(settings config.AskSettings, logger *slog.Logger, opts Options) (*Service, error) { + if !settings.Active { + return nil, ErrNotActive + } + chat := opts.Chatter + if chat == nil { + key, err := loadKey(settings.APIKeyFile) + if err != nil { + return nil, err + } + chat = NewClient(opts.BaseURL, settings.ProjectID, settings.IntegrationID, key, + time.Duration(settings.Limits.TimeoutSeconds)*time.Second) + } + if logger == nil { + logger = slog.Default() + } + s := &Service{ + settings: settings, + chat: chat, + lim: NewLimiter(settings.Limits, opts.Now), + br: NewBreaker(breakerThreshold, breakerCooldown, opts.Now), + threads: newThreads(opts.ThreadTTL, opts.Now), + logger: logger, + counts: map[string]int64{}, + lastLimitLog: map[string]time.Time{}, + now: opts.Now, + } + if s.now == nil { + s.now = time.Now + } + s.br.OnOpen = func() { logger.Warn("audit: ask circuit breaker opened") } + s.br.OnClose = func() { logger.Info("audit: ask circuit breaker closed") } + return s, nil +} + +func loadKey(path string) (string, error) { + if !config.SecretFilePathAllowed(path) { + return "", ErrKeyUnreadable + } + b, err := keystore.ReadFile(context.Background(), path) + if err != nil { + return "", ErrKeyUnreadable + } + key := strings.TrimSpace(string(b)) + if key == "" { + return "", ErrKeyUnreadable + } + return key, nil +} + +// Limits returns the resolved limits. It is the only part of the settings the +// service exposes: the vendor values stay private to the client. +func (s *Service) Limits() config.AskLimits { return s.settings.Limits } + +// Counters returns requests by outcome since start, plus the number admitted +// to the vendor today. +func (s *Service) Counters() (byOutcome map[string]int64, today int) { + s.mu.Lock() + out := make(map[string]int64, len(s.counts)) + for k, v := range s.counts { + out[k] = v + } + s.mu.Unlock() + return out, s.lim.DailyCount() +} + +// Forget drops the person's conversation handles (consent withdrawn). +func (s *Service) Forget(person string) { s.threads.forget(person) } + +// RecordNotConsented counts and audits a request refused before reaching +// Ask because the person has no current consent. +func (s *Service) RecordNotConsented(person, hive string, questionLen int) Result { + return s.finish(person, hive, Result{Outcome: OutcomeNotConsented}, questionLen, 0, time.Time{}) +} + +// Ask answers one question. +func (s *Service) Ask(ctx context.Context, req Request) Result { + start := s.now() + qlen := utf8.RuneCountInString(req.Question) + q := strings.TrimSpace(req.Question) + switch { + case q == "": + return s.finish(req.Person, req.Hive, Result{Outcome: OutcomeRefusedEmpty}, qlen, 0, start) + case utf8.RuneCountInString(q) > s.settings.Limits.QuestionChars: + return s.finish(req.Person, req.Hive, Result{Outcome: OutcomeRefusedLength}, qlen, 0, start) + } + if _, found := logscrub.FindSecret(q); found { + return s.finish(req.Person, req.Hive, Result{Outcome: OutcomeRefusedSecret}, qlen, 0, start) + } + + release, denial, wait := s.lim.Acquire(req.Person, req.Hive) + if denial != DenyNone { + r := Result{Outcome: OutcomeRateLimited, RetryAfterSeconds: ceilSeconds(wait)} + if denial == DenyBudget { + r.Outcome = OutcomeBudgetSpent + } + return s.finish(req.Person, req.Hive, r, qlen, 0, start) + } + defer release() + if !s.br.Allow() { + return s.finish(req.Person, req.Hive, Result{Outcome: OutcomeBreakerOpen}, qlen, 0, start) + } + + query := q + if s.settings.SendSectionLabel && len(req.Section) <= maxSectionIDChars { + if label := SectionLabel(req.Section); label != "" { + query = headerSectionLabelPrefix + label + "\n\n" + q + } + } + handle := req.Thread + if len(handle) > maxThreadHandleChars { + handle = "" + } + vendorThread := s.threads.lookup(handle, req.Person, req.Hive) + + callCtx, cancel := context.WithTimeout(ctx, time.Duration(s.settings.Limits.TimeoutSeconds)*time.Second) + defer cancel() + resp, err := s.chat.Chat(callCtx, query, vendorThread) + if err != nil { + s.br.Failure() + r := Result{Outcome: OutcomeUpstreamError} + if errors.Is(err, ErrRateLimited) { + r = Result{Outcome: OutcomeRateLimited, RetryAfterSeconds: 30} + } + return s.finish(req.Person, req.Hive, r, qlen, 0, start) + } + s.br.Success() + + sources := FilterSources(resp.RelevantSources) + answer := PlainText(resp.Answer, s.settings.Limits.AnswerChars) + if len(sources) == 0 || answer == "" { + return s.finish(req.Person, req.Hive, Result{Outcome: OutcomeNoCitedAnswer}, qlen, 0, start) + } + out := Result{Outcome: OutcomeOK, Answer: answer, Sources: sources, Uncertain: resp.IsUncertain} + if resp.ThreadID != "" { + out.Thread = s.threads.store(handle, req.Person, req.Hive, resp.ThreadID) + } + return s.finish(req.Person, req.Hive, out, qlen, len(sources), start) +} + +func ceilSeconds(d time.Duration) int { + n := int((d + time.Second - 1) / time.Second) + if n < 1 { + n = 1 + } + return n +} + +// finish counts and audits one request and fills the fixed message. The audit +// line carries the outcome and counts only: never the question, the answer, the +// sources, a thread id, the key, the project id or the integration id. +func (s *Service) finish(person, hive string, r Result, qlen, nsources int, start time.Time) Result { + if r.Message == "" { + r.Message = outcomeMessages[r.Outcome] + } + now := s.now() + s.mu.Lock() + s.counts[r.Outcome]++ + logIt := true + if r.Outcome == OutcomeRateLimited || r.Outcome == OutcomeBudgetSpent { + // One audit line per person per window, not one per request. + key := r.Outcome + "|" + person + if last, ok := s.lastLimitLog[key]; ok && now.Sub(last) < rateLimitAuditWindow { + logIt = false + } else { + s.lastLimitLog[key] = now + if len(s.lastLimitLog) > 4096 { + s.lastLimitLog = map[string]time.Time{key: now} + } + } + } + s.mu.Unlock() + if logIt { + var latency int64 + if !start.IsZero() { + latency = now.Sub(start).Milliseconds() + } + attrs := []any{"person", person, "outcome", r.Outcome, "question_len", qlen, + "sources", nsources, "latency_ms", latency} + if hive != "" { + attrs = append(attrs, "hive", hive) + } + s.logger.Info("audit: ask request", attrs...) + } + return r +} diff --git a/src/pkg/ask/service_test.go b/src/pkg/ask/service_test.go new file mode 100644 index 0000000000..31b31c0fda --- /dev/null +++ b/src/pkg/ask/service_test.go @@ -0,0 +1,531 @@ +package ask + +import ( + "bytes" + "context" + "errors" + "log/slog" + "net/http" + "net/http/httptest" + "os" + "path/filepath" + "strings" + "sync" + "testing" + "time" + + cruxconfig "github.com/hivecommons/hive/crux/config" + "github.com/hivecommons/hive/pkg/config" +) + +type fakeChat struct { + mu sync.Mutex + queries []string + threads []string + resp ChatResponse + err error + block chan struct{} +} + +func (f *fakeChat) Chat(ctx context.Context, query, threadID string) (ChatResponse, error) { + f.mu.Lock() + f.queries = append(f.queries, query) + f.threads = append(f.threads, threadID) + f.mu.Unlock() + if f.block != nil { + <-f.block + } + return f.resp, f.err +} + +func (f *fakeChat) calls() int { + f.mu.Lock() + defer f.mu.Unlock() + return len(f.queries) +} + +func goodResponse() ChatResponse { + return ChatResponse{ + Answer: "Open Settings, then Features.", ThreadID: "vendor-thread-1", IsUncertain: false, + RelevantSources: []VendorSource{{SourceURL: "https://docs.hivecommons.dev/docs/dashboard", Title: "Dashboard"}}, + } +} + +type clock struct { + mu sync.Mutex + t time.Time +} + +func (c *clock) now() time.Time { c.mu.Lock(); defer c.mu.Unlock(); return c.t } +func (c *clock) advance(d time.Duration) { + c.mu.Lock() + c.t = c.t.Add(d) + c.mu.Unlock() +} + +func newClock() *clock { return &clock{t: time.Date(2026, 11, 19, 10, 0, 0, 0, time.UTC)} } + +func testSettings() config.AskSettings { + return config.AskSettings{ + Active: true, Mode: config.AskModeDirect, SendSectionLabel: true, + Limits: config.AskLimits{ + PersonBurst: 3, PersonPerHour: 30, HivePerHour: 120, GlobalPerMinute: 8, + InFlight: 4, Daily: 500, QuestionChars: 200, AnswerChars: 60, TimeoutSeconds: 5, + }, + } +} + +func newService(t *testing.T, st config.AskSettings, f *fakeChat, c *clock) (*Service, *bytes.Buffer) { + t.Helper() + var logs bytes.Buffer + logger := slog.New(slog.NewTextHandler(&logs, nil)) + var now func() time.Time + if c != nil { + now = c.now + } + svc, err := Open(st, logger, Options{Chatter: f, Now: now}) + if err != nil { + t.Fatal(err) + } + return svc, &logs +} + +func TestAskReturnsCitedAnswer(t *testing.T) { + f := &fakeChat{resp: goodResponse()} + svc, _ := newService(t, testSettings(), f, nil) + res := svc.Ask(context.Background(), Request{Person: "alice", Question: "where is Features?"}) + if res.Outcome != OutcomeOK || res.Answer == "" || len(res.Sources) != 1 || res.Thread == "" || res.Message != "" { + t.Fatalf("result = %+v", res) + } + if res.Thread == "vendor-thread-1" { + t.Fatal("the vendor thread id must not reach the browser") + } +} + +func TestOnlyTheQuestionAndSectionLabelAreSent(t *testing.T) { + f := &fakeChat{resp: goodResponse()} + svc, _ := newService(t, testSettings(), f, nil) + // Seeded identifiers that exist in the process but must never be sent. + person, hive := "secret-login-bob", "secret-hive-name" + svc.Ask(context.Background(), Request{ + Person: person, Hive: hive, Question: "where is the toggle?", Section: "settings-features", + }) + want := "Dashboard section: Settings, Features\n\nwhere is the toggle?" + if len(f.queries) != 1 || f.queries[0] != want { + t.Fatalf("query = %q, want %q", f.queries, want) + } + for _, seeded := range []string{person, hive} { + if strings.Contains(f.queries[0], seeded) { + t.Fatalf("query carries %q", seeded) + } + } +} + +func TestSectionLabelDroppedWhenUnknownOrSwitchedOff(t *testing.T) { + f := &fakeChat{resp: goodResponse()} + svc, _ := newService(t, testSettings(), f, nil) + svc.Ask(context.Background(), Request{Person: "a", Question: "q1", Section: "not a section; repo=private/repo"}) + st := testSettings() + st.SendSectionLabel = false + svc2, _ := newService(t, st, f, nil) + svc2.Ask(context.Background(), Request{Person: "a", Question: "q2", Section: "settings-features"}) + if f.queries[0] != "q1" || f.queries[1] != "q2" { + t.Fatalf("queries = %q", f.queries) + } +} + +func TestCredentialShapedInputIsRefusedAndNeverSent(t *testing.T) { + f := &fakeChat{resp: goodResponse()} + svc, logs := newService(t, testSettings(), f, nil) + token := "ghp_" + strings.Repeat("a1B2c3D4e5", 4) + res := svc.Ask(context.Background(), Request{Person: "alice", Question: "why does " + token + " fail?"}) + if res.Outcome != OutcomeRefusedSecret || res.Message == "" || res.Answer != "" { + t.Fatalf("result = %+v", res) + } + if f.calls() != 0 { + t.Fatal("a credential-shaped question reached the vendor") + } + if strings.Contains(logs.String(), token) || strings.Contains(res.Message, token) { + t.Fatal("the credential was echoed") + } +} + +func TestLengthAndEmptyCaps(t *testing.T) { + f := &fakeChat{resp: goodResponse()} + svc, _ := newService(t, testSettings(), f, nil) + if r := svc.Ask(context.Background(), Request{Person: "a", Question: " "}); r.Outcome != OutcomeRefusedEmpty { + t.Fatalf("empty: %+v", r) + } + if r := svc.Ask(context.Background(), Request{Person: "a", Question: strings.Repeat("é", 201)}); r.Outcome != OutcomeRefusedLength { + t.Fatalf("long: %+v", r) + } + if r := svc.Ask(context.Background(), Request{Person: "a", Question: strings.Repeat("é", 200)}); r.Outcome != OutcomeOK { + t.Fatalf("at the cap: %+v", r) + } + if f.calls() != 1 { + t.Fatalf("vendor calls = %d", f.calls()) + } +} + +func TestNoAllowlistedSourceIsNoCitedAnswer(t *testing.T) { + f := &fakeChat{resp: ChatResponse{ + Answer: "Click here", ThreadID: "t", + RelevantSources: []VendorSource{{SourceURL: "https://evil.example/x", Title: "x"}}, + }} + svc, _ := newService(t, testSettings(), f, nil) + res := svc.Ask(context.Background(), Request{Person: "a", Question: "q"}) + if res.Outcome != OutcomeNoCitedAnswer || res.Answer != "" || len(res.Sources) != 0 || res.Thread != "" || res.Message == "" { + t.Fatalf("uncited text must not be shown as an answer: %+v", res) + } +} + +func TestAnswerIsPlainCappedText(t *testing.T) { + r := goodResponse() + r.Answer = "\x1b[31m" + strings.Repeat("x", 500) + svc, _ := newService(t, testSettings(), &fakeChat{resp: r}, nil) + res := svc.Ask(context.Background(), Request{Person: "a", Question: "q"}) + if res.Outcome != OutcomeOK || len([]rune(res.Answer)) > 60 || strings.ContainsRune(res.Answer, '\x1b') { + t.Fatalf("answer = %q", res.Answer) + } +} + +func TestThreadHandleIsBoundToThePerson(t *testing.T) { + f := &fakeChat{resp: goodResponse()} + svc, _ := newService(t, testSettings(), f, nil) + first := svc.Ask(context.Background(), Request{Person: "alice", Question: "one"}) + // Alice continues her thread. + second := svc.Ask(context.Background(), Request{Person: "alice", Question: "two", Thread: first.Thread}) + // Bob presents Alice's handle: it must start a new conversation. + third := svc.Ask(context.Background(), Request{Person: "bob", Question: "three", Thread: first.Thread}) + if f.threads[0] != "" || f.threads[1] != "vendor-thread-1" || f.threads[2] != "" { + t.Fatalf("vendor threads used = %q", f.threads) + } + if second.Thread != first.Thread { + t.Fatalf("a continued conversation keeps its handle: %q vs %q", second.Thread, first.Thread) + } + if third.Thread == first.Thread || third.Thread == "" { + t.Fatalf("bob must get his own handle, got %q", third.Thread) + } + // Withdrawal drops the conversation. + svc.Forget("alice") + svc.Ask(context.Background(), Request{Person: "alice", Question: "four", Thread: first.Thread}) + if f.threads[3] != "" { + t.Fatalf("thread survived Forget: %q", f.threads) + } +} + +func TestThreadHandleExpires(t *testing.T) { + clk := newClock() + f := &fakeChat{resp: goodResponse()} + svc, _ := newService(t, testSettings(), f, clk) + first := svc.Ask(context.Background(), Request{Person: "alice", Question: "one"}) + clk.advance(DefaultThreadTTL + time.Minute) + svc.Ask(context.Background(), Request{Person: "alice", Question: "two", Thread: first.Thread}) + if f.threads[1] != "" { + t.Fatalf("expired thread was used: %q", f.threads) + } +} + +func TestPersonLimits(t *testing.T) { + clk := newClock() + f := &fakeChat{resp: goodResponse()} + svc, _ := newService(t, testSettings(), f, clk) + ask := func(p string) Result { + return svc.Ask(context.Background(), Request{Person: p, Question: "q"}) + } + for i := 0; i < 3; i++ { + if r := ask("alice"); r.Outcome != OutcomeOK { + t.Fatalf("request %d: %+v", i, r) + } + } + r := ask("alice") + if r.Outcome != OutcomeRateLimited || r.RetryAfterSeconds < 1 || r.RetryAfterSeconds > 60 { + t.Fatalf("burst should be limited with a retry hint: %+v", r) + } + if ask("bob").Outcome != OutcomeOK { + t.Fatal("one person's burst limited another") + } + clk.advance(61 * time.Second) + if ask("alice").Outcome != OutcomeOK { + t.Fatal("burst window did not reopen") + } + // Hourly cap: 30 per hour, with the burst respected by spacing requests. + hourly := testSettings() + hourly.Limits.PersonPerHour = 4 + svc2, _ := newService(t, hourly, f, clk) + for i := 0; i < 4; i++ { + if r := svc2.Ask(context.Background(), Request{Person: "carol", Question: "q"}); r.Outcome != OutcomeOK { + t.Fatalf("hourly %d: %+v", i, r) + } + clk.advance(61 * time.Second) + } + if r := svc2.Ask(context.Background(), Request{Person: "carol", Question: "q"}); r.Outcome != OutcomeRateLimited { + t.Fatalf("hourly cap did not hold: %+v", r) + } +} + +func TestHiveGlobalAndDailyLimits(t *testing.T) { + clk := newClock() + f := &fakeChat{resp: goodResponse()} + st := testSettings() + st.Limits.HivePerHour = 2 + st.Limits.PersonBurst = 100 + st.Limits.GlobalPerMinute = 5 + svc, _ := newService(t, st, f, clk) + ask := func(person, hive string) Result { + return svc.Ask(context.Background(), Request{Person: person, Hive: hive, Question: "q"}) + } + if ask("a", "hive-1").Outcome != OutcomeOK || ask("b", "hive-1").Outcome != OutcomeOK { + t.Fatal("hive allowance") + } + if r := ask("c", "hive-1"); r.Outcome != OutcomeRateLimited { + t.Fatalf("one hive exceeded its allowance: %+v", r) + } + if ask("c", "hive-2").Outcome != OutcomeOK { + t.Fatal("one hive's limit spent another hive's allowance") + } + // Global: 3 used so far, 2 more allowed this minute. + if ask("d", "hive-3").Outcome != OutcomeOK || ask("e", "hive-4").Outcome != OutcomeOK { + t.Fatal("global allowance") + } + if r := ask("f", "hive-5"); r.Outcome != OutcomeRateLimited { + t.Fatalf("hub-wide per-minute cap did not hold: %+v", r) + } + + // Daily budget. + daily := testSettings() + daily.Limits.Daily = 2 + daily.Limits.PersonBurst = 100 + daily.Limits.GlobalPerMinute = 100 + svc2, _ := newService(t, daily, f, clk) + for i := 0; i < 2; i++ { + svc2.Ask(context.Background(), Request{Person: "a", Question: "q"}) + } + r := svc2.Ask(context.Background(), Request{Person: "a", Question: "q"}) + if r.Outcome != OutcomeBudgetSpent || r.RetryAfterSeconds < 1 { + t.Fatalf("daily budget: %+v", r) + } + clk.advance(24 * time.Hour) + if r := svc2.Ask(context.Background(), Request{Person: "a", Question: "q"}); r.Outcome != OutcomeOK { + t.Fatalf("budget did not reset: %+v", r) + } + if _, today := svc2.Counters(); today != 1 { + t.Fatalf("sent today = %d", today) + } +} + +func TestInFlightLimits(t *testing.T) { + f := &fakeChat{resp: goodResponse(), block: make(chan struct{})} + st := testSettings() + st.Limits.InFlight = 2 + svc, _ := newService(t, st, f, nil) + var wg sync.WaitGroup + start := func(person string) { + wg.Add(1) + go func() { + defer wg.Done() + svc.Ask(context.Background(), Request{Person: person, Question: "q"}) + }() + } + start("a") + start("b") + deadline := time.Now().Add(2 * time.Second) + for f.calls() < 2 && time.Now().Before(deadline) { + time.Sleep(5 * time.Millisecond) + } + if r := svc.Ask(context.Background(), Request{Person: "c", Question: "q"}); r.Outcome != OutcomeRateLimited { + t.Fatalf("hub-wide in-flight cap did not hold: %+v", r) + } + if r := svc.Ask(context.Background(), Request{Person: "a", Question: "again"}); r.Outcome != OutcomeRateLimited { + t.Fatalf("a second request from the same person while one is in flight: %+v", r) + } + close(f.block) + wg.Wait() + if r := svc.Ask(context.Background(), Request{Person: "c", Question: "q"}); r.Outcome != OutcomeOK { + t.Fatalf("slots were not released: %+v", r) + } +} + +func TestVendor429AndTimeoutGiveFallback(t *testing.T) { + f := &fakeChat{err: ErrRateLimited} + svc, _ := newService(t, testSettings(), f, nil) + r := svc.Ask(context.Background(), Request{Person: "a", Question: "q"}) + if r.Outcome != OutcomeRateLimited || r.RetryAfterSeconds < 1 || r.Message == "" || r.Answer != "" { + t.Fatalf("429: %+v", r) + } + f.err = ErrUnavailable + r = svc.Ask(context.Background(), Request{Person: "b", Question: "q"}) + if r.Outcome != OutcomeUpstreamError || r.Message == "" { + t.Fatalf("unavailable: %+v", r) + } +} + +func TestCircuitBreakerOpensAndCloses(t *testing.T) { + clk := newClock() + f := &fakeChat{err: ErrUnavailable} + st := testSettings() + st.Limits.PersonBurst = 1000 + st.Limits.PersonPerHour = 1000 + st.Limits.GlobalPerMinute = 1000 + svc, logs := newService(t, st, f, clk) + ask := func() Result { return svc.Ask(context.Background(), Request{Person: "a", Question: "q"}) } + for i := 0; i < breakerThreshold; i++ { + if r := ask(); r.Outcome != OutcomeUpstreamError { + t.Fatalf("failure %d: %+v", i, r) + } + } + before := f.calls() + if r := ask(); r.Outcome != OutcomeBreakerOpen { + t.Fatalf("breaker should be open: %+v", r) + } + if f.calls() != before { + t.Fatal("an open breaker still called the vendor") + } + if !strings.Contains(logs.String(), "circuit breaker opened") { + t.Fatal("no audit line when the breaker opened") + } + // After the cool-down one probe is let through; a failed probe re-opens. + clk.advance(breakerCooldown + time.Second) + if r := ask(); r.Outcome != OutcomeUpstreamError { + t.Fatalf("probe: %+v", r) + } + if r := ask(); r.Outcome != OutcomeBreakerOpen { + t.Fatalf("a failed probe must re-open: %+v", r) + } + // A successful probe closes it. + clk.advance(breakerCooldown + time.Second) + f.err, f.resp = nil, goodResponse() + if r := ask(); r.Outcome != OutcomeOK { + t.Fatalf("good probe: %+v", r) + } + if r := ask(); r.Outcome != OutcomeOK { + t.Fatalf("closed breaker: %+v", r) + } + if !strings.Contains(logs.String(), "circuit breaker closed") { + t.Fatal("no audit line when the breaker closed") + } +} + +func TestAuditLinesAndCountersCarryNoText(t *testing.T) { + f := &fakeChat{resp: goodResponse()} + svc, logs := newService(t, testSettings(), f, nil) + question := "unique-question-text-zzz" + svc.Ask(context.Background(), Request{Person: "alice", Hive: "hive-1", Question: question}) + svc.Ask(context.Background(), Request{Person: "alice", Question: "ghp_" + strings.Repeat("a1B2c3D4e5", 4)}) + svc.RecordNotConsented("alice", "", 12) + out := logs.String() + for _, banned := range []string{question, "Open Settings", "docs.hivecommons.dev", "vendor-thread-1", "ghp_"} { + if strings.Contains(out, banned) { + t.Fatalf("audit output carries %q:\n%s", banned, out) + } + } + for _, want := range []string{"outcome=ok", "outcome=refused_secret_shape", "outcome=not_consented", "question_len=", "sources=1", "person=alice", "hive=hive-1"} { + if !strings.Contains(out, want) { + t.Fatalf("audit output missing %q:\n%s", want, out) + } + } + counts, _ := svc.Counters() + if counts[OutcomeOK] != 1 || counts[OutcomeRefusedSecret] != 1 || counts[OutcomeNotConsented] != 1 { + t.Fatalf("counters = %v", counts) + } +} + +func TestRateLimitAuditIsOncePerWindow(t *testing.T) { + clk := newClock() + st := testSettings() + st.Limits.PersonBurst = 1 + svc, logs := newService(t, st, &fakeChat{resp: goodResponse()}, clk) + for i := 0; i < 6; i++ { + svc.Ask(context.Background(), Request{Person: "alice", Question: "q"}) + } + if n := strings.Count(logs.String(), "outcome=rate_limited"); n != 1 { + t.Fatalf("rate limit audit lines = %d, want 1", n) + } + if counts, _ := svc.Counters(); counts[OutcomeRateLimited] != 5 { + t.Fatalf("every limited request still counts: %v", counts) + } +} + +func TestOpenStaysOffWithoutAUsableKey(t *testing.T) { + st := testSettings() + if _, err := Open(config.AskSettings{}, nil, Options{}); !errors.Is(err, ErrNotActive) { + t.Fatalf("inactive settings: %v", err) + } + dir := t.TempDir() + defer cruxconfig.SetSecretFileRootsForTest(dir)() + st.ProjectID, st.IntegrationID = testProjectID, testIntegrationID + + st.APIKeyFile = filepath.Join(dir, "missing") + if _, err := Open(st, nil, Options{}); !errors.Is(err, ErrKeyUnreadable) || strings.Contains(err.Error(), dir) { + t.Fatalf("missing key file: %v", err) + } + empty := filepath.Join(dir, "empty") + if err := os.WriteFile(empty, []byte(" \n"), 0o600); err != nil { + t.Fatal(err) + } + st.APIKeyFile = empty + if _, err := Open(st, nil, Options{}); !errors.Is(err, ErrKeyUnreadable) { + t.Fatalf("empty key file: %v", err) + } + st.APIKeyFile = filepath.Join(t.TempDir(), "outside") // not under the allowed root + if err := os.WriteFile(st.APIKeyFile, []byte(testAPIKey), 0o600); err != nil { + t.Fatal(err) + } + if _, err := Open(st, nil, Options{}); !errors.Is(err, ErrKeyUnreadable) { + t.Fatalf("key outside the allowed directory must not be read: %v", err) + } +} + +// TestEndToEndAgainstFakeVendorHidesSeededValues drives the real client against +// a fake vendor with seeded secrets, repo names, hive names and logins in the +// surrounding process, and checks none reaches the wire, a response or a log. +func TestEndToEndAgainstFakeVendorHidesSeededValues(t *testing.T) { + dir := t.TempDir() + defer cruxconfig.SetSecretFileRootsForTest(dir)() + keyFile := filepath.Join(dir, "kapa-key") + if err := os.WriteFile(keyFile, []byte(testAPIKey+"\n"), 0o600); err != nil { + t.Fatal(err) + } + var seen []recorded + vendor := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) { + var b bytes.Buffer + _, _ = b.ReadFrom(r.Body) + seen = append(seen, recorded{r.URL.Path, r.Method, r.Header.Get("X-API-KEY"), "", b.Bytes()}) + _, _ = w.Write([]byte(contractResponse)) + })) + defer vendor.Close() + + st := testSettings() + st.ProjectID, st.IntegrationID, st.APIKeyFile = testProjectID, testIntegrationID, keyFile + var logs bytes.Buffer + svc, err := Open(st, slog.New(slog.NewTextHandler(&logs, nil)), Options{BaseURL: vendor.URL}) + if err != nil { + t.Fatal(err) + } + seeded := []string{"acme/private-repo", "hive-prod-eu", "login-of-bob", "ghs_" + strings.Repeat("Z9y8X7w6V5", 3)} + res := svc.Ask(context.Background(), Request{ + Person: "login-of-bob", Hive: "hive-prod-eu", Question: "how do I add a repo?", Section: "settings-repos", + }) + if res.Outcome != OutcomeOK { + t.Fatalf("result = %+v", res) + } + if len(seen) != 1 || seen[0].key != testAPIKey { + t.Fatalf("vendor saw %+v", seen) + } + wire := string(seen[0].body) + if !strings.Contains(wire, "how do I add a repo?") || !strings.Contains(wire, "Settings, Repositories") { + t.Fatalf("wire body missing the question or label: %s", wire) + } + for _, v := range seeded { + if strings.Contains(wire, v) { + t.Fatalf("wire body carries %q: %s", v, wire) + } + } + // Nothing credential-like in a response or the logs. + out := logs.String() + res.Message + res.Answer + res.Thread + for _, v := range []string{testAPIKey, testProjectID, testIntegrationID, vendor.URL} { + if strings.Contains(out, v) { + t.Fatalf("output carries %q", v) + } + } +} diff --git a/src/pkg/ask/sources.go b/src/pkg/ask/sources.go new file mode 100644 index 0000000000..3b93804309 --- /dev/null +++ b/src/pkg/ask/sources.go @@ -0,0 +1,93 @@ +package ask + +import ( + "net/url" + "strings" + "unicode" +) + +const ( + // DocsHost is the published docs site. + DocsHost = "docs.hivecommons.dev" + // githubHost and githubDocsPrefix name the one GitHub path help links + // already point at (dashboardDocsHref): the docs under hivecommons/hive. + githubHost = "github.com" + + maxSources = 5 + maxTitleChars = 120 + maxSourceChars = 512 +) + +// AllowedSourceURL reports whether raw is a link an answer may cite: https, +// no credentials, no explicit port, and either the docs site host or a docs +// page under hivecommons/hive on GitHub. Anything else, including a lookalike +// host, a javascript: or data: URL, or a path with a dot segment, is refused. +func AllowedSourceURL(raw string) bool { + if raw == "" || len(raw) > maxSourceChars || strings.ContainsFunc(raw, unicode.IsControl) { + return false + } + u, err := url.Parse(raw) + if err != nil || u.Scheme != "https" || u.User != nil || u.Opaque != "" || u.Port() != "" { + return false + } + for _, seg := range strings.Split(u.Path, "/") { + if seg == ".." || seg == "." { + return false + } + } + switch strings.ToLower(u.Hostname()) { + case DocsHost: + return true + case githubHost: + // /hivecommons/hive/(blob|tree)//src/docs/... + parts := strings.Split(strings.TrimPrefix(u.Path, "/"), "/") + return len(parts) >= 6 && + parts[0] == "hivecommons" && parts[1] == "hive" && + (parts[2] == "blob" || parts[2] == "tree") && + parts[3] != "" && parts[4] == "src" && parts[5] == "docs" + } + return false +} + +// FilterSources keeps the allowlisted, de-duplicated sources of a vendor +// response, drops any marked as internal data, caps the count and cleans each +// title into plain text. +func FilterSources(in []VendorSource) []Source { + var out []Source + seen := map[string]bool{} + for _, v := range in { + if v.ContainsInternalData || !AllowedSourceURL(v.SourceURL) || seen[v.SourceURL] { + continue + } + seen[v.SourceURL] = true + title := PlainText(v.Title, maxTitleChars) + if title == "" { + title = v.SourceURL + } + out = append(out, Source{URL: v.SourceURL, Title: title}) + if len(out) == maxSources { + break + } + } + return out +} + +// PlainText removes control characters (keeping newline and tab), collapses +// nothing else, and caps the result at max runes. The answer is returned as a +// JSON string and rendered as text by the panel; this is defence in depth, not +// HTML sanitising. +func PlainText(s string, max int) string { + var b strings.Builder + n := 0 + for _, r := range strings.TrimSpace(s) { + if r == '�' || (unicode.IsControl(r) && r != '\n' && r != '\t') { + continue + } + if n >= max { + break + } + b.WriteRune(r) + n++ + } + return strings.TrimSpace(b.String()) +} diff --git a/src/pkg/ask/sources_test.go b/src/pkg/ask/sources_test.go new file mode 100644 index 0000000000..7796e4123b --- /dev/null +++ b/src/pkg/ask/sources_test.go @@ -0,0 +1,91 @@ +package ask + +import ( + "strings" + "testing" +) + +func TestAllowedSourceURL(t *testing.T) { + good := []string{ + "https://docs.hivecommons.dev/docs/dashboard-sections", + "https://DOCS.hivecommons.dev/docs/x#anchor", + "https://github.com/hivecommons/hive/blob/v5/src/docs/dashboard-sections.md#knowledge", + "https://github.com/hivecommons/hive/tree/v5/src/docs/design", + } + bad := []string{ + "", + "http://docs.hivecommons.dev/docs/x", + "javascript:alert(1)", + "data:text/html,", + "//docs.hivecommons.dev/x", + "https://docs.hivecommons.dev.evil.example/x", + "https://evil.example/docs.hivecommons.dev", + "https://user:pw@docs.hivecommons.dev/x", + "https://docs.hivecommons.dev:8443/x", + "https://docs.hivecommons.dev/a/../../x", + "https://docs.hivecommons.dev/x\nSet-Cookie: a=b", + "https://github.com/hivecommons/hive/issues/1", + "https://github.com/hivecommons/hive/blob/v5/src/pkg/hub/ask.go", + "https://github.com/hivecommons/other/blob/v5/src/docs/x.md", + "https://github.com/evil/hivecommons/hive/blob/v5/src/docs/x.md", + "https://github.com/hivecommons/hive/blob/v5/src/docs", + "https://docs.hivecommons.dev/" + strings.Repeat("a", 600), + } + for _, u := range good { + if !AllowedSourceURL(u) { + t.Errorf("should allow %q", u) + } + } + for _, u := range bad { + if AllowedSourceURL(u) { + t.Errorf("should refuse %q", u) + } + } +} + +func TestFilterSources(t *testing.T) { + in := []VendorSource{ + {SourceURL: "https://evil.example/x", Title: "Evil"}, + {SourceURL: "https://docs.hivecommons.dev/docs/a", Title: "A\x07\x1b[31m"}, + {SourceURL: "https://docs.hivecommons.dev/docs/a", Title: "duplicate"}, + {SourceURL: "https://docs.hivecommons.dev/docs/internal", Title: "Internal", ContainsInternalData: true}, + {SourceURL: "https://docs.hivecommons.dev/docs/b", Title: ""}, + } + got := FilterSources(in) + if len(got) != 2 { + t.Fatalf("got %+v", got) + } + if strings.ContainsAny(got[0].Title, "\x07\x1b") { + t.Fatalf("control characters survived: %q", got[0].Title) + } + if got[1].Title != "https://docs.hivecommons.dev/docs/b" { + t.Fatalf("empty title should fall back to the URL: %+v", got[1]) + } + var many []VendorSource + for i := 0; i < 20; i++ { + many = append(many, VendorSource{SourceURL: "https://docs.hivecommons.dev/docs/p" + string(rune('a'+i)), Title: "t"}) + } + if n := len(FilterSources(many)); n != maxSources { + t.Fatalf("source cap = %d", n) + } +} + +func TestPlainTextCapsAndCleans(t *testing.T) { + if got := PlainText(" a\x00b\x1bc\n\td ", 100); got != "abc\n\td" { + t.Fatalf("got %q", got) + } + if got := PlainText("héllo wörld", 5); got != "héllo" { + t.Fatalf("rune-safe cap: %q", got) + } +} + +func TestSectionLabelIsAClosedVocabulary(t *testing.T) { + if SectionLabel("settings-features") != "Settings, Features" { + t.Fatal("known id lost its label") + } + for _, id := range []string{"", "https://example.com", "ghp_x", "Settings, Features", "customers-section/alice"} { + if SectionLabel(id) != "" { + t.Fatalf("%q must not resolve", id) + } + } +} diff --git a/src/pkg/ask/threads.go b/src/pkg/ask/threads.go new file mode 100644 index 0000000000..bbe161703f --- /dev/null +++ b/src/pkg/ask/threads.go @@ -0,0 +1,105 @@ +package ask + +import ( + "crypto/rand" + "encoding/hex" + "sync" + "time" +) + +const ( + // DefaultThreadTTL is how long an idle conversation handle stays valid. + DefaultThreadTTL = 30 * time.Minute + maxThreads = 10000 +) + +type threadEntry struct { + person string + hive string + vendor string + expires time.Time +} + +// threads maps an opaque handle the browser holds to the vendor's thread id, +// which never leaves the server. A handle only resolves for the person (and +// hive) it was issued to, and expires. In memory only: closing the panel or a +// restart drops the conversation, as the design record says. +type threads struct { + mu sync.Mutex + now func() time.Time + ttl time.Duration + m map[string]threadEntry +} + +func newThreads(ttl time.Duration, now func() time.Time) *threads { + if now == nil { + now = time.Now + } + if ttl <= 0 { + ttl = DefaultThreadTTL + } + return &threads{now: now, ttl: ttl, m: map[string]threadEntry{}} +} + +// lookup returns the vendor thread id for handle, or "" when the handle is +// unknown, expired or belongs to someone else. +func (t *threads) lookup(handle, person, hive string) string { + if handle == "" { + return "" + } + t.mu.Lock() + defer t.mu.Unlock() + e, ok := t.m[handle] + if !ok { + return "" + } + if !t.now().Before(e.expires) { + delete(t.m, handle) + return "" + } + if e.person != person || e.hive != hive { + return "" + } + return e.vendor +} + +// store records vendor under handle (a fresh handle when handle does not +// resolve for this person) and returns the handle to give the browser. +func (t *threads) store(handle, person, hive, vendor string) string { + t.mu.Lock() + defer t.mu.Unlock() + now := t.now() + if e, ok := t.m[handle]; !ok || e.person != person || e.hive != hive { + handle = "" + } + if handle == "" { + if len(t.m) >= maxThreads { + for k, e := range t.m { + if !now.Before(e.expires) { + delete(t.m, k) + } + } + if len(t.m) >= maxThreads { + return "" + } + } + var b [16]byte + if _, err := rand.Read(b[:]); err != nil { + return "" + } + handle = hex.EncodeToString(b[:]) + } + t.m[handle] = threadEntry{person: person, hive: hive, vendor: vendor, expires: now.Add(t.ttl)} + return handle +} + +// forget drops every handle held for person (consent withdrawn, user deleted). +func (t *threads) forget(person string) { + t.mu.Lock() + defer t.mu.Unlock() + for k, e := range t.m { + if e.person == person { + delete(t.m, k) + } + } +} diff --git a/src/pkg/config/ask.go b/src/pkg/config/ask.go new file mode 100644 index 0000000000..4b7f229c46 --- /dev/null +++ b/src/pkg/config/ask.go @@ -0,0 +1,243 @@ +package config + +import ( + "os" + "strconv" + "strings" +) + +// Ask AI help (#19177, phase 1 #19180). Off by default and inert: nothing in +// this file opens a route, makes a request or changes a header unless the +// operator switches it on AND supplies every value the vendor call needs. +// See src/docs/design/dashboard-ask-ai-help.md. +const ( + // EnvAskEnabled overrides ask.enabled when it holds a recognised boolean. + EnvAskEnabled = "HIVE_ASK_ENABLED" + // EnvAskMode overrides ask.mode (off, direct or hub). + EnvAskMode = "HIVE_ASK_MODE" + // EnvAskKapaProjectID overrides ask.kapa.project_id. + EnvAskKapaProjectID = "HIVE_ASK_KAPA_PROJECT_ID" + // EnvAskKapaIntegrationID overrides ask.kapa.integration_id. + EnvAskKapaIntegrationID = "HIVE_ASK_KAPA_INTEGRATION_ID" + // EnvAskKapaAPIKeyFile overrides ask.kapa.api_key_file. There is no env + // var that holds the key itself, by design. + EnvAskKapaAPIKeyFile = "HIVE_ASK_KAPA_API_KEY_FILE" + // EnvAskSendSectionLabel overrides ask.send_section_label. + EnvAskSendSectionLabel = "HIVE_ASK_SEND_SECTION_LABEL" + + EnvAskLimitPersonBurst = "HIVE_ASK_LIMIT_PERSON_BURST" + EnvAskLimitPersonPerHour = "HIVE_ASK_LIMIT_PERSON_PER_HOUR" + EnvAskLimitHivePerHour = "HIVE_ASK_LIMIT_HIVE_PER_HOUR" + EnvAskLimitGlobalPerMinute = "HIVE_ASK_LIMIT_GLOBAL_PER_MINUTE" + EnvAskLimitInFlight = "HIVE_ASK_LIMIT_IN_FLIGHT" + EnvAskLimitDaily = "HIVE_ASK_LIMIT_DAILY" + EnvAskLimitQuestionChars = "HIVE_ASK_LIMIT_QUESTION_CHARS" + EnvAskLimitAnswerChars = "HIVE_ASK_LIMIT_ANSWER_CHARS" + EnvAskLimitTimeoutSeconds = "HIVE_ASK_LIMIT_TIMEOUT_SECONDS" +) + +// Ask modes. +const ( + AskModeOff = "off" + AskModeDirect = "direct" + AskModeHub = "hub" +) + +// Default Ask limits. Every number is a proposal from the design record, not +// a measurement; each is overridable by config and env. The hub-wide numbers +// sit well under the vendor's 20-per-minute per-team chat limit, which the +// docs site's own widget shares. +const ( + DefaultAskPersonBurst = 3 + DefaultAskPersonPerHour = 30 + DefaultAskHivePerHour = 120 + DefaultAskGlobalPerMinute = 8 + DefaultAskInFlight = 4 + DefaultAskDaily = 500 + DefaultAskQuestionChars = 1000 + DefaultAskAnswerChars = 4000 + DefaultAskTimeoutSeconds = 30 +) + +// AskConfig (top-level `ask:`) is the optional Ask AI help block. Zero value = +// off. +type AskConfig struct { + // Enabled is the switch. nil or false = off: no route, no request. + Enabled *bool `yaml:"enabled,omitempty" json:"enabled,omitempty"` + // Mode is "off", "direct" (this process calls the vendor with its own + // key) or "hub" (a hive calls the hub relay, a later phase). Empty means + // the process default: direct on the hub. + Mode string `yaml:"mode,omitempty" json:"mode,omitempty"` + // Kapa holds the vendor project values. None is ever returned by an API. + Kapa AskKapaConfig `yaml:"kapa,omitempty" json:"kapa,omitempty"` + // SendSectionLabel includes the closed-vocabulary dashboard section label + // in the question. nil = true. + SendSectionLabel *bool `yaml:"send_section_label,omitempty" json:"send_section_label,omitempty"` + // Limits tunes the named limits. Zero = the default. + Limits AskLimitsConfig `yaml:"limits,omitempty" json:"limits,omitempty"` +} + +// AskKapaConfig is the vendor block. The project and integration ids are +// configuration, never in the repo, never logged. Their JSON tags are "-" so +// no JSON API response built from Config can carry them; the config export +// path redacts them from the YAML form (see pkg/dashboard). +type AskKapaConfig struct { + ProjectID string `yaml:"project_id,omitempty" json:"-"` + IntegrationID string `yaml:"integration_id,omitempty" json:"-"` + // APIKeyFile is a path under the allowed secret directory, mounted from + // a Secret. There is no inline key and no env var holding the key. + APIKeyFile string `yaml:"api_key_file,omitempty" json:"-"` +} + +// AskLimitsConfig holds optional overrides for the Ask limits. +type AskLimitsConfig struct { + PersonBurst int `yaml:"person_burst,omitempty" json:"person_burst,omitempty"` + PersonPerHour int `yaml:"person_per_hour,omitempty" json:"person_per_hour,omitempty"` + HivePerHour int `yaml:"hive_per_hour,omitempty" json:"hive_per_hour,omitempty"` + GlobalPerMinute int `yaml:"global_per_minute,omitempty" json:"global_per_minute,omitempty"` + InFlight int `yaml:"in_flight,omitempty" json:"in_flight,omitempty"` + Daily int `yaml:"daily,omitempty" json:"daily,omitempty"` + QuestionChars int `yaml:"question_chars,omitempty" json:"question_chars,omitempty"` + AnswerChars int `yaml:"answer_chars,omitempty" json:"answer_chars,omitempty"` + TimeoutSeconds int `yaml:"timeout_seconds,omitempty" json:"timeout_seconds,omitempty"` +} + +// AskLimits is the resolved set of limits. +type AskLimits struct { + PersonBurst int + PersonPerHour int + HivePerHour int + GlobalPerMinute int + InFlight int + Daily int + QuestionChars int + AnswerChars int + TimeoutSeconds int +} + +// AskSettings is the resolved Ask configuration for one process. When Active +// is false nothing else is meaningful except Reason, which names why the +// feature is off without carrying any configured value. +type AskSettings struct { + // Active is true only when the switch is on and every required value is + // present and usable. + Active bool + // Requested is true when the operator switched the feature on, whether or + // not it could start. Callers log one warning when Requested && !Active. + Requested bool + // Reason is a log-safe explanation, with no configured value in it. + Reason string + + Mode string + ProjectID string + IntegrationID string + APIKeyFile string + SendSectionLabel bool + Limits AskLimits +} + +// ResolveAskSettings merges ask: with the env overrides (env wins) and checks +// the values a vendor call needs. isHub is true for the hub process, whose +// only valid mode is direct. cfg may be nil (env only). The key file is +// checked for placement under the allowed secret directory; whether it is +// readable is checked by the caller that loads the key. +func ResolveAskSettings(cfg *Config, isHub bool) AskSettings { + var ask AskConfig + if cfg != nil { + ask = cfg.Ask + } + s := AskSettings{ + Mode: strings.ToLower(strings.TrimSpace(ask.Mode)), + ProjectID: strings.TrimSpace(ask.Kapa.ProjectID), + IntegrationID: strings.TrimSpace(ask.Kapa.IntegrationID), + APIKeyFile: strings.TrimSpace(ask.Kapa.APIKeyFile), + SendSectionLabel: ask.SendSectionLabel == nil || *ask.SendSectionLabel, + } + enabled := ask.Enabled != nil && *ask.Enabled + if v, ok := parseBoolEnv(EnvAskEnabled); ok { + enabled = v + } + if v := strings.ToLower(strings.TrimSpace(os.Getenv(EnvAskMode))); v != "" { + s.Mode = v + } + if v := strings.TrimSpace(os.Getenv(EnvAskKapaProjectID)); v != "" { + s.ProjectID = v + } + if v := strings.TrimSpace(os.Getenv(EnvAskKapaIntegrationID)); v != "" { + s.IntegrationID = v + } + if v := strings.TrimSpace(os.Getenv(EnvAskKapaAPIKeyFile)); v != "" { + s.APIKeyFile = v + } + if v, ok := parseBoolEnv(EnvAskSendSectionLabel); ok { + s.SendSectionLabel = v + } + s.Limits = resolveAskLimits(ask.Limits) + s.Requested = enabled + if !enabled { + s.Reason = "ask is not enabled" + return s.inert() + } + + if s.Mode == "" { + if isHub { + s.Mode = AskModeDirect + } else { + s.Mode = AskModeOff + } + } + switch { + case s.Mode == AskModeOff: + s.Reason = "ask.mode is off" + case s.Mode != AskModeDirect && s.Mode != AskModeHub: + s.Reason = "ask.mode is not one of off, direct or hub" + case s.Mode == AskModeHub && isHub: + s.Reason = "the hub always uses direct mode" + case s.Mode == AskModeHub: + s.Reason = "hub relay mode is not available yet" + case s.ProjectID == "": + s.Reason = "ask.kapa.project_id is missing" + case s.IntegrationID == "": + s.Reason = "ask.kapa.integration_id is missing" + case s.APIKeyFile == "": + s.Reason = "ask.kapa.api_key_file is missing" + case !SecretFilePathAllowed(s.APIKeyFile): + s.Reason = "ask.kapa.api_key_file is outside the allowed secret directories" + } + if s.Reason != "" { + return s.inert() + } + s.Active = true + return s +} + +// inert returns s with every vendor value cleared, so an inactive setting can +// never leak one by accident. +func (s AskSettings) inert() AskSettings { + s.Active = false + s.ProjectID, s.IntegrationID, s.APIKeyFile = "", "", "" + return s +} + +func resolveAskLimits(c AskLimitsConfig) AskLimits { + pick := func(env string, cfg, def int) int { + if v, err := strconv.Atoi(strings.TrimSpace(os.Getenv(env))); err == nil && v > 0 { + return v + } + if cfg > 0 { + return cfg + } + return def + } + return AskLimits{ + PersonBurst: pick(EnvAskLimitPersonBurst, c.PersonBurst, DefaultAskPersonBurst), + PersonPerHour: pick(EnvAskLimitPersonPerHour, c.PersonPerHour, DefaultAskPersonPerHour), + HivePerHour: pick(EnvAskLimitHivePerHour, c.HivePerHour, DefaultAskHivePerHour), + GlobalPerMinute: pick(EnvAskLimitGlobalPerMinute, c.GlobalPerMinute, DefaultAskGlobalPerMinute), + InFlight: pick(EnvAskLimitInFlight, c.InFlight, DefaultAskInFlight), + Daily: pick(EnvAskLimitDaily, c.Daily, DefaultAskDaily), + QuestionChars: pick(EnvAskLimitQuestionChars, c.QuestionChars, DefaultAskQuestionChars), + AnswerChars: pick(EnvAskLimitAnswerChars, c.AnswerChars, DefaultAskAnswerChars), + TimeoutSeconds: pick(EnvAskLimitTimeoutSeconds, c.TimeoutSeconds, DefaultAskTimeoutSeconds), + } +} diff --git a/src/pkg/config/ask_test.go b/src/pkg/config/ask_test.go new file mode 100644 index 0000000000..7bd4b9cc13 --- /dev/null +++ b/src/pkg/config/ask_test.go @@ -0,0 +1,158 @@ +package config + +import ( + "encoding/json" + "path/filepath" + "strings" + "testing" + + "gopkg.in/yaml.v3" +) + +func clearAskEnv(t *testing.T) { + t.Helper() + for _, name := range []string{ + EnvAskEnabled, EnvAskMode, EnvAskKapaProjectID, EnvAskKapaIntegrationID, EnvAskKapaAPIKeyFile, + EnvAskSendSectionLabel, EnvAskLimitPersonBurst, EnvAskLimitPersonPerHour, EnvAskLimitHivePerHour, + EnvAskLimitGlobalPerMinute, EnvAskLimitInFlight, EnvAskLimitDaily, EnvAskLimitQuestionChars, + EnvAskLimitAnswerChars, EnvAskLimitTimeoutSeconds, + } { + t.Setenv(name, "") + } +} + +func askTestConfig(t *testing.T, keyFile string) *Config { + t.Helper() + var cfg Config + doc := "ask:\n enabled: true\n kapa:\n project_id: proj-secret\n integration_id: integ-secret\n api_key_file: " + keyFile + "\n" + if err := yaml.Unmarshal([]byte(doc), &cfg); err != nil { + t.Fatal(err) + } + return &cfg +} + +func TestAskIsOffByDefault(t *testing.T) { + clearAskEnv(t) + for _, cfg := range []*Config{nil, {}} { + s := ResolveAskSettings(cfg, true) + if s.Active || s.Requested { + t.Fatalf("default must be off and not requested: %+v", s) + } + } +} + +func TestAskActiveOnHubDefaultsToDirect(t *testing.T) { + clearAskEnv(t) + root := keyFileTestRoot(t) + s := ResolveAskSettings(askTestConfig(t, filepath.Join(root, "kapa")), true) + if !s.Active || s.Mode != AskModeDirect || s.ProjectID != "proj-secret" || s.IntegrationID != "integ-secret" || !s.SendSectionLabel { + t.Fatalf("settings = %+v", s) + } + if s.Limits.PersonBurst != DefaultAskPersonBurst || s.Limits.Daily != DefaultAskDaily || s.Limits.TimeoutSeconds != DefaultAskTimeoutSeconds { + t.Fatalf("limits = %+v", s.Limits) + } +} + +func TestAskStaysOffWhenAValueIsMissingOrUnusable(t *testing.T) { + clearAskEnv(t) + root := keyFileTestRoot(t) + key := filepath.Join(root, "kapa") + + cases := map[string]func(*Config){ + "ask.kapa.project_id is missing": func(c *Config) { c.Ask.Kapa.ProjectID = "" }, + "ask.kapa.integration_id is missing": func(c *Config) { c.Ask.Kapa.IntegrationID = "" }, + "ask.kapa.api_key_file is missing": func(c *Config) { c.Ask.Kapa.APIKeyFile = "" }, + "outside the allowed secret": func(c *Config) { c.Ask.Kapa.APIKeyFile = "/etc/passwd" }, + "ask.mode is off": func(c *Config) { c.Ask.Mode = "off" }, + "not one of": func(c *Config) { c.Ask.Mode = "bogus" }, + "always uses direct mode": func(c *Config) { c.Ask.Mode = "hub" }, + } + for want, mutate := range cases { + cfg := askTestConfig(t, key) + mutate(cfg) + s := ResolveAskSettings(cfg, true) + if s.Active || !s.Requested || !strings.Contains(s.Reason, want) { + t.Fatalf("%q: %+v", want, s) + } + // Whatever the reason, an inactive setting carries no vendor value, and + // the reason itself names none. + if s.ProjectID != "" || s.IntegrationID != "" || s.APIKeyFile != "" { + t.Fatalf("%q: inactive settings still hold values: %+v", want, s) + } + for _, v := range []string{"proj-secret", "integ-secret", key} { + if strings.Contains(s.Reason, v) { + t.Fatalf("reason %q carries %q", s.Reason, v) + } + } + } +} + +func TestAskHiveModeWaitsForALaterPhase(t *testing.T) { + clearAskEnv(t) + cfg := askTestConfig(t, filepath.Join(keyFileTestRoot(t), "kapa")) + // A non-hub process with no mode set stays off; hub mode is not built yet. + if s := ResolveAskSettings(cfg, false); s.Active || !strings.Contains(s.Reason, "ask.mode is off") { + t.Fatalf("hive with no mode: %+v", s) + } + cfg.Ask.Mode = "hub" + if s := ResolveAskSettings(cfg, false); s.Active { + t.Fatalf("hub relay mode must stay off until it exists: %+v", s) + } + cfg.Ask.Mode = "direct" + if s := ResolveAskSettings(cfg, false); !s.Active { + t.Fatalf("direct mode on a hive: %+v", s) + } +} + +func TestAskEnvOverridesConfig(t *testing.T) { + clearAskEnv(t) + root := keyFileTestRoot(t) + cfg := askTestConfig(t, filepath.Join(root, "kapa")) + + t.Setenv(EnvAskEnabled, "false") + if s := ResolveAskSettings(cfg, true); s.Active { + t.Fatalf("env false must win over config true: %+v", s) + } + t.Setenv(EnvAskEnabled, "") + t.Setenv(EnvAskKapaProjectID, "env-proj") + t.Setenv(EnvAskKapaIntegrationID, "env-integ") + t.Setenv(EnvAskKapaAPIKeyFile, filepath.Join(root, "other")) + t.Setenv(EnvAskMode, "DIRECT") + t.Setenv(EnvAskSendSectionLabel, "false") + t.Setenv(EnvAskLimitPersonBurst, "7") + t.Setenv(EnvAskLimitDaily, "42") + t.Setenv(EnvAskLimitInFlight, "not-a-number") + cfg.Ask.Limits.InFlight = 9 + s := ResolveAskSettings(cfg, true) + if !s.Active || s.ProjectID != "env-proj" || s.IntegrationID != "env-integ" || + s.APIKeyFile != filepath.Join(root, "other") || s.SendSectionLabel { + t.Fatalf("env values not applied: %+v", s) + } + if s.Limits.PersonBurst != 7 || s.Limits.Daily != 42 || s.Limits.InFlight != 9 { + t.Fatalf("limits: %+v", s.Limits) + } + // With the switch only in env and no config file at all. + t.Setenv(EnvAskEnabled, "true") + if s := ResolveAskSettings(nil, true); !s.Active { + t.Fatalf("env alone should be enough: %+v", s) + } +} + +func TestAskVendorValuesNeverAppearInJSON(t *testing.T) { + clearAskEnv(t) + cfg := askTestConfig(t, "/secrets/kapa") + b, err := json.Marshal(cfg) + if err != nil { + t.Fatal(err) + } + for _, v := range []string{"proj-secret", "integ-secret", "/secrets/kapa"} { + if strings.Contains(string(b), v) { + t.Fatalf("JSON form carries %q: %s", v, b) + } + } + // The YAML form is what persists, so it must keep them. + y, err := yaml.Marshal(cfg.Ask) + if err != nil || !strings.Contains(string(y), "proj-secret") { + t.Fatalf("YAML form lost the project id: %s %v", y, err) + } +} diff --git a/src/pkg/config/config.go b/src/pkg/config/config.go index 0c9656ad71..bd6fcbe3a0 100644 --- a/src/pkg/config/config.go +++ b/src/pkg/config/config.go @@ -204,6 +204,8 @@ type Config struct { // Customers gates the Customers section and its people store (default // on). Nothing it stores leaves the hive. Customers CustomersConfig `yaml:"customers,omitempty" json:"customers,omitempty"` + // Ask is the optional Ask AI help block (#19177). Default off and inert. + Ask AskConfig `yaml:"ask,omitempty" json:"ask,omitempty"` // DuplicateSweep gates the cross-PR duplicate suggestion pass // (hivecommons/hive#7469 capability B). Default off → zero behaviour // change and no GitHub traffic. diff --git a/src/pkg/dashboard/api_config_export.go b/src/pkg/dashboard/api_config_export.go index e8b4606d8e..69abe9c476 100644 --- a/src/pkg/dashboard/api_config_export.go +++ b/src/pkg/dashboard/api_config_export.go @@ -259,6 +259,14 @@ func redactConfigValue(v any, key string) any { case map[string]any: out := make(map[string]any, len(x)) for k, val := range x { + // ask.kapa.project_id and ask.kapa.integration_id are vendor + // identifiers that must never leave the process (#19180). Their + // names do not match the secret-key pattern, so they are named here, + // scoped to the kapa block so other keys of that name are untouched. + if key == "kapa" && (k == "project_id" || k == "integration_id") && hasConfiguredValue(val) { + out[k] = redactMarker(fmt.Sprint(val)) + continue + } out[k] = redactConfigValue(val, k) } return out diff --git a/src/pkg/dashboard/api_config_export_ask_test.go b/src/pkg/dashboard/api_config_export_ask_test.go new file mode 100644 index 0000000000..cb4c3307ab --- /dev/null +++ b/src/pkg/dashboard/api_config_export_ask_test.go @@ -0,0 +1,41 @@ +package dashboard + +import ( + "encoding/json" + "strings" + "testing" +) + +// The Ask AI vendor identifiers must never appear in a config export (#19180). +func TestRedactConfigValueHidesAskKapaIdentifiers(t *testing.T) { + in := map[string]any{ + "ask": map[string]any{ + "enabled": true, + "kapa": map[string]any{ + "project_id": "proj-secret-1", + "integration_id": "integ-secret-2", + "api_key_file": "/secrets/kapa-key", + }, + }, + // An unrelated block with a key of the same name is left alone. + "other": map[string]any{"project_id": "visible-project"}, + } + b, err := json.Marshal(redactConfigValue(in, "")) + if err != nil { + t.Fatal(err) + } + out := string(b) + for _, hidden := range []string{"proj-secret-1", "integ-secret-2", "/secrets/kapa-key"} { + if strings.Contains(out, hidden) { + t.Fatalf("export carries %q: %s", hidden, out) + } + } + if !strings.Contains(out, "visible-project") || !strings.Contains(out, `"enabled":true`) { + t.Fatalf("redaction was too broad: %s", out) + } + // An unset value stays unset rather than becoming a marker. + empty := map[string]any{"kapa": map[string]any{"project_id": ""}} + if got := redactConfigValue(empty, "ask").(map[string]any)["kapa"].(map[string]any)["project_id"]; got != "" { + t.Fatalf("empty project_id = %v", got) + } +} diff --git a/src/pkg/hub/ask.go b/src/pkg/hub/ask.go new file mode 100644 index 0000000000..7c3633d861 --- /dev/null +++ b/src/pkg/hub/ask.go @@ -0,0 +1,349 @@ +package hub + +import ( + "encoding/json" + "errors" + "io" + "net/http" + "os" + "path/filepath" + "strconv" + "strings" + "sync" + "time" + + "github.com/hivecommons/hive/pkg/ask" + "github.com/hivecommons/hive/pkg/config" +) + +// Ask AI help, hub backend (#19180, phase 1 of #19177). Off by default and +// inert: with the switch off every route below answers 404, no consent prompt +// exists, no request leaves the hub and no header changes. See +// src/docs/design/dashboard-ask-ai-help.md. + +const ( + // askConsentTextVersion is bumped for a material change to the disclosure + // text. The page submits the version it displayed, so a person never + // consents to text they did not see. + askConsentTextVersion = "1" + // askConsentText is the disclosure the panel shows (phase 2). It is served + // by the consent route so the wording lives in one place. + askConsentText = "Your question, and the section you are in, are sent to Kapa.ai, an AI service. " + + "The maintainers of this project can read it in their analytics. " + + "Do not include secrets or private details." + maxAskBodyBytes = 16 << 10 + maxAskConsentBytes = 1 << 10 +) + +// askConsentMu serialises consent writes with each other and with user +// deletion, like customerConsentMu. +var askConsentMu sync.Mutex + +type askConsent struct { + Person string `json:"person"` + TextVersion string `json:"text_version"` + GrantedAt time.Time `json:"granted_at"` +} + +func askConsentDir() string { + return filepath.Join(filepath.Dir(saasUsersDir), "ask-consents") +} + +// askConsentPath keys the record by the same file stem the person's user record +// uses, so deleting the user finds it. One file per person, separate from every +// other record. +func askConsentPath(u *SaaSUser) (string, error) { + userPath, err := saveSaaSUserPath(u) + if err != nil { + return "", err + } + return filepath.Join(askConsentDir(), filepath.Base(userPath)), nil +} + +func askPersonStem(path string) string { + return strings.TrimSuffix(filepath.Base(path), ".json") +} + +func readAskConsent(path string) (*askConsent, error) { + data, err := os.ReadFile(path) + if errors.Is(err, os.ErrNotExist) { + return nil, nil + } + if err != nil { + return nil, err + } + var c askConsent + if err := json.Unmarshal(data, &c); err != nil { + return nil, err + } + return &c, nil +} + +// writeAskConsent writes the record atomically with owner-only permissions. +func writeAskConsent(path string, c askConsent) error { + data, err := json.MarshalIndent(c, "", " ") + if err != nil { + return err + } + dir := filepath.Dir(path) + if err := os.MkdirAll(dir, 0o700); err != nil { + return err + } + tmp, err := os.CreateTemp(dir, ".ask-consent-*") // mode 0600 + if err != nil { + return err + } + defer os.Remove(tmp.Name()) + _, writeErr := tmp.Write(data) + closeErr := tmp.Close() + if err := errors.Join(writeErr, closeErr); err != nil { + return err + } + if err := os.Chmod(tmp.Name(), 0o600); err != nil { + return err + } + return os.Rename(tmp.Name(), path) +} + +func removeAskConsent(path string) error { + if err := os.Remove(path); err != nil && !errors.Is(err, os.ErrNotExist) { + return err + } + return nil +} + +// askService resolves the Ask settings once per process and returns the +// service, or nil when the feature is off or could not start. When the operator +// switched it on but a value is missing or the key file is unusable, it stays +// off and logs ONE warning naming the reason, never a value. +func (s *HubServer) askService() *ask.Service { + s.askOnce.Do(func() { + var cfg *config.Config + if strings.TrimSpace(s.configPath) != "" { + loaded, err := config.LoadWithDashboardOverlayForHub(s.configPath) + if err == nil { + cfg = loaded + } else if !errors.Is(err, os.ErrNotExist) && s.logger != nil { + s.logger.Warn("ask: could not load hub config; using env only") + } + } + settings := config.ResolveAskSettings(cfg, true) + if !settings.Active { + if settings.Requested && s.logger != nil { + s.logger.Warn("ask: enabled but not started", "reason", settings.Reason) + } + return + } + svc, err := ask.Open(settings, s.logger, ask.Options{}) + if err != nil { + if s.logger != nil { + s.logger.Warn("ask: enabled but not started", "reason", err.Error()) + } + return + } + s.askSvc = svc + }) + return s.askSvc +} + +// requireAsk gates every Ask route: 404 when off (so a probe learns nothing), +// then a browser session only (no bearer fallback, like profile sharing: the +// consent is a person's decision), no impersonation, and the normal auth. +func (s *HubServer) requireAsk(next http.HandlerFunc) http.HandlerFunc { + auth := s.requireAuth(next) + return func(w http.ResponseWriter, r *http.Request) { + w.Header().Set("Cache-Control", "no-store") + if s.askService() == nil { + http.NotFound(w, r) + return + } + if r.Header.Get("Authorization") != "" { + writeJSONError(w, http.StatusUnauthorized, "hub session required") + return + } + if s.customerShareSession(r) == "" { + writeJSONError(w, http.StatusUnauthorized, "hub session required") + return + } + if _, err := r.Cookie(impersonateCookieName); err == nil { + if _, _, impersonating := s.resolveIdentity(r); impersonating { + writeJSONError(w, http.StatusForbidden, "exit impersonation to use Ask") + return + } + } + auth(w, r) + } +} + +func (s *HubServer) registerAskRoutes() { + s.mux.HandleFunc("GET /api/ask/consent", s.requireAsk(s.handleAskConsent)) + s.mux.HandleFunc("PUT /api/ask/consent", s.requireAsk(s.handleAskConsent)) + s.mux.HandleFunc("DELETE /api/ask/consent", s.requireAsk(s.handleAskConsent)) + s.mux.HandleFunc("POST /api/ask", s.requireAsk(s.handleAsk)) + s.mux.HandleFunc("GET /api/saas/admin/ask/stats", s.requireAdmin(s.handleAskStats)) +} + +// askUser resolves the signed-in person and their consent path, answering the +// request itself on failure. +func (s *HubServer) askUser(w http.ResponseWriter, r *http.Request) (*SaaSUser, string, bool) { + user := loadSaaSUser(s.customerShareSession(r)) + if user == nil || user.Blocked { + writeJSONError(w, http.StatusForbidden, "account unavailable") + return nil, "", false + } + path, err := askConsentPath(user) + if err != nil { + s.askStoreError(w, err) + return nil, "", false + } + return user, path, true +} + +func (s *HubServer) askStoreError(w http.ResponseWriter, err error) { + s.logger.Warn("ask consent store failed", "error", err) + writeJSONError(w, http.StatusInternalServerError, "could not access consent records") +} + +func (s *HubServer) handleAskConsent(w http.ResponseWriter, r *http.Request) { + var body struct { + TextVersion string `json:"text_version"` + } + if r.Method == http.MethodPut { + d := json.NewDecoder(http.MaxBytesReader(w, r.Body, maxAskConsentBytes)) + d.DisallowUnknownFields() + if d.Decode(&body) != nil || d.Decode(new(json.RawMessage)) != io.EOF { + writeJSONError(w, http.StatusBadRequest, "invalid consent JSON") + return + } + if body.TextVersion != askConsentTextVersion { + writeJSONError(w, http.StatusConflict, "disclosure version changed; reload and confirm again") + return + } + } + + askConsentMu.Lock() + defer askConsentMu.Unlock() + _, path, ok := s.askUser(w, r) + if !ok { + return + } + person := askPersonStem(path) + switch r.Method { + case http.MethodPut: + now := time.Now().UTC() + if err := writeAskConsent(path, askConsent{Person: person, TextVersion: askConsentTextVersion, GrantedAt: now}); err != nil { + s.askStoreError(w, err) + return + } + s.logger.Info("audit: ask consent grant", "person", person, "text_version", askConsentTextVersion) + w.WriteHeader(http.StatusNoContent) + case http.MethodDelete: + existing, err := readAskConsent(path) + if err == nil { + err = removeAskConsent(path) + } + if err != nil { + s.askStoreError(w, err) + return + } + // Withdrawal takes effect on the next request, and the conversation + // the person had open is dropped with it. + s.askService().Forget(person) + if existing != nil { + s.logger.Info("audit: ask consent withdraw", "person", person, "text_version", existing.TextVersion) + } + w.WriteHeader(http.StatusNoContent) + default: + c, err := readAskConsent(path) + if err != nil { + s.askStoreError(w, err) + return + } + w.Header().Set("Content-Type", "application/json") + _ = json.NewEncoder(w).Encode(struct { + Consented bool `json:"consented"` + TextVersion string `json:"text_version"` + Text string `json:"text"` + }{c != nil && c.TextVersion == askConsentTextVersion, askConsentTextVersion, askConsentText}) + } +} + +type askRequestBody struct { + Question string `json:"question"` + Section string `json:"section"` + Thread string `json:"thread"` +} + +func (s *HubServer) handleAsk(w http.ResponseWriter, r *http.Request) { + var body askRequestBody + d := json.NewDecoder(http.MaxBytesReader(w, r.Body, maxAskBodyBytes)) + d.DisallowUnknownFields() + if d.Decode(&body) != nil || d.Decode(new(json.RawMessage)) != io.EOF { + writeJSONError(w, http.StatusBadRequest, "invalid ask JSON") + return + } + svc := s.askService() + _, path, ok := s.askUser(w, r) + if !ok { + return + } + person := askPersonStem(path) + + // Consent gates every request: a missing record, a withdrawn one and an + // old text version all stop here, before anything else runs. + askConsentMu.Lock() + c, err := readAskConsent(path) + askConsentMu.Unlock() + if err != nil { + s.askStoreError(w, err) + return + } + if c == nil || c.TextVersion != askConsentTextVersion { + writeAskResult(w, svc.RecordNotConsented(person, "", len([]rune(body.Question)))) + return + } + + res := svc.Ask(r.Context(), ask.Request{ + Person: person, Question: body.Question, Section: body.Section, Thread: body.Thread, + }) + writeAskResult(w, res) +} + +func writeAskResult(w http.ResponseWriter, res ask.Result) { + status := http.StatusOK + switch res.Outcome { + case ask.OutcomeOK, ask.OutcomeNoCitedAnswer: + case ask.OutcomeRefusedSecret, ask.OutcomeRefusedLength, ask.OutcomeRefusedEmpty: + status = http.StatusUnprocessableEntity + case ask.OutcomeNotConsented: + status = http.StatusForbidden + case ask.OutcomeRateLimited, ask.OutcomeBudgetSpent: + status = http.StatusTooManyRequests + if res.RetryAfterSeconds > 0 { + w.Header().Set("Retry-After", strconv.Itoa(res.RetryAfterSeconds)) + } + default: // breaker_open, upstream_error: the panel shows the docs link + status = http.StatusServiceUnavailable + } + w.Header().Set("Content-Type", "application/json") + w.WriteHeader(status) + _ = json.NewEncoder(w).Encode(res) +} + +// handleAskStats is the hub operator's counter: requests by outcome since the +// process started and the number sent to the vendor today. +func (s *HubServer) handleAskStats(w http.ResponseWriter, r *http.Request) { + w.Header().Set("Cache-Control", "no-store") + svc := s.askService() + if svc == nil { + w.Header().Set("Content-Type", "application/json") + _ = json.NewEncoder(w).Encode(map[string]any{"enabled": false}) + return + } + outcomes, today := svc.Counters() + lim := svc.Limits() + w.Header().Set("Content-Type", "application/json") + _ = json.NewEncoder(w).Encode(map[string]any{ + "enabled": true, "outcomes": outcomes, "sent_today": today, "daily_budget": lim.Daily, + }) +} diff --git a/src/pkg/hub/ask_test.go b/src/pkg/hub/ask_test.go new file mode 100644 index 0000000000..84a621bea5 --- /dev/null +++ b/src/pkg/hub/ask_test.go @@ -0,0 +1,526 @@ +package hub + +import ( + "bytes" + "context" + "encoding/json" + "errors" + "log/slog" + "net/http" + "net/http/httptest" + "os" + "path/filepath" + "strings" + "sync" + "testing" + + cruxconfig "github.com/hivecommons/hive/crux/config" + "github.com/hivecommons/hive/pkg/ask" + "github.com/hivecommons/hive/pkg/config" +) + +// Ask AI help hub backend (#19180). + +const ( + askTestOrigin = "https://hub.example.com" + askAPI = "/api/ask" + askConsentAPI = "/api/ask/consent" +) + +type askVendor struct { + mu sync.Mutex + queries []string + resp ask.ChatResponse + err error +} + +func (v *askVendor) Chat(_ context.Context, query, _ string) (ask.ChatResponse, error) { + v.mu.Lock() + defer v.mu.Unlock() + v.queries = append(v.queries, query) + return v.resp, v.err +} + +func (v *askVendor) calls() int { + v.mu.Lock() + defer v.mu.Unlock() + return len(v.queries) +} + +func askTestSettings() config.AskSettings { + return config.AskSettings{ + Active: true, Mode: config.AskModeDirect, SendSectionLabel: true, + Limits: config.AskLimits{ + PersonBurst: 3, PersonPerHour: 30, HivePerHour: 120, GlobalPerMinute: 8, + InFlight: 4, Daily: 500, QuestionChars: 200, AnswerChars: 500, TimeoutSeconds: 5, + }, + } +} + +func clearAskEnv(t *testing.T) { + t.Helper() + for _, name := range []string{ + config.EnvAskEnabled, config.EnvAskMode, config.EnvAskKapaProjectID, + config.EnvAskKapaIntegrationID, config.EnvAskKapaAPIKeyFile, + } { + t.Setenv(name, "") + } +} + +// newAskTestHub builds a hub with the Ask routes registered. With vendor nil +// the feature is off (as in production with no configuration). +func newAskTestHub(t *testing.T, vendor *askVendor) (*HubServer, *bytes.Buffer) { + t.Helper() + clearAskEnv(t) + t.Setenv("HIVE_HUB_PUBLIC_URL", askTestOrigin) + s := newHandlerHub() + s.mux = http.NewServeMux() + var logs bytes.Buffer + s.logger = slog.New(slog.NewTextHandler(&logs, nil)) + if vendor != nil { + svc, err := ask.Open(askTestSettings(), s.logger, ask.Options{Chatter: vendor}) + if err != nil { + t.Fatal(err) + } + s.askOnce.Do(func() {}) + s.askSvc = svc + } + s.registerAskRoutes() + s.mux.HandleFunc("DELETE /admin/{username}", s.requireAdmin(s.handleAdminDeleteUser)) + return s, &logs +} + +func askDo(s *HubServer, method, path, body, login string) *httptest.ResponseRecorder { + r := reqWithUser(method, path, body, login) + if login == "" { + r.Header.Del("Cookie") + } + if method != http.MethodGet { + r.Header.Set("Origin", askTestOrigin) + } + w := httptest.NewRecorder() + s.mux.ServeHTTP(w, r) + return w +} + +func goodAskResponse() ask.ChatResponse { + return ask.ChatResponse{ + Answer: "Open Settings, then Features.", ThreadID: "vendor-thread-1", + RelevantSources: []ask.VendorSource{{SourceURL: "https://docs.hivecommons.dev/docs/dashboard", Title: "Dashboard"}}, + } +} + +func grantAskConsent(t *testing.T, s *HubServer, login string) { + t.Helper() + w := askDo(s, "PUT", askConsentAPI, `{"text_version":"`+askConsentTextVersion+`"}`, login) + if w.Code != http.StatusNoContent { + t.Fatalf("grant: %d %s", w.Code, w.Body) + } +} + +func TestAskOffMeansNoRouteAndNoRequest(t *testing.T) { + defer helperSetupTempDirs(t)() + s, logs := newAskTestHub(t, nil) + mkUser(t, "alice") + mkUser(t, hubAdminUsername) + for _, tc := range []struct{ method, path, body string }{ + {"GET", askConsentAPI, ""}, + {"PUT", askConsentAPI, `{"text_version":"1"}`}, + {"DELETE", askConsentAPI, ""}, + {"POST", askAPI, `{"question":"hi"}`}, + } { + // Signed in or not, the answer is the same 404: a probe learns nothing. + for _, login := range []string{"alice", ""} { + if w := askDo(s, tc.method, tc.path, tc.body, login); w.Code != http.StatusNotFound { + t.Fatalf("%s %s as %q = %d, want 404", tc.method, tc.path, login, w.Code) + } + } + } + if w := askDo(s, "GET", "/api/saas/admin/ask/stats", "", hubAdminUsername); w.Code != http.StatusOK || + !strings.Contains(w.Body.String(), `"enabled":false`) { + t.Fatalf("stats when off: %d %s", w.Code, w.Body) + } + if _, err := os.Stat(askConsentDir()); !errors.Is(err, os.ErrNotExist) { + t.Fatalf("off must write nothing: %v", err) + } + if strings.Contains(logs.String(), "ask:") { + t.Fatalf("off must be silent when never requested: %s", logs.String()) + } +} + +func TestAskAddsNoContentSecurityPolicy(t *testing.T) { + defer helperSetupTempDirs(t)() + if strings.Contains(strings.ToLower(activityPageCSP), "kapa") { + t.Fatal("the hub CSP names the vendor") + } + vendor := &askVendor{resp: goodAskResponse()} + s, _ := newAskTestHub(t, vendor) + mkUser(t, "alice") + grantAskConsent(t, s, "alice") + for _, w := range []*httptest.ResponseRecorder{ + askDo(s, "GET", askConsentAPI, "", "alice"), + askDo(s, "POST", askAPI, `{"question":"hi"}`, "alice"), + } { + if csp := w.Header().Get("Content-Security-Policy"); csp != "" { + t.Fatalf("Ask routes must not set a CSP, got %q", csp) + } + if w.Header().Get("Cache-Control") != "no-store" { + t.Fatal("Ask responses must not be cached") + } + } +} + +func TestAskRequiresABrowserSession(t *testing.T) { + defer helperSetupTempDirs(t)() + vendor := &askVendor{resp: goodAskResponse()} + s, _ := newAskTestHub(t, vendor) + mkUser(t, "alice") + grantAskConsent(t, s, "alice") + + if w := askDo(s, "POST", askAPI, `{"question":"hi"}`, ""); w.Code != http.StatusUnauthorized { + t.Fatalf("anonymous = %d", w.Code) + } + // A bearer token is not a person's consent. + r := reqWithUser("POST", askAPI, `{"question":"hi"}`, "alice") + r.Header.Set("Origin", askTestOrigin) + r.Header.Set("Authorization", "Bearer something") + w := httptest.NewRecorder() + s.mux.ServeHTTP(w, r) + if w.Code != http.StatusUnauthorized { + t.Fatalf("bearer = %d", w.Code) + } + // Cross-origin browser POST fails the CSRF check. + r = reqWithUser("POST", askAPI, `{"question":"hi"}`, "alice") + r.Header.Set("Origin", "https://evil.example.com") + w = httptest.NewRecorder() + s.mux.ServeHTTP(w, r) + if w.Code != http.StatusForbidden { + t.Fatalf("cross-origin = %d", w.Code) + } + // A blocked account is refused. + if err := saveSaaSUser(&SaaSUser{GitHubUsername: "mallory", Blocked: true, Hives: map[string]string{}}); err != nil { + t.Fatal(err) + } + if w := askDo(s, "POST", askAPI, `{"question":"hi"}`, "mallory"); w.Code != http.StatusForbidden { + t.Fatalf("blocked = %d", w.Code) + } + if vendor.calls() != 0 { + t.Fatalf("vendor was called %d times by refused requests", vendor.calls()) + } +} + +func TestAskConsentGatesEveryRequest(t *testing.T) { + defer helperSetupTempDirs(t)() + vendor := &askVendor{resp: goodAskResponse()} + s, logs := newAskTestHub(t, vendor) + mkUser(t, "alice") + mkUser(t, "bob") + postAsk := func(login string) *httptest.ResponseRecorder { + return askDo(s, "POST", askAPI, `{"question":"where is Features?"}`, login) + } + + // No consent yet. + if w := postAsk("alice"); w.Code != http.StatusForbidden || !strings.Contains(w.Body.String(), "not_consented") { + t.Fatalf("before consent: %d %s", w.Code, w.Body) + } + if vendor.calls() != 0 { + t.Fatal("a request without consent reached the vendor") + } + + // The page must submit the version it displayed. + if w := askDo(s, "PUT", askConsentAPI, `{"text_version":"old"}`, "alice"); w.Code != http.StatusConflict { + t.Fatalf("stale text version = %d", w.Code) + } + if w := askDo(s, "PUT", askConsentAPI, `{"text_version":"1","person":"bob"}`, "alice"); w.Code != http.StatusBadRequest { + t.Fatalf("unknown field = %d", w.Code) + } + grantAskConsent(t, s, "alice") + if w := postAsk("alice"); w.Code != http.StatusOK || !strings.Contains(w.Body.String(), `"outcome":"ok"`) { + t.Fatalf("after consent: %d %s", w.Code, w.Body) + } + if vendor.calls() != 1 { + t.Fatalf("vendor calls = %d", vendor.calls()) + } + // One person's consent does not cover another. + if w := postAsk("bob"); w.Code != http.StatusForbidden { + t.Fatalf("bob without consent = %d", w.Code) + } + + // The record: one file per person, owner-only, with the text version. + u := loadSaaSUser("alice") + path, err := askConsentPath(u) + if err != nil { + t.Fatal(err) + } + info, err := os.Stat(path) + if err != nil { + t.Fatal(err) + } + if info.Mode().Perm() != 0o600 { + t.Fatalf("consent permissions = %o", info.Mode().Perm()) + } + c, err := readAskConsent(path) + if err != nil || c == nil || c.TextVersion != askConsentTextVersion || c.Person != "alice" { + t.Fatalf("record = %+v, %v", c, err) + } + if bobPath, _ := askConsentPath(loadSaaSUser("bob")); bobPath == path { + t.Fatal("two people share a consent file") + } + + // GET reports state and serves the disclosure text. + w := askDo(s, "GET", askConsentAPI, "", "alice") + var state struct { + Consented bool `json:"consented"` + TextVersion string `json:"text_version"` + Text string `json:"text"` + } + if err := json.Unmarshal(w.Body.Bytes(), &state); err != nil || !state.Consented || state.Text != askConsentText { + t.Fatalf("state = %+v %v", state, err) + } + if state2 := askDo(s, "GET", askConsentAPI, "", "bob"); strings.Contains(state2.Body.String(), `"consented":true`) { + t.Fatal("bob appears consented") + } + + // A text change blocks the next request until confirmed again. + if err := writeAskConsent(path, askConsent{Person: "alice", TextVersion: "0"}); err != nil { + t.Fatal(err) + } + if w := postAsk("alice"); w.Code != http.StatusForbidden { + t.Fatalf("old text version = %d", w.Code) + } + grantAskConsent(t, s, "alice") + + // Withdrawal takes effect on the very next request. + if w := askDo(s, "DELETE", askConsentAPI, "", "alice"); w.Code != http.StatusNoContent { + t.Fatalf("withdraw = %d", w.Code) + } + if w := postAsk("alice"); w.Code != http.StatusForbidden { + t.Fatalf("after withdrawal = %d", w.Code) + } + if _, err := os.Stat(path); !errors.Is(err, os.ErrNotExist) { + t.Fatalf("withdrawn consent must not linger: %v", err) + } + // Withdrawing again is harmless. + if w := askDo(s, "DELETE", askConsentAPI, "", "alice"); w.Code != http.StatusNoContent { + t.Fatalf("second withdraw = %d", w.Code) + } + for _, want := range []string{"audit: ask consent grant", "audit: ask consent withdraw"} { + if !strings.Contains(logs.String(), want) { + t.Fatalf("missing audit line %q:\n%s", want, logs.String()) + } + } +} + +func TestAskRefusesCredentialsAndBadBodies(t *testing.T) { + defer helperSetupTempDirs(t)() + vendor := &askVendor{resp: goodAskResponse()} + s, logs := newAskTestHub(t, vendor) + mkUser(t, "alice") + grantAskConsent(t, s, "alice") + + token := "ghp_" + strings.Repeat("a1B2c3D4e5", 4) + w := askDo(s, "POST", askAPI, `{"question":"why does `+token+` fail"}`, "alice") + if w.Code != http.StatusUnprocessableEntity || !strings.Contains(w.Body.String(), "refused_secret_shape") { + t.Fatalf("credential: %d %s", w.Code, w.Body) + } + if strings.Contains(w.Body.String(), token) || strings.Contains(logs.String(), token) { + t.Fatal("the credential was echoed") + } + long := strings.Repeat("x", 201) + if w := askDo(s, "POST", askAPI, `{"question":"`+long+`"}`, "alice"); w.Code != http.StatusUnprocessableEntity { + t.Fatalf("too long = %d", w.Code) + } + for _, body := range []string{`not json`, `{"question":"hi","user":"alice"}`, `{"question":"hi"} {}`} { + if w := askDo(s, "POST", askAPI, body, "alice"); w.Code != http.StatusBadRequest { + t.Fatalf("body %q = %d", body, w.Code) + } + } + if w := askDo(s, "POST", askAPI, `{"question":"`+strings.Repeat("x", maxAskBodyBytes)+`"}`, "alice"); w.Code != http.StatusBadRequest { + t.Fatalf("oversized body = %d", w.Code) + } + if vendor.calls() != 0 { + t.Fatalf("refused input reached the vendor %d times", vendor.calls()) + } +} + +func TestAskSendsOnlyQuestionAndSectionLabel(t *testing.T) { + defer helperSetupTempDirs(t)() + vendor := &askVendor{resp: goodAskResponse()} + s, _ := newAskTestHub(t, vendor) + if err := saveSaaSUser(&SaaSUser{ + GitHubUsername: "alice", FullName: "Alice Private", Email: "alice@private.example", + Hives: map[string]string{"secret-hive-id": "owner"}, + }); err != nil { + t.Fatal(err) + } + grantAskConsent(t, s, "alice") + w := askDo(s, "POST", askAPI, `{"question":"where is it?","section":"settings-features"}`, "alice") + if w.Code != http.StatusOK { + t.Fatalf("%d %s", w.Code, w.Body) + } + if got := vendor.queries[0]; got != "Dashboard section: Settings, Features\n\nwhere is it?" { + t.Fatalf("vendor query = %q", got) + } + for _, seeded := range []string{"alice", "Alice Private", "private.example", "secret-hive-id"} { + if strings.Contains(vendor.queries[0], seeded) { + t.Fatalf("vendor query carries %q", seeded) + } + } + // The response carries the cited answer and an opaque handle only. + var res ask.Result + if err := json.Unmarshal(w.Body.Bytes(), &res); err != nil { + t.Fatal(err) + } + if res.Outcome != ask.OutcomeOK || len(res.Sources) != 1 || res.Thread == "" || strings.Contains(w.Body.String(), "vendor-thread-1") { + t.Fatalf("result = %+v body=%s", res, w.Body) + } +} + +func TestAskLimitsAndFallbackStatuses(t *testing.T) { + defer helperSetupTempDirs(t)() + vendor := &askVendor{resp: goodAskResponse()} + s, _ := newAskTestHub(t, vendor) + mkUser(t, "alice") + grantAskConsent(t, s, "alice") + post := func() *httptest.ResponseRecorder { + return askDo(s, "POST", askAPI, `{"question":"q"}`, "alice") + } + for i := 0; i < 3; i++ { + if w := post(); w.Code != http.StatusOK { + t.Fatalf("request %d = %d", i, w.Code) + } + } + w := post() + if w.Code != http.StatusTooManyRequests || w.Header().Get("Retry-After") == "" || !strings.Contains(w.Body.String(), "rate_limited") { + t.Fatalf("burst: %d %v %s", w.Code, w.Header(), w.Body) + } + if vendor.calls() != 3 { + t.Fatalf("a limited request reached the vendor: %d", vendor.calls()) + } + + // A vendor outage is a 503 with the fallback message, never an answer. + vendor2 := &askVendor{err: ask.ErrUnavailable} + s2, _ := newAskTestHub(t, vendor2) + grantAskConsent(t, s2, "alice") + w = askDo(s2, "POST", askAPI, `{"question":"q"}`, "alice") + if w.Code != http.StatusServiceUnavailable || !strings.Contains(w.Body.String(), "upstream_error") || strings.Contains(w.Body.String(), `"answer"`) { + t.Fatalf("outage: %d %s", w.Code, w.Body) + } +} + +func TestAskUserDeletionRemovesConsent(t *testing.T) { + defer helperSetupTempDirs(t)() + vendor := &askVendor{resp: goodAskResponse()} + s, _ := newAskTestHub(t, vendor) + mkUser(t, hubAdminUsername) + mkUser(t, "alice") + grantAskConsent(t, s, "alice") + path, err := askConsentPath(loadSaaSUser("alice")) + if err != nil { + t.Fatal(err) + } + if _, err := os.Stat(path); err != nil { + t.Fatal(err) + } + if w := askDo(s, "DELETE", "/admin/alice", "", hubAdminUsername); w.Code != http.StatusOK { + t.Fatalf("delete user = %d %s", w.Code, w.Body) + } + if _, err := os.Stat(path); !errors.Is(err, os.ErrNotExist) || loadSaaSUser("alice") != nil { + t.Fatalf("user deletion must remove the Ask consent: %v", err) + } + if filepath.Dir(path) != askConsentDir() { + t.Fatalf("consent lives outside its own directory: %s", path) + } +} + +func TestAskStatsCountOutcomesForAdminsOnly(t *testing.T) { + defer helperSetupTempDirs(t)() + vendor := &askVendor{resp: goodAskResponse()} + s, _ := newAskTestHub(t, vendor) + mkUser(t, hubAdminUsername) + mkUser(t, "alice") + grantAskConsent(t, s, "alice") + askDo(s, "POST", askAPI, `{"question":"q"}`, "alice") + askDo(s, "POST", askAPI, `{"question":" "}`, "alice") + + if w := askDo(s, "GET", "/api/saas/admin/ask/stats", "", "alice"); w.Code == http.StatusOK { + t.Fatalf("a non-admin read the stats: %d", w.Code) + } + w := askDo(s, "GET", "/api/saas/admin/ask/stats", "", hubAdminUsername) + var stats struct { + Enabled bool `json:"enabled"` + Outcomes map[string]int64 `json:"outcomes"` + SentToday int `json:"sent_today"` + } + if err := json.Unmarshal(w.Body.Bytes(), &stats); err != nil { + t.Fatal(err) + } + if !stats.Enabled || stats.Outcomes["ok"] != 1 || stats.Outcomes["refused_empty"] != 1 || stats.SentToday != 1 { + t.Fatalf("stats = %+v", stats) + } +} + +func TestAskServiceStartsFromEnvAndStaysOffQuietlyOtherwise(t *testing.T) { + defer helperSetupTempDirs(t)() + dir := t.TempDir() + defer cruxconfig.SetSecretFileRootsForTest(dir)() + keyFile := filepath.Join(dir, "kapa-key") + if err := os.WriteFile(keyFile, []byte("kapa-key-value\n"), 0o600); err != nil { + t.Fatal(err) + } + + newHub := func() (*HubServer, *bytes.Buffer) { + s := newHandlerHub() + var logs bytes.Buffer + s.logger = slog.New(slog.NewTextHandler(&logs, nil)) + return s, &logs + } + + // Switched on with every value: starts. + clearAskEnv(t) + t.Setenv(config.EnvAskEnabled, "true") + t.Setenv(config.EnvAskKapaProjectID, "proj-secret-value") + t.Setenv(config.EnvAskKapaIntegrationID, "integ-secret-value") + t.Setenv(config.EnvAskKapaAPIKeyFile, keyFile) + s, logs := newHub() + if s.askService() == nil { + t.Fatalf("a fully configured Ask did not start: %s", logs) + } + + // Switched on but a value is missing: off, ONE warning, no value in it. + t.Setenv(config.EnvAskKapaIntegrationID, "") + s, logs = newHub() + for i := 0; i < 3; i++ { + if s.askService() != nil { + t.Fatal("Ask started with a missing value") + } + } + if n := strings.Count(logs.String(), "ask: enabled but not started"); n != 1 { + t.Fatalf("warnings = %d, want 1:\n%s", n, logs) + } + for _, v := range []string{"proj-secret-value", "kapa-key-value", keyFile} { + if strings.Contains(logs.String(), v) { + t.Fatalf("warning carries %q", v) + } + } + + // Switched on, every value present, key file unreadable: off, one warning. + t.Setenv(config.EnvAskKapaIntegrationID, "integ-secret-value") + t.Setenv(config.EnvAskKapaAPIKeyFile, filepath.Join(dir, "missing")) + s, logs = newHub() + if s.askService() != nil || strings.Count(logs.String(), "ask: enabled but not started") != 1 { + t.Fatalf("unreadable key file:\n%s", logs) + } + for _, v := range []string{"proj-secret-value", "integ-secret-value", dir} { + if strings.Contains(logs.String(), v) { + t.Fatalf("warning carries %q", v) + } + } + + // Never switched on: off and silent. + clearAskEnv(t) + s, logs = newHub() + if s.askService() != nil || logs.Len() != 0 { + t.Fatalf("default must be off and silent: %s", logs) + } +} diff --git a/src/pkg/hub/saas.go b/src/pkg/hub/saas.go index 44baa88388..2c0acc956f 100644 --- a/src/pkg/hub/saas.go +++ b/src/pkg/hub/saas.go @@ -538,6 +538,7 @@ func (s *HubServer) registerSaaSRoutes() { s.mux.HandleFunc("GET /api/saas/admin/journey-status", s.requireAdmin(s.handleJourneyStatus)) s.registerBuildContainsRoutes() s.registerCustomerSharingRoutes() + s.registerAskRoutes() } func pendingAgentRestartResetsForHeartbeat(hiveID string) []string { diff --git a/src/pkg/hub/saas_users_admin.go b/src/pkg/hub/saas_users_admin.go index 9b79f668d6..0f0225773f 100644 --- a/src/pkg/hub/saas_users_admin.go +++ b/src/pkg/hub/saas_users_admin.go @@ -672,6 +672,22 @@ func (s *HubServer) handleAdminDeleteUser(w http.ResponseWriter, r *http.Request writeJSONError(w, http.StatusInternalServerError, "failed to delete user's consent records") return } + // The Ask AI consent record goes with the user (#19180), and any open + // conversation with it. + askConsentMu.Lock() + askPath, askErr := askConsentPath(u) + if askErr == nil { + askErr = removeAskConsent(askPath) + } + askConsentMu.Unlock() + if askErr != nil { + s.logger.Warn("admin delete user: ask consent removal failed", "target", username, "error", askErr) + writeJSONError(w, http.StatusInternalServerError, "failed to delete user's consent records") + return + } + if svc := s.askService(); svc != nil { + svc.Forget(askPersonStem(askPath)) + } // Delete both canonical and legacy representations; otherwise a leftover // fallback would become the live account as soon as the first file is gone. for _, path := range saaSUserFilePaths(username) { diff --git a/src/pkg/hub/server.go b/src/pkg/hub/server.go index d41192d0a0..76d7b61850 100644 --- a/src/pkg/hub/server.go +++ b/src/pkg/hub/server.go @@ -27,6 +27,7 @@ import ( "github.com/hivecommons/hive/crux" "github.com/hivecommons/hive/crux/openrouter" + "github.com/hivecommons/hive/pkg/ask" "github.com/hivecommons/hive/pkg/auth" "github.com/hivecommons/hive/pkg/dashboard/webstatic" "github.com/hivecommons/hive/pkg/delegation" @@ -1020,6 +1021,12 @@ type HubServer struct { configPath string envGitHubToken string discordRequests discordRequestNotificationState + + // askOnce/askSvc hold the optional Ask AI help service (#19180); askSvc is + // nil when the feature is off or could not start. + askOnce sync.Once + askSvc *ask.Service + // saveLoopStop / saveLoopDone make the debounced saveLoop goroutine // joinable (#4774). A hub built by NewHubServer used to leak its saveLoop // forever: in tests, the loop could wake up to registrySaveDelay after the From 33b5d097050839a78e4883921e9fef1ff320a617 Mon Sep 17 00:00:00 2001 From: Danathar Date: Thu, 19 Nov 2026 01:43:05 +0000 Subject: [PATCH 2/2] =?UTF-8?q?=F0=9F=93=96=20docs:=20refresh=20hub-server?= =?UTF-8?q?=20api-reference=20citations=20after=20the=20Ask=20AI=20route?= =?UTF-8?q?=20registration?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit The Ask AI routes shifted pkg/hub/server.go registration lines, so 34 citations in api-reference/hub-server.md drifted. Rewritten with src/scripts/check-api-reference-citations.sh --fix; the checker now passes. Hive-Run: hivecommons/hive#19180 Co-Authored-By: Claude Sonnet 5.5 Signed-off-by: Danathar --- src/docs/api-reference/hub-server.md | 68 ++++++++++++++-------------- 1 file changed, 34 insertions(+), 34 deletions(-) diff --git a/src/docs/api-reference/hub-server.md b/src/docs/api-reference/hub-server.md index a3223de629..66b77229a4 100644 --- a/src/docs/api-reference/hub-server.md +++ b/src/docs/api-reference/hub-server.md @@ -35,29 +35,29 @@ Part of the [Dashboard REST API reference](../api-reference.md); see that page f | `POST` | `/api/benchmarks/ingest` | Per-hive heartbeat bearer | Spoke push target for anonymized [benchmark records](../benchmarks.md): body `{hive_id, records[]}`; the hub recomputes the hive hash from the authenticated id, rejects records whose `source.hive_id` differs (`401` bad bearer, `403` unregistered hive, `400` malformed), merges accepted records by id into `/data/benchmarks//.json` and returns `202 {accepted, rejected, files}` | `pkg/hub/benchmarks.go` | | `GET` | `/api/hub/benchmarks` | Public | Benchmark publisher status: `last_ingest_at`, `last_publish_at`, `last_publish_files`, `last_error`, `pending_files`, `hive_count`, `record_count` and per-hive-hash `hives{}` counts; the hub commits pending month files hourly to `hivecommons/benchmarks` (`HIVE_BENCHMARKS_REPO`) via its App identity | `pkg/hub/benchmarks.go` | | `GET` | `/api/reach` | Hub admin | PR Reach Report (?pr=NNN or ?recent=K) | `pkg/hub/saas.go:511` | -| `GET` | `/fleet` | Hub handler-specific | My-Hives Fleet Page (static; data via `/api/saas/my-hives`) | `pkg/hub/server.go:1829` | -| `GET` | `/my-hives` | Hub handler-specific | 301 redirect to `/fleet` (query preserved) | `pkg/hub/server.go:1830` | +| `GET` | `/fleet` | Hub handler-specific | My-Hives Fleet Page (static; data via `/api/saas/my-hives`) | `pkg/hub/server.go:1836` | +| `GET` | `/my-hives` | Hub handler-specific | 301 redirect to `/fleet` (query preserved) | `pkg/hub/server.go:1837` | | `GET` | `/activity` | Signed-in user | Fleet activity page ([#16063](https://github.com/hivecommons/hive/issues/16063)): read-only, data-free shell that polls `/api/hub/activity` every 15 s with `If-None-Match`. No session → `307` to `/login`; blocked user → `403`. Own Content-Security-Policy (scripts and styles from `'self'` only, `script-src-attr`/`style-src-attr` `'none'`), `Cache-Control: no-cache` + content `ETag`. See [Hub Fleet activity page](../hub-fleet-activity.md) | `pkg/hub/activity_page.go` | | `GET` | `/activity/public` | Public | Public-safe Fleet activity page ([#15966](https://github.com/hivecommons/hive/issues/15966)): the same data-free shell as `/activity`, served without a session; on this path the page polls `/api/hub/activity/public` with `credentials: 'omit'`. Source of the `twitch.tv/hivecommonsfleet` broadcast. Same CSP and cache headers as `/activity`. See [Hub Fleet activity page](../hub-fleet-activity.md#public-safe-view) | `pkg/hub/activity_public.go` | | `GET` | `/activity/assets/{name}` | Public | The shared tracker assets the Fleet activity page loads, byte-identical: the row asset (`pkg/webassets/trackerrow`, [#16062](https://github.com/hivecommons/hive/issues/16062)) and the `am.*` URL-view and display-playlist helpers (`pkg/webassets/trackerview`, [#16064](https://github.com/hivecommons/hive/issues/16064)); only names in `trackerrow.Files` or `trackerview.Files`, else `404`. `Cache-Control: no-cache` + content `ETag` | `pkg/hub/activity_page.go` | -| `POST` | `/api/shared-agent-pool/leases` | Hub handler-specific | Acquire or renew a shared CLI session lease | `pkg/hub/server.go:1748` | -| `PUT` | `/api/shared-agent-pool/leases` | Hub handler-specific | Renew an active shared CLI session lease | `pkg/hub/server.go:1749` | -| `DELETE` | `/api/shared-agent-pool/leases` | Hub handler-specific | Release a shared CLI session lease | `pkg/hub/server.go:1750` | -| `POST` | `/api/heartbeat` | Hub handler-specific | Heartbeat. A request carrying `X-Hive-Activity-Update` is an activity-only update ([#15963](https://github.com/hivecommons/hive/issues/15963)): body `{"activity_update":{"hive_id","activity"}}`, the same per-hive bearer, accepted only for a hive already in the registry (`404` otherwise); it replaces the in-memory activity digest and nothing else, and the signed response carries `ok`, `activity_etag` and `activity_events_ack` (the highest activity event sequence number the hub holds for the hive, [#15964](https://github.com/hivecommons/hive/issues/15964); full heartbeat responses carry it too) | `pkg/hub/server.go:1751` | -| `POST` | `/api/task-status` | Hub handler-specific | Task Status | `pkg/hub/server.go:1752` | -| `GET` | `/api/registry` | Hub handler-specific | Registry | `pkg/hub/server.go:1763` | -| `GET` | `/api/hub/leaderboard` | Hub handler-specific | Leaderboard | `pkg/hub/server.go:1764` | -| `GET` | `/api/hub/stats` | Hub handler-specific | Stats | `pkg/hub/server.go:1765` | -| `GET` | `/api/fleet-stats` | Hub handler-specific | Fleet Stats | `pkg/hub/server.go:1766` | +| `POST` | `/api/shared-agent-pool/leases` | Hub handler-specific | Acquire or renew a shared CLI session lease | `pkg/hub/server.go:1755` | +| `PUT` | `/api/shared-agent-pool/leases` | Hub handler-specific | Renew an active shared CLI session lease | `pkg/hub/server.go:1756` | +| `DELETE` | `/api/shared-agent-pool/leases` | Hub handler-specific | Release a shared CLI session lease | `pkg/hub/server.go:1757` | +| `POST` | `/api/heartbeat` | Hub handler-specific | Heartbeat. A request carrying `X-Hive-Activity-Update` is an activity-only update ([#15963](https://github.com/hivecommons/hive/issues/15963)): body `{"activity_update":{"hive_id","activity"}}`, the same per-hive bearer, accepted only for a hive already in the registry (`404` otherwise); it replaces the in-memory activity digest and nothing else, and the signed response carries `ok`, `activity_etag` and `activity_events_ack` (the highest activity event sequence number the hub holds for the hive, [#15964](https://github.com/hivecommons/hive/issues/15964); full heartbeat responses carry it too) | `pkg/hub/server.go:1758` | +| `POST` | `/api/task-status` | Hub handler-specific | Task Status | `pkg/hub/server.go:1759` | +| `GET` | `/api/registry` | Hub handler-specific | Registry | `pkg/hub/server.go:1770` | +| `GET` | `/api/hub/leaderboard` | Hub handler-specific | Leaderboard | `pkg/hub/server.go:1771` | +| `GET` | `/api/hub/stats` | Hub handler-specific | Stats | `pkg/hub/server.go:1772` | +| `GET` | `/api/fleet-stats` | Hub handler-specific | Fleet Stats | `pkg/hub/server.go:1773` | | `GET` | `/api/hub/evidence` | Hub admin | Read-only cross-hive evidence drill-down from each spoke's stored heartbeat `evidence` summary ([#12116](https://github.com/hivecommons/hive/issues/12116)); non-admins get `403`. Returns `spokes`, `reporting`, pooled `calibration` (bucket 0–5 → `{n, bad, rate, lo, hi}`; the hub view folds the six buckets into four reviewer-confidence ranges), `findings[]` (the plain-language *What this says* sentences, generated from the same fields; [#12456](https://github.com/hivecommons/hive/issues/12456)), `by_acmm_level` (`level`, `hives`, `merged_90d`, `revert_rate`, and `time_to_merge` — `median_of_hive_medians_s`, `hives`, `n` — the median of the level's per-hive median time-to-merge, omitted when no hive at the level reports one; [#12464](https://github.com/hivecommons/hive/issues/12464)), `family_false_approve_rate` (`same`/`cross`/`mixed`, hives classed by their reviewer vs author family sets), fleet `record_coverage` (merged PRs with an evidence record over `prs_merged_90d`), fleet panels ([#12370](https://github.com/hivecommons/hive/issues/12370)) `outcomes_by_author_kind[]` (`kind`, `hives`, `n`, `merged`, `reverted`, `merge_rate`, `revert_rate`), `write_counters[]` (`tier`, `hives`, `attempted`, `blocked`, `block_rate`) and `shadow_gate` (`review_gate` / `intent` → `held` / `passed` outcomes with `revert_rate`), each omitted while no spoke reports it and repeated per `by_acmm_level[]` row, and per hive `hives[]` with `acmm_level`, `prs_merged_90d`, `record_coverage`, `family_mix`, false-approve / false-block / revert rates, families, a calibration series the raw `counts` behind them (including `by_author_kind`) so the drill-down can re-pool a hive / ACMM-level selection client-side, and `criteria[]`: the per-axis criteria table derived hub-side from the quadrant criteria (`axis`, `label`, `scored`, `criteria[]` with `name`, `label`, `status` `met`/`unmet`/`not_reported`, `display`/`score` when measured, `reason` otherwise; offline, evidence-less or unscored hives are `not_reported`, never `unmet`; [#12463](https://github.com/hivecommons/hive/issues/12463)). Every rate is `{n, rate, lo, hi}` with a 95% Wilson interval; `rate`/`lo`/`hi` are null when `n` < 10. Window selection ([RFC #12062](https://github.com/hivecommons/hive/issues/12062)): `?hours=24|168|720` pools each hive's matching heartbeat `windows` block instead of its 90-day block (any other or absent `hours` → 90 days); `?from=YYYY-MM-DD|RFC3339[&to=…]` (date-only `to` is inclusive, `to` defaults to now, `to` ≥ `from`, span ≤ 366 days, else `400`) is served from the smallest reported window that reaches back to `from` and flagged `window.approximate`. The response carries `window` (`hours`, `key`, `label`, `from`, `to`, `approximate`, `coverage_available` — `record_coverage` is only computed for the 90-day window because `prs_merged_90d` is a 90-day count), `window_unreported` and per-hive `window_unreported` for hives whose spoke predates per-window reporting. `?hive=[,…]` (repeatable) and `?level=` restrict the registry server-side to those hives / that ACMM level before pooling. Hub admins open it from the Fleet quadrant card or its **🔬 Evidence** button ([#12117](https://github.com/hivecommons/hive/issues/12117)), pick a window (24h / 7d / 30d / 90d / custom), filter it by hive and ACMM level and read each section's ⓘ explainer; the per-hive author-kind revert counts behind the *Revert rate by author kind* panel are `hives[].counts.by_author_kind` ([#12326](https://github.com/hivecommons/hive/issues/12326)). Never consulted by merge decisions | `pkg/hub/evidence.go` | | `GET` | `/api/hub/evidence/report` | Hub admin | Plain-language report of the hub evidence view ([#12456](https://github.com/hivecommons/hive/issues/12456)), rendered server-side from the same `EvidenceView` as `/api/hub/evidence` so every figure reconciles with the modal; non-admins get `403`. Same `hours=` / `from=` / `to=` window and `hive=` / `level=` filter parameters. `format=md` (default, `text/markdown`) or `format=html` (standalone page, inline CSS, no scripts or external assets, hive names escaped); any other value → `400`. `download=1` sets `Content-Disposition: attachment; filename="hive-evidence---.md|html"`, otherwise inline. Sections: Summary (the generated findings), What we measured and why, The numbers, How to read these, What is not reported yet and why, Grade per section, footer link to `review-evidence.md`. `Cache-Control: no-store`. Read-only; never consulted by merge decisions | `pkg/hub/evidence_report.go` | | `GET` | `/api/hub/activity` | Signed-in user | Read-only hub-wide activity view ([#15951](https://github.com/hivecommons/hive/issues/15951), RFC [#15861](https://github.com/hivecommons/hive/issues/15861)) merged from each spoke's heartbeat `activity` digest, held in hub memory only. Authorization is per user and per hive before aggregation: hub admins see every registered hive; everyone else only hives they own or hold an unexpired grant for; anonymous → `401`. Returns `hives[]` (`hive_id`, `name`, `status` `ok`/`stale`/`not_reporting`/`missing`, `reporting`, `stale` once the block is older than two heartbeats, `last_heartbeat`, `received_at`, `generated_at`, `totals` — `items`, `by_stage`, `by_modifier`, counting private and truncated items too — `truncated` and `events_dropped`), `items[]` (`hive_id`, `repo`, `n`, `kind`, `title`, `author_kind`, `stage`, `stage_n`, `since`, `mods`, `private`) ordered needs-human/needs-decision, stalled, furthest stage, oldest, a response-level `truncated` (cap 2000), and `events[]` ([#15964](https://github.com/hivecommons/hive/issues/15964): `hive_id`, `received_at`, `seq`, `type` `opened`/`stage`/`merged`, `repo`, `n`, `kind`, `title`, `from`, `stage`, `at`, `private`), oldest first, from a fleet-wide in-memory ring of at most 1,000 events, none held longer than 24 hours (ageing out is not counted in `events_dropped`; when full, the oldest event of the hive holding the most events is dropped, counted in that hive's `events_dropped`, so one hive cannot evict the others' events). Events are filtered per user and per hive exactly like items, before aggregation, so an event for a hive the viewer cannot see is never delivered; `mod`/`author_kind` filters keep only events whose item is still on the hive's board. A spoke block carries at most 200 events; drops are disclosed per hive as `events_dropped`. New events change the `ETag`. Each digest carries at most 100 items by default and never more than the 250-item hub ceiling; private-repository items appear only when the spoke opts in. Filters: `hive`, `repo`, `stage`, `mod`, `author_kind`, `q` (title substring or `#`), each ≤ 200 characters (else `400`). `ETag` + `If-None-Match` → `304`; `Cache-Control: private, no-cache`, `Vary: Cookie, Authorization`. No actions | `pkg/hub/activity.go` | | `GET` | `/api/hub/activity/public` | Public | Public-safe activity view ([#15966](https://github.com/hivecommons/hive/issues/15966)): the `/api/hub/activity` shape for anyone, never reading a cookie, token or `Authorization` header (credentials do not change the response). Only hives the registry lists as public on github.com, and only items and events not marked `private`; `totals`, per-hive `truncated` and `events_dropped`, hive owners and per-user grants are omitted. Same filters (`400` on an invalid one). The body is built at most once per 2 s per parsed filter and shared by every anonymous caller, so rows, `generated_at` and the `ETag` change at most every 2 s; query strings that parse to the default view share one entry. Toggling a hive's visibility drops the cache at once. `ETag` + `If-None-Match` → `304`; `Cache-Control: no-cache` | `pkg/hub/activity_public.go` | -| `GET` | `/api/hub/version` | Hub handler-specific | Hub Version | `pkg/hub/server.go:1774` | -| `DELETE` | `/api/hub/registry/{id}` | Hub handler-specific | Registry Delete | `pkg/hub/server.go:1784` | -| `GET` | `/api/hub/selfservice/status` | Hub auth | Self-service registration status: `{enabled, max_per_user, count, expiry}`; always answers so the dashboard can decide whether **Register your own hive** opens the form or the self-hosting guide ([#18060](https://github.com/hivecommons/hive/issues/18060)) | `pkg/hub/server.go:1788` | -| `GET` | `/api/hub/selfservice/hives` | Hub auth | The caller's own self-service (`byo-`) hives: `{hives[]}` of `{id, owner, org, repos, dashboard_url, created_at, ownership_checked_at, last_accepted_heartbeat_at, online}`; never a key or a hash. `404` while `HIVE_HUB_SELFSERVICE_REGISTRATION` is off | `pkg/hub/server.go:1789` | -| `POST` | `/api/hub/selfservice/hives` | Hub auth | Register a self-hosted hive: body `{name, org, repos[], dashboard_url}`; the ID is `byo---` (the double hyphen keeps IDs unambiguous, since logins contain hyphens but never `--`). Proves `permissions.admin` on every `owner/name` repo with the caller's own stored GitHub token (`403` naming the refused repo, nothing stored); `400` invalid or reserved name, a login that contains `--` or starts or ends with `-`, bare-org or `.`/`..` repo, or non-`https`/private `dashboard_url`; `409` ID already registered; `429` over `HIVE_HUB_SELFSERVICE_MAX_PER_USER` (enforced again under the store lock, so concurrent registrations cannot exceed it). `201 {hive, key, hub_url, snippets{docker_compose, podman, kubernetes}}`: the raw key appears here once and only its hash is stored. `404` while the flag is off | `pkg/hub/server.go:1790` | +| `GET` | `/api/hub/version` | Hub handler-specific | Hub Version | `pkg/hub/server.go:1781` | +| `DELETE` | `/api/hub/registry/{id}` | Hub handler-specific | Registry Delete | `pkg/hub/server.go:1791` | +| `GET` | `/api/hub/selfservice/status` | Hub auth | Self-service registration status: `{enabled, max_per_user, count, expiry}`; always answers so the dashboard can decide whether **Register your own hive** opens the form or the self-hosting guide ([#18060](https://github.com/hivecommons/hive/issues/18060)) | `pkg/hub/server.go:1795` | +| `GET` | `/api/hub/selfservice/hives` | Hub auth | The caller's own self-service (`byo-`) hives: `{hives[]}` of `{id, owner, org, repos, dashboard_url, created_at, ownership_checked_at, last_accepted_heartbeat_at, online}`; never a key or a hash. `404` while `HIVE_HUB_SELFSERVICE_REGISTRATION` is off | `pkg/hub/server.go:1796` | +| `POST` | `/api/hub/selfservice/hives` | Hub auth | Register a self-hosted hive: body `{name, org, repos[], dashboard_url}`; the ID is `byo---` (the double hyphen keeps IDs unambiguous, since logins contain hyphens but never `--`). Proves `permissions.admin` on every `owner/name` repo with the caller's own stored GitHub token (`403` naming the refused repo, nothing stored); `400` invalid or reserved name, a login that contains `--` or starts or ends with `-`, bare-org or `.`/`..` repo, or non-`https`/private `dashboard_url`; `409` ID already registered; `429` over `HIVE_HUB_SELFSERVICE_MAX_PER_USER` (enforced again under the store lock, so concurrent registrations cannot exceed it). `201 {hive, key, hub_url, snippets{docker_compose, podman, kubernetes}}`: the raw key appears here once and only its hash is stored. `404` while the flag is off | `pkg/hub/server.go:1797` | | `GET` | `/api/hub/v1` | Public | Hub registration protocol v1 discovery ([#18642](https://github.com/hivecommons/hive/issues/18642)): `{protocol, version, base_path, operations[], patch_fields[], limits{max_repos, max_per_user, expiry, key_rotation_overlap}, proof_methods[]}`. Readable without signing in so a client learns the limits before it hits a `4xx`. `proof_methods` is `["hub-session-token"]`: ownership is proven with the GitHub token stored at hub sign-in. `max_repos` is `HIVE_HUB_SELFSERVICE_MAX_REPOS`. `404` while `HIVE_HUB_SELFSERVICE_REGISTRATION` is off | `pkg/hub/selfservice_v1_api.go` | | `POST` | `/api/hub/v1/registrations` | Hub auth | Register a self-hosted hive as a resource ([#18642](https://github.com/hivecommons/hive/issues/18642)): the same body, proof and refusals as `POST /api/hub/selfservice/hives` (including `429` over `HIVE_HUB_SELFSERVICE_MAX_PER_USER`), except that a repo list over `HIVE_HUB_SELFSERVICE_MAX_REPOS` answers `413 {error, limit: "max_repos", max_repos}` instead of `400`. `201 {registration, key, hub_url, snippets}` with `Location` and a strong `ETag` header; `registration` is the legacy view plus `etag` and `repo_proofs[{repo, checked_at}]` (when the owner's admin permission on each repo was last proven). The raw key appears once. `404` while `HIVE_HUB_SELFSERVICE_REGISTRATION` is off | `pkg/hub/selfservice_v1_api.go` | | `GET` | `/api/hub/v1/registrations` | Hub auth | The caller's own registrations: `{registrations[]}`, each with its `etag` and `repo_proofs`; never a key or a hash | `pkg/hub/selfservice_v1_api.go` | @@ -65,26 +65,26 @@ Part of the [Dashboard REST API reference](../api-reference.md); see that page f | `PATCH` | `/api/hub/v1/registrations/{id}` | Hub auth | Owner only, by hub session (a registration key is not accepted): change `repos` and/or `dashboard_url` in place with the heartbeat key kept valid ([#18642](https://github.com/hivecommons/hive/issues/18642)). Body: `repos` is a full list (replacement) or `{add[], remove[]}`; `dashboard_url` is an `https` URL or `""` to clear it. `If-Match` is required: `428` when absent, `412` carrying the current `ETag` when stale (checked before any GitHub call and again under the store lock). Only added repos are proven (`403` naming the first refused repo, nothing applied); kept repos keep their proof time and removed repos lose theirs. `org`, the owner, the ID and the keys are not editable: a body naming `org` or any other field is `400`, as is a list that would leave no repo. Over `HIVE_HUB_SELFSERVICE_MAX_REPOS` is `413 {error, limit: "max_repos", max_repos}`. Removing a repo the registration does not list is ignored; an edit that changes nothing answers `200` with the current `ETag` and does not bump the revision. `200 {registration}` with the new `ETag`; the key never appears. `404` for a missing hive, another user's hive and a hub admin who does not own it; `404` while `HIVE_HUB_SELFSERVICE_REGISTRATION` is off | `pkg/hub/selfservice_v1_patch.go` | | `POST` | `/api/hub/v1/registrations/{id}/keys` | Hub auth | Owner only: rotate the heartbeat key; the previous key keeps authenticating for the 10 minute overlap. `201 {id, key, hub_url, overlap, snippets}` (the legacy rotate answers `200`) with the new `ETag`; the raw key appears once. Optional `If-Match` (`412` when stale). A body asking for a grace window is refused with `400`: caller-chosen grace is a later rollout step | `pkg/hub/selfservice_v1_api.go` | | `DELETE` | `/api/hub/v1/registrations/{id}` | Hub auth | Owner or hub admin: revoke the key and remove the registry entry. `200 {removed, id}`. Optional `If-Match` (`412` when stale, carrying the current `ETag`) | `pkg/hub/selfservice_v1_api.go` | -| `POST` | `/api/hub/selfservice/hives/{id}/rotate` | Hub auth | Owner only: mint a new heartbeat key for `{id}`; the previous key keeps authenticating for `overlap` (10 minutes). `200 {id, key, hub_url, overlap, snippets}`, raw key once. Another user's hive, a missing hive and a hub admin who does not own it all get `404` | `pkg/hub/server.go:1791` | -| `DELETE` | `/api/hub/selfservice/hives/{id}` | Hub auth | Owner or hub admin: revoke the key and remove the registry entry. `200 {removed, id}`; another user's hive is `404` | `pkg/hub/server.go:1792` | -| `POST` | `/api/contribute/register` | Hub handler-specific | Contribute Proxy | `pkg/hub/server.go:1796` | -| `GET` | `/api/contribute/status` | Hub handler-specific | Contribute Status | `pkg/hub/server.go:1797` | -| `GET` | `/api/contribute/ws` | Hub handler-specific | Contribute WSProxy | `pkg/hub/server.go:1798` | -| `POST` | `/api/github/webhook` | Hub handler-specific | GitHub Webhook | `pkg/hub/server.go:1799` | +| `POST` | `/api/hub/selfservice/hives/{id}/rotate` | Hub auth | Owner only: mint a new heartbeat key for `{id}`; the previous key keeps authenticating for `overlap` (10 minutes). `200 {id, key, hub_url, overlap, snippets}`, raw key once. Another user's hive, a missing hive and a hub admin who does not own it all get `404` | `pkg/hub/server.go:1798` | +| `DELETE` | `/api/hub/selfservice/hives/{id}` | Hub auth | Owner or hub admin: revoke the key and remove the registry entry. `200 {removed, id}`; another user's hive is `404` | `pkg/hub/server.go:1799` | +| `POST` | `/api/contribute/register` | Hub handler-specific | Contribute Proxy | `pkg/hub/server.go:1803` | +| `GET` | `/api/contribute/status` | Hub handler-specific | Contribute Status | `pkg/hub/server.go:1804` | +| `GET` | `/api/contribute/ws` | Hub handler-specific | Contribute WSProxy | `pkg/hub/server.go:1805` | +| `POST` | `/api/github/webhook` | Hub handler-specific | GitHub Webhook | `pkg/hub/server.go:1806` | | `GET` | `/api/mail/hosts` | hive-mx bearer (`HIVE_HUB_MAIL_TOKEN`) | Hosts the mail receiver accepts `hive@` for ([#18395](https://github.com/hivecommons/hive/issues/18395)): `{hosts[]}`, sorted, lower-case: every hosted hive's subdomain host, vanity URL host and reported dashboard URL host. A host claimed by two hives is left out. `401` bad token, `404` while `HIVE_HUB_MAIL_TOKEN` is unset. `Cache-Control: no-store` | `pkg/hub/mail_relay.go` | | `POST` | `/api/mail/inbound` | hive-mx bearer (`HIVE_HUB_MAIL_TOKEN`) | One parsed mail delivery from hive-mx (`pkg/mailin/wire.Delivery`): JSON, or `multipart/form-data` for a large files delivery, routed by the `X-Hive-Mail-Delivery`, `X-Hive-Mail-Kind` and `X-Hive-Mail-Recipient` headers (at most 48 MiB, `HIVE_HUB_MAIL_MAX_BODY_BYTES`, this route only; without the headers, JSON of at most 8 MiB). The body is spooled to disk, never held whole in memory. The hub matches the recipient host to a hosted hive, signs the body byte for byte with that hive's relay key (`X-Hive-Relay-Timestamp`, `X-Hive-Relay-Signature-256` bound to `X-Hive-Relay-Delivery`) and forwards it to the spoke's `/api/meetings/invites`, `/api/inbox/email`, `/api/meetings/calendar-emails/proof` or `/api/inbox/email/report` (a refusal report, [#18699](https://github.com/hivecommons/hive/issues/18699)) by `kind`, behind the webhook relay's private-address guard and without following redirects. A files delivery that is multipart or larger than one piece (`HIVE_HUB_MAIL_CHUNK_BYTES`, 768 KiB) goes to `/api/inbox/email/chunk` in signed pieces instead ([#18564](https://github.com/hivecommons/hive/issues/18564)). Stores nothing once relayed; logs the delivery id only. `202` delivered; `400` invalid delivery; `404` no such hive; `413` too large; `422` the spoke refused it (permanent); `502`/`503` hive not reachable yet or unavailable (hive-mx asks the sender to retry) | `pkg/hub/mail_relay.go` | -| `GET` | `/gh-setup` | Hub handler-specific | GitHub App Setup Router | `pkg/hub/server.go:1802` | -| `GET` | `/learn` | Hub handler-specific | Static HTML page | `pkg/hub/server.go:1803` | -| `GET` | `/get-started` | Hub handler-specific | Static HTML page | `pkg/hub/server.go:1804` | -| `GET` | `/api/docs` | Hub handler-specific | Static HTML page | `pkg/hub/server.go:1807` | -| `GET` | `/api/reading-list` | Hub handler-specific | Reading List | `pkg/hub/server.go:1810` | -| `GET` | `/reading` | Hub handler-specific | Static HTML page | `pkg/hub/server.go:1811` | -| `GET` | `/cncf-reference-architecture` | Hub handler-specific | Static HTML page | `pkg/hub/server.go:1818` | -| `GET` | `/cncf-reference-architecture/console` | Hub handler-specific | Static HTML page | `pkg/hub/server.go:1819` | -| `GET` | `/cncf-reference-architecture/bluefin` | Hub handler-specific | Static HTML page | `pkg/hub/server.go:1820` | -| `GET` | `/{$}` | Hub handler-specific | Static HTML page | `pkg/hub/server.go:1837` | -| `GET` | `/og-card.png` | Hub handler-specific | OGCard | `pkg/hub/server.go:1842` | -| `GET` | `/` | Public | Static asset fallback (`http.FileServerFS` over the embedded `static/` tree) for any path no other route claims | `pkg/hub/server.go:1844` | +| `GET` | `/gh-setup` | Hub handler-specific | GitHub App Setup Router | `pkg/hub/server.go:1809` | +| `GET` | `/learn` | Hub handler-specific | Static HTML page | `pkg/hub/server.go:1810` | +| `GET` | `/get-started` | Hub handler-specific | Static HTML page | `pkg/hub/server.go:1811` | +| `GET` | `/api/docs` | Hub handler-specific | Static HTML page | `pkg/hub/server.go:1814` | +| `GET` | `/api/reading-list` | Hub handler-specific | Reading List | `pkg/hub/server.go:1817` | +| `GET` | `/reading` | Hub handler-specific | Static HTML page | `pkg/hub/server.go:1818` | +| `GET` | `/cncf-reference-architecture` | Hub handler-specific | Static HTML page | `pkg/hub/server.go:1825` | +| `GET` | `/cncf-reference-architecture/console` | Hub handler-specific | Static HTML page | `pkg/hub/server.go:1826` | +| `GET` | `/cncf-reference-architecture/bluefin` | Hub handler-specific | Static HTML page | `pkg/hub/server.go:1827` | +| `GET` | `/{$}` | Hub handler-specific | Static HTML page | `pkg/hub/server.go:1844` | +| `GET` | `/og-card.png` | Hub handler-specific | OGCard | `pkg/hub/server.go:1849` | +| `GET` | `/` | Public | Static asset fallback (`http.FileServerFS` over the embedded `static/` tree) for any path no other route claims | `pkg/hub/server.go:1851` | | `GET` | `/api/marketplace/workflows` | Public | Workflow-model marketplace index ([#17620](https://github.com/hivecommons/hive/issues/17620) phase 7): `{generatedAt, entries:[{name, version, hash, tier, description, source, states, columns, publishedAt, publishedBy, sourceSHA?, newerIngestAvailable, adoption:{}, ratings:{}, benchmarks:{}}]}`. Strong `ETag` (sha256 of the body) + `Cache-Control: no-cache`; `If-None-Match` → `304`. Tiers: `built-in`, `curated`, `ingested-unreviewed`, `local-published`. `404` when `HIVE_HUB_MARKETPLACE_DISABLED` is set | `pkg/hub/marketplace.go:36` | | `GET` | `/api/marketplace/workflows/{name}/{version}/bundle` | Public | One immutable signed bundle (`{apiVersion, model, profile, hash, createdAt, createdBy, tier}`), `Cache-Control: immutable`; `404` unknown, `400` malformed key | `pkg/hub/marketplace.go:37` | | `POST` | `/api/marketplace/workflows` | Spoke heartbeat bearer + `X-Hive-ID`, or Hub admin session | Publish a bundle. Body = bundle (≤ 4 MiB); the hub re-verifies the hash and recompiles the model before storing. Publisher may assert tier `local`/`local-published` (normalised to `local-published`) or `ingested-unreviewed`; `curated`/`built-in` require a hub admin session (`403` otherwise). `201 {published, created:true, entry}`; identical republish `200 created:false`; different content under an existing `name@version` `409`; verification failure `422`; unknown hive `403`; no hub secret `503` | `pkg/hub/marketplace.go:38` |