✨ add Spek recheck outward discovery sources

changelog.d/added-10683.md

@@ -0,0 +1 @@
+- add Spek recheck outward discovery sources (#10683)

src/changelog.d/added-spek-recheck-discovery.md

@@ -0,0 +1 @@
+- Add opt-in outward discovery sources for Spektacular recheck revisions, surfacing bounded external evidence in drift metadata and the Campaigns dashboard.

src/docs/adr/0020-spek-continuous-convergence.md

@@ -35,10 +35,22 @@ Each revision records drift evidence:
 - `prior_revision`
 - `delta_count`
 - `recheck_reason` (`cadence` or `manual`)
-
-The evidence source interface is `RecheckEvidenceSource`. This ADR accepts only
-the codebase-head source for now; external upstream/competitor/standards
-discovery remains a follow-up.
+- optional `external[]`, `external_count`, and `sources_failed[]` discovery
+  evidence when outward discovery is enabled
+
+The evidence source interface is `RecheckEvidenceSource`.
+
+### Discovery sources
+
+The outward-looking half is opt-in under
+`runs.spektacular.recheck.discovery`. Operators declare bounded sources of kind
+`upstream_release`, `repo_activity`, `standards_feed`, or `landscape`; Hive
+rejects undeclared kinds and hosts outside `variables.security.http_allowlist`.
+Discovery runs before the recheck `spec` stage generator, records evidence on
+the linked revision's `drift.external` block, and includes that evidence in the
+spec-stage prompt as context. It is read-only and non-fatal: source failures are
+recorded in `sources_failed` and never block the recheck. Evidence is never
+auto-applied; the existing spec and plan human checkpoints remain authoritative.
 
 ## Consequences
 

src/docs/api-reference.md

@@ -37,8 +37,8 @@ Auth levels are derived from dashboard middleware (`isPublicPath`, dashboard tok
 | `POST` | `/api/runs/{key}/checkpoint` | Owner only | Resolve a compact checkpoint decision. Body `{"action":"approve"|"reject","gen":<lease generation>}`; refuses stale generations with 409 and non-owner callers with 403 before mutating the plan. | `pkg/dashboard/api.go:83` |
 | `GET` | `/api/runs/{key}` | Dashboard auth/session | One run by URL-escaped canonical run key (`owner/repo%23number`); early lease-shaped keys (`repo!run-key:stage`) and queued Wavefront run-stage keys (`owner/repo!graph:node`) are also accepted. Includes timeline-derived stage history in the same `Run` shape as `/api/runs` when a lease/timeline exists. History entries that came from an owner reset carry the owner's `reason`. When the key belongs to a wired convergence campaign, the detail response also includes an optional `burndown` block with `source`, `satisfied`, `remaining`, `unknown`, and `scope`; the list endpoint omits this block. | `pkg/dashboard/api.go:84` |
 | `POST` | `/api/runs/{key}/reset` | Owner only | Move a run's lease back to an earlier stage (for example `implement` to `plan` after a rejected plan). Body `{"to": "<stage>", "reason": "<why>"}`, both required. Mints a new generation so the relay holding the old one can no longer resume, persists before answering, and records the reason on the agent audit sink, the lifecycle timeline, and the `stage_completed` hook payload (`attrs.reset = "true"`). Returns `{ok, key, stage_from, stage, gen, reason}`. 400 for the same or a later stage or a missing reason, 404 when no live staged lease holds the key, 409 when the lease expired, 500 when the registry could not be written. | `pkg/dashboard/api.go:85` |
-| `GET` | `/api/campaigns` | Dashboard auth/session | List resumable campaigns from archived Inception sessions and retained Spektacular (Spek) runs. Query `search`/`q`, `repo`, `stage`, `status`, and `owner`; each campaign is keyed by stable spec/session id and includes title, source, repos, current stage/step, artifact links, linked issues/PRs, contributors, last activity, status, engine, type, run link when applicable, and recheck status (`enabled`, `interval`, `last_at`, `next_at`, `in_flight`, `last_delta_count`). | `pkg/dashboard/api.go:86` |
-| `GET` | `/api/campaigns/{id}` | Dashboard auth/session | Return one campaign by URL-escaped stable campaign id or run key, using the same campaign shape as `/api/campaigns`; recheck revisions may include a `drift` block with `codebase_changed`, `prior_revision`, `delta_count`, and `recheck_reason`. | `pkg/dashboard/api.go:87` |
+| `GET` | `/api/campaigns` | Dashboard auth/session | List resumable campaigns from archived Inception sessions and retained Spektacular (Spek) runs. Query `search`/`q`, `repo`, `stage`, `status`, and `owner`; each campaign is keyed by stable spec/session id and includes title, source, repos, current stage/step, artifact links, linked issues/PRs, contributors, last activity, status, engine, type, run link when applicable, and recheck status (`enabled`, `interval`, `last_at`, `next_at`, `in_flight`, `last_delta_count`, `external_count`, `sources_failed`). | `pkg/dashboard/api.go:86` |
+| `GET` | `/api/campaigns/{id}` | Dashboard auth/session | Return one campaign by URL-escaped stable campaign id or run key, using the same campaign shape as `/api/campaigns`; recheck revisions may include a `drift` block with `codebase_changed`, `prior_revision`, `delta_count`, `recheck_reason`, `external[]` (`source`, `kind`, `title`, `url`, `published_at`, `summary`), `external_count`, and `sources_failed`. | `pkg/dashboard/api.go:87` |
 | `POST` | `/api/campaigns/{id}/resume` | Owner only | Resume or pick up a campaign. Archived Inception campaigns take a durable campaign lease, restore their saved state and wiki working files; Spek campaigns return the retained run detail plus a CLI status command that reloads state from Spek artifacts. Body may include `{ "surface": "chat"|"contributor"|"cli"|"dashboard" }`; 409 means another user holds the lease. | `pkg/dashboard/api.go:88` |
 | `POST` | `/api/campaigns/{id}/release` | Owner only | Release the caller's Inception campaign lease and write a `campaign_release` audit entry so another user can continue it. Run-backed Spektacular campaigns return `409 Conflict`; this endpoint never revokes their run-lifecycle stage leases. | `pkg/dashboard/api.go:89` |
 | `POST` | `/api/campaigns/{id}/revise` | Owner only | Create a linked revision campaign (`revision_of`, `revision`) for revisiting shipped or parked Inception/Spek work without overwriting the previous campaign. The new revision is leased to the caller and audit-logged as `campaign_revise`. | `pkg/dashboard/api.go:90` |

src/docs/spektacular.md

@@ -274,14 +274,64 @@ imports the first capped set and records `campaign_recheck_delta_capped`.
 Each revision carries a `drift` evidence block with `codebase_changed`,
 `prior_revision`, `delta_count`, and `recheck_reason` (`cadence` or `manual`).
 The codebase signal compares the prior recorded head SHA with the current Git
-HEAD when available. Outward-looking external-source discovery (upstreams,
-competitors, standards trackers, landscape entries) is deliberately not in
-this PR; the Go extension point is `RecheckEvidenceSource`, with only the
-codebase-head source implemented. Follow-up: [#10667](https://github.com/hivecommons/hive/issues/10667).
+HEAD when available.
+
+### Outward-looking discovery
+
+Outward discovery is opt-in under `runs.spektacular.recheck.discovery`. Hive
+only reads operator-declared sources, bounded by `timeout` (default `30s`),
+each source's `max_items` (default `20`), and `max_total_items` (default
+`100`). It never searches the web, never crawls arbitrary or competitor
+websites, never applies findings automatically, and rejects any configured
+host that is not listed in `variables.security.http_allowlist`. GitHub-backed
+sources use the existing authenticated GitHub client; feed sources use GET
+through the normal proxy/egress path.
+
+Supported source kinds are:
+
+- `upstream_release`: GitHub releases for a named `owner/repo` since the prior
+  revision timestamp.
+- `repo_activity`: merged PRs and tagged releases for a named `owner/repo`
+  since the prior timestamp.
+- `standards_feed`: an Atom or RSS URL such as a spec repo's `releases.atom`
+  or a standards-body feed.
+- `landscape`: GitHub repositories linked from `docs/landscape.md`, checked
+  for new releases using the same release reader as `upstream_release`.
+
+Example:
+
+```yaml
+runs:
+  spektacular:
+    recheck:
+      discovery:
+        enabled: true
+        timeout: 30s
+        max_total_items: 100
+        sources:
+          - {kind: upstream_release, name: kubernetes, url_or_repo: kubernetes/kubernetes, max_items: 10}
+          - {kind: repo_activity, name: competitor-runtime, url_or_repo: example/runtime, allow_prerelease: true}
+          - {kind: standards_feed, name: ietf-http, url_or_repo: https://www.ietf.org/archive/id/atom.xml}
+          - {kind: landscape, name: project-landscape, max_items: 20}
+variables:
+  security:
+    http_allowlist: [api.github.com, www.ietf.org]
+```
+
+Findings are recorded on the revision as
+`drift.external[] = {source, kind, title, url, published_at, summary}` with
+summaries capped at 280 characters, plus `external_count` and
+`sources_failed[] = {name, reason}`. A failed source is non-fatal: Hive records
+the typed reason and continues with the rest. The same evidence is included in
+the `spec` stage assignment prompt as context so the spec generator can decide
+what, if anything, belongs in the revised spec; the spec and plan human
+checkpoints remain the only approval gates.
 
 Campaign list/detail JSON includes
-`recheck: {enabled, interval, last_at, next_at, in_flight, last_delta_count}`;
-the dashboard Campaigns panel renders the cadence and provides an owner-only
+`recheck: {enabled, interval, last_at, next_at, in_flight, last_delta_count,
+external_count, sources_failed}`; recheck revisions also include the `drift`
+block above. The dashboard Campaigns panel renders the cadence, shows
+`N external signals` with an expandable list, and provides an owner-only
 Recheck action.
 
 ## Defensive handling of the open questions

src/hive.yaml.example

@@ -569,6 +569,15 @@ governor:
 #       enabled: false
 #       default_interval: 168h
 #       max_delta_tasks: 50
+#       discovery:
+#         enabled: false
+#         timeout: 30s
+#         max_total_items: 100
+#         sources:
+#           - {kind: upstream_release, name: kubernetes, url_or_repo: kubernetes/kubernetes, max_items: 10}
+#           - {kind: repo_activity, name: competitor-runtime, url_or_repo: example/runtime, allow_prerelease: true}
+#           - {kind: standards_feed, name: ietf-http, url_or_repo: https://www.ietf.org/archive/id/atom.xml}
+#           - {kind: landscape, name: project-landscape, max_items: 20}
 #   max_stage_retries: 2
 #   max_worktrees: 8
 

src/pkg/config/runs_config.go

@@ -29,6 +29,12 @@ const (
 	// DefaultSpektacularMaxDeltaTasks caps how many changed/new plan tasks a
 	// recheck imports into the planner in one generation.
 	DefaultSpektacularMaxDeltaTasks = 50
+	// DefaultSpektacularDiscoveryTimeoutS bounds each outward discovery pass.
+	DefaultSpektacularDiscoveryTimeoutS = 30
+	// DefaultSpektacularDiscoveryMaxItems caps one configured discovery source.
+	DefaultSpektacularDiscoveryMaxItems = 20
+	// DefaultSpektacularDiscoveryMaxTotalItems caps all discovery evidence for a revision.
+	DefaultSpektacularDiscoveryMaxTotalItems = 100
 
 	// DefaultRunsWaitTimeoutSeconds is how long a run checkpoint may wait for
 	// owner approval before escalation. It also floors how far a held plan
@@ -165,6 +171,26 @@ type SpektacularRecheckConfig struct {
 	// MaxDeltaTasks caps imported changed/new tasks. Zero or negative means
 	// DefaultSpektacularMaxDeltaTasks.
 	MaxDeltaTasks int `yaml:"max_delta_tasks,omitempty" json:"max_delta_tasks,omitempty"`
+	// Discovery configures opt-in outward evidence gathered before recheck spec.
+	Discovery SpektacularRecheckDiscoveryConfig `yaml:"discovery,omitempty" json:"discovery,omitempty"`
+}
+
+// SpektacularRecheckDiscoveryConfig is the bounded, declared-source-only
+// outward discovery step for recheck spec revisions.
+type SpektacularRecheckDiscoveryConfig struct {
+	Enabled       bool                                `yaml:"enabled,omitempty" json:"enabled,omitempty"`
+	Sources       []SpektacularRecheckDiscoverySource `yaml:"sources,omitempty" json:"sources,omitempty"`
+	Timeout       time.Duration                       `yaml:"timeout,omitempty" json:"timeout,omitempty"`
+	MaxTotalItems int                                 `yaml:"max_total_items,omitempty" json:"max_total_items,omitempty"`
+}
+
+// SpektacularRecheckDiscoverySource declares one operator-approved source.
+type SpektacularRecheckDiscoverySource struct {
+	Kind            string `yaml:"kind,omitempty" json:"kind,omitempty"`
+	Name            string `yaml:"name,omitempty" json:"name,omitempty"`
+	URLOrRepo       string `yaml:"url_or_repo,omitempty" json:"url_or_repo,omitempty"`
+	AllowPrerelease bool   `yaml:"allow_prerelease,omitempty" json:"allow_prerelease,omitempty"`
+	MaxItems        int    `yaml:"max_items,omitempty" json:"max_items,omitempty"`
 }
 
 // MaxStageRetriesOrDefault returns the configured retry budget or the default.
@@ -238,3 +264,24 @@ func (s SpektacularConfig) MaxDeltaTasks() int {
 	}
 	return s.Recheck.MaxDeltaTasks
 }
+
+func (s SpektacularConfig) DiscoveryTimeout() time.Duration {
+	if s.Recheck.Discovery.Timeout <= 0 {
+		return time.Duration(DefaultSpektacularDiscoveryTimeoutS) * time.Second
+	}
+	return s.Recheck.Discovery.Timeout
+}
+
+func (s SpektacularConfig) DiscoveryMaxTotalItems() int {
+	if s.Recheck.Discovery.MaxTotalItems <= 0 {
+		return DefaultSpektacularDiscoveryMaxTotalItems
+	}
+	return s.Recheck.Discovery.MaxTotalItems
+}
+
+func (s SpektacularRecheckDiscoverySource) EffectiveMaxItems() int {
+	if s.MaxItems <= 0 {
+		return DefaultSpektacularDiscoveryMaxItems
+	}
+	return s.MaxItems
+}

src/pkg/config/spektacular_recheck_discovery_test.go

@@ -0,0 +1,74 @@
+package config
+
+import (
+	"strings"
+	"testing"
+	"time"
+)
+
+func TestSpektacularRecheckDiscoveryDefaults(t *testing.T) {
+	var cfg SpektacularConfig
+	if got := cfg.DiscoveryTimeout(); got != 30*time.Second {
+		t.Fatalf("DiscoveryTimeout = %s, want 30s", got)
+	}
+	if got := cfg.DiscoveryMaxTotalItems(); got != 100 {
+		t.Fatalf("DiscoveryMaxTotalItems = %d, want 100", got)
+	}
+	if got := (SpektacularRecheckDiscoverySource{}).EffectiveMaxItems(); got != 20 {
+		t.Fatalf("EffectiveMaxItems = %d, want 20", got)
+	}
+}
+
+func TestValidateSpektacularRecheckDiscoveryRejectsBadKind(t *testing.T) {
+	cfg := Config{
+		Project: ProjectConfig{Org: "org"},
+		GitHub:  GitHubConfig{Token: "x"},
+		Agents:  map[string]AgentConfig{"scanner": {Backend: "claude"}},
+	}
+	cfg.Runs.Spektacular.Recheck.Discovery = SpektacularRecheckDiscoveryConfig{
+		Enabled: true,
+		Sources: []SpektacularRecheckDiscoverySource{{Kind: "search_engine", Name: "bad", URLOrRepo: "https://example.com/feed.xml"}},
+	}
+	err := cfg.validate()
+	if err == nil || !strings.Contains(err.Error(), "invalid kind") {
+		t.Fatalf("validate() = %v, want invalid kind", err)
+	}
+}
+
+func TestValidateSpektacularRecheckDiscoveryRejectsDisallowedHost(t *testing.T) {
+	cfg := Config{
+		Project: ProjectConfig{Org: "org"},
+		GitHub:  GitHubConfig{Token: "x"},
+		Agents:  map[string]AgentConfig{"scanner": {Backend: "claude"}},
+	}
+	cfg.Variables.Security.HTTPAllowlist = []string{"allowed.example.com"}
+	cfg.Runs.Spektacular.Recheck.Discovery = SpektacularRecheckDiscoveryConfig{
+		Enabled: true,
+		Sources: []SpektacularRecheckDiscoverySource{{Kind: "standards_feed", Name: "ietf", URLOrRepo: "https://blocked.example.com/feed.xml"}},
+	}
+	err := cfg.validate()
+	if err == nil || !strings.Contains(err.Error(), "blocked.example.com") || !strings.Contains(err.Error(), "http_allowlist") {
+		t.Fatalf("validate() = %v, want host allowlist error", err)
+	}
+}
+
+func TestValidateSpektacularRecheckDiscoveryAcceptsAllowlistedSources(t *testing.T) {
+	cfg := Config{
+		Project: ProjectConfig{Org: "org"},
+		GitHub:  GitHubConfig{Token: "x"},
+		Agents:  map[string]AgentConfig{"scanner": {Backend: "claude"}},
+	}
+	cfg.Variables.Security.HTTPAllowlist = []string{"api.github.com", "standards.example.com"}
+	cfg.Runs.Spektacular.Recheck.Discovery = SpektacularRecheckDiscoveryConfig{
+		Enabled: true,
+		Sources: []SpektacularRecheckDiscoverySource{
+			{Kind: "upstream_release", Name: "up", URLOrRepo: "owner/repo"},
+			{Kind: "repo_activity", Name: "activity", URLOrRepo: "owner/other"},
+			{Kind: "standards_feed", Name: "feed", URLOrRepo: "https://standards.example.com/feed.xml"},
+			{Kind: "landscape", Name: "landscape"},
+		},
+	}
+	if err := cfg.validate(); err != nil {
+		t.Fatalf("validate() = %v, want nil", err)
+	}
+}

src/pkg/config/validate.go

@@ -16,6 +16,69 @@ func (c *Config) Validate() error {
 	return c.ValidateWithOptions(ValidateOptions{RequireAgents: true})
 }
 
+func (c *Config) validateSpektacularRecheckDiscovery() error {
+	d := c.Runs.Spektacular.Recheck.Discovery
+	if !d.Enabled && len(d.Sources) == 0 {
+		return nil
+	}
+	allowedKinds := map[string]bool{
+		"upstream_release": true,
+		"repo_activity":    true,
+		"standards_feed":   true,
+		"landscape":        true,
+	}
+	for i, src := range d.Sources {
+		kind := strings.TrimSpace(src.Kind)
+		if !allowedKinds[kind] {
+			return fmt.Errorf("runs.spektacular.recheck.discovery.sources[%d]: invalid kind %q (must be upstream_release, repo_activity, standards_feed, or landscape)", i, src.Kind)
+		}
+		if strings.TrimSpace(src.Name) == "" {
+			return fmt.Errorf("runs.spektacular.recheck.discovery.sources[%d]: name is required", i)
+		}
+		if kind != "landscape" && strings.TrimSpace(src.URLOrRepo) == "" {
+			return fmt.Errorf("runs.spektacular.recheck.discovery.sources[%d]: url_or_repo is required", i)
+		}
+		for _, host := range discoverySourceHosts(kind, src.URLOrRepo) {
+			if !hostAllowedByHTTPList(host, c.Variables.Security.HTTPAllowlist) {
+				return fmt.Errorf("runs.spektacular.recheck.discovery.sources[%d]: host %q is not covered by variables.security.http_allowlist", i, host)
+			}
+		}
+	}
+	return nil
+}
+
+func discoverySourceHosts(kind, value string) []string {
+	value = strings.TrimSpace(value)
+	switch kind {
+	case "upstream_release", "repo_activity":
+		return []string{"api.github.com"}
+	case "standards_feed":
+		u, err := url.Parse(value)
+		if err != nil || u.Hostname() == "" {
+			return []string{value}
+		}
+		return []string{strings.ToLower(u.Hostname())}
+	case "landscape":
+		return []string{"api.github.com"}
+	default:
+		return nil
+	}
+}
+
+func hostAllowedByHTTPList(host string, allowlist []string) bool {
+	host = strings.ToLower(strings.TrimSpace(host))
+	if host == "" {
+		return false
+	}
+	for _, allowed := range allowlist {
+		allowed = strings.ToLower(strings.TrimSpace(allowed))
+		if allowed == host || allowed == "*" {
+			return true
+		}
+	}
+	return false
+}
+
 func (c *Config) ValidateWithOptions(opts ValidateOptions) error {
 	if c.Project.Org == "" {
 		return fmt.Errorf("project.org is required")
@@ -63,6 +126,9 @@ func (c *Config) ValidateWithOptions(opts ValidateOptions) error {
 	if err := c.validateUpstreamWatch(); err != nil {
 		return err
 	}
+	if err := c.validateSpektacularRecheckDiscovery(); err != nil {
+		return err
+	}
 	if normalized, err := ValidateSnapshotFrameAncestors(c.Dashboard.SnapshotFrameAncestors); err != nil {
 		return err
 	} else {