feat: receive and deduplicate weekly ClawHub search intelligence

docs/clawhub-search-intelligence.md

@@ -0,0 +1,113 @@
+# ClawHub weekly search intelligence receiver
+
+Companion to [CLAW-768](https://linear.app/my-openclaw/issue/CLAW-768) under
+[CLAW-724](https://linear.app/my-openclaw/issue/CLAW-724).
+
+## Boundary and ownership
+
+`POST /api/clawhub-search-intelligence/weekly` accepts ClawHub's frozen
+`plugin_search_weekly` digest. It uses the existing `CLAWHUB_HERMIT_TOKEN`
+(fallback `CLAWHUB_BAN_APPEALS_TOKEN`) and `CLAWHUB_SITE_URL` trusted-origin
+configuration. Destination is `formSettings.clawhubAppealReviewChannelId`, the
+`maintainer-clawhub` channel. No role or user is mentioned.
+
+ClawHub owns query normalization, aggregate privacy thresholds, authoritative
+`isOfficial === true` gap calculations, enrichment/classification, UTC week
+selection, and the Monday 09:00 America/Los_Angeles schedule. Hermit validates and
+delivers facts; it never classifies a query or assigns official status.
+
+The request has a 64 KiB streaming byte limit and exact recursive field
+allowlists. Digest source counts use `clawhubWeb` / `openclawControlUi` keys (the
+observation source enum remains hyphenated in ClawHub). Counts are nonnegative
+safe integers and source counts must sum to the total. Rows are capped at five
+per section; query text at 256 UTF-16 code units, company/display names at 120,
+package names at 160, and same-origin credential-free HTTP(S) links at 2048.
+All ordinary rows require current-week searches >= 3; gap/company rows also
+require official gaps >= 3. Movers require at least three searches in either
+whole week, including drops to zero. Rare-in-both-weeks movers are suppressed.
+The company confidence floor is 0.8. No extra user/device/session/request fields
+are accepted, retained, or logged.
+
+Coverage is required: `dataThrough`, `collectionStartedAt`, `gapStart`, `gapEnd`
+are nullable timestamps, with gap endpoints paired. The message identifies
+unknown/partial collection history, explicit gaps, unavailable enrichment, and
+capped input. Empty initial history is not described as a complete-week total.
+
+Carbon V2 `Container` / `TextDisplay` components carry all content. The message
+stays below 4000 text characters; whole rows that do not fit are replaced by a
+dashboard pointer, never cut links or Markdown. Text is escaped, mentions are
+neutralized, and `allowed_mentions.parse` is empty. A localhost trusted dashboard
+origin produces a visible **LOCAL PREVIEW** heading.
+
+## Delivery state and failure semantics
+
+No migration is needed. The existing D1 `keyValue` primary key stores one receipt
+per trusted origin and UTC week. The record contains a version, canonical digest
+hash, delivery status, start timestamp, and confirmed Discord message ID; it
+contains no query text. Reads use a `first-primary` D1 session. Atomic
+`INSERT ... ON CONFLICT DO NOTHING RETURNING` and compare-and-swap updates fence
+concurrent requests and freeze the weekly payload.
+
+| State / event | Receiver behavior |
+| --- | --- |
+| First request | Claim durably **before** Discord POST. |
+| Concurrent fresh claim | HTTP 409, no additional POST. |
+| Confirmed receipt | HTTP 200 `{ok:true, delivered:true, weekEnd}`; no additional POST, even after the sender loses its HTTP response. |
+| Changed payload for the same week | HTTP 409, no POST. |
+| Explicit Discord 4xx rejection, excluding 408 | Persist retryable state; HTTP 502. A subsequent request may atomically claim another attempt. |
+| Timeout, network/5xx failure, or malformed success | Persist uncertainty; HTTP 503, never blindly repost. |
+| Crash after POST / failed receipt save | The durable sending claim remains. After two minutes, retries reconcile channel history read-only. |
+| Uncertain/stale claim, matching bot message found | Compare expected component text, configured bot author, and send time; persist its ID and return success. |
+| History missing, inaccessible, truncated, or unmatched | HTTP 503; no POST. Requires operational reconciliation, not clearing the weekly key and retrying. |
+| D1 claim/write unavailable | Non-2xx; no success claim without a persisted receipt. |
+
+History reconciliation scans at most five pages of 100 messages, ignores messages
+older than the claim (with a one-minute clock allowance), and rejects copies from
+other authors. The service needs channel View/Read Message History permissions
+as well as Send Messages. The existing Carbon client uses `queueRequests:false`.
+
+Each POST also uses a stable 25-character nonce and `enforce_nonce:true`.
+[Discord documents this deduplication only for the past few minutes](https://docs.discord.com/developers/resources/message#create-message).
+It is defense in depth, **not durable exact-once delivery**. In particular, an
+uncertain request with no discoverable receipt may remain blocked rather than
+risk a duplicate. Do not expire or reset these keys as routine cleanup.
+
+## Validation and proof
+
+Public-handler tests use actual SQLite-backed D1 and replace only Discord HTTP.
+They cover authentication, nested field/URL/count/threshold rejection, real Carbon
+serialization, simultaneous delivery, repeat delivery, rejected-send retry,
+response loss, pre-send D1 failure, post-send receipt failure, bounded/negative
+history reconciliation, coverage, preview labels, and render limits.
+
+Commands:
+
+```sh
+bun test tests/searchIntelligenceApi.test.ts
+bun run typecheck
+bun run test
+bun run deploy:dry-run
+```
+
+Local validation: 13 focused receiver tests (105 assertions), typecheck, and
+deployment dry-run pass. After installing the existing artwork suite's
+ImageMagick prerequisite, the full Hermit suite passes: 300 tests across 35
+files, 184,958 assertions (114.70 seconds). No artwork source changes were needed.
+
+Executable real-service proof (never deploys or registers commands):
+
+```sh
+bun scripts/proof-search-intelligence.ts --prepare /tmp/claw-768-hermit-proof
+# Run the following only through the managed token flow; reuse the SAME directory.
+bun scripts/proof-search-intelligence.ts --send /tmp/claw-768-hermit-proof /path/to/frozen-digest.json
+```
+
+`--prepare` has passed against real local Wrangler D1 without sending anything.
+`--send` requires the frozen digest's dashboard/links to use the trusted localhost
+origin. It verifies the configured test bot identity and actual destination,
+invokes this production handler twice, reads the real Discord message and durable
+receipt, and saves `evidence.json` with status codes, components, no-mention facts,
+and the Discord link. It does not print tokens or message headers. Proof uses
+Patrick's configured OpenClaw test bot, not the deployed production Hermit bot.
+Production Hermit token/configuration and deployment remain separate gates;
+local/test-bot proof must never be presented as production deployment proof.

scripts/proof-search-intelligence.ts

@@ -0,0 +1,196 @@
+/** Local Hermit + real Discord proof; never deploys or registers commands.
+ * Prepare: bun scripts/proof-search-intelligence.ts --prepare /tmp/claw-768-hermit-proof
+ * Send: bun scripts/proof-search-intelligence.ts --send /tmp/claw-768-hermit-proof /path/to/frozen-digest.json
+ * Run --send only through the managed credential flow. Reuse the same proof directory.
+ */
+import { mkdir, readFile, writeFile } from "node:fs/promises"
+import { resolve } from "node:path"
+import { Client, Routes } from "@buape/carbon"
+import { getPlatformProxy } from "wrangler"
+import { formSettings } from "../forms.config.js"
+import { handleSearchIntelligenceApiRequest } from "../src/clawhubSearchIntelligence/api.js"
+import { setRuntimeEnv } from "../src/runtime/env.js"
+
+const [mode, directoryArg, payloadPath] = process.argv.slice(2)
+if (!["--prepare", "--send"].includes(mode ?? "") || !directoryArg)
+	throw new Error(
+		"Expected --prepare <proof-directory> or --send <same-proof-directory> <frozen-digest.json>"
+	)
+const directory = resolve(directoryArg)
+await mkdir(directory, { recursive: true })
+const configPath = resolve(directory, "wrangler.json")
+await writeFile(
+	configPath,
+	JSON.stringify({
+		name: "claw-768-hermit-local-proof",
+		compatibility_date: "2026-09-08",
+		compatibility_flags: ["nodejs_compat"],
+		d1_databases: [
+			{
+				binding: "DB",
+				database_name: "claw-768-local-proof",
+				database_id: "00000000-0000-0000-0000-000000000768"
+			}
+		]
+	})
+)
+const proxy = await getPlatformProxy<{ DB: D1Database }>({
+	configPath,
+	envFiles: [],
+	remoteBindings: false,
+	persist: { path: resolve(directory, "d1-state") }
+})
+let proofStep = "prepare-local-d1"
+try {
+	await proxy.env.DB.prepare(
+		"CREATE TABLE IF NOT EXISTS keyValue (key TEXT PRIMARY KEY NOT NULL, value TEXT NOT NULL, createdAt INTEGER NOT NULL, updatedAt INTEGER NOT NULL)"
+	).run()
+	if (mode === "--prepare") {
+		console.log(
+			JSON.stringify({
+				prepared: true,
+				database: "local D1",
+				directory,
+				sent: false
+			})
+		)
+	} else {
+		proofStep = "read-frozen-payload-and-managed-token"
+		if (!payloadPath || !process.env.DISCORD_BOT_TOKEN)
+			throw new Error(
+				"Frozen payload and managed DISCORD_BOT_TOKEN are required"
+			)
+		const digest = JSON.parse(await readFile(payloadPath, "utf8"))
+		const origin = new URL(digest.dashboardUrl).origin
+		if (!["localhost", "127.0.0.1", "[::1]"].includes(new URL(origin).hostname))
+			throw new Error(
+				"Proof requires a localhost dashboard origin so Discord is visibly labeled LOCAL PREVIEW"
+			)
+		const botId = "1501672484095660143"
+		const client = new Client(
+			{
+				baseUrl: "http://localhost:4312",
+				clientId: botId,
+				publicKey: "0".repeat(64),
+				token: process.env.DISCORD_BOT_TOKEN,
+				autoDeploy: false,
+				disableDeployRoute: true,
+				requestOptions: { queueRequests: false }
+			},
+			{}
+		)
+		proofStep = "verify-approved-bot"
+		const identity = (await client.rest.get(Routes.user("@me"))) as {
+			id: string
+		}
+		if (identity.id !== botId)
+			throw new Error("Configured token is not the approved proof bot")
+		proofStep = "verify-maintainer-channel"
+		const channel = (await client.rest.get(
+			Routes.channel(formSettings.clawhubAppealReviewChannelId)
+		)) as { id: string; name: string; guild_id: string }
+		if (
+			channel.name !== "maintainer-clawhub" ||
+			channel.guild_id !== "1456350064065904867"
+		)
+			throw new Error("Unexpected proof channel")
+		const localToken = crypto.randomUUID()
+		setRuntimeEnv({
+			DB: proxy.env.DB,
+			CLAWHUB_HERMIT_TOKEN: localToken,
+			CLAWHUB_SITE_URL: origin,
+			DISCORD_CLIENT_ID: botId
+		} as Env)
+		const request = () =>
+			new Request(
+				"http://localhost:4312/api/clawhub-search-intelligence/weekly",
+				{
+					method: "POST",
+					headers: {
+						Authorization: `Bearer ${localToken}`,
+						"Content-Type": "application/json"
+					},
+					body: JSON.stringify(digest)
+				}
+			)
+		proofStep = "deliver-and-check-duplicate"
+		const first = await handleSearchIntelligenceApiRequest(request(), client)
+		const replay = await handleSearchIntelligenceApiRequest(request(), client)
+		const key = `clawhub-search-weekly:${origin}:${digest.weekStart}`
+		const row = await proxy.env.DB.withSession("first-primary")
+			.prepare("SELECT value FROM keyValue WHERE key = ?")
+			.bind(key)
+			.first<{ value: string }>()
+		const receipt = row ? JSON.parse(row.value) : null
+		proofStep = "read-confirmed-discord-message"
+		const message = receipt?.messageId
+			? ((await client.rest.get(
+					Routes.channelMessage(channel.id, receipt.messageId)
+				)) as {
+					id: string
+					author: { id: string }
+					flags: number
+					components: unknown[]
+					mentions: unknown[]
+					mention_roles: unknown[]
+					mention_everyone: boolean
+				})
+			: null
+		const evidence = {
+			mode: "local Hermit production handler + local persistent D1 + real Discord test bot; NOT deployed production Hermit",
+			first: { status: first?.status, body: await first?.json() },
+			duplicate: { status: replay?.status, body: await replay?.json() },
+			receipt,
+			discord: message
+				? {
+						url: `https://discord.com/channels/${channel.guild_id}/${channel.id}/${message.id}`,
+						botId: message.author.id,
+						flags: message.flags,
+						components: message.components,
+						mentionCount: message.mentions.length,
+						roleMentionCount: message.mention_roles.length,
+						mentionEveryone: message.mention_everyone
+					}
+				: null
+		}
+		await writeFile(
+			resolve(directory, "evidence.json"),
+			JSON.stringify(evidence, null, 2)
+		)
+		console.log(
+			JSON.stringify({
+				firstStatus: first?.status,
+				duplicateStatus: replay?.status,
+				delivered: receipt?.status === "sent",
+				discordUrl: evidence.discord?.url,
+				evidencePath: resolve(directory, "evidence.json")
+			})
+		)
+		if (
+			first?.status !== 200 ||
+			replay?.status !== 200 ||
+			!message ||
+			message.flags !== 32768 ||
+			message.author.id !== botId ||
+			message.mentions.length ||
+			message.mention_roles.length ||
+			message.mention_everyone
+		)
+			process.exitCode = 1
+	}
+} catch (error) {
+	const status =
+		error &&
+		typeof error === "object" &&
+		"status" in error &&
+		typeof error.status === "number"
+			? error.status
+			: null
+	// Never print the SDK error object, request, headers, or protected environment.
+	console.error(
+		JSON.stringify({ error: "Proof failed", step: proofStep, status })
+	)
+	process.exitCode = 1
+} finally {
+	await proxy.dispose()
+}