diff --git a/docs/clawhub-search-intelligence.md b/docs/clawhub-search-intelligence.md new file mode 100644 index 0000000..d3552b2 --- /dev/null +++ b/docs/clawhub-search-intelligence.md @@ -0,0 +1,113 @@ +# ClawHub weekly search intelligence receiver + +Companion to [CLAW-768](https://linear.app/my-openclaw/issue/CLAW-768) under +[CLAW-724](https://linear.app/my-openclaw/issue/CLAW-724). + +## Boundary and ownership + +`POST /api/clawhub-search-intelligence/weekly` accepts ClawHub's frozen +`plugin_search_weekly` digest. It uses the existing `CLAWHUB_HERMIT_TOKEN` +(fallback `CLAWHUB_BAN_APPEALS_TOKEN`) and `CLAWHUB_SITE_URL` trusted-origin +configuration. Destination is `formSettings.clawhubAppealReviewChannelId`, the +`maintainer-clawhub` channel. No role or user is mentioned. + +ClawHub owns query normalization, aggregate privacy thresholds, authoritative +`isOfficial === true` gap calculations, enrichment/classification, UTC week +selection, and the Monday 09:00 America/Los_Angeles schedule. Hermit validates and +delivers facts; it never classifies a query or assigns official status. + +The request has a 64 KiB streaming byte limit and exact recursive field +allowlists. Digest source counts use `clawhubWeb` / `openclawControlUi` keys (the +observation source enum remains hyphenated in ClawHub). Counts are nonnegative +safe integers and source counts must sum to the total. Rows are capped at five +per section; query text at 256 UTF-16 code units, company/display names at 120, +package names at 160, and same-origin credential-free HTTP(S) links at 2048. +All ordinary rows require current-week searches >= 3; gap/company rows also +require official gaps >= 3. Movers require at least three searches in either +whole week, including drops to zero. Rare-in-both-weeks movers are suppressed. +The company confidence floor is 0.8. No extra user/device/session/request fields +are accepted, retained, or logged. + +Coverage is required: `dataThrough`, `collectionStartedAt`, `gapStart`, `gapEnd` +are nullable timestamps, with gap endpoints paired. The message identifies +unknown/partial collection history, explicit gaps, unavailable enrichment, and +capped input. Empty initial history is not described as a complete-week total. + +Carbon V2 `Container` / `TextDisplay` components carry all content. The message +stays below 4000 text characters; whole rows that do not fit are replaced by a +dashboard pointer, never cut links or Markdown. Text is escaped, mentions are +neutralized, and `allowed_mentions.parse` is empty. A localhost trusted dashboard +origin produces a visible **LOCAL PREVIEW** heading. + +## Delivery state and failure semantics + +No migration is needed. The existing D1 `keyValue` primary key stores one receipt +per trusted origin and UTC week. The record contains a version, canonical digest +hash, delivery status, start timestamp, and confirmed Discord message ID; it +contains no query text. Reads use a `first-primary` D1 session. Atomic +`INSERT ... ON CONFLICT DO NOTHING RETURNING` and compare-and-swap updates fence +concurrent requests and freeze the weekly payload. + +| State / event | Receiver behavior | +| --- | --- | +| First request | Claim durably **before** Discord POST. | +| Concurrent fresh claim | HTTP 409, no additional POST. | +| Confirmed receipt | HTTP 200 `{ok:true, delivered:true, weekEnd}`; no additional POST, even after the sender loses its HTTP response. | +| Changed payload for the same week | HTTP 409, no POST. | +| Explicit Discord 4xx rejection, excluding 408 | Persist retryable state; HTTP 502. A subsequent request may atomically claim another attempt. | +| Timeout, network/5xx failure, or malformed success | Persist uncertainty; HTTP 503, never blindly repost. | +| Crash after POST / failed receipt save | The durable sending claim remains. After two minutes, retries reconcile channel history read-only. | +| Uncertain/stale claim, matching bot message found | Compare expected component text, configured bot author, and send time; persist its ID and return success. | +| History missing, inaccessible, truncated, or unmatched | HTTP 503; no POST. Requires operational reconciliation, not clearing the weekly key and retrying. | +| D1 claim/write unavailable | Non-2xx; no success claim without a persisted receipt. | + +History reconciliation scans at most five pages of 100 messages, ignores messages +older than the claim (with a one-minute clock allowance), and rejects copies from +other authors. The service needs channel View/Read Message History permissions +as well as Send Messages. The existing Carbon client uses `queueRequests:false`. + +Each POST also uses a stable 25-character nonce and `enforce_nonce:true`. +[Discord documents this deduplication only for the past few minutes](https://docs.discord.com/developers/resources/message#create-message). +It is defense in depth, **not durable exact-once delivery**. In particular, an +uncertain request with no discoverable receipt may remain blocked rather than +risk a duplicate. Do not expire or reset these keys as routine cleanup. + +## Validation and proof + +Public-handler tests use actual SQLite-backed D1 and replace only Discord HTTP. +They cover authentication, nested field/URL/count/threshold rejection, real Carbon +serialization, simultaneous delivery, repeat delivery, rejected-send retry, +response loss, pre-send D1 failure, post-send receipt failure, bounded/negative +history reconciliation, coverage, preview labels, and render limits. + +Commands: + +```sh +bun test tests/searchIntelligenceApi.test.ts +bun run typecheck +bun run test +bun run deploy:dry-run +``` + +Local validation: 13 focused receiver tests (105 assertions), typecheck, and +deployment dry-run pass. After installing the existing artwork suite's +ImageMagick prerequisite, the full Hermit suite passes: 300 tests across 35 +files, 184,958 assertions (114.70 seconds). No artwork source changes were needed. + +Executable real-service proof (never deploys or registers commands): + +```sh +bun scripts/proof-search-intelligence.ts --prepare /tmp/claw-768-hermit-proof +# Run the following only through the managed token flow; reuse the SAME directory. +bun scripts/proof-search-intelligence.ts --send /tmp/claw-768-hermit-proof /path/to/frozen-digest.json +``` + +`--prepare` has passed against real local Wrangler D1 without sending anything. +`--send` requires the frozen digest's dashboard/links to use the trusted localhost +origin. It verifies the configured test bot identity and actual destination, +invokes this production handler twice, reads the real Discord message and durable +receipt, and saves `evidence.json` with status codes, components, no-mention facts, +and the Discord link. It does not print tokens or message headers. Proof uses +Patrick's configured OpenClaw test bot, not the deployed production Hermit bot. +Production Hermit token/configuration and deployment remain separate gates; +local/test-bot proof must never be presented as production deployment proof. diff --git a/scripts/proof-search-intelligence.ts b/scripts/proof-search-intelligence.ts new file mode 100644 index 0000000..81711eb --- /dev/null +++ b/scripts/proof-search-intelligence.ts @@ -0,0 +1,196 @@ +/** Local Hermit + real Discord proof; never deploys or registers commands. + * Prepare: bun scripts/proof-search-intelligence.ts --prepare /tmp/claw-768-hermit-proof + * Send: bun scripts/proof-search-intelligence.ts --send /tmp/claw-768-hermit-proof /path/to/frozen-digest.json + * Run --send only through the managed credential flow. Reuse the same proof directory. + */ +import { mkdir, readFile, writeFile } from "node:fs/promises" +import { resolve } from "node:path" +import { Client, Routes } from "@buape/carbon" +import { getPlatformProxy } from "wrangler" +import { formSettings } from "../forms.config.js" +import { handleSearchIntelligenceApiRequest } from "../src/clawhubSearchIntelligence/api.js" +import { setRuntimeEnv } from "../src/runtime/env.js" + +const [mode, directoryArg, payloadPath] = process.argv.slice(2) +if (!["--prepare", "--send"].includes(mode ?? "") || !directoryArg) + throw new Error( + "Expected --prepare or --send " + ) +const directory = resolve(directoryArg) +await mkdir(directory, { recursive: true }) +const configPath = resolve(directory, "wrangler.json") +await writeFile( + configPath, + JSON.stringify({ + name: "claw-768-hermit-local-proof", + compatibility_date: "2026-09-08", + compatibility_flags: ["nodejs_compat"], + d1_databases: [ + { + binding: "DB", + database_name: "claw-768-local-proof", + database_id: "00000000-0000-0000-0000-000000000768" + } + ] + }) +) +const proxy = await getPlatformProxy<{ DB: D1Database }>({ + configPath, + envFiles: [], + remoteBindings: false, + persist: { path: resolve(directory, "d1-state") } +}) +let proofStep = "prepare-local-d1" +try { + await proxy.env.DB.prepare( + "CREATE TABLE IF NOT EXISTS keyValue (key TEXT PRIMARY KEY NOT NULL, value TEXT NOT NULL, createdAt INTEGER NOT NULL, updatedAt INTEGER NOT NULL)" + ).run() + if (mode === "--prepare") { + console.log( + JSON.stringify({ + prepared: true, + database: "local D1", + directory, + sent: false + }) + ) + } else { + proofStep = "read-frozen-payload-and-managed-token" + if (!payloadPath || !process.env.DISCORD_BOT_TOKEN) + throw new Error( + "Frozen payload and managed DISCORD_BOT_TOKEN are required" + ) + const digest = JSON.parse(await readFile(payloadPath, "utf8")) + const origin = new URL(digest.dashboardUrl).origin + if (!["localhost", "127.0.0.1", "[::1]"].includes(new URL(origin).hostname)) + throw new Error( + "Proof requires a localhost dashboard origin so Discord is visibly labeled LOCAL PREVIEW" + ) + const botId = "1501672484095660143" + const client = new Client( + { + baseUrl: "http://localhost:4312", + clientId: botId, + publicKey: "0".repeat(64), + token: process.env.DISCORD_BOT_TOKEN, + autoDeploy: false, + disableDeployRoute: true, + requestOptions: { queueRequests: false } + }, + {} + ) + proofStep = "verify-approved-bot" + const identity = (await client.rest.get(Routes.user("@me"))) as { + id: string + } + if (identity.id !== botId) + throw new Error("Configured token is not the approved proof bot") + proofStep = "verify-maintainer-channel" + const channel = (await client.rest.get( + Routes.channel(formSettings.clawhubAppealReviewChannelId) + )) as { id: string; name: string; guild_id: string } + if ( + channel.name !== "maintainer-clawhub" || + channel.guild_id !== "1456350064065904867" + ) + throw new Error("Unexpected proof channel") + const localToken = crypto.randomUUID() + setRuntimeEnv({ + DB: proxy.env.DB, + CLAWHUB_HERMIT_TOKEN: localToken, + CLAWHUB_SITE_URL: origin, + DISCORD_CLIENT_ID: botId + } as Env) + const request = () => + new Request( + "http://localhost:4312/api/clawhub-search-intelligence/weekly", + { + method: "POST", + headers: { + Authorization: `Bearer ${localToken}`, + "Content-Type": "application/json" + }, + body: JSON.stringify(digest) + } + ) + proofStep = "deliver-and-check-duplicate" + const first = await handleSearchIntelligenceApiRequest(request(), client) + const replay = await handleSearchIntelligenceApiRequest(request(), client) + const key = `clawhub-search-weekly:${origin}:${digest.weekStart}` + const row = await proxy.env.DB.withSession("first-primary") + .prepare("SELECT value FROM keyValue WHERE key = ?") + .bind(key) + .first<{ value: string }>() + const receipt = row ? JSON.parse(row.value) : null + proofStep = "read-confirmed-discord-message" + const message = receipt?.messageId + ? ((await client.rest.get( + Routes.channelMessage(channel.id, receipt.messageId) + )) as { + id: string + author: { id: string } + flags: number + components: unknown[] + mentions: unknown[] + mention_roles: unknown[] + mention_everyone: boolean + }) + : null + const evidence = { + mode: "local Hermit production handler + local persistent D1 + real Discord test bot; NOT deployed production Hermit", + first: { status: first?.status, body: await first?.json() }, + duplicate: { status: replay?.status, body: await replay?.json() }, + receipt, + discord: message + ? { + url: `https://discord.com/channels/${channel.guild_id}/${channel.id}/${message.id}`, + botId: message.author.id, + flags: message.flags, + components: message.components, + mentionCount: message.mentions.length, + roleMentionCount: message.mention_roles.length, + mentionEveryone: message.mention_everyone + } + : null + } + await writeFile( + resolve(directory, "evidence.json"), + JSON.stringify(evidence, null, 2) + ) + console.log( + JSON.stringify({ + firstStatus: first?.status, + duplicateStatus: replay?.status, + delivered: receipt?.status === "sent", + discordUrl: evidence.discord?.url, + evidencePath: resolve(directory, "evidence.json") + }) + ) + if ( + first?.status !== 200 || + replay?.status !== 200 || + !message || + message.flags !== 32768 || + message.author.id !== botId || + message.mentions.length || + message.mention_roles.length || + message.mention_everyone + ) + process.exitCode = 1 + } +} catch (error) { + const status = + error && + typeof error === "object" && + "status" in error && + typeof error.status === "number" + ? error.status + : null + // Never print the SDK error object, request, headers, or protected environment. + console.error( + JSON.stringify({ error: "Proof failed", step: proofStep, status }) + ) + process.exitCode = 1 +} finally { + await proxy.dispose() +} diff --git a/src/clawhubSearchIntelligence/api.ts b/src/clawhubSearchIntelligence/api.ts new file mode 100644 index 0000000..89a233b --- /dev/null +++ b/src/clawhubSearchIntelligence/api.ts @@ -0,0 +1,415 @@ +import { + Container, + TextDisplay, + serializePayload, + type Client +} from "@buape/carbon" +import { getRuntimeEnv } from "../runtime/env.js" +import { + publisherAbuseDigestApiToken, + publisherAbuseDigestTrustedOrigins +} from "../clawhubPublisherAbuse/api.js" +import { deliverWeeklyDigest } from "./delivery.js" + +const apiPath = "/api/clawhub-search-intelligence/weekly" +type Row = { + query: string + searches: number + previousSearches: number + officialGaps: number + searchUrl: string +} +type Digest = { + kind: "plugin_search_weekly" + weekStart: number + weekEnd: number + minimumSearches: 3 + dashboardUrl: string + totalSearches: number + sourceCounts: { clawhubWeb: number; openclawControlUi: number } + classificationStatus: "available" | "partial" | "unavailable" + currentMetadataStatus: "available" | "unavailable" + truncated: boolean + coverage: { + dataThrough: number | null + collectionStartedAt: number | null + gapStart: number | null + gapEnd: number | null + } + companyOpportunities: (Row & { + companyProductName?: string + confidence: number + })[] + officialGaps: Row[] + featuredCandidates: (Row & { + package: { name: string; displayName: string; url: string } + })[] + movers: Row[] +} +const record = (value: unknown): value is Record => + !!value && typeof value === "object" && !Array.isArray(value) +const fields = ( + value: unknown, + required: string[], + optional: string[] = [] +): value is Record => + record(value) && + required.every((key) => Object.hasOwn(value, key)) && + Object.keys(value).every( + (key) => required.includes(key) || optional.includes(key) + ) +const count = (value: unknown): value is number => + typeof value === "number" && Number.isSafeInteger(value) && value >= 0 +const timestamp = (value: unknown): value is number => + count(value) && value <= 8_640_000_000_000_000 +const string = (value: unknown, max: number): value is string => + typeof value === "string" && + value.length > 0 && + value.trim() === value && + value.length <= max && + !/[\u0000-\u001f\u007f]/.test(value) +const validUrl = (value: unknown, origins: string[]) => { + if (!string(value, 2048)) return false + try { + const url = new URL(value) + return ( + ["http:", "https:"].includes(url.protocol) && + !url.username && + !url.password && + origins.includes(url.origin) && + url.toString().length <= 2048 + ) + } catch { + return false + } +} +const parseDigest = (value: unknown, origins: string[]): Digest | null => { + if ( + !fields(value, [ + "kind", + "weekStart", + "weekEnd", + "minimumSearches", + "dashboardUrl", + "totalSearches", + "sourceCounts", + "classificationStatus", + "currentMetadataStatus", + "truncated", + "coverage", + "companyOpportunities", + "officialGaps", + "featuredCandidates", + "movers" + ]) + ) + return null + if ( + value.kind !== "plugin_search_weekly" || + value.minimumSearches !== 3 || + !timestamp(value.weekStart) || + !timestamp(value.weekEnd) || + value.weekEnd - value.weekStart !== 604_800_000 || + value.weekEnd % 86_400_000 !== 0 || + new Date(value.weekEnd).getUTCDay() !== 1 || + !validUrl(value.dashboardUrl, origins) || + !count(value.totalSearches) || + typeof value.truncated !== "boolean" + ) + return null + if ( + typeof value.classificationStatus !== "string" || + !["available", "partial", "unavailable"].includes( + value.classificationStatus + ) || + typeof value.currentMetadataStatus !== "string" || + !["available", "unavailable"].includes(value.currentMetadataStatus) + ) + return null + const sources = value.sourceCounts + if ( + !fields(sources, ["clawhubWeb", "openclawControlUi"]) || + !count(sources["clawhubWeb"]) || + !count(sources["openclawControlUi"]) || + sources["clawhubWeb"] + sources["openclawControlUi"] !== value.totalSearches + ) + return null + const coverage = value.coverage + if ( + !fields(coverage, [ + "dataThrough", + "collectionStartedAt", + "gapStart", + "gapEnd" + ]) || + !Object.values(coverage).every( + (time) => time === null || timestamp(time) + ) || + (coverage.gapStart === null) !== (coverage.gapEnd === null) || + (typeof coverage.gapStart === "number" && + typeof coverage.gapEnd === "number" && + coverage.gapStart >= coverage.gapEnd) + ) + return null + const rowFields = [ + "query", + "searches", + "previousSearches", + "officialGaps", + "searchUrl" + ] + const validRows = ( + rows: unknown, + kind: "company" | "gap" | "featured" | "mover" + ) => + Array.isArray(rows) && + rows.length <= 5 && + rows.every((row) => { + if ( + !fields( + row, + [ + ...rowFields, + ...(kind === "company" + ? ["confidence"] + : kind === "featured" + ? ["package"] + : []) + ], + kind === "company" ? ["companyProductName"] : [] + ) || + !string(row.query, 256) || + !count(row.searches) || + row.searches > (value.totalSearches as number) || + !count(row.previousSearches) || + !count(row.officialGaps) || + row.officialGaps > row.searches || + !validUrl(row.searchUrl, origins) + ) + return false + if ( + (kind === "mover" + ? Math.max(row.searches, row.previousSearches) + : row.searches) < 3 + ) + return false + if ((kind === "company" || kind === "gap") && row.officialGaps < 3) + return false + if ( + kind === "company" && + (typeof row.confidence !== "number" || + !Number.isFinite(row.confidence) || + row.confidence < 0.8 || + row.confidence > 1 || + (row.companyProductName !== undefined && + !string(row.companyProductName, 120))) + ) + return false + if ( + kind === "featured" && + (!fields(row.package, ["name", "displayName", "url"]) || + !string(row.package.name, 160) || + !string(row.package.displayName, 120) || + !validUrl(row.package.url, origins)) + ) + return false + return true + }) + if ( + !validRows(value.companyOpportunities, "company") || + !validRows(value.officialGaps, "gap") || + !validRows(value.featuredCandidates, "featured") || + !validRows(value.movers, "mover") + ) + return null + if ( + (value.classificationStatus === "unavailable" && + (value.companyOpportunities as unknown[]).length) || + (value.currentMetadataStatus === "unavailable" && + (value.featuredCandidates as unknown[]).length) + ) + return null + return value as unknown as Digest +} +// Bound bytes while consuming the stream, not after allocating an arbitrary body. +const readBody = async (request: Request): Promise => { + const reader = request.body?.getReader() + if (!reader) return null + const chunks: Uint8Array[] = [] + let size = 0 + try { + while (true) { + const { value, done } = await reader.read() + if (done) break + size += value.byteLength + if (size > 65_536) { + await reader.cancel() + throw new RangeError("Body too large") + } + chunks.push(value) + } + } finally { + reader.releaseLock() + } + const bytes = new Uint8Array(size) + let offset = 0 + for (const chunk of chunks) { + bytes.set(chunk, offset) + offset += chunk.byteLength + } + return JSON.parse(new TextDecoder().decode(bytes)) +} +const json = (value: unknown, status = 200) => + new Response(JSON.stringify(value), { + status, + headers: { "content-type": "application/json" } + }) +const safe = (value: string) => + value + .replace(/[\u0000-\u001f\u007f]/g, " ") + .replace(/([\\`*_~|>\[\]()#])/g, "\\$1") + .replace(/@/g, "@\u200b") +const link = (value: string) => + `<${new URL(value).toString().replace(//g, "%3E")}>` +const date = (value: number) => new Date(value).toISOString().slice(0, 10) +const render = (digest: Digest) => { + const preview = ["localhost", "127.0.0.1", "[::1]"].includes( + new URL(digest.dashboardUrl).hostname + ) + const coverage = digest.coverage + const incomplete = + coverage.dataThrough === null || + coverage.dataThrough < digest.weekEnd || + coverage.collectionStartedAt === null || + coverage.collectionStartedAt > digest.weekStart || + coverage.gapStart !== null + const header = [ + `### ${preview ? "LOCAL PREVIEW · " : ""}ClawHub weekly search intelligence`, + `${date(digest.weekStart)} – ${date(digest.weekEnd)} (UTC, end exclusive)`, + `${digest.totalSearches} searches · Web ${digest.sourceCounts["clawhubWeb"]} · Control UI ${digest.sourceCounts["openclawControlUi"]}`, + `Data through: ${coverage.dataThrough === null ? "unknown" : date(coverage.dataThrough)} · Collection started: ${coverage.collectionStartedAt === null ? "unknown" : date(coverage.collectionStartedAt)}`, + ...(incomplete + ? ["**Incomplete collection history; not a complete-week demand total.**"] + : []), + ...(coverage.gapStart !== null && coverage.gapEnd !== null + ? [ + `Collection gap: ${date(coverage.gapStart)} – ${date(coverage.gapEnd)} UTC` + ] + : []), + `[Open search intelligence](${link(digest.dashboardUrl)})` + ].join("\n") + const footer = [ + "At least 3 searches required: either week for movers, current week for other rows. Official gaps are deterministic; company classification is advisory.", + ...(digest.classificationStatus !== "available" + ? [ + digest.classificationStatus === "unavailable" + ? "Classification unavailable." + : "Classification partially available." + ] + : []), + ...(digest.currentMetadataStatus === "unavailable" + ? ["Current package metadata unavailable."] + : []), + ...(digest.truncated ? ["Input capped; rankings may be incomplete."] : []) + ].join("\n") + const brief = (value: string) => + safe(value.length > 80 ? `${value.slice(0, 79)}…` : value) + const rowText = (row: Row) => + `[${brief(row.query)}](${link(row.searchUrl)}) · ${row.searches} searches · ${row.officialGaps} gaps · previous ${row.previousSearches}` + // Reserve equal space for each section; never cut links/Markdown mid-row. + const budget = Math.floor((3900 - header.length - footer.length) / 4) + const section = (title: string, rows: string[], empty: string) => { + let text = `**${title}**` + if (!rows.length) return `${text}\n${empty}` + for (const row of rows) { + if (text.length + row.length + 34 > budget) + return `${text}\nMore rows on the dashboard.` + text += `\n${row}` + } + return text + } + return serializePayload({ + components: [ + new Container([ + new TextDisplay(header), + new TextDisplay( + section( + "Company plugin opportunities", + digest.companyOpportunities.map( + (row) => + `${rowText(row)}${row.companyProductName ? ` · ${brief(row.companyProductName)}` : ""}` + ), + digest.classificationStatus === "unavailable" + ? "Classification unavailable." + : "No threshold-qualified opportunities." + ) + ), + new TextDisplay( + section( + "Official gaps", + digest.officialGaps.map(rowText), + "No threshold-qualified gaps." + ) + ), + new TextDisplay( + section( + "Featured candidates", + digest.featuredCandidates.map( + (row) => + `${rowText(row)} · [${brief(row.package.displayName)}](${link(row.package.url)})` + ), + digest.currentMetadataStatus === "unavailable" + ? "Current package metadata unavailable." + : "No eligible candidates." + ) + ), + new TextDisplay( + section( + "Week-over-week movers", + digest.movers.map(rowText), + "No threshold-qualified movers." + ) + ), + new TextDisplay(footer) + ]) + ], + allowedMentions: { parse: [] } + }) +} +export const handleSearchIntelligenceApiRequest = async ( + request: Request, + client: Client +): Promise => { + if (new URL(request.url).pathname !== apiPath) return null + const token = publisherAbuseDigestApiToken(getRuntimeEnv()) + if ( + !token || + request.headers.get("authorization")?.match(/^Bearer\s+(.+)$/i)?.[1] !== + token + ) + return json({ error: "Unauthorized" }, 401) + if (request.method !== "POST") + return json({ error: "Method not allowed" }, 405) + let body: unknown + try { + body = await readBody(request) + } catch (error) { + return json( + { + error: error instanceof RangeError ? "Body too large" : "Invalid JSON" + }, + error instanceof RangeError ? 413 : 400 + ) + } + const digest = parseDigest( + body, + publisherAbuseDigestTrustedOrigins(getRuntimeEnv()) + ) + if (!digest) + return json({ error: "Invalid search intelligence payload" }, 400) + try { + return await deliverWeeklyDigest(client, digest, render(digest)) + } catch { + return json({ error: "Delivery state unavailable" }, 503) + } +} diff --git a/src/clawhubSearchIntelligence/delivery.ts b/src/clawhubSearchIntelligence/delivery.ts new file mode 100644 index 0000000..becffe9 --- /dev/null +++ b/src/clawhubSearchIntelligence/delivery.ts @@ -0,0 +1,200 @@ +import { Routes, type Client, type serializePayload } from "@buape/carbon" +import { formSettings } from "../../forms.config.js" +import { getRuntimeEnv } from "../runtime/env.js" + +type Delivery = { + version: 1 + hash: string + status: "sending" | "sent" | "retryable" | "uncertain" + startedAt: number + messageId?: string +} +const canonical = (value: unknown): string => { + if (Array.isArray(value)) return `[${value.map(canonical).join(",")}]` + if (value && typeof value === "object") + return ( + "{" + + Object.entries(value) + .sort(([a], [b]) => (a < b ? -1 : a > b ? 1 : 0)) + .map(([key, entry]) => `${JSON.stringify(key)}:${canonical(entry)}`) + .join(",") + + "}" + ) + return JSON.stringify(value) +} +const hash = async (value: string) => + Array.from( + new Uint8Array( + await crypto.subtle.digest("SHA-256", new TextEncoder().encode(value)) + ), + (byte) => byte.toString(16).padStart(2, "0") + ).join("") +const response = (value: unknown, status = 200) => + new Response(JSON.stringify(value), { + status, + headers: { "content-type": "application/json" } + }) + +const componentText = (value: unknown): string[] => { + if (!value || typeof value !== "object") return [] + const row = value as { content?: unknown; components?: unknown } + return [ + ...(typeof row.content === "string" ? [row.content] : []), + ...(Array.isArray(row.components) + ? row.components.flatMap(componentText) + : []) + ] +} +// Discord's enforce_nonce is only a few-minute safeguard. An uncertain send is +// reconciled read-only; absence from bounded history is NOT permission to resend. +const findDeliveredMessage = async ( + client: Client, + body: ReturnType, + startedAt: number +): Promise => { + const botId = getRuntimeEnv().DISCORD_CLIENT_ID + if (!botId) return null + const expected = JSON.stringify( + (body.components ?? []).flatMap(componentText) + ) + let before: string | undefined + for (let page = 0; page < 5; page++) { + const messages = await client.rest.get( + Routes.channelMessages(formSettings.clawhubAppealReviewChannelId), + { limit: 100, ...(before ? { before } : {}) } + ) + if (!Array.isArray(messages)) return null + for (const message of messages) { + if (!message || typeof message !== "object") continue + const time = Date.parse(message.timestamp) + if (!Number.isFinite(time) || time < startedAt - 60_000) continue + if ( + message.author?.id === botId && + message.author?.bot === true && + typeof message.id === "string" && + Array.isArray(message.components) && + JSON.stringify(message.components.flatMap(componentText)) === expected + ) + return message.id + } + const oldest = messages.at(-1) + if ( + messages.length < 100 || + !oldest || + typeof oldest.id !== "string" || + oldest.id === before || + Date.parse(oldest.timestamp) < startedAt - 60_000 + ) + return null + before = oldest.id + } + return null +} + +export const deliverWeeklyDigest = async ( + client: Client, + digest: { weekStart: number; weekEnd: number; dashboardUrl: string }, + body: ReturnType +): Promise => { + const db = getRuntimeEnv().DB.withSession("first-primary") + const key = `clawhub-search-weekly:${new URL(digest.dashboardUrl).origin}:${digest.weekStart}` + const payloadHash = await hash(canonical(digest)) + const claim: Delivery = { + version: 1, + hash: payloadHash, + status: "sending", + startedAt: Date.now() + } + const serialized = JSON.stringify(claim) + const inserted = await db + .prepare( + "INSERT INTO keyValue (key, value, createdAt, updatedAt) VALUES (?, ?, ?, ?) ON CONFLICT(key) DO NOTHING RETURNING key" + ) + .bind(key, serialized, claim.startedAt, claim.startedAt) + .first() + const success = () => + response({ ok: true, delivered: true, weekEnd: digest.weekEnd }) + const save = (next: Delivery, previous = serialized) => + db + .prepare( + "UPDATE keyValue SET value = ?, updatedAt = ? WHERE key = ? AND value = ? RETURNING key" + ) + .bind(JSON.stringify(next), Date.now(), key, previous) + .first() + if (!inserted) { + const existing = await db + .prepare("SELECT value FROM keyValue WHERE key = ?") + .bind(key) + .first<{ value: string }>() + if (!existing) return response({ error: "Delivery state unavailable" }, 503) + const state = JSON.parse(existing.value) as Delivery + if (state.hash !== payloadHash) + return response({ error: "Weekly payload conflict" }, 409) + if (state.status === "sent") return success() + if (state.status !== "retryable") { + if (state.status === "sending" && Date.now() - state.startedAt < 120_000) + return response({ error: "Delivery pending" }, 409) + const messageId = await findDeliveredMessage( + client, + body, + state.startedAt + ) + if (!messageId) + return response( + { error: "Delivery uncertain; reconciliation required" }, + 503 + ) + return (await save( + { ...state, status: "sent", messageId }, + existing.value + )) + ? success() + : response({ error: "Delivery receipt pending" }, 503) + } + if (!(await save(claim, existing.value))) + return response({ error: "Delivery pending" }, 409) + } + let message: unknown + try { + message = await client.rest.post( + Routes.channelMessages(formSettings.clawhubAppealReviewChannelId), + { + body: { + ...body, + nonce: (await hash(key)).slice(0, 25), + enforce_nonce: true + } + } + ) + } catch (error) { + const status = + error && typeof error === "object" && "status" in error + ? error.status + : null + // Only an explicit rejection proves Discord did not accept a message. + const rejected = + typeof status === "number" && + status >= 400 && + status < 500 && + status !== 408 + await save({ ...claim, status: rejected ? "retryable" : "uncertain" }) + return response( + { error: rejected ? "Discord rejected delivery" : "Delivery uncertain" }, + rejected ? 502 : 503 + ) + } + if ( + !message || + typeof message !== "object" || + !("id" in message) || + typeof message.id !== "string" || + !message.id + ) { + await save({ ...claim, status: "uncertain" }) + return response({ error: "Delivery uncertain" }, 503) + } + const saved = await save({ ...claim, status: "sent", messageId: message.id }) + return saved + ? success() + : response({ error: "Delivery receipt pending" }, 503) +} diff --git a/src/index.ts b/src/index.ts index 19e319c..1f3862d 100644 --- a/src/index.ts +++ b/src/index.ts @@ -46,6 +46,7 @@ import { import { runNominationCardSyncRecovery } from "./services/nominationCardSync.js" import { runThreadLengthMonitor } from "./services/threadLengthMonitor.js" import { handleContentRightsApiRequest } from "./clawhubContentRights/api.js" +import { handleSearchIntelligenceApiRequest } from "./clawhubSearchIntelligence/api.js" import { handlePublisherAbuseDigestApiRequest } from "./clawhubPublisherAbuse/api.js" import { handleLobsterDossierRequest } from "./lobsterDossiers/server.js" @@ -152,6 +153,8 @@ export default { if (publisherAbuseDigestResponse) { return publisherAbuseDigestResponse } + const searchIntelligenceResponse = await handleSearchIntelligenceApiRequest(request, client) + if (searchIntelligenceResponse) return searchIntelligenceResponse const formsResponse = await handleFormsRequest(request, client) if (formsResponse) { return formsResponse diff --git a/tests/searchIntelligenceApi.test.ts b/tests/searchIntelligenceApi.test.ts new file mode 100644 index 0000000..2a581ec --- /dev/null +++ b/tests/searchIntelligenceApi.test.ts @@ -0,0 +1,607 @@ +import { afterEach, describe, expect, it, spyOn } from "bun:test" +import { readFileSync } from "node:fs" +import type { Client } from "@buape/carbon" +import { handleSearchIntelligenceApiRequest } from "../src/clawhubSearchIntelligence/api.js" +import { setRuntimeEnv } from "../src/runtime/env.js" +import { SqliteD1Database } from "./helpers/sqliteD1.js" + +const validPayload = { + kind: "plugin_search_weekly", + weekStart: Date.UTC(2026, 7, 31), + weekEnd: Date.UTC(2026, 8, 7), + minimumSearches: 3, + coverage: { + dataThrough: Date.UTC(2026, 8, 7), + collectionStartedAt: Date.UTC(2026, 7, 1), + gapStart: null, + gapEnd: null + }, + dashboardUrl: + "https://clawhub.ai/management/search-insights?endDay=1788739200000", + totalSearches: 12, + sourceCounts: { clawhubWeb: 8, openclawControlUi: 4 }, + classificationStatus: "available", + currentMetadataStatus: "available", + truncated: false, + companyOpportunities: [ + { + query: "notion", + searches: 5, + previousSearches: 3, + officialGaps: 5, + searchUrl: "https://clawhub.ai/plugins?q=notion", + companyProductName: "Notion", + confidence: 0.95 + } + ], + officialGaps: [ + { + query: "notion", + searches: 5, + previousSearches: 3, + officialGaps: 5, + searchUrl: "https://clawhub.ai/plugins?q=notion" + } + ], + featuredCandidates: [ + { + query: "memory", + searches: 4, + previousSearches: 2, + officialGaps: 0, + searchUrl: "https://clawhub.ai/plugins?q=memory", + package: { + name: "memory-kit", + displayName: "Memory Kit", + url: "https://clawhub.ai/plugins/memory-kit" + } + } + ], + movers: [ + { + query: "notion", + searches: 5, + previousSearches: 3, + officialGaps: 5, + searchUrl: "https://clawhub.ai/plugins?q=notion" + } + ] +} +let mockClock: ReturnType> | undefined +const owners: SqliteD1Database[] = [] +const setup = () => { + const owner = new SqliteD1Database() + owner.database.exec( + readFileSync( + new URL("../drizzle/0000_productive_tinkerer.sql", import.meta.url), + "utf8" + ) + ) + owners.push(owner) + setRuntimeEnv({ + DB: owner as unknown as D1Database, + CLAWHUB_HERMIT_TOKEN: "test-service-token", + DISCORD_CLIENT_ID: "bot-user" + } as Env) + const posts: Array<{ route: string; body: Record }> = [] + const client = { + rest: { + post: async ( + route: string, + { body }: { body: Record } + ) => { + posts.push({ route, body }) + return { id: "message-1" } + }, + get: async () => [] + } + } as unknown as Client + return { owner, client, posts } +} +const request = ( + payload: unknown = validPayload, + authorization = "Bearer test-service-token" +) => + new Request( + "https://forms.openclaw.ai/api/clawhub-search-intelligence/weekly", + { + method: "POST", + headers: { + Authorization: authorization, + "Content-Type": "application/json" + }, + body: JSON.stringify(payload) + } + ) +const texts = (component: unknown): string[] => { + if (!component || typeof component !== "object") return [] + const row = component as { content?: string; components?: unknown[] } + return [ + ...(typeof row.content === "string" ? [row.content] : []), + ...(row.components ?? []).flatMap(texts) + ] +} +afterEach(() => { + mockClock?.mockRestore() + mockClock = undefined + for (const owner of owners.splice(0)) owner.close() +}) + +describe("ClawHub weekly search intelligence receiver", () => { + it("delivers bounded aggregate facts with Carbon V2 and no mentions", async () => { + const { client, posts } = setup() + const response = await handleSearchIntelligenceApiRequest(request(), client) + expect(response?.status).toBe(200) + expect(posts).toHaveLength(1) + expect(posts[0].route).toBe("/channels/1498032057337647295/messages") + expect(posts[0].body.allowed_mentions).toEqual({ parse: [] }) + expect(posts[0].body.flags).toBe(32768) + expect(posts[0].body).not.toHaveProperty("content") + expect(posts[0].body.embeds).toBeUndefined() + const text = (posts[0].body.components as unknown[]) + .flatMap(texts) + .join("\n") + expect(text).toContain("Company plugin opportunities") + expect(text).toContain("Official gaps") + expect(text).toContain("Featured candidates") + expect(text).toContain("Week-over-week movers") + expect(text).toContain("12 searches") + expect(text).toContain("Notion") + }) + it("authenticates before parsing and only handles its POST endpoint", async () => { + const { client, posts } = setup() + expect( + await handleSearchIntelligenceApiRequest( + new Request("https://example.com/unrelated"), + client + ) + ).toBeNull() + expect( + ( + await handleSearchIntelligenceApiRequest( + request(validPayload, "Bearer wrong"), + client + ) + )?.status + ).toBe(401) + expect( + ( + await handleSearchIntelligenceApiRequest( + new Request( + "https://example.com/api/clawhub-search-intelligence/weekly", + { headers: { Authorization: "Bearer test-service-token" } } + ), + client + ) + )?.status + ).toBe(405) + expect(posts).toHaveLength(0) + }) + + it("rejects unknown, private, unbounded, inconsistent or untrusted aggregate payloads", async () => { + const { client, posts } = setup() + const invalid = [ + { ...validPayload, userId: "forbidden" }, + { + ...validPayload, + coverage: { ...validPayload.coverage, deviceId: "forbidden" } + }, + { + ...validPayload, + sourceCounts: { ...validPayload.sourceCounts, api: 1 } + }, + { + ...validPayload, + sourceCounts: { ...validPayload.sourceCounts, clawhubWeb: 9 } + }, + { ...validPayload, minimumSearches: 1 }, + { ...validPayload, weekEnd: validPayload.weekEnd + 1 }, + { ...validPayload, totalSearches: -1 }, + { ...validPayload, totalSearches: Number.MAX_SAFE_INTEGER + 1 }, + { ...validPayload, dashboardUrl: "https://evil.example/" }, + { ...validPayload, dashboardUrl: "https://secret@clawhub.ai/" }, + { + ...validPayload, + dashboardUrl: "https://clawhub.ai/" + "a".repeat(2049) + }, + { + ...validPayload, + companyOpportunities: [ + { ...validPayload.companyOpportunities[0], confidence: 0.4 } + ] + }, + { + ...validPayload, + companyOpportunities: [ + { ...validPayload.companyOpportunities[0], officialGaps: 6 } + ] + }, + { + ...validPayload, + officialGaps: [ + { ...validPayload.officialGaps[0], searches: 2, officialGaps: 2 } + ] + }, + { + ...validPayload, + officialGaps: Array(6).fill(validPayload.officialGaps[0]) + }, + { + ...validPayload, + officialGaps: [ + { ...validPayload.officialGaps[0], query: "a".repeat(257) } + ] + }, + { + ...validPayload, + featuredCandidates: [ + { + ...validPayload.featuredCandidates[0], + package: { + ...validPayload.featuredCandidates[0].package, + isOfficial: true + } + } + ] + }, + { ...validPayload, classificationStatus: "unavailable" }, + { ...validPayload, classificationStatus: ["available"] }, + { ...validPayload, currentMetadataStatus: ["available"] }, + { + ...validPayload, + dashboardUrl: "https://clawhub.ai/" + "<".repeat(1000) + }, + { ...validPayload, currentMetadataStatus: "unavailable" }, + { + ...validPayload, + coverage: { ...validPayload.coverage, gapStart: validPayload.weekStart } + } + ] + for (const payload of invalid) { + expect( + (await handleSearchIntelligenceApiRequest(request(payload), client)) + ?.status + ).toBe(400) + } + const malformed = request() + expect( + ( + await handleSearchIntelligenceApiRequest( + new Request(malformed.url, { + method: "POST", + headers: malformed.headers, + body: "{" + }), + client + ) + )?.status + ).toBe(400) + expect( + ( + await handleSearchIntelligenceApiRequest( + request({ padding: "x".repeat(65537) }), + client + ) + )?.status + ).toBe(413) + expect(posts).toHaveLength(0) + }) + + it("persists one immutable receipt per week across duplicate requests", async () => { + const { client, posts, owner } = setup() + const first = await handleSearchIntelligenceApiRequest(request(), client) + const reordered = Object.fromEntries(Object.entries(validPayload).reverse()) + const replay = await handleSearchIntelligenceApiRequest( + request(reordered), + client + ) + expect(await first?.json()).toEqual({ + ok: true, + delivered: true, + weekEnd: validPayload.weekEnd + }) + expect(await replay?.json()).toEqual({ + ok: true, + delivered: true, + weekEnd: validPayload.weekEnd + }) + expect(posts).toHaveLength(1) + expect(posts[0].body.enforce_nonce).toBe(true) + expect(String(posts[0].body.nonce).length).toBeLessThanOrEqual(25) + const changed = { ...validPayload, truncated: true } + expect( + (await handleSearchIntelligenceApiRequest(request(changed), client)) + ?.status + ).toBe(409) + expect(posts).toHaveLength(1) + const rows = owner.database + .query("SELECT value FROM keyValue") + .all() as Array<{ value: string }> + expect(rows).toHaveLength(1) + expect(JSON.parse(rows[0].value).messageId).toBe("message-1") + expect(rows[0].value).not.toContain("notion") + }) + + it("holds concurrent duplicates behind the durable claim", async () => { + const { client, posts } = setup() + let release!: () => void + let started!: () => void + const sending = new Promise((resolve) => { + started = resolve + }) + const blocked = new Promise((resolve) => { + release = resolve + }) + const original = client.rest.post.bind(client.rest) + client.rest.post = (async (...args: Parameters) => { + started() + await blocked + return original(...args) + }) as typeof client.rest.post + const first = handleSearchIntelligenceApiRequest(request(), client) + await sending + expect( + (await handleSearchIntelligenceApiRequest(request(), client))?.status + ).toBe(409) + release() + expect((await first)?.status).toBe(200) + expect(posts).toHaveLength(1) + }) + it("retries only confirmed Discord rejection, preserving the weekly nonce", async () => { + const { client, posts } = setup() + const original = client.rest.post.bind(client.rest) + let attempts = 0 + client.rest.post = (async (...args: Parameters) => { + if (++attempts === 1) + throw Object.assign(new Error("Forbidden"), { status: 403 }) + return original(...args) + }) as typeof client.rest.post + expect( + (await handleSearchIntelligenceApiRequest(request(), client))?.status + ).toBe(502) + expect( + (await handleSearchIntelligenceApiRequest(request(), client))?.status + ).toBe(200) + expect(attempts).toBe(2) + expect(posts).toHaveLength(1) + }) + + it("reconciles an accepted message after a lost Discord response without reposting", async () => { + const { client, posts } = setup() + const original = client.rest.post.bind(client.rest) + client.rest.post = (async (...args: Parameters) => { + await original(...args) + throw new Error("Response lost") + }) as typeof client.rest.post + expect( + (await handleSearchIntelligenceApiRequest(request(), client))?.status + ).toBe(503) + client.rest.get = async () => [ + { + id: "message-1", + author: { id: "bot-user", bot: true }, + timestamp: new Date().toISOString(), + components: posts[0].body.components + } + ] + expect( + (await handleSearchIntelligenceApiRequest(request(), client))?.status + ).toBe(200) + expect(posts).toHaveLength(1) + }) + it("never blindly replays an uncertain message when channel history cannot confirm it", async () => { + const { client } = setup() + let posts = 0 + client.rest.post = async () => { + posts++ + throw new Error("Timed out") + } + expect( + (await handleSearchIntelligenceApiRequest(request(), client))?.status + ).toBe(503) + for (let retry = 0; retry < 3; retry++) + expect( + (await handleSearchIntelligenceApiRequest(request(), client))?.status + ).toBe(503) + expect(posts).toBe(1) + }) + + it("shows incomplete coverage, unavailable enrichments and a localhost preview label", async () => { + const { client, posts, owner } = setup() + setRuntimeEnv({ + DB: owner as unknown as D1Database, + CLAWHUB_HERMIT_TOKEN: "test-service-token", + CLAWHUB_SITE_URL: "http://localhost:4311", + DISCORD_CLIENT_ID: "bot-user" + } as Env) + const payload = { + ...validPayload, + dashboardUrl: "http://localhost:4311/management/search-insights", + totalSearches: 0, + sourceCounts: { clawhubWeb: 0, openclawControlUi: 0 }, + coverage: { + dataThrough: null, + collectionStartedAt: null, + gapStart: validPayload.weekStart, + gapEnd: validPayload.weekEnd + }, + classificationStatus: "unavailable", + currentMetadataStatus: "unavailable", + companyOpportunities: [], + officialGaps: [], + featuredCandidates: [], + movers: [] + } + expect( + (await handleSearchIntelligenceApiRequest(request(payload), client)) + ?.status + ).toBe(200) + const text = (posts[0].body.components as unknown[]) + .flatMap(texts) + .join("\n") + expect(text).toContain("LOCAL PREVIEW") + expect(text).toContain("Data through: unknown") + expect(text).toContain("Collection started: unknown") + expect(text).toContain("Collection gap") + expect(text).toContain("Incomplete collection history") + expect(text).toContain("Classification unavailable") + expect(text).toContain("Current package metadata unavailable") + }) + it("caps rendered text while retaining sections, coverage and the dashboard link", async () => { + const { client, posts } = setup() + const query = "@everyone [click](https://evil.example) ".repeat(5) + const rows = Array.from({ length: 5 }, (_, index) => ({ + ...validPayload.officialGaps[0], + query: query + index, + searchUrl: "https://clawhub.ai/plugins?q=" + "x".repeat(1000) + })) + const payload = { + ...validPayload, + truncated: true, + companyOpportunities: rows.map((row) => ({ ...row, confidence: 0.9 })), + officialGaps: rows, + movers: rows, + featuredCandidates: rows.map((row) => ({ + ...row, + package: validPayload.featuredCandidates[0].package + })) + } + expect( + (await handleSearchIntelligenceApiRequest(request(payload), client)) + ?.status + ).toBe(200) + const text = (posts[0].body.components as unknown[]) + .flatMap(texts) + .join("\n") + expect(text.length).toBeLessThanOrEqual(4000) + expect(text).toContain("Company plugin opportunities") + expect(text).toContain("Official gaps") + expect(text).toContain("Featured candidates") + expect(text).toContain("Week-over-week movers") + expect(text).toContain(validPayload.dashboardUrl) + expect(text).toContain("More rows on the dashboard") + expect(text).toContain("Input capped") + expect(text).not.toContain("@everyone") + expect(text).not.toContain("[click](https://evil.example)") + }) + + it("does not send without the durable claim and reconciles a post-send receipt failure", async () => { + const { client, posts, owner } = setup() + owner.database.exec( + "CREATE TRIGGER fail_claim BEFORE INSERT ON keyValue BEGIN SELECT RAISE(FAIL, 'storage down'); END" + ) + expect( + (await handleSearchIntelligenceApiRequest(request(), client))?.status + ).toBe(503) + expect(posts).toHaveLength(0) + owner.database.exec( + "DROP TRIGGER fail_claim; CREATE TRIGGER fail_receipt BEFORE UPDATE ON keyValue BEGIN SELECT RAISE(FAIL, 'storage down'); END" + ) + expect( + (await handleSearchIntelligenceApiRequest(request(), client))?.status + ).toBe(503) + expect(posts).toHaveLength(1) + owner.database.exec("DROP TRIGGER fail_receipt") + const now = Date.now() + mockClock = spyOn(Date, "now").mockReturnValue(now + 300_000) + client.rest.get = async () => [ + { + id: "message-1", + author: { id: "bot-user", bot: true }, + timestamp: new Date(now).toISOString(), + components: posts[0].body.components + } + ] + expect( + (await handleSearchIntelligenceApiRequest(request(), client))?.status + ).toBe(200) + expect( + (await handleSearchIntelligenceApiRequest(request(), client))?.status + ).toBe(200) + expect(posts).toHaveLength(1) + }) + it("rejects copied or stale history and bounds reconciliation reads", async () => { + const { client, posts } = setup() + const original = client.rest.post.bind(client.rest) + client.rest.post = (async (...args: Parameters) => { + await original(...args) + throw Object.assign(new Error("Gateway failure"), { status: 502 }) + }) as typeof client.rest.post + expect( + (await handleSearchIntelligenceApiRequest(request(), client))?.status + ).toBe(503) + const row = { + id: "message-1", + author: { id: "another-user", bot: true }, + timestamp: new Date().toISOString(), + components: posts[0].body.components + } + client.rest.get = async () => [row] + expect( + (await handleSearchIntelligenceApiRequest(request(), client))?.status + ).toBe(503) + client.rest.get = async () => [ + { + ...row, + author: { id: "bot-user", bot: true }, + timestamp: "2020-01-01T00:00:00.000Z" + } + ] + expect( + (await handleSearchIntelligenceApiRequest(request(), client))?.status + ).toBe(503) + let reads = 0 + client.rest.get = async () => { + reads++ + return Array.from({ length: 100 }, (_, i) => ({ + ...row, + id: `${reads}-${i}` + })) + } + expect( + (await handleSearchIntelligenceApiRequest(request(), client))?.status + ).toBe(503) + expect(reads).toBe(5) + client.rest.get = async () => { + throw Object.assign(new Error("Missing history permission"), { + status: 403 + }) + } + expect( + (await handleSearchIntelligenceApiRequest(request(), client))?.status + ).toBe(503) + expect(posts).toHaveLength(1) + }) + + it("includes threshold-qualified movers that dropped to zero without exposing rare weeks", async () => { + const { client, posts } = setup() + const payload = { + ...validPayload, + movers: [ + { + ...validPayload.movers[0], + searches: 0, + previousSearches: 4, + officialGaps: 0 + } + ] + } + expect( + (await handleSearchIntelligenceApiRequest(request(payload), client)) + ?.status + ).toBe(200) + const text = (posts[0].body.components as unknown[]) + .flatMap(texts) + .join("\n") + expect(text).toContain("0 searches · 0 gaps · previous 4") + const tooRare = { + ...payload, + movers: [{ ...payload.movers[0], searches: 1, previousSearches: 2 }] + } + expect( + (await handleSearchIntelligenceApiRequest(request(tooRare), client)) + ?.status + ).toBe(400) + expect(posts).toHaveLength(1) + }) +})