From b8ce60dd72c59d8442b2610f5bd0432fa3669f9f Mon Sep 17 00:00:00 2001 From: Claude Date: Tue, 21 Jul 2026 05:48:50 +0000 Subject: [PATCH] Sensor conformance kit: capture validator, fake engine, honesty checklist MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Builder-facing test tooling so sensor authors catch the silent-emptiness failure classes at build time instead of in production: - tools/check-capture.mjs — validates capture bodies/files against SPEC.md: rejects nothing-would-land captures, errors on malformed event time, warns on missing emitted_at (ingest-time stamping), missing/unstable thread_id, missing source_kind, malformed location, and wrong drop-file suffix. - tools/fake-engine.mjs — a local test double serving both doors (POST /intake/capture + --watch drop-file folder). Demonstrates the SPEC §4 semantics honestly: 401 that names a rejected token (never 'unreachable'), 400 on empty captures, per-thread content dedup on re-send (turns_landed vs turns_deduped), partial-write guidance. - SENSOR-CHECKLIST.md — the five honesty rules (event time; cursor advances only after acked writes and caps must resume; say every skip out loud; 401 is not an outage; re-send whole threads and read the counts) + a troubleshooting table. Zero dependencies, zero engine code — everything derives from SPEC.md. Co-Authored-By: Claude Fable 5 Claude-Session: https://claude.ai/code/session_013ejbYvWjhjfx8nrasS8p7D --- README.md | 8 ++ SENSOR-CHECKLIST.md | 101 ++++++++++++++++++++++++++ tools/check-capture.mjs | 60 +++++++++++++++ tools/fake-engine.mjs | 157 ++++++++++++++++++++++++++++++++++++++++ tools/lib/check.mjs | 111 ++++++++++++++++++++++++++++ 5 files changed, 437 insertions(+) create mode 100644 SENSOR-CHECKLIST.md create mode 100644 tools/check-capture.mjs create mode 100644 tools/fake-engine.mjs create mode 100644 tools/lib/check.mjs diff --git a/README.md b/README.md index 1d15fa7..6737064 100644 --- a/README.md +++ b/README.md @@ -9,6 +9,14 @@ engine. If you can write a JSON file, you can build a Bevia sensor. - **[schema/](schema/)** — JSON Schemas for validation. - **[examples/](examples/)** — real capture files: a chat session, a robot maintenance session, a captured note. +- **[tools/](tools/)** — zero-dependency conformance tools: + `check-capture.mjs` validates your capture bodies against the spec, + and `fake-engine.mjs` is a local test double (HTTP + drop-file + watcher) that answers honestly — what landed, what deduped, why + something was rejected — so you can build a sensor without an engine. +- **[SENSOR-CHECKLIST.md](SENSOR-CHECKLIST.md)** — the five honesty + rules every sensor should meet, with a troubleshooting table for + "my data isn't landing." - **[reference/](reference/)** — reference sensor implementations. ## The one-paragraph version diff --git a/SENSOR-CHECKLIST.md b/SENSOR-CHECKLIST.md new file mode 100644 index 0000000..95ef526 --- /dev/null +++ b/SENSOR-CHECKLIST.md @@ -0,0 +1,101 @@ +# The sensor honesty checklist + +Five rules that keep a sensor trustworthy, learned the hard way from +auditing real intake lanes. Every one of them guards against the same +failure: **silent emptiness** — the sensor reports success, the map +stays empty, and nobody finds out for weeks. + +Test against them locally with the tools in [`tools/`](tools/): + +```sh +# 1. Validate your capture bodies before touching any engine: +node tools/check-capture.mjs my-session.bevia-capture.json + +# 2. Run the fake engine and point your sensor at it: +node tools/fake-engine.mjs --token TESTKEY --watch ./drops +``` + +--- + +## Rule 1 — Carry event time on every turn + +`emitted_at` is when the thing *happened*, not when you exported it. +A turn without it gets stamped at ingest time — history delivered late +looks like it happened today, and every time-shaped question the map +answers about it is wrong forever. + +**Test:** `check-capture.mjs` warns per-turn when event time is missing. + +## Rule 2 — Never advance your cursor before the write is acknowledged + +Whatever your sensor uses to remember "I already sent this" — a +watermark, a hash, a page cursor — advance it only **after** delivery +succeeds. Advance-then-deliver means one failed delivery strands that +data forever: the cursor says done, the map never saw it. + +Corollary: **caps must resume, never strand.** If you batch (send at +most N per run), the un-sent remainder must still be behind the cursor +so the next run picks it up. A cap that skips the tail while the cursor +moves past it is the same bug in slow motion. + +**Test:** kill the fake engine mid-run; restart it; run your sensor +again. Everything unsent must arrive. Nothing should double-land +(that's what thread-scoped dedup is for — see Rule 5). + +## Rule 3 — Say every skip out loud + +Anything your sensor decides not to send — an unsupported format, an +entry with no id, a malformed record — must be visible in its log. +A silent `continue` is invisible under-ingest: the user believes the +map covers their world, and it quietly doesn't. + +**Test:** feed your sensor something it can't handle and read its log. +If the log looks the same as a clean run, you have this bug. + +## Rule 4 — A rejected credential is not "unreachable" + +An unreachable engine self-heals: retry and eventually it's back. A +wrong token **never** self-heals. If your sensor logs a 401/403 as +"engine down, will retry", it will retry forever and the user reads +"connecting…" while the real fix — check the key — is never named. + +**Test:** `fake-engine.mjs --token RIGHT`, run your sensor with the +wrong token. Its log must say the credential was rejected, not that +the engine was down. + +## Rule 5 — Re-send whole threads; trust dedup; read the counts + +Delivery is repeat-safe (SPEC §4.3): re-sending a capture, or a session +that grew, is always safe within a `thread_id`. So prefer re-sending +the whole thread over fragile delta tracking — and build `thread_id` +deterministically from source identifiers, never randomly per delivery +(a random thread id defeats dedup and doubles your map). + +Then **read the response counts, not just the status code**. A `200` +with zero turns landed means dedup (fine on a re-send) — or an empty +capture (a bug on a first send). Only the count tells you which. + +**Test:** POST the same body to the fake engine twice. First response: +`turns_landed: N`. Second: `turns_deduped: N`. If you see +`turns_landed: 0` on the *first* send, your capture is empty. + +--- + +## Troubleshooting: "my data isn't landing" + +| Symptom | Likely cause | Check | +|---|---|---| +| Delivery succeeds, map empty | Every turn has empty `text` | `check-capture.mjs` — it 400-classes this exactly | +| Drop files ignored, no error | Wrong suffix — only `*.bevia-capture.json` is delivered | File name (SPEC §2) | +| Drop files half-read | Partial write raced the watcher | Write temp name, then rename (SPEC §2) | +| History all dated today | Missing `emitted_at` | Rule 1 | +| Same session lands twice | Random / unstable `thread_id` | Rule 5 | +| Sensor says "retrying" forever | 401 treated as outage | Rule 4 | +| Old items never arrive | Cursor advanced past a failed or capped batch | Rule 2 | +| Location missing on the map | Malformed `location` object (dropped silently, by design) | `check-capture.mjs` warns on it | + +--- + +Everything here derives from [SPEC.md](SPEC.md) — the public contract. +None of it depends on how any engine works inside, and the tools in +`tools/` contain no engine code: they encode the spec, nothing else. diff --git a/tools/check-capture.mjs b/tools/check-capture.mjs new file mode 100644 index 0000000..f6a9a37 --- /dev/null +++ b/tools/check-capture.mjs @@ -0,0 +1,60 @@ +#!/usr/bin/env node +// Bevia Capture Protocol — capture validator (v1). +// +// Checks capture files (or stdin) against SPEC.md and says, before you +// ever touch an engine, exactly what would land, what would be skipped, +// and what you're silently losing (event time, thread identity). +// +// Usage: +// node tools/check-capture.mjs my-session.bevia-capture.json [more…] +// cat body.json | node tools/check-capture.mjs - +// +// Exit code: 0 when every input would land (warnings allowed), +// 1 when any input would be rejected or can't be read. +// +// Zero dependencies. Encodes only the public contract in SPEC.md. + +import { readFileSync } from 'node:fs'; +import { checkCaptureBody, formatReport } from './lib/check.mjs'; + +const args = process.argv.slice(2); +if (args.length === 0 || args.includes('--help') || args.includes('-h')) { + console.log('usage: check-capture.mjs '); + console.log('Validates capture bodies against the Bevia Capture Protocol v1 (SPEC.md).'); + process.exit(args.length === 0 ? 1 : 0); +} + +let failed = false; + +for (const arg of args) { + const name = arg === '-' ? '' : arg; + let raw; + try { + raw = arg === '-' ? readFileSync(0, 'utf8') : readFileSync(arg, 'utf8'); + } catch (err) { + console.log(`${name}: ERROR cannot read (${err.message})`); + failed = true; + continue; + } + + let body; + try { + body = JSON.parse(raw); + } catch (err) { + console.log(`${name}: ERROR not valid JSON (${err.message})`); + failed = true; + continue; + } + + const result = checkCaptureBody(body); + if (arg !== '-' && !arg.endsWith('.bevia-capture.json')) { + result.warnings.push( + 'file name does not end in ".bevia-capture.json" — folder-watching engines deliver ' + + 'ONLY that suffix (SPEC §2). A drop file with the wrong name is ignored without an error.', + ); + } + console.log(formatReport(name, result)); + if (result.errors.length > 0) failed = true; +} + +process.exit(failed ? 1 : 0); diff --git a/tools/fake-engine.mjs b/tools/fake-engine.mjs new file mode 100644 index 0000000..b380c76 --- /dev/null +++ b/tools/fake-engine.mjs @@ -0,0 +1,157 @@ +#!/usr/bin/env node +// Bevia Capture Protocol — the fake engine (v1). +// +// A local test double for sensor builders: it accepts captures the two +// ways a real Bevia engine does — HTTP POST and drop files — validates +// them against SPEC.md, and answers HONESTLY: what landed, what was +// skipped, what deduped, and why something was rejected. Build your +// sensor against this before you ever point it at a real engine. +// +// What it is NOT: a Bevia engine. It stores nothing durable, computes +// nothing, and its response body is its own (the real engine's accept/ +// reject SEMANTICS match SPEC §4; its response fields may differ). +// +// Usage: +// node tools/fake-engine.mjs # HTTP on :8787 +// node tools/fake-engine.mjs --port 9000 +// node tools/fake-engine.mjs --token SECRET # require Authorization: Bearer SECRET +// node tools/fake-engine.mjs --watch ./drops # also validate drop files +// +// Then: +// curl -s -X POST localhost:8787/intake/capture -d @examples/chat-session.bevia-capture.json +// +// Teaching notes baked in: +// • 401 ≠ unreachable. A wrong token gets a 401 that SAYS the token +// was rejected — if your sensor logs that as "engine down, will +// retry", it will retry forever. Distinguish them. +// • Repeat-safe delivery (SPEC §4.3): re-POST the same thread and the +// response shows turns_deduped instead of double-landing. Re-send +// whole threads; don't build fragile delta tracking. +// • Silent emptiness: a capture whose every turn is empty "succeeds" +// as a delivery and lands nothing. The fake engine 400s it and +// names the failure so you fix it at build time. +// +// Zero dependencies. Encodes only the public contract in SPEC.md. + +import { createServer } from 'node:http'; +import { watch, readFileSync, existsSync, statSync } from 'node:fs'; +import { join } from 'node:path'; +import { checkCaptureBody, formatReport } from './lib/check.mjs'; + +const args = process.argv.slice(2); +const flag = (name) => { + const i = args.indexOf(name); + return i >= 0 ? args[i + 1] : undefined; +}; +const PORT = Number(flag('--port') ?? 8787); +const TOKEN = flag('--token'); +const WATCH_DIR = flag('--watch'); + +// Per-thread content memory — enough to demonstrate SPEC §4.3 +// (content-based dedup within a thread) without being an engine. +const seenByThread = new Map(); // thread_id -> Set + +function landCapture(body, via) { + const result = checkCaptureBody(body); + if (result.errors.length > 0) { + return { status: 400, report: result, landed: 0, deduped: 0 }; + } + const threadKey = result.threadId ?? `(no-thread-id:${via})`; + const seen = seenByThread.get(threadKey) ?? new Set(); + let landed = 0; + let deduped = 0; + for (const turn of body.capture.conversation) { + const text = typeof turn?.text === 'string' ? turn.text.trim() : ''; + if (!text) continue; + if (seen.has(text)) deduped++; + else { + seen.add(text); + landed++; + } + } + seenByThread.set(threadKey, seen); + return { status: 200, report: result, landed, deduped }; +} + +function log(line) { + console.log(`[fake-engine] ${line}`); +} + +const server = createServer((req, res) => { + const answer = (status, obj) => { + res.writeHead(status, { 'content-type': 'application/json' }); + res.end(JSON.stringify(obj, null, 2)); + }; + + if (req.method !== 'POST' || req.url !== '/intake/capture') { + return answer(404, { error: 'POST /intake/capture is the only door here (SPEC §1). Drop files are the other door (SPEC §2) — use --watch.' }); + } + + if (TOKEN) { + const auth = req.headers.authorization ?? ''; + if (auth !== `Bearer ${TOKEN}`) { + log('401 — token rejected (this is NOT "unreachable"; do not log it as a retryable outage)'); + return answer(401, { + error: 'token rejected', + hint: 'A 401 means your credential is wrong and will never self-heal. Log it as "key rejected — check the token", never as "engine unreachable — will retry".', + }); + } + } + + let raw = ''; + req.on('data', (c) => { raw += c; }); + req.on('end', () => { + let body; + try { + body = JSON.parse(raw || 'null'); + } catch (err) { + log(`400 — body is not JSON (${err.message})`); + return answer(400, { error: `body is not valid JSON: ${err.message}` }); + } + const { status, report, landed, deduped } = landCapture(body, 'http'); + console.log(formatReport('POST /intake/capture', report)); + if (status !== 200) { + return answer(400, { error: 'capture rejected — nothing would land', reasons: report.errors, warnings: report.warnings }); + } + return answer(200, { + ok: true, + thread_id: report.threadId ?? null, + turns_landed: landed, + turns_deduped: deduped, + turns_skipped_empty: report.skippedEmpty, + warnings: report.warnings, + note: 'fake engine: semantics match SPEC §4; a real engine\'s response fields may differ. Read counts, not just status codes — 200 with turns_landed:0 on a re-send is dedup working, but 200 with turns_landed:0 on FIRST send means your capture is empty.', + }); + }); +}); + +server.listen(PORT, '127.0.0.1', () => { + log(`listening on http://127.0.0.1:${PORT}/intake/capture${TOKEN ? ' (Bearer token required)' : ''}`); + log('POST capture bodies at it; every response says what landed and why.'); +}); + +if (WATCH_DIR) { + if (!existsSync(WATCH_DIR)) { + log(`--watch: ${WATCH_DIR} does not exist`); + process.exit(1); + } + log(`watching ${WATCH_DIR} for *.bevia-capture.json drop files`); + const seenAt = new Map(); // path -> mtimeMs, so rewrites re-validate + watch(WATCH_DIR, (_event, file) => { + if (!file || !file.endsWith('.bevia-capture.json')) return; + const path = join(WATCH_DIR, file); + try { + const mtime = statSync(path).mtimeMs; + if (seenAt.get(path) === mtime) return; + seenAt.set(path, mtime); + const body = JSON.parse(readFileSync(path, 'utf8')); + const { report, landed, deduped } = landCapture(body, path); + console.log(formatReport(file, report)); + if (report.errors.length === 0) log(`${file}: ${landed} landed, ${deduped} deduped (rewrite-with-growth is the correct update path, SPEC §2)`); + } catch (err) { + // Partial write (SPEC §2 says temp-name-then-rename) or vanish — + // say it, don't silently skip. + log(`${file}: could not read/parse (${err.message}) — if you saw this mid-write, write to a temp name and rename (SPEC §2).`); + } + }); +} diff --git a/tools/lib/check.mjs b/tools/lib/check.mjs new file mode 100644 index 0000000..00a7866 --- /dev/null +++ b/tools/lib/check.mjs @@ -0,0 +1,111 @@ +// Bevia Capture Protocol — shared conformance checks (v1). +// +// Everything in this file derives from SPEC.md alone. It encodes the +// public wire contract and the honesty rules a sensor should meet — +// nothing about how any engine turns captures into understanding. +// +// Used by tools/check-capture.mjs (CLI) and tools/fake-engine.mjs +// (local test double). Zero dependencies. + +/** RFC 3339-ish: what Date.parse accepts AND carries an explicit date. */ +export function isEventTime(s) { + if (typeof s !== 'string' || s.length < 10) return false; + const t = Date.parse(s); + return Number.isFinite(t); +} + +/** + * Check one capture body against the v1 contract. + * Returns { errors, warnings, landable, skippedEmpty, threadId }. + * + * errors → the capture would be rejected (nothing lands). + * warnings → the capture lands, but the sensor is losing something it + * will wish it had (event time, thread identity, …). + */ +export function checkCaptureBody(body) { + const errors = []; + const warnings = []; + + if (body === null || typeof body !== 'object' || Array.isArray(body)) { + return { errors: ['body is not a JSON object'], warnings, landable: 0, skippedEmpty: 0, threadId: undefined }; + } + + const capture = body.capture; + if (capture === null || capture === undefined || typeof capture !== 'object' || Array.isArray(capture)) { + errors.push('missing "capture" object — the body must be { "capture": { … } }'); + return { errors, warnings, landable: 0, skippedEmpty: 0, threadId: undefined }; + } + + const conv = capture.conversation; + if (!Array.isArray(conv) || conv.length === 0) { + errors.push('"capture.conversation" must be a non-empty array of turns'); + return { errors, warnings, landable: 0, skippedEmpty: 0, threadId: capture.thread_id }; + } + + let landable = 0; + let skippedEmpty = 0; + let missingTime = 0; + conv.forEach((turn, i) => { + if (turn === null || typeof turn !== 'object' || Array.isArray(turn)) { + warnings.push(`turn[${i}] is not an object — it will be skipped`); + skippedEmpty++; + return; + } + const hasText = typeof turn.text === 'string' && turn.text.trim().length > 0; + if (!hasText) { + // SPEC §1: a turn without non-empty text is skipped; the capture + // still lands. Say it out loud so nobody is surprised. + warnings.push(`turn[${i}] has no non-empty "text" — it will be skipped (the rest of the capture still lands)`); + skippedEmpty++; + } else { + landable++; + } + if (turn.emitted_at !== undefined && !isEventTime(turn.emitted_at)) { + errors.push(`turn[${i}].emitted_at is not a parseable RFC 3339 timestamp: ${JSON.stringify(turn.emitted_at)}`); + } else if (turn.emitted_at === undefined && hasText) { + missingTime++; + } + }); + + if (landable === 0) { + errors.push('no turn carries non-empty "text" — NOTHING would land. This is the silent-emptiness failure: the delivery "succeeds" and the map stays empty.'); + } + + if (missingTime > 0) { + warnings.push( + `${missingTime} turn(s) have no "emitted_at" — they will be stamped at INGEST time. ` + + 'History delivered late will look like it happened today. If your source knows when ' + + 'things happened, carry it (SPEC §4.2: event time is honored).', + ); + } + + const threadId = capture.thread_id; + if (threadId === undefined || String(threadId).trim() === '') { + warnings.push( + 'no "capture.thread_id" — the conversation has no stable identity. Re-sending a session ' + + 'that grew cannot dedup against its earlier delivery. Build it deterministically from ' + + 'source identifiers (SPEC §5), never randomly per delivery.', + ); + } + + if (capture.source_kind === undefined) { + warnings.push('no "capture.source_kind" — set it explicitly (lowercase snake_case, SPEC §5) so evidence stays attributable to its kind of source.'); + } + + const loc = capture.location; + if (loc !== undefined && (loc === null || typeof loc !== 'object' || Array.isArray(loc))) { + warnings.push('"capture.location" is not an object — engines drop malformed locations silently (SPEC §3: advisory, never rejecting). You lose the WHERE without an error.'); + } + + return { errors, warnings, landable, skippedEmpty, threadId: typeof threadId === 'string' ? threadId : undefined }; +} + +/** Render one file/body report as human-readable lines. */ +export function formatReport(name, result) { + const lines = []; + const verdict = result.errors.length > 0 ? 'REJECTED' : (result.warnings.length > 0 ? 'LANDS (with warnings)' : 'LANDS'); + lines.push(`${name}: ${verdict} — ${result.landable} turn(s) would land, ${result.skippedEmpty} skipped`); + for (const e of result.errors) lines.push(` ERROR ${e}`); + for (const w of result.warnings) lines.push(` warning ${w}`); + return lines.join('\n'); +}