Sensor conformance kit: capture validator, fake engine, honesty checklist

README.md

@@ -9,6 +9,14 @@ engine. If you can write a JSON file, you can build a Bevia sensor.
 - **[schema/](schema/)** — JSON Schemas for validation.
 - **[examples/](examples/)** — real capture files: a chat session, a
   robot maintenance session, a captured note.
+- **[tools/](tools/)** — zero-dependency conformance tools:
+  `check-capture.mjs` validates your capture bodies against the spec,
+  and `fake-engine.mjs` is a local test double (HTTP + drop-file
+  watcher) that answers honestly — what landed, what deduped, why
+  something was rejected — so you can build a sensor without an engine.
+- **[SENSOR-CHECKLIST.md](SENSOR-CHECKLIST.md)** — the five honesty
+  rules every sensor should meet, with a troubleshooting table for
+  "my data isn't landing."
 - **[reference/](reference/)** — reference sensor implementations.
 
 ## The one-paragraph version

SENSOR-CHECKLIST.md

@@ -0,0 +1,101 @@
+# The sensor honesty checklist
+
+Five rules that keep a sensor trustworthy, learned the hard way from
+auditing real intake lanes. Every one of them guards against the same
+failure: **silent emptiness** — the sensor reports success, the map
+stays empty, and nobody finds out for weeks.
+
+Test against them locally with the tools in [`tools/`](tools/):
+
+```sh
+# 1. Validate your capture bodies before touching any engine:
+node tools/check-capture.mjs my-session.bevia-capture.json
+
+# 2. Run the fake engine and point your sensor at it:
+node tools/fake-engine.mjs --token TESTKEY --watch ./drops
+```
+
+---
+
+## Rule 1 — Carry event time on every turn
+
+`emitted_at` is when the thing *happened*, not when you exported it.
+A turn without it gets stamped at ingest time — history delivered late
+looks like it happened today, and every time-shaped question the map
+answers about it is wrong forever.
+
+**Test:** `check-capture.mjs` warns per-turn when event time is missing.
+
+## Rule 2 — Never advance your cursor before the write is acknowledged
+
+Whatever your sensor uses to remember "I already sent this" — a
+watermark, a hash, a page cursor — advance it only **after** delivery
+succeeds. Advance-then-deliver means one failed delivery strands that
+data forever: the cursor says done, the map never saw it.
+
+Corollary: **caps must resume, never strand.** If you batch (send at
+most N per run), the un-sent remainder must still be behind the cursor
+so the next run picks it up. A cap that skips the tail while the cursor
+moves past it is the same bug in slow motion.
+
+**Test:** kill the fake engine mid-run; restart it; run your sensor
+again. Everything unsent must arrive. Nothing should double-land
+(that's what thread-scoped dedup is for — see Rule 5).
+
+## Rule 3 — Say every skip out loud
+
+Anything your sensor decides not to send — an unsupported format, an
+entry with no id, a malformed record — must be visible in its log.
+A silent `continue` is invisible under-ingest: the user believes the
+map covers their world, and it quietly doesn't.
+
+**Test:** feed your sensor something it can't handle and read its log.
+If the log looks the same as a clean run, you have this bug.
+
+## Rule 4 — A rejected credential is not "unreachable"
+
+An unreachable engine self-heals: retry and eventually it's back. A
+wrong token **never** self-heals. If your sensor logs a 401/403 as
+"engine down, will retry", it will retry forever and the user reads
+"connecting…" while the real fix — check the key — is never named.
+
+**Test:** `fake-engine.mjs --token RIGHT`, run your sensor with the
+wrong token. Its log must say the credential was rejected, not that
+the engine was down.
+
+## Rule 5 — Re-send whole threads; trust dedup; read the counts
+
+Delivery is repeat-safe (SPEC §4.3): re-sending a capture, or a session
+that grew, is always safe within a `thread_id`. So prefer re-sending
+the whole thread over fragile delta tracking — and build `thread_id`
+deterministically from source identifiers, never randomly per delivery
+(a random thread id defeats dedup and doubles your map).
+
+Then **read the response counts, not just the status code**. A `200`
+with zero turns landed means dedup (fine on a re-send) — or an empty
+capture (a bug on a first send). Only the count tells you which.
+
+**Test:** POST the same body to the fake engine twice. First response:
+`turns_landed: N`. Second: `turns_deduped: N`. If you see
+`turns_landed: 0` on the *first* send, your capture is empty.
+
+---
+
+## Troubleshooting: "my data isn't landing"
+
+| Symptom | Likely cause | Check |
+|---|---|---|
+| Delivery succeeds, map empty | Every turn has empty `text` | `check-capture.mjs` — it 400-classes this exactly |
+| Drop files ignored, no error | Wrong suffix — only `*.bevia-capture.json` is delivered | File name (SPEC §2) |
+| Drop files half-read | Partial write raced the watcher | Write temp name, then rename (SPEC §2) |
+| History all dated today | Missing `emitted_at` | Rule 1 |
+| Same session lands twice | Random / unstable `thread_id` | Rule 5 |
+| Sensor says "retrying" forever | 401 treated as outage | Rule 4 |
+| Old items never arrive | Cursor advanced past a failed or capped batch | Rule 2 |
+| Location missing on the map | Malformed `location` object (dropped silently, by design) | `check-capture.mjs` warns on it |
+
+---
+
+Everything here derives from [SPEC.md](SPEC.md) — the public contract.
+None of it depends on how any engine works inside, and the tools in
+`tools/` contain no engine code: they encode the spec, nothing else.

tools/check-capture.mjs

@@ -0,0 +1,60 @@
+#!/usr/bin/env node
+// Bevia Capture Protocol — capture validator (v1).
+//
+// Checks capture files (or stdin) against SPEC.md and says, before you
+// ever touch an engine, exactly what would land, what would be skipped,
+// and what you're silently losing (event time, thread identity).
+//
+// Usage:
+//   node tools/check-capture.mjs my-session.bevia-capture.json [more…]
+//   cat body.json | node tools/check-capture.mjs -
+//
+// Exit code: 0 when every input would land (warnings allowed),
+//            1 when any input would be rejected or can't be read.
+//
+// Zero dependencies. Encodes only the public contract in SPEC.md.
+
+import { readFileSync } from 'node:fs';
+import { checkCaptureBody, formatReport } from './lib/check.mjs';
+
+const args = process.argv.slice(2);
+if (args.length === 0 || args.includes('--help') || args.includes('-h')) {
+  console.log('usage: check-capture.mjs <file.bevia-capture.json …|->');
+  console.log('Validates capture bodies against the Bevia Capture Protocol v1 (SPEC.md).');
+  process.exit(args.length === 0 ? 1 : 0);
+}
+
+let failed = false;
+
+for (const arg of args) {
+  const name = arg === '-' ? '<stdin>' : arg;
+  let raw;
+  try {
+    raw = arg === '-' ? readFileSync(0, 'utf8') : readFileSync(arg, 'utf8');
+  } catch (err) {
+    console.log(`${name}: ERROR cannot read (${err.message})`);
+    failed = true;
+    continue;
+  }
+
+  let body;
+  try {
+    body = JSON.parse(raw);
+  } catch (err) {
+    console.log(`${name}: ERROR not valid JSON (${err.message})`);
+    failed = true;
+    continue;
+  }
+
+  const result = checkCaptureBody(body);
+  if (arg !== '-' && !arg.endsWith('.bevia-capture.json')) {
+    result.warnings.push(
+      'file name does not end in ".bevia-capture.json" — folder-watching engines deliver ' +
+      'ONLY that suffix (SPEC §2). A drop file with the wrong name is ignored without an error.',
+    );
+  }
+  console.log(formatReport(name, result));
+  if (result.errors.length > 0) failed = true;
+}
+
+process.exit(failed ? 1 : 0);

tools/fake-engine.mjs

@@ -0,0 +1,157 @@
+#!/usr/bin/env node
+// Bevia Capture Protocol — the fake engine (v1).
+//
+// A local test double for sensor builders: it accepts captures the two
+// ways a real Bevia engine does — HTTP POST and drop files — validates
+// them against SPEC.md, and answers HONESTLY: what landed, what was
+// skipped, what deduped, and why something was rejected. Build your
+// sensor against this before you ever point it at a real engine.
+//
+// What it is NOT: a Bevia engine. It stores nothing durable, computes
+// nothing, and its response body is its own (the real engine's accept/
+// reject SEMANTICS match SPEC §4; its response fields may differ).
+//
+// Usage:
+//   node tools/fake-engine.mjs                    # HTTP on :8787
+//   node tools/fake-engine.mjs --port 9000
+//   node tools/fake-engine.mjs --token SECRET     # require Authorization: Bearer SECRET
+//   node tools/fake-engine.mjs --watch ./drops    # also validate drop files
+//
+// Then:
+//   curl -s -X POST localhost:8787/intake/capture -d @examples/chat-session.bevia-capture.json
+//
+// Teaching notes baked in:
+//   • 401 ≠ unreachable. A wrong token gets a 401 that SAYS the token
+//     was rejected — if your sensor logs that as "engine down, will
+//     retry", it will retry forever. Distinguish them.
+//   • Repeat-safe delivery (SPEC §4.3): re-POST the same thread and the
+//     response shows turns_deduped instead of double-landing. Re-send
+//     whole threads; don't build fragile delta tracking.
+//   • Silent emptiness: a capture whose every turn is empty "succeeds"
+//     as a delivery and lands nothing. The fake engine 400s it and
+//     names the failure so you fix it at build time.
+//
+// Zero dependencies. Encodes only the public contract in SPEC.md.
+
+import { createServer } from 'node:http';
+import { watch, readFileSync, existsSync, statSync } from 'node:fs';
+import { join } from 'node:path';
+import { checkCaptureBody, formatReport } from './lib/check.mjs';
+
+const args = process.argv.slice(2);
+const flag = (name) => {
+  const i = args.indexOf(name);
+  return i >= 0 ? args[i + 1] : undefined;
+};
+const PORT = Number(flag('--port') ?? 8787);
+const TOKEN = flag('--token');
+const WATCH_DIR = flag('--watch');
+
+// Per-thread content memory — enough to demonstrate SPEC §4.3
+// (content-based dedup within a thread) without being an engine.
+const seenByThread = new Map(); // thread_id -> Set<turn text>
+
+function landCapture(body, via) {
+  const result = checkCaptureBody(body);
+  if (result.errors.length > 0) {
+    return { status: 400, report: result, landed: 0, deduped: 0 };
+  }
+  const threadKey = result.threadId ?? `(no-thread-id:${via})`;
+  const seen = seenByThread.get(threadKey) ?? new Set();
+  let landed = 0;
+  let deduped = 0;
+  for (const turn of body.capture.conversation) {
+    const text = typeof turn?.text === 'string' ? turn.text.trim() : '';
+    if (!text) continue;
+    if (seen.has(text)) deduped++;
+    else {
+      seen.add(text);
+      landed++;
+    }
+  }
+  seenByThread.set(threadKey, seen);
+  return { status: 200, report: result, landed, deduped };
+}
+
+function log(line) {
+  console.log(`[fake-engine] ${line}`);
+}
+
+const server = createServer((req, res) => {
+  const answer = (status, obj) => {
+    res.writeHead(status, { 'content-type': 'application/json' });
+    res.end(JSON.stringify(obj, null, 2));
+  };
+
+  if (req.method !== 'POST' || req.url !== '/intake/capture') {
+    return answer(404, { error: 'POST /intake/capture is the only door here (SPEC §1). Drop files are the other door (SPEC §2) — use --watch.' });
+  }
+
+  if (TOKEN) {
+    const auth = req.headers.authorization ?? '';
+    if (auth !== `Bearer ${TOKEN}`) {
+      log('401 — token rejected (this is NOT "unreachable"; do not log it as a retryable outage)');
+      return answer(401, {
+        error: 'token rejected',
+        hint: 'A 401 means your credential is wrong and will never self-heal. Log it as "key rejected — check the token", never as "engine unreachable — will retry".',
+      });
+    }
+  }
+
+  let raw = '';
+  req.on('data', (c) => { raw += c; });
+  req.on('end', () => {
+    let body;
+    try {
+      body = JSON.parse(raw || 'null');
+    } catch (err) {
+      log(`400 — body is not JSON (${err.message})`);
+      return answer(400, { error: `body is not valid JSON: ${err.message}` });
+    }
+    const { status, report, landed, deduped } = landCapture(body, 'http');
+    console.log(formatReport('POST /intake/capture', report));
+    if (status !== 200) {
+      return answer(400, { error: 'capture rejected — nothing would land', reasons: report.errors, warnings: report.warnings });
+    }
+    return answer(200, {
+      ok: true,
+      thread_id: report.threadId ?? null,
+      turns_landed: landed,
+      turns_deduped: deduped,
+      turns_skipped_empty: report.skippedEmpty,
+      warnings: report.warnings,
+      note: 'fake engine: semantics match SPEC §4; a real engine\'s response fields may differ. Read counts, not just status codes — 200 with turns_landed:0 on a re-send is dedup working, but 200 with turns_landed:0 on FIRST send means your capture is empty.',
+    });
+  });
+});
+
+server.listen(PORT, '127.0.0.1', () => {
+  log(`listening on http://127.0.0.1:${PORT}/intake/capture${TOKEN ? ' (Bearer token required)' : ''}`);
+  log('POST capture bodies at it; every response says what landed and why.');
+});
+
+if (WATCH_DIR) {
+  if (!existsSync(WATCH_DIR)) {
+    log(`--watch: ${WATCH_DIR} does not exist`);
+    process.exit(1);
+  }
+  log(`watching ${WATCH_DIR} for *.bevia-capture.json drop files`);
+  const seenAt = new Map(); // path -> mtimeMs, so rewrites re-validate
+  watch(WATCH_DIR, (_event, file) => {
+    if (!file || !file.endsWith('.bevia-capture.json')) return;
+    const path = join(WATCH_DIR, file);
+    try {
+      const mtime = statSync(path).mtimeMs;
+      if (seenAt.get(path) === mtime) return;
+      seenAt.set(path, mtime);
+      const body = JSON.parse(readFileSync(path, 'utf8'));
+      const { report, landed, deduped } = landCapture(body, path);
+      console.log(formatReport(file, report));
+      if (report.errors.length === 0) log(`${file}: ${landed} landed, ${deduped} deduped (rewrite-with-growth is the correct update path, SPEC §2)`);
+    } catch (err) {
+      // Partial write (SPEC §2 says temp-name-then-rename) or vanish —
+      // say it, don't silently skip.
+      log(`${file}: could not read/parse (${err.message}) — if you saw this mid-write, write to a temp name and rename (SPEC §2).`);
+    }
+  });
+}