`pollLogs` dedup silently drops distinct events sharing a timestamp — remove the dedup pass

[kcarnold]

What

pollLogs (backend/src/logging.ts) dedupes each user's JSONL before slicing:

const key = `${entry.timestamp}|${entry.username}`;

The docstring above it says the key is timestamp|interaction|username. interaction doesn't exist in the schema any more — it's a leftover from the FastAPI-era log shape. And since the file is already per-username, username in the key is a constant, so the effective rule is one entry per timestamp, per user.

Why it matters

Two different events that land on the same millisecond collapse to one, and the loser is gone from the live viewer permanently — not just for that poll, since the dedup re-runs over the whole file each time. Timestamps are Date.now() / 1000 stamped in useLog at fetch-construction time, so collisions are entirely ordinary: Revise fires features_run and then a visualization_requested per selected feature in a tight loop, all in the same tick.

Severity is bounded, and worth stating plainly: the JSONL on disk is untouched, and zipLogs / the viewer's file mode don't dedupe. No data is lost — the raw logs and the analysis scripts in scripts/ see everything. It's the live study-log viewer (frontend/src/logs/index.tsx, polling /api/logs_poll) that under-reports.

Suggested fix: delete the dedup, don't repair the key

pollLogs itself should stay — it's live, it's gated behind LOG_SECRET, and the log viewer depends on it. The stale part isn't the endpoint, it's the dedup pass, and the right move is to remove it rather than fix its key:

• There's no known duplicate source. appendLog writes one line per request, and useLog does a single fetch with no retry logic — the swallowed-error path drops the event, it doesn't resend it.
• Position slicing works better on raw lines: the file is append-only, so a raw line index is a stable cursor, whereas a deduped index shifts under the client whenever the dedup drops something.
• A dedup that silently drops distinct events is worse than the duplicates it was guarding against.

If duplicates ever show up for real, add dedup back then, keyed on something that actually identifies an event (timestamp|event|page or a client-generated event id).

Also

• backend/src/__tests__/logging.test.ts has a test asserting the dedup behavior ("dedupes by timestamp+username") — it needs updating alongside.
• One-time cosmetic effect on deploy: a viewer holding n deduped entries will re-receive the few entries the dedup had been swallowing, and append them without dedup (the client appends blind). Harmless in a live viewer; worth knowing so it isn't mistaken for a new bug.
• While in there, validateUsername permits the empty string, so logFilePath('') resolves to .jsonl — which pollLogs then explicitly skips. Writable but unreadable. Probably worth rejecting empty outright.