fix(dashboard): serialize events.jsonl append and compaction (#804)

CHANGELOG.md

@@ -34,6 +34,7 @@ All notable changes to this project will be documented in this file. See [standa
 
 - A misspelled top-level key in `mcp/mcp.yaml` or `hooks/hooks.yaml` (`server:` for `servers:`, `hook:` for `hooks:`) no longer removes every installed team MCP server or hook: such a file read as empty. It now fails like a file that does not parse, so pull keeps what is installed, and pull and `teamai doctor` name the file, the keys found and the key expected. An extra top-level key beside `servers:` or `hooks:` is still ignored (for [#822](https://github.com/Tencent/teamai-cli/issues/822)).
 - The closing line of `teamai recall` output is in English (for [#822](https://github.com/Tencent/teamai-cli/issues/822)).
+- The dashboard event log no longer drops events to a compaction race. `~/.teamai/dashboard/events.jsonl` is appended by every dashboard hook on the machine, and compaction shrank it by a lock-free read → filter → temp-file → rename, so an event appended between the rewrite's read and its rename was overwritten and lost: the dashboard's sessions, interventions and prompt counts under-counted until the next rebuild. Every writer that may modify the file now takes `events.jsonl.lock` (the `acquireLock` retry the usage file took for the same lost update, [#803](https://github.com/Tencent/teamai-cli/issues/803)): a hook append waits up to ~250 ms, inside its foreground budget, and one that gives up records its line in an `events.pending-<uuid>.jsonl` side file that the next lock holder folds into the file before it writes, so an event is late, never gone; the line's `pendingId` keeps a fold from appending a side file twice, identical events in separate side files are both kept, and the id stays in the raw file — a compaction keeps it too, as the usage file's rewrite does — until the side file itself is gone, while no reader ever sees it. Side files fold in their events' own time order, and a compaction classifies sessions in time order as every reader does, so a side file that outlived newer appends cannot place an older event after them and re-mark a live session stopped. A compaction that finds the log below its threshold and no side files to fold still runs lock-free, so the common case costs one read and no lock. A rewrite waits up to ~5 s for a peer's rewrite and skips — leaving the file as it is for the next compaction — when a live holder outlasts the wait, and a lock whose owner is gone is reclaimed (for [#804](https://github.com/Tencent/teamai-cli/issues/804)).
 - A broken team file no longer wipes or downgrades what a member has installed. An `mcp/mcp.yaml` or `hooks/hooks.yaml` that did not parse reconciled to an empty set and removed every team MCP server or hook from every tool, and two active namespaces defining one skill or agent aborted the pull for the whole scope, skipping rules, env, docs and cleanup. Now a file in the active set that does not parse or cannot be read, a name repeated inside one file, or one name in two active namespaces stops only that resource type for the run: env keeps `env.sh`, hooks and MCP keep their entries (the built-in hooks, with the session-start pull, are still installed where missing: with the root hooks file's `builtin:` overrides when it parses, and otherwise with their defaults only in a tool that has none yet; `teamai init` says the team hooks were not installed), model profiles leave switched agents alone, skills and agents keep what is installed, and every other type still syncs. The warning names the file or both files and the fix, and for env, hooks, MCP and models is also written to `~/.teamai/debug.log` for session-start pulls (for [#707](https://github.com/Tencent/teamai-cli/issues/707)).
 - In single-repo (`kind: self`) mode, every checkout of the business repo now publishes learnings and reports. The `teamai-learnings` and `teamai-reports` checkouts and the queue of unpublished learnings lived in each checkout's own `.teamai/`, and git checks a branch out in one worktree only, so the first checkout to create them locked every other one out (`'teamai-learnings' is already used by worktree`), and a learning queued in a linked worktree was deleted with it by a plain `git worktree remove`. They now live in the project partition that every checkout shares, and the search index is kept per checkout, so `recall` no longer serves another checkout's index with paths into it. On upgrade, `init`, `pull`, `push`, `contribute` and `import --from-mr` migrate a checkout's old install, moving its queue into the partition without overwriting anything (or into `pending-learnings.self` when another checkout has since switched the project to another kind; an old git-mode install beside the knowledge of a switch to single-repo mode goes to `.teamai.bak`, its queue to `pending-learnings.git`); `contribute` and `import --from-mr` stop, saving nothing, while that data cannot move out of the checkout; and the first command that needs a side-branch checkout removes the old one from `.teamai/`; one with uncommitted changes is kept, a warning names it and says what to do, and `recall maintenance` and `recall promote` stop until then, while another command holds the learnings or reports lock, and when a checkout cannot be created, naming the cause. A git-mode install of the same project uses the same partition paths, so a checkout that belongs to the other repository is refused, never reused or removed, with the command that clears it, and so is one whose repository was moved or deleted; its votes are not counted, `init` deletes the search indexes built for the old repository, learnings the old install still had queued are set aside in `pending-learnings.<old kind>` rather than published by the new one (and in `pending-learnings.<kind>-<repo>` when `init` points an install at another team repository of the same kind, which on `main` published them to the new one; the same repository written another way keeps them, [#823](https://github.com/Tencent/teamai-cli/issues/823) item 13), and `uninstall` lists every unpublished queue before it asks. (for [#808](https://github.com/Tencent/teamai-cli/issues/808)).
 - A scope's usage file no longer grows without bound where it is never reported: an http source, a team with `usageReport: false`, or a remote that rejects every push. `teamai pull` keeps its newest 5,000 events, so `teamai stats` still shows recent usage there, the local file being its only source. In a reporting scope whose report does not complete while it holds more than 5,000 events, the dropped events were never reported. The cap runs after the report has removed the events it sent, so it cannot shift that removal onto events not yet reported, and a file at or below the cap is not rewritten. Hook appends, the report's truncate and the cap now share one lock beside the usage file, and a rewrite goes through a temp file that keeps the file's mode, so a rewrite no longer loses an event recorded while it runs, nor the file on a kill or a full disk. A hook that cannot take the lock within ~250 ms records its event in a `*.pending-<id>.jsonl` file beside it, which the next lock holder appends; a rewrite that cannot take it within ~5 s leaves the file as it is. A pending file gets no wider mode than the usage file (owner-only while there is none), and an in-workspace `.teamai/.gitignore` ignores the lock, a rewrite's temp copy and the pending files; `pull` and `push` add those entries to an existing single-repo one, and the first pending file or rewrite adds them to an existing project-scope one (for [#788](https://github.com/Tencent/teamai-cli/issues/788)).

docs/designs/data-directory-layout.md

@@ -800,6 +800,37 @@ what the machine had reported by then, never from another scope's later report.
 The seed holds a session's whole total, so a session still running at the
 upgrade goes on from the reported total, as before.
 
+Every writer that may modify `events.jsonl` — each hook's append, and a
+compaction that finds the log past its threshold or side files to fold —
+takes `events.jsonl.lock` beside it (#804), the pattern the usage
+file took for the same lost update (#803): compaction's
+read → filter → temp-file → rename cannot drop an append that lands while it
+runs, and two compactions cannot interleave their rewrites. A compaction
+whose log is below the threshold and holds no side files does the common
+case lock-free: one read, and it returns without touching the lock, so a
+state directory that is being torn down concurrently never meets a lock
+creation. A hook append waits
+up to ~250 ms, inside its foreground budget, and one that gives up records its
+line in an `events.pending-<uuid>.jsonl` side file, with the file's mode, that
+the next lock holder folds into the file before it writes — so an event is
+late, never gone. Side files fold in their events' own time order, not
+readdir's, and the compaction classifies sessions in time order too (the
+order every reader rebuilds from), so a side file that outlived newer appends
+cannot place an older event after them and re-mark a live session stopped.
+The line carries a `pendingId`, so a fold never appends a side file twice (a
+holder that died after appending it but before removing it leaves it for the
+next one), two side files of identical events are both kept, and the id stays
+in the raw file — a rewrite keeps it, as the usage file's does — until the
+side file itself is gone, while no reader ever sees it: `readEvents` drops it.
+A side file without its trailing
+newline is still being written and waits for the next holder. A compaction
+waits up to ~5 s for a peer's rewrite, and skips — leaving the file as it is
+for the next compaction — when a live holder outlasts the wait; a lock whose
+owner is gone is reclaimed, and a rewrite's temp copy left by a killed
+compaction (`events.jsonl.<pid>.<hex>.tmp`) is removed by the next one. The
+side files and the lock live in `~/.teamai/dashboard/` beside the log, which
+no repository tracks, so nothing needs adding to a workspace `.gitignore`.
+
 **`anchor` on save.** Previously only migration wrote a partition's `anchor`
 reverse-lookup file, so freshly-init'd partitions had none. `saveLocalConfigForScope`
 now writes it whenever the config lands in a partition (via the shared

src/__tests__/dashboard-collector.test.ts

@@ -1,4 +1,5 @@
 import { describe, it, expect, beforeEach, afterEach, vi } from 'vitest';
+import { spawnSync } from 'node:child_process';
 import fs from 'node:fs';
 import path from 'node:path';
 import os from 'node:os';
@@ -2146,6 +2147,225 @@ describe('compactEvents', () => {
   });
 });
 
+// ─── events file lock (#804) ────────────────────────────
+
+describe('events file lock (#804)', () => {
+  const eventsPath = () => path.join(tmpDir, '.teamai', 'dashboard', 'events.jsonl');
+  const event = (sessionId: string, extra: Partial<DashboardEvent> = {}): DashboardEvent =>
+    ({ type: 'session_start', timestamp: '2026-01-01T00:00:00Z', sessionId, tool: 'claude', ...extra });
+  const ensureEventsDir = async () => fs.promises.mkdir(path.dirname(eventsPath()), { recursive: true });
+  const ids = async () => (await readEvents(eventsPath())).map((e) => e.sessionId);
+  const writeLock = async (pid: number) => {
+    await ensureEventsDir();
+    await fs.promises.writeFile(
+      `${eventsPath()}.lock`,
+      JSON.stringify({ pid, startedAt: '2026-01-01T00:00:00Z', owner: 'other' }),
+      'utf-8',
+    );
+  };
+  const pendingFiles = async () =>
+    (await fs.promises.readdir(path.dirname(eventsPath()))).filter((n) => n.startsWith('events.pending-'));
+  // The seed every compaction test needs: past the threshold, and every
+  // session alive at its own monitor (process.pid), so compaction keeps it.
+  const seedActive = async () => {
+    await ensureEventsDir();
+    const line = (sessionId: string) =>
+      JSON.stringify({ type: 'session_start', timestamp: '2026-01-01T00:00:00Z', sessionId, tool: 'copilot', monitorPid: process.pid });
+    await fs.promises.writeFile(eventsPath(), Array.from({ length: 5_000 }, (_, i) => line(`filler-${i}`)).join('\n') + '\n', 'utf-8');
+  };
+
+  it('reclaims a lock whose owner is gone', async () => {
+    await writeLock(spawnSync(process.execPath, ['-e', '']).pid ?? 0);
+
+    await appendEvent(event('a'));
+
+    expect(await ids()).toEqual(['a']);
+    expect(fs.existsSync(`${eventsPath()}.lock`)).toBe(false);
+  });
+
+  it('records an event beside the file within the hook budget while the lock stays held, and folds it in later', async () => {
+    await appendEvent(event('a'));
+    await writeLock(process.pid);
+
+    const started = Date.now();
+    await appendEvent(event('b'));
+
+    expect(Date.now() - started).toBeLessThan(1000);
+    expect(await ids()).toEqual(['a']);
+    expect(await pendingFiles()).toHaveLength(1);
+
+    await fs.promises.rm(`${eventsPath()}.lock`);
+    await appendEvent(event('c'));
+
+    expect(await ids()).toEqual(['a', 'b', 'c']);
+    expect(await pendingFiles()).toEqual([]);
+  });
+
+  it('keeps an append that lands between compaction\'s read and its rename', async () => {
+    await seedActive();
+
+    // While compaction holds its snapshot, an append lands. The lock makes the
+    // append record a side file the next holder folds in, so the rewrite
+    // cannot drop it, as it did before the lock (#804). The append lands after
+    // compaction's second read under the lock — the snapshot the rewrite is
+    // built from — so the old read-all-then-overwrite lost it at the rename.
+    // Reads are counted only while the lock file exists: compaction's
+    // lock-free pre-check read runs before it, and the realpath'd target it
+    // reads under the lock differs from eventsPath() on macOS (/private
+    // prefix), so both are matched by file name.
+    const realReadFile = fs.promises.readFile;
+    let reads = 0;
+    const spy = vi.spyOn(fs.promises, 'readFile').mockImplementation(async (file, options) => {
+      const content = await realReadFile(file, options);
+      if (path.basename(String(file)) === 'events.jsonl' && fs.existsSync(`${eventsPath()}.lock`) && ++reads === 2) {
+        await appendEvent(event('live-b', { monitorPid: process.pid }));
+      }
+      return content;
+    });
+    try {
+      await compactEvents(eventsPath());
+    } finally {
+      spy.mockRestore();
+    }
+
+    // The next holder folds the side file in; the event was never lost.
+    await appendEvent(event('live-c', { monitorPid: process.pid }));
+
+    const sessionIds = await ids();
+    expect(sessionIds).toContain('live-b');
+    expect(sessionIds).toContain('live-c');
+    expect(await pendingFiles()).toEqual([]);
+  });
+
+  it('keeps a folded side file recognizable through a compaction, and its id out of readers', async () => {
+    await seedActive();
+    await writeLock(process.pid);
+    await appendEvent(event('live-b', { monitorPid: process.pid }));
+    expect(await pendingFiles()).toHaveLength(1);
+    await fs.promises.rm(`${eventsPath()}.lock`);
+
+    await compactEvents(eventsPath());
+
+    // The rewrite keeps the folded line's id, so a side file that survived its
+    // fold (its rm failed) is still recognized and never appended twice.
+    const content = fs.readFileSync(eventsPath(), 'utf-8');
+    expect(content).toContain('live-b');
+    expect(content).toContain('pendingId');
+    const events = await readEvents(eventsPath());
+    expect(events.some((e) => 'pendingId' in e)).toBe(false);
+    expect(await pendingFiles()).toEqual([]);
+  });
+
+  it('does not append a surviving side file twice across a compaction', async () => {
+    await seedActive();
+    await appendEvent(event('a', { monitorPid: process.pid }));
+    await writeLock(process.pid);
+    await appendEvent(event('b', { monitorPid: process.pid }));
+    await fs.promises.rm(`${eventsPath()}.lock`);
+    // The fold appends b but cannot remove its side file; a compaction runs
+    // while it survives.
+    const realRm = fs.promises.rm;
+    const spy = vi.spyOn(fs.promises, 'rm').mockImplementation(async (file, options) => {
+      if (String(file).includes('.pending-')) throw Object.assign(new Error('EPERM: operation not permitted'), { code: 'EPERM' });
+      return realRm(file, options);
+    });
+    try {
+      await appendEvent(event('c', { monitorPid: process.pid }));
+      await compactEvents(eventsPath());
+    } finally {
+      spy.mockRestore();
+    }
+    expect(await pendingFiles()).toHaveLength(1);
+
+    await appendEvent(event('d', { monitorPid: process.pid }));
+
+    expect(await ids()).toEqual([...Array.from({ length: 5_000 }, (_, i) => `filler-${i}`), 'a', 'b', 'c', 'd']);
+    expect(await pendingFiles()).toEqual([]);
+  });
+
+  it('classifies sessions in time order, so a late older event cannot stop a live one', async () => {
+    await seedActive();
+    // A session whose newer prompt sits before an older end in the raw file —
+    // the placement a late-folding side file can produce. Classified in raw
+    // file order the session reads as long stopped and the rewrite would drop
+    // its events; in time order it is still running.
+    const now = new Date().toISOString();
+    const old = new Date(Date.now() - 3_600_000).toISOString();
+    const line = (o: Record<string, unknown>) => JSON.stringify(o);
+    await fs.promises.appendFile(eventsPath(), [
+      line({ type: 'prompt_submit', timestamp: now, sessionId: 'x', tool: 'claude' }),
+      line({ type: 'session_end', timestamp: old, sessionId: 'x', tool: 'claude' }),
+    ].join('\n') + '\n', 'utf-8');
+
+    await compactEvents(eventsPath());
+
+    const kept = fs.readFileSync(eventsPath(), 'utf-8').split('\n').filter(Boolean)
+      .map((l) => JSON.parse(l) as { sessionId: string })
+      .filter((e) => e.sessionId === 'x');
+    expect(kept).toHaveLength(2);
+  });
+
+  it('folds a side file even when the file is below the compaction threshold', async () => {
+    await appendEvent(event('a'));
+    await writeLock(process.pid);
+    await appendEvent(event('b'));
+    await fs.promises.rm(`${eventsPath()}.lock`);
+
+    await compactEvents(eventsPath());
+
+    expect(await ids()).toEqual(['a', 'b']);
+    expect(await pendingFiles()).toEqual([]);
+    expect(fs.existsSync(`${eventsPath()}.lock`)).toBe(false);
+  });
+
+  it('folds a side file once when it outlives its append', async () => {
+    await appendEvent(event('a'));
+    await writeLock(process.pid);
+    await appendEvent(event('b'));
+    await fs.promises.rm(`${eventsPath()}.lock`);
+    // The side file cannot be removed once its event is in the file (or the holder dies there).
+    const realRm = fs.promises.rm;
+    const spy = vi.spyOn(fs.promises, 'rm').mockImplementation(async (file, options) => {
+      if (String(file).includes('.pending-')) throw Object.assign(new Error('EPERM: operation not permitted'), { code: 'EPERM' });
+      return realRm(file, options);
+    });
+    try {
+      await appendEvent(event('c'));
+    } finally {
+      spy.mockRestore();
+    }
+
+    await appendEvent(event('d'));
+
+    expect(await ids()).toEqual(['a', 'b', 'c', 'd']);
+    expect(await pendingFiles()).toEqual([]);
+  });
+
+  it('keeps two identical events recorded in side files at once', async () => {
+    await writeLock(process.pid);
+    await Promise.all([appendEvent(event('x')), appendEvent(event('x'))]);
+    expect(await pendingFiles()).toHaveLength(2);
+    await fs.promises.rm(`${eventsPath()}.lock`);
+
+    await appendEvent(event('y'));
+
+    expect(await ids()).toEqual(['x', 'x', 'y']);
+    expect(await pendingFiles()).toEqual([]);
+  });
+
+  it('keeps the side file id out of what readers see', async () => {
+    await writeLock(process.pid);
+    await appendEvent(event('a'));
+    await fs.promises.rm(`${eventsPath()}.lock`);
+    await appendEvent(event('b'));
+    expect(await fs.promises.readFile(eventsPath(), 'utf-8')).toContain('"pendingId"');
+
+    const events = await readEvents(eventsPath());
+    expect(events.map((e) => e.sessionId)).toEqual(['a', 'b']);
+    expect(events.some((e) => 'pendingId' in e)).toBe(false);
+  });
+});
+
 // ─── countInterventions (CodeBuddy index.json) ─────────
 
 describe('countInterventions (CodeBuddy index.json)', () => {