From 20bf78e75410873b6c14be09f2a744fcc7f1cc5b Mon Sep 17 00:00:00 2001 From: Saul Moro Date: Thu, 24 Sep 2026 10:12:05 +0200 Subject: [PATCH 01/41] fix(report): each scope reports only the dashboard sessions recorded in it (#785) Every scope read one machine-wide events.jsonl and picked its sessions out by cwd prefix. The user scope excluded nothing, so a user-scope pull reported every project's sessions (and, through the shared reported snapshots, took them from the project's own report); Copilot sends no cwd, so a project never reported its Copilot sessions; and a raw cwd under a symlink or /tmp never matched the realpath'd projectRoot. The hook now stamps each event's dataHome with the data home of the scope the dispatcher resolved (the key the per-scope usage file already uses), and a report keeps only its own scope's events, comparing realpath'd keys. A project also owns its in-repo .teamai key, where hooks record until migration moves it to a partition. Events written before this carry no dataHome: a project keeps those whose realpath'd cwd is under its root, the user scope never reports them. The log stays machine-wide for the dashboard UI, stats --by-repo, session save and the contribute check. Removes the excludeProjectRoots option, which pull only ever passed as [] (the user target exists only when no project config resolved), and the projectRoot option now carried by selfConfig. The usage guide documents how to remove by hand a skill an earlier release pushed into stats/.yaml from another project. --- CHANGELOG.md | 1 + docs/designs/data-directory-layout.md | 9 +- docs/usage-guide.md | 26 ++- docs/usage-guide.zh-CN.md | 21 ++- src/__tests__/dashboard-report-scope.test.ts | 163 ++++++++++++++++ src/__tests__/e2e/report-timeout.test.ts | 8 +- src/__tests__/git-kind-reports.test.ts | 12 +- src/__tests__/scope-filter.test.ts | 178 +++++++++--------- .../self-mode-no-business-reset.test.ts | 6 +- src/__tests__/team-push-interventions.test.ts | 4 +- src/dashboard-collector.ts | 7 +- src/dashboard.ts | 2 + src/hook-handlers.ts | 6 + src/pull.ts | 8 +- src/team-push.ts | 66 ++++--- src/types.ts | 10 +- 16 files changed, 394 insertions(+), 133 deletions(-) create mode 100644 src/__tests__/dashboard-report-scope.test.ts diff --git a/CHANGELOG.md b/CHANGELOG.md index 2fa0cbfc5..36e24837a 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -36,6 +36,7 @@ All notable changes to this project will be documented in this file. See [standa - A `manifest/roles.yaml` that exists but does not parse now fails the pull for that scope instead of warning and syncing with no role filter at all, for a member with no role as much as for one with a role. The same applies to `init` and `push`, which each fell back to a guess at the namespaces when any error came out of the loader. The legacy role migration skips with a warning instead of failing, so every command, `pull` included, still loads the config and can fetch the fixed manifest; until it can run, the member holds no role rather than every role, so hooks, MCP servers and env variables scoped by `roles:` reach them no more than skills do. For a member with no active project that fallback meant an unfiltered sync, so a broken manifest delivered every namespace it was written to gate. Only an absent manifest still means "this team does not use roles"; an unreadable or empty file is an error, as it now is for `manifest/projects.yaml` too. `push` stops at its scan for such a manifest (exit 2) even with `--role `, since the scan needs it to tell which namespaces are the member's. - `teamai members list` and `teamai projects members` read the roster registered before the reports switch, so a team upgrading past the orphan-branch split no longer sees "No team members registered" while its `members/` still lives on the default branch. The default-branch copy becomes a read-only inherited root, the way learnings' already was: listed in union with the `teamai-reports` copy, with the branch copy winning when the same file exists on both; nothing is copied or deleted, and a cold `members list` still does not publish the reports branch. Member registration merges against the inherited copy too, so a re-init keeps the original `registeredAt` and projects. Fixes [#735](https://github.com/Tencent/teamai-cli/issues/735). - Cache GC now rejects partial integers such as `12abc`, decimals, zero and unsafe integers for `--max-bytes` and `--stale-days` before deleting anything. An invalid `TEAMAI_CACHE_MAX_BYTES` value falls back to the default 5 GB limit instead of using a numeric prefix. +- Each scope reports only the dashboard sessions recorded in it. Every scope read one machine-wide event log and picked its sessions out by the `cwd` they started in, so a user-scope pull reported every project's sessions to the user team, a project never reported its Copilot sessions (Copilot sends no `cwd`), and a session started under a symlinked path (or `/tmp` for `/private/tmp` on macOS) never matched its project. Each event now records the data directory of the scope `hook-dispatch` resolved, and a report keeps only its own scope's. Events recorded by an earlier release go to the project whose root holds their `cwd`, never to the user scope. The usage guide explains how to remove by hand a skill an earlier release reported into `stats/.yaml` from another project (for [#785](https://github.com/Tencent/teamai-cli/issues/785)). - Usage reporting scopes sessions by path on Windows too. The project/user scope filter compared an event's `cwd` against `projectRoot` with a hard-coded `/` separator, so on Windows only a session started in the project root itself matched: every session started in a subdirectory was dropped from the project team's report and counted in the user scope's instead, which is the isolation the usage guide promises. Windows paths are also compared case-insensitively, so a drive letter or a directory name spelled with different case in the two sources no longer leaks a project session into the user scope. POSIX paths keep their own rules: case-sensitive, and a `\` in a filename stays part of the name. - `teamai tags subscribe` and `teamai tags unsubscribe` now invalidate the pull revision cache, as `teamai skill exclude` already does, so the next `teamai pull` applies the new subscriptions instead of reporting "Already synced" when the team repo has not changed. - `teamai pull` now deletes a tombstoned agent under all three render extensions, so the Codex `.toml` and Kiro `.json` copies of a removed agent no longer survive on other machines. The cleanup also runs when the team repo rev is unchanged, so an upgrade reaches machines that already pulled the tombstone with an older CLI. `teamai remove agents ` also honours `enabledAgents` and no longer deletes from excluded tools. Fixes [#576](https://github.com/Tencent/teamai-cli/issues/576). diff --git a/docs/designs/data-directory-layout.md b/docs/designs/data-directory-layout.md index 46df6a2f0..d4d503adf 100644 --- a/docs/designs/data-directory-layout.md +++ b/docs/designs/data-directory-layout.md @@ -312,9 +312,12 @@ already route through `getDataHome()`. Skill usage moved there too (#748): project's report carry every project's skills. The user scope records in `~/.teamai/user-usage.jsonl`, not that old shared `~/.teamai/usage.jsonl`, which an earlier release still writes after a rollback; the shared file is never -read. The dashboard is likewise an A2 singleton -(events carry `cwd`/`sessionId`); "two projects' events don't mix" is satisfied by -`getEventsPath()` reading `HOME` at call time, not by per-project dirs. +read. The dashboard stays an A2 singleton: `teamai dashboard`, `stats --by-repo`, +`session save` and the contribute check read across scopes. Each event instead +carries `dataHome`, the `getDataHome()` of the scope the hook resolved (#785), and a +scope's report keeps only its own. An event written before that field existed is +attributed by its `cwd`, realpath'd, to the project whose root holds it, never to +the user scope. **`anchor` on save.** Previously only migration wrote a partition's `anchor` reverse-lookup file, so freshly-init'd partitions had none. `saveLocalConfigForScope` diff --git a/docs/usage-guide.md b/docs/usage-guide.md index 61bdab923..b1c6b8ae8 100644 --- a/docs/usage-guide.md +++ b/docs/usage-guide.md @@ -370,6 +370,23 @@ branch, exactly where they are. Nothing is copied, deleted or migrated: that directory is still read, so every existing learning keeps coming back from `teamai recall`. New learnings go to `teamai-learnings`. +**Removing a skill another project reported into your `stats/`.** Before skill +usage was kept per scope, whichever project pulled next reported every +project's skills, so `stats/.yaml` on `teamai-reports` can count a skill +that belongs to an unrelated repo. Those events recorded no directory, so +teamai cannot attribute them and never rewrites the file. Remove the entry by +hand, from a clone of your own so teamai's `reports-wt/` checkout is untouched: + +```bash +git clone --branch teamai-reports --single-branch teamai-reports +cd teamai-reports +# delete the skill's entry under `skills:` in stats/.yaml +git commit -am "stats: remove reported from another project" +git push origin teamai-reports +``` + +The next report reads the branch first, so the entry does not come back. + **Minimum Git permissions with a protected default branch.** A member needs to: @@ -1726,7 +1743,14 @@ Pull waits up to 5 seconds for the reporting batch, then continues its other work while reporting finishes. A late successful push still updates the local reported snapshots. Skill usage is recorded per scope, in the data directory of the project teamai is set up for where the session ran (or the user scope), so -each target reports only its own; a directory without teamai records none. A +each target reports only its own; a directory without teamai records none. +Dashboard sessions stay in one machine-wide `~/.teamai/dashboard/events.jsonl`, +but each event records the data home of its scope, so a scope reports only the +sessions recorded in it: a user-scope pull no longer reports a project's +sessions, and a project reports its Copilot sessions and sessions started under +a symlinked path. Events recorded by an earlier release carry no data home: a +project reports those whose directory lies under its root, the user scope +reports none of them. A target removes its usage events only after it confirms success; failed pushes preserve them. The affected sync locks remain held until reporting finishes, preventing another pull from racing the report. diff --git a/docs/usage-guide.zh-CN.md b/docs/usage-guide.zh-CN.md index 43c2c99f0..0758aaf6f 100644 --- a/docs/usage-guide.zh-CN.md +++ b/docs/usage-guide.zh-CN.md @@ -346,6 +346,21 @@ learnings 迁到独立分支之前团队已经写下的内容，原地留在默 不迁移：该目录仍会被读取，所有既有 learning 依然能从 `teamai recall` 中找回。新的 learning 写入 `teamai-learnings`。 +**删除其他项目上报进你 `stats/` 的 skill。** 在 skill 使用按 scope 记录之前，下一个 +执行 pull 的项目会上报所有项目的 skill，因此 `teamai-reports` 上的 `stats/.yaml` 可能 +统计了属于无关仓库的 skill。这些事件没有记录目录，teamai 无法归属，也不会改写该 +文件。请手动删除该条目，在单独的 clone 中操作，不要动 teamai 的 `reports-wt/` 检出： + +```bash +git clone --branch teamai-reports --single-branch teamai-reports +cd teamai-reports +# 删除 stats/.yaml 中 `skills:` 下该 skill 的条目 +git commit -am "stats: remove reported from another project" +git push origin teamai-reports +``` + +下一次上报会先读取该分支，所以条目不会再出现。 + **默认分支受保护时所需的最小 Git 权限。** 成员需要： @@ -1669,7 +1684,11 @@ teamai remove rules --force # 跳过确认，用于脚本和 CI Pull 对整批统计上报最多等待 5 秒，之后继续其他工作，上报任务仍会完成。 超时后推送成功，仍会更新本地已上报快照。skill 使用按 scope 记录：写入会话所在 目录对应的已配置 teamai 项目的数据目录（或 user scope），因此每个目标只上报 -自己的使用；未配置 teamai 的目录不记录。目标确认成功后才清理自己的使用事件， +自己的使用；未配置 teamai 的目录不记录。Dashboard 会话仍写入整机共用的 +`~/.teamai/dashboard/events.jsonl`，但每条事件都记下所属 scope 的数据目录（data home），因此每个 scope +只上报在其中记录的会话：user scope 的 pull 不再上报项目的会话，项目会上报自己的 +Copilot 会话以及从软链接路径启动的会话。旧版本记录的事件没有数据目录：项目上报目录 +位于其根目录下的那些，user scope 一条也不上报。目标确认成功后才清理自己的使用事件， 推送失败会保留事件。上报完成前继续持有相关同步锁， 避免另一次 Pull 与尚未完成的上报竞争。 diff --git a/src/__tests__/dashboard-report-scope.test.ts b/src/__tests__/dashboard-report-scope.test.ts new file mode 100644 index 000000000..09d5b5b9c --- /dev/null +++ b/src/__tests__/dashboard-report-scope.test.ts @@ -0,0 +1,163 @@ +/** + * Which scope reports a dashboard session (#785): the real dispatcher records + * the sessions, the real report reads them, observed through the stats file + * each scope pushes. Only the reports-branch push and the handlers that reach + * the network or spawn processes are stubbed. + */ +import { afterEach, beforeEach, describe, expect, it, vi } from 'vitest'; +import { execFileSync } from 'node:child_process'; +import fs from 'node:fs'; +import os from 'node:os'; +import path from 'node:path'; +import YAML from 'yaml'; +import type { LocalConfig } from '../types.js'; + +vi.mock('node:child_process', async (importOriginal) => ({ + ...(await importOriginal()), + // The detached background pass: run inline instead (bgOnly) so its writes are observable. + spawn: vi.fn(() => ({ on: vi.fn(), stdin: { on: vi.fn(), end: vi.fn((_: string, done: () => void) => done()) }, unref: vi.fn() })), +})); +vi.mock('../pull.js', () => ({ pull: vi.fn(async () => undefined) })); +vi.mock('../update.js', () => ({ doUpdate: vi.fn(async () => undefined) })); +vi.mock('../local-agent.js', () => ({ reportAndSyncFromHook: vi.fn(async () => null) })); +// Each scope's reports branch is a plain directory next to its team repo. +vi.mock('../utils/reports-branch.js', () => ({ + updateReports: vi.fn(async (cfg: LocalConfig, write: (wt: string) => Promise) => { + const dir = path.join(path.dirname(cfg.repo.localPath), 'reports-wt'); + fs.mkdirSync(dir, { recursive: true }); + return (await write(dir)) != null; + }), +})); + +const { hookDispatchCli } = await import('../hook-dispatch-cli.js'); +const { resolveProjectDataHome, saveLocalConfigForScope, resolveConfigForDir, loadLocalConfig } = await import('../config.js'); +const { reportUsageToTeam } = await import('../team-push.js'); + +let tmp: string; +let originalHome: string | undefined; +let originalCwd: string; + +beforeEach(() => { + tmp = fs.realpathSync(fs.mkdtempSync(path.join(os.tmpdir(), 'teamai-dashboard-scope-'))); + originalHome = process.env.HOME; + originalCwd = process.cwd(); + process.env.HOME = path.join(tmp, 'home'); + fs.mkdirSync(process.env.HOME); +}); + +afterEach(() => { + process.chdir(originalCwd); + if (originalHome === undefined) delete process.env.HOME; + else process.env.HOME = originalHome; + fs.rmSync(tmp, { recursive: true, force: true }); +}); + +const teamaiHome = () => path.join(tmp, 'home', '.teamai'); + +/** Run one hook event the way a host does: foreground pass, then the background pass. */ +async function hook(event: string, tool: string, payload: Record): Promise { + for (const bgOnly of [false, true]) { + const stdinFile = path.join(tmp, `stdin-${Date.now()}-${Math.random()}.json`); + fs.writeFileSync(stdinFile, JSON.stringify(payload)); + await hookDispatchCli(event, tool, '*', { bgOnly, stdinFile }); + } +} + +/** One complete session: start, a prompt, stop. */ +async function session(tool: string, base: Record): Promise { + await hook('session-start', tool, { ...base, hook_event_name: 'SessionStart' }); + await hook('prompt-submit', tool, { ...base, hook_event_name: 'UserPromptSubmit', prompt: 'hello' }); + await hook('stop', tool, { ...base, hook_event_name: 'Stop' }); +} + +async function setup(): Promise<{ root: string; user: LocalConfig; project: LocalConfig }> { + const userRepo = path.join(teamaiHome(), 'team-repo'); + fs.mkdirSync(userRepo, { recursive: true }); + fs.writeFileSync(path.join(teamaiHome(), 'config.yaml'), + `repo:\n localPath: ${userRepo}\n remote: https://example.test/acme/user-team.git\n kind: git\nusername: tester\nscope: user\n`); + const root = path.join(tmp, 'project-p'); + fs.mkdirSync(root); + execFileSync('git', ['init', '-q'], { cwd: root }); + const dataHome = await resolveProjectDataHome(root); + fs.mkdirSync(path.join(dataHome, 'team-repo'), { recursive: true }); + await saveLocalConfigForScope({ + repo: { localPath: path.join(dataHome, 'team-repo'), remote: 'https://example.test/acme/team-p.git', kind: 'git' }, + username: 'tester', scope: 'project', projectRoot: root, additionalRoles: [], dataHome, + }); + // The configs as pull resolves them. + const user = await loadLocalConfig(); + const project = await resolveConfigForDir(root); + if (!user || !project) throw new Error('fixture configs did not resolve'); + return { root, user, project }; +} + +/** Report the way pull does, and return the sessions that scope's stats now hold. */ +async function reportedSessions(config: LocalConfig): Promise { + await reportUsageToTeam(config.repo.localPath, config.username, { skipTruncate: true, selfConfig: config }); + const statsPath = path.join(path.dirname(config.repo.localPath), 'reports-wt', 'stats', `${config.username}.yaml`); + if (!fs.existsSync(statsPath)) return 0; + const stats: unknown = YAML.parse(fs.readFileSync(statsPath, 'utf-8')); + const daily = stats && typeof stats === 'object' && 'daily' in stats && stats.daily && typeof stats.daily === 'object' ? stats.daily : {}; + return Object.values(daily).reduce((sum: number, day: unknown) => + sum + (day && typeof day === 'object' && 'sessionsEnded' in day && typeof day.sessionsEnded === 'number' ? day.sessionsEnded : 0), 0); +} + +describe('each scope reports only the dashboard sessions recorded in it (#785)', () => { + it('a user-scope report leaves out a session recorded in a project, and the project reports it', async () => { + const { root, user, project } = await setup(); + await session('claude', { session_id: 'sid-p', cwd: root }); + + expect(await reportedSessions(user)).toBe(0); + expect(await reportedSessions(project)).toBe(1); + }); + + it('a user-scope session is reported by the user scope only', async () => { + const { user, project } = await setup(); + const elsewhere = path.join(tmp, 'elsewhere'); + fs.mkdirSync(elsewhere); + await session('claude', { session_id: 'sid-u', cwd: elsewhere }); + + expect(await reportedSessions(project)).toBe(0); + expect(await reportedSessions(user)).toBe(1); + }); + + it('a Copilot session, which records no cwd, is reported by its project', async () => { + const { root, user, project } = await setup(); + await session('copilot', { session_id: 'copilot-p', cwd: root }); + + expect(await reportedSessions(user)).toBe(0); + expect(await reportedSessions(project)).toBe(1); + }); + + it('a session started under a symlinked path of the project is reported by the project', async () => { + const { root, user, project } = await setup(); + const link = path.join(tmp, 'link-p'); + fs.symlinkSync(root, link, 'dir'); + await session('claude', { session_id: 'sid-link', cwd: link }); + + expect(await reportedSessions(user)).toBe(0); + expect(await reportedSessions(project)).toBe(1); + }); + + it('events recorded before sessions carried a data home go to their project, never to the user scope', async () => { + const { root, user, project } = await setup(); + const link = path.join(tmp, 'link-p'); + fs.symlinkSync(root, link, 'dir'); + const timestamp = new Date().toISOString(); + const old = (sessionId: string, cwd: string | undefined) => [ + { type: 'session_start', timestamp, sessionId, tool: 'claude', cwd }, + { type: 'stop', timestamp, sessionId, tool: 'claude', cwd }, + ]; + const eventsPath = path.join(teamaiHome(), 'dashboard', 'events.jsonl'); + fs.mkdirSync(path.dirname(eventsPath), { recursive: true }); + fs.writeFileSync(eventsPath, [ + ...old('old-in-p', path.join(root, 'src')), + ...old('old-via-link', link), + ...old('old-elsewhere', path.join(tmp, 'elsewhere')), + ...old('old-no-cwd', undefined), + ].map((e) => JSON.stringify(e)).join('\n') + '\n'); + + expect(await reportedSessions(user)).toBe(0); + expect(await reportedSessions(project)).toBe(2); + }); +}); diff --git a/src/__tests__/e2e/report-timeout.test.ts b/src/__tests__/e2e/report-timeout.test.ts index 585372f04..f945243bc 100644 --- a/src/__tests__/e2e/report-timeout.test.ts +++ b/src/__tests__/e2e/report-timeout.test.ts @@ -41,14 +41,16 @@ function fixture(agent: keyof typeof agents, provider: string) { const dashboard = path.join(home, '.teamai/dashboard'); const timestamp = new Date().toISOString(); const usageLine = JSON.stringify({ skill: 'review', tool: agent, timestamp }) + '\n'; + // A session the user scope recorded (#785). + const dataHome = path.join(home, '.teamai'); function seedEvents() { fs.mkdirSync(dashboard, { recursive: true }); fs.writeFileSync(usage, usageLine); fs.writeFileSync(path.join(dashboard, 'events.jsonl'), [ - { type: 'session_start', timestamp, sessionId: 's1', tool: agent, cwd: sandbox }, - { type: 'prompt_submit', timestamp, sessionId: 's1', tool: agent, promptSummary: 'review' }, + { type: 'session_start', timestamp, sessionId: 's1', tool: agent, cwd: sandbox, dataHome }, + { type: 'prompt_submit', timestamp, sessionId: 's1', tool: agent, promptSummary: 'review', dataHome }, { type: 'stop', timestamp, sessionId: 's1', tool: agent, interventions: { interrupt: 1, toolReject: 0 }, - tokens: { input: 10, output: 5, cacheRead: 0, cacheCreation: 0 } }, + tokens: { input: 10, output: 5, cacheRead: 0, cacheCreation: 0 }, dataHome }, ].map((e) => JSON.stringify(e)).join('\n') + '\n'); } function receiver(mode: 'slow' | 'reject' | 'normal') { diff --git a/src/__tests__/git-kind-reports.test.ts b/src/__tests__/git-kind-reports.test.ts index 4d7ff3cfe..5e29a8ad2 100644 --- a/src/__tests__/git-kind-reports.test.ts +++ b/src/__tests__/git-kind-reports.test.ts @@ -8,7 +8,7 @@ import os from 'node:os'; import path from 'node:path'; import { simpleGit } from 'simple-git'; -import { getReportsDir, REPORTS_WORKTREE_DIRNAME, type LocalConfig } from '../types.js'; +import { getDataHome, getReportsDir, REPORTS_WORKTREE_DIRNAME, type LocalConfig } from '../types.js'; import { commitAndPushReports, ensureReportsWorktree, refreshReportsWorktree, updateReports } from '../utils/reports-branch.js'; import { pushRepoDirectly } from '../utils/git.js'; import { reportUsageToTeam } from '../team-push.js'; @@ -105,11 +105,13 @@ describe('git-kind reports branch', () => { const ts = new Date().toISOString(); const eventsDir = path.join(process.env.HOME!, '.teamai', 'dashboard'); + // A session the user scope recorded. + const dataHome = getDataHome(cfg); fs.mkdirSync(eventsDir, { recursive: true }); fs.writeFileSync( path.join(eventsDir, 'events.jsonl'), - `${JSON.stringify({ type: 'session_start', timestamp: ts, sessionId: 's1', tool: 'claude', cwd: '/p' })}\n` + - `${JSON.stringify({ type: 'stop', timestamp: ts, sessionId: 's1', tool: 'claude', interventions: { interrupt: 1, toolReject: 0 } })}\n`, + `${JSON.stringify({ type: 'session_start', timestamp: ts, sessionId: 's1', tool: 'claude', cwd: '/p', dataHome })}\n` + + `${JSON.stringify({ type: 'stop', timestamp: ts, sessionId: 's1', tool: 'claude', dataHome, interventions: { interrupt: 1, toolReject: 0 } })}\n`, ); await reportUsageToTeam(clone, 'alice', { skipTruncate: true, selfConfig: cfg }); @@ -672,10 +674,10 @@ describe('skill usage stays in the scope that recorded it (#748)', () => { await useSkill(a.root, 'skill-a'); await useSkill(c.root, 'skill-c'); - await reportUsageToTeam(a.config.repo.localPath, 'alice', { projectRoot: a.root, selfConfig: a.config }); + await reportUsageToTeam(a.config.repo.localPath, 'alice', { selfConfig: a.config }); expect(await reportedSkills(a.origin)).toEqual(['skill-a']); - await reportUsageToTeam(c.config.repo.localPath, 'alice', { projectRoot: c.root, selfConfig: c.config }); + await reportUsageToTeam(c.config.repo.localPath, 'alice', { selfConfig: c.config }); expect(await reportedSkills(c.origin)).toEqual(['skill-c']); }); }); diff --git a/src/__tests__/scope-filter.test.ts b/src/__tests__/scope-filter.test.ts index 17794855b..0a039fb58 100644 --- a/src/__tests__/scope-filter.test.ts +++ b/src/__tests__/scope-filter.test.ts @@ -1,9 +1,27 @@ import { describe, it, expect } from 'vitest'; +import fs from 'node:fs'; +import os from 'node:os'; +import path from 'node:path'; import { filterEventsByScope } from '../team-push.js'; -import type { DashboardEvent } from '../types.js'; +import type { DashboardEvent, LocalConfig } from '../types.js'; -function makeEvent(cwd: string | undefined, sessionId = 's1'): DashboardEvent { - return { type: 'prompt_submit', timestamp: new Date().toISOString(), sessionId, tool: 'claude', cwd }; +/** An event; without `dataHome` it is one written before events carried one, attributed by cwd. */ +function makeEvent(cwd: string | undefined, sessionId = 's1', dataHome?: string): DashboardEvent { + return { type: 'prompt_submit', timestamp: new Date().toISOString(), sessionId, tool: 'claude', cwd, dataHome }; +} + +const repo = { localPath: '/unused/team-repo', remote: 'https://example.test/team.git' }; + +function projectScope(projectRoot: string, dataHome = '/home/jeff/.teamai/projects/p'): LocalConfig { + return { repo, username: 'jeff', scope: 'project', additionalRoles: [], projectRoot, dataHome }; +} + +function userScope(dataHome = '/home/jeff/.teamai'): LocalConfig { + return { repo, username: 'jeff', scope: 'user', additionalRoles: [], dataHome }; +} + +async function ids(events: DashboardEvent[], config?: LocalConfig): Promise { + return (await filterEventsByScope(events, config)).map((e) => e.sessionId); } describe('filterEventsByScope', () => { @@ -15,50 +33,80 @@ describe('filterEventsByScope', () => { makeEvent(undefined, 's5'), ]; - it('returns all events when no filter is provided', () => { - expect(filterEventsByScope(events)).toEqual(events); - expect(filterEventsByScope(events, {})).toEqual(events); + it('returns all events when no scope is given', async () => { + expect(await filterEventsByScope(events)).toEqual(events); }); - it('filters to projectRoot (exact match and subdirectories)', () => { - const result = filterEventsByScope(events, { projectRoot: '/Users/jeff/project-a' }); - expect(result.map((e) => e.sessionId)).toEqual(['s1', 's2']); - }); + describe('events that carry a data home', () => { + const scoped: DashboardEvent[] = [ + makeEvent('/Users/jeff/project-a', 'p1', '/home/jeff/.teamai/projects/p'), + makeEvent(undefined, 'p2', '/home/jeff/.teamai/projects/p'), + makeEvent('/Users/jeff/project-a', 'u1', '/home/jeff/.teamai'), + makeEvent('/Users/jeff/project-a', 'q1', '/home/jeff/.teamai/projects/q'), + ]; + + it('a project keeps its own, whatever their cwd', async () => { + expect(await ids(scoped, projectScope('/Users/jeff/project-a'))).toEqual(['p1', 'p2']); + }); + + it('the user scope keeps its own and no project\'s', async () => { + expect(await ids(scoped, userScope())).toEqual(['u1']); + }); + + it('a project keeps what it recorded under its in-repo .teamai before moving to a partition', async () => { + const legacy = [makeEvent(undefined, 'l1', '/Users/jeff/project-a/.teamai')]; + expect(await ids(legacy, projectScope('/Users/jeff/project-a'))).toEqual(['l1']); + expect(await ids(legacy, userScope())).toEqual([]); + }); + + it('a project rooted at HOME does not take the user scope\'s events', async () => { + const home = os.homedir(); + const evts = [makeEvent(undefined, 'u1', path.join(home, '.teamai'))]; + expect(await ids(evts, projectScope(home))).toEqual([]); + }); - it('projectRoot with trailing slash works the same', () => { - const result = filterEventsByScope(events, { projectRoot: '/Users/jeff/project-a/' }); - expect(result.map((e) => e.sessionId)).toEqual(['s1', 's2']); + it('matches a data home reached through a symlink', async () => { + const tmp = fs.realpathSync(fs.mkdtempSync(path.join(os.tmpdir(), 'teamai-scope-key-'))); + try { + fs.mkdirSync(path.join(tmp, 'real', '.teamai'), { recursive: true }); + fs.symlinkSync(path.join(tmp, 'real'), path.join(tmp, 'link'), 'dir'); + const evts = [makeEvent(undefined, 'k1', path.join(tmp, 'link', '.teamai'))]; + const config = projectScope(path.join(tmp, 'real'), path.join(tmp, 'real', '.teamai')); + expect(await ids(evts, config)).toEqual(['k1']); + } finally { + fs.rmSync(tmp, { recursive: true, force: true }); + } + }); + + it('a Windows data home matches whatever its case or separators', async () => { + const win = [makeEvent(undefined, 'w1', 'C:\\Users\\Jeff\\.teamai'), makeEvent(undefined, 'w2', 'C:\\Users\\jeff\\.teamai\\projects\\p')]; + expect(await ids(win, userScope('c:/users/jeff/.teamai'))).toEqual(['w1']); + }); }); - it('excludeProjectRoots removes matching events and keeps the rest', () => { - const result = filterEventsByScope(events, { excludeProjectRoots: ['/Users/jeff/project-a'] }); - expect(result.map((e) => e.sessionId)).toEqual(['s3', 's4', 's5']); + it('the user scope never keeps events without a data home', async () => { + expect(await ids(events, userScope())).toEqual([]); }); - it('excludeProjectRoots with multiple roots', () => { - const result = filterEventsByScope(events, { - excludeProjectRoots: ['/Users/jeff/project-a', '/Users/jeff/project-b'], - }); - expect(result.map((e) => e.sessionId)).toEqual(['s3', 's5']); + // Events without a data home go to the project whose root holds their cwd. + it('filters to projectRoot (exact match and subdirectories)', async () => { + expect(await ids(events, projectScope('/Users/jeff/project-a'))).toEqual(['s1', 's2']); }); - it('events with undefined cwd are kept by excludeProjectRoots', () => { - const result = filterEventsByScope(events, { excludeProjectRoots: ['/Users/jeff/project-a'] }); - expect(result.find((e) => e.sessionId === 's5')).toBeDefined(); + it('projectRoot with trailing slash works the same', async () => { + expect(await ids(events, projectScope('/Users/jeff/project-a/'))).toEqual(['s1', 's2']); }); - it('events with undefined cwd are excluded by projectRoot', () => { - const result = filterEventsByScope(events, { projectRoot: '/Users/jeff/project-a' }); - expect(result.find((e) => e.sessionId === 's5')).toBeUndefined(); + it('events with undefined cwd are excluded by projectRoot', async () => { + expect(await ids(events, projectScope('/Users/jeff/project-a'))).not.toContain('s5'); }); - it('does not match partial directory name prefixes', () => { + it('does not match partial directory name prefixes', async () => { const evts = [ makeEvent('/Users/jeff/project-ab', 'x1'), makeEvent('/Users/jeff/project-a', 'x2'), ]; - const result = filterEventsByScope(evts, { projectRoot: '/Users/jeff/project-a' }); - expect(result.map((e) => e.sessionId)).toEqual(['x2']); + expect(await ids(evts, projectScope('/Users/jeff/project-a'))).toEqual(['x2']); }); // Windows paths are plain strings here, so these run on the ubuntu CI too. @@ -73,83 +121,43 @@ describe('filterEventsByScope', () => { makeEvent('C:\\Users\\jeff\\other-work', 'w4'), ]; - it('filters to projectRoot including subdirectories', () => { - const result = filterEventsByScope(winEvents, { - projectRoot: 'C:\\Users\\jeff\\project-a', - }); - expect(result.map((e) => e.sessionId)).toEqual(['w1', 'w2']); + it('filters to projectRoot including subdirectories', async () => { + expect(await ids(winEvents, projectScope('C:\\Users\\jeff\\project-a'))).toEqual(['w1', 'w2']); }); - it('excludeProjectRoots removes subdirectory sessions too', () => { - const result = filterEventsByScope(winEvents, { - excludeProjectRoots: ['C:\\Users\\jeff\\project-a'], - }); - expect(result.map((e) => e.sessionId)).toEqual(['w3', 'w4']); + it('matches a root and a cwd that disagree on separator style', async () => { + expect(await ids(winEvents, projectScope('C:/Users/jeff/project-a'))).toEqual(['w1', 'w2']); }); - it('matches a root and a cwd that disagree on separator style', () => { - const result = filterEventsByScope(winEvents, { - projectRoot: 'C:/Users/jeff/project-a', - }); - expect(result.map((e) => e.sessionId)).toEqual(['w1', 'w2']); + it('trailing backslash on the root works the same', async () => { + expect(await ids(winEvents, projectScope('C:\\Users\\jeff\\project-a\\'))).toEqual(['w1', 'w2']); }); - it('trailing backslash on the root works the same', () => { - const result = filterEventsByScope(winEvents, { - projectRoot: 'C:\\Users\\jeff\\project-a\\', - }); - expect(result.map((e) => e.sessionId)).toEqual(['w1', 'w2']); + it('ignores drive-letter and directory casing', async () => { + expect(await ids(winEvents, projectScope('c:\\users\\JEFF\\Project-A'))).toEqual(['w1', 'w2']); }); - it('ignores drive-letter and directory casing', () => { - const result = filterEventsByScope(winEvents, { - projectRoot: 'c:\\users\\JEFF\\Project-A', - }); - expect(result.map((e) => e.sessionId)).toEqual(['w1', 'w2']); - }); - - it('excludeProjectRoots ignores casing too', () => { - const result = filterEventsByScope(winEvents, { - excludeProjectRoots: ['c:/users/jeff/project-a'], - }); - expect(result.map((e) => e.sessionId)).toEqual(['w3', 'w4']); - }); - - it('matches a UNC root whatever its case or separators', () => { + it('matches a UNC root whatever its case or separators', async () => { const uncEvents: DashboardEvent[] = [ makeEvent('\\\\Server\\Share\\Proj', 'u1'), makeEvent('\\\\server\\share\\proj\\src', 'u2'), makeEvent('\\\\server\\share\\other', 'u3'), ]; - const result = filterEventsByScope(uncEvents, { - projectRoot: '\\\\SERVER\\SHARE\\proj', - }); - expect(result.map((e) => e.sessionId)).toEqual(['u1', 'u2']); + expect(await ids(uncEvents, projectScope('\\\\SERVER\\SHARE\\proj'))).toEqual(['u1', 'u2']); }); }); // A POSIX path is case-sensitive, and `\` is a legal character in a POSIX // filename, so neither folding may be applied to one. describe('POSIX paths keep their own rules', () => { - it('does not fold case', () => { - const result = filterEventsByScope(events, { projectRoot: '/users/jeff/PROJECT-A' }); - expect(result.map((e) => e.sessionId)).toEqual([]); - }); - - it('treats a backslash in a filename as part of the name', () => { - const evts = [makeEvent('/work/a\\b', 'p1'), makeEvent('/work/a/b', 'p2')]; - expect( - filterEventsByScope(evts, { projectRoot: '/work/a/b' }).map((e) => e.sessionId), - ).toEqual(['p2']); - expect( - filterEventsByScope(evts, { projectRoot: '/work/a\\b' }).map((e) => e.sessionId), - ).toEqual(['p1']); + it('does not fold case', async () => { + expect(await ids(events, projectScope('/users/jeff/PROJECT-A'))).toEqual([]); }); - it('does not let a backslash filename escape an excluded root', () => { + it('treats a backslash in a filename as part of the name', async () => { const evts = [makeEvent('/work/a\\b', 'p1'), makeEvent('/work/a/b', 'p2')]; - const result = filterEventsByScope(evts, { excludeProjectRoots: ['/work/a/b'] }); - expect(result.map((e) => e.sessionId)).toEqual(['p1']); + expect(await ids(evts, projectScope('/work/a/b'))).toEqual(['p2']); + expect(await ids(evts, projectScope('/work/a\\b'))).toEqual(['p1']); }); }); }); diff --git a/src/__tests__/self-mode-no-business-reset.test.ts b/src/__tests__/self-mode-no-business-reset.test.ts index cb0924596..d080649ee 100644 --- a/src/__tests__/self-mode-no-business-reset.test.ts +++ b/src/__tests__/self-mode-no-business-reset.test.ts @@ -95,11 +95,13 @@ describe('E2E self-mode: business repo working tree is never reset', () => { const ts = new Date().toISOString(); const eventsDir = path.join(process.env.HOME!, '.teamai', 'dashboard'); + // A session the user scope recorded (#785). + const dataHome = path.join(tmp, 'home', '.teamai'); fs.mkdirSync(eventsDir, { recursive: true }); fs.writeFileSync( path.join(eventsDir, 'events.jsonl'), - `${JSON.stringify({ type: 'session_start', timestamp: ts, sessionId: 's1', tool: 'claude', cwd: '/p' })}\n` + - `${JSON.stringify({ type: 'stop', timestamp: ts, sessionId: 's1', tool: 'claude', interventions: { interrupt: 1, toolReject: 0 } })}\n`, + `${JSON.stringify({ type: 'session_start', timestamp: ts, sessionId: 's1', tool: 'claude', cwd: '/p', dataHome })}\n` + + `${JSON.stringify({ type: 'stop', timestamp: ts, sessionId: 's1', tool: 'claude', dataHome, interventions: { interrupt: 1, toolReject: 0 } })}\n`, ); const cfg: LocalConfig = { diff --git a/src/__tests__/team-push-interventions.test.ts b/src/__tests__/team-push-interventions.test.ts index 64890fa01..6542148d5 100644 --- a/src/__tests__/team-push-interventions.test.ts +++ b/src/__tests__/team-push-interventions.test.ts @@ -75,10 +75,12 @@ afterEach(() => { fs.rmSync(tmpDir, { recursive: true, force: true }); }); +/** Sessions the user scope recorded (#785), the scope `gitConfig()` reports. */ function writeDashboardEvents(lines: object[]): void { const p = path.join(tmpDir, '.teamai', 'dashboard', 'events.jsonl'); fs.mkdirSync(path.dirname(p), { recursive: true }); - fs.writeFileSync(p, lines.map((l) => JSON.stringify(l)).join('\n') + '\n'); + const dataHome = path.join(tmpDir, '.teamai'); + fs.writeFileSync(p, lines.map((l) => JSON.stringify({ ...l, dataHome })).join('\n') + '\n'); } describe('reportUsageToTeam — intervention reporting', () => { diff --git a/src/dashboard-collector.ts b/src/dashboard-collector.ts index ff2363e99..a80e55500 100644 --- a/src/dashboard-collector.ts +++ b/src/dashboard-collector.ts @@ -23,6 +23,7 @@ import { TRANSCRIPT_REJECT_MARKERS, COPILOT_TOOL_ID, getCopilotHome, + getDataHome, emptyTokenUsage, addTokenUsage, type DashboardEvent, @@ -46,7 +47,7 @@ import { estimateClaudeRequest } from './model-pricing.js'; // │ extract: session_id / cwd / tool_name / prompt // ▼ // DashboardEvent -// │ +// │ dataHome = data home of the scope the hook resolved // ▼ // appendEvent(event) → events.jsonl // @@ -1957,7 +1958,8 @@ export async function dashboardReport(toolArg?: string): Promise { // parseHookEvent reports the malformed payload below; nothing to gate on yet. } const { resolveConfigForDir } = await import('./config.js'); - if (!(await resolveConfigForDir(resolveHookCwd(hookData)))) { + const config = await resolveConfigForDir(resolveHookCwd(hookData)); + if (!config) { log.debug('dashboard-report: teamai is not set up here, skipping'); return; } @@ -1965,6 +1967,7 @@ export async function dashboardReport(toolArg?: string): Promise { const event = await parseHookEvent(raw, toolArg ?? 'claude'); if (!event) return; + event.dataHome = getDataHome(config); await appendEvent(event); // Trigger compaction check (non-blocking) diff --git a/src/dashboard.ts b/src/dashboard.ts index 598e39f37..83c338402 100644 --- a/src/dashboard.ts +++ b/src/dashboard.ts @@ -123,6 +123,8 @@ export async function startDashboard(port?: number): Promise { sessionId: session.sessionId, tool: session.tool, cwd: session.cwd, + // The session's own data home, so its scope's report still sees it end. + dataHome: events.find((e) => e.sessionId === session.sessionId && e.dataHome)?.dataHome, }; await appendEvent(exitEvent); log.info( diff --git a/src/hook-handlers.ts b/src/hook-handlers.ts index 30fe12116..9b237bb3a 100644 --- a/src/hook-handlers.ts +++ b/src/hook-handlers.ts @@ -173,13 +173,19 @@ async function userModelAliases(stdin: Record): Promise {}); diff --git a/src/pull.ts b/src/pull.ts index 5de3020c0..4e830d405 100644 --- a/src/pull.ts +++ b/src/pull.ts @@ -2014,14 +2014,14 @@ export async function pull( // scope's own file (`/usage.jsonl`, the user scope's // `~/.teamai/user-usage.jsonl`), so each target reports and then // truncates only its own file. Dashboard sessions live in one shared file - // and are filtered instead: project scope only gets sessions whose cwd is - // under projectRoot; user scope excludes those sessions. + // and are filtered instead: each target gets the sessions its scope + // recorded (#785). if (!options.dryRun && !pendingUsageReport) { pendingUsageReport = (async () => { try { const { reportUsageToTeam } = await import('./team-push.js'); const { truncateUsageAfterReport, readUsageEvents } = await import('./usage-tracker.js'); - const targets: Array<{ repoPath: string; username: string; opts: { skipTruncate: true; projectRoot?: string; excludeProjectRoots?: string[]; selfConfig: LocalConfig } }> = []; + const targets: Array<{ repoPath: string; username: string; opts: { skipTruncate: true; selfConfig: LocalConfig } }> = []; // Per-target opt-out (teamai.yaml `usageReport: false`): a repo that // disables stat commits is dropped from the targets — e.g. teams // pulling from a read-only remote never accumulate unpushable commits. @@ -2032,7 +2032,6 @@ export async function pull( username: reconcileProject.username, opts: { skipTruncate: true, - projectRoot: reconcileProject.projectRoot, // Non-HTTP repos route stats/votes to the teamai-reports orphan branch. selfConfig: reconcileProject, }, @@ -2045,7 +2044,6 @@ export async function pull( username: reconcileUser.username, opts: { skipTruncate: true, - excludeProjectRoots: projectConfig?.projectRoot ? [projectConfig.projectRoot] : [], // Non-HTTP repos route stats/votes to the teamai-reports orphan branch — // never reset/pull the default branch (or, in self mode, the business tree). selfConfig: reconcileUser, diff --git a/src/team-push.ts b/src/team-push.ts index fbb91efe8..6ed47b1a5 100644 --- a/src/team-push.ts +++ b/src/team-push.ts @@ -1,4 +1,5 @@ import YAML from 'yaml'; +import fs from 'node:fs'; import path from 'node:path'; import { readUsageEvents, truncateUsageAfterReport } from './usage-tracker.js'; import { aggregateUsage } from './stats.js'; @@ -14,7 +15,7 @@ import { import { writeFile, readFileSafe, ensureDir, pathExists, readJson, writeJson } from './utils/fs.js'; import { log } from './utils/logger.js'; import type { UserStats, UserInterventionStats, SessionMetrics, TokenUsage, DashboardEvent, LocalConfig } from './types.js'; -import { getUserVotesDir, emptyTokenUsage, addTokenUsage, usesBranchWorktree } from './types.js'; +import { getUserVotesDir, getDataHome, emptyTokenUsage, addTokenUsage, usesBranchWorktree } from './types.js'; import { getUserHome } from './utils/home.js'; import { aggregateDailySessions, @@ -352,28 +353,46 @@ function isUnderScopeRoot(cwd: string, root: ScopeRoot): boolean { } /** - * Filter dashboard events by scope: - * - projectRoot set: keep only events whose cwd is under that root. - * - excludeProjectRoots set: exclude events whose cwd is under any listed root. - * - Neither: return all events (backward-compatible). + * The dashboard events a scope reports (#785): those recorded in it, keyed by + * its data home. A project also owns the key of its in-repo `.teamai`, where a + * hook recorded until migration moved the project to a partition. An event + * written before events carried a data home belongs to the project whose root holds + * its cwd, never to the user scope. `projectRoot` is realpath'd, but a cwd is + * raw as the host sent it (a symlinked checkout, macOS `/tmp` vs + * `/private/tmp`) and so is a non-git project's data home, so both are + * realpath'd while they still exist. A caller without a scope config reads the + * whole log. */ -export function filterEventsByScope( +export async function filterEventsByScope( events: DashboardEvent[], - opts?: { projectRoot?: string; excludeProjectRoots?: string[] }, -): DashboardEvent[] { - if (!opts) return events; - if (opts.projectRoot) { - const root = scopeRoot(opts.projectRoot); - return events.filter((e) => !!e.cwd && isUnderScopeRoot(e.cwd, root)); - } - if (opts.excludeProjectRoots && opts.excludeProjectRoots.length > 0) { - const roots = opts.excludeProjectRoots.map(scopeRoot); - return events.filter((e) => { - if (!e.cwd) return true; - return !roots.some((root) => isUnderScopeRoot(e.cwd!, root)); - }); + config?: LocalConfig, +): Promise { + if (!config) return events; + const realPaths = new Map>(); + const realPath = (dir: string): Promise => { + let real = realPaths.get(dir); + if (!real) { + real = fs.promises.realpath(dir).catch(() => dir); + realPaths.set(dir, real); + } + return real; + }; + const keyOf = async (home: string): Promise => scopeRoot(await realPath(home)); + const homeRoots = [await keyOf(getDataHome(config))]; + if (config.projectRoot) { + // Unless the project is rooted at HOME, where that is the user scope's. + const legacy = await keyOf(path.join(config.projectRoot, '.teamai')); + if (legacy.key !== (await keyOf(path.join(getUserHome(), '.teamai'))).key) homeRoots.push(legacy); } - return events; + const root = config.projectRoot ? scopeRoot(config.projectRoot) : undefined; + const kept = await Promise.all(events.map(async (e) => { + if (e.dataHome !== undefined) { + const dataHome = await realPath(e.dataHome); + return homeRoots.some((home) => scopeKey(dataHome, home.windows) === home.key); + } + return !!root && !!e.cwd && isUnderScopeRoot(await realPath(e.cwd), root); + })); + return events.filter((_, i) => kept[i]); } /** @@ -388,7 +407,7 @@ export function filterEventsByScope( export async function reportUsageToTeam( repoPath: string, username: string, - options?: { skipTruncate?: boolean; projectRoot?: string; excludeProjectRoots?: string[]; selfConfig?: LocalConfig }, + options?: { skipTruncate?: boolean; selfConfig?: LocalConfig }, ): Promise { // Non-HTTP repos: stats + votes are report data → the teamai-reports orphan // branch (isolated worktree). We must NOT resetToCleanMaster / pullRepo / @@ -409,9 +428,8 @@ export async function reportUsageToTeam( // Fold the local dashboard event log into per-session metrics once, then derive // both the intervention delta and the prompt-count/token delta from it. - // Filter by scope so project repos only receive project sessions and vice versa. - const allDashboardEvents = await readEvents(); - const dashboardEvents = filterEventsByScope(allDashboardEvents, options); + // Only the sessions recorded in this scope (#785). + const dashboardEvents = await filterEventsByScope(await readEvents(), reportsConfig); const metrics = aggregateSessionMetrics(dashboardEvents); const currentInterventions = new Map( diff --git a/src/types.ts b/src/types.ts index 1252417f3..4c741de92 100644 --- a/src/types.ts +++ b/src/types.ts @@ -1219,6 +1219,12 @@ export interface DashboardEvent { tool: string; /** Working directory of the session */ cwd?: string; + /** + * Data home (`getDataHome`) of the scope that recorded the event, the key its + * report filters on (#785). Absent on events written before this field + * existed; the report then attributes them by `cwd`. + */ + dataHome?: string; /** First user prompt (captured from UserPromptSubmit) */ promptSummary?: string; /** @@ -1333,7 +1339,9 @@ export interface DashboardSession { // code consumed — dashboard read/write go through runtime helpers that inline // getUserHome() (dashboard-collector.ts getEventsPath, dashboard.ts), so HOME // isolation already works there. Removed (issue #374 P3). The dashboard is an -// A2 machine-level singleton keyed by event cwd/sessionId, not per-project. +// A2 machine-level singleton keyed by sessionId, not per-project; each event +// carries the data home of the scope that recorded it, which is what a scope's +// report filters on. export const DASHBOARD_DEFAULT_PORT = 3721; /** Sessions with no activity for this long (ms) are marked idle */ export const DASHBOARD_IDLE_TIMEOUT_MS = 5 * 60 * 1000; From 1d6ff131a6c0f29a459cf6a38b86eb101625deed Mon Sep 17 00:00:00 2001 From: Saul Moro Date: Thu, 24 Sep 2026 11:52:55 +0200 Subject: [PATCH 02/41] fix(report): address pre-review findings (#785) - Events record `dataHomeKey`, a hash of the realpath'd data home, instead of the path. A Copilot event persisted a workspace path through its data home (the raw root for a non-git project, the path-derived partition name otherwise), breaking the path-free Copilot contract from #666. - A data home that no longer exists (an in-repo .teamai removed after migration) keys through its parent's realpath, so it still matches the key recorded while it existed. - A non-git project's root is realpath'd before older events' cwd is matched against it, as the cwd already was. - A key that is not a string (a hand-edited log) counts as absent instead of throwing and skipping the whole report. - The legacy `dashboard-report` command's stamping is asserted. - CHANGELOG and the comment say teamai does not record Copilot's cwd, not that Copilot sends none. --- CHANGELOG.md | 2 +- docs/designs/data-directory-layout.md | 3 +- docs/usage-guide.md | 8 +- docs/usage-guide.zh-CN.md | 4 +- src/__tests__/dashboard-report-gate.test.ts | 4 +- src/__tests__/dashboard-report-scope.test.ts | 12 +++ src/__tests__/e2e/report-timeout.test.ts | 15 ++-- src/__tests__/git-kind-reports.test.ts | 7 +- src/__tests__/hook-handlers.test.ts | 1 + src/__tests__/scope-filter.test.ts | 74 +++++++++++++++---- .../self-mode-no-business-reset.test.ts | 7 +- src/__tests__/team-push-interventions.test.ts | 9 ++- src/dashboard-collector.ts | 20 ++++- src/dashboard.ts | 2 +- src/hook-handlers.ts | 8 +- src/team-push.ts | 32 ++++---- src/types.ts | 12 +-- 17 files changed, 148 insertions(+), 72 deletions(-) diff --git a/CHANGELOG.md b/CHANGELOG.md index 36e24837a..9f7c37939 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -36,7 +36,7 @@ All notable changes to this project will be documented in this file. See [standa - A `manifest/roles.yaml` that exists but does not parse now fails the pull for that scope instead of warning and syncing with no role filter at all, for a member with no role as much as for one with a role. The same applies to `init` and `push`, which each fell back to a guess at the namespaces when any error came out of the loader. The legacy role migration skips with a warning instead of failing, so every command, `pull` included, still loads the config and can fetch the fixed manifest; until it can run, the member holds no role rather than every role, so hooks, MCP servers and env variables scoped by `roles:` reach them no more than skills do. For a member with no active project that fallback meant an unfiltered sync, so a broken manifest delivered every namespace it was written to gate. Only an absent manifest still means "this team does not use roles"; an unreadable or empty file is an error, as it now is for `manifest/projects.yaml` too. `push` stops at its scan for such a manifest (exit 2) even with `--role `, since the scan needs it to tell which namespaces are the member's. - `teamai members list` and `teamai projects members` read the roster registered before the reports switch, so a team upgrading past the orphan-branch split no longer sees "No team members registered" while its `members/` still lives on the default branch. The default-branch copy becomes a read-only inherited root, the way learnings' already was: listed in union with the `teamai-reports` copy, with the branch copy winning when the same file exists on both; nothing is copied or deleted, and a cold `members list` still does not publish the reports branch. Member registration merges against the inherited copy too, so a re-init keeps the original `registeredAt` and projects. Fixes [#735](https://github.com/Tencent/teamai-cli/issues/735). - Cache GC now rejects partial integers such as `12abc`, decimals, zero and unsafe integers for `--max-bytes` and `--stale-days` before deleting anything. An invalid `TEAMAI_CACHE_MAX_BYTES` value falls back to the default 5 GB limit instead of using a numeric prefix. -- Each scope reports only the dashboard sessions recorded in it. Every scope read one machine-wide event log and picked its sessions out by the `cwd` they started in, so a user-scope pull reported every project's sessions to the user team, a project never reported its Copilot sessions (Copilot sends no `cwd`), and a session started under a symlinked path (or `/tmp` for `/private/tmp` on macOS) never matched its project. Each event now records the data directory of the scope `hook-dispatch` resolved, and a report keeps only its own scope's. Events recorded by an earlier release go to the project whose root holds their `cwd`, never to the user scope. The usage guide explains how to remove by hand a skill an earlier release reported into `stats/.yaml` from another project (for [#785](https://github.com/Tencent/teamai-cli/issues/785)). +- Each scope reports only the dashboard sessions recorded in it. Every scope read one machine-wide event log and picked its sessions out by the `cwd` they started in, so a user-scope pull reported every project's sessions to the user team, a project never reported its Copilot sessions (teamai does not record Copilot's `cwd`), and a session started under a symlinked path (or `/tmp` for `/private/tmp` on macOS) never matched its project. Each event now records a key (a hash, so no path is stored) of the data directory of the scope `hook-dispatch` resolved, and a report keeps only its own scope's. Events recorded by an earlier release go to the project whose root holds their `cwd`, never to the user scope. The usage guide explains how to remove by hand a skill an earlier release reported into `stats/.yaml` from another project (for [#785](https://github.com/Tencent/teamai-cli/issues/785)). - Usage reporting scopes sessions by path on Windows too. The project/user scope filter compared an event's `cwd` against `projectRoot` with a hard-coded `/` separator, so on Windows only a session started in the project root itself matched: every session started in a subdirectory was dropped from the project team's report and counted in the user scope's instead, which is the isolation the usage guide promises. Windows paths are also compared case-insensitively, so a drive letter or a directory name spelled with different case in the two sources no longer leaks a project session into the user scope. POSIX paths keep their own rules: case-sensitive, and a `\` in a filename stays part of the name. - `teamai tags subscribe` and `teamai tags unsubscribe` now invalidate the pull revision cache, as `teamai skill exclude` already does, so the next `teamai pull` applies the new subscriptions instead of reporting "Already synced" when the team repo has not changed. - `teamai pull` now deletes a tombstoned agent under all three render extensions, so the Codex `.toml` and Kiro `.json` copies of a removed agent no longer survive on other machines. The cleanup also runs when the team repo rev is unchanged, so an upgrade reaches machines that already pulled the tombstone with an older CLI. `teamai remove agents ` also honours `enabledAgents` and no longer deletes from excluded tools. Fixes [#576](https://github.com/Tencent/teamai-cli/issues/576). diff --git a/docs/designs/data-directory-layout.md b/docs/designs/data-directory-layout.md index d4d503adf..4c3ff26b3 100644 --- a/docs/designs/data-directory-layout.md +++ b/docs/designs/data-directory-layout.md @@ -314,7 +314,8 @@ project's report carry every project's skills. The user scope records in an earlier release still writes after a rollback; the shared file is never read. The dashboard stays an A2 singleton: `teamai dashboard`, `stats --by-repo`, `session save` and the contribute check read across scopes. Each event instead -carries `dataHome`, the `getDataHome()` of the scope the hook resolved (#785), and a +carries `dataHomeKey`, a hash of the realpath'd `getDataHome()` of the scope the +hook resolved (#785; a hash, so a Copilot event still stores no path), and a scope's report keeps only its own. An event written before that field existed is attributed by its `cwd`, realpath'd, to the project whose root holds it, never to the user scope. diff --git a/docs/usage-guide.md b/docs/usage-guide.md index b1c6b8ae8..c6eeedcc9 100644 --- a/docs/usage-guide.md +++ b/docs/usage-guide.md @@ -1745,10 +1745,10 @@ reported snapshots. Skill usage is recorded per scope, in the data directory of the project teamai is set up for where the session ran (or the user scope), so each target reports only its own; a directory without teamai records none. Dashboard sessions stay in one machine-wide `~/.teamai/dashboard/events.jsonl`, -but each event records the data home of its scope, so a scope reports only the -sessions recorded in it: a user-scope pull no longer reports a project's -sessions, and a project reports its Copilot sessions and sessions started under -a symlinked path. Events recorded by an earlier release carry no data home: a +but each event records a key of its scope's data home (a hash, not the path), +so a scope reports only the sessions recorded in it: a user-scope pull no +longer reports a project's sessions, and a project reports its Copilot sessions and sessions started under +a symlinked path. Events recorded by an earlier release carry no key: a project reports those whose directory lies under its root, the user scope reports none of them. A target removes its usage events only after it confirms success; failed pushes diff --git a/docs/usage-guide.zh-CN.md b/docs/usage-guide.zh-CN.md index 0758aaf6f..54909a751 100644 --- a/docs/usage-guide.zh-CN.md +++ b/docs/usage-guide.zh-CN.md @@ -1685,9 +1685,9 @@ Pull 对整批统计上报最多等待 5 秒，之后继续其他工作，上报 超时后推送成功，仍会更新本地已上报快照。skill 使用按 scope 记录：写入会话所在 目录对应的已配置 teamai 项目的数据目录（或 user scope），因此每个目标只上报 自己的使用；未配置 teamai 的目录不记录。Dashboard 会话仍写入整机共用的 -`~/.teamai/dashboard/events.jsonl`，但每条事件都记下所属 scope 的数据目录（data home），因此每个 scope +`~/.teamai/dashboard/events.jsonl`，但每条事件都记下所属 scope 数据目录（data home）的键（哈希值，不是路径），因此每个 scope 只上报在其中记录的会话：user scope 的 pull 不再上报项目的会话，项目会上报自己的 -Copilot 会话以及从软链接路径启动的会话。旧版本记录的事件没有数据目录：项目上报目录 +Copilot 会话以及从软链接路径启动的会话。旧版本记录的事件没有该键：项目上报目录 位于其根目录下的那些，user scope 一条也不上报。目标确认成功后才清理自己的使用事件， 推送失败会保留事件。上报完成前继续持有相关同步锁， 避免另一次 Pull 与尚未完成的上报竞争。 diff --git a/src/__tests__/dashboard-report-gate.test.ts b/src/__tests__/dashboard-report-gate.test.ts index 6c070bc4c..ecb7d3e39 100644 --- a/src/__tests__/dashboard-report-gate.test.ts +++ b/src/__tests__/dashboard-report-gate.test.ts @@ -3,7 +3,7 @@ import fs from 'node:fs'; import os from 'node:os'; import path from 'node:path'; import { Readable } from 'node:stream'; -import { dashboardReport } from '../dashboard-collector.js'; +import { dashboardReport, dataHomeKey } from '../dashboard-collector.js'; import { _setLogFilePath, _resetState } from '../utils/logger.js'; // ─── legacy `dashboard-report` config gate ─── @@ -110,6 +110,8 @@ describe('legacy dashboard-report config gate', () => { expect(events).toHaveLength(1); expect(events[0].type).toBe('session_start'); expect(events[0].sessionId).toBe('sess-legacy-1'); + // Keyed to the scope it resolved, so that scope's report keeps it (#785). + expect(events[0].dataHomeKey).toBe(await dataHomeKey(userConfigDir)); }); it('records nothing when no cwd can be resolved from the payload', async () => { diff --git a/src/__tests__/dashboard-report-scope.test.ts b/src/__tests__/dashboard-report-scope.test.ts index 09d5b5b9c..a32456926 100644 --- a/src/__tests__/dashboard-report-scope.test.ts +++ b/src/__tests__/dashboard-report-scope.test.ts @@ -32,6 +32,7 @@ vi.mock('../utils/reports-branch.js', () => ({ const { hookDispatchCli } = await import('../hook-dispatch-cli.js'); const { resolveProjectDataHome, saveLocalConfigForScope, resolveConfigForDir, loadLocalConfig } = await import('../config.js'); const { reportUsageToTeam } = await import('../team-push.js'); +const { getDataHome } = await import('../types.js'); let tmp: string; let originalHome: string | undefined; @@ -129,6 +130,17 @@ describe('each scope reports only the dashboard sessions recorded in it (#785)', expect(await reportedSessions(project)).toBe(1); }); + it('a Copilot event records its scope without persisting a path (#666)', async () => { + const { root, project } = await setup(); + await session('copilot', { session_id: 'copilot-p', cwd: root }); + + const log = fs.readFileSync(path.join(teamaiHome(), 'dashboard', 'events.jsonl'), 'utf-8'); + const partition = getDataHome(project); + expect(log).not.toContain(root); + expect(log).not.toContain(partition); + expect(log).not.toContain(path.basename(partition)); + }); + it('a session started under a symlinked path of the project is reported by the project', async () => { const { root, user, project } = await setup(); const link = path.join(tmp, 'link-p'); diff --git a/src/__tests__/e2e/report-timeout.test.ts b/src/__tests__/e2e/report-timeout.test.ts index f945243bc..705362783 100644 --- a/src/__tests__/e2e/report-timeout.test.ts +++ b/src/__tests__/e2e/report-timeout.test.ts @@ -5,6 +5,7 @@ import os from 'node:os'; import path from 'node:path'; import { fileURLToPath } from 'node:url'; import YAML from 'yaml'; +import { dataHomeKey } from '../../dashboard-collector.js'; const root = path.resolve(path.dirname(fileURLToPath(import.meta.url)), '../../..'); const cli = path.join(root, 'dist/index.js'); @@ -42,15 +43,15 @@ function fixture(agent: keyof typeof agents, provider: string) { const timestamp = new Date().toISOString(); const usageLine = JSON.stringify({ skill: 'review', tool: agent, timestamp }) + '\n'; // A session the user scope recorded (#785). - const dataHome = path.join(home, '.teamai'); - function seedEvents() { + async function seedEvents() { + const key = await dataHomeKey(path.join(home, '.teamai')); fs.mkdirSync(dashboard, { recursive: true }); fs.writeFileSync(usage, usageLine); fs.writeFileSync(path.join(dashboard, 'events.jsonl'), [ - { type: 'session_start', timestamp, sessionId: 's1', tool: agent, cwd: sandbox, dataHome }, - { type: 'prompt_submit', timestamp, sessionId: 's1', tool: agent, promptSummary: 'review', dataHome }, + { type: 'session_start', timestamp, sessionId: 's1', tool: agent, cwd: sandbox, dataHomeKey: key }, + { type: 'prompt_submit', timestamp, sessionId: 's1', tool: agent, promptSummary: 'review', dataHomeKey: key }, { type: 'stop', timestamp, sessionId: 's1', tool: agent, interventions: { interrupt: 1, toolReject: 0 }, - tokens: { input: 10, output: 5, cacheRead: 0, cacheCreation: 0 }, dataHome }, + tokens: { input: 10, output: 5, cacheRead: 0, cacheCreation: 0 }, dataHomeKey: key }, ].map((e) => JSON.stringify(e)).join('\n') + '\n'); } function receiver(mode: 'slow' | 'reject' | 'normal') { @@ -101,7 +102,7 @@ describe('real CLI report completion', () => { it(`acknowledges slow pushes once: ${provider}/${agent}`, async () => { const f = fixture(agent, provider); await f.pull(); // Warm reports worktree without any session data. - f.seedEvents(); + await f.seedEvents(); f.receiver('slow'); const output = await f.pull(() => { expect(fs.readFileSync(f.usage, 'utf8')).toBe(f.usageLine); @@ -134,7 +135,7 @@ describe('real CLI report completion', () => { it('retains a rejected report and retries its committed tree without counting twice', async () => { const f = fixture('codex', 'git'); await f.pull(); - f.seedEvents(); + await f.seedEvents(); f.receiver('reject'); await f.pull(); expect(fs.readFileSync(f.usage, 'utf8')).toBe(f.usageLine); diff --git a/src/__tests__/git-kind-reports.test.ts b/src/__tests__/git-kind-reports.test.ts index 5e29a8ad2..ac46a3d34 100644 --- a/src/__tests__/git-kind-reports.test.ts +++ b/src/__tests__/git-kind-reports.test.ts @@ -12,6 +12,7 @@ import { getDataHome, getReportsDir, REPORTS_WORKTREE_DIRNAME, type LocalConfig import { commitAndPushReports, ensureReportsWorktree, refreshReportsWorktree, updateReports } from '../utils/reports-branch.js'; import { pushRepoDirectly } from '../utils/git.js'; import { reportUsageToTeam } from '../team-push.js'; +import { dataHomeKey } from '../dashboard-collector.js'; import { listMembers } from '../members.js'; import { resolveProjectDataHome, saveLocalConfigForScope } from '../config.js'; import { buildHandlerRegistry } from '../hook-handlers.js'; @@ -106,12 +107,12 @@ describe('git-kind reports branch', () => { const ts = new Date().toISOString(); const eventsDir = path.join(process.env.HOME!, '.teamai', 'dashboard'); // A session the user scope recorded. - const dataHome = getDataHome(cfg); + const key = await dataHomeKey(getDataHome(cfg)); fs.mkdirSync(eventsDir, { recursive: true }); fs.writeFileSync( path.join(eventsDir, 'events.jsonl'), - `${JSON.stringify({ type: 'session_start', timestamp: ts, sessionId: 's1', tool: 'claude', cwd: '/p', dataHome })}\n` + - `${JSON.stringify({ type: 'stop', timestamp: ts, sessionId: 's1', tool: 'claude', dataHome, interventions: { interrupt: 1, toolReject: 0 } })}\n`, + `${JSON.stringify({ type: 'session_start', timestamp: ts, sessionId: 's1', tool: 'claude', cwd: '/p', dataHomeKey: key })}\n` + + `${JSON.stringify({ type: 'stop', timestamp: ts, sessionId: 's1', tool: 'claude', dataHomeKey: key, interventions: { interrupt: 1, toolReject: 0 } })}\n`, ); await reportUsageToTeam(clone, 'alice', { skipTruncate: true, selfConfig: cfg }); diff --git a/src/__tests__/hook-handlers.test.ts b/src/__tests__/hook-handlers.test.ts index 3e074fd3c..63da9c2e2 100644 --- a/src/__tests__/hook-handlers.test.ts +++ b/src/__tests__/hook-handlers.test.ts @@ -36,6 +36,7 @@ vi.mock('../dashboard-collector.js', () => ({ appendEvent: mockAppendEvent, compactEvents: vi.fn().mockResolvedValue(undefined), dashboardReport: mockDashboardReport, + dataHomeKey: vi.fn().mockResolvedValue('key'), })); // Use the REAL resolveSkillUse (pure Skill/Read+SKILL.md logic, no I/O) so the diff --git a/src/__tests__/scope-filter.test.ts b/src/__tests__/scope-filter.test.ts index 0a039fb58..cb65ba307 100644 --- a/src/__tests__/scope-filter.test.ts +++ b/src/__tests__/scope-filter.test.ts @@ -3,11 +3,17 @@ import fs from 'node:fs'; import os from 'node:os'; import path from 'node:path'; import { filterEventsByScope } from '../team-push.js'; +import { dataHomeKey } from '../dashboard-collector.js'; import type { DashboardEvent, LocalConfig } from '../types.js'; -/** An event; without `dataHome` it is one written before events carried one, attributed by cwd. */ -function makeEvent(cwd: string | undefined, sessionId = 's1', dataHome?: string): DashboardEvent { - return { type: 'prompt_submit', timestamp: new Date().toISOString(), sessionId, tool: 'claude', cwd, dataHome }; +/** An event written before events carried a data home key, attributed by cwd. */ +function makeEvent(cwd: string | undefined, sessionId = 's1'): DashboardEvent { + return { type: 'prompt_submit', timestamp: new Date().toISOString(), sessionId, tool: 'claude', cwd }; +} + +/** An event recorded by the scope whose data home is `dataHome`. */ +async function scopedEvent(cwd: string | undefined, sessionId: string, dataHome: string): Promise { + return { ...makeEvent(cwd, sessionId), dataHomeKey: await dataHomeKey(dataHome) }; } const repo = { localPath: '/unused/team-repo', remote: 'https://example.test/team.git' }; @@ -37,31 +43,31 @@ describe('filterEventsByScope', () => { expect(await filterEventsByScope(events)).toEqual(events); }); - describe('events that carry a data home', () => { - const scoped: DashboardEvent[] = [ - makeEvent('/Users/jeff/project-a', 'p1', '/home/jeff/.teamai/projects/p'), - makeEvent(undefined, 'p2', '/home/jeff/.teamai/projects/p'), - makeEvent('/Users/jeff/project-a', 'u1', '/home/jeff/.teamai'), - makeEvent('/Users/jeff/project-a', 'q1', '/home/jeff/.teamai/projects/q'), - ]; + describe('events that carry a data home key', () => { + const scoped = (): Promise => Promise.all([ + scopedEvent('/Users/jeff/project-a', 'p1', '/home/jeff/.teamai/projects/p'), + scopedEvent(undefined, 'p2', '/home/jeff/.teamai/projects/p'), + scopedEvent('/Users/jeff/project-a', 'u1', '/home/jeff/.teamai'), + scopedEvent('/Users/jeff/project-a', 'q1', '/home/jeff/.teamai/projects/q'), + ]); it('a project keeps its own, whatever their cwd', async () => { - expect(await ids(scoped, projectScope('/Users/jeff/project-a'))).toEqual(['p1', 'p2']); + expect(await ids(await scoped(), projectScope('/Users/jeff/project-a'))).toEqual(['p1', 'p2']); }); it('the user scope keeps its own and no project\'s', async () => { - expect(await ids(scoped, userScope())).toEqual(['u1']); + expect(await ids(await scoped(), userScope())).toEqual(['u1']); }); it('a project keeps what it recorded under its in-repo .teamai before moving to a partition', async () => { - const legacy = [makeEvent(undefined, 'l1', '/Users/jeff/project-a/.teamai')]; + const legacy = [await scopedEvent(undefined, 'l1', '/Users/jeff/project-a/.teamai')]; expect(await ids(legacy, projectScope('/Users/jeff/project-a'))).toEqual(['l1']); expect(await ids(legacy, userScope())).toEqual([]); }); it('a project rooted at HOME does not take the user scope\'s events', async () => { const home = os.homedir(); - const evts = [makeEvent(undefined, 'u1', path.join(home, '.teamai'))]; + const evts = [await scopedEvent(undefined, 'u1', path.join(home, '.teamai'))]; expect(await ids(evts, projectScope(home))).toEqual([]); }); @@ -70,7 +76,7 @@ describe('filterEventsByScope', () => { try { fs.mkdirSync(path.join(tmp, 'real', '.teamai'), { recursive: true }); fs.symlinkSync(path.join(tmp, 'real'), path.join(tmp, 'link'), 'dir'); - const evts = [makeEvent(undefined, 'k1', path.join(tmp, 'link', '.teamai'))]; + const evts = [await scopedEvent(undefined, 'k1', path.join(tmp, 'link', '.teamai'))]; const config = projectScope(path.join(tmp, 'real'), path.join(tmp, 'real', '.teamai')); expect(await ids(evts, config)).toEqual(['k1']); } finally { @@ -78,12 +84,48 @@ describe('filterEventsByScope', () => { } }); + it('keeps what a project recorded in its in-repo .teamai after migration removed it', async () => { + const tmp = fs.realpathSync(fs.mkdtempSync(path.join(os.tmpdir(), 'teamai-scope-gone-'))); + try { + fs.mkdirSync(path.join(tmp, 'real', '.teamai'), { recursive: true }); + fs.symlinkSync(path.join(tmp, 'real'), path.join(tmp, 'link'), 'dir'); + const evts = [await scopedEvent(undefined, 'g1', path.join(tmp, 'link', '.teamai'))]; + fs.rmSync(path.join(tmp, 'real', '.teamai'), { recursive: true }); + expect(await ids(evts, projectScope(path.join(tmp, 'link')))).toEqual(['g1']); + } finally { + fs.rmSync(tmp, { recursive: true, force: true }); + } + }); + it('a Windows data home matches whatever its case or separators', async () => { - const win = [makeEvent(undefined, 'w1', 'C:\\Users\\Jeff\\.teamai'), makeEvent(undefined, 'w2', 'C:\\Users\\jeff\\.teamai\\projects\\p')]; + const win = await Promise.all([scopedEvent(undefined, 'w1', 'C:\\Users\\Jeff\\.teamai'), scopedEvent(undefined, 'w2', 'C:\\Users\\jeff\\.teamai\\projects\\p')]); expect(await ids(win, userScope('c:/users/jeff/.teamai'))).toEqual(['w1']); }); }); + it('an event whose key is not a string counts as written before keys existed', async () => { + // A hand-edited or corrupted line in the shared log. + const corrupt: DashboardEvent[] = JSON.parse(JSON.stringify([ + { ...makeEvent('/Users/jeff/project-a', 'c1'), dataHomeKey: null }, + { ...makeEvent('/Users/jeff/other-work', 'c2'), dataHomeKey: 42 }, + ])); + expect(await ids(corrupt, projectScope('/Users/jeff/project-a'))).toEqual(['c1']); + expect(await ids(corrupt, userScope())).toEqual([]); + }); + + it('matches an older event\'s real cwd to a project whose root was set through a symlink', async () => { + // A non-git project's root is the directory as given, not realpath'd. + const tmp = fs.realpathSync(fs.mkdtempSync(path.join(os.tmpdir(), 'teamai-scope-root-'))); + try { + fs.mkdirSync(path.join(tmp, 'real')); + fs.symlinkSync(path.join(tmp, 'real'), path.join(tmp, 'link'), 'dir'); + const evts = [makeEvent(path.join(tmp, 'real'), 'r1')]; + expect(await ids(evts, projectScope(path.join(tmp, 'link')))).toEqual(['r1']); + } finally { + fs.rmSync(tmp, { recursive: true, force: true }); + } + }); + it('the user scope never keeps events without a data home', async () => { expect(await ids(events, userScope())).toEqual([]); }); diff --git a/src/__tests__/self-mode-no-business-reset.test.ts b/src/__tests__/self-mode-no-business-reset.test.ts index d080649ee..19f0f65ea 100644 --- a/src/__tests__/self-mode-no-business-reset.test.ts +++ b/src/__tests__/self-mode-no-business-reset.test.ts @@ -10,6 +10,7 @@ import os from 'node:os'; import { simpleGit } from 'simple-git'; import { reportUsageToTeam } from '../team-push.js'; +import { dataHomeKey } from '../dashboard-collector.js'; import type { LocalConfig } from '../types.js'; let tmp: string; @@ -96,12 +97,12 @@ describe('E2E self-mode: business repo working tree is never reset', () => { const ts = new Date().toISOString(); const eventsDir = path.join(process.env.HOME!, '.teamai', 'dashboard'); // A session the user scope recorded (#785). - const dataHome = path.join(tmp, 'home', '.teamai'); + const key = await dataHomeKey(path.join(tmp, 'home', '.teamai')); fs.mkdirSync(eventsDir, { recursive: true }); fs.writeFileSync( path.join(eventsDir, 'events.jsonl'), - `${JSON.stringify({ type: 'session_start', timestamp: ts, sessionId: 's1', tool: 'claude', cwd: '/p', dataHome })}\n` + - `${JSON.stringify({ type: 'stop', timestamp: ts, sessionId: 's1', tool: 'claude', dataHome, interventions: { interrupt: 1, toolReject: 0 } })}\n`, + `${JSON.stringify({ type: 'session_start', timestamp: ts, sessionId: 's1', tool: 'claude', cwd: '/p', dataHomeKey: key })}\n` + + `${JSON.stringify({ type: 'stop', timestamp: ts, sessionId: 's1', tool: 'claude', dataHomeKey: key, interventions: { interrupt: 1, toolReject: 0 } })}\n`, ); const cfg: LocalConfig = { diff --git a/src/__tests__/team-push-interventions.test.ts b/src/__tests__/team-push-interventions.test.ts index 6542148d5..892b7b5d6 100644 --- a/src/__tests__/team-push-interventions.test.ts +++ b/src/__tests__/team-push-interventions.test.ts @@ -35,11 +35,14 @@ vi.mock('../utils/fs.js', async (importActual) => { }); import { reportUsageToTeam } from '../team-push.js'; +import { dataHomeKey } from '../dashboard-collector.js'; import { withTimeout } from '../utils/async.js'; let tmpDir: string; let repoDir: string; let originalHome: string; +/** `dataHomeKey()` of the user scope that `gitConfig()` reports. */ +let userKey: string; function gitConfig(): LocalConfig { return { @@ -54,10 +57,11 @@ function reportsStatsPath(): string { return path.join(tmpDir, 'reports-wt', 'stats', 'me.yaml'); } -beforeEach(() => { +beforeEach(async () => { tmpDir = fs.mkdtempSync(path.join(os.tmpdir(), 'teamai-tp-iv-')); originalHome = process.env.HOME ?? ''; process.env.HOME = tmpDir; + userKey = await dataHomeKey(path.join(tmpDir, '.teamai')); repoDir = path.join(tmpDir, 'repo'); fs.mkdirSync(repoDir, { recursive: true }); pushRepoDirectly.mockReset().mockResolvedValue(undefined); @@ -79,8 +83,7 @@ afterEach(() => { function writeDashboardEvents(lines: object[]): void { const p = path.join(tmpDir, '.teamai', 'dashboard', 'events.jsonl'); fs.mkdirSync(path.dirname(p), { recursive: true }); - const dataHome = path.join(tmpDir, '.teamai'); - fs.writeFileSync(p, lines.map((l) => JSON.stringify({ ...l, dataHome })).join('\n') + '\n'); + fs.writeFileSync(p, lines.map((l) => JSON.stringify({ ...l, dataHomeKey: userKey })).join('\n') + '\n'); } describe('reportUsageToTeam — intervention reporting', () => { diff --git a/src/dashboard-collector.ts b/src/dashboard-collector.ts index a80e55500..9ce30cd01 100644 --- a/src/dashboard-collector.ts +++ b/src/dashboard-collector.ts @@ -47,7 +47,7 @@ import { estimateClaudeRequest } from './model-pricing.js'; // │ extract: session_id / cwd / tool_name / prompt // ▼ // DashboardEvent -// │ dataHome = data home of the scope the hook resolved +// │ dataHomeKey = key of the data home of the scope the hook resolved // ▼ // appendEvent(event) → events.jsonl // @@ -1480,6 +1480,22 @@ export async function reconcileRequestLog( } } +/** + * The key an event records for the scope that wrote it, and that a scope's + * report matches (#785): a hash of the data home, so the log stores no path + * (Copilot events persist none, #666). The data home is realpath'd, so a + * symlinked checkout or macOS `/tmp` vs `/private/tmp` keys the same; a + * Windows path also folds separators and case, as the report's cwd rule does. + * A data home that is gone (an in-repo `.teamai` removed after migration) + * keys through its parent's realpath. + */ +export async function dataHomeKey(dataHome: string): Promise { + const real = await fs.promises.realpath(dataHome).catch(() => !path.isAbsolute(dataHome) ? dataHome + : fs.promises.realpath(path.dirname(dataHome)).then((p) => path.join(p, path.basename(dataHome)), () => dataHome)); + const norm = /^(?:[A-Za-z]:[\\/]|\\\\)/.test(real) ? real.replace(/\\/g, '/').toLowerCase() : real; + return createHash('sha256').update(norm.replace(/\/+$/, '')).digest('hex').slice(0, 16); +} + /** * Append a DashboardEvent to the events JSONL file. * Silently fails on I/O errors to avoid disrupting the AI session. @@ -1967,7 +1983,7 @@ export async function dashboardReport(toolArg?: string): Promise { const event = await parseHookEvent(raw, toolArg ?? 'claude'); if (!event) return; - event.dataHome = getDataHome(config); + event.dataHomeKey = await dataHomeKey(getDataHome(config)); await appendEvent(event); // Trigger compaction check (non-blocking) diff --git a/src/dashboard.ts b/src/dashboard.ts index 83c338402..845a9a390 100644 --- a/src/dashboard.ts +++ b/src/dashboard.ts @@ -124,7 +124,7 @@ export async function startDashboard(port?: number): Promise { tool: session.tool, cwd: session.cwd, // The session's own data home, so its scope's report still sees it end. - dataHome: events.find((e) => e.sessionId === session.sessionId && e.dataHome)?.dataHome, + dataHomeKey: events.find((e) => e.sessionId === session.sessionId && e.dataHomeKey)?.dataHomeKey, }; await appendEvent(exitEvent); log.info( diff --git a/src/hook-handlers.ts b/src/hook-handlers.ts index 9b237bb3a..eeb0250f6 100644 --- a/src/hook-handlers.ts +++ b/src/hook-handlers.ts @@ -175,7 +175,7 @@ const dashboardReportHandler: HookHandler = { async execute(stdin, tool, config) { // Registered with requiresConfig: a session outside any scope is not recorded. if (!config) return null; - const { parseHookEvent, appendEvent, compactEvents } = await import('./dashboard-collector.js'); + const { parseHookEvent, appendEvent, compactEvents, dataHomeKey } = await import('./dashboard-collector.js'); const { getDataHome } = await import('./types.js'); const raw = JSON.stringify(stdin); const event = await parseHookEvent(raw, tool, { @@ -183,9 +183,9 @@ const dashboardReportHandler: HookHandler = { modelAliases: await userModelAliases(stdin), }); if (event) { - // The dispatcher's scope, which knows the project even when the host - // sends no cwd (Copilot) or a symlinked one (#785). - event.dataHome = getDataHome(config); + // The dispatcher's scope, which knows the project even when the event + // records no cwd (Copilot) or a symlinked one (#785). + event.dataHomeKey = await dataHomeKey(getDataHome(config)); await appendEvent(event); // Non-blocking compaction compactEvents().catch(() => {}); diff --git a/src/team-push.ts b/src/team-push.ts index 6ed47b1a5..721c2486e 100644 --- a/src/team-push.ts +++ b/src/team-push.ts @@ -3,7 +3,7 @@ import fs from 'node:fs'; import path from 'node:path'; import { readUsageEvents, truncateUsageAfterReport } from './usage-tracker.js'; import { aggregateUsage } from './stats.js'; -import { readEvents, aggregateSessionMetrics } from './dashboard-collector.js'; +import { readEvents, aggregateSessionMetrics, dataHomeKey } from './dashboard-collector.js'; import { createGit, pushRepoDirectly, @@ -354,14 +354,13 @@ function isUnderScopeRoot(cwd: string, root: ScopeRoot): boolean { /** * The dashboard events a scope reports (#785): those recorded in it, keyed by - * its data home. A project also owns the key of its in-repo `.teamai`, where a - * hook recorded until migration moved the project to a partition. An event - * written before events carried a data home belongs to the project whose root holds - * its cwd, never to the user scope. `projectRoot` is realpath'd, but a cwd is - * raw as the host sent it (a symlinked checkout, macOS `/tmp` vs - * `/private/tmp`) and so is a non-git project's data home, so both are - * realpath'd while they still exist. A caller without a scope config reads the - * whole log. + * its data home (`dataHomeKey`). A project also owns the key of its in-repo + * `.teamai`, where a hook recorded until migration moved the project to a + * partition. An event written before events carried a key belongs to the + * project whose root holds its cwd, never to the user scope. A cwd is raw as + * the host sent it (a symlinked checkout, macOS `/tmp` vs `/private/tmp`), and + * so is a non-git project's root, so both are realpath'd while they still exist. + * A caller without a scope config reads the whole log. */ export async function filterEventsByScope( events: DashboardEvent[], @@ -377,19 +376,16 @@ export async function filterEventsByScope( } return real; }; - const keyOf = async (home: string): Promise => scopeRoot(await realPath(home)); - const homeRoots = [await keyOf(getDataHome(config))]; + const keys = new Set([await dataHomeKey(getDataHome(config))]); if (config.projectRoot) { // Unless the project is rooted at HOME, where that is the user scope's. - const legacy = await keyOf(path.join(config.projectRoot, '.teamai')); - if (legacy.key !== (await keyOf(path.join(getUserHome(), '.teamai'))).key) homeRoots.push(legacy); + const legacy = await dataHomeKey(path.join(config.projectRoot, '.teamai')); + if (legacy !== (await dataHomeKey(path.join(getUserHome(), '.teamai')))) keys.add(legacy); } - const root = config.projectRoot ? scopeRoot(config.projectRoot) : undefined; + const root = config.projectRoot ? scopeRoot(await realPath(config.projectRoot)) : undefined; const kept = await Promise.all(events.map(async (e) => { - if (e.dataHome !== undefined) { - const dataHome = await realPath(e.dataHome); - return homeRoots.some((home) => scopeKey(dataHome, home.windows) === home.key); - } + // The log is hand-editable: a key that is not a string counts as absent. + if (typeof e.dataHomeKey === 'string') return keys.has(e.dataHomeKey); return !!root && !!e.cwd && isUnderScopeRoot(await realPath(e.cwd), root); })); return events.filter((_, i) => kept[i]); diff --git a/src/types.ts b/src/types.ts index 4c741de92..ea93a69e9 100644 --- a/src/types.ts +++ b/src/types.ts @@ -1220,11 +1220,11 @@ export interface DashboardEvent { /** Working directory of the session */ cwd?: string; /** - * Data home (`getDataHome`) of the scope that recorded the event, the key its - * report filters on (#785). Absent on events written before this field - * existed; the report then attributes them by `cwd`. + * `dataHomeKey()` of the data home of the scope that recorded the event, the + * key its report filters on (#785). Absent on events written before this + * field existed; the report then attributes them by `cwd`. */ - dataHome?: string; + dataHomeKey?: string; /** First user prompt (captured from UserPromptSubmit) */ promptSummary?: string; /** @@ -1340,8 +1340,8 @@ export interface DashboardSession { // getUserHome() (dashboard-collector.ts getEventsPath, dashboard.ts), so HOME // isolation already works there. Removed (issue #374 P3). The dashboard is an // A2 machine-level singleton keyed by sessionId, not per-project; each event -// carries the data home of the scope that recorded it, which is what a scope's -// report filters on. +// carries the key of the data home of the scope that recorded it, which is what +// a scope's report filters on. export const DASHBOARD_DEFAULT_PORT = 3721; /** Sessions with no activity for this long (ms) are marked idle */ export const DASHBOARD_IDLE_TIMEOUT_MS = 5 * 60 * 1000; From aa8513cb6dab257c57f231fce0e071e108121860 Mon Sep 17 00:00:00 2001 From: Saul Moro Date: Thu, 24 Sep 2026 12:05:42 +0200 Subject: [PATCH 03/41] docs(report): place the stats cleanup under usage reporting (#785) The manual `stats/.yaml` cleanup sat under single-repo mode, but the pre-#748 leak hit every team with a git-kind repo, so it moves to "Usage reporting" and notes where an `http` team repo keeps the file. The guide also says the scope key is per event: hooks that run outside the project (a worktree removed before the session ends) report to the scope they ran in. --- docs/usage-guide.md | 45 +++++++++++++++++++++------------------ docs/usage-guide.zh-CN.md | 33 ++++++++++++++-------------- 2 files changed, 41 insertions(+), 37 deletions(-) diff --git a/docs/usage-guide.md b/docs/usage-guide.md index c6eeedcc9..3e9cdbb78 100644 --- a/docs/usage-guide.md +++ b/docs/usage-guide.md @@ -370,23 +370,6 @@ branch, exactly where they are. Nothing is copied, deleted or migrated: that directory is still read, so every existing learning keeps coming back from `teamai recall`. New learnings go to `teamai-learnings`. -**Removing a skill another project reported into your `stats/`.** Before skill -usage was kept per scope, whichever project pulled next reported every -project's skills, so `stats/.yaml` on `teamai-reports` can count a skill -that belongs to an unrelated repo. Those events recorded no directory, so -teamai cannot attribute them and never rewrites the file. Remove the entry by -hand, from a clone of your own so teamai's `reports-wt/` checkout is untouched: - -```bash -git clone --branch teamai-reports --single-branch teamai-reports -cd teamai-reports -# delete the skill's entry under `skills:` in stats/.yaml -git commit -am "stats: remove reported from another project" -git push origin teamai-reports -``` - -The next report reads the branch first, so the entry does not come back. - **Minimum Git permissions with a protected default branch.** A member needs to: @@ -1747,10 +1730,12 @@ each target reports only its own; a directory without teamai records none. Dashboard sessions stay in one machine-wide `~/.teamai/dashboard/events.jsonl`, but each event records a key of its scope's data home (a hash, not the path), so a scope reports only the sessions recorded in it: a user-scope pull no -longer reports a project's sessions, and a project reports its Copilot sessions and sessions started under -a symlinked path. Events recorded by an earlier release carry no key: a -project reports those whose directory lies under its root, the user scope -reports none of them. A +longer reports a project's sessions, and a project reports its Copilot sessions +and sessions started under a symlinked path. The key is per event: hooks that +run outside the project (for example in a worktree removed before the session +ends) report their events to the scope they ran in. Events recorded by an +earlier release carry no key: a project reports those whose directory lies +under its root, the user scope reports none of them. A target removes its usage events only after it confirms success; failed pushes preserve them. The affected sync locks remain held until reporting finishes, preventing another pull from racing the report. @@ -1768,6 +1753,24 @@ can turn this off in `teamai.yaml`: usageReport: false ``` +**Removing a skill another project reported into your `stats/`.** Before skill +usage was kept per scope, whichever project pulled next reported every +project's skills, so `stats/.yaml` on `teamai-reports` can count a skill +that belongs to an unrelated repo. Those events recorded no directory, so +teamai cannot attribute them and never rewrites the file. Remove the entry by +hand, from a clone of your own so teamai's `reports-wt/` checkout is untouched: + +```bash +git clone --branch teamai-reports --single-branch teamai-reports +cd teamai-reports +# delete the skill's entry under `skills:` in stats/.yaml +git commit -am "stats: remove reported from another project" +git push origin teamai-reports +``` + +The next report reads the branch first, so the entry does not come back. For +an `http` team repo, `stats/.yaml` is on the default branch instead. + ### Git submodules If your team distributes skills as git submodules, opt in with `submodules: true` diff --git a/docs/usage-guide.zh-CN.md b/docs/usage-guide.zh-CN.md index 54909a751..68c1d4313 100644 --- a/docs/usage-guide.zh-CN.md +++ b/docs/usage-guide.zh-CN.md @@ -346,21 +346,6 @@ learnings 迁到独立分支之前团队已经写下的内容，原地留在默 不迁移：该目录仍会被读取，所有既有 learning 依然能从 `teamai recall` 中找回。新的 learning 写入 `teamai-learnings`。 -**删除其他项目上报进你 `stats/` 的 skill。** 在 skill 使用按 scope 记录之前，下一个 -执行 pull 的项目会上报所有项目的 skill，因此 `teamai-reports` 上的 `stats/.yaml` 可能 -统计了属于无关仓库的 skill。这些事件没有记录目录，teamai 无法归属，也不会改写该 -文件。请手动删除该条目，在单独的 clone 中操作，不要动 teamai 的 `reports-wt/` 检出： - -```bash -git clone --branch teamai-reports --single-branch teamai-reports -cd teamai-reports -# 删除 stats/.yaml 中 `skills:` 下该 skill 的条目 -git commit -am "stats: remove reported from another project" -git push origin teamai-reports -``` - -下一次上报会先读取该分支，所以条目不会再出现。 - **默认分支受保护时所需的最小 Git 权限。** 成员需要： @@ -1687,7 +1672,8 @@ Pull 对整批统计上报最多等待 5 秒，之后继续其他工作，上报 自己的使用；未配置 teamai 的目录不记录。Dashboard 会话仍写入整机共用的 `~/.teamai/dashboard/events.jsonl`，但每条事件都记下所属 scope 数据目录（data home）的键（哈希值，不是路径），因此每个 scope 只上报在其中记录的会话：user scope 的 pull 不再上报项目的会话，项目会上报自己的 -Copilot 会话以及从软链接路径启动的会话。旧版本记录的事件没有该键：项目上报目录 +Copilot 会话以及从软链接路径启动的会话。该键按事件记录：在项目之外运行的 hook +（例如会话结束前已删除的 worktree 中）会把事件上报给其所在的 scope。旧版本记录的事件没有该键：项目上报目录 位于其根目录下的那些，user scope 一条也不上报。目标确认成功后才清理自己的使用事件， 推送失败会保留事件。上报完成前继续持有相关同步锁， 避免另一次 Pull 与尚未完成的上报竞争。 @@ -1703,6 +1689,21 @@ Copilot 会话以及从软链接路径启动的会话。旧版本记录的事件 usageReport: false ``` +**删除其他项目上报进你 `stats/` 的 skill。** 在 skill 使用按 scope 记录之前，下一个 +执行 pull 的项目会上报所有项目的 skill，因此 `teamai-reports` 上的 `stats/.yaml` 可能 +统计了属于无关仓库的 skill。这些事件没有记录目录，teamai 无法归属，也不会改写该 +文件。请手动删除该条目，在单独的 clone 中操作，不要动 teamai 的 `reports-wt/` 检出： + +```bash +git clone --branch teamai-reports --single-branch teamai-reports +cd teamai-reports +# 删除 stats/.yaml 中 `skills:` 下该 skill 的条目 +git commit -am "stats: remove reported from another project" +git push origin teamai-reports +``` + +下一次上报会先读取该分支，所以条目不会再出现。若团队仓为 `http` 类型，`stats/.yaml` 位于默认分支。 + ### Git 子模块 若团队以 git submodule 形式分发 skill，在 `teamai.yaml` 中开启 `submodules: true`： From 703ca67f082d26ebd02722cd6cbc1ce42d604377 Mon Sep 17 00:00:00 2001 From: Saul Moro Date: Thu, 24 Sep 2026 12:22:20 +0200 Subject: [PATCH 04/41] docs(report): name where unattributed sessions go (#785) The CHANGELOG now says a session in a directory that resolves to no project (a non-git project's subdirectory, a submodule or nested clone) is the user scope's, as for skill usage. The usage guide drops the line on http team repos: pull does not report usage to them, so no stats file there needs cleaning. --- CHANGELOG.md | 2 +- docs/usage-guide.md | 3 +-- docs/usage-guide.zh-CN.md | 2 +- 3 files changed, 3 insertions(+), 4 deletions(-) diff --git a/CHANGELOG.md b/CHANGELOG.md index 9f7c37939..9ba1b5ef3 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -36,7 +36,7 @@ All notable changes to this project will be documented in this file. See [standa - A `manifest/roles.yaml` that exists but does not parse now fails the pull for that scope instead of warning and syncing with no role filter at all, for a member with no role as much as for one with a role. The same applies to `init` and `push`, which each fell back to a guess at the namespaces when any error came out of the loader. The legacy role migration skips with a warning instead of failing, so every command, `pull` included, still loads the config and can fetch the fixed manifest; until it can run, the member holds no role rather than every role, so hooks, MCP servers and env variables scoped by `roles:` reach them no more than skills do. For a member with no active project that fallback meant an unfiltered sync, so a broken manifest delivered every namespace it was written to gate. Only an absent manifest still means "this team does not use roles"; an unreadable or empty file is an error, as it now is for `manifest/projects.yaml` too. `push` stops at its scan for such a manifest (exit 2) even with `--role `, since the scan needs it to tell which namespaces are the member's. - `teamai members list` and `teamai projects members` read the roster registered before the reports switch, so a team upgrading past the orphan-branch split no longer sees "No team members registered" while its `members/` still lives on the default branch. The default-branch copy becomes a read-only inherited root, the way learnings' already was: listed in union with the `teamai-reports` copy, with the branch copy winning when the same file exists on both; nothing is copied or deleted, and a cold `members list` still does not publish the reports branch. Member registration merges against the inherited copy too, so a re-init keeps the original `registeredAt` and projects. Fixes [#735](https://github.com/Tencent/teamai-cli/issues/735). - Cache GC now rejects partial integers such as `12abc`, decimals, zero and unsafe integers for `--max-bytes` and `--stale-days` before deleting anything. An invalid `TEAMAI_CACHE_MAX_BYTES` value falls back to the default 5 GB limit instead of using a numeric prefix. -- Each scope reports only the dashboard sessions recorded in it. Every scope read one machine-wide event log and picked its sessions out by the `cwd` they started in, so a user-scope pull reported every project's sessions to the user team, a project never reported its Copilot sessions (teamai does not record Copilot's `cwd`), and a session started under a symlinked path (or `/tmp` for `/private/tmp` on macOS) never matched its project. Each event now records a key (a hash, so no path is stored) of the data directory of the scope `hook-dispatch` resolved, and a report keeps only its own scope's. Events recorded by an earlier release go to the project whose root holds their `cwd`, never to the user scope. The usage guide explains how to remove by hand a skill an earlier release reported into `stats/.yaml` from another project (for [#785](https://github.com/Tencent/teamai-cli/issues/785)). +- Each scope reports only the dashboard sessions recorded in it. Every scope read one machine-wide event log and picked its sessions out by the `cwd` they started in, so a user-scope pull reported every project's sessions to the user team, a project never reported its Copilot sessions (teamai does not record Copilot's `cwd`), and a session started under a symlinked path (or `/tmp` for `/private/tmp` on macOS) never matched its project. Each event now records a key (a hash, so no path is stored) of the data directory of the scope `hook-dispatch` resolved, and a report keeps only its own scope's. A session in a directory that resolves to no project (a subdirectory of a non-git project, a git submodule or nested clone) is the user scope's, as for skill usage. Events recorded by an earlier release go to the project whose root holds their `cwd`, never to the user scope. The usage guide explains how to remove by hand a skill an earlier release reported into `stats/.yaml` from another project (for [#785](https://github.com/Tencent/teamai-cli/issues/785)). - Usage reporting scopes sessions by path on Windows too. The project/user scope filter compared an event's `cwd` against `projectRoot` with a hard-coded `/` separator, so on Windows only a session started in the project root itself matched: every session started in a subdirectory was dropped from the project team's report and counted in the user scope's instead, which is the isolation the usage guide promises. Windows paths are also compared case-insensitively, so a drive letter or a directory name spelled with different case in the two sources no longer leaks a project session into the user scope. POSIX paths keep their own rules: case-sensitive, and a `\` in a filename stays part of the name. - `teamai tags subscribe` and `teamai tags unsubscribe` now invalidate the pull revision cache, as `teamai skill exclude` already does, so the next `teamai pull` applies the new subscriptions instead of reporting "Already synced" when the team repo has not changed. - `teamai pull` now deletes a tombstoned agent under all three render extensions, so the Codex `.toml` and Kiro `.json` copies of a removed agent no longer survive on other machines. The cleanup also runs when the team repo rev is unchanged, so an upgrade reaches machines that already pulled the tombstone with an older CLI. `teamai remove agents ` also honours `enabledAgents` and no longer deletes from excluded tools. Fixes [#576](https://github.com/Tencent/teamai-cli/issues/576). diff --git a/docs/usage-guide.md b/docs/usage-guide.md index 3e9cdbb78..5cf77f893 100644 --- a/docs/usage-guide.md +++ b/docs/usage-guide.md @@ -1768,8 +1768,7 @@ git commit -am "stats: remove reported from another project" git push origin teamai-reports ``` -The next report reads the branch first, so the entry does not come back. For -an `http` team repo, `stats/.yaml` is on the default branch instead. +The next report reads the branch first, so the entry does not come back. ### Git submodules diff --git a/docs/usage-guide.zh-CN.md b/docs/usage-guide.zh-CN.md index 68c1d4313..88a2c8f6b 100644 --- a/docs/usage-guide.zh-CN.md +++ b/docs/usage-guide.zh-CN.md @@ -1702,7 +1702,7 @@ git commit -am "stats: remove reported from another project" git push origin teamai-reports ``` -下一次上报会先读取该分支，所以条目不会再出现。若团队仓为 `http` 类型，`stats/.yaml` 位于默认分支。 +下一次上报会先读取该分支，所以条目不会再出现。 ### Git 子模块 From 9e70d66f517305bf910a9034ff6064d2028afe16 Mon Sep 17 00:00:00 2001 From: Saul Moro Date: Thu, 24 Sep 2026 10:49:44 +0200 Subject: [PATCH 05/41] fix(report): each scope keeps its own reported dashboard snapshots (#786) The report sends per-session deltas against reported-*.json snapshots that every scope shared. A session whose events belong to two scopes (a cd into another project mid-session) was then reported by the first scope, and the second compared its own part with the first scope's totals and sent nothing. Each scope now keeps its snapshots in /dashboard/, and the user scope, whose data home holds the shared files, in user-reported-*.json. The first time a scope needs one it copies the shared file, so the first report after the upgrade sends nothing already reported; after that it reads only its own. The user scope moves too, unlike the ticket proposed: had it kept writing the shared file, a project seeding later would copy the user scope's part of a split session and report nothing for its own. The shared file is no longer written, except by an earlier release after a rollback, which only a scope not yet seeded reads. --- CHANGELOG.md | 1 + docs/designs/data-directory-layout.md | 12 +- docs/usage-guide.md | 5 +- docs/usage-guide.zh-CN.md | 4 +- src/__tests__/dashboard-report-scope.test.ts | 124 +++++++++++++++++- src/__tests__/e2e/report-timeout.test.ts | 13 +- src/__tests__/team-push-interventions.test.ts | 22 ++-- src/team-push.ts | 108 +++++++++------ 8 files changed, 227 insertions(+), 62 deletions(-) diff --git a/CHANGELOG.md b/CHANGELOG.md index 9ba1b5ef3..3b526b480 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -37,6 +37,7 @@ All notable changes to this project will be documented in this file. See [standa - `teamai members list` and `teamai projects members` read the roster registered before the reports switch, so a team upgrading past the orphan-branch split no longer sees "No team members registered" while its `members/` still lives on the default branch. The default-branch copy becomes a read-only inherited root, the way learnings' already was: listed in union with the `teamai-reports` copy, with the branch copy winning when the same file exists on both; nothing is copied or deleted, and a cold `members list` still does not publish the reports branch. Member registration merges against the inherited copy too, so a re-init keeps the original `registeredAt` and projects. Fixes [#735](https://github.com/Tencent/teamai-cli/issues/735). - Cache GC now rejects partial integers such as `12abc`, decimals, zero and unsafe integers for `--max-bytes` and `--stale-days` before deleting anything. An invalid `TEAMAI_CACHE_MAX_BYTES` value falls back to the default 5 GB limit instead of using a numeric prefix. - Each scope reports only the dashboard sessions recorded in it. Every scope read one machine-wide event log and picked its sessions out by the `cwd` they started in, so a user-scope pull reported every project's sessions to the user team, a project never reported its Copilot sessions (teamai does not record Copilot's `cwd`), and a session started under a symlinked path (or `/tmp` for `/private/tmp` on macOS) never matched its project. Each event now records a key (a hash, so no path is stored) of the data directory of the scope `hook-dispatch` resolved, and a report keeps only its own scope's. A session in a directory that resolves to no project (a subdirectory of a non-git project, a git submodule or nested clone) is the user scope's, as for skill usage. Events recorded by an earlier release go to the project whose root holds their `cwd`, never to the user scope. The usage guide explains how to remove by hand a skill an earlier release reported into `stats/.yaml` from another project (for [#785](https://github.com/Tencent/teamai-cli/issues/785)). +- A session whose events belong to two scopes is reported to both teams with the part recorded in each. The snapshots of what was already reported were machine-wide, so after a `cd` into another project mid-session, the second scope to report compared its part with the first scope's totals and sent nothing. Each scope now keeps its own (`/dashboard/reported-*.json`, and `~/.teamai/dashboard/user-reported-*.json` for the user scope), seeded the first time from the shared `~/.teamai/dashboard/reported-*.json`, so the first report after the upgrade sends nothing already reported. The shared file is no longer written; after a rollback an earlier release writes it again, and only a scope not yet seeded reads it (for [#786](https://github.com/Tencent/teamai-cli/issues/786)). - Usage reporting scopes sessions by path on Windows too. The project/user scope filter compared an event's `cwd` against `projectRoot` with a hard-coded `/` separator, so on Windows only a session started in the project root itself matched: every session started in a subdirectory was dropped from the project team's report and counted in the user scope's instead, which is the isolation the usage guide promises. Windows paths are also compared case-insensitively, so a drive letter or a directory name spelled with different case in the two sources no longer leaks a project session into the user scope. POSIX paths keep their own rules: case-sensitive, and a `\` in a filename stays part of the name. - `teamai tags subscribe` and `teamai tags unsubscribe` now invalidate the pull revision cache, as `teamai skill exclude` already does, so the next `teamai pull` applies the new subscriptions instead of reporting "Already synced" when the team repo has not changed. - `teamai pull` now deletes a tombstoned agent under all three render extensions, so the Codex `.toml` and Kiro `.json` copies of a removed agent no longer survive on other machines. The cleanup also runs when the team repo rev is unchanged, so an upgrade reaches machines that already pulled the tombstone with an older CLI. `teamai remove agents ` also honours `enabledAgents` and no longer deletes from excluded tools. Fixes [#576](https://github.com/Tencent/teamai-cli/issues/576). diff --git a/docs/designs/data-directory-layout.md b/docs/designs/data-directory-layout.md index 4c3ff26b3..d2912d2e9 100644 --- a/docs/designs/data-directory-layout.md +++ b/docs/designs/data-directory-layout.md @@ -318,7 +318,17 @@ carries `dataHomeKey`, a hash of the realpath'd `getDataHome()` of the scope the hook resolved (#785; a hash, so a Copilot event still stores no path), and a scope's report keeps only its own. An event written before that field existed is attributed by its `cwd`, realpath'd, to the project whose root holds it, never to -the user scope. +the user scope. The snapshots of what was already reported are per scope too +(#786), because a session can record events in two scopes (a `cd` mid-session): +`/dashboard/reported-*.json`, and `~/.teamai/dashboard/user-reported-*.json` +for the user scope. The first time a scope needs one it copies the shared +`~/.teamai/dashboard/reported-*.json`, so nothing reported before the upgrade is +sent again; after that it reads only its own. The shared file is no longer +written, except by an earlier release after a rollback, so every scope seeds from +what the machine had reported by then, never from another scope's later report. +The seed holds a session's whole total, so a session still running at the upgrade +that later records in a second scope reports nothing there until that scope's +part exceeds it. **`anchor` on save.** Previously only migration wrote a partition's `anchor` reverse-lookup file, so freshly-init'd partitions had none. `saveLocalConfigForScope` diff --git a/docs/usage-guide.md b/docs/usage-guide.md index 5cf77f893..f5af28531 100644 --- a/docs/usage-guide.md +++ b/docs/usage-guide.md @@ -1735,7 +1735,10 @@ and sessions started under a symlinked path. The key is per event: hooks that run outside the project (for example in a worktree removed before the session ends) report their events to the scope they ran in. Events recorded by an earlier release carry no key: a project reports those whose directory lies -under its root, the user scope reports none of them. A +under its root, the user scope reports none of them. Each scope also keeps its own snapshot of what it already +reported, so a session that moved into another project mid-session reaches both +teams with the part recorded in each; the first report after upgrading starts +from the snapshot every scope used to share, so nothing is reported twice. A target removes its usage events only after it confirms success; failed pushes preserve them. The affected sync locks remain held until reporting finishes, preventing another pull from racing the report. diff --git a/docs/usage-guide.zh-CN.md b/docs/usage-guide.zh-CN.md index 88a2c8f6b..3bcb3e7eb 100644 --- a/docs/usage-guide.zh-CN.md +++ b/docs/usage-guide.zh-CN.md @@ -1674,7 +1674,9 @@ Pull 对整批统计上报最多等待 5 秒，之后继续其他工作，上报 只上报在其中记录的会话：user scope 的 pull 不再上报项目的会话，项目会上报自己的 Copilot 会话以及从软链接路径启动的会话。该键按事件记录：在项目之外运行的 hook （例如会话结束前已删除的 worktree 中）会把事件上报给其所在的 scope。旧版本记录的事件没有该键：项目上报目录 -位于其根目录下的那些，user scope 一条也不上报。目标确认成功后才清理自己的使用事件， +位于其根目录下的那些，user scope 一条也不上报。每个 scope 还各自保存已上报快照， +因此会话中途切换到另一个项目时，两个团队各自收到在其中记录的那部分；升级后的首次上报 +从原先各 scope 共用的快照开始，不会重复上报。目标确认成功后才清理自己的使用事件， 推送失败会保留事件。上报完成前继续持有相关同步锁， 避免另一次 Pull 与尚未完成的上报竞争。 diff --git a/src/__tests__/dashboard-report-scope.test.ts b/src/__tests__/dashboard-report-scope.test.ts index a32456926..0bed92af6 100644 --- a/src/__tests__/dashboard-report-scope.test.ts +++ b/src/__tests__/dashboard-report-scope.test.ts @@ -92,12 +92,16 @@ async function setup(): Promise<{ root: string; user: LocalConfig; project: Loca return { root, user, project }; } -/** Report the way pull does, and return the sessions that scope's stats now hold. */ -async function reportedSessions(config: LocalConfig): Promise { +/** Report the way pull does, and return the stats that scope has pushed so far. */ +async function report(config: LocalConfig): Promise { await reportUsageToTeam(config.repo.localPath, config.username, { skipTruncate: true, selfConfig: config }); const statsPath = path.join(path.dirname(config.repo.localPath), 'reports-wt', 'stats', `${config.username}.yaml`); - if (!fs.existsSync(statsPath)) return 0; - const stats: unknown = YAML.parse(fs.readFileSync(statsPath, 'utf-8')); + return fs.existsSync(statsPath) ? YAML.parse(fs.readFileSync(statsPath, 'utf-8')) : null; +} + +/** Report the way pull does, and return the sessions that scope's stats now hold. */ +async function reportedSessions(config: LocalConfig): Promise { + const stats = await report(config); const daily = stats && typeof stats === 'object' && 'daily' in stats && stats.daily && typeof stats.daily === 'object' ? stats.daily : {}; return Object.values(daily).reduce((sum: number, day: unknown) => sum + (day && typeof day === 'object' && 'sessionsEnded' in day && typeof day.sessionsEnded === 'number' ? day.sessionsEnded : 0), 0); @@ -173,3 +177,115 @@ describe('each scope reports only the dashboard sessions recorded in it (#785)', expect(await reportedSessions(project)).toBe(2); }); }); + +describe('each scope keeps its own reported snapshot (#786)', () => { + /** The prompts a scope's stats hold, 0 before its first push. */ + async function reportedPrompts(config: LocalConfig): Promise { + const stats = await report(config); + return stats && typeof stats === 'object' && 'prompts' in stats && typeof stats.prompts === 'number' ? stats.prompts : 0; + } + + const shared = (name: string) => path.join(teamaiHome(), 'dashboard', `reported-${name}.json`); + const SNAPSHOTS = ['interventions', 'prompt-tokens', 'daily-sessions']; + + /** The shared snapshots as a release before #786 left them, with `prompts` reported per session. */ + function writeSharedSnapshots(prompts: Record, date: string): void { + const entries = (value: (n: number) => unknown) => + Object.fromEntries(Object.entries(prompts).map(([sid, n]) => [sid, value(n)])); + const values: Record = { + interventions: entries(() => ({ interrupt: 0, toolReject: 0, correction: 0 })), + 'prompt-tokens': entries((n) => ({ prompts: n, tokens: { input: 0, output: 0, cacheRead: 0, cacheCreation: 0 } })), + // Longer than any fixture session, so no duration is left to report. + 'daily-sessions': entries((n) => ({ date, prompts: n, durationMs: 3_600_000, succeeded: 1, corrected: 0 })), + }; + fs.mkdirSync(path.join(teamaiHome(), 'dashboard'), { recursive: true }); + for (const name of SNAPSHOTS) fs.writeFileSync(shared(name), JSON.stringify(values[name])); + } + + async function prompts(sessionId: string, cwd: string, count: number): Promise { + for (let i = 0; i < count; i++) { + await hook('prompt-submit', 'claude', { session_id: sessionId, cwd, hook_event_name: 'UserPromptSubmit', prompt: `p${i}` }); + } + } + + it('a session split across the user scope and a project reaches both teams with its own counts', async () => { + const { root, user, project } = await setup(); + const elsewhere = path.join(tmp, 'elsewhere'); + fs.mkdirSync(elsewhere); + await hook('session-start', 'claude', { session_id: 'split', cwd: elsewhere, hook_event_name: 'SessionStart' }); + await prompts('split', elsewhere, 3); + await prompts('split', root, 2); // `cd` into the project mid-session + await hook('stop', 'claude', { session_id: 'split', cwd: root, hook_event_name: 'Stop' }); + + expect(await reportedPrompts(user)).toBe(3); + expect(await reportedPrompts(project)).toBe(2); + }); + + it('a session split across two projects reaches both teams with its own counts', async () => { + const { root, project } = await setup(); + const rootQ = path.join(tmp, 'project-q'); + fs.mkdirSync(rootQ); + execFileSync('git', ['init', '-q'], { cwd: rootQ }); + const dataHomeQ = await resolveProjectDataHome(rootQ); + fs.mkdirSync(path.join(dataHomeQ, 'team-repo'), { recursive: true }); + await saveLocalConfigForScope({ + repo: { localPath: path.join(dataHomeQ, 'team-repo'), remote: 'https://example.test/acme/team-q.git', kind: 'git' }, + username: 'tester', scope: 'project', projectRoot: rootQ, additionalRoles: [], dataHome: dataHomeQ, + }); + const projectQ = await resolveConfigForDir(rootQ); + if (!projectQ) throw new Error('fixture config Q did not resolve'); + await hook('session-start', 'claude', { session_id: 'split', cwd: root, hook_event_name: 'SessionStart' }); + await prompts('split', root, 3); + await prompts('split', rootQ, 2); + await hook('stop', 'claude', { session_id: 'split', cwd: rootQ, hook_event_name: 'Stop' }); + + expect(await reportedPrompts(project)).toBe(3); + expect(await reportedPrompts(projectQ)).toBe(2); + }); + + it('the first report after the upgrade sends nothing a shared snapshot already reported', async () => { + const { root, user, project } = await setup(); + const elsewhere = path.join(tmp, 'elsewhere'); + fs.mkdirSync(elsewhere); + await session('claude', { session_id: 'old-p', cwd: root }); + await session('claude', { session_id: 'old-u', cwd: elsewhere }); + writeSharedSnapshots({ 'old-p': 1, 'old-u': 1 }, new Date().toISOString().slice(0, 10)); + + expect(await report(user)).toBeNull(); + expect(await report(project)).toBeNull(); + }); + + it('once seeded, a scope reads and writes only its own snapshot, even after a rollback rewrites the shared one', async () => { + const { root, user, project } = await setup(); + const elsewhere = path.join(tmp, 'elsewhere'); + fs.mkdirSync(elsewhere); + const today = new Date().toISOString().slice(0, 10); + writeSharedSnapshots({ 'old-p': 1 }, today); + const before = SNAPSHOTS.map((name) => fs.readFileSync(shared(name), 'utf-8')); + + await session('claude', { session_id: 'new-p', cwd: root }); + expect(await reportedPrompts(project)).toBe(1); + await session('claude', { session_id: 'new-u', cwd: elsewhere }); + expect(await reportedPrompts(user)).toBe(1); + expect(SNAPSHOTS.map((name) => fs.readFileSync(shared(name), 'utf-8'))).toEqual(before); + + // An earlier release, after a rollback, records and reports two more + // sessions and writes the shared snapshots again. + await session('claude', { session_id: 'rollback-p', cwd: root }); + await session('claude', { session_id: 'rollback-u', cwd: elsewhere }); + writeSharedSnapshots({ 'old-p': 1, 'rollback-p': 1, 'rollback-u': 1 }, today); + + // Both scopes were seeded before the rollback and read only their own + // snapshot, so each reports its session again, as the ticket asks. + expect(await reportedPrompts(project)).toBe(2); + expect(await reportedPrompts(user)).toBe(2); + }); + + it('a scope first seeded after a rollback skips what the earlier release reported', async () => { + const { root, project } = await setup(); + await session('claude', { session_id: 'rollback-p', cwd: root }); + writeSharedSnapshots({ 'rollback-p': 1 }, new Date().toISOString().slice(0, 10)); + + expect(await report(project)).toBeNull(); + }); +}); diff --git a/src/__tests__/e2e/report-timeout.test.ts b/src/__tests__/e2e/report-timeout.test.ts index 705362783..08b03a8c6 100644 --- a/src/__tests__/e2e/report-timeout.test.ts +++ b/src/__tests__/e2e/report-timeout.test.ts @@ -87,7 +87,12 @@ function fixture(agent: keyof typeof agents, provider: string) { function stats() { return YAML.parse(git(['show', 'teamai-reports:stats/alice.yaml'], remote, env)); } - return { home, usage, usageLine, dashboard, seedEvents, receiver, pull, stats }; + /** The user scope's reported snapshot (#786), `{}` before the report writes one. */ + function snapshot(name: string): Record { + const p = path.join(dashboard, `user-reported-${name}.json`); + return fs.existsSync(p) ? JSON.parse(fs.readFileSync(p, 'utf8')) : {}; + } + return { home, usage, usageLine, seedEvents, receiver, pull, stats, snapshot }; } afterEach(() => { @@ -106,7 +111,7 @@ describe('real CLI report completion', () => { f.receiver('slow'); const output = await f.pull(() => { expect(fs.readFileSync(f.usage, 'utf8')).toBe(f.usageLine); - expect(fs.existsSync(path.join(f.dashboard, 'reported-prompt-tokens.json'))).toBe(false); + expect(f.snapshot('prompt-tokens').s1).toBeUndefined(); expect(fs.existsSync(path.join(f.home, '.teamai/.sync-lock'))).toBe(true); // An event arriving during the push must survive cleanup of the batch. fs.appendFileSync(f.usage, f.usageLine); @@ -115,7 +120,7 @@ describe('real CLI report completion', () => { expect(f.stats().skills.review.count).toBe(1); expect(fs.readFileSync(f.usage, 'utf8')).toBe(f.usageLine); for (const name of ['interventions', 'prompt-tokens', 'daily-sessions']) { - expect(JSON.parse(fs.readFileSync(path.join(f.dashboard, `reported-${name}.json`), 'utf8')).s1).toBeDefined(); + expect(f.snapshot(name).s1).toBeDefined(); } expect(fs.existsSync(path.join(f.home, '.teamai/.sync-lock'))).toBe(false); f.receiver('normal'); @@ -139,7 +144,7 @@ describe('real CLI report completion', () => { f.receiver('reject'); await f.pull(); expect(fs.readFileSync(f.usage, 'utf8')).toBe(f.usageLine); - expect(fs.existsSync(path.join(f.dashboard, 'reported-prompt-tokens.json'))).toBe(false); + expect(f.snapshot('prompt-tokens').s1).toBeUndefined(); f.receiver('normal'); await f.pull(); const stats = f.stats(); diff --git a/src/__tests__/team-push-interventions.test.ts b/src/__tests__/team-push-interventions.test.ts index 892b7b5d6..39ff73e05 100644 --- a/src/__tests__/team-push-interventions.test.ts +++ b/src/__tests__/team-push-interventions.test.ts @@ -57,6 +57,15 @@ function reportsStatsPath(): string { return path.join(tmpDir, 'reports-wt', 'stats', 'me.yaml'); } +/** + * A reported snapshot as the report left it, `{}` when it wrote none: the user + * scope's own (#786), or the shared one for a caller without a scope config. + */ +function reportedSnapshot(name: string, scoped = true): Record { + const p = path.join(tmpDir, '.teamai', 'dashboard', `${scoped ? 'user-' : ''}reported-${name}.json`); + return fs.existsSync(p) ? JSON.parse(fs.readFileSync(p, 'utf-8')) : {}; +} + beforeEach(async () => { tmpDir = fs.mkdtempSync(path.join(os.tmpdir(), 'teamai-tp-iv-')); originalHome = process.env.HOME ?? ''; @@ -127,8 +136,7 @@ describe('reportUsageToTeam — intervention reporting', () => { await vi.advanceTimersByTimeAsync(5000); await timeout; expect(fs.readFileSync(usagePath, 'utf-8')).toContain('review'); - const dashboard = path.join(tmpDir, '.teamai', 'dashboard'); - expect(fs.existsSync(path.join(dashboard, 'reported-prompt-tokens.json'))).toBe(false); + expect(reportedSnapshot('prompt-tokens', backend === 'reports').slow).toBeUndefined(); finish(true); expect(await operation).toBe(true); @@ -136,7 +144,7 @@ describe('reportUsageToTeam — intervention reporting', () => { // so it leaves the user-scope file for the scope that owns it. expect(fs.readFileSync(usagePath, 'utf-8')).toBe(backend === 'reports' ? '' : seeded); for (const name of ['interventions', 'prompt-tokens', 'daily-sessions']) { - expect(JSON.parse(fs.readFileSync(path.join(dashboard, `reported-${name}.json`), 'utf-8')).slow).toBeDefined(); + expect(reportedSnapshot(name, backend === 'reports').slow).toBeDefined(); } const statsPath = backend === 'reports' ? reportsStatsPath() : path.join(repoDir, 'stats', 'me.yaml'); const before = fs.readFileSync(statsPath, 'utf-8'); @@ -152,7 +160,7 @@ describe('reportUsageToTeam — intervention reporting', () => { expect(await reportUsageToTeam(repoDir, 'me', { selfConfig: gitConfig() })).toBe(false); expect(fs.readFileSync(usagePath, 'utf-8')).toBe(before); for (const name of ['interventions', 'prompt-tokens', 'daily-sessions']) { - expect(fs.existsSync(path.join(tmpDir, '.teamai', 'dashboard', `reported-${name}.json`))).toBe(false); + expect(reportedSnapshot(name).slow).toBeUndefined(); } expect(await reportUsageToTeam(repoDir, 'me', { selfConfig: gitConfig() })).toBe(true); const stats = YAML.parse(fs.readFileSync(reportsStatsPath(), 'utf-8')); @@ -181,12 +189,10 @@ describe('reportUsageToTeam — intervention reporting', () => { expect(reportsMocks.updateReports).toHaveBeenCalledTimes(1); // reported snapshot persisted so a second run reports nothing new - const reportedPath = path.join(tmpDir, '.teamai', 'dashboard', 'reported-interventions.json'); - expect(JSON.parse(fs.readFileSync(reportedPath, 'utf-8'))).toEqual({ + expect(reportedSnapshot('interventions')).toEqual({ s1: { interrupt: 2, toolReject: 1, correction: 0 }, }); - const dailyPath = path.join(tmpDir, '.teamai', 'dashboard', 'reported-daily-sessions.json'); - expect(JSON.parse(fs.readFileSync(dailyPath, 'utf-8')).s1.date).toBe(ts.slice(0, 10)); + expect(reportedSnapshot('daily-sessions').s1).toMatchObject({ date: ts.slice(0, 10) }); reportsMocks.updateReports.mockClear(); await reportUsageToTeam(repoDir, 'me', { selfConfig: gitConfig() }); diff --git a/src/team-push.ts b/src/team-push.ts index 721c2486e..c0b8201eb 100644 --- a/src/team-push.ts +++ b/src/team-push.ts @@ -15,7 +15,7 @@ import { import { writeFile, readFileSafe, ensureDir, pathExists, readJson, writeJson } from './utils/fs.js'; import { log } from './utils/logger.js'; import type { UserStats, UserInterventionStats, SessionMetrics, TokenUsage, DashboardEvent, LocalConfig } from './types.js'; -import { getUserVotesDir, getDataHome, emptyTokenUsage, addTokenUsage, usesBranchWorktree } from './types.js'; +import { getUserVotesDir, getDataHome, getTeamaiHomeDir, emptyTokenUsage, addTokenUsage, usesBranchWorktree } from './types.js'; import { getUserHome } from './utils/home.js'; import { aggregateDailySessions, @@ -137,19 +137,58 @@ export function mergeStats( // double-counts a session, since we only add the positive change since last report. // -/** Path to the local reported-interventions snapshot (evaluated at call time for tests). */ -function getReportedInterventionsPath(): string { - return path.join(getUserHome(), '.teamai', 'dashboard', 'reported-interventions.json'); +// ─── Reported snapshots, one set per scope (#786) ────── +// +// /dashboard/reported-.json project scope +// ~/.teamai/dashboard/user-reported-.json user scope +// ~/.teamai/dashboard/reported-.json shared, written before #786 +// +// A session can record events in two scopes (a `cd` mid-session), so each +// scope compares against what it reported itself. The first time a scope needs +// a snapshot, it copies the shared one, so nothing an earlier release reported +// is sent again; after that only its own file is read. No scope writes the +// shared file any more, only an earlier release after a rollback (and a caller +// without a scope config, which reads the whole log and reports into it). +// + +type ReportedSnapshotName = 'interventions' | 'prompt-tokens' | 'daily-sessions'; + +/** The machine-level snapshot every scope shared before #786 (evaluated at call time for tests). */ +function sharedSnapshotPath(name: ReportedSnapshotName): string { + return path.join(getTeamaiHomeDir(), 'dashboard', `reported-${name}.json`); +} + +/** A scope's own snapshot. The user scope's data home holds the shared one, hence its prefix. */ +function scopeSnapshotPath(name: ReportedSnapshotName, config: LocalConfig | undefined): string { + if (!config) return sharedSnapshotPath(name); + const dataHome = getDataHome(config); + const file = `reported-${name}.json`; + if (path.resolve(dataHome) !== path.resolve(getTeamaiHomeDir())) return path.join(dataHome, 'dashboard', file); + return path.join(dataHome, 'dashboard', `user-${file}`); +} + +/** A scope's snapshot, seeded from the shared one when the scope has none yet. */ +async function readSnapshot(name: ReportedSnapshotName, config: LocalConfig | undefined): Promise { + const own = scopeSnapshotPath(name, config); + if (!config || await pathExists(own)) return readJson(own); + const seed = await readJson(sharedSnapshotPath(name)); + try { + await writeJson(own, seed ?? {}); + } catch (e) { + // Seeded again next time: the shared file is not written any more. + log.debug(`Could not seed ${own}: ${(e as Error).message}`); + } + return seed; } -async function readReportedInterventions(): Promise { - const parsed = await readJson(getReportedInterventionsPath()); +async function readReportedInterventions(config: LocalConfig | undefined): Promise { + const parsed = await readSnapshot('interventions', config); return parsed && typeof parsed === 'object' ? parsed : {}; } -async function writeReportedInterventions(data: ReportedInterventions): Promise { +async function writeReportedInterventions(data: ReportedInterventions, config: LocalConfig | undefined): Promise { try { - await writeJson(getReportedInterventionsPath(), data); + await writeJson(scopeSnapshotPath('interventions', config), data); } catch (e) { log.error(`Failed to persist reported interventions: ${(e as Error).message}`); } @@ -208,27 +247,14 @@ function hasInterventionDelta(d: UserInterventionStats): boolean { // Separate snapshot from interventions so each metric stays independently idempotent. // -/** Path to the local prompt/token reported snapshot (evaluated at call time for tests). */ -function getReportedPromptTokensPath(): string { - return path.join(getUserHome(), '.teamai', 'dashboard', 'reported-prompt-tokens.json'); -} - -async function readReportedPromptTokens(): Promise { - try { - const content = await readFileSafe(getReportedPromptTokensPath()); - if (!content) return {}; - const parsed = JSON.parse(content); - return parsed && typeof parsed === 'object' ? parsed : {}; - } catch { - return {}; - } +async function readReportedPromptTokens(config: LocalConfig | undefined): Promise { + const parsed = await readSnapshot('prompt-tokens', config); + return parsed && typeof parsed === 'object' ? parsed : {}; } -async function writeReportedPromptTokens(data: ReportedPromptTokens): Promise { +async function writeReportedPromptTokens(data: ReportedPromptTokens, config: LocalConfig | undefined): Promise { try { - const p = getReportedPromptTokensPath(); - await ensureDir(path.dirname(p)); - await writeFile(p, JSON.stringify(data)); + await writeJson(scopeSnapshotPath('prompt-tokens', config), data); } catch (e) { log.error(`Failed to persist reported prompt/token snapshot: ${(e as Error).message}`); } @@ -285,16 +311,12 @@ function hasPromptTokenDelta(d: PromptTokenDelta): boolean { || d.tokens.cacheRead > 0 || d.tokens.cacheCreation > 0; } -function getReportedDailySessionsPath(): string { - return path.join(getUserHome(), '.teamai', 'dashboard', 'reported-daily-sessions.json'); -} - -async function readReportedDailySessions(): Promise { - return (await readJson(getReportedDailySessionsPath())) ?? {}; +async function readReportedDailySessions(config: LocalConfig | undefined): Promise { + return (await readSnapshot('daily-sessions', config)) ?? {}; } -async function writeReportedDailySessions(data: ReportedDailySessions): Promise { - await writeJson(getReportedDailySessionsPath(), data); +async function writeReportedDailySessions(data: ReportedDailySessions, config: LocalConfig | undefined): Promise { + await writeJson(scopeSnapshotPath('daily-sessions', config), data); } function hasDailyDelta(delta: ReturnType['delta']): boolean { @@ -431,18 +453,18 @@ export async function reportUsageToTeam( const currentInterventions = new Map( [...metrics].map(([sid, m]) => [sid, { interrupt: m.interrupt, toolReject: m.toolReject, correction: m.correction }]), ); - const reportedInterventions = await readReportedInterventions(); + const reportedInterventions = await readReportedInterventions(reportsConfig); const { delta: interventionDelta, nextReported } = computeInterventionDelta( currentInterventions, reportedInterventions, ); - const reportedPromptTokens = await readReportedPromptTokens(); + const reportedPromptTokens = await readReportedPromptTokens(reportsConfig); const { delta: promptTokenDelta, nextReported: nextReportedPromptTokens } = computePromptTokenDelta( metrics, reportedPromptTokens, ); - const reportedDailySessions = await readReportedDailySessions(); + const reportedDailySessions = await readReportedDailySessions(reportsConfig); const { delta: dailyDelta, nextReported: nextReportedDailySessions } = computeDailyStatsDelta( aggregateDailySessions(dashboardEvents), reportedDailySessions, @@ -598,18 +620,18 @@ export async function reportUsageToTeam( // Success — advance the reported snapshots so we don't re-count. // Merge (not overwrite) because each scope only touches its own sessions. if (hasInterventions) { - const existingIv = await readReportedInterventions(); - await writeReportedInterventions({ ...existingIv, ...nextReported }); + const existingIv = await readReportedInterventions(reportsConfig); + await writeReportedInterventions({ ...existingIv, ...nextReported }, reportsConfig); log.debug(`Reported intervention delta (${interventionDelta.sessions} new sessions) to team repo`); } if (hasPromptTokens) { - const existingPt = await readReportedPromptTokens(); - await writeReportedPromptTokens({ ...existingPt, ...nextReportedPromptTokens }); + const existingPt = await readReportedPromptTokens(reportsConfig); + await writeReportedPromptTokens({ ...existingPt, ...nextReportedPromptTokens }, reportsConfig); log.debug(`Reported prompt/token delta (${promptTokenDelta.prompts} prompts) to team repo`); } if (hasDaily) { - const existingDaily = await readReportedDailySessions(); - await writeReportedDailySessions({ ...existingDaily, ...nextReportedDailySessions }); + const existingDaily = await readReportedDailySessions(reportsConfig); + await writeReportedDailySessions({ ...existingDaily, ...nextReportedDailySessions }, reportsConfig); log.debug(`Reported daily session trends (${Object.keys(dailyDelta).length} UTC day buckets) to team repo`); } if (!hasUsage && !hasInterventions && !hasPromptTokens && !hasDaily) { From bd48780c9cd5fd597e369ffd00bb6aacecfbe763 Mon Sep 17 00:00:00 2001 From: Saul Moro Date: Thu, 24 Sep 2026 13:03:34 +0200 Subject: [PATCH 06/41] fix(report): report each dashboard session once, from the scope it started in (#785, #786) A Stop carries the whole transcript's totals (prompts, tokens, interventions, request cost). Filtered per event, a session that moved into another scope mid-session was reported whole again by the scope holding the later Stop: 3 user-scope prompts then 2 in P reported 3 to the user team and 5 to P. Each session is now decided once, by its first keyed event, and reported whole by that scope. This replaces #786's "a split session reaches both teams with its part"; per-scope snapshots stay, so a session ID another scope already reported (Copilot's PID fallback) still counts as new. Unkeyed sessions from before the upgrade are decided by their first cwd. The user scope now takes those whose directory still exists and resolves to it (resolveConfigForDir, the dispatcher's rule) instead of dropping its whole backlog; no cwd, or one removed since, is still no scope's. The Copilot test also runs a payload without cwd from a hook in the project. --- CHANGELOG.md | 4 +- docs/designs/data-directory-layout.md | 18 +++--- docs/usage-guide.md | 19 +++--- docs/usage-guide.zh-CN.md | 11 ++-- src/__tests__/dashboard-report-scope.test.ts | 61 ++++++++++++++------ src/__tests__/scope-filter.test.ts | 21 +++++++ src/team-push.ts | 56 +++++++++++++----- 7 files changed, 136 insertions(+), 54 deletions(-) diff --git a/CHANGELOG.md b/CHANGELOG.md index 3b526b480..8c9ee7395 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -36,8 +36,8 @@ All notable changes to this project will be documented in this file. See [standa - A `manifest/roles.yaml` that exists but does not parse now fails the pull for that scope instead of warning and syncing with no role filter at all, for a member with no role as much as for one with a role. The same applies to `init` and `push`, which each fell back to a guess at the namespaces when any error came out of the loader. The legacy role migration skips with a warning instead of failing, so every command, `pull` included, still loads the config and can fetch the fixed manifest; until it can run, the member holds no role rather than every role, so hooks, MCP servers and env variables scoped by `roles:` reach them no more than skills do. For a member with no active project that fallback meant an unfiltered sync, so a broken manifest delivered every namespace it was written to gate. Only an absent manifest still means "this team does not use roles"; an unreadable or empty file is an error, as it now is for `manifest/projects.yaml` too. `push` stops at its scan for such a manifest (exit 2) even with `--role `, since the scan needs it to tell which namespaces are the member's. - `teamai members list` and `teamai projects members` read the roster registered before the reports switch, so a team upgrading past the orphan-branch split no longer sees "No team members registered" while its `members/` still lives on the default branch. The default-branch copy becomes a read-only inherited root, the way learnings' already was: listed in union with the `teamai-reports` copy, with the branch copy winning when the same file exists on both; nothing is copied or deleted, and a cold `members list` still does not publish the reports branch. Member registration merges against the inherited copy too, so a re-init keeps the original `registeredAt` and projects. Fixes [#735](https://github.com/Tencent/teamai-cli/issues/735). - Cache GC now rejects partial integers such as `12abc`, decimals, zero and unsafe integers for `--max-bytes` and `--stale-days` before deleting anything. An invalid `TEAMAI_CACHE_MAX_BYTES` value falls back to the default 5 GB limit instead of using a numeric prefix. -- Each scope reports only the dashboard sessions recorded in it. Every scope read one machine-wide event log and picked its sessions out by the `cwd` they started in, so a user-scope pull reported every project's sessions to the user team, a project never reported its Copilot sessions (teamai does not record Copilot's `cwd`), and a session started under a symlinked path (or `/tmp` for `/private/tmp` on macOS) never matched its project. Each event now records a key (a hash, so no path is stored) of the data directory of the scope `hook-dispatch` resolved, and a report keeps only its own scope's. A session in a directory that resolves to no project (a subdirectory of a non-git project, a git submodule or nested clone) is the user scope's, as for skill usage. Events recorded by an earlier release go to the project whose root holds their `cwd`, never to the user scope. The usage guide explains how to remove by hand a skill an earlier release reported into `stats/.yaml` from another project (for [#785](https://github.com/Tencent/teamai-cli/issues/785)). -- A session whose events belong to two scopes is reported to both teams with the part recorded in each. The snapshots of what was already reported were machine-wide, so after a `cd` into another project mid-session, the second scope to report compared its part with the first scope's totals and sent nothing. Each scope now keeps its own (`/dashboard/reported-*.json`, and `~/.teamai/dashboard/user-reported-*.json` for the user scope), seeded the first time from the shared `~/.teamai/dashboard/reported-*.json`, so the first report after the upgrade sends nothing already reported. The shared file is no longer written; after a rollback an earlier release writes it again, and only a scope not yet seeded reads it (for [#786](https://github.com/Tencent/teamai-cli/issues/786)). +- Each scope reports only the dashboard sessions recorded in it. Every scope read one machine-wide event log and picked its sessions out by the `cwd` they started in, so a user-scope pull reported every project's sessions to the user team, a project never reported its Copilot sessions (teamai does not record Copilot's `cwd`), and a session started under a symlinked path (or `/tmp` for `/private/tmp` on macOS) never matched its project. Each event now records a key (a hash, so no path is stored) of the data directory of the scope `hook-dispatch` resolved, and a report keeps only its own scope's. A session in a directory that resolves to no project (a subdirectory of a non-git project, a git submodule or nested clone) is the user scope's, as for skill usage. A session is reported once, whole, by the scope it started in, even after a `cd` into another project: its Stop carries the whole transcript's totals, so reporting its later part elsewhere would count them twice. Events recorded by an earlier release go to the project whose root holds their `cwd`, or to the user scope when that directory still exists and belongs to no project; events with no `cwd`, or one removed since, are reported by no one. The usage guide explains how to remove by hand a skill an earlier release reported into `stats/.yaml` from another project (for [#785](https://github.com/Tencent/teamai-cli/issues/785)). +- Each scope keeps its own snapshots of the dashboard sessions it already reported. They were machine-wide and keyed by session ID, so a session ID reused in another scope (Copilot's fallback ID is the parent process ID) was taken as already reported there and sent nothing. Each scope now keeps its own (`/dashboard/reported-*.json`, and `~/.teamai/dashboard/user-reported-*.json` for the user scope), seeded the first time from the shared `~/.teamai/dashboard/reported-*.json`, so the first report after the upgrade sends nothing already reported. The shared file is no longer written; after a rollback an earlier release writes it again, and only a scope not yet seeded reads it (for [#786](https://github.com/Tencent/teamai-cli/issues/786)). - Usage reporting scopes sessions by path on Windows too. The project/user scope filter compared an event's `cwd` against `projectRoot` with a hard-coded `/` separator, so on Windows only a session started in the project root itself matched: every session started in a subdirectory was dropped from the project team's report and counted in the user scope's instead, which is the isolation the usage guide promises. Windows paths are also compared case-insensitively, so a drive letter or a directory name spelled with different case in the two sources no longer leaks a project session into the user scope. POSIX paths keep their own rules: case-sensitive, and a `\` in a filename stays part of the name. - `teamai tags subscribe` and `teamai tags unsubscribe` now invalidate the pull revision cache, as `teamai skill exclude` already does, so the next `teamai pull` applies the new subscriptions instead of reporting "Already synced" when the team repo has not changed. - `teamai pull` now deletes a tombstoned agent under all three render extensions, so the Codex `.toml` and Kiro `.json` copies of a removed agent no longer survive on other machines. The cleanup also runs when the team repo rev is unchanged, so an upgrade reaches machines that already pulled the tombstone with an older CLI. `teamai remove agents ` also honours `enabledAgents` and no longer deletes from excluded tools. Fixes [#576](https://github.com/Tencent/teamai-cli/issues/576). diff --git a/docs/designs/data-directory-layout.md b/docs/designs/data-directory-layout.md index d2912d2e9..33ed90b17 100644 --- a/docs/designs/data-directory-layout.md +++ b/docs/designs/data-directory-layout.md @@ -316,19 +316,23 @@ read. The dashboard stays an A2 singleton: `teamai dashboard`, `stats --by-repo` `session save` and the contribute check read across scopes. Each event instead carries `dataHomeKey`, a hash of the realpath'd `getDataHome()` of the scope the hook resolved (#785; a hash, so a Copilot event still stores no path), and a -scope's report keeps only its own. An event written before that field existed is -attributed by its `cwd`, realpath'd, to the project whose root holds it, never to -the user scope. The snapshots of what was already reported are per scope too -(#786), because a session can record events in two scopes (a `cd` mid-session): +scope's report keeps the sessions whose first keyed event is its own, whole: a +Stop carries the whole transcript's totals, so a session that moved scope (a `cd` +mid-session) is reported once, where it started. A session written before that +field existed is attributed by its first `cwd`: to the project whose root holds +it (realpath'd), or to the user scope when that directory still exists and +`resolveConfigForDir` resolves it to the user scope; no `cwd`, or one removed +since, is no scope's. The snapshots of what was already reported are per scope +too (#786), because a session ID can recur in another scope (Copilot's fallback +ID is the parent PID): `/dashboard/reported-*.json`, and `~/.teamai/dashboard/user-reported-*.json` for the user scope. The first time a scope needs one it copies the shared `~/.teamai/dashboard/reported-*.json`, so nothing reported before the upgrade is sent again; after that it reads only its own. The shared file is no longer written, except by an earlier release after a rollback, so every scope seeds from what the machine had reported by then, never from another scope's later report. -The seed holds a session's whole total, so a session still running at the upgrade -that later records in a second scope reports nothing there until that scope's -part exceeds it. +The seed holds a session's whole total, so a session still running at the +upgrade goes on from the reported total, as before. **`anchor` on save.** Previously only migration wrote a partition's `anchor` reverse-lookup file, so freshly-init'd partitions had none. `saveLocalConfigForScope` diff --git a/docs/usage-guide.md b/docs/usage-guide.md index f5af28531..453d66645 100644 --- a/docs/usage-guide.md +++ b/docs/usage-guide.md @@ -1731,14 +1731,17 @@ Dashboard sessions stay in one machine-wide `~/.teamai/dashboard/events.jsonl`, but each event records a key of its scope's data home (a hash, not the path), so a scope reports only the sessions recorded in it: a user-scope pull no longer reports a project's sessions, and a project reports its Copilot sessions -and sessions started under a symlinked path. The key is per event: hooks that -run outside the project (for example in a worktree removed before the session -ends) report their events to the scope they ran in. Events recorded by an -earlier release carry no key: a project reports those whose directory lies -under its root, the user scope reports none of them. Each scope also keeps its own snapshot of what it already -reported, so a session that moved into another project mid-session reaches both -teams with the part recorded in each; the first report after upgrading starts -from the snapshot every scope used to share, so nothing is reported twice. A +and sessions started under a symlinked path. A session is reported once, whole, +by the scope it started in, even if it later moves into another project: its +Stop carries the whole transcript's totals, so a second scope would count them +again. Events recorded by an earlier release carry no key: a project reports +those whose directory lies under its root, and the user scope those whose +directory still exists and belongs to no project; events with no directory, or +one removed since, are reported by no one. Each scope also keeps its own +snapshot of what it already reported, so a session ID another scope already +reported (Copilot reuses its fallback IDs) still counts as new here; the first +report after upgrading starts from the snapshot every scope used to share, so +nothing is reported twice. A target removes its usage events only after it confirms success; failed pushes preserve them. The affected sync locks remain held until reporting finishes, preventing another pull from racing the report. diff --git a/docs/usage-guide.zh-CN.md b/docs/usage-guide.zh-CN.md index 3bcb3e7eb..23bdd987c 100644 --- a/docs/usage-guide.zh-CN.md +++ b/docs/usage-guide.zh-CN.md @@ -1672,11 +1672,12 @@ Pull 对整批统计上报最多等待 5 秒，之后继续其他工作，上报 自己的使用；未配置 teamai 的目录不记录。Dashboard 会话仍写入整机共用的 `~/.teamai/dashboard/events.jsonl`，但每条事件都记下所属 scope 数据目录（data home）的键（哈希值，不是路径），因此每个 scope 只上报在其中记录的会话：user scope 的 pull 不再上报项目的会话，项目会上报自己的 -Copilot 会话以及从软链接路径启动的会话。该键按事件记录：在项目之外运行的 hook -（例如会话结束前已删除的 worktree 中）会把事件上报给其所在的 scope。旧版本记录的事件没有该键：项目上报目录 -位于其根目录下的那些，user scope 一条也不上报。每个 scope 还各自保存已上报快照， -因此会话中途切换到另一个项目时，两个团队各自收到在其中记录的那部分；升级后的首次上报 -从原先各 scope 共用的快照开始，不会重复上报。目标确认成功后才清理自己的使用事件， +Copilot 会话以及从软链接路径启动的会话。每个会话只上报一次，整体归属其开始时所在的 +scope，即使之后切换到另一个项目：它的 Stop 带有整份 transcript 的累计值，第二个 scope +会重复计算。旧版本记录的事件没有该键：项目上报目录位于其根目录下的那些，user scope +上报目录仍存在且不属于任何项目的那些；没有目录或目录已删除的事件不由任何 scope 上报。 +每个 scope 还各自保存已上报快照，因此其他 scope 已上报过的会话 ID（Copilot 会复用其回退 +ID）在这里仍算新会话；升级后的首次上报从原先各 scope 共用的快照开始，不会重复上报。目标确认成功后才清理自己的使用事件， 推送失败会保留事件。上报完成前继续持有相关同步锁， 避免另一次 Pull 与尚未完成的上报竞争。 diff --git a/src/__tests__/dashboard-report-scope.test.ts b/src/__tests__/dashboard-report-scope.test.ts index 0bed92af6..dfaa0c4cb 100644 --- a/src/__tests__/dashboard-report-scope.test.ts +++ b/src/__tests__/dashboard-report-scope.test.ts @@ -126,9 +126,13 @@ describe('each scope reports only the dashboard sessions recorded in it (#785)', expect(await reportedSessions(user)).toBe(1); }); - it('a Copilot session, which records no cwd, is reported by its project', async () => { + it.each([ + ['names its cwd', true], + ['sends no cwd, from a hook running in the project', false], + ])('a Copilot session, whose events record no cwd, is reported by its project: payload %s', async (_, withCwd) => { const { root, user, project } = await setup(); - await session('copilot', { session_id: 'copilot-p', cwd: root }); + if (!withCwd) process.chdir(root); + await session('copilot', withCwd ? { session_id: 'copilot-p', cwd: root } : { session_id: 'copilot-p' }); expect(await reportedSessions(user)).toBe(0); expect(await reportedSessions(project)).toBe(1); @@ -155,10 +159,12 @@ describe('each scope reports only the dashboard sessions recorded in it (#785)', expect(await reportedSessions(project)).toBe(1); }); - it('events recorded before sessions carried a data home go to their project, never to the user scope', async () => { + it('events recorded before sessions carried a data home go to the scope their cwd resolves to now', async () => { const { root, user, project } = await setup(); const link = path.join(tmp, 'link-p'); fs.symlinkSync(root, link, 'dir'); + fs.mkdirSync(path.join(root, 'src')); + fs.mkdirSync(path.join(tmp, 'elsewhere')); const timestamp = new Date().toISOString(); const old = (sessionId: string, cwd: string | undefined) => [ { type: 'session_start', timestamp, sessionId, tool: 'claude', cwd }, @@ -170,10 +176,12 @@ describe('each scope reports only the dashboard sessions recorded in it (#785)', ...old('old-in-p', path.join(root, 'src')), ...old('old-via-link', link), ...old('old-elsewhere', path.join(tmp, 'elsewhere')), + // Nothing can tell whose these were, so no scope reports them. + ...old('old-gone', path.join(tmp, 'deleted-worktree')), ...old('old-no-cwd', undefined), ].map((e) => JSON.stringify(e)).join('\n') + '\n'); - expect(await reportedSessions(user)).toBe(0); + expect(await reportedSessions(user)).toBe(1); expect(await reportedSessions(project)).toBe(2); }); }); @@ -208,20 +216,23 @@ describe('each scope keeps its own reported snapshot (#786)', () => { } } - it('a session split across the user scope and a project reaches both teams with its own counts', async () => { + // A Stop carries the whole transcript's totals, so a session is reported once, + // whole, by the scope it started in; split per event, the later scope would + // count the earlier scope's part again. + it('a session that moves from the user scope into a project is reported once, by the user scope', async () => { const { root, user, project } = await setup(); const elsewhere = path.join(tmp, 'elsewhere'); fs.mkdirSync(elsewhere); - await hook('session-start', 'claude', { session_id: 'split', cwd: elsewhere, hook_event_name: 'SessionStart' }); - await prompts('split', elsewhere, 3); - await prompts('split', root, 2); // `cd` into the project mid-session - await hook('stop', 'claude', { session_id: 'split', cwd: root, hook_event_name: 'Stop' }); + await hook('session-start', 'claude', { session_id: 'moved', cwd: elsewhere, hook_event_name: 'SessionStart' }); + await prompts('moved', elsewhere, 3); + await prompts('moved', root, 2); // `cd` into the project mid-session + await hook('stop', 'claude', { session_id: 'moved', cwd: root, hook_event_name: 'Stop' }); - expect(await reportedPrompts(user)).toBe(3); - expect(await reportedPrompts(project)).toBe(2); + expect(await reportedPrompts(project)).toBe(0); + expect(await reportedPrompts(user)).toBe(5); }); - it('a session split across two projects reaches both teams with its own counts', async () => { + it('a session that moves from one project into another is reported once, by the first', async () => { const { root, project } = await setup(); const rootQ = path.join(tmp, 'project-q'); fs.mkdirSync(rootQ); @@ -234,13 +245,27 @@ describe('each scope keeps its own reported snapshot (#786)', () => { }); const projectQ = await resolveConfigForDir(rootQ); if (!projectQ) throw new Error('fixture config Q did not resolve'); - await hook('session-start', 'claude', { session_id: 'split', cwd: root, hook_event_name: 'SessionStart' }); - await prompts('split', root, 3); - await prompts('split', rootQ, 2); - await hook('stop', 'claude', { session_id: 'split', cwd: rootQ, hook_event_name: 'Stop' }); + await hook('session-start', 'claude', { session_id: 'moved', cwd: root, hook_event_name: 'SessionStart' }); + await prompts('moved', root, 3); + await prompts('moved', rootQ, 2); + await hook('stop', 'claude', { session_id: 'moved', cwd: rootQ, hook_event_name: 'Stop' }); + + expect(await reportedPrompts(projectQ)).toBe(0); + expect(await reportedPrompts(project)).toBe(5); + }); - expect(await reportedPrompts(project)).toBe(3); - expect(await reportedPrompts(projectQ)).toBe(2); + it('a session ID another scope already reported is a new session in this scope (Copilot PID fallback)', async () => { + const { root, user, project } = await setup(); + const elsewhere = path.join(tmp, 'elsewhere'); + fs.mkdirSync(elsewhere); + // No session ID: Copilot falls back to `pid-`, reused by the next session. + await session('copilot', { cwd: elsewhere }); + expect(await reportedSessions(user)).toBe(1); + // Compaction dropped the ended session; a later one in P gets the same ID. + fs.writeFileSync(path.join(teamaiHome(), 'dashboard', 'events.jsonl'), ''); + await session('copilot', { cwd: root }); + + expect(await reportedSessions(project)).toBe(1); }); it('the first report after the upgrade sends nothing a shared snapshot already reported', async () => { diff --git a/src/__tests__/scope-filter.test.ts b/src/__tests__/scope-filter.test.ts index cb65ba307..048e6aec5 100644 --- a/src/__tests__/scope-filter.test.ts +++ b/src/__tests__/scope-filter.test.ts @@ -59,6 +59,27 @@ describe('filterEventsByScope', () => { expect(await ids(await scoped(), userScope())).toEqual(['u1']); }); + it('a session goes whole to the scope of its first keyed event', async () => { + // A Stop carries the whole transcript's totals, so P must not count them again. + const moved = [ + makeEvent('/Users/jeff/other-work', 'm1'), + await scopedEvent('/Users/jeff/other-work', 'm1', '/home/jeff/.teamai'), + { ...(await scopedEvent('/Users/jeff/project-a', 'm1', '/home/jeff/.teamai/projects/p')), type: 'stop' as const, prompts: 5 }, + ]; + expect(await ids(moved, userScope())).toEqual(['m1', 'm1', 'm1']); + expect(await ids(moved, projectScope('/Users/jeff/project-a'))).toEqual([]); + }); + + it('a keyed event decides a session over an earlier unkeyed one', async () => { + // Recorded across the upgrade: the unkeyed event's cwd is P's, the key the user scope's. + const upgraded = [ + makeEvent('/Users/jeff/project-a', 'x1'), + await scopedEvent('/Users/jeff/project-a', 'x1', '/home/jeff/.teamai'), + ]; + expect(await ids(upgraded, projectScope('/Users/jeff/project-a'))).toEqual([]); + expect(await ids(upgraded, userScope())).toEqual(['x1', 'x1']); + }); + it('a project keeps what it recorded under its in-repo .teamai before moving to a partition', async () => { const legacy = [await scopedEvent(undefined, 'l1', '/Users/jeff/project-a/.teamai')]; expect(await ids(legacy, projectScope('/Users/jeff/project-a'))).toEqual(['l1']); diff --git a/src/team-push.ts b/src/team-push.ts index c0b8201eb..22859acd5 100644 --- a/src/team-push.ts +++ b/src/team-push.ts @@ -375,14 +375,20 @@ function isUnderScopeRoot(cwd: string, root: ScopeRoot): boolean { } /** - * The dashboard events a scope reports (#785): those recorded in it, keyed by - * its data home (`dataHomeKey`). A project also owns the key of its in-repo - * `.teamai`, where a hook recorded until migration moved the project to a - * partition. An event written before events carried a key belongs to the - * project whose root holds its cwd, never to the user scope. A cwd is raw as - * the host sent it (a symlinked checkout, macOS `/tmp` vs `/private/tmp`), and - * so is a non-git project's root, so both are realpath'd while they still exist. - * A caller without a scope config reads the whole log. + * The dashboard events a scope reports (#785): every event of the sessions + * recorded in it. A session is decided once, whole, by its first event keyed + * with a data home (`dataHomeKey`), because a Stop carries the whole + * transcript's totals: split per event, a session that moved into another + * scope would count there again the part recorded before the move. A project + * also owns the key of its in-repo `.teamai`, where a hook recorded until + * migration moved the project to a partition. A session written before events + * carried a key is decided by its first cwd: a project's when the root holds + * it, the user scope's when the directory still exists and resolves to the + * user scope now, the dispatcher's rule; one with no cwd, or a cwd gone since, + * is no scope's. A cwd is raw as the host sent it (a symlinked checkout, macOS + * `/tmp` vs `/private/tmp`), and so is a non-git project's root, so both are + * realpath'd while they still exist. A caller without a scope config reads the + * whole log. */ export async function filterEventsByScope( events: DashboardEvent[], @@ -398,19 +404,41 @@ export async function filterEventsByScope( } return real; }; - const keys = new Set([await dataHomeKey(getDataHome(config))]); + const ownKey = await dataHomeKey(getDataHome(config)); + const keys = new Set([ownKey]); if (config.projectRoot) { // Unless the project is rooted at HOME, where that is the user scope's. const legacy = await dataHomeKey(path.join(config.projectRoot, '.teamai')); if (legacy !== (await dataHomeKey(path.join(getUserHome(), '.teamai')))) keys.add(legacy); } const root = config.projectRoot ? scopeRoot(await realPath(config.projectRoot)) : undefined; - const kept = await Promise.all(events.map(async (e) => { - // The log is hand-editable: a key that is not a string counts as absent. - if (typeof e.dataHomeKey === 'string') return keys.has(e.dataHomeKey); - return !!root && !!e.cwd && isUnderScopeRoot(await realPath(e.cwd), root); + const { resolveConfigForDir } = await import('./config.js'); + const resolvesHere = new Map>(); + const ownsCwd = (cwd: string): Promise => { + let owns = resolvesHere.get(cwd); + if (!owns) { + owns = root ? realPath(cwd).then((real) => isUnderScopeRoot(real, root)) + : pathExists(cwd).then(async (exists) => { + const resolved = exists ? await resolveConfigForDir(cwd) : null; + return !!resolved && (await dataHomeKey(getDataHome(resolved))) === ownKey; + }); + resolvesHere.set(cwd, owns); + } + return owns; + }; + // The log is hand-editable: a key that is not a string counts as absent. + const keyed = (e: DashboardEvent | undefined): boolean => typeof e?.dataHomeKey === 'string'; + const deciding = new Map(); + for (const e of events) { + const current = deciding.get(e.sessionId); + if (keyed(e) ? !keyed(current) : !current && !!e.cwd) deciding.set(e.sessionId, e); + } + const owned = new Set(); + await Promise.all([...deciding].map(async ([sessionId, e]) => { + const mine = typeof e.dataHomeKey === 'string' ? keys.has(e.dataHomeKey) : !!e.cwd && await ownsCwd(e.cwd); + if (mine) owned.add(sessionId); })); - return events.filter((_, i) => kept[i]); + return events.filter((e) => owned.has(e.sessionId)); } /** From d91027db3cbf62ffe2e33342c1de963dc1f002b6 Mon Sep 17 00:00:00 2001 From: Saul Moro Date: Thu, 24 Sep 2026 13:37:07 +0200 Subject: [PATCH 07/41] fix(stats): read the scope's own dashboard filter and snapshots (#785, #786) `teamai stats` (#771) still called filterEventsByScope with the old { projectRoot, excludeProjectRoots } options, synchronously, after #795 made it async and keyed by the scope config, so main no longer type-checks and stats-scope fails. It also subtracted the shared reported-*.json, which no scope writes since #786. stats now filters with the config it resolved and subtracts that scope's own snapshots (readReportedInterventions / readReportedPromptTokens, the report's readers), so what it shows matches what pull reports. The user scope leaves a project's older sessions out, as the report does (#785); the stats-scope case that pinned "no exclusion in the user scope" now expects that. --- src/__tests__/stats-scope.test.ts | 14 +++++++----- src/stats.ts | 38 ++++++++----------------------- src/team-push.ts | 4 ++-- 3 files changed, 19 insertions(+), 37 deletions(-) diff --git a/src/__tests__/stats-scope.test.ts b/src/__tests__/stats-scope.test.ts index 630438707..da0a0c639 100644 --- a/src/__tests__/stats-scope.test.ts +++ b/src/__tests__/stats-scope.test.ts @@ -297,11 +297,13 @@ describe('showStats scope and idempotency', () => { expect(outputNumber(out, 'Conversation turns:')).toBe(1); }); - it('applies no project exclusion in the user scope when no project resolves', async () => { - // A user-scope run from a plain directory: detectProjectConfig() finds no - // project here, exactly as `pull` sees it from the same directory, so the - // report path passes no exclusion list either. The display side matches. + it('leaves a project\'s sessions out of the user scope even when no project resolves for the cwd', async () => { + // A user-scope run from a plain directory. The user scope reports only its + // own sessions (#785): an older unkeyed session goes to the scope its cwd + // resolves to now, so P's stays out and the one elsewhere counts. await seedUserScopeWithProject(); + fs.mkdirSync(DIRS.project, { recursive: true }); + fs.mkdirSync(DIRS.other, { recursive: true }); await appendEvents([ ...session('sess-1', DIRS.project), ...session('sess-2', DIRS.other), @@ -321,8 +323,8 @@ describe('showStats scope and idempotency', () => { fs.mkdirSync(plainDir, { recursive: true }); const out = await showStatsFromPlainDir(plainDir); - expect(outputNumber(out, 'Sessions:')).toBe(2); - expect(outputNumber(out, 'Conversation turns:')).toBe(2); + expect(outputNumber(out, 'Sessions:')).toBe(1); + expect(outputNumber(out, 'Conversation turns:')).toBe(1); }); it('still shows local sessions when the team stats file is missing', async () => { diff --git a/src/stats.ts b/src/stats.ts index 4eb4ae6cc..08cf563ed 100644 --- a/src/stats.ts +++ b/src/stats.ts @@ -4,11 +4,10 @@ import { readUsageEvents } from './usage-tracker.js'; import { readFileSafe } from './utils/fs.js'; import { resolveConfigForDir } from './config.js'; import { readEvents, aggregateSessionMetrics } from './dashboard-collector.js'; -import { getUserHome } from './utils/home.js'; import { totalTokens, addTokenUsage, emptyTokenUsage } from './types.js'; import { attributeByRepo, timeAnalytics, renderHourSparkline } from './session-analytics.js'; import { formatTokenCount } from './digest.js'; -import type { UsageEvent, UserStats, TokenUsage, SessionMetrics } from './types.js'; +import type { UsageEvent, UserStats, TokenUsage, SessionMetrics, LocalConfig } from './types.js'; interface SkillStats { name: string; @@ -150,21 +149,16 @@ function aggregateDashboardStats(metrics: Map): Aggregat * reported + unreported rather than reported + everything. * * The caller passes the scope's own metrics, already filtered the way the - * report path filters them. + * report path filters them, and the scope whose snapshots the report keeps (#786). */ async function unreportedDashboardStats( metrics: Map, + config: LocalConfig, ): Promise { - const { computeInterventionDelta, computePromptTokenDelta } = await import('./team-push.js'); - const { readJson } = await import('./utils/fs.js'); - const dashboardDir = path.join(getUserHome(), '.teamai', 'dashboard'); - - const interventions = (await readJson[1]>( - path.join(dashboardDir, 'reported-interventions.json'), - )) ?? {}; - const promptTokens = (await readJson[1]>( - path.join(dashboardDir, 'reported-prompt-tokens.json'), - )) ?? {}; + const { computeInterventionDelta, computePromptTokenDelta, readReportedInterventions, readReportedPromptTokens } = + await import('./team-push.js'); + const interventions = await readReportedInterventions(config); + const promptTokens = await readReportedPromptTokens(config); const interventionDelta = computeInterventionDelta( new Map([...metrics].map(([sid, m]) => [sid, { interrupt: m.interrupt, toolReject: m.toolReject, correction: m.correction }])), @@ -232,22 +226,8 @@ export async function showStats(options: ShowStatsOptions = {}): Promise { // and only the part of them not already reported (reported sessions stay in // events.jsonl until compaction, so counting the full local aggregate would // count each one twice and pull in other projects' sessions). - // - // The project root is resolved on its own, exactly as `pull` does: it reads - // `detectProjectConfig()`, never the projectRoot of the scope config, because - // a user-scope config carries no projectRoot at all (the field is attached - // only when a PROJECT config is detected). Same call, same directory, same - // answer as the report path. - const { detectProjectConfig } = await import('./config.js'); - const projectConfig = await detectProjectConfig(); - const projectRoot = config?.scope === 'project' ? config.projectRoot : projectConfig?.projectRoot; - const scopeFilter = projectRoot - ? (config?.scope === 'project' - ? { projectRoot } - : { excludeProjectRoots: [projectRoot] }) - : undefined; const { filterEventsByScope } = await import('./team-push.js'); - const scopedEvents = filterEventsByScope(await readEvents(), scopeFilter); + const scopedEvents = await filterEventsByScope(await readEvents(), config ?? undefined); const metricsMap = aggregateSessionMetrics(scopedEvents); // Only subtract what the team already holds. Two guards, because the local // snapshots are machine-global while the team file is per user: @@ -267,7 +247,7 @@ export async function showStats(options: ShowStatsOptions = {}): Promise { || totalTokens(reported.tokens ?? emptyTokenUsage()) > 0 ); const localDashboard = config && teamHasReported - ? await unreportedDashboardStats(metricsMap) + ? await unreportedDashboardStats(metricsMap, config) : aggregateDashboardStats(metricsMap); const dashboard = mergeDashboardAndReported(localDashboard, reported); const hasDashboardData = diff --git a/src/team-push.ts b/src/team-push.ts index 22859acd5..1b4f753a2 100644 --- a/src/team-push.ts +++ b/src/team-push.ts @@ -181,7 +181,7 @@ async function readSnapshot(name: ReportedSnapshotName, config: LocalConfig | return seed; } -async function readReportedInterventions(config: LocalConfig | undefined): Promise { +export async function readReportedInterventions(config: LocalConfig | undefined): Promise { const parsed = await readSnapshot('interventions', config); return parsed && typeof parsed === 'object' ? parsed : {}; } @@ -247,7 +247,7 @@ function hasInterventionDelta(d: UserInterventionStats): boolean { // Separate snapshot from interventions so each metric stays independently idempotent. // -async function readReportedPromptTokens(config: LocalConfig | undefined): Promise { +export async function readReportedPromptTokens(config: LocalConfig | undefined): Promise { const parsed = await readSnapshot('prompt-tokens', config); return parsed && typeof parsed === 'object' ? parsed : {}; } From dd3bc9f2ff6fd0f643030565789cdca6a363cc10 Mon Sep 17 00:00:00 2001 From: Saul Moro Date: Thu, 24 Sep 2026 14:46:04 +0200 Subject: [PATCH 08/41] fix(stats): address CI review (#785) A session ID now names one run up to its session_end or process_exit. A PID-fallback ID (Copilot) comes back for a later run, maybe in another scope, and the log keeps the ended run below the compaction threshold, so grouping by ID alone gave the later run to the first run's scope. Each run is still decided whole by its first keyed event. Events written by main since #795 record the data home as a path (`dataHome`); the report now keys them the way the writer derives `dataHomeKey`, so pending Copilot sessions (no cwd) are not dropped. --- CHANGELOG.md | 2 +- docs/designs/data-directory-layout.md | 4 +- src/__tests__/dashboard-report-scope.test.ts | 14 ++++ src/__tests__/scope-filter.test.ts | 24 +++++++ src/team-push.ts | 69 +++++++++++++------- src/types.ts | 5 ++ 6 files changed, 93 insertions(+), 25 deletions(-) diff --git a/CHANGELOG.md b/CHANGELOG.md index 3fa0dd0fd..02dd9bb54 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -40,7 +40,7 @@ All notable changes to this project will be documented in this file. See [standa - A `manifest/roles.yaml` that exists but does not parse now fails the pull for that scope instead of warning and syncing with no role filter at all, for a member with no role as much as for one with a role. The same applies to `init` and `push`, which each fell back to a guess at the namespaces when any error came out of the loader. The legacy role migration skips with a warning instead of failing, so every command, `pull` included, still loads the config and can fetch the fixed manifest; until it can run, the member holds no role rather than every role, so hooks, MCP servers and env variables scoped by `roles:` reach them no more than skills do. For a member with no active project that fallback meant an unfiltered sync, so a broken manifest delivered every namespace it was written to gate. Only an absent manifest still means "this team does not use roles"; an unreadable or empty file is an error, as it now is for `manifest/projects.yaml` too. `push` stops at its scan for such a manifest (exit 2) even with `--role `, since the scan needs it to tell which namespaces are the member's. - `teamai members list` and `teamai projects members` read the roster registered before the reports switch, so a team upgrading past the orphan-branch split no longer sees "No team members registered" while its `members/` still lives on the default branch. The default-branch copy becomes a read-only inherited root, the way learnings' already was: listed in union with the `teamai-reports` copy, with the branch copy winning when the same file exists on both; nothing is copied or deleted, and a cold `members list` still does not publish the reports branch. Member registration merges against the inherited copy too, so a re-init keeps the original `registeredAt` and projects. Fixes [#735](https://github.com/Tencent/teamai-cli/issues/735). - Cache GC now rejects partial integers such as `12abc`, decimals, zero and unsafe integers for `--max-bytes` and `--stale-days` before deleting anything. An invalid `TEAMAI_CACHE_MAX_BYTES` value falls back to the default 5 GB limit instead of using a numeric prefix. -- Each scope reports only the dashboard sessions recorded in it. Every scope read one machine-wide event log and picked its sessions out by the `cwd` they started in, so a user-scope pull reported every project's sessions to the user team, a project never reported its Copilot sessions (teamai does not record Copilot's `cwd`), and a session started under a symlinked path (or `/tmp` for `/private/tmp` on macOS) never matched its project. Each event now records a key (a hash, so no path is stored) of the data directory of the scope `hook-dispatch` resolved, and a report keeps only its own scope's. A session in a directory that resolves to no project (a subdirectory of a non-git project, a git submodule or nested clone) is the user scope's, as for skill usage. A session is reported once, whole, by the scope it started in, even after a `cd` into another project: its Stop carries the whole transcript's totals, so reporting its later part elsewhere would count them twice. Events recorded by an earlier release go to the project whose root holds their `cwd`, or to the user scope when that directory still exists and belongs to no project; events with no `cwd`, or one removed since, are reported by no one. The usage guide explains how to remove by hand a skill an earlier release reported into `stats/.yaml` from another project (for [#785](https://github.com/Tencent/teamai-cli/issues/785)). +- Each scope reports only the dashboard sessions recorded in it. Every scope read one machine-wide event log and picked its sessions out by the `cwd` they started in, so a user-scope pull reported every project's sessions to the user team, a project never reported its Copilot sessions (teamai does not record Copilot's `cwd`), and a session started under a symlinked path (or `/tmp` for `/private/tmp` on macOS) never matched its project. Each event now records a key (a hash, so no path is stored) of the data directory of the scope `hook-dispatch` resolved, and a report keeps only its own scope's. A session in a directory that resolves to no project (a subdirectory of a non-git project, a git submodule or nested clone) is the user scope's, as for skill usage. A session is reported once, whole, by the scope it started in, even after a `cd` into another project: its Stop carries the whole transcript's totals, so reporting its later part elsewhere would count them twice. A session ends with its session end or process exit, so a later session that reuses its ID (Copilot's fallback ID is the parent process ID) is attributed on its own. Events recorded by an earlier release go to the project whose root holds their `cwd`, or to the user scope when that directory still exists and belongs to no project; events with no `cwd`, or one removed since, are reported by no one. The usage guide explains how to remove by hand a skill an earlier release reported into `stats/.yaml` from another project (for [#785](https://github.com/Tencent/teamai-cli/issues/785)). - Each scope keeps its own snapshots of the dashboard sessions it already reported. They were machine-wide and keyed by session ID, so a session ID reused in another scope (Copilot's fallback ID is the parent process ID) was taken as already reported there and sent nothing. Each scope now keeps its own (`/dashboard/reported-*.json`, and `~/.teamai/dashboard/user-reported-*.json` for the user scope), seeded the first time from the shared `~/.teamai/dashboard/reported-*.json`, so the first report after the upgrade sends nothing already reported. The shared file is no longer written; after a rollback an earlier release writes it again, and only a scope not yet seeded reads it (for [#786](https://github.com/Tencent/teamai-cli/issues/786)). - Usage reporting scopes sessions by path on Windows too. The project/user scope filter compared an event's `cwd` against `projectRoot` with a hard-coded `/` separator, so on Windows only a session started in the project root itself matched: every session started in a subdirectory was dropped from the project team's report and counted in the user scope's instead, which is the isolation the usage guide promises. Windows paths are also compared case-insensitively, so a drive letter or a directory name spelled with different case in the two sources no longer leaks a project session into the user scope. POSIX paths keep their own rules: case-sensitive, and a `\` in a filename stays part of the name. - `teamai tags subscribe` and `teamai tags unsubscribe` now invalidate the pull revision cache, as `teamai skill exclude` already does, so the next `teamai pull` applies the new subscriptions instead of reporting "Already synced" when the team repo has not changed. diff --git a/docs/designs/data-directory-layout.md b/docs/designs/data-directory-layout.md index 339424e10..73b64ca4e 100644 --- a/docs/designs/data-directory-layout.md +++ b/docs/designs/data-directory-layout.md @@ -331,7 +331,9 @@ carries `dataHomeKey`, a hash of the realpath'd `getDataHome()` of the scope the hook resolved (#785; a hash, so a Copilot event still stores no path), and a scope's report keeps the sessions whose first keyed event is its own, whole: a Stop carries the whole transcript's totals, so a session that moved scope (a `cd` -mid-session) is reported once, where it started. A session written before that +mid-session) is reported once, where it started. A session is one ID's run up to +its `session_end` or `process_exit`, so a later run that reuses the ID (Copilot's +fallback ID is the parent PID) is decided on its own. A session written before that field existed is attributed by its first `cwd`: to the project whose root holds it (realpath'd), or to the user scope when that directory still exists and `resolveConfigForDir` resolves it to the user scope; no `cwd`, or one removed diff --git a/src/__tests__/dashboard-report-scope.test.ts b/src/__tests__/dashboard-report-scope.test.ts index dfaa0c4cb..f71d6e11b 100644 --- a/src/__tests__/dashboard-report-scope.test.ts +++ b/src/__tests__/dashboard-report-scope.test.ts @@ -268,6 +268,20 @@ describe('each scope keeps its own reported snapshot (#786)', () => { expect(await reportedSessions(project)).toBe(1); }); + it('a session ID reused after its session ended goes to the scope that reuses it, while the log still holds the ended one', async () => { + const { root, user, project } = await setup(); + const elsewhere = path.join(tmp, 'elsewhere'); + fs.mkdirSync(elsewhere); + await session('copilot', { cwd: elsewhere }); + await hook('session-end', 'copilot', { cwd: elsewhere, hook_event_name: 'SessionEnd' }); + expect(await reportedSessions(user)).toBe(1); + // Below the compaction threshold the ended `pid-` session stays; the next one in P reuses its ID. + await session('copilot', { cwd: root }); + + expect(await reportedSessions(project)).toBe(1); + expect(await reportedSessions(user)).toBe(1); + }); + it('the first report after the upgrade sends nothing a shared snapshot already reported', async () => { const { root, user, project } = await setup(); const elsewhere = path.join(tmp, 'elsewhere'); diff --git a/src/__tests__/scope-filter.test.ts b/src/__tests__/scope-filter.test.ts index 048e6aec5..932083340 100644 --- a/src/__tests__/scope-filter.test.ts +++ b/src/__tests__/scope-filter.test.ts @@ -70,6 +70,30 @@ describe('filterEventsByScope', () => { expect(await ids(moved, projectScope('/Users/jeff/project-a'))).toEqual([]); }); + it('a session ID reused after its session ended is decided anew', async () => { + // A PID-fallback ID: the user-scope run ended, a later run in P reuses the ID. + for (const end of ['session_end', 'process_exit'] as const) { + const reused = [ + await scopedEvent(undefined, 'pid-1', '/home/jeff/.teamai'), + { ...(await scopedEvent(undefined, 'pid-1', '/home/jeff/.teamai')), type: end }, + await scopedEvent(undefined, 'pid-1', '/home/jeff/.teamai/projects/p'), + ]; + const project = await filterEventsByScope(reused, projectScope('/Users/jeff/project-a')); + const user = await filterEventsByScope(reused, userScope()); + expect(project).toEqual([reused[2]]); + expect(user).toEqual([reused[0], reused[1]]); + } + }); + + it('an event that records its data home as a path, before keys were hashed, is keyed by it', async () => { + const unhashed: DashboardEvent[] = [ + { ...makeEvent(undefined, 'c1'), tool: 'copilot', dataHome: '/home/jeff/.teamai/projects/p' }, + { ...makeEvent('/Users/jeff/project-a', 'u1'), dataHome: '/home/jeff/.teamai' }, + ]; + expect(await ids(unhashed, projectScope('/Users/jeff/project-a'))).toEqual(['c1']); + expect(await ids(unhashed, userScope())).toEqual(['u1']); + }); + it('a keyed event decides a session over an earlier unkeyed one', async () => { // Recorded across the upgrade: the unkeyed event's cwd is P's, the key the user scope's. const upgraded = [ diff --git a/src/team-push.ts b/src/team-push.ts index 042ab1f34..cf4d4756c 100644 --- a/src/team-push.ts +++ b/src/team-push.ts @@ -376,18 +376,20 @@ function isUnderScopeRoot(cwd: string, root: ScopeRoot): boolean { /** * The dashboard events a scope reports (#785): every event of the sessions - * recorded in it. A session is decided once, whole, by its first event keyed - * with a data home (`dataHomeKey`), because a Stop carries the whole - * transcript's totals: split per event, a session that moved into another - * scope would count there again the part recorded before the move. A project - * also owns the key of its in-repo `.teamai`, where a hook recorded until - * migration moved the project to a partition. A session written before events - * carried a key is decided by its first cwd: a project's when the root holds - * it, the user scope's when the directory still exists and resolves to the - * user scope now, the dispatcher's rule; one with no cwd, or a cwd gone since, - * is no scope's. A cwd is raw as the host sent it (a symlinked checkout, macOS - * `/tmp` vs `/private/tmp`), and so is a non-git project's root, so both are - * realpath'd while they still exist. A caller without a scope config reads the + * recorded in it. A session is the run of one ID up to its session_end or + * process_exit, since a PID-fallback ID comes back for a later run. It is + * decided once, whole, by its first event keyed with a data home + * (`dataHomeKey`, or the path an unreleased build of #795 wrote), because a + * Stop carries the whole transcript's totals: split per event, a session that + * moved into another scope would count there again the part recorded before + * the move. A project also owns the key of its in-repo `.teamai`, where a + * hook recorded until migration moved the project to a partition. A session + * written before events carried a key is decided by its first cwd: a + * project's when the root holds it, the user scope's when the directory still + * exists and resolves to the user scope now, the dispatcher's rule; one with + * no cwd, or a cwd gone since, is no scope's. A cwd is raw as the host sent + * it (a symlinked checkout, macOS `/tmp` vs `/private/tmp`), and so is a + * non-git project's root, so both are realpath'd while they still exist. A caller without a scope config reads the * whole log. */ export async function filterEventsByScope( @@ -427,18 +429,39 @@ export async function filterEventsByScope( return owns; }; // The log is hand-editable: a key that is not a string counts as absent. - const keyed = (e: DashboardEvent | undefined): boolean => typeof e?.dataHomeKey === 'string'; - const deciding = new Map(); - for (const e of events) { - const current = deciding.get(e.sessionId); - if (keyed(e) ? !keyed(current) : !current && !!e.cwd) deciding.set(e.sessionId, e); - } - const owned = new Set(); - await Promise.all([...deciding].map(async ([sessionId, e]) => { - const mine = typeof e.dataHomeKey === 'string' ? keys.has(e.dataHomeKey) : !!e.cwd && await ownsCwd(e.cwd); - if (mine) owned.add(sessionId); + // An unreleased build of #795 recorded the data home path in place of its key. + const hashes = new Map>(); + const eventKeys = await Promise.all(events.map((e) => { + if (typeof e.dataHomeKey === 'string') return e.dataHomeKey; + if (typeof e.dataHome !== 'string') return undefined; + let key = hashes.get(e.dataHome); + if (!key) { + key = dataHomeKey(e.dataHome); + hashes.set(e.dataHome, key); + } + return key; + })); + // A session ID names one run until it ends: a PID-fallback ID comes back for + // a later run, maybe in another scope, so each run is decided on its own. + const runOf: number[] = []; + const openRun = new Map(); + const deciding: Array = []; + events.forEach((e, i) => { + const run = openRun.get(e.sessionId) ?? deciding.push(undefined) - 1; + openRun.set(e.sessionId, run); + if (e.type === 'session_end' || e.type === 'process_exit') openRun.delete(e.sessionId); + runOf.push(run); + const current = deciding[run]; + if (eventKeys[i] !== undefined ? current === undefined || eventKeys[current] === undefined + : current === undefined && !!e.cwd) deciding[run] = i; + }); + const owned = await Promise.all(deciding.map(async (i) => { + if (i === undefined) return false; + const key = eventKeys[i]; + const cwd = events[i].cwd; + return key !== undefined ? keys.has(key) : !!cwd && await ownsCwd(cwd); })); - return events.filter((e) => owned.has(e.sessionId)); + return events.filter((_, i) => owned[runOf[i]]); } /** diff --git a/src/types.ts b/src/types.ts index fae370448..8c4f71b12 100644 --- a/src/types.ts +++ b/src/types.ts @@ -1225,6 +1225,11 @@ export interface DashboardEvent { * field existed; the report then attributes them by `cwd`. */ dataHomeKey?: string; + /** + * The data home itself, as a path, which unreleased builds of #795 wrote in + * place of `dataHomeKey`. Read only: the report keys it with `dataHomeKey()`. + */ + dataHome?: string; /** First user prompt (captured from UserPromptSubmit) */ promptSummary?: string; /** From 13985fb96a5b3ad8385dda97d345ab80a0d57b0d Mon Sep 17 00:00:00 2001 From: Saul Moro Date: Thu, 24 Sep 2026 15:03:30 +0200 Subject: [PATCH 09/41] fix(stats): address CI review (#785) A later run of a reused session ID (Copilot's PID fallback) was decided on its own but returned under the same ID, so aggregation and the per-scope snapshots merged two runs in one scope back into one session. The filter now returns a later run as `@`; the first run keeps the bare ID, so existing snapshots still match. An unkeyed event's cwd under a project root counted even when the directory was gone (realpath fell back to the raw path). It now counts only while it exists, as the docs and the user-scope rule already say. --- CHANGELOG.md | 2 +- docs/designs/data-directory-layout.md | 4 ++- src/__tests__/dashboard-report-scope.test.ts | 12 +++++++ src/__tests__/scope-filter.test.ts | 27 +++++++++++++-- src/__tests__/stats-scope.test.ts | 3 ++ src/team-push.ts | 35 ++++++++++++-------- 6 files changed, 65 insertions(+), 18 deletions(-) diff --git a/CHANGELOG.md b/CHANGELOG.md index 02dd9bb54..ef3f528c6 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -40,7 +40,7 @@ All notable changes to this project will be documented in this file. See [standa - A `manifest/roles.yaml` that exists but does not parse now fails the pull for that scope instead of warning and syncing with no role filter at all, for a member with no role as much as for one with a role. The same applies to `init` and `push`, which each fell back to a guess at the namespaces when any error came out of the loader. The legacy role migration skips with a warning instead of failing, so every command, `pull` included, still loads the config and can fetch the fixed manifest; until it can run, the member holds no role rather than every role, so hooks, MCP servers and env variables scoped by `roles:` reach them no more than skills do. For a member with no active project that fallback meant an unfiltered sync, so a broken manifest delivered every namespace it was written to gate. Only an absent manifest still means "this team does not use roles"; an unreadable or empty file is an error, as it now is for `manifest/projects.yaml` too. `push` stops at its scan for such a manifest (exit 2) even with `--role `, since the scan needs it to tell which namespaces are the member's. - `teamai members list` and `teamai projects members` read the roster registered before the reports switch, so a team upgrading past the orphan-branch split no longer sees "No team members registered" while its `members/` still lives on the default branch. The default-branch copy becomes a read-only inherited root, the way learnings' already was: listed in union with the `teamai-reports` copy, with the branch copy winning when the same file exists on both; nothing is copied or deleted, and a cold `members list` still does not publish the reports branch. Member registration merges against the inherited copy too, so a re-init keeps the original `registeredAt` and projects. Fixes [#735](https://github.com/Tencent/teamai-cli/issues/735). - Cache GC now rejects partial integers such as `12abc`, decimals, zero and unsafe integers for `--max-bytes` and `--stale-days` before deleting anything. An invalid `TEAMAI_CACHE_MAX_BYTES` value falls back to the default 5 GB limit instead of using a numeric prefix. -- Each scope reports only the dashboard sessions recorded in it. Every scope read one machine-wide event log and picked its sessions out by the `cwd` they started in, so a user-scope pull reported every project's sessions to the user team, a project never reported its Copilot sessions (teamai does not record Copilot's `cwd`), and a session started under a symlinked path (or `/tmp` for `/private/tmp` on macOS) never matched its project. Each event now records a key (a hash, so no path is stored) of the data directory of the scope `hook-dispatch` resolved, and a report keeps only its own scope's. A session in a directory that resolves to no project (a subdirectory of a non-git project, a git submodule or nested clone) is the user scope's, as for skill usage. A session is reported once, whole, by the scope it started in, even after a `cd` into another project: its Stop carries the whole transcript's totals, so reporting its later part elsewhere would count them twice. A session ends with its session end or process exit, so a later session that reuses its ID (Copilot's fallback ID is the parent process ID) is attributed on its own. Events recorded by an earlier release go to the project whose root holds their `cwd`, or to the user scope when that directory still exists and belongs to no project; events with no `cwd`, or one removed since, are reported by no one. The usage guide explains how to remove by hand a skill an earlier release reported into `stats/.yaml` from another project (for [#785](https://github.com/Tencent/teamai-cli/issues/785)). +- Each scope reports only the dashboard sessions recorded in it. Every scope read one machine-wide event log and picked its sessions out by the `cwd` they started in, so a user-scope pull reported every project's sessions to the user team, a project never reported its Copilot sessions (teamai does not record Copilot's `cwd`), and a session started under a symlinked path (or `/tmp` for `/private/tmp` on macOS) never matched its project. Each event now records a key (a hash, so no path is stored) of the data directory of the scope `hook-dispatch` resolved, and a report keeps only its own scope's. A session in a directory that resolves to no project (a subdirectory of a non-git project, a git submodule or nested clone) is the user scope's, as for skill usage. A session is reported once, whole, by the scope it started in, even after a `cd` into another project: its Stop carries the whole transcript's totals, so reporting its later part elsewhere would count them twice. A session ends with its session end or process exit, so a later session that reuses its ID (Copilot's fallback ID is the parent process ID) is attributed and counted on its own. Events recorded by an earlier release go to the project whose root holds their `cwd`, or to the user scope when that directory still exists and belongs to no project; events with no `cwd`, or one removed since, are reported by no one. The usage guide explains how to remove by hand a skill an earlier release reported into `stats/.yaml` from another project (for [#785](https://github.com/Tencent/teamai-cli/issues/785)). - Each scope keeps its own snapshots of the dashboard sessions it already reported. They were machine-wide and keyed by session ID, so a session ID reused in another scope (Copilot's fallback ID is the parent process ID) was taken as already reported there and sent nothing. Each scope now keeps its own (`/dashboard/reported-*.json`, and `~/.teamai/dashboard/user-reported-*.json` for the user scope), seeded the first time from the shared `~/.teamai/dashboard/reported-*.json`, so the first report after the upgrade sends nothing already reported. The shared file is no longer written; after a rollback an earlier release writes it again, and only a scope not yet seeded reads it (for [#786](https://github.com/Tencent/teamai-cli/issues/786)). - Usage reporting scopes sessions by path on Windows too. The project/user scope filter compared an event's `cwd` against `projectRoot` with a hard-coded `/` separator, so on Windows only a session started in the project root itself matched: every session started in a subdirectory was dropped from the project team's report and counted in the user scope's instead, which is the isolation the usage guide promises. Windows paths are also compared case-insensitively, so a drive letter or a directory name spelled with different case in the two sources no longer leaks a project session into the user scope. POSIX paths keep their own rules: case-sensitive, and a `\` in a filename stays part of the name. - `teamai tags subscribe` and `teamai tags unsubscribe` now invalidate the pull revision cache, as `teamai skill exclude` already does, so the next `teamai pull` applies the new subscriptions instead of reporting "Already synced" when the team repo has not changed. diff --git a/docs/designs/data-directory-layout.md b/docs/designs/data-directory-layout.md index 73b64ca4e..4d7e347da 100644 --- a/docs/designs/data-directory-layout.md +++ b/docs/designs/data-directory-layout.md @@ -333,7 +333,9 @@ scope's report keeps the sessions whose first keyed event is its own, whole: a Stop carries the whole transcript's totals, so a session that moved scope (a `cd` mid-session) is reported once, where it started. A session is one ID's run up to its `session_end` or `process_exit`, so a later run that reuses the ID (Copilot's -fallback ID is the parent PID) is decided on its own. A session written before that +fallback ID is the parent PID) is decided on its own, and reported and snapshotted +as its own session under `@` (the first run keeps the +bare ID, so snapshots written before still match). A session written before that field existed is attributed by its first `cwd`: to the project whose root holds it (realpath'd), or to the user scope when that directory still exists and `resolveConfigForDir` resolves it to the user scope; no `cwd`, or one removed diff --git a/src/__tests__/dashboard-report-scope.test.ts b/src/__tests__/dashboard-report-scope.test.ts index f71d6e11b..f71da317f 100644 --- a/src/__tests__/dashboard-report-scope.test.ts +++ b/src/__tests__/dashboard-report-scope.test.ts @@ -282,6 +282,18 @@ describe('each scope keeps its own reported snapshot (#786)', () => { expect(await reportedSessions(user)).toBe(1); }); + it('two sessions that reuse one session ID in the same scope are two sessions, while the log holds both', async () => { + const { root, project } = await setup(); + await session('copilot', { cwd: root }); + await hook('session-end', 'copilot', { cwd: root, hook_event_name: 'SessionEnd' }); + expect(await reportedSessions(project)).toBe(1); + await session('copilot', { cwd: root }); + await hook('session-end', 'copilot', { cwd: root, hook_event_name: 'SessionEnd' }); + + expect(await reportedSessions(project)).toBe(2); + expect(await reportedPrompts(project)).toBe(2); + }); + it('the first report after the upgrade sends nothing a shared snapshot already reported', async () => { const { root, user, project } = await setup(); const elsewhere = path.join(tmp, 'elsewhere'); diff --git a/src/__tests__/scope-filter.test.ts b/src/__tests__/scope-filter.test.ts index 932083340..5d105d29a 100644 --- a/src/__tests__/scope-filter.test.ts +++ b/src/__tests__/scope-filter.test.ts @@ -1,4 +1,4 @@ -import { describe, it, expect } from 'vitest'; +import { afterEach, beforeEach, describe, it, expect, vi } from 'vitest'; import fs from 'node:fs'; import os from 'node:os'; import path from 'node:path'; @@ -31,6 +31,16 @@ async function ids(events: DashboardEvent[], config?: LocalConfig): Promise { + // The fixed paths below stand for directories that exist: an older event's + // cwd counts only while it does. Real temp directories resolve for real. + const FIXED_PATH = /^(?:\/Users\/jeff\/|\/work\/|[A-Za-z]:|\\\\)/; + beforeEach(() => { + const realpath = fs.promises.realpath; + vi.spyOn(fs.promises, 'realpath').mockImplementation(async (p) => + typeof p === 'string' && FIXED_PATH.test(p) ? p : realpath(p)); + }); + afterEach(() => vi.restoreAllMocks()); + const events: DashboardEvent[] = [ makeEvent('/Users/jeff/project-a', 's1'), makeEvent('/Users/jeff/project-a/src', 's2'), @@ -80,7 +90,8 @@ describe('filterEventsByScope', () => { ]; const project = await filterEventsByScope(reused, projectScope('/Users/jeff/project-a')); const user = await filterEventsByScope(reused, userScope()); - expect(project).toEqual([reused[2]]); + // The later run is its own session, under an ID of its own. + expect(project).toEqual([{ ...reused[2], sessionId: `pid-1@${reused[2].timestamp}` }]); expect(user).toEqual([reused[0], reused[1]]); } }); @@ -171,6 +182,18 @@ describe('filterEventsByScope', () => { } }); + it('does not give a project an older event whose cwd under its root is gone', async () => { + // A removed worktree may have been a nested clone, whose sessions were never the project's. + const tmp = fs.realpathSync(fs.mkdtempSync(path.join(os.tmpdir(), 'teamai-scope-cwd-'))); + try { + fs.mkdirSync(path.join(tmp, 'src')); + const evts = [makeEvent(path.join(tmp, 'src'), 'e1'), makeEvent(path.join(tmp, 'removed-worktree'), 'g1')]; + expect(await ids(evts, projectScope(tmp))).toEqual(['e1']); + } finally { + fs.rmSync(tmp, { recursive: true, force: true }); + } + }); + it('the user scope never keeps events without a data home', async () => { expect(await ids(events, userScope())).toEqual([]); }); diff --git a/src/__tests__/stats-scope.test.ts b/src/__tests__/stats-scope.test.ts index 79d3445f5..e4b912900 100644 --- a/src/__tests__/stats-scope.test.ts +++ b/src/__tests__/stats-scope.test.ts @@ -209,6 +209,9 @@ beforeEach(() => { consoleLog = vi.spyOn(console, 'log').mockImplementation(() => undefined); initWorkspace(); DIRS = projectDirs(); + // An older, unkeyed event counts only while its cwd exists (#785). + fs.mkdirSync(DIRS.project, { recursive: true }); + fs.mkdirSync(DIRS.other, { recursive: true }); }); afterEach(() => { diff --git a/src/team-push.ts b/src/team-push.ts index cf4d4756c..132f64b20 100644 --- a/src/team-push.ts +++ b/src/team-push.ts @@ -377,7 +377,8 @@ function isUnderScopeRoot(cwd: string, root: ScopeRoot): boolean { /** * The dashboard events a scope reports (#785): every event of the sessions * recorded in it. A session is the run of one ID up to its session_end or - * process_exit, since a PID-fallback ID comes back for a later run. It is + * process_exit, since a PID-fallback ID comes back for a later run; a later + * run is returned under an ID of its own, so it is counted on its own. It is * decided once, whole, by its first event keyed with a data home * (`dataHomeKey`, or the path an unreleased build of #795 wrote), because a * Stop carries the whole transcript's totals: split per event, a session that @@ -397,15 +398,6 @@ export async function filterEventsByScope( config?: LocalConfig, ): Promise { if (!config) return events; - const realPaths = new Map>(); - const realPath = (dir: string): Promise => { - let real = realPaths.get(dir); - if (!real) { - real = fs.promises.realpath(dir).catch(() => dir); - realPaths.set(dir, real); - } - return real; - }; const ownKey = await dataHomeKey(getDataHome(config)); const keys = new Set([ownKey]); if (config.projectRoot) { @@ -413,13 +405,14 @@ export async function filterEventsByScope( const legacy = await dataHomeKey(path.join(config.projectRoot, '.teamai')); if (legacy !== (await dataHomeKey(path.join(getUserHome(), '.teamai')))) keys.add(legacy); } - const root = config.projectRoot ? scopeRoot(await realPath(config.projectRoot)) : undefined; + const projectRoot = config.projectRoot; + const root = projectRoot ? scopeRoot(await fs.promises.realpath(projectRoot).catch(() => projectRoot)) : undefined; const { resolveConfigForDir } = await import('./config.js'); const resolvesHere = new Map>(); const ownsCwd = (cwd: string): Promise => { let owns = resolvesHere.get(cwd); if (!owns) { - owns = root ? realPath(cwd).then((real) => isUnderScopeRoot(real, root)) + owns = root ? fs.promises.realpath(cwd).then((real) => isUnderScopeRoot(real, root), () => false) : pathExists(cwd).then(async (exists) => { const resolved = exists ? await resolveConfigForDir(cwd) : null; return !!resolved && (await dataHomeKey(getDataHome(resolved))) === ownKey; @@ -443,11 +436,21 @@ export async function filterEventsByScope( })); // A session ID names one run until it ends: a PID-fallback ID comes back for // a later run, maybe in another scope, so each run is decided on its own. + // A later run is also reported as its own session: it is returned under the + // ID plus its first event's timestamp. The first run keeps the bare ID, so + // the snapshots already written for it still match. const runOf: number[] = []; const openRun = new Map(); const deciding: Array = []; + const runIds: string[] = []; + const seen = new Set(); events.forEach((e, i) => { - const run = openRun.get(e.sessionId) ?? deciding.push(undefined) - 1; + let run = openRun.get(e.sessionId); + if (run === undefined) { + run = deciding.push(undefined) - 1; + runIds.push(seen.has(e.sessionId) ? `${e.sessionId}@${e.timestamp}` : e.sessionId); + seen.add(e.sessionId); + } openRun.set(e.sessionId, run); if (e.type === 'session_end' || e.type === 'process_exit') openRun.delete(e.sessionId); runOf.push(run); @@ -461,7 +464,11 @@ export async function filterEventsByScope( const cwd = events[i].cwd; return key !== undefined ? keys.has(key) : !!cwd && await ownsCwd(cwd); })); - return events.filter((_, i) => owned[runOf[i]]); + return events.flatMap((e, i) => { + const run = runOf[i]; + if (!owned[run]) return []; + return runIds[run] === e.sessionId ? [e] : [{ ...e, sessionId: runIds[run] }]; + }); } /** From d28f86f06428ba518ec7725bfc38bcec1b2aa7a5 Mon Sep 17 00:00:00 2001 From: Saul Moro Date: Thu, 24 Sep 2026 15:30:54 +0200 Subject: [PATCH 10/41] fix(stats): address CI review (#785) Run identity no longer depends on which earlier runs compaction kept: every run is `@`, so a reused PID-fallback ID is a new session even when the scope's snapshot still names the run compaction dropped. Snapshot entries keyed by the bare ID (written by earlier builds) are adopted by the first run of that ID in the log, so the upgrade re-sends nothing; the next snapshot holds only run IDs. An unkeyed event's cwd is now owned by the scope resolveConfigForDir resolves it to, for projects as for the user scope, so a nested clone under a project is no longer reported by both. The lexical root matcher and its string-level tests go; the cases move to real repositories. --- CHANGELOG.md | 2 +- docs/designs/data-directory-layout.md | 15 +- docs/usage-guide.md | 13 +- docs/usage-guide.zh-CN.md | 8 +- src/__tests__/dashboard-report-scope.test.ts | 24 ++- src/__tests__/e2e/report-timeout.test.ts | 10 +- src/__tests__/scope-filter.test.ts | 152 +++++------------- src/__tests__/team-push-interventions.test.ts | 21 +-- src/stats.ts | 6 +- src/team-push.ts | 114 +++++-------- 10 files changed, 138 insertions(+), 227 deletions(-) diff --git a/CHANGELOG.md b/CHANGELOG.md index fcf5bf3d6..2bb21ddf4 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -41,7 +41,7 @@ All notable changes to this project will be documented in this file. See [standa - A `manifest/roles.yaml` that exists but does not parse now fails the pull for that scope instead of warning and syncing with no role filter at all, for a member with no role as much as for one with a role. The same applies to `init` and `push`, which each fell back to a guess at the namespaces when any error came out of the loader. The legacy role migration skips with a warning instead of failing, so every command, `pull` included, still loads the config and can fetch the fixed manifest; until it can run, the member holds no role rather than every role, so hooks, MCP servers and env variables scoped by `roles:` reach them no more than skills do. For a member with no active project that fallback meant an unfiltered sync, so a broken manifest delivered every namespace it was written to gate. Only an absent manifest still means "this team does not use roles"; an unreadable or empty file is an error, as it now is for `manifest/projects.yaml` too. `push` stops at its scan for such a manifest (exit 2) even with `--role `, since the scan needs it to tell which namespaces are the member's. - `teamai members list` and `teamai projects members` read the roster registered before the reports switch, so a team upgrading past the orphan-branch split no longer sees "No team members registered" while its `members/` still lives on the default branch. The default-branch copy becomes a read-only inherited root, the way learnings' already was: listed in union with the `teamai-reports` copy, with the branch copy winning when the same file exists on both; nothing is copied or deleted, and a cold `members list` still does not publish the reports branch. Member registration merges against the inherited copy too, so a re-init keeps the original `registeredAt` and projects. Fixes [#735](https://github.com/Tencent/teamai-cli/issues/735). - Cache GC now rejects partial integers such as `12abc`, decimals, zero and unsafe integers for `--max-bytes` and `--stale-days` before deleting anything. An invalid `TEAMAI_CACHE_MAX_BYTES` value falls back to the default 5 GB limit instead of using a numeric prefix. -- Each scope reports only the dashboard sessions recorded in it. Every scope read one machine-wide event log and picked its sessions out by the `cwd` they started in, so a user-scope pull reported every project's sessions to the user team, a project never reported its Copilot sessions (teamai does not record Copilot's `cwd`), and a session started under a symlinked path (or `/tmp` for `/private/tmp` on macOS) never matched its project. Each event now records a key (a hash, so no path is stored) of the data directory of the scope `hook-dispatch` resolved, and a report keeps only its own scope's. A session in a directory that resolves to no project (a subdirectory of a non-git project, a git submodule or nested clone) is the user scope's, as for skill usage. A session is reported once, whole, by the scope it started in, even after a `cd` into another project: its Stop carries the whole transcript's totals, so reporting its later part elsewhere would count them twice. A session ends with its session end or process exit, so a later session that reuses its ID (Copilot's fallback ID is the parent process ID) is attributed and counted on its own. Events recorded by an earlier release go to the project whose root holds their `cwd`, or to the user scope when that directory still exists and belongs to no project; events with no `cwd`, or one removed since, are reported by no one. The usage guide explains how to remove by hand a skill an earlier release reported into `stats/.yaml` from another project (for [#785](https://github.com/Tencent/teamai-cli/issues/785)). +- Each scope reports only the dashboard sessions recorded in it. Every scope read one machine-wide event log and picked its sessions out by the `cwd` they started in, so a user-scope pull reported every project's sessions to the user team, a project never reported its Copilot sessions (teamai does not record Copilot's `cwd`), and a session started under a symlinked path (or `/tmp` for `/private/tmp` on macOS) never matched its project. Each event now records a key (a hash, so no path is stored) of the data directory of the scope `hook-dispatch` resolved, and a report keeps only its own scope's. A session in a directory that resolves to no project (a subdirectory of a non-git project, a git submodule or nested clone) is the user scope's, as for skill usage. A session is reported once, whole, by the scope it started in, even after a `cd` into another project: its Stop carries the whole transcript's totals, so reporting its later part elsewhere would count them twice. A session ends with its session end or process exit, so a later session that reuses its ID (Copilot's fallback ID is the parent process ID) is attributed and counted on its own. Events recorded by an earlier release go to the scope their `cwd` resolves to now (a nested clone under a project is not the project's); events with no `cwd`, or one removed since, are reported by no one. The usage guide explains how to remove by hand a skill an earlier release reported into `stats/.yaml` from another project (for [#785](https://github.com/Tencent/teamai-cli/issues/785)). - Each scope keeps its own snapshots of the dashboard sessions it already reported. They were machine-wide and keyed by session ID, so a session ID reused in another scope (Copilot's fallback ID is the parent process ID) was taken as already reported there and sent nothing. Each scope now keeps its own (`/dashboard/reported-*.json`, and `~/.teamai/dashboard/user-reported-*.json` for the user scope), seeded the first time from the shared `~/.teamai/dashboard/reported-*.json`, so the first report after the upgrade sends nothing already reported. The shared file is no longer written; after a rollback an earlier release writes it again, and only a scope not yet seeded reads it (for [#786](https://github.com/Tencent/teamai-cli/issues/786)). - Usage reporting scopes sessions by path on Windows too. The project/user scope filter compared an event's `cwd` against `projectRoot` with a hard-coded `/` separator, so on Windows only a session started in the project root itself matched: every session started in a subdirectory was dropped from the project team's report and counted in the user scope's instead, which is the isolation the usage guide promises. Windows paths are also compared case-insensitively, so a drive letter or a directory name spelled with different case in the two sources no longer leaks a project session into the user scope. POSIX paths keep their own rules: case-sensitive, and a `\` in a filename stays part of the name. - `teamai tags subscribe` and `teamai tags unsubscribe` now invalidate the pull revision cache, as `teamai skill exclude` already does, so the next `teamai pull` applies the new subscriptions instead of reporting "Already synced" when the team repo has not changed. diff --git a/docs/designs/data-directory-layout.md b/docs/designs/data-directory-layout.md index 4d7e347da..8c8127eff 100644 --- a/docs/designs/data-directory-layout.md +++ b/docs/designs/data-directory-layout.md @@ -333,13 +333,14 @@ scope's report keeps the sessions whose first keyed event is its own, whole: a Stop carries the whole transcript's totals, so a session that moved scope (a `cd` mid-session) is reported once, where it started. A session is one ID's run up to its `session_end` or `process_exit`, so a later run that reuses the ID (Copilot's -fallback ID is the parent PID) is decided on its own, and reported and snapshotted -as its own session under `@` (the first run keeps the -bare ID, so snapshots written before still match). A session written before that -field existed is attributed by its first `cwd`: to the project whose root holds -it (realpath'd), or to the user scope when that directory still exists and -`resolveConfigForDir` resolves it to the user scope; no `cwd`, or one removed -since, is no scope's. The snapshots of what was already reported are per scope +fallback ID is the parent PID) is decided on its own. Every run is reported and +snapshotted as `@`, which does not change when +compaction drops earlier runs; a snapshot entry keyed by the bare ID (written +before) goes to the first run of that ID in the log, until the scope reports +again. A session written before that field existed is attributed by its first +`cwd`: to the scope `resolveConfigForDir` resolves that directory to now, the +dispatcher's rule, so a nested clone under a project is not the project's; no +`cwd`, or one removed since, is no scope's. The snapshots of what was already reported are per scope too (#786), because a session ID can recur in another scope (Copilot's fallback ID is the parent PID): `/dashboard/reported-*.json`, and `~/.teamai/dashboard/user-reported-*.json` diff --git a/docs/usage-guide.md b/docs/usage-guide.md index 9ab436b23..278effe8d 100644 --- a/docs/usage-guide.md +++ b/docs/usage-guide.md @@ -1757,12 +1757,13 @@ longer reports a project's sessions, and a project reports its Copilot sessions and sessions started under a symlinked path. A session is reported once, whole, by the scope it started in, even if it later moves into another project: its Stop carries the whole transcript's totals, so a second scope would count them -again. Events recorded by an earlier release carry no key: a project reports -those whose directory lies under its root, and the user scope those whose -directory still exists and belongs to no project; events with no directory, or -one removed since, are reported by no one. Each scope also keeps its own -snapshot of what it already reported, so a session ID another scope already -reported (Copilot reuses its fallback IDs) still counts as new here; the first +again. Events recorded by an earlier release carry no key: the scope their +directory resolves to now reports them (a nested clone under a project resolves +to its own project or the user scope, not the enclosing project); events with +no directory, or one removed since, are reported by no one. Each scope also +keeps its own snapshot of what it already reported, and a session that reuses +an earlier session's ID (Copilot reuses its fallback IDs) counts as new, +whichever scope reported the earlier one; the first report after upgrading starts from the snapshot every scope used to share, so nothing is reported twice. A target removes its usage events only after it confirms success; failed pushes diff --git a/docs/usage-guide.zh-CN.md b/docs/usage-guide.zh-CN.md index 205d3a6e0..6170534d4 100644 --- a/docs/usage-guide.zh-CN.md +++ b/docs/usage-guide.zh-CN.md @@ -1692,10 +1692,10 @@ Pull 对整批统计上报最多等待 5 秒，之后继续其他工作，上报 只上报在其中记录的会话：user scope 的 pull 不再上报项目的会话，项目会上报自己的 Copilot 会话以及从软链接路径启动的会话。每个会话只上报一次，整体归属其开始时所在的 scope，即使之后切换到另一个项目：它的 Stop 带有整份 transcript 的累计值，第二个 scope -会重复计算。旧版本记录的事件没有该键：项目上报目录位于其根目录下的那些，user scope -上报目录仍存在且不属于任何项目的那些；没有目录或目录已删除的事件不由任何 scope 上报。 -每个 scope 还各自保存已上报快照，因此其他 scope 已上报过的会话 ID（Copilot 会复用其回退 -ID）在这里仍算新会话；升级后的首次上报从原先各 scope 共用的快照开始，不会重复上报。目标确认成功后才清理自己的使用事件， +会重复计算。旧版本记录的事件没有该键：由其目录当前解析到的 scope 上报（项目下的嵌套 +clone 解析到它自己的项目或 user scope，而不是外层项目）；没有目录或目录已删除的事件不由任何 scope 上报。 +每个 scope 还各自保存已上报快照，且复用先前会话 ID 的会话（Copilot 会复用其回退 ID） +总算作新会话，无论先前那个由哪个 scope 上报；升级后的首次上报从原先各 scope 共用的快照开始，不会重复上报。目标确认成功后才清理自己的使用事件， 推送失败会保留事件。上报完成前继续持有相关同步锁， 避免另一次 Pull 与尚未完成的上报竞争。 diff --git a/src/__tests__/dashboard-report-scope.test.ts b/src/__tests__/dashboard-report-scope.test.ts index f71da317f..427c8410f 100644 --- a/src/__tests__/dashboard-report-scope.test.ts +++ b/src/__tests__/dashboard-report-scope.test.ts @@ -165,6 +165,11 @@ describe('each scope reports only the dashboard sessions recorded in it (#785)', fs.symlinkSync(root, link, 'dir'); fs.mkdirSync(path.join(root, 'src')); fs.mkdirSync(path.join(tmp, 'elsewhere')); + // A nested clone under P resolves to no project, so it is the user scope's, not P's too. + fs.mkdirSync(path.join(root, 'nested')); + execFileSync('git', ['init', '-q'], { cwd: path.join(root, 'nested') }); + // A sibling whose name only starts with P's. + fs.mkdirSync(`${root}-ab`); const timestamp = new Date().toISOString(); const old = (sessionId: string, cwd: string | undefined) => [ { type: 'session_start', timestamp, sessionId, tool: 'claude', cwd }, @@ -176,12 +181,14 @@ describe('each scope reports only the dashboard sessions recorded in it (#785)', ...old('old-in-p', path.join(root, 'src')), ...old('old-via-link', link), ...old('old-elsewhere', path.join(tmp, 'elsewhere')), + ...old('old-nested', path.join(root, 'nested')), + ...old('old-sibling', `${root}-ab`), // Nothing can tell whose these were, so no scope reports them. - ...old('old-gone', path.join(tmp, 'deleted-worktree')), + ...old('old-gone', path.join(root, 'removed-worktree')), ...old('old-no-cwd', undefined), ].map((e) => JSON.stringify(e)).join('\n') + '\n'); - expect(await reportedSessions(user)).toBe(1); + expect(await reportedSessions(user)).toBe(3); expect(await reportedSessions(project)).toBe(2); }); }); @@ -294,6 +301,19 @@ describe('each scope keeps its own reported snapshot (#786)', () => { expect(await reportedPrompts(project)).toBe(2); }); + it('a session ID reused in the same scope after compaction dropped the run it reported is a new session', async () => { + const { root, project } = await setup(); + await session('copilot', { cwd: root }); + await hook('session-end', 'copilot', { cwd: root, hook_event_name: 'SessionEnd' }); + expect(await reportedSessions(project)).toBe(1); + // Compaction dropped the ended run before P reported again; the next run in P reuses its ID. + fs.writeFileSync(path.join(teamaiHome(), 'dashboard', 'events.jsonl'), ''); + await session('copilot', { cwd: root }); + + expect(await reportedSessions(project)).toBe(2); + expect(await reportedPrompts(project)).toBe(2); + }); + it('the first report after the upgrade sends nothing a shared snapshot already reported', async () => { const { root, user, project } = await setup(); const elsewhere = path.join(tmp, 'elsewhere'); diff --git a/src/__tests__/e2e/report-timeout.test.ts b/src/__tests__/e2e/report-timeout.test.ts index 08b03a8c6..5129c61d7 100644 --- a/src/__tests__/e2e/report-timeout.test.ts +++ b/src/__tests__/e2e/report-timeout.test.ts @@ -92,7 +92,9 @@ function fixture(agent: keyof typeof agents, provider: string) { const p = path.join(dashboard, `user-reported-${name}.json`); return fs.existsSync(p) ? JSON.parse(fs.readFileSync(p, 'utf8')) : {}; } - return { home, usage, usageLine, seedEvents, receiver, pull, stats, snapshot }; + // The seeded session's snapshot key: its run ID (#785). + const run = `s1@${timestamp}`; + return { home, usage, usageLine, seedEvents, receiver, pull, stats, snapshot, run }; } afterEach(() => { @@ -111,7 +113,7 @@ describe('real CLI report completion', () => { f.receiver('slow'); const output = await f.pull(() => { expect(fs.readFileSync(f.usage, 'utf8')).toBe(f.usageLine); - expect(f.snapshot('prompt-tokens').s1).toBeUndefined(); + expect(f.snapshot('prompt-tokens')[f.run]).toBeUndefined(); expect(fs.existsSync(path.join(f.home, '.teamai/.sync-lock'))).toBe(true); // An event arriving during the push must survive cleanup of the batch. fs.appendFileSync(f.usage, f.usageLine); @@ -120,7 +122,7 @@ describe('real CLI report completion', () => { expect(f.stats().skills.review.count).toBe(1); expect(fs.readFileSync(f.usage, 'utf8')).toBe(f.usageLine); for (const name of ['interventions', 'prompt-tokens', 'daily-sessions']) { - expect(f.snapshot(name).s1).toBeDefined(); + expect(f.snapshot(name)[f.run]).toBeDefined(); } expect(fs.existsSync(path.join(f.home, '.teamai/.sync-lock'))).toBe(false); f.receiver('normal'); @@ -144,7 +146,7 @@ describe('real CLI report completion', () => { f.receiver('reject'); await f.pull(); expect(fs.readFileSync(f.usage, 'utf8')).toBe(f.usageLine); - expect(f.snapshot('prompt-tokens').s1).toBeUndefined(); + expect(f.snapshot('prompt-tokens')[f.run]).toBeUndefined(); f.receiver('normal'); await f.pull(); const stats = f.stats(); diff --git a/src/__tests__/scope-filter.test.ts b/src/__tests__/scope-filter.test.ts index 5d105d29a..e0d9e1a42 100644 --- a/src/__tests__/scope-filter.test.ts +++ b/src/__tests__/scope-filter.test.ts @@ -1,8 +1,8 @@ -import { afterEach, beforeEach, describe, it, expect, vi } from 'vitest'; +import { describe, it, expect } from 'vitest'; import fs from 'node:fs'; import os from 'node:os'; import path from 'node:path'; -import { filterEventsByScope } from '../team-push.js'; +import { adoptBareKeys, filterEventsByScope } from '../team-push.js'; import { dataHomeKey } from '../dashboard-collector.js'; import type { DashboardEvent, LocalConfig } from '../types.js'; @@ -26,21 +26,12 @@ function userScope(dataHome = '/home/jeff/.teamai'): LocalConfig { return { repo, username: 'jeff', scope: 'user', additionalRoles: [], dataHome }; } +/** The session of each event a scope keeps: its run ID without the run's start. */ async function ids(events: DashboardEvent[], config?: LocalConfig): Promise { - return (await filterEventsByScope(events, config)).map((e) => e.sessionId); + return (await filterEventsByScope(events, config)).map((e) => e.sessionId.slice(0, e.sessionId.lastIndexOf('@'))); } describe('filterEventsByScope', () => { - // The fixed paths below stand for directories that exist: an older event's - // cwd counts only while it does. Real temp directories resolve for real. - const FIXED_PATH = /^(?:\/Users\/jeff\/|\/work\/|[A-Za-z]:|\\\\)/; - beforeEach(() => { - const realpath = fs.promises.realpath; - vi.spyOn(fs.promises, 'realpath').mockImplementation(async (p) => - typeof p === 'string' && FIXED_PATH.test(p) ? p : realpath(p)); - }); - afterEach(() => vi.restoreAllMocks()); - const events: DashboardEvent[] = [ makeEvent('/Users/jeff/project-a', 's1'), makeEvent('/Users/jeff/project-a/src', 's2'), @@ -90,12 +81,25 @@ describe('filterEventsByScope', () => { ]; const project = await filterEventsByScope(reused, projectScope('/Users/jeff/project-a')); const user = await filterEventsByScope(reused, userScope()); - // The later run is its own session, under an ID of its own. + // Each run is its own session, under the ID plus its first event's timestamp. + const first = `pid-1@${reused[0].timestamp}`; expect(project).toEqual([{ ...reused[2], sessionId: `pid-1@${reused[2].timestamp}` }]); - expect(user).toEqual([reused[0], reused[1]]); + expect(user).toEqual([{ ...reused[0], sessionId: first }, { ...reused[1], sessionId: first }]); } }); + it('a run keeps its ID after compaction drops the earlier runs of its ID', async () => { + const ended = [ + await scopedEvent(undefined, 'pid-1', '/home/jeff/.teamai/projects/p'), + { ...(await scopedEvent(undefined, 'pid-1', '/home/jeff/.teamai/projects/p')), type: 'session_end' as const }, + ]; + const later = { ...(await scopedEvent(undefined, 'pid-1', '/home/jeff/.teamai/projects/p')), timestamp: '2099-01-01T00:00:00.000Z' }; + const before = await filterEventsByScope([...ended, later], projectScope('/Users/jeff/project-a')); + const after = await filterEventsByScope([later], projectScope('/Users/jeff/project-a')); + expect(after.map((e) => e.sessionId)).toEqual(['pid-1@2099-01-01T00:00:00.000Z']); + expect(before[2].sessionId).toBe(after[0].sessionId); + }); + it('an event that records its data home as a path, before keys were hashed, is keyed by it', async () => { const unhashed: DashboardEvent[] = [ { ...makeEvent(undefined, 'c1'), tool: 'copilot', dataHome: '/home/jeff/.teamai/projects/p' }, @@ -160,114 +164,30 @@ describe('filterEventsByScope', () => { }); it('an event whose key is not a string counts as written before keys existed', async () => { - // A hand-edited or corrupted line in the shared log. + // A hand-edited or corrupted line in the shared log: a later keyed event decides the session. const corrupt: DashboardEvent[] = JSON.parse(JSON.stringify([ { ...makeEvent('/Users/jeff/project-a', 'c1'), dataHomeKey: null }, - { ...makeEvent('/Users/jeff/other-work', 'c2'), dataHomeKey: 42 }, + { ...makeEvent('/Users/jeff/project-a', 'c2'), dataHomeKey: 42 }, ])); - expect(await ids(corrupt, projectScope('/Users/jeff/project-a'))).toEqual(['c1']); - expect(await ids(corrupt, userScope())).toEqual([]); - }); - - it('matches an older event\'s real cwd to a project whose root was set through a symlink', async () => { - // A non-git project's root is the directory as given, not realpath'd. - const tmp = fs.realpathSync(fs.mkdtempSync(path.join(os.tmpdir(), 'teamai-scope-root-'))); - try { - fs.mkdirSync(path.join(tmp, 'real')); - fs.symlinkSync(path.join(tmp, 'real'), path.join(tmp, 'link'), 'dir'); - const evts = [makeEvent(path.join(tmp, 'real'), 'r1')]; - expect(await ids(evts, projectScope(path.join(tmp, 'link')))).toEqual(['r1']); - } finally { - fs.rmSync(tmp, { recursive: true, force: true }); - } - }); - - it('does not give a project an older event whose cwd under its root is gone', async () => { - // A removed worktree may have been a nested clone, whose sessions were never the project's. - const tmp = fs.realpathSync(fs.mkdtempSync(path.join(os.tmpdir(), 'teamai-scope-cwd-'))); - try { - fs.mkdirSync(path.join(tmp, 'src')); - const evts = [makeEvent(path.join(tmp, 'src'), 'e1'), makeEvent(path.join(tmp, 'removed-worktree'), 'g1')]; - expect(await ids(evts, projectScope(tmp))).toEqual(['e1']); - } finally { - fs.rmSync(tmp, { recursive: true, force: true }); - } - }); - - it('the user scope never keeps events without a data home', async () => { - expect(await ids(events, userScope())).toEqual([]); - }); - - // Events without a data home go to the project whose root holds their cwd. - it('filters to projectRoot (exact match and subdirectories)', async () => { - expect(await ids(events, projectScope('/Users/jeff/project-a'))).toEqual(['s1', 's2']); - }); - - it('projectRoot with trailing slash works the same', async () => { - expect(await ids(events, projectScope('/Users/jeff/project-a/'))).toEqual(['s1', 's2']); - }); - - it('events with undefined cwd are excluded by projectRoot', async () => { - expect(await ids(events, projectScope('/Users/jeff/project-a'))).not.toContain('s5'); - }); - - it('does not match partial directory name prefixes', async () => { - const evts = [ - makeEvent('/Users/jeff/project-ab', 'x1'), - makeEvent('/Users/jeff/project-a', 'x2'), + const decided = [ + corrupt[0], await scopedEvent(undefined, 'c1', '/home/jeff/.teamai/projects/p'), + corrupt[1], await scopedEvent(undefined, 'c2', '/home/jeff/.teamai/projects/p'), ]; - expect(await ids(evts, projectScope('/Users/jeff/project-a'))).toEqual(['x2']); + expect(await ids(decided, projectScope('/Users/jeff/project-a'))).toEqual(['c1', 'c1', 'c2', 'c2']); + expect(await ids(decided, userScope())).toEqual([]); }); +}); - // Windows paths are plain strings here, so these run on the ubuntu CI too. - // Both sides of the comparison are native paths in production: projectRoot is - // path.resolve(cwd) from init, and cwd is whatever the tool's hook payload - // carried. - describe('Windows paths', () => { - const winEvents: DashboardEvent[] = [ - makeEvent('C:\\Users\\jeff\\project-a', 'w1'), - makeEvent('C:\\Users\\jeff\\project-a\\src', 'w2'), - makeEvent('C:\\Users\\jeff\\project-ab', 'w3'), - makeEvent('C:\\Users\\jeff\\other-work', 'w4'), - ]; - - it('filters to projectRoot including subdirectories', async () => { - expect(await ids(winEvents, projectScope('C:\\Users\\jeff\\project-a'))).toEqual(['w1', 'w2']); - }); - - it('matches a root and a cwd that disagree on separator style', async () => { - expect(await ids(winEvents, projectScope('C:/Users/jeff/project-a'))).toEqual(['w1', 'w2']); - }); - - it('trailing backslash on the root works the same', async () => { - expect(await ids(winEvents, projectScope('C:\\Users\\jeff\\project-a\\'))).toEqual(['w1', 'w2']); - }); - - it('ignores drive-letter and directory casing', async () => { - expect(await ids(winEvents, projectScope('c:\\users\\JEFF\\Project-A'))).toEqual(['w1', 'w2']); - }); - - it('matches a UNC root whatever its case or separators', async () => { - const uncEvents: DashboardEvent[] = [ - makeEvent('\\\\Server\\Share\\Proj', 'u1'), - makeEvent('\\\\server\\share\\proj\\src', 'u2'), - makeEvent('\\\\server\\share\\other', 'u3'), - ]; - expect(await ids(uncEvents, projectScope('\\\\SERVER\\SHARE\\proj'))).toEqual(['u1', 'u2']); - }); +describe('adoptBareKeys', () => { + it('gives a snapshot entry keyed by the bare session ID to the first run of that ID', () => { + const reported = { 'pid-1': 3, 's2@t0': 1 }; + const adopted = adoptBareKeys(reported, ['pid-1@t1', 'pid-1@t2', 's2@t0', 's3@t3']); + expect(adopted['pid-1@t1']).toBe(3); + expect(adopted['pid-1@t2']).toBeUndefined(); + expect(adopted['s3@t3']).toBeUndefined(); }); - // A POSIX path is case-sensitive, and `\` is a legal character in a POSIX - // filename, so neither folding may be applied to one. - describe('POSIX paths keep their own rules', () => { - it('does not fold case', async () => { - expect(await ids(events, projectScope('/users/jeff/PROJECT-A'))).toEqual([]); - }); - - it('treats a backslash in a filename as part of the name', async () => { - const evts = [makeEvent('/work/a\\b', 'p1'), makeEvent('/work/a/b', 'p2')]; - expect(await ids(evts, projectScope('/work/a/b'))).toEqual(['p2']); - expect(await ids(evts, projectScope('/work/a\\b'))).toEqual(['p1']); - }); + it('never overrides a run\'s own entry', () => { + expect(adoptBareKeys({ 'pid-1': 3, 'pid-1@t1': 5 }, ['pid-1@t1'])['pid-1@t1']).toBe(5); }); }); diff --git a/src/__tests__/team-push-interventions.test.ts b/src/__tests__/team-push-interventions.test.ts index 39ff73e05..8a34b3da0 100644 --- a/src/__tests__/team-push-interventions.test.ts +++ b/src/__tests__/team-push-interventions.test.ts @@ -96,7 +96,8 @@ function writeDashboardEvents(lines: object[]): void { } describe('reportUsageToTeam — intervention reporting', () => { - function seedReport(): string { + /** The usage file, and the snapshot key of the seeded session under a scope config (its run ID, #785). */ + function seedReport(): { usagePath: string; run: string } { const timestamp = new Date().toISOString(); writeDashboardEvents([ { type: 'session_start', timestamp, sessionId: 'slow', tool: 'claude', cwd: '/p' }, @@ -107,12 +108,14 @@ describe('reportUsageToTeam — intervention reporting', () => { ]); const usagePath = path.join(tmpDir, '.teamai', 'user-usage.jsonl'); fs.writeFileSync(usagePath, JSON.stringify({ skill: 'review', timestamp, tool: 'claude' }) + '\n'); - return usagePath; + return { usagePath, run: `slow@${timestamp}` }; } it.each(['reports', 'legacy'])('finishes acknowledgement after the caller times out (%s)', async (backend) => { vi.useFakeTimers({ toFake: ['setTimeout', 'clearTimeout'] }); - const usagePath = seedReport(); + const { usagePath, run } = seedReport(); + // A caller without a scope config reads the whole log, under the bare session IDs. + const key = backend === 'reports' ? run : 'slow'; const seeded = fs.readFileSync(usagePath, 'utf-8'); let finish!: (value: boolean) => void; let started!: () => void; @@ -136,7 +139,7 @@ describe('reportUsageToTeam — intervention reporting', () => { await vi.advanceTimersByTimeAsync(5000); await timeout; expect(fs.readFileSync(usagePath, 'utf-8')).toContain('review'); - expect(reportedSnapshot('prompt-tokens', backend === 'reports').slow).toBeUndefined(); + expect(reportedSnapshot('prompt-tokens', backend === 'reports')[key]).toBeUndefined(); finish(true); expect(await operation).toBe(true); @@ -144,7 +147,7 @@ describe('reportUsageToTeam — intervention reporting', () => { // so it leaves the user-scope file for the scope that owns it. expect(fs.readFileSync(usagePath, 'utf-8')).toBe(backend === 'reports' ? '' : seeded); for (const name of ['interventions', 'prompt-tokens', 'daily-sessions']) { - expect(reportedSnapshot(name, backend === 'reports').slow).toBeDefined(); + expect(reportedSnapshot(name, backend === 'reports')[key]).toBeDefined(); } const statsPath = backend === 'reports' ? reportsStatsPath() : path.join(repoDir, 'stats', 'me.yaml'); const before = fs.readFileSync(statsPath, 'utf-8'); @@ -153,14 +156,14 @@ describe('reportUsageToTeam — intervention reporting', () => { }); it.each(['false', 'rejection'])('retains events and snapshots when push returns %s', async (failure) => { - const usagePath = seedReport(); + const { usagePath, run } = seedReport(); const before = fs.readFileSync(usagePath, 'utf-8'); if (failure === 'false') reportsMocks.updateReports.mockResolvedValueOnce(false); else reportsMocks.updateReports.mockRejectedValueOnce(new Error('offline')); expect(await reportUsageToTeam(repoDir, 'me', { selfConfig: gitConfig() })).toBe(false); expect(fs.readFileSync(usagePath, 'utf-8')).toBe(before); for (const name of ['interventions', 'prompt-tokens', 'daily-sessions']) { - expect(reportedSnapshot(name).slow).toBeUndefined(); + expect(reportedSnapshot(name)[run]).toBeUndefined(); } expect(await reportUsageToTeam(repoDir, 'me', { selfConfig: gitConfig() })).toBe(true); const stats = YAML.parse(fs.readFileSync(reportsStatsPath(), 'utf-8')); @@ -190,9 +193,9 @@ describe('reportUsageToTeam — intervention reporting', () => { // reported snapshot persisted so a second run reports nothing new expect(reportedSnapshot('interventions')).toEqual({ - s1: { interrupt: 2, toolReject: 1, correction: 0 }, + [`s1@${ts}`]: { interrupt: 2, toolReject: 1, correction: 0 }, }); - expect(reportedSnapshot('daily-sessions').s1).toMatchObject({ date: ts.slice(0, 10) }); + expect(reportedSnapshot('daily-sessions')[`s1@${ts}`]).toMatchObject({ date: ts.slice(0, 10) }); reportsMocks.updateReports.mockClear(); await reportUsageToTeam(repoDir, 'me', { selfConfig: gitConfig() }); diff --git a/src/stats.ts b/src/stats.ts index 8a226ec4a..6f788b355 100644 --- a/src/stats.ts +++ b/src/stats.ts @@ -156,11 +156,11 @@ async function unreportedDashboardStats( config: LocalConfig, ): Promise { const { - computeInterventionDelta, computePromptTokenDelta, readReportedInterventions, readReportedPromptTokens, + adoptBareKeys, computeInterventionDelta, computePromptTokenDelta, readReportedInterventions, readReportedPromptTokens, } = await import('./team-push.js'); // The scope's own snapshots, the ones its report compares against (#786). - const interventions = await readReportedInterventions(config); - const promptTokens = await readReportedPromptTokens(config); + const interventions = adoptBareKeys(await readReportedInterventions(config), metrics.keys()); + const promptTokens = adoptBareKeys(await readReportedPromptTokens(config), metrics.keys()); const interventionDelta = computeInterventionDelta( new Map([...metrics].map(([sid, m]) => [sid, { interrupt: m.interrupt, toolReject: m.toolReject, correction: m.correction }])), diff --git a/src/team-push.ts b/src/team-push.ts index 132f64b20..b9e729bb0 100644 --- a/src/team-push.ts +++ b/src/team-push.ts @@ -330,68 +330,21 @@ function hasDailyDelta(delta: ReturnType['delta'] ); } -/** - * A Windows-style root: a drive letter (`C:\`, `c:/`) or a UNC share - * (`\\server\share`). Tested per path rather than per platform, because the - * dashboard event log is shared — a team repo can hold events pushed from - * Windows and from Linux in the same file. - */ -const WINDOWS_ROOT = /^(?:[A-Za-z]:[\\/]|\\\\)/; - -type ScopeRoot = { key: string; windows: boolean }; - -/** - * Normalize a directory path for scope comparison. - * - * Both sides are native paths: `projectRoot` is stored as `path.resolve(cwd)` - * at init time, and an event's `cwd` is whatever the AI tool put in its hook - * payload. On Windows both use backslashes, so a literal `root + '/'` prefix - * can never match a subdirectory, and the two sources can also disagree on the - * case of the drive letter or of any directory along the way. Windows paths - * therefore get their separators unified and their case folded. - * - * POSIX paths keep both distinctions: they are case-sensitive, and `\` is a - * legal character in a POSIX filename, so `/work/a\b` and `/work/a/b` are two - * different directories and must not collapse onto one key. - * - * The root decides which set of rules applies to both sides, so a Windows root - * still matches a cwd the tool reported with forward slashes, and a POSIX root - * never has a backslash rewritten underneath it. - */ -function scopeKey(dir: string, windows: boolean): string { - if (!windows) return dir.replace(/\/+$/, ''); - return dir.replace(/\\/g, '/').replace(/\/+$/, '').toLowerCase(); -} - -function scopeRoot(dir: string): ScopeRoot { - const windows = WINDOWS_ROOT.test(dir); - return { key: scopeKey(dir, windows), windows }; -} - -/** True when `cwd` is the root itself or sits below it. */ -function isUnderScopeRoot(cwd: string, root: ScopeRoot): boolean { - const key = scopeKey(cwd, root.windows); - return key === root.key || key.startsWith(root.key + '/'); -} - /** * The dashboard events a scope reports (#785): every event of the sessions * recorded in it. A session is the run of one ID up to its session_end or - * process_exit, since a PID-fallback ID comes back for a later run; a later - * run is returned under an ID of its own, so it is counted on its own. It is + * process_exit, since a PID-fallback ID comes back for a later run; each run + * is returned under its own ID (see {@link adoptBareKeys}). It is * decided once, whole, by its first event keyed with a data home * (`dataHomeKey`, or the path an unreleased build of #795 wrote), because a * Stop carries the whole transcript's totals: split per event, a session that * moved into another scope would count there again the part recorded before * the move. A project also owns the key of its in-repo `.teamai`, where a * hook recorded until migration moved the project to a partition. A session - * written before events carried a key is decided by its first cwd: a - * project's when the root holds it, the user scope's when the directory still - * exists and resolves to the user scope now, the dispatcher's rule; one with - * no cwd, or a cwd gone since, is no scope's. A cwd is raw as the host sent - * it (a symlinked checkout, macOS `/tmp` vs `/private/tmp`), and so is a - * non-git project's root, so both are realpath'd while they still exist. A caller without a scope config reads the - * whole log. + * written before events carried a key is decided by its first cwd, by the + * dispatcher's rule: the scope that directory resolves to now, so a nested + * clone under a project is not the project's; one with no cwd, or a cwd gone + * since, is no scope's. A caller without a scope config reads the whole log. */ export async function filterEventsByScope( events: DashboardEvent[], @@ -405,18 +358,15 @@ export async function filterEventsByScope( const legacy = await dataHomeKey(path.join(config.projectRoot, '.teamai')); if (legacy !== (await dataHomeKey(path.join(getUserHome(), '.teamai')))) keys.add(legacy); } - const projectRoot = config.projectRoot; - const root = projectRoot ? scopeRoot(await fs.promises.realpath(projectRoot).catch(() => projectRoot)) : undefined; const { resolveConfigForDir } = await import('./config.js'); const resolvesHere = new Map>(); const ownsCwd = (cwd: string): Promise => { let owns = resolvesHere.get(cwd); if (!owns) { - owns = root ? fs.promises.realpath(cwd).then((real) => isUnderScopeRoot(real, root), () => false) - : pathExists(cwd).then(async (exists) => { - const resolved = exists ? await resolveConfigForDir(cwd) : null; - return !!resolved && (await dataHomeKey(getDataHome(resolved))) === ownKey; - }); + owns = pathExists(cwd).then(async (exists) => { + const resolved = exists ? await resolveConfigForDir(cwd) : null; + return !!resolved && keys.has(await dataHomeKey(getDataHome(resolved))); + }); resolvesHere.set(cwd, owns); } return owns; @@ -435,21 +385,18 @@ export async function filterEventsByScope( return key; })); // A session ID names one run until it ends: a PID-fallback ID comes back for - // a later run, maybe in another scope, so each run is decided on its own. - // A later run is also reported as its own session: it is returned under the - // ID plus its first event's timestamp. The first run keeps the bare ID, so - // the snapshots already written for it still match. + // a later run, maybe in another scope, so each run is decided on its own and + // returned under the ID plus its first event's timestamp, which stays the + // same whichever earlier runs compaction has dropped. const runOf: number[] = []; const openRun = new Map(); const deciding: Array = []; const runIds: string[] = []; - const seen = new Set(); events.forEach((e, i) => { let run = openRun.get(e.sessionId); if (run === undefined) { run = deciding.push(undefined) - 1; - runIds.push(seen.has(e.sessionId) ? `${e.sessionId}@${e.timestamp}` : e.sessionId); - seen.add(e.sessionId); + runIds.push(`${e.sessionId}@${e.timestamp}`); } openRun.set(e.sessionId, run); if (e.type === 'session_end' || e.type === 'process_exit') openRun.delete(e.sessionId); @@ -464,11 +411,28 @@ export async function filterEventsByScope( const cwd = events[i].cwd; return key !== undefined ? keys.has(key) : !!cwd && await ownsCwd(cwd); })); - return events.flatMap((e, i) => { - const run = runOf[i]; - if (!owned[run]) return []; - return runIds[run] === e.sessionId ? [e] : [{ ...e, sessionId: runIds[run] }]; - }); + return events.flatMap((e, i) => (owned[runOf[i]] ? [{ ...e, sessionId: runIds[runOf[i]] }] : [])); +} + +/** + * A reported snapshot as the run IDs of {@link filterEventsByScope} read it. + * Snapshots written before runs had their own IDs are keyed by the bare + * session ID; the first run of that ID in the log (`current` is in log order) + * takes that entry when it has none of its own. The next snapshot holds only + * run IDs, so a bare entry is read at most until the scope reports again. + */ +export function adoptBareKeys(reported: Record, current: Iterable): Record { + const adopted = { ...reported }; + const seen = new Set(); + for (const runId of current) { + const at = runId.lastIndexOf('@'); + if (at < 0) continue; + const id = runId.slice(0, at); + if (seen.has(id)) continue; + seen.add(id); + if (!Object.hasOwn(adopted, runId) && Object.hasOwn(reported, id)) adopted[runId] = reported[id]; + } + return adopted; } /** @@ -513,18 +477,18 @@ export async function reportUsageToTeam( const currentInterventions = new Map( [...metrics].map(([sid, m]) => [sid, { interrupt: m.interrupt, toolReject: m.toolReject, correction: m.correction }]), ); - const reportedInterventions = await readReportedInterventions(reportsConfig); + const reportedInterventions = adoptBareKeys(await readReportedInterventions(reportsConfig), metrics.keys()); const { delta: interventionDelta, nextReported } = computeInterventionDelta( currentInterventions, reportedInterventions, ); - const reportedPromptTokens = await readReportedPromptTokens(reportsConfig); + const reportedPromptTokens = adoptBareKeys(await readReportedPromptTokens(reportsConfig), metrics.keys()); const { delta: promptTokenDelta, nextReported: nextReportedPromptTokens } = computePromptTokenDelta( metrics, reportedPromptTokens, ); - const reportedDailySessions = await readReportedDailySessions(reportsConfig); + const reportedDailySessions = adoptBareKeys(await readReportedDailySessions(reportsConfig), metrics.keys()); const { delta: dailyDelta, nextReported: nextReportedDailySessions } = computeDailyStatsDelta( aggregateDailySessions(dashboardEvents), reportedDailySessions, From 78763ff9bc6b5e5647de01eb94937ae5cd29cb6f Mon Sep 17 00:00:00 2001 From: Saul Moro Date: Thu, 24 Sep 2026 15:49:52 +0200 Subject: [PATCH 11/41] fix(stats): address CI review (#785) adoptBareKeys() read a legacy bare `pid-N` snapshot entry as the first run's, but only in memory: the success writes merge into the file, and with nothing new to report nothing was written, so the bare entry stayed. Once compaction dropped that run, the next run reusing `pid-N` read it and was suppressed. The report now writes each snapshot as soon as a bare entry is retired, under the run ID only, even when there is no delta. --- docs/designs/data-directory-layout.md | 4 +-- src/__tests__/dashboard-report-scope.test.ts | 22 ++++++++++++++ src/__tests__/scope-filter.test.ts | 10 +++++++ src/team-push.ts | 30 +++++++++++++++----- 4 files changed, 57 insertions(+), 9 deletions(-) diff --git a/docs/designs/data-directory-layout.md b/docs/designs/data-directory-layout.md index 8c8127eff..6cc378c05 100644 --- a/docs/designs/data-directory-layout.md +++ b/docs/designs/data-directory-layout.md @@ -336,8 +336,8 @@ its `session_end` or `process_exit`, so a later run that reuses the ID (Copilot' fallback ID is the parent PID) is decided on its own. Every run is reported and snapshotted as `@`, which does not change when compaction drops earlier runs; a snapshot entry keyed by the bare ID (written -before) goes to the first run of that ID in the log, until the scope reports -again. A session written before that field existed is attributed by its first +before) is rewritten under the first run of that ID in the log the next time +the scope reports, and removed, so no later run of that ID reads it. A session written before that field existed is attributed by its first `cwd`: to the scope `resolveConfigForDir` resolves that directory to now, the dispatcher's rule, so a nested clone under a project is not the project's; no `cwd`, or one removed since, is no scope's. The snapshots of what was already reported are per scope diff --git a/src/__tests__/dashboard-report-scope.test.ts b/src/__tests__/dashboard-report-scope.test.ts index 427c8410f..42c364dff 100644 --- a/src/__tests__/dashboard-report-scope.test.ts +++ b/src/__tests__/dashboard-report-scope.test.ts @@ -352,6 +352,28 @@ describe('each scope keeps its own reported snapshot (#786)', () => { expect(await reportedPrompts(user)).toBe(2); }); + it.each([ + ['nothing new', 1], + ['a new prompt', 0], + ])('a bare snapshot entry a run adopts is written back under the run ID only, with %s to report', async (_, reported) => { + const { root, project } = await setup(); + const pid = `pid-${process.ppid}`; + await session('copilot', { cwd: root }); + // An earlier release reported this run under its bare session ID. + writeSharedSnapshots({ [pid]: reported }, new Date().toISOString().slice(0, 10)); + await report(project); + + for (const name of SNAPSHOTS) { + const own = JSON.parse(fs.readFileSync(path.join(getDataHome(project), 'dashboard', `reported-${name}.json`), 'utf-8')); + expect(Object.keys(own)).toEqual([expect.stringMatching(new RegExp(`^${pid}@`))]); + } + // Compaction dropped that run; the next one in P reuses its ID and is a new session. + fs.writeFileSync(path.join(teamaiHome(), 'dashboard', 'events.jsonl'), ''); + await session('copilot', { cwd: root }); + + expect(await reportedSessions(project)).toBe(1); + }); + it('a scope first seeded after a rollback skips what the earlier release reported', async () => { const { root, project } = await setup(); await session('claude', { session_id: 'rollback-p', cwd: root }); diff --git a/src/__tests__/scope-filter.test.ts b/src/__tests__/scope-filter.test.ts index e0d9e1a42..b34936420 100644 --- a/src/__tests__/scope-filter.test.ts +++ b/src/__tests__/scope-filter.test.ts @@ -190,4 +190,14 @@ describe('adoptBareKeys', () => { it('never overrides a run\'s own entry', () => { expect(adoptBareKeys({ 'pid-1': 3, 'pid-1@t1': 5 }, ['pid-1@t1'])['pid-1@t1']).toBe(5); }); + + it('retires the bare entry the first run of its ID reads, so it is never read again', () => { + expect(adoptBareKeys({ 'pid-1': 3, s2: 1 }, ['pid-1@t1', 'pid-1@t2'])).toEqual({ 'pid-1@t1': 3, s2: 1 }); + expect(adoptBareKeys({ 'pid-1': 3, 'pid-1@t1': 5 }, ['pid-1@t1'])).toEqual({ 'pid-1@t1': 5 }); + }); + + it('returns the snapshot itself when it holds no bare entry to retire', () => { + const reported = { 'pid-1@t1': 5, s2: 1 }; + expect(adoptBareKeys(reported, ['pid-1@t1', 's3@t3'])).toBe(reported); + }); }); diff --git a/src/team-push.ts b/src/team-push.ts index b9e729bb0..1e5d732e4 100644 --- a/src/team-push.ts +++ b/src/team-push.ts @@ -418,21 +418,26 @@ export async function filterEventsByScope( * A reported snapshot as the run IDs of {@link filterEventsByScope} read it. * Snapshots written before runs had their own IDs are keyed by the bare * session ID; the first run of that ID in the log (`current` is in log order) - * takes that entry when it has none of its own. The next snapshot holds only - * run IDs, so a bare entry is read at most until the scope reports again. + * takes that entry when it has none of its own, and the bare entry is retired + * either way, so no later run of that ID reads it. Returns `reported` itself + * when there is nothing to retire; otherwise the caller persists the result. */ export function adoptBareKeys(reported: Record, current: Iterable): Record { const adopted = { ...reported }; const seen = new Set(); + let retired = false; for (const runId of current) { const at = runId.lastIndexOf('@'); if (at < 0) continue; const id = runId.slice(0, at); if (seen.has(id)) continue; seen.add(id); - if (!Object.hasOwn(adopted, runId) && Object.hasOwn(reported, id)) adopted[runId] = reported[id]; + if (!Object.hasOwn(reported, id)) continue; + if (!Object.hasOwn(adopted, runId)) adopted[runId] = reported[id]; + delete adopted[id]; + retired = true; } - return adopted; + return retired ? adopted : reported; } /** @@ -477,18 +482,29 @@ export async function reportUsageToTeam( const currentInterventions = new Map( [...metrics].map(([sid, m]) => [sid, { interrupt: m.interrupt, toolReject: m.toolReject, correction: m.correction }]), ); - const reportedInterventions = adoptBareKeys(await readReportedInterventions(reportsConfig), metrics.keys()); + // A retired bare entry is written out now, even with nothing to report, so + // the success writes below, which merge into the file, cannot bring it back. + const adopt = async ( + read: (config: LocalConfig | undefined) => Promise>, + write: (data: Record, config: LocalConfig | undefined) => Promise, + ): Promise> => { + const stored = await read(reportsConfig); + const adopted = adoptBareKeys(stored, metrics.keys()); + if (adopted !== stored) await write(adopted, reportsConfig); + return adopted; + }; + const reportedInterventions = await adopt(readReportedInterventions, writeReportedInterventions); const { delta: interventionDelta, nextReported } = computeInterventionDelta( currentInterventions, reportedInterventions, ); - const reportedPromptTokens = adoptBareKeys(await readReportedPromptTokens(reportsConfig), metrics.keys()); + const reportedPromptTokens = await adopt(readReportedPromptTokens, writeReportedPromptTokens); const { delta: promptTokenDelta, nextReported: nextReportedPromptTokens } = computePromptTokenDelta( metrics, reportedPromptTokens, ); - const reportedDailySessions = adoptBareKeys(await readReportedDailySessions(reportsConfig), metrics.keys()); + const reportedDailySessions = await adopt(readReportedDailySessions, writeReportedDailySessions); const { delta: dailyDelta, nextReported: nextReportedDailySessions } = computeDailyStatsDelta( aggregateDailySessions(dashboardEvents), reportedDailySessions, From bde86040f22e9e9e51dfb241ecf1b171b4139931 Mon Sep 17 00:00:00 2001 From: Saul Moro Date: Thu, 24 Sep 2026 17:00:41 +0200 Subject: [PATCH 12/41] fix(stats): address CI review (#785) A bare snapshot entry is given only to a run an earlier release recorded (its first event has no dataHomeKey). Only earlier releases wrote bare entries, and a seeded one may be another scope's run under a reused PID-fallback ID, so a run this release recorded takes none. A marker of the seed time would miss the common case: a scope seeds at its first report, usually the pull its first session's SessionStart triggers. A second end of a run with nothing recorded since the first (the dashboard monitor's process_exit after SessionEnd) joins the run it closed instead of opening a terminal-only run counted as a session. --- CHANGELOG.md | 2 +- docs/designs/data-directory-layout.md | 9 ++- src/__tests__/dashboard-report-scope.test.ts | 84 ++++++++++++++++++-- src/__tests__/scope-filter.test.ts | 45 +++++++++-- src/stats.ts | 13 +-- src/team-push.ts | 33 +++++--- 6 files changed, 156 insertions(+), 30 deletions(-) diff --git a/CHANGELOG.md b/CHANGELOG.md index 2bb21ddf4..6ee90fcb3 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -41,7 +41,7 @@ All notable changes to this project will be documented in this file. See [standa - A `manifest/roles.yaml` that exists but does not parse now fails the pull for that scope instead of warning and syncing with no role filter at all, for a member with no role as much as for one with a role. The same applies to `init` and `push`, which each fell back to a guess at the namespaces when any error came out of the loader. The legacy role migration skips with a warning instead of failing, so every command, `pull` included, still loads the config and can fetch the fixed manifest; until it can run, the member holds no role rather than every role, so hooks, MCP servers and env variables scoped by `roles:` reach them no more than skills do. For a member with no active project that fallback meant an unfiltered sync, so a broken manifest delivered every namespace it was written to gate. Only an absent manifest still means "this team does not use roles"; an unreadable or empty file is an error, as it now is for `manifest/projects.yaml` too. `push` stops at its scan for such a manifest (exit 2) even with `--role `, since the scan needs it to tell which namespaces are the member's. - `teamai members list` and `teamai projects members` read the roster registered before the reports switch, so a team upgrading past the orphan-branch split no longer sees "No team members registered" while its `members/` still lives on the default branch. The default-branch copy becomes a read-only inherited root, the way learnings' already was: listed in union with the `teamai-reports` copy, with the branch copy winning when the same file exists on both; nothing is copied or deleted, and a cold `members list` still does not publish the reports branch. Member registration merges against the inherited copy too, so a re-init keeps the original `registeredAt` and projects. Fixes [#735](https://github.com/Tencent/teamai-cli/issues/735). - Cache GC now rejects partial integers such as `12abc`, decimals, zero and unsafe integers for `--max-bytes` and `--stale-days` before deleting anything. An invalid `TEAMAI_CACHE_MAX_BYTES` value falls back to the default 5 GB limit instead of using a numeric prefix. -- Each scope reports only the dashboard sessions recorded in it. Every scope read one machine-wide event log and picked its sessions out by the `cwd` they started in, so a user-scope pull reported every project's sessions to the user team, a project never reported its Copilot sessions (teamai does not record Copilot's `cwd`), and a session started under a symlinked path (or `/tmp` for `/private/tmp` on macOS) never matched its project. Each event now records a key (a hash, so no path is stored) of the data directory of the scope `hook-dispatch` resolved, and a report keeps only its own scope's. A session in a directory that resolves to no project (a subdirectory of a non-git project, a git submodule or nested clone) is the user scope's, as for skill usage. A session is reported once, whole, by the scope it started in, even after a `cd` into another project: its Stop carries the whole transcript's totals, so reporting its later part elsewhere would count them twice. A session ends with its session end or process exit, so a later session that reuses its ID (Copilot's fallback ID is the parent process ID) is attributed and counted on its own. Events recorded by an earlier release go to the scope their `cwd` resolves to now (a nested clone under a project is not the project's); events with no `cwd`, or one removed since, are reported by no one. The usage guide explains how to remove by hand a skill an earlier release reported into `stats/.yaml` from another project (for [#785](https://github.com/Tencent/teamai-cli/issues/785)). +- Each scope reports only the dashboard sessions recorded in it. Every scope read one machine-wide event log and picked its sessions out by the `cwd` they started in, so a user-scope pull reported every project's sessions to the user team, a project never reported its Copilot sessions (teamai does not record Copilot's `cwd`), and a session started under a symlinked path (or `/tmp` for `/private/tmp` on macOS) never matched its project. Each event now records a key (a hash, so no path is stored) of the data directory of the scope `hook-dispatch` resolved, and a report keeps only its own scope's. A session in a directory that resolves to no project (a subdirectory of a non-git project, a git submodule or nested clone) is the user scope's, as for skill usage. A session is reported once, whole, by the scope it started in, even after a `cd` into another project: its Stop carries the whole transcript's totals, so reporting its later part elsewhere would count them twice. A session ends with its session end or process exit (a process exit recorded right after its session end is the same session), so a later session that reuses its ID (Copilot's fallback ID is the parent process ID) is attributed and counted on its own, even when an earlier release reported that ID for another scope. Events recorded by an earlier release go to the scope their `cwd` resolves to now (a nested clone under a project is not the project's); events with no `cwd`, or one removed since, are reported by no one. The usage guide explains how to remove by hand a skill an earlier release reported into `stats/.yaml` from another project (for [#785](https://github.com/Tencent/teamai-cli/issues/785)). - Each scope keeps its own snapshots of the dashboard sessions it already reported. They were machine-wide and keyed by session ID, so a session ID reused in another scope (Copilot's fallback ID is the parent process ID) was taken as already reported there and sent nothing. Each scope now keeps its own (`/dashboard/reported-*.json`, and `~/.teamai/dashboard/user-reported-*.json` for the user scope), seeded the first time from the shared `~/.teamai/dashboard/reported-*.json`, so the first report after the upgrade sends nothing already reported. The shared file is no longer written; after a rollback an earlier release writes it again, and only a scope not yet seeded reads it (for [#786](https://github.com/Tencent/teamai-cli/issues/786)). - Usage reporting scopes sessions by path on Windows too. The project/user scope filter compared an event's `cwd` against `projectRoot` with a hard-coded `/` separator, so on Windows only a session started in the project root itself matched: every session started in a subdirectory was dropped from the project team's report and counted in the user scope's instead, which is the isolation the usage guide promises. Windows paths are also compared case-insensitively, so a drive letter or a directory name spelled with different case in the two sources no longer leaks a project session into the user scope. POSIX paths keep their own rules: case-sensitive, and a `\` in a filename stays part of the name. - `teamai tags subscribe` and `teamai tags unsubscribe` now invalidate the pull revision cache, as `teamai skill exclude` already does, so the next `teamai pull` applies the new subscriptions instead of reporting "Already synced" when the team repo has not changed. diff --git a/docs/designs/data-directory-layout.md b/docs/designs/data-directory-layout.md index 0952eb9e4..6be6cbe6e 100644 --- a/docs/designs/data-directory-layout.md +++ b/docs/designs/data-directory-layout.md @@ -353,11 +353,16 @@ scope's report keeps the sessions whose first keyed event is its own, whole: a Stop carries the whole transcript's totals, so a session that moved scope (a `cd` mid-session) is reported once, where it started. A session is one ID's run up to its `session_end` or `process_exit`, so a later run that reuses the ID (Copilot's -fallback ID is the parent PID) is decided on its own. Every run is reported and +fallback ID is the parent PID) is decided on its own; a second end with nothing +recorded since the first (the dashboard monitor's `process_exit` after +`SessionEnd`) belongs to the run it closed. Every run is reported and snapshotted as `@`, which does not change when compaction drops earlier runs; a snapshot entry keyed by the bare ID (written before) is rewritten under the first run of that ID in the log the next time -the scope reports, and removed, so no later run of that ID reads it. A session written before that field existed is attributed by its first +the scope reports, and removed, so no later run of that ID reads it. Only an +earlier release wrote bare entries, and a seeded one may be another scope's, so +a run whose first event carries `dataHomeKey` (recorded by this release) takes +none. A session written before that field existed is attributed by its first `cwd`: to the scope `resolveConfigForDir` resolves that directory to now, the dispatcher's rule, so a nested clone under a project is not the project's; no `cwd`, or one removed since, is no scope's. The snapshots of what was already reported are per scope diff --git a/src/__tests__/dashboard-report-scope.test.ts b/src/__tests__/dashboard-report-scope.test.ts index 42c364dff..7a8ffc99d 100644 --- a/src/__tests__/dashboard-report-scope.test.ts +++ b/src/__tests__/dashboard-report-scope.test.ts @@ -200,6 +200,14 @@ describe('each scope keeps its own reported snapshot (#786)', () => { return stats && typeof stats === 'object' && 'prompts' in stats && typeof stats.prompts === 'number' ? stats.prompts : 0; } + /** The sessions a scope's intervention stats hold, 0 before its first push. */ + async function reportedInterventionSessions(config: LocalConfig): Promise { + const stats = await report(config); + const interventions = stats && typeof stats === 'object' && 'interventions' in stats ? stats.interventions : undefined; + return interventions && typeof interventions === 'object' && 'sessions' in interventions + && typeof interventions.sessions === 'number' ? interventions.sessions : 0; + } + const shared = (name: string) => path.join(teamaiHome(), 'dashboard', `reported-${name}.json`); const SNAPSHOTS = ['interventions', 'prompt-tokens', 'daily-sessions']; @@ -223,6 +231,21 @@ describe('each scope keeps its own reported snapshot (#786)', () => { } } + /** + * Run `record` as an earlier release would have: before #785 its events carry + * no data home, and main since #795 wrote the data home path, `dataHome`. + */ + async function asEarlierRelease(record: () => Promise, dataHome?: string): Promise { + const log = path.join(teamaiHome(), 'dashboard', 'events.jsonl'); + const before = fs.existsSync(log) ? fs.readFileSync(log, 'utf-8') : ''; + await record(); + const added = fs.readFileSync(log, 'utf-8').slice(before.length).split('\n').filter(Boolean).map((line) => { + const event = Object.fromEntries(Object.entries(JSON.parse(line)).filter(([field]) => field !== 'dataHomeKey')); + return `${JSON.stringify(dataHome === undefined ? event : { ...event, dataHome })}\n`; + }); + fs.writeFileSync(log, before + added.join('')); + } + // A Stop carries the whole transcript's totals, so a session is reported once, // whole, by the scope it started in; split per event, the later scope would // count the earlier scope's part again. @@ -314,12 +337,63 @@ describe('each scope keeps its own reported snapshot (#786)', () => { expect(await reportedPrompts(project)).toBe(2); }); + it('a session that ends twice, SessionEnd then the dashboard monitor\'s process_exit, is one session', async () => { + const { root, project } = await setup(); + await session('copilot', { cwd: root }); + await hook('session-end', 'copilot', { cwd: root, hook_event_name: 'SessionEnd' }); + // The monitor read the session as still running while SessionEnd was appended. + const log = path.join(teamaiHome(), 'dashboard', 'events.jsonl'); + const last = Object.fromEntries(Object.entries(JSON.parse(fs.readFileSync(log, 'utf-8').trim().split('\n').at(-1) ?? '{}'))); + fs.appendFileSync(log, `${JSON.stringify({ ...last, type: 'process_exit', timestamp: new Date(Date.now() + 1000).toISOString() })}\n`); + + expect(await reportedInterventionSessions(project)).toBe(1); + expect(await reportedSessions(project)).toBe(1); + }); + + it.each([ + ['after the project first reported', true], + ['before the project first reported', false], + ])('a session ID an earlier release reported for the user scope is a new session when this build records it in a project, %s', async (_, reportFirst) => { + const { root, project } = await setup(); + const elsewhere = path.join(tmp, 'elsewhere'); + fs.mkdirSync(elsewhere); + const pid = `pid-${process.ppid}`; + // The ended user-scope run stays in the log, below the compaction threshold. + await asEarlierRelease(async () => { + await session('copilot', { cwd: elsewhere }); + await hook('session-end', 'copilot', { cwd: elsewhere, hook_event_name: 'SessionEnd' }); + }); + writeSharedSnapshots({ [pid]: 1 }, new Date().toISOString().slice(0, 10)); + if (reportFirst) expect(await report(project)).toBeNull(); + await session('copilot', { cwd: root }); + + expect(await reportedSessions(project)).toBe(1); + expect(await reportedPrompts(project)).toBe(1); + }); + + it('a run in progress across the upgrade is not reported again for what the earlier release reported', async () => { + const { root, project } = await setup(); + const pid = `pid-${process.ppid}`; + await asEarlierRelease(async () => { + await hook('session-start', 'copilot', { cwd: root, hook_event_name: 'SessionStart' }); + await prompts(pid, root, 1); + }); + writeSharedSnapshots({ [pid]: 1 }, new Date().toISOString().slice(0, 10)); + await hook('prompt-submit', 'copilot', { cwd: root, hook_event_name: 'UserPromptSubmit', prompt: 'after the upgrade' }); + await hook('stop', 'copilot', { cwd: root, hook_event_name: 'Stop' }); + + expect(await reportedPrompts(project)).toBe(1); + expect(await reportedInterventionSessions(project)).toBe(0); + }); + it('the first report after the upgrade sends nothing a shared snapshot already reported', async () => { const { root, user, project } = await setup(); const elsewhere = path.join(tmp, 'elsewhere'); fs.mkdirSync(elsewhere); - await session('claude', { session_id: 'old-p', cwd: root }); - await session('claude', { session_id: 'old-u', cwd: elsewhere }); + await asEarlierRelease(async () => { + await session('claude', { session_id: 'old-p', cwd: root }); + await session('claude', { session_id: 'old-u', cwd: elsewhere }); + }); writeSharedSnapshots({ 'old-p': 1, 'old-u': 1 }, new Date().toISOString().slice(0, 10)); expect(await report(user)).toBeNull(); @@ -358,8 +432,8 @@ describe('each scope keeps its own reported snapshot (#786)', () => { ])('a bare snapshot entry a run adopts is written back under the run ID only, with %s to report', async (_, reported) => { const { root, project } = await setup(); const pid = `pid-${process.ppid}`; - await session('copilot', { cwd: root }); - // An earlier release reported this run under its bare session ID. + // Main since #795 recorded and reported this run under its bare session ID. + await asEarlierRelease(() => session('copilot', { cwd: root }), getDataHome(project)); writeSharedSnapshots({ [pid]: reported }, new Date().toISOString().slice(0, 10)); await report(project); @@ -376,7 +450,7 @@ describe('each scope keeps its own reported snapshot (#786)', () => { it('a scope first seeded after a rollback skips what the earlier release reported', async () => { const { root, project } = await setup(); - await session('claude', { session_id: 'rollback-p', cwd: root }); + await asEarlierRelease(() => session('claude', { session_id: 'rollback-p', cwd: root })); writeSharedSnapshots({ 'rollback-p': 1 }, new Date().toISOString().slice(0, 10)); expect(await report(project)).toBeNull(); diff --git a/src/__tests__/scope-filter.test.ts b/src/__tests__/scope-filter.test.ts index b34936420..cb5199d69 100644 --- a/src/__tests__/scope-filter.test.ts +++ b/src/__tests__/scope-filter.test.ts @@ -88,6 +88,23 @@ describe('filterEventsByScope', () => { } }); + it('a second end of a run, with nothing recorded between the two, joins the run that just closed', async () => { + // The dashboard monitor appends process_exit after the SessionEnd hook's session_end. + const p = '/home/jeff/.teamai/projects/p'; + const at = (e: DashboardEvent, timestamp: string): DashboardEvent => ({ ...e, timestamp }); + const log = [ + at(await scopedEvent(undefined, 'pid-1', p), '2026-01-01T00:00:00.000Z'), + at({ ...(await scopedEvent(undefined, 'pid-1', p)), type: 'session_end' }, '2026-01-01T00:01:00.000Z'), + at({ ...(await scopedEvent(undefined, 'pid-1', p)), type: 'process_exit' }, '2026-01-01T00:01:05.000Z'), + at(await scopedEvent(undefined, 'pid-1', p), '2026-01-01T00:02:00.000Z'), + at({ ...(await scopedEvent(undefined, 'pid-1', p)), type: 'process_exit' }, '2026-01-01T00:03:00.000Z'), + ]; + expect((await filterEventsByScope(log, projectScope('/Users/jeff/project-a'))).map((e) => e.sessionId)).toEqual([ + 'pid-1@2026-01-01T00:00:00.000Z', 'pid-1@2026-01-01T00:00:00.000Z', 'pid-1@2026-01-01T00:00:00.000Z', + 'pid-1@2026-01-01T00:02:00.000Z', 'pid-1@2026-01-01T00:02:00.000Z', + ]); + }); + it('a run keeps its ID after compaction drops the earlier runs of its ID', async () => { const ended = [ await scopedEvent(undefined, 'pid-1', '/home/jeff/.teamai/projects/p'), @@ -179,25 +196,43 @@ describe('filterEventsByScope', () => { }); describe('adoptBareKeys', () => { + /** A run's event as filterEventsByScope returns it; an earlier release's unless `fields` key it. */ + const run = (sessionId: string, fields: Partial = {}): DashboardEvent => + ({ type: 'prompt_submit', timestamp: 't', sessionId, tool: 'copilot', ...fields }); + it('gives a snapshot entry keyed by the bare session ID to the first run of that ID', () => { const reported = { 'pid-1': 3, 's2@t0': 1 }; - const adopted = adoptBareKeys(reported, ['pid-1@t1', 'pid-1@t2', 's2@t0', 's3@t3']); + const adopted = adoptBareKeys(reported, [run('pid-1@t1'), run('pid-1@t2'), run('s2@t0'), run('s3@t3')]); expect(adopted['pid-1@t1']).toBe(3); expect(adopted['pid-1@t2']).toBeUndefined(); expect(adopted['s3@t3']).toBeUndefined(); }); it('never overrides a run\'s own entry', () => { - expect(adoptBareKeys({ 'pid-1': 3, 'pid-1@t1': 5 }, ['pid-1@t1'])['pid-1@t1']).toBe(5); + expect(adoptBareKeys({ 'pid-1': 3, 'pid-1@t1': 5 }, [run('pid-1@t1')])['pid-1@t1']).toBe(5); }); it('retires the bare entry the first run of its ID reads, so it is never read again', () => { - expect(adoptBareKeys({ 'pid-1': 3, s2: 1 }, ['pid-1@t1', 'pid-1@t2'])).toEqual({ 'pid-1@t1': 3, s2: 1 }); - expect(adoptBareKeys({ 'pid-1': 3, 'pid-1@t1': 5 }, ['pid-1@t1'])).toEqual({ 'pid-1@t1': 5 }); + expect(adoptBareKeys({ 'pid-1': 3, s2: 1 }, [run('pid-1@t1'), run('pid-1@t2')])).toEqual({ 'pid-1@t1': 3, s2: 1 }); + expect(adoptBareKeys({ 'pid-1': 3, 'pid-1@t1': 5 }, [run('pid-1@t1')])).toEqual({ 'pid-1@t1': 5 }); }); it('returns the snapshot itself when it holds no bare entry to retire', () => { const reported = { 'pid-1@t1': 5, s2: 1 }; - expect(adoptBareKeys(reported, ['pid-1@t1', 's3@t3'])).toBe(reported); + expect(adoptBareKeys(reported, [run('pid-1@t1'), run('s3@t3')])).toBe(reported); + }); + + it('gives no bare entry to a run this build recorded: only an earlier release wrote them', () => { + // Another scope's run, maybe: the bare entry says nothing about which scope reported it. + const reported = { 'pid-1': 3 }; + expect(adoptBareKeys(reported, [run('pid-1@t1', { dataHomeKey: 'k' }), run('pid-1@t2')])).toBe(reported); + }); + + it('a run recorded by main since #795, which wrote the data home path, is an earlier release\'s', () => { + expect(adoptBareKeys({ 'pid-1': 3 }, [run('pid-1@t1', { dataHome: '/home/jeff/.teamai' })])).toEqual({ 'pid-1@t1': 3 }); + }); + + it('a run in progress across the upgrade is an earlier release\'s, by its first event', () => { + expect(adoptBareKeys({ 'pid-1': 3 }, [run('pid-1@t1'), run('pid-1@t1', { dataHomeKey: 'k' })])).toEqual({ 'pid-1@t1': 3 }); }); }); diff --git a/src/stats.ts b/src/stats.ts index 6f788b355..adbad9a29 100644 --- a/src/stats.ts +++ b/src/stats.ts @@ -7,7 +7,7 @@ import { readEvents, aggregateSessionMetrics } from './dashboard-collector.js'; import { totalTokens, addTokenUsage, emptyTokenUsage } from './types.js'; import { attributeByRepo, timeAnalytics, renderHourSparkline } from './session-analytics.js'; import { formatTokenCount } from './digest.js'; -import type { UsageEvent, UserStats, TokenUsage, SessionMetrics, LocalConfig } from './types.js'; +import type { UsageEvent, UserStats, TokenUsage, SessionMetrics, LocalConfig, DashboardEvent } from './types.js'; interface SkillStats { name: string; @@ -148,10 +148,11 @@ function aggregateDashboardStats(metrics: Map): Aggregat * per-session delta `teamai pull` pushes, so the displayed total is * reported + unreported rather than reported + everything. * - * The caller passes the scope's own metrics, already filtered the way the - * report path filters them. + * The caller passes the scope's own events and their metrics, already + * filtered the way the report path filters them. */ async function unreportedDashboardStats( + events: DashboardEvent[], metrics: Map, config: LocalConfig, ): Promise { @@ -159,8 +160,8 @@ async function unreportedDashboardStats( adoptBareKeys, computeInterventionDelta, computePromptTokenDelta, readReportedInterventions, readReportedPromptTokens, } = await import('./team-push.js'); // The scope's own snapshots, the ones its report compares against (#786). - const interventions = adoptBareKeys(await readReportedInterventions(config), metrics.keys()); - const promptTokens = adoptBareKeys(await readReportedPromptTokens(config), metrics.keys()); + const interventions = adoptBareKeys(await readReportedInterventions(config), events); + const promptTokens = adoptBareKeys(await readReportedPromptTokens(config), events); const interventionDelta = computeInterventionDelta( new Map([...metrics].map(([sid, m]) => [sid, { interrupt: m.interrupt, toolReject: m.toolReject, correction: m.correction }])), @@ -251,7 +252,7 @@ export async function showStats(options: ShowStatsOptions = {}): Promise { || totalTokens(reported.tokens ?? emptyTokenUsage()) > 0 ); const localDashboard = config && teamHasReported - ? await unreportedDashboardStats(metricsMap, config) + ? await unreportedDashboardStats(scopedEvents, metricsMap, config) : aggregateDashboardStats(metricsMap); const dashboard = mergeDashboardAndReported(localDashboard, reported); const hasDashboardData = diff --git a/src/team-push.ts b/src/team-push.ts index 1e5d732e4..f35c8fe2a 100644 --- a/src/team-push.ts +++ b/src/team-push.ts @@ -388,18 +388,26 @@ export async function filterEventsByScope( // a later run, maybe in another scope, so each run is decided on its own and // returned under the ID plus its first event's timestamp, which stays the // same whichever earlier runs compaction has dropped. + // A second end with nothing recorded since the first (the dashboard + // monitor's process_exit after SessionEnd) belongs to the run just closed. const runOf: number[] = []; const openRun = new Map(); + const closedRun = new Map(); const deciding: Array = []; const runIds: string[] = []; events.forEach((e, i) => { - let run = openRun.get(e.sessionId); + const ends = e.type === 'session_end' || e.type === 'process_exit'; + let run = openRun.get(e.sessionId) ?? (ends ? closedRun.get(e.sessionId) : undefined); if (run === undefined) { run = deciding.push(undefined) - 1; runIds.push(`${e.sessionId}@${e.timestamp}`); } - openRun.set(e.sessionId, run); - if (e.type === 'session_end' || e.type === 'process_exit') openRun.delete(e.sessionId); + if (ends) { + openRun.delete(e.sessionId); + closedRun.set(e.sessionId, run); + } else { + openRun.set(e.sessionId, run); + } runOf.push(run); const current = deciding[run]; if (eventKeys[i] !== undefined ? current === undefined || eventKeys[current] === undefined @@ -417,22 +425,25 @@ export async function filterEventsByScope( /** * A reported snapshot as the run IDs of {@link filterEventsByScope} read it. * Snapshots written before runs had their own IDs are keyed by the bare - * session ID; the first run of that ID in the log (`current` is in log order) - * takes that entry when it has none of its own, and the bare entry is retired - * either way, so no later run of that ID reads it. Returns `reported` itself - * when there is nothing to retire; otherwise the caller persists the result. + * session ID; the first run of that ID in `events` (the scope's, as that + * filter returns them) takes that entry when it has none of its own, and the + * bare entry is retired either way, so no later run of that ID reads it. Only + * an earlier release wrote bare entries, and only for runs it recorded, so a + * run whose first event carries a `dataHomeKey` takes none: the entry may be + * another scope's run under a reused PID-fallback ID. Returns `reported` + * itself when there is nothing to retire; otherwise the caller persists the result. */ -export function adoptBareKeys(reported: Record, current: Iterable): Record { +export function adoptBareKeys(reported: Record, events: Iterable): Record { const adopted = { ...reported }; const seen = new Set(); let retired = false; - for (const runId of current) { + for (const { sessionId: runId, dataHomeKey } of events) { const at = runId.lastIndexOf('@'); if (at < 0) continue; const id = runId.slice(0, at); if (seen.has(id)) continue; seen.add(id); - if (!Object.hasOwn(reported, id)) continue; + if (typeof dataHomeKey === 'string' || !Object.hasOwn(reported, id)) continue; if (!Object.hasOwn(adopted, runId)) adopted[runId] = reported[id]; delete adopted[id]; retired = true; @@ -489,7 +500,7 @@ export async function reportUsageToTeam( write: (data: Record, config: LocalConfig | undefined) => Promise, ): Promise> => { const stored = await read(reportsConfig); - const adopted = adoptBareKeys(stored, metrics.keys()); + const adopted = adoptBareKeys(stored, dashboardEvents); if (adopted !== stored) await write(adopted, reportsConfig); return adopted; }; From f37dd22559798249594b234f9a26ec53d80af588 Mon Sep 17 00:00:00 2001 From: Saul Moro Date: Thu, 24 Sep 2026 17:28:31 +0200 Subject: [PATCH 13/41] fix(stats): address CI review (#785) A scope's first snapshot is seeded only with the shared entries of its own runs in the log, under their run IDs, and none for a run recorded with a dataHome path: that release already kept per-scope snapshots, so a shared entry under the same ID is another scope's. An unmatched entry is dropped instead of copied, so a later reuse of the ID cannot inherit it. The dashboard monitor records processExitAfter, the last event it observed, and the scope filter closes only that run. A delayed exit appended after the next run of the same ID began no longer ends it and splits it in two; an exit whose run compaction dropped is ignored. --- docs/designs/data-directory-layout.md | 13 ++- src/__tests__/dashboard-report-scope.test.ts | 51 +++++++- src/__tests__/e2e/report-scope-reuse.test.ts | 115 +++++++++++++++++++ src/__tests__/scope-filter.test.ts | 26 +++++ src/dashboard.ts | 1 + src/team-push.ts | 69 +++++++---- src/types.ts | 2 + 7 files changed, 250 insertions(+), 27 deletions(-) create mode 100644 src/__tests__/e2e/report-scope-reuse.test.ts diff --git a/docs/designs/data-directory-layout.md b/docs/designs/data-directory-layout.md index 6be6cbe6e..c9a8b0fc8 100644 --- a/docs/designs/data-directory-layout.md +++ b/docs/designs/data-directory-layout.md @@ -355,7 +355,10 @@ mid-session) is reported once, where it started. A session is one ID's run up to its `session_end` or `process_exit`, so a later run that reuses the ID (Copilot's fallback ID is the parent PID) is decided on its own; a second end with nothing recorded since the first (the dashboard monitor's `process_exit` after -`SessionEnd`) belongs to the run it closed. Every run is reported and +`SessionEnd`) belongs to the run it closed. The monitor's `process_exit` also +records `processExitAfter`, the last event it observed, and closes only that +run: an exit appended after the next run of the same ID began does not end it, +and one whose run compaction dropped is ignored. Every run is reported and snapshotted as `@`, which does not change when compaction drops earlier runs; a snapshot entry keyed by the bare ID (written before) is rewritten under the first run of that ID in the log the next time @@ -369,9 +372,13 @@ dispatcher's rule, so a nested clone under a project is not the project's; no too (#786), because a session ID can recur in another scope (Copilot's fallback ID is the parent PID): `/dashboard/reported-*.json`, and `~/.teamai/dashboard/user-reported-*.json` -for the user scope. The first time a scope needs one it copies the shared +for the user scope. The first time a scope needs one it seeds it from the shared `~/.teamai/dashboard/reported-*.json`, so nothing reported before the upgrade is -sent again; after that it reads only its own. The shared file is no longer +sent again; after that it reads only its own. The seed takes only the entries of +the scope's runs in the log, under their run IDs, and none for a run recorded +with a `dataHome` path: that release already kept per-scope snapshots, so a +shared entry under its ID is another scope's. An unmatched entry is dropped, so a +later reuse of the ID cannot inherit it. The shared file is no longer written, except by an earlier release after a rollback, so every scope seeds from what the machine had reported by then, never from another scope's later report. The seed holds a session's whole total, so a session still running at the diff --git a/src/__tests__/dashboard-report-scope.test.ts b/src/__tests__/dashboard-report-scope.test.ts index 7a8ffc99d..b935d3fd8 100644 --- a/src/__tests__/dashboard-report-scope.test.ts +++ b/src/__tests__/dashboard-report-scope.test.ts @@ -212,7 +212,7 @@ describe('each scope keeps its own reported snapshot (#786)', () => { const SNAPSHOTS = ['interventions', 'prompt-tokens', 'daily-sessions']; /** The shared snapshots as a release before #786 left them, with `prompts` reported per session. */ - function writeSharedSnapshots(prompts: Record, date: string): void { + function writeSharedSnapshots(prompts: Record, date: string, config?: LocalConfig): void { const entries = (value: (n: number) => unknown) => Object.fromEntries(Object.entries(prompts).map(([sid, n]) => [sid, value(n)])); const values: Record = { @@ -221,8 +221,9 @@ describe('each scope keeps its own reported snapshot (#786)', () => { // Longer than any fixture session, so no duration is left to report. 'daily-sessions': entries((n) => ({ date, prompts: n, durationMs: 3_600_000, succeeded: 1, corrected: 0 })), }; - fs.mkdirSync(path.join(teamaiHome(), 'dashboard'), { recursive: true }); - for (const name of SNAPSHOTS) fs.writeFileSync(shared(name), JSON.stringify(values[name])); + const dir = path.join(config ? getDataHome(config) : teamaiHome(), 'dashboard'); + fs.mkdirSync(dir, { recursive: true }); + for (const name of SNAPSHOTS) fs.writeFileSync(path.join(dir, `reported-${name}.json`), JSON.stringify(values[name])); } async function prompts(sessionId: string, cwd: string, count: number): Promise { @@ -386,6 +387,47 @@ describe('each scope keeps its own reported snapshot (#786)', () => { expect(await reportedInterventionSessions(project)).toBe(0); }); + it.each([ + ['retained', false, false], + ['compacted', true, false], + ['seeded before reuse', false, true], + ])('a path-keyed project run does not inherit a shared user snapshot (%s)', async (_, compact, seedFirst) => { + const { root, project } = await setup(); + const pid = `pid-${process.ppid}`; + await asEarlierRelease(async () => { + await session('copilot', { cwd: tmp }); + await hook('session-end', 'copilot', { cwd: tmp, hook_event_name: 'SessionEnd' }); + }); + writeSharedSnapshots({ [pid]: 1 }, new Date().toISOString().slice(0, 10)); + if (seedFirst) expect(await report(project)).toBeNull(); + if (compact) fs.writeFileSync(path.join(teamaiHome(), 'dashboard', 'events.jsonl'), ''); + await asEarlierRelease(() => session('copilot', { cwd: root }), getDataHome(project)); + + expect(await reportedSessions(project)).toBe(1); + expect(await reportedInterventionSessions(project)).toBe(1); + expect(await reportedPrompts(project)).toBe(1); + }); + + it('a delayed monitor exit does not split the next invocation with the same ID', async () => { + const { root, project } = await setup(); + const log = path.join(teamaiHome(), 'dashboard', 'events.jsonl'); + await session('copilot', { cwd: root }); + const observed = JSON.parse(fs.readFileSync(log, 'utf8').trim().split('\n').at(-1) ?? '{}'); + await hook('session-end', 'copilot', { cwd: root, hook_event_name: 'SessionEnd' }); + await hook('session-start', 'copilot', { cwd: root, hook_event_name: 'SessionStart' }); + fs.appendFileSync(log, JSON.stringify({ + type: 'process_exit', sessionId: observed.sessionId, tool: 'copilot', + timestamp: new Date().toISOString(), dataHomeKey: observed.dataHomeKey, + processExitAfter: observed.timestamp, + }) + '\n'); + await hook('prompt-submit', 'copilot', { cwd: root, hook_event_name: 'UserPromptSubmit', prompt: 'second run' }); + await hook('stop', 'copilot', { cwd: root, hook_event_name: 'Stop' }); + + expect(await reportedInterventionSessions(project)).toBe(2); + expect(await reportedSessions(project)).toBe(2); + expect(await reportedPrompts(project)).toBe(2); + }); + it('the first report after the upgrade sends nothing a shared snapshot already reported', async () => { const { root, user, project } = await setup(); const elsewhere = path.join(tmp, 'elsewhere'); @@ -434,7 +476,8 @@ describe('each scope keeps its own reported snapshot (#786)', () => { const pid = `pid-${process.ppid}`; // Main since #795 recorded and reported this run under its bare session ID. await asEarlierRelease(() => session('copilot', { cwd: root }), getDataHome(project)); - writeSharedSnapshots({ [pid]: reported }, new Date().toISOString().slice(0, 10)); + // Main writes this acknowledgement to P's own snapshot, not the shared file. + writeSharedSnapshots({ [pid]: reported }, new Date().toISOString().slice(0, 10), project); await report(project); for (const name of SNAPSHOTS) { diff --git a/src/__tests__/e2e/report-scope-reuse.test.ts b/src/__tests__/e2e/report-scope-reuse.test.ts new file mode 100644 index 000000000..15ac417fa --- /dev/null +++ b/src/__tests__/e2e/report-scope-reuse.test.ts @@ -0,0 +1,115 @@ +import { afterEach, expect, it } from 'vitest'; +import { execFileSync, spawn, type ChildProcess } from 'node:child_process'; +import { once } from 'node:events'; +import fs from 'node:fs'; +import os from 'node:os'; +import path from 'node:path'; +import YAML from 'yaml'; +import type { DashboardEvent } from '../../types.js'; + +const cli = path.resolve('dist/index.js'); +let sandbox = ''; +let dashboard: ChildProcess | undefined; + +async function stopDashboard(): Promise { + if (dashboard && dashboard.exitCode === null) { + const exited = once(dashboard, 'exit'); + dashboard.kill('SIGTERM'); + await exited; + } + dashboard = undefined; +} + +afterEach(async () => { + await stopDashboard(); + if (sandbox) fs.rmSync(sandbox, { recursive: true, force: true }); +}); + +it('real CLI preserves a path-keyed reuse and replays a delayed exit emitted by the real monitor', async () => { + sandbox = fs.realpathSync(fs.mkdtempSync(path.join(os.tmpdir(), 'teamai-scope-reuse-'))); + const home = path.join(sandbox, 'home'); + const project = path.join(sandbox, 'project'); + const dataHome = path.join(project, '.teamai'); + const remote = path.join(sandbox, 'remote.git'); + const clone = path.join(dataHome, 'team-repo'); + const logDir = path.join(home, '.teamai', 'dashboard'); + const logPath = path.join(logDir, 'events.jsonl'); + fs.mkdirSync(path.join(project, '.claude'), { recursive: true }); + fs.mkdirSync(logDir, { recursive: true }); + const env = { + ...process.env, HOME: home, USERPROFILE: home, XDG_CONFIG_HOME: path.join(home, '.config'), + GIT_CONFIG_GLOBAL: path.join(home, '.gitconfig'), GIT_CONFIG_NOSYSTEM: '1', + GIT_AUTHOR_NAME: 'Scope Test', GIT_AUTHOR_EMAIL: 'test@example.invalid', + GIT_COMMITTER_NAME: 'Scope Test', GIT_COMMITTER_EMAIL: 'test@example.invalid', NO_COLOR: '1', + }; + const git = (...args: string[]) => execFileSync('git', args, { cwd: project, env, encoding: 'utf8', stdio: 'pipe' }); + git('init', '-q', '-b', 'main'); + fs.writeFileSync(path.join(project, 'teamai.yaml'), YAML.stringify({ team: 'scope-test', repo: remote, provider: 'git' })); + git('add', 'teamai.yaml'); + git('commit', '-qm', 'fixture'); + git('clone', '-q', '--bare', project, remote); + git('clone', '-q', remote, clone); + fs.writeFileSync(path.join(dataHome, 'config.yaml'), YAML.stringify({ + repo: { localPath: clone, remote, kind: 'git' }, username: 'tester', scope: 'project', projectRoot: project, + updatePolicy: 'skip', enabledAgents: ['claude'], additionalRoles: [], + })); + const now = Date.now(); + const timestamp = (offset: number) => new Date(now + offset).toISOString(); + const event = (type: DashboardEvent['type'], offset: number): DashboardEvent => ({ + type, timestamp: timestamp(offset), sessionId: 'pid-123', tool: 'copilot', + }); + const oldUser = [event('prompt_submit', -120_000), event('session_end', -119_000)]; + const firstRun = [ + { ...event('session_start', -60_000), dataHome, monitorPid: 99999999 }, + { ...event('prompt_submit', -59_000), dataHome }, + { ...event('stop', -58_000), dataHome }, + ]; + const writeLog = (events: DashboardEvent[]) => fs.writeFileSync(logPath, events.map(e => JSON.stringify(e)).join('\n') + '\n'); + writeLog([...oldUser, ...firstRun]); + const snapshots = { + interventions: { interrupt: 0, toolReject: 0, correction: 0 }, + 'prompt-tokens': { prompts: 1, tokens: { input: 0, output: 0, cacheRead: 0, cacheCreation: 0 } }, + 'daily-sessions': { date: timestamp(0).slice(0, 10), prompts: 1, durationMs: 3_600_000, succeeded: 1, corrected: 0 }, + }; + for (const [name, value] of Object.entries(snapshots)) { + fs.writeFileSync(path.join(logDir, `reported-${name}.json`), JSON.stringify({ 'pid-123': value })); + } + const pull = () => execFileSync(process.execPath, [cli, 'pull'], { cwd: project, env, encoding: 'utf8', stdio: 'pipe', timeout: 30_000 }); + const stats = () => YAML.parse(git('--git-dir', remote, 'show', 'teamai-reports:stats/tester.yaml')); + pull(); + expect(stats()).toMatchObject({ prompts: 1, interventions: { sessions: 1 } }); + + // Capture an actual monitor event, then replay the race: SessionEnd and the + // next invocation arrive after the monitor read but before its exit append. + dashboard = spawn(process.execPath, [cli, 'dashboard', '--port', '0'], { cwd: project, env, stdio: 'pipe' }); + let output = ''; + dashboard.stdout?.on('data', data => { output += data.toString(); }); + dashboard.stderr?.on('data', data => { output += data.toString(); }); + let exit: DashboardEvent | undefined; + const deadline = Date.now() + 25_000; + while (!exit && Date.now() < deadline) { + if (dashboard.exitCode !== null) throw new Error(output); + const events: DashboardEvent[] = fs.readFileSync(logPath, 'utf8').trim().split('\n').map(line => JSON.parse(line)); + exit = events.find(e => e.type === 'process_exit'); + if (!exit) await new Promise(resolve => setTimeout(resolve, 100)); + } + await stopDashboard(); + expect(exit?.processExitAfter, output).toBe(firstRun[2].timestamp); + if (!exit) throw new Error('The dashboard monitor did not record an exit.'); + writeLog([ + ...oldUser, ...firstRun, + { ...event('session_end', -57_000), dataHome }, + { ...event('session_start', -56_000), dataHome }, + exit, + { ...event('prompt_submit', 0), timestamp: new Date(Date.parse(exit.timestamp) + 1000).toISOString(), dataHome }, + { ...event('stop', 0), timestamp: new Date(Date.parse(exit.timestamp) + 2000).toISOString(), dataHome }, + ]); + pull(); + expect(stats()).toMatchObject({ prompts: 2, interventions: { sessions: 2 } }); + const reportedHead = git('--git-dir', remote, 'rev-parse', 'teamai-reports'); + pull(); + expect(git('--git-dir', remote, 'rev-parse', 'teamai-reports')).toBe(reportedHead); + const shown = execFileSync(process.execPath, [cli, 'stats'], { cwd: project, env, encoding: 'utf8', timeout: 15_000 }); + expect(shown).toMatch(/Sessions:\s+2/); + expect(shown).toMatch(/Conversation turns:\s+2/); +}, 60_000); diff --git a/src/__tests__/scope-filter.test.ts b/src/__tests__/scope-filter.test.ts index cb5199d69..e55ece29a 100644 --- a/src/__tests__/scope-filter.test.ts +++ b/src/__tests__/scope-filter.test.ts @@ -117,6 +117,25 @@ describe('filterEventsByScope', () => { expect(before[2].sessionId).toBe(after[0].sessionId); }); + it('a delayed observed exit targets the earlier run across scopes and is ignored after that run is compacted', async () => { + const old = { ...(await scopedEvent(undefined, 'pid-1', '/home/jeff/.teamai')), timestamp: '2026-01-01T00:00:00Z' }; + const ended = { ...old, type: 'session_end' as const, timestamp: '2026-01-01T00:01:00Z' }; + const next = { ...(await scopedEvent(undefined, 'pid-1', '/home/jeff/.teamai/projects/p')), timestamp: '2026-01-01T00:02:00Z' }; + const delayed = { ...old, type: 'process_exit' as const, timestamp: '2026-01-01T00:03:00Z', processExitAfter: old.timestamp }; + const prompt = { ...next, timestamp: '2026-01-01T00:04:00Z' }; + const exit = { ...next, type: 'process_exit' as const, timestamp: '2026-01-01T00:05:00Z', processExitAfter: prompt.timestamp }; + const later = { ...next, timestamp: '2026-01-01T00:06:00Z' }; + for (const history of [[old, ended], []]) { + const log = [...history, next, delayed, prompt, exit, later]; + const project = await filterEventsByScope(log, projectScope('/Users/jeff/project-a')); + expect(project.map(e => e.sessionId)).toEqual([ + `pid-1@${next.timestamp}`, `pid-1@${next.timestamp}`, `pid-1@${next.timestamp}`, `pid-1@${later.timestamp}`, + ]); + const user = await filterEventsByScope(log, userScope()); + expect(user).toHaveLength(history.length ? 3 : 0); + } + }); + it('an event that records its data home as a path, before keys were hashed, is keyed by it', async () => { const unhashed: DashboardEvent[] = [ { ...makeEvent(undefined, 'c1'), tool: 'copilot', dataHome: '/home/jeff/.teamai/projects/p' }, @@ -232,6 +251,13 @@ describe('adoptBareKeys', () => { expect(adoptBareKeys({ 'pid-1': 3 }, [run('pid-1@t1', { dataHome: '/home/jeff/.teamai' })])).toEqual({ 'pid-1@t1': 3 }); }); + it('does not give a path-keyed run a shared bare entry, but still adopts its own scope entry', () => { + const reported = { 'pid-1': 3 }; + const events = [run('pid-1@t1', { dataHome: '/home/jeff/.teamai/projects/p' })]; + expect(adoptBareKeys(reported, events, 'shared')).toBe(reported); + expect(adoptBareKeys(reported, events, 'scope')).toEqual({ 'pid-1@t1': 3 }); + }); + it('a run in progress across the upgrade is an earlier release\'s, by its first event', () => { expect(adoptBareKeys({ 'pid-1': 3 }, [run('pid-1@t1'), run('pid-1@t1', { dataHomeKey: 'k' })])).toEqual({ 'pid-1@t1': 3 }); }); diff --git a/src/dashboard.ts b/src/dashboard.ts index 845a9a390..eeb52f183 100644 --- a/src/dashboard.ts +++ b/src/dashboard.ts @@ -121,6 +121,7 @@ export async function startDashboard(port?: number): Promise { type: 'process_exit', timestamp: new Date().toISOString(), sessionId: session.sessionId, + processExitAfter: session.lastActivity, tool: session.tool, cwd: session.cwd, // The session's own data home, so its scope's report still sees it end. diff --git a/src/team-push.ts b/src/team-push.ts index f35c8fe2a..641323fdc 100644 --- a/src/team-push.ts +++ b/src/team-push.ts @@ -145,10 +145,11 @@ export function mergeStats( // // A session can record events in two scopes (a `cd` mid-session), so each // scope compares against what it reported itself. The first time a scope needs -// a snapshot, it copies the shared one, so nothing an earlier release reported -// is sent again; after that only its own file is read. No scope writes the -// shared file any more, only an earlier release after a rollback (and a caller -// without a scope config, which reads the whole log and reports into it). +// a snapshot, it migrates the shared entries for its retained unkeyed runs; +// path-keyed releases already wrote scope snapshots. After that only its own +// file is read. No scope writes the shared file any more, only an earlier +// release after a rollback (and a caller without a scope config, which reads +// the whole log and reports into it). // type ReportedSnapshotName = 'interventions' | 'prompt-tokens' | 'daily-sessions'; @@ -168,12 +169,20 @@ function scopeSnapshotPath(name: ReportedSnapshotName, config: LocalConfig | und } /** A scope's snapshot, seeded from the shared one when the scope has none yet. */ -async function readSnapshot(name: ReportedSnapshotName, config: LocalConfig | undefined): Promise { +async function readSnapshot(name: ReportedSnapshotName, config: LocalConfig | undefined): Promise | null> { const own = scopeSnapshotPath(name, config); - if (!config || await pathExists(own)) return readJson(own); - const seed = await readJson(sharedSnapshotPath(name)); + if (!config || await pathExists(own)) return readJson>(own); + const shared = await readJson>(sharedSnapshotPath(name)); + const events = await filterEventsByScope(await readEvents(), config); + const adopted = adoptBareKeys(shared ?? {}, events, 'shared'); + // Copy only resolved run IDs. An unmatched bare entry cannot be allowed to + // attach to a future reuse after the source of the snapshot has been lost. + const seed: Record = {}; + for (const { sessionId } of events) { + if (Object.hasOwn(adopted, sessionId)) seed[sessionId] = adopted[sessionId]; + } try { - await writeJson(own, seed ?? {}); + await writeJson(own, seed); } catch (e) { // Seeded again next time: the shared file is not written any more. log.debug(`Could not seed ${own}: ${(e as Error).message}`); @@ -182,7 +191,7 @@ async function readSnapshot(name: ReportedSnapshotName, config: LocalConfig | } export async function readReportedInterventions(config: LocalConfig | undefined): Promise { - const parsed = await readSnapshot('interventions', config); + const parsed = await readSnapshot('interventions', config); return parsed && typeof parsed === 'object' ? parsed : {}; } @@ -248,7 +257,7 @@ function hasInterventionDelta(d: UserInterventionStats): boolean { // export async function readReportedPromptTokens(config: LocalConfig | undefined): Promise { - const parsed = await readSnapshot('prompt-tokens', config); + const parsed = await readSnapshot('prompt-tokens', config); return parsed && typeof parsed === 'object' ? parsed : {}; } @@ -312,7 +321,7 @@ function hasPromptTokenDelta(d: PromptTokenDelta): boolean { } async function readReportedDailySessions(config: LocalConfig | undefined): Promise { - return (await readSnapshot('daily-sessions', config)) ?? {}; + return (await readSnapshot('daily-sessions', config)) ?? {}; } async function writeReportedDailySessions(data: ReportedDailySessions, config: LocalConfig | undefined): Promise { @@ -390,25 +399,35 @@ export async function filterEventsByScope( // same whichever earlier runs compaction has dropped. // A second end with nothing recorded since the first (the dashboard // monitor's process_exit after SessionEnd) belongs to the run just closed. - const runOf: number[] = []; + const runOf: Array = []; + const observedRuns = new Map(); const openRun = new Map(); const closedRun = new Map(); const deciding: Array = []; const runIds: string[] = []; events.forEach((e, i) => { const ends = e.type === 'session_end' || e.type === 'process_exit'; - let run = openRun.get(e.sessionId) ?? (ends ? closedRun.get(e.sessionId) : undefined); + const observed = e.type === 'process_exit' && typeof e.processExitAfter === 'string'; + let run = observed ? observedRuns.get(`${e.sessionId}@${e.processExitAfter}`) + : openRun.get(e.sessionId) ?? (ends ? closedRun.get(e.sessionId) : undefined); + // The observed run may have been compacted. Its delayed exit must not + // manufacture a new session or close a later reuse of the same ID. + if (observed && run === undefined) { + runOf.push(undefined); + return; + } if (run === undefined) { run = deciding.push(undefined) - 1; runIds.push(`${e.sessionId}@${e.timestamp}`); } - if (ends) { + if (ends && (!observed || openRun.get(e.sessionId) === run)) { openRun.delete(e.sessionId); closedRun.set(e.sessionId, run); - } else { + } else if (!ends) { openRun.set(e.sessionId, run); } runOf.push(run); + observedRuns.set(`${e.sessionId}@${e.timestamp}`, run); const current = deciding[run]; if (eventKeys[i] !== undefined ? current === undefined || eventKeys[current] === undefined : current === undefined && !!e.cwd) deciding[run] = i; @@ -419,7 +438,10 @@ export async function filterEventsByScope( const cwd = events[i].cwd; return key !== undefined ? keys.has(key) : !!cwd && await ownsCwd(cwd); })); - return events.flatMap((e, i) => (owned[runOf[i]] ? [{ ...e, sessionId: runIds[runOf[i]] }] : [])); + return events.flatMap((e, i) => { + const run = runOf[i]; + return run !== undefined && owned[run] ? [{ ...e, sessionId: runIds[run] }] : []; + }); } /** @@ -430,20 +452,27 @@ export async function filterEventsByScope( * bare entry is retired either way, so no later run of that ID reads it. Only * an earlier release wrote bare entries, and only for runs it recorded, so a * run whose first event carries a `dataHomeKey` takes none: the entry may be - * another scope's run under a reused PID-fallback ID. Returns `reported` - * itself when there is nothing to retire; otherwise the caller persists the result. + * another scope's run under a reused PID-fallback ID. Shared snapshots also + * exclude path-keyed runs: that release already had per-scope snapshots. + * Returns `reported` itself when there is nothing to retire; otherwise the + * caller persists the result. */ -export function adoptBareKeys(reported: Record, events: Iterable): Record { +export function adoptBareKeys( + reported: Record, + events: Iterable, + source: 'scope' | 'shared' = 'scope', +): Record { const adopted = { ...reported }; const seen = new Set(); let retired = false; - for (const { sessionId: runId, dataHomeKey } of events) { + for (const { sessionId: runId, dataHomeKey, dataHome } of events) { const at = runId.lastIndexOf('@'); if (at < 0) continue; const id = runId.slice(0, at); if (seen.has(id)) continue; seen.add(id); if (typeof dataHomeKey === 'string' || !Object.hasOwn(reported, id)) continue; + if (source === 'shared' && typeof dataHome === 'string') continue; if (!Object.hasOwn(adopted, runId)) adopted[runId] = reported[id]; delete adopted[id]; retired = true; diff --git a/src/types.ts b/src/types.ts index 703e48711..82170e1ab 100644 --- a/src/types.ts +++ b/src/types.ts @@ -1262,6 +1262,8 @@ export interface DashboardEvent { transcriptPath?: string; /** Resolved PID of the AI tool main process (for liveness monitoring) */ monitorPid?: number; + /** Last event timestamp the PID monitor observed, so a delayed exit targets that run. */ + processExitAfter?: string; /** Byte boundary captured at Copilot SessionStart; private log path is never stored. */ copilotRunStartOffset?: number; /** Opaque marker metadata retained for events written by older collector versions. */ From 93d4c0c7f0e6091bf1da0173950dec511d5cc9d7 Mon Sep 17 00:00:00 2001 From: Saul Moro Date: Thu, 24 Sep 2026 18:00:02 +0200 Subject: [PATCH 14/41] fix(stats): address CI review (#785) An earlier release summed every run of a reused ID under its bare snapshot entry, but only the first retained run adopted it, so the next one was reported again. Each of those runs in the log but the last is now taken as reported at its own totals and the last takes the entry, in the report, in teamai stats and in the seed from the shared file. The last run is undercounted by at most the other runs' share, once. A session_start on a fallback ID from another monitorPid than its open run's begins a new run, so a run that crashed with no dashboard running no longer takes the next invocation, maybe another scope's. A tool's own ID is not split: Claude fires SessionStart again on resume, in a new process, and its Stop carries the whole transcript. --- docs/designs/data-directory-layout.md | 22 ++-- src/__tests__/dashboard-report-scope.test.ts | 39 +++++++ src/__tests__/e2e/report-scope-reuse.test.ts | 75 ++++++++++--- src/__tests__/scope-filter.test.ts | 69 +++++++++--- src/stats.ts | 16 +-- src/team-push.ts | 106 +++++++++++++------ 6 files changed, 257 insertions(+), 70 deletions(-) diff --git a/docs/designs/data-directory-layout.md b/docs/designs/data-directory-layout.md index c9a8b0fc8..0cfee4855 100644 --- a/docs/designs/data-directory-layout.md +++ b/docs/designs/data-directory-layout.md @@ -355,17 +355,25 @@ mid-session) is reported once, where it started. A session is one ID's run up to its `session_end` or `process_exit`, so a later run that reuses the ID (Copilot's fallback ID is the parent PID) is decided on its own; a second end with nothing recorded since the first (the dashboard monitor's `process_exit` after -`SessionEnd`) belongs to the run it closed. The monitor's `process_exit` also +`SessionEnd`) belongs to the run it closed. A `session_start` on a fallback ID +(`pid-…`) whose `monitorPid` differs from its open run's begins a new run even +though nothing ended that one (a crash with no dashboard running); a tool's own +ID is not split this way, since Claude fires SessionStart again on resume, in a +new process, and its Stop carries the whole transcript. The monitor's `process_exit` also records `processExitAfter`, the last event it observed, and closes only that run: an exit appended after the next run of the same ID began does not end it, and one whose run compaction dropped is ignored. Every run is reported and snapshotted as `@`, which does not change when -compaction drops earlier runs; a snapshot entry keyed by the bare ID (written -before) is rewritten under the first run of that ID in the log the next time -the scope reports, and removed, so no later run of that ID reads it. Only an -earlier release wrote bare entries, and a seeded one may be another scope's, so -a run whose first event carries `dataHomeKey` (recorded by this release) takes -none. A session written before that field existed is attributed by its first +compaction drops earlier runs. A snapshot entry keyed by the bare ID (written +before) sums every run of that ID the earlier release saw, so the next time the +scope reports, each of those runs in the log but the last is taken as reported +at its own totals, the last takes the entry, and the entry is removed, so no run +is sent again and no later run of that ID reads it. What the last run records +beyond the entry is reported; it is undercounted by at most the other runs' +share of the sum, once (when it was still growing at the upgrade, or had not been +reported yet). Only an earlier release wrote bare entries, and a seeded one +may be another scope's, so a run whose first event carries `dataHomeKey` +(recorded by this release), and every later run of its ID, takes none. A session written before that field existed is attributed by its first `cwd`: to the scope `resolveConfigForDir` resolves that directory to now, the dispatcher's rule, so a nested clone under a project is not the project's; no `cwd`, or one removed since, is no scope's. The snapshots of what was already reported are per scope diff --git a/src/__tests__/dashboard-report-scope.test.ts b/src/__tests__/dashboard-report-scope.test.ts index b935d3fd8..50977ee2f 100644 --- a/src/__tests__/dashboard-report-scope.test.ts +++ b/src/__tests__/dashboard-report-scope.test.ts @@ -428,6 +428,45 @@ describe('each scope keeps its own reported snapshot (#786)', () => { expect(await reportedPrompts(project)).toBe(2); }); + it.each([ + ['a shared snapshot, before #785', false], + ['the scope\'s own snapshot, main since #795', true], + ])('two runs an earlier release reported under one bare ID are not sent again: %s', async (_, ownSnapshot) => { + const { root, project } = await setup(); + // No session ID: a fallback ID the second run reuses. That release summed + // both runs under it; Claude's fallback carries the cwd, Copilot's does not. + const [tool, payload] = ownSnapshot ? ['copilot', {}] : ['claude', { cwd: root }]; + const run = async () => { + await session(tool, { ...payload, cwd: root }); + await hook('session-end', tool, { ...payload, cwd: root, hook_event_name: 'SessionEnd' }); + }; + await asEarlierRelease(async () => { await run(); await run(); }, ownSnapshot ? getDataHome(project) : undefined); + const events = fs.readFileSync(path.join(teamaiHome(), 'dashboard', 'events.jsonl'), 'utf-8'); + const id: string = JSON.parse(events.split('\n')[0]).sessionId; + writeSharedSnapshots({ [id]: 2 }, new Date().toISOString().slice(0, 10), ownSnapshot ? project : undefined); + + expect(await report(project)).toBeNull(); + }); + + it('a fallback ID reused in a project after a user-scope run that never ended is the project\'s', async () => { + const { root, user, project } = await setup(); + const elsewhere = path.join(tmp, 'elsewhere'); + fs.mkdirSync(elsewhere); + const log = path.join(teamaiHome(), 'dashboard', 'events.jsonl'); + await session('copilot', { cwd: elsewhere }); + // That invocation crashed with no dashboard running: nothing ended it. Its + // process, recorded at SessionStart, is not the next invocation's. + fs.writeFileSync(log, fs.readFileSync(log, 'utf-8').split('\n').filter(Boolean) + .map((line) => JSON.stringify({ ...JSON.parse(line), ...(line.includes('"monitorPid"') ? { monitorPid: 99999999 } : {}) })) + .join('\n') + '\n'); + await session('copilot', { cwd: root }); + + expect(await reportedSessions(project)).toBe(1); + expect(await reportedPrompts(project)).toBe(1); + expect(await reportedSessions(user)).toBe(1); + expect(await reportedPrompts(user)).toBe(1); + }); + it('the first report after the upgrade sends nothing a shared snapshot already reported', async () => { const { root, user, project } = await setup(); const elsewhere = path.join(tmp, 'elsewhere'); diff --git a/src/__tests__/e2e/report-scope-reuse.test.ts b/src/__tests__/e2e/report-scope-reuse.test.ts index 15ac417fa..252ade399 100644 --- a/src/__tests__/e2e/report-scope-reuse.test.ts +++ b/src/__tests__/e2e/report-scope-reuse.test.ts @@ -25,7 +25,8 @@ afterEach(async () => { if (sandbox) fs.rmSync(sandbox, { recursive: true, force: true }); }); -it('real CLI preserves a path-keyed reuse and replays a delayed exit emitted by the real monitor', async () => { +/** A git project P whose data home lives in the project, and the real CLI to drive it. */ +function fixture() { sandbox = fs.realpathSync(fs.mkdtempSync(path.join(os.tmpdir(), 'teamai-scope-reuse-'))); const home = path.join(sandbox, 'home'); const project = path.join(sandbox, 'project'); @@ -53,6 +54,29 @@ it('real CLI preserves a path-keyed reuse and replays a delayed exit emitted by repo: { localPath: clone, remote, kind: 'git' }, username: 'tester', scope: 'project', projectRoot: project, updatePolicy: 'skip', enabledAgents: ['claude'], additionalRoles: [], })); + const writeLog = (events: DashboardEvent[]) => fs.writeFileSync(logPath, events.map(e => JSON.stringify(e)).join('\n') + '\n'); + /** The shared snapshots an earlier release left, `prompts` reported per bare session ID. */ + const writeSharedSnapshots = (prompts: Record, date: string) => { + const entries = (value: (n: number) => unknown) => Object.fromEntries(Object.entries(prompts).map(([sid, n]) => [sid, value(n)])); + const snapshots = { + interventions: entries(() => ({ interrupt: 0, toolReject: 0, correction: 0 })), + 'prompt-tokens': entries((n) => ({ prompts: n, tokens: { input: 0, output: 0, cacheRead: 0, cacheCreation: 0 } })), + 'daily-sessions': entries((n) => ({ date, prompts: n, durationMs: 3_600_000, succeeded: 1, corrected: 0 })), + }; + for (const [name, value] of Object.entries(snapshots)) { + fs.writeFileSync(path.join(logDir, `reported-${name}.json`), JSON.stringify(value)); + } + }; + const pull = () => execFileSync(process.execPath, [cli, 'pull'], { cwd: project, env, encoding: 'utf8', stdio: 'pipe', timeout: 30_000 }); + const stats = () => YAML.parse(git('--git-dir', remote, 'show', 'teamai-reports:stats/tester.yaml')); + const reportsHead = () => { + try { return git('--git-dir', remote, 'rev-parse', '--verify', '-q', 'teamai-reports'); } catch { return ''; } + }; + return { project, dataHome, remote, logPath, env, git, writeLog, writeSharedSnapshots, pull, stats, reportsHead }; +} + +it('real CLI preserves a path-keyed reuse and replays a delayed exit emitted by the real monitor', async () => { + const { project, dataHome, logPath, env, git, remote, writeLog, writeSharedSnapshots, pull, stats } = fixture(); const now = Date.now(); const timestamp = (offset: number) => new Date(now + offset).toISOString(); const event = (type: DashboardEvent['type'], offset: number): DashboardEvent => ({ @@ -64,18 +88,8 @@ it('real CLI preserves a path-keyed reuse and replays a delayed exit emitted by { ...event('prompt_submit', -59_000), dataHome }, { ...event('stop', -58_000), dataHome }, ]; - const writeLog = (events: DashboardEvent[]) => fs.writeFileSync(logPath, events.map(e => JSON.stringify(e)).join('\n') + '\n'); writeLog([...oldUser, ...firstRun]); - const snapshots = { - interventions: { interrupt: 0, toolReject: 0, correction: 0 }, - 'prompt-tokens': { prompts: 1, tokens: { input: 0, output: 0, cacheRead: 0, cacheCreation: 0 } }, - 'daily-sessions': { date: timestamp(0).slice(0, 10), prompts: 1, durationMs: 3_600_000, succeeded: 1, corrected: 0 }, - }; - for (const [name, value] of Object.entries(snapshots)) { - fs.writeFileSync(path.join(logDir, `reported-${name}.json`), JSON.stringify({ 'pid-123': value })); - } - const pull = () => execFileSync(process.execPath, [cli, 'pull'], { cwd: project, env, encoding: 'utf8', stdio: 'pipe', timeout: 30_000 }); - const stats = () => YAML.parse(git('--git-dir', remote, 'show', 'teamai-reports:stats/tester.yaml')); + writeSharedSnapshots({ 'pid-123': 1 }, timestamp(0).slice(0, 10)); pull(); expect(stats()).toMatchObject({ prompts: 1, interventions: { sessions: 1 } }); @@ -113,3 +127,40 @@ it('real CLI preserves a path-keyed reuse and replays a delayed exit emitted by expect(shown).toMatch(/Sessions:\s+2/); expect(shown).toMatch(/Conversation turns:\s+2/); }, 60_000); + +it('real CLI sends no run of an aggregated bare ID again and splits a reused ID a crash left open', () => { + const { project, dataHome, writeLog, writeSharedSnapshots, pull, stats, reportsHead } = fixture(); + const now = Date.now(); + let offset = -600_000; + const event = (type: DashboardEvent['type'], sessionId: string, fields: Partial = {}): DashboardEvent => + ({ type, timestamp: new Date(now + (offset += 1000)).toISOString(), sessionId, tool: 'claude', ...fields }); + const legacyId = `pid-321-${project}`; + const legacyRun = () => [ + event('session_start', legacyId, { cwd: project }), event('prompt_submit', legacyId, { cwd: project }), + event('stop', legacyId, { cwd: project }), event('session_end', legacyId, { cwd: project }), + ]; + // An earlier release recorded two runs of one fallback ID and reported them as one. + const legacy = [...legacyRun(), ...legacyRun()]; + writeLog(legacy); + writeSharedSnapshots({ [legacyId]: 2 }, new Date(now).toISOString().slice(0, 10)); + pull(); + expect(reportsHead()).toBe(''); + + // A run that crashed with no dashboard running: nothing ends it. + const crashed = [ + event('session_start', 'pid-9', { dataHome, monitorPid: 99999998 }), + event('prompt_submit', 'pid-9', { dataHome }), event('stop', 'pid-9', { dataHome }), + ]; + writeLog([...legacy, ...crashed]); + pull(); + expect(stats()).toMatchObject({ prompts: 1, interventions: { sessions: 1 } }); + + // The next invocation reuses the ID from another process. + const next = [ + event('session_start', 'pid-9', { dataHome, monitorPid: 99999999 }), + event('prompt_submit', 'pid-9', { dataHome }), event('stop', 'pid-9', { dataHome }), + ]; + writeLog([...legacy, ...crashed, ...next]); + pull(); + expect(stats()).toMatchObject({ prompts: 2, interventions: { sessions: 2 } }); +}, 60_000); diff --git a/src/__tests__/scope-filter.test.ts b/src/__tests__/scope-filter.test.ts index e55ece29a..26036ae61 100644 --- a/src/__tests__/scope-filter.test.ts +++ b/src/__tests__/scope-filter.test.ts @@ -136,6 +136,37 @@ describe('filterEventsByScope', () => { } }); + it('a fallback ID started by another process is a new run, though the last one never ended', async () => { + // The user-scope run crashed with no dashboard running, so nothing ended it. + const start = (sessionId: string, dataHome: string, monitorPid: number) => scopedEvent(undefined, sessionId, dataHome) + .then((e) => ({ ...e, type: 'session_start' as const, monitorPid })); + for (const id of ['pid-1', 'pid-1-/Users/jeff/project-a']) { + const log = [ + await start(id, '/home/jeff/.teamai', 100), + await scopedEvent(undefined, id, '/home/jeff/.teamai'), + await start(id, '/home/jeff/.teamai/projects/p', 200), + await scopedEvent(undefined, id, '/home/jeff/.teamai/projects/p'), + ]; + expect(await ids(log, projectScope('/Users/jeff/project-a'))).toEqual([id, id]); + expect(await ids(log, userScope())).toEqual([id, id]); + } + }); + + it('a start from the same process, or under a tool\'s own session ID, stays in the open run', async () => { + // Claude fires SessionStart again on compact and on resume, a new process for + // the same session; its Stop carries the whole transcript, so it must stay one run. + const cases: Array<[string, number]> = [['pid-1', 100], ['claude-uuid', 200]]; + for (const [id, pid] of cases) { + const log = [ + { ...(await scopedEvent(undefined, id, '/home/jeff/.teamai')), type: 'session_start' as const, monitorPid: 100 }, + { ...(await scopedEvent(undefined, id, '/home/jeff/.teamai/projects/p')), type: 'session_start' as const, monitorPid: pid }, + await scopedEvent(undefined, id, '/home/jeff/.teamai/projects/p'), + ]; + expect(await ids(log, userScope())).toEqual([id, id, id]); + expect(await ids(log, projectScope('/Users/jeff/project-a'))).toEqual([]); + } + }); + it('an event that records its data home as a path, before keys were hashed, is keyed by it', async () => { const unhashed: DashboardEvent[] = [ { ...makeEvent(undefined, 'c1'), tool: 'copilot', dataHome: '/home/jeff/.teamai/projects/p' }, @@ -219,46 +250,54 @@ describe('adoptBareKeys', () => { const run = (sessionId: string, fields: Partial = {}): DashboardEvent => ({ type: 'prompt_submit', timestamp: 't', sessionId, tool: 'copilot', ...fields }); - it('gives a snapshot entry keyed by the bare session ID to the first run of that ID', () => { + it('gives a snapshot entry keyed by the bare session ID to the last run of that ID', () => { + // An earlier release summed every run of the ID under it: the earlier runs are + // reported at their own totals, the last takes the sum, so none is sent again. const reported = { 'pid-1': 3, 's2@t0': 1 }; - const adopted = adoptBareKeys(reported, [run('pid-1@t1'), run('pid-1@t2'), run('s2@t0'), run('s3@t3')]); - expect(adopted['pid-1@t1']).toBe(3); - expect(adopted['pid-1@t2']).toBeUndefined(); - expect(adopted['s3@t3']).toBeUndefined(); + const current = { 'pid-1@t1': 1, 'pid-1@t2': 2, 's2@t0': 1, 's3@t3': 4 }; + const adopted = adoptBareKeys(reported, [run('pid-1@t1'), run('pid-1@t2'), run('s2@t0'), run('s3@t3')], current); + expect(adopted).toEqual({ 'pid-1@t1': 1, 'pid-1@t2': 3, 's2@t0': 1 }); + }); + + it('a single run of the ID takes the bare entry', () => { + expect(adoptBareKeys({ 'pid-1': 3 }, [run('pid-1@t1')], { 'pid-1@t1': 5 })).toEqual({ 'pid-1@t1': 3 }); }); it('never overrides a run\'s own entry', () => { - expect(adoptBareKeys({ 'pid-1': 3, 'pid-1@t1': 5 }, [run('pid-1@t1')])['pid-1@t1']).toBe(5); + expect(adoptBareKeys({ 'pid-1': 3, 'pid-1@t1': 5 }, [run('pid-1@t1')], {})['pid-1@t1']).toBe(5); + expect(adoptBareKeys({ 'pid-1': 3, 'pid-1@t1': 5 }, [run('pid-1@t1'), run('pid-1@t2')], { 'pid-1@t1': 7 })) + .toEqual({ 'pid-1@t1': 5, 'pid-1@t2': 3 }); }); - it('retires the bare entry the first run of its ID reads, so it is never read again', () => { - expect(adoptBareKeys({ 'pid-1': 3, s2: 1 }, [run('pid-1@t1'), run('pid-1@t2')])).toEqual({ 'pid-1@t1': 3, s2: 1 }); - expect(adoptBareKeys({ 'pid-1': 3, 'pid-1@t1': 5 }, [run('pid-1@t1')])).toEqual({ 'pid-1@t1': 5 }); + it('retires the bare entry its runs read, so it is never read again', () => { + expect(adoptBareKeys({ 'pid-1': 3, s2: 1 }, [run('pid-1@t1'), run('pid-1@t2')], { 'pid-1@t1': 1 })) + .toEqual({ 'pid-1@t1': 1, 'pid-1@t2': 3, s2: 1 }); + expect(adoptBareKeys({ 'pid-1': 3, 'pid-1@t1': 5 }, [run('pid-1@t1')], {})).toEqual({ 'pid-1@t1': 5 }); }); it('returns the snapshot itself when it holds no bare entry to retire', () => { const reported = { 'pid-1@t1': 5, s2: 1 }; - expect(adoptBareKeys(reported, [run('pid-1@t1'), run('s3@t3')])).toBe(reported); + expect(adoptBareKeys(reported, [run('pid-1@t1'), run('s3@t3')], {})).toBe(reported); }); it('gives no bare entry to a run this build recorded: only an earlier release wrote them', () => { // Another scope's run, maybe: the bare entry says nothing about which scope reported it. const reported = { 'pid-1': 3 }; - expect(adoptBareKeys(reported, [run('pid-1@t1', { dataHomeKey: 'k' }), run('pid-1@t2')])).toBe(reported); + expect(adoptBareKeys(reported, [run('pid-1@t1', { dataHomeKey: 'k' }), run('pid-1@t2')], {})).toBe(reported); }); it('a run recorded by main since #795, which wrote the data home path, is an earlier release\'s', () => { - expect(adoptBareKeys({ 'pid-1': 3 }, [run('pid-1@t1', { dataHome: '/home/jeff/.teamai' })])).toEqual({ 'pid-1@t1': 3 }); + expect(adoptBareKeys({ 'pid-1': 3 }, [run('pid-1@t1', { dataHome: '/home/jeff/.teamai' })], {})).toEqual({ 'pid-1@t1': 3 }); }); it('does not give a path-keyed run a shared bare entry, but still adopts its own scope entry', () => { const reported = { 'pid-1': 3 }; const events = [run('pid-1@t1', { dataHome: '/home/jeff/.teamai/projects/p' })]; - expect(adoptBareKeys(reported, events, 'shared')).toBe(reported); - expect(adoptBareKeys(reported, events, 'scope')).toEqual({ 'pid-1@t1': 3 }); + expect(adoptBareKeys(reported, events, {}, 'shared')).toBe(reported); + expect(adoptBareKeys(reported, events, {}, 'scope')).toEqual({ 'pid-1@t1': 3 }); }); it('a run in progress across the upgrade is an earlier release\'s, by its first event', () => { - expect(adoptBareKeys({ 'pid-1': 3 }, [run('pid-1@t1'), run('pid-1@t1', { dataHomeKey: 'k' })])).toEqual({ 'pid-1@t1': 3 }); + expect(adoptBareKeys({ 'pid-1': 3 }, [run('pid-1@t1'), run('pid-1@t1', { dataHomeKey: 'k' })], {})).toEqual({ 'pid-1@t1': 3 }); }); }); diff --git a/src/stats.ts b/src/stats.ts index adbad9a29..e169a805f 100644 --- a/src/stats.ts +++ b/src/stats.ts @@ -160,13 +160,17 @@ async function unreportedDashboardStats( adoptBareKeys, computeInterventionDelta, computePromptTokenDelta, readReportedInterventions, readReportedPromptTokens, } = await import('./team-push.js'); // The scope's own snapshots, the ones its report compares against (#786). - const interventions = adoptBareKeys(await readReportedInterventions(config), events); - const promptTokens = adoptBareKeys(await readReportedPromptTokens(config), events); - - const interventionDelta = computeInterventionDelta( - new Map([...metrics].map(([sid, m]) => [sid, { interrupt: m.interrupt, toolReject: m.toolReject, correction: m.correction }])), - interventions, + const currentInterventions = new Map( + [...metrics].map(([sid, m]) => [sid, { interrupt: m.interrupt, toolReject: m.toolReject, correction: m.correction }]), + ); + const interventions = adoptBareKeys( + await readReportedInterventions(config), events, Object.fromEntries(currentInterventions), ); + const promptTokens = adoptBareKeys( + await readReportedPromptTokens(config), events, computePromptTokenDelta(metrics, {}).nextReported, + ); + + const interventionDelta = computeInterventionDelta(currentInterventions, interventions); const promptTokenDelta = computePromptTokenDelta(metrics, promptTokens); return { diff --git a/src/team-push.ts b/src/team-push.ts index 641323fdc..e8283e4b6 100644 --- a/src/team-push.ts +++ b/src/team-push.ts @@ -168,13 +168,20 @@ function scopeSnapshotPath(name: ReportedSnapshotName, config: LocalConfig | und return path.join(dataHome, 'dashboard', `user-${file}`); } -/** A scope's snapshot, seeded from the shared one when the scope has none yet. */ -async function readSnapshot(name: ReportedSnapshotName, config: LocalConfig | undefined): Promise | null> { +/** + * A scope's snapshot, seeded from the shared one when the scope has none yet. + * `current` gives each run's entry at its own totals, for {@link adoptBareKeys}. + */ +async function readSnapshot( + name: ReportedSnapshotName, + config: LocalConfig | undefined, + current: (events: DashboardEvent[]) => Record, +): Promise | null> { const own = scopeSnapshotPath(name, config); if (!config || await pathExists(own)) return readJson>(own); const shared = await readJson>(sharedSnapshotPath(name)); const events = await filterEventsByScope(await readEvents(), config); - const adopted = adoptBareKeys(shared ?? {}, events, 'shared'); + const adopted = adoptBareKeys(shared ?? {}, events, current(events), 'shared'); // Copy only resolved run IDs. An unmatched bare entry cannot be allowed to // attach to a future reuse after the source of the snapshot has been lost. const seed: Record = {}; @@ -191,7 +198,8 @@ async function readSnapshot(name: ReportedSnapshotName, config: LocalConfig | } export async function readReportedInterventions(config: LocalConfig | undefined): Promise { - const parsed = await readSnapshot('interventions', config); + const parsed = await readSnapshot('interventions', config, + (events) => Object.fromEntries(interventionCounts(aggregateSessionMetrics(events)))); return parsed && typeof parsed === 'object' ? parsed : {}; } @@ -203,6 +211,11 @@ async function writeReportedInterventions(data: ReportedInterventions, config: L } } +/** Each session's intervention counts, the shape its snapshot entry holds. */ +function interventionCounts(metrics: Map): Map { + return new Map([...metrics].map(([sid, m]) => [sid, { interrupt: m.interrupt, toolReject: m.toolReject, correction: m.correction }])); +} + /** * Compute the intervention delta to report: for each current session, the positive * change since it was last reported. A session not seen before contributes +1 to @@ -257,7 +270,8 @@ function hasInterventionDelta(d: UserInterventionStats): boolean { // export async function readReportedPromptTokens(config: LocalConfig | undefined): Promise { - const parsed = await readSnapshot('prompt-tokens', config); + const parsed = await readSnapshot('prompt-tokens', config, + (events) => computePromptTokenDelta(aggregateSessionMetrics(events), {}).nextReported); return parsed && typeof parsed === 'object' ? parsed : {}; } @@ -321,7 +335,8 @@ function hasPromptTokenDelta(d: PromptTokenDelta): boolean { } async function readReportedDailySessions(config: LocalConfig | undefined): Promise { - return (await readSnapshot('daily-sessions', config)) ?? {}; + return (await readSnapshot('daily-sessions', config, + (events) => computeDailyStatsDelta(aggregateDailySessions(events), {}).nextReported)) ?? {}; } async function writeReportedDailySessions(data: ReportedDailySessions, config: LocalConfig | undefined): Promise { @@ -399,7 +414,12 @@ export async function filterEventsByScope( // same whichever earlier runs compaction has dropped. // A second end with nothing recorded since the first (the dashboard // monitor's process_exit after SessionEnd) belongs to the run just closed. + // A start on a fallback ID from another process than its open run's begins + // a new run, though nothing ended that one (a crash with no dashboard + // running). A tool's own ID is left alone: Claude fires SessionStart again + // on resume, in a new process, and its Stop carries the whole transcript. const runOf: Array = []; + const runPid: Array = []; const observedRuns = new Map(); const openRun = new Map(); const closedRun = new Map(); @@ -408,6 +428,10 @@ export async function filterEventsByScope( events.forEach((e, i) => { const ends = e.type === 'session_end' || e.type === 'process_exit'; const observed = e.type === 'process_exit' && typeof e.processExitAfter === 'string'; + const starts = e.type === 'session_start' && typeof e.monitorPid === 'number'; + const open = openRun.get(e.sessionId); + if (starts && open !== undefined && e.sessionId.startsWith('pid-') + && typeof runPid[open] === 'number' && runPid[open] !== e.monitorPid) openRun.delete(e.sessionId); let run = observed ? observedRuns.get(`${e.sessionId}@${e.processExitAfter}`) : openRun.get(e.sessionId) ?? (ends ? closedRun.get(e.sessionId) : undefined); // The observed run may have been compacted. Its delayed exit must not @@ -426,6 +450,7 @@ export async function filterEventsByScope( } else if (!ends) { openRun.set(e.sessionId, run); } + if (starts) runPid[run] ??= e.monitorPid; runOf.push(run); observedRuns.set(`${e.sessionId}@${e.timestamp}`, run); const current = deciding[run]; @@ -447,37 +472,52 @@ export async function filterEventsByScope( /** * A reported snapshot as the run IDs of {@link filterEventsByScope} read it. * Snapshots written before runs had their own IDs are keyed by the bare - * session ID; the first run of that ID in `events` (the scope's, as that - * filter returns them) takes that entry when it has none of its own, and the - * bare entry is retired either way, so no later run of that ID reads it. Only - * an earlier release wrote bare entries, and only for runs it recorded, so a - * run whose first event carries a `dataHomeKey` takes none: the entry may be - * another scope's run under a reused PID-fallback ID. Shared snapshots also - * exclude path-keyed runs: that release already had per-scope snapshots. + * session ID, summed over every run of that ID the release saw. Of those runs + * in `events` (the scope's, as that filter returns them), each but the last is + * taken as reported at its own totals, from `current`, and the last takes the + * bare entry, so none is sent again; a run's own entry is never replaced. The + * bare entry is retired, so no later run of that ID reads it. Only an earlier + * release wrote bare entries, and only for runs it recorded, so a run whose + * first event carries a `dataHomeKey`, and every later run of its ID, takes + * none: the entry may be another scope's run under a reused PID-fallback ID. + * Shared snapshots also exclude path-keyed runs: that release already had + * per-scope snapshots. * Returns `reported` itself when there is nothing to retire; otherwise the * caller persists the result. */ export function adoptBareKeys( reported: Record, events: Iterable, + current: Record, source: 'scope' | 'shared' = 'scope', ): Record { - const adopted = { ...reported }; + const legacyRuns = new Map(); + const recorded = new Set(); const seen = new Set(); - let retired = false; for (const { sessionId: runId, dataHomeKey, dataHome } of events) { + if (seen.has(runId)) continue; + seen.add(runId); const at = runId.lastIndexOf('@'); if (at < 0) continue; const id = runId.slice(0, at); - if (seen.has(id)) continue; - seen.add(id); - if (typeof dataHomeKey === 'string' || !Object.hasOwn(reported, id)) continue; - if (source === 'shared' && typeof dataHome === 'string') continue; - if (!Object.hasOwn(adopted, runId)) adopted[runId] = reported[id]; + if (recorded.has(id) || !Object.hasOwn(reported, id)) continue; + if (typeof dataHomeKey === 'string' || (source === 'shared' && typeof dataHome === 'string')) { + recorded.add(id); + continue; + } + legacyRuns.set(id, [...(legacyRuns.get(id) ?? []), runId]); + } + if (legacyRuns.size === 0) return reported; + const adopted = { ...reported }; + for (const [id, runs] of legacyRuns) { + runs.forEach((runId, i) => { + if (Object.hasOwn(adopted, runId)) return; + if (i === runs.length - 1) adopted[runId] = reported[id]; + else if (Object.hasOwn(current, runId)) adopted[runId] = current[runId]; + }); delete adopted[id]; - retired = true; } - return retired ? adopted : reported; + return adopted; } /** @@ -519,34 +559,40 @@ export async function reportUsageToTeam( const dashboardEvents = await filterEventsByScope(await readEvents(), reportsConfig); const metrics = aggregateSessionMetrics(dashboardEvents); - const currentInterventions = new Map( - [...metrics].map(([sid, m]) => [sid, { interrupt: m.interrupt, toolReject: m.toolReject, correction: m.correction }]), - ); + const currentInterventions = interventionCounts(metrics); + const currentDaily = aggregateDailySessions(dashboardEvents); // A retired bare entry is written out now, even with nothing to report, so // the success writes below, which merge into the file, cannot bring it back. const adopt = async ( read: (config: LocalConfig | undefined) => Promise>, write: (data: Record, config: LocalConfig | undefined) => Promise, + current: Record, ): Promise> => { const stored = await read(reportsConfig); - const adopted = adoptBareKeys(stored, dashboardEvents); + const adopted = adoptBareKeys(stored, dashboardEvents, current); if (adopted !== stored) await write(adopted, reportsConfig); return adopted; }; - const reportedInterventions = await adopt(readReportedInterventions, writeReportedInterventions); + const reportedInterventions = await adopt( + readReportedInterventions, writeReportedInterventions, Object.fromEntries(currentInterventions), + ); const { delta: interventionDelta, nextReported } = computeInterventionDelta( currentInterventions, reportedInterventions, ); - const reportedPromptTokens = await adopt(readReportedPromptTokens, writeReportedPromptTokens); + const reportedPromptTokens = await adopt( + readReportedPromptTokens, writeReportedPromptTokens, computePromptTokenDelta(metrics, {}).nextReported, + ); const { delta: promptTokenDelta, nextReported: nextReportedPromptTokens } = computePromptTokenDelta( metrics, reportedPromptTokens, ); - const reportedDailySessions = await adopt(readReportedDailySessions, writeReportedDailySessions); + const reportedDailySessions = await adopt( + readReportedDailySessions, writeReportedDailySessions, computeDailyStatsDelta(currentDaily, {}).nextReported, + ); const { delta: dailyDelta, nextReported: nextReportedDailySessions } = computeDailyStatsDelta( - aggregateDailySessions(dashboardEvents), + currentDaily, reportedDailySessions, ); From 18dcd77a6084e249c2b01dabe5f04ccb76e8602a Mon Sep 17 00:00:00 2001 From: Saul Moro Date: Thu, 24 Sep 2026 18:13:02 +0200 Subject: [PATCH 15/41] fix(stats): address CI review (#785) MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit An end splits runs only on a fallback ID (pid-…). A tool's own session ID is one session whatever ends it records: claude --resume continues it in a new process, and its Stop carries the whole transcript, so a second run counted it again, maybe in another scope. --- docs/designs/data-directory-layout.md | 8 +++--- docs/usage-guide.md | 7 ++--- docs/usage-guide.zh-CN.md | 4 +-- src/__tests__/dashboard-report-scope.test.ts | 15 +++++++++++ src/__tests__/scope-filter.test.ts | 16 ++++++++++++ src/team-push.ts | 27 ++++++++++---------- 6 files changed, 56 insertions(+), 21 deletions(-) diff --git a/docs/designs/data-directory-layout.md b/docs/designs/data-directory-layout.md index 0cfee4855..f1ac1b561 100644 --- a/docs/designs/data-directory-layout.md +++ b/docs/designs/data-directory-layout.md @@ -351,9 +351,11 @@ carries `dataHomeKey`, a hash of the realpath'd `getDataHome()` of the scope the hook resolved (#785; a hash, so a Copilot event still stores no path), and a scope's report keeps the sessions whose first keyed event is its own, whole: a Stop carries the whole transcript's totals, so a session that moved scope (a `cd` -mid-session) is reported once, where it started. A session is one ID's run up to -its `session_end` or `process_exit`, so a later run that reuses the ID (Copilot's -fallback ID is the parent PID) is decided on its own; a second end with nothing +mid-session) is reported once, where it started. A tool's own session ID is one +session whatever ends it records: `claude --resume` continues it, in a new +process, and its Stop carries the whole transcript. A fallback ID (`pid-…`; +Copilot's is the parent PID) names one run up to its `session_end` or +`process_exit`, so a later run that reuses it is decided on its own; a second end with nothing recorded since the first (the dashboard monitor's `process_exit` after `SessionEnd`) belongs to the run it closed. A `session_start` on a fallback ID (`pid-…`) whose `monitorPid` differs from its open run's begins a new run even diff --git a/docs/usage-guide.md b/docs/usage-guide.md index 1eed08c63..056a89f2a 100644 --- a/docs/usage-guide.md +++ b/docs/usage-guide.md @@ -1763,9 +1763,10 @@ again. Events recorded by an earlier release carry no key: the scope their directory resolves to now reports them (a nested clone under a project resolves to its own project or the user scope, not the enclosing project); events with no directory, or one removed since, are reported by no one. Each scope also -keeps its own snapshot of what it already reported, and a session that reuses -an earlier session's ID (Copilot reuses its fallback IDs) counts as new, -whichever scope reported the earlier one; the first +keeps its own snapshot of what it already reported, and a session under a +reused fallback ID (Copilot's PID-based ID when it sends none) counts as new, +whichever scope reported the earlier one, while a resumed session +(`claude --resume`) keeps its ID and stays one session; the first report after upgrading starts from the snapshot every scope used to share, so nothing is reported twice. A target removes its usage events only after it confirms success; failed pushes diff --git a/docs/usage-guide.zh-CN.md b/docs/usage-guide.zh-CN.md index 3b1dcb81d..1ab490444 100644 --- a/docs/usage-guide.zh-CN.md +++ b/docs/usage-guide.zh-CN.md @@ -1695,8 +1695,8 @@ Copilot 会话以及从软链接路径启动的会话。每个会话只上报一 scope，即使之后切换到另一个项目：它的 Stop 带有整份 transcript 的累计值，第二个 scope 会重复计算。旧版本记录的事件没有该键：由其目录当前解析到的 scope 上报（项目下的嵌套 clone 解析到它自己的项目或 user scope，而不是外层项目）；没有目录或目录已删除的事件不由任何 scope 上报。 -每个 scope 还各自保存已上报快照，且复用先前会话 ID 的会话（Copilot 会复用其回退 ID） -总算作新会话，无论先前那个由哪个 scope 上报；升级后的首次上报从原先各 scope 共用的快照开始，不会重复上报。目标确认成功后才清理自己的使用事件， +每个 scope 还各自保存已上报快照，且复用回退 ID 的新会话（Copilot 未提供会话 ID 时基于 PID 的 ID） +总算作新会话，无论先前那个由哪个 scope 上报；恢复的会话（`claude --resume`）保留原 ID，仍是同一个会话；升级后的首次上报从原先各 scope 共用的快照开始，不会重复上报。目标确认成功后才清理自己的使用事件， 推送失败会保留事件（最多保留最新 5,000 条，见下文）。上报完成前继续持有相关同步锁， 避免另一次 Pull 与尚未完成的上报竞争。 diff --git a/src/__tests__/dashboard-report-scope.test.ts b/src/__tests__/dashboard-report-scope.test.ts index 50977ee2f..40709dbe7 100644 --- a/src/__tests__/dashboard-report-scope.test.ts +++ b/src/__tests__/dashboard-report-scope.test.ts @@ -448,6 +448,21 @@ describe('each scope keeps its own reported snapshot (#786)', () => { expect(await report(project)).toBeNull(); }); + it('a session resumed in a project after it ended is still the one the user scope reported', async () => { + const { root, user, project } = await setup(); + const elsewhere = path.join(tmp, 'elsewhere'); + fs.mkdirSync(elsewhere); + await session('claude', { session_id: 'resumed', cwd: elsewhere }); + await hook('session-end', 'claude', { session_id: 'resumed', cwd: elsewhere, hook_event_name: 'SessionEnd' }); + expect(await reportedSessions(user)).toBe(1); + // `claude --resume` in P: the same session ID, whose Stop carries the whole transcript. + await session('claude', { session_id: 'resumed', cwd: root }); + + expect(await reportedSessions(project)).toBe(0); + expect(await reportedSessions(user)).toBe(1); + expect(await reportedPrompts(user)).toBe(2); + }); + it('a fallback ID reused in a project after a user-scope run that never ended is the project\'s', async () => { const { root, user, project } = await setup(); const elsewhere = path.join(tmp, 'elsewhere'); diff --git a/src/__tests__/scope-filter.test.ts b/src/__tests__/scope-filter.test.ts index 26036ae61..3e6901c66 100644 --- a/src/__tests__/scope-filter.test.ts +++ b/src/__tests__/scope-filter.test.ts @@ -136,6 +136,22 @@ describe('filterEventsByScope', () => { } }); + it('a tool\'s own session ID stays one run across an end and a resume', async () => { + // `claude --resume` after the dashboard recorded the exit: its Stop carries + // the whole transcript, so a second run would count it again. + for (const end of ['session_end', 'process_exit'] as const) { + const log = [ + await scopedEvent(undefined, 'claude-uuid', '/home/jeff/.teamai'), + { ...(await scopedEvent(undefined, 'claude-uuid', '/home/jeff/.teamai')), type: end }, + { ...(await scopedEvent(undefined, 'claude-uuid', '/home/jeff/.teamai/projects/p')), type: 'session_start' as const }, + await scopedEvent(undefined, 'claude-uuid', '/home/jeff/.teamai/projects/p'), + ]; + const user = await filterEventsByScope(log, userScope()); + expect(user.map((e) => e.sessionId)).toEqual(Array(4).fill(`claude-uuid@${log[0].timestamp}`)); + expect(await ids(log, projectScope('/Users/jeff/project-a'))).toEqual([]); + } + }); + it('a fallback ID started by another process is a new run, though the last one never ended', async () => { // The user-scope run crashed with no dashboard running, so nothing ended it. const start = (sessionId: string, dataHome: string, monitorPid: number) => scopedEvent(undefined, sessionId, dataHome) diff --git a/src/team-push.ts b/src/team-push.ts index e8283e4b6..e7104fd7a 100644 --- a/src/team-push.ts +++ b/src/team-push.ts @@ -408,16 +408,16 @@ export async function filterEventsByScope( } return key; })); - // A session ID names one run until it ends: a PID-fallback ID comes back for - // a later run, maybe in another scope, so each run is decided on its own and - // returned under the ID plus its first event's timestamp, which stays the - // same whichever earlier runs compaction has dropped. - // A second end with nothing recorded since the first (the dashboard - // monitor's process_exit after SessionEnd) belongs to the run just closed. - // A start on a fallback ID from another process than its open run's begins - // a new run, though nothing ended that one (a crash with no dashboard - // running). A tool's own ID is left alone: Claude fires SessionStart again - // on resume, in a new process, and its Stop carries the whole transcript. + // Each run is returned under its session ID plus its first event's + // timestamp, which stays the same whichever earlier runs compaction has + // dropped. A tool's own session ID is one run, whatever ends it records: + // `claude --resume` continues it, in a new process, and its Stop carries the + // whole transcript. A PID-fallback ID (`pid-…`) names one run until it ends, + // then comes back for a later one, maybe in another scope, so each run is + // decided on its own. A second end with nothing recorded since the first + // (the dashboard monitor's process_exit after SessionEnd) belongs to the run + // just closed. A start from another process than its open run's begins a new + // run, though nothing ended that one (a crash with no dashboard running). const runOf: Array = []; const runPid: Array = []; const observedRuns = new Map(); @@ -426,11 +426,12 @@ export async function filterEventsByScope( const deciding: Array = []; const runIds: string[] = []; events.forEach((e, i) => { + const fallback = e.sessionId.startsWith('pid-'); const ends = e.type === 'session_end' || e.type === 'process_exit'; const observed = e.type === 'process_exit' && typeof e.processExitAfter === 'string'; const starts = e.type === 'session_start' && typeof e.monitorPid === 'number'; const open = openRun.get(e.sessionId); - if (starts && open !== undefined && e.sessionId.startsWith('pid-') + if (starts && open !== undefined && fallback && typeof runPid[open] === 'number' && runPid[open] !== e.monitorPid) openRun.delete(e.sessionId); let run = observed ? observedRuns.get(`${e.sessionId}@${e.processExitAfter}`) : openRun.get(e.sessionId) ?? (ends ? closedRun.get(e.sessionId) : undefined); @@ -444,10 +445,10 @@ export async function filterEventsByScope( run = deciding.push(undefined) - 1; runIds.push(`${e.sessionId}@${e.timestamp}`); } - if (ends && (!observed || openRun.get(e.sessionId) === run)) { + if (ends && fallback && (!observed || openRun.get(e.sessionId) === run)) { openRun.delete(e.sessionId); closedRun.set(e.sessionId, run); - } else if (!ends) { + } else if (!ends || !fallback) { openRun.set(e.sessionId, run); } if (starts) runPid[run] ??= e.monitorPid; From ae037b36fd019b69978d8067de7df30e53c66887 Mon Sep 17 00:00:00 2001 From: Saul Moro Date: Thu, 24 Sep 2026 18:36:58 +0200 Subject: [PATCH 16/41] fix(stats): address CI review (#785) A tool's own session ID is keyed by the ID itself again, as on main, not by its first event's timestamp, so a session resumed after compaction dropped its events still reads what its scope reported. Only PID-fallback runs carry the timestamp. A bare fallback entry is the sum of the runs of its ID in the log at the earlier release's last report, and compaction keeps or drops an ID's runs together. Those runs now consume it in log order, each up to its own totals, so a later run that release never reported is sent instead of taking the whole entry. The prompt-token snapshot decides which runs it covered; interventions and daily follow it, and the first run always takes a share. --- docs/designs/data-directory-layout.md | 29 +-- src/__tests__/dashboard-report-scope.test.ts | 34 ++++ src/__tests__/e2e/report-timeout.test.ts | 4 +- src/__tests__/scope-filter.test.ts | 69 ++++--- src/__tests__/team-push-interventions.test.ts | 21 +-- src/session-trends.ts | 107 +++++++++-- src/stats.ts | 9 +- src/team-push.ts | 177 +++++++++++++----- 8 files changed, 332 insertions(+), 118 deletions(-) diff --git a/docs/designs/data-directory-layout.md b/docs/designs/data-directory-layout.md index f1ac1b561..be57071c0 100644 --- a/docs/designs/data-directory-layout.md +++ b/docs/designs/data-directory-layout.md @@ -364,18 +364,23 @@ ID is not split this way, since Claude fires SessionStart again on resume, in a new process, and its Stop carries the whole transcript. The monitor's `process_exit` also records `processExitAfter`, the last event it observed, and closes only that run: an exit appended after the next run of the same ID began does not end it, -and one whose run compaction dropped is ignored. Every run is reported and -snapshotted as `@`, which does not change when -compaction drops earlier runs. A snapshot entry keyed by the bare ID (written -before) sums every run of that ID the earlier release saw, so the next time the -scope reports, each of those runs in the log but the last is taken as reported -at its own totals, the last takes the entry, and the entry is removed, so no run -is sent again and no later run of that ID reads it. What the last run records -beyond the entry is reported; it is undercounted by at most the other runs' -share of the sum, once (when it was still growing at the upgrade, or had not been -reported yet). Only an earlier release wrote bare entries, and a seeded one -may be another scope's, so a run whose first event carries `dataHomeKey` -(recorded by this release), and every later run of its ID, takes none. A session written before that field existed is attributed by its first +and one whose run compaction dropped is ignored. A tool's own session ID is +reported and snapshotted under the ID itself, as before, so a session resumed +after compaction dropped its events still reads what its scope reported; a +fallback run is reported and snapshotted as `@`, +which does not change when compaction drops earlier runs. A snapshot entry keyed +by a bare fallback ID (written before) is the sum of the runs of that ID in the +log at the earlier release's last report, and compaction keeps or drops the +runs of an ID together. So the next time the scope reports, those runs consume +the entry in log order, each taking up to its own totals of what is left; once +the prompt-token entry is used up, the later runs were not reported and count +as new sessions (the interventions and daily snapshots follow the prompt-token +one, since their counts say nothing when they run out). The first run always +takes a share, as the entry means that release reported it. The entry is then +removed, so no later run of that ID reads it. Only an earlier release wrote bare +entries, and a seeded one may be another scope's, so a run whose first event +carries `dataHomeKey` (recorded by this release), and every later run of its ID, +takes none. A session written before that field existed is attributed by its first `cwd`: to the scope `resolveConfigForDir` resolves that directory to now, the dispatcher's rule, so a nested clone under a project is not the project's; no `cwd`, or one removed since, is no scope's. The snapshots of what was already reported are per scope diff --git a/src/__tests__/dashboard-report-scope.test.ts b/src/__tests__/dashboard-report-scope.test.ts index 40709dbe7..6aa9551c9 100644 --- a/src/__tests__/dashboard-report-scope.test.ts +++ b/src/__tests__/dashboard-report-scope.test.ts @@ -463,6 +463,40 @@ describe('each scope keeps its own reported snapshot (#786)', () => { expect(await reportedPrompts(user)).toBe(2); }); + it('a session resumed after compaction dropped its events is still the one already reported', async () => { + const { root, project } = await setup(); + await session('claude', { session_id: 'resumed', cwd: root }); + await hook('session-end', 'claude', { session_id: 'resumed', cwd: root, hook_event_name: 'SessionEnd' }); + expect(await reportedSessions(project)).toBe(1); + fs.writeFileSync(path.join(teamaiHome(), 'dashboard', 'events.jsonl'), ''); + // `claude --resume`: the same session ID, compared against what P reported. + await session('claude', { session_id: 'resumed', cwd: root }); + + expect(await reportedSessions(project)).toBe(1); + expect(await reportedPrompts(project)).toBe(1); + }); + + it.each([ + ['a shared snapshot, before #785', false], + ['the scope\'s own snapshot, main since #795', true], + ])('a later run of a bare ID that release never reported is sent: %s', async (_, ownSnapshot) => { + const { root, project } = await setup(); + const [tool, payload] = ownSnapshot ? ['copilot', {}] : ['claude', { cwd: root }]; + const run = async () => { + await session(tool, { ...payload, cwd: root }); + await hook('session-end', tool, { ...payload, cwd: root, hook_event_name: 'SessionEnd' }); + }; + // That release reported the first run; the second came after its last report. + await asEarlierRelease(async () => { await run(); await run(); }, ownSnapshot ? getDataHome(project) : undefined); + const events = fs.readFileSync(path.join(teamaiHome(), 'dashboard', 'events.jsonl'), 'utf-8'); + const id: string = JSON.parse(events.split('\n')[0]).sessionId; + writeSharedSnapshots({ [id]: 1 }, new Date().toISOString().slice(0, 10), ownSnapshot ? project : undefined); + + expect(await reportedSessions(project)).toBe(1); + expect(await reportedInterventionSessions(project)).toBe(1); + expect(await reportedPrompts(project)).toBe(1); + }); + it('a fallback ID reused in a project after a user-scope run that never ended is the project\'s', async () => { const { root, user, project } = await setup(); const elsewhere = path.join(tmp, 'elsewhere'); diff --git a/src/__tests__/e2e/report-timeout.test.ts b/src/__tests__/e2e/report-timeout.test.ts index ad0e025d2..4eed66517 100644 --- a/src/__tests__/e2e/report-timeout.test.ts +++ b/src/__tests__/e2e/report-timeout.test.ts @@ -92,8 +92,8 @@ function fixture(agent: keyof typeof agents, provider: string, team: Record { - return (await filterEventsByScope(events, config)).map((e) => e.sessionId.slice(0, e.sessionId.lastIndexOf('@'))); + return (await filterEventsByScope(events, config)).map((e) => + e.sessionId.startsWith('pid-') ? e.sessionId.slice(0, e.sessionId.lastIndexOf('@')) : e.sessionId); } describe('filterEventsByScope', () => { @@ -147,7 +148,8 @@ describe('filterEventsByScope', () => { await scopedEvent(undefined, 'claude-uuid', '/home/jeff/.teamai/projects/p'), ]; const user = await filterEventsByScope(log, userScope()); - expect(user.map((e) => e.sessionId)).toEqual(Array(4).fill(`claude-uuid@${log[0].timestamp}`)); + // Keyed by the ID alone, so it stays the same session after compaction drops its events. + expect(user.map((e) => e.sessionId)).toEqual(Array(4).fill('claude-uuid')); expect(await ids(log, projectScope('/Users/jeff/project-a'))).toEqual([]); } }); @@ -265,55 +267,72 @@ describe('adoptBareKeys', () => { /** A run's event as filterEventsByScope returns it; an earlier release's unless `fields` key it. */ const run = (sessionId: string, fields: Partial = {}): DashboardEvent => ({ type: 'prompt_submit', timestamp: 't', sessionId, tool: 'copilot', ...fields }); + /** A counter's share: up to the run's own, while any of the sum is left. */ + const take = (own: number, left: number) => (left > 0 ? { taken: Math.min(own, left), left: left - Math.min(own, left) } : undefined); + + it('consumes a bare entry, the sum of its runs, across them in order', () => { + // An earlier release summed every run of the ID under it: each run takes up + // to its own totals of what is left, so none is sent again. + const events = [run('pid-1@t1'), run('pid-1@t2'), run('s2@t0'), run('s3@t3')]; + const reported = { 'pid-1': 6, 's2@t0': 1 }; + expect(adoptBareKeys(reported, events, { 'pid-1@t1': 5, 'pid-1@t2': 3, 's3@t3': 4 }, take)) + .toEqual({ 'pid-1@t1': 5, 'pid-1@t2': 1, 's2@t0': 1 }); + }); - it('gives a snapshot entry keyed by the bare session ID to the last run of that ID', () => { - // An earlier release summed every run of the ID under it: the earlier runs are - // reported at their own totals, the last takes the sum, so none is sent again. - const reported = { 'pid-1': 3, 's2@t0': 1 }; - const current = { 'pid-1@t1': 1, 'pid-1@t2': 2, 's2@t0': 1, 's3@t3': 4 }; - const adopted = adoptBareKeys(reported, [run('pid-1@t1'), run('pid-1@t2'), run('s2@t0'), run('s3@t3')], current); - expect(adopted).toEqual({ 'pid-1@t1': 1, 'pid-1@t2': 3, 's2@t0': 1 }); + it('a later run the sum does not reach takes nothing: it was not reported', () => { + // Run 1 reported as 5; run 2 came after the last report of that release. + const adopted = adoptBareKeys({ 'pid-1': 5 }, [run('pid-1@t1'), run('pid-1@t2')], { 'pid-1@t1': 5, 'pid-1@t2': 3 }, take); + expect(adopted).toEqual({ 'pid-1@t1': 5 }); }); - it('a single run of the ID takes the bare entry', () => { - expect(adoptBareKeys({ 'pid-1': 3 }, [run('pid-1@t1')], { 'pid-1@t1': 5 })).toEqual({ 'pid-1@t1': 3 }); + it('the first run takes the entry even when nothing of it is left to share', () => { + // A bare entry means that release reported the ID, so its first run is not new. + expect(adoptBareKeys({ 'pid-1': 0 }, [run('pid-1@t1'), run('pid-1@t2')], { 'pid-1@t1': 2, 'pid-1@t2': 1 }, take)) + .toEqual({ 'pid-1@t1': 0 }); }); - it('never overrides a run\'s own entry', () => { - expect(adoptBareKeys({ 'pid-1': 3, 'pid-1@t1': 5 }, [run('pid-1@t1')], {})['pid-1@t1']).toBe(5); - expect(adoptBareKeys({ 'pid-1': 3, 'pid-1@t1': 5 }, [run('pid-1@t1'), run('pid-1@t2')], { 'pid-1@t1': 7 })) - .toEqual({ 'pid-1@t1': 5, 'pid-1@t2': 3 }); + it('never overrides a run\'s own entry, which still takes its share', () => { + expect(adoptBareKeys({ 'pid-1': 3, 'pid-1@t1': 5 }, [run('pid-1@t1')], { 'pid-1@t1': 5 }, take)['pid-1@t1']).toBe(5); + expect(adoptBareKeys({ 'pid-1': 3, 'pid-1@t1': 5 }, [run('pid-1@t1'), run('pid-1@t2')], { 'pid-1@t1': 2, 'pid-1@t2': 4 }, take)) + .toEqual({ 'pid-1@t1': 5, 'pid-1@t2': 1 }); }); it('retires the bare entry its runs read, so it is never read again', () => { - expect(adoptBareKeys({ 'pid-1': 3, s2: 1 }, [run('pid-1@t1'), run('pid-1@t2')], { 'pid-1@t1': 1 })) - .toEqual({ 'pid-1@t1': 1, 'pid-1@t2': 3, s2: 1 }); - expect(adoptBareKeys({ 'pid-1': 3, 'pid-1@t1': 5 }, [run('pid-1@t1')], {})).toEqual({ 'pid-1@t1': 5 }); + expect(adoptBareKeys({ 'pid-1': 3, s2: 1 }, [run('pid-1@t1'), run('pid-1@t2')], { 'pid-1@t1': 1, 'pid-1@t2': 5 }, take)) + .toEqual({ 'pid-1@t1': 1, 'pid-1@t2': 2, s2: 1 }); + }); + + it('leaves a tool\'s own session ID alone: its run is keyed by the ID itself', () => { + const reported = { 'claude-uuid': 3 }; + expect(adoptBareKeys(reported, [run('claude-uuid')], { 'claude-uuid': 4 }, take)).toBe(reported); }); it('returns the snapshot itself when it holds no bare entry to retire', () => { const reported = { 'pid-1@t1': 5, s2: 1 }; - expect(adoptBareKeys(reported, [run('pid-1@t1'), run('s3@t3')], {})).toBe(reported); + expect(adoptBareKeys(reported, [run('pid-1@t1'), run('s3@t3')], { 'pid-1@t1': 5 }, take)).toBe(reported); }); it('gives no bare entry to a run this build recorded: only an earlier release wrote them', () => { // Another scope's run, maybe: the bare entry says nothing about which scope reported it. const reported = { 'pid-1': 3 }; - expect(adoptBareKeys(reported, [run('pid-1@t1', { dataHomeKey: 'k' }), run('pid-1@t2')], {})).toBe(reported); + expect(adoptBareKeys(reported, [run('pid-1@t1', { dataHomeKey: 'k' }), run('pid-1@t2')], { 'pid-1@t1': 1, 'pid-1@t2': 1 }, take)) + .toBe(reported); }); it('a run recorded by main since #795, which wrote the data home path, is an earlier release\'s', () => { - expect(adoptBareKeys({ 'pid-1': 3 }, [run('pid-1@t1', { dataHome: '/home/jeff/.teamai' })], {})).toEqual({ 'pid-1@t1': 3 }); + expect(adoptBareKeys({ 'pid-1': 3 }, [run('pid-1@t1', { dataHome: '/home/jeff/.teamai' })], { 'pid-1@t1': 3 }, take)) + .toEqual({ 'pid-1@t1': 3 }); }); it('does not give a path-keyed run a shared bare entry, but still adopts its own scope entry', () => { const reported = { 'pid-1': 3 }; const events = [run('pid-1@t1', { dataHome: '/home/jeff/.teamai/projects/p' })]; - expect(adoptBareKeys(reported, events, {}, 'shared')).toBe(reported); - expect(adoptBareKeys(reported, events, {}, 'scope')).toEqual({ 'pid-1@t1': 3 }); + expect(adoptBareKeys(reported, events, { 'pid-1@t1': 3 }, take, 'shared')).toBe(reported); + expect(adoptBareKeys(reported, events, { 'pid-1@t1': 3 }, take, 'scope')).toEqual({ 'pid-1@t1': 3 }); }); it('a run in progress across the upgrade is an earlier release\'s, by its first event', () => { - expect(adoptBareKeys({ 'pid-1': 3 }, [run('pid-1@t1'), run('pid-1@t1', { dataHomeKey: 'k' })], {})).toEqual({ 'pid-1@t1': 3 }); + expect(adoptBareKeys({ 'pid-1': 3 }, [run('pid-1@t1'), run('pid-1@t1', { dataHomeKey: 'k' })], { 'pid-1@t1': 4 }, take)) + .toEqual({ 'pid-1@t1': 3 }); }); }); diff --git a/src/__tests__/team-push-interventions.test.ts b/src/__tests__/team-push-interventions.test.ts index a997c4676..040be75bf 100644 --- a/src/__tests__/team-push-interventions.test.ts +++ b/src/__tests__/team-push-interventions.test.ts @@ -96,8 +96,7 @@ function writeDashboardEvents(lines: object[]): void { } describe('reportUsageToTeam — intervention reporting', () => { - /** The usage file, and the snapshot key of the seeded session under a scope config (its run ID, #785). */ - function seedReport(): { usagePath: string; run: string } { + function seedReport(): string { const timestamp = new Date().toISOString(); writeDashboardEvents([ { type: 'session_start', timestamp, sessionId: 'slow', tool: 'claude', cwd: '/p' }, @@ -108,14 +107,12 @@ describe('reportUsageToTeam — intervention reporting', () => { ]); const usagePath = path.join(tmpDir, '.teamai', 'user-usage.jsonl'); fs.writeFileSync(usagePath, JSON.stringify({ skill: 'review', timestamp, tool: 'claude' }) + '\n'); - return { usagePath, run: `slow@${timestamp}` }; + return usagePath; } it.each(['reports', 'legacy'])('finishes acknowledgement after the caller times out (%s)', async (backend) => { vi.useFakeTimers({ toFake: ['setTimeout', 'clearTimeout'] }); - const { usagePath, run } = seedReport(); - // A caller without a scope config reads the whole log, under the bare session IDs. - const key = backend === 'reports' ? run : 'slow'; + const usagePath = seedReport(); const seeded = fs.readFileSync(usagePath, 'utf-8'); let finish!: (value: boolean) => void; let started!: () => void; @@ -139,7 +136,7 @@ describe('reportUsageToTeam — intervention reporting', () => { await vi.advanceTimersByTimeAsync(5000); await timeout; expect(fs.readFileSync(usagePath, 'utf-8')).toContain('review'); - expect(reportedSnapshot('prompt-tokens', backend === 'reports')[key]).toBeUndefined(); + expect(reportedSnapshot('prompt-tokens', backend === 'reports').slow).toBeUndefined(); finish(true); expect(await operation).toBe(true); @@ -147,7 +144,7 @@ describe('reportUsageToTeam — intervention reporting', () => { // so it leaves the user-scope file for the scope that owns it. expect(fs.readFileSync(usagePath, 'utf-8')).toBe(backend === 'reports' ? '' : seeded); for (const name of ['interventions', 'prompt-tokens', 'daily-sessions']) { - expect(reportedSnapshot(name, backend === 'reports')[key]).toBeDefined(); + expect(reportedSnapshot(name, backend === 'reports').slow).toBeDefined(); } const statsPath = backend === 'reports' ? reportsStatsPath() : path.join(repoDir, 'stats', 'me.yaml'); const before = fs.readFileSync(statsPath, 'utf-8'); @@ -156,14 +153,14 @@ describe('reportUsageToTeam — intervention reporting', () => { }); it.each(['false', 'rejection'])('retains events and snapshots when push returns %s', async (failure) => { - const { usagePath, run } = seedReport(); + const usagePath = seedReport(); const before = fs.readFileSync(usagePath, 'utf-8'); if (failure === 'false') reportsMocks.updateReports.mockResolvedValueOnce(false); else reportsMocks.updateReports.mockRejectedValueOnce(new Error('offline')); expect(await reportUsageToTeam(repoDir, 'me', { selfConfig: gitConfig() })).toBe(false); expect(fs.readFileSync(usagePath, 'utf-8')).toBe(before); for (const name of ['interventions', 'prompt-tokens', 'daily-sessions']) { - expect(reportedSnapshot(name)[run]).toBeUndefined(); + expect(reportedSnapshot(name).slow).toBeUndefined(); } expect(await reportUsageToTeam(repoDir, 'me', { selfConfig: gitConfig() })).toBe(true); const stats = YAML.parse(fs.readFileSync(reportsStatsPath(), 'utf-8')); @@ -193,9 +190,9 @@ describe('reportUsageToTeam — intervention reporting', () => { // reported snapshot persisted so a second run reports nothing new expect(reportedSnapshot('interventions')).toEqual({ - [`s1@${ts}`]: { interrupt: 2, toolReject: 1, correction: 0 }, + s1: { interrupt: 2, toolReject: 1, correction: 0 }, }); - expect(reportedSnapshot('daily-sessions')[`s1@${ts}`]).toMatchObject({ date: ts.slice(0, 10) }); + expect(reportedSnapshot('daily-sessions').s1).toMatchObject({ date: ts.slice(0, 10) }); reportsMocks.updateReports.mockClear(); await reportUsageToTeam(repoDir, 'me', { selfConfig: gitConfig() }); diff --git a/src/session-trends.ts b/src/session-trends.ts index b36299bed..d3eb8b35d 100644 --- a/src/session-trends.ts +++ b/src/session-trends.ts @@ -97,6 +97,88 @@ function positiveDelta(current: number, previous: number | undefined): number { return Math.max(0, current - (previous ?? 0)); } +/** A reported snapshot's request costs per day, from its session-level fields if it predates them. */ +function reportedRequestDaily(previous: DailySessionSnapshot | undefined): Record { + return previous?.requestDaily ?? ( + previous?.pricedRequests || previous?.costMicros || previous?.cacheReadTokens || previous?.cacheEligibleInputTokens + ? { [previous.date]: { + pricedRequests: previous.pricedRequests ?? 0, + costMicros: previous.costMicros ?? 0, + cacheReadTokens: previous.cacheReadTokens ?? 0, + cacheEligibleInputTokens: previous.cacheEligibleInputTokens ?? 0, + priceVersion: previous.priceVersion ?? '', + } } + : {} + ); +} + +/** + * A reported snapshot's session cache tokens. A session reported under the old + * code has no sessionCache* fields but does have requestDaily cache: summed as + * the baseline, so the first post-upgrade report doesn't re-add counted tokens. + */ +function reportedSessionCache( + previous: DailySessionSnapshot | undefined, + requestDaily: Record, +): { read: number; eligible: number } { + return { + read: previous?.sessionCacheReadTokens ?? Object.values(requestDaily).reduce((sum, r) => sum + r.cacheReadTokens, 0), + eligible: previous?.sessionCacheEligibleTokens + ?? Object.values(requestDaily).reduce((sum, r) => sum + r.cacheEligibleInputTokens, 0), + }; +} + +/** + * `run`'s share of `left`, what remains of the snapshot an earlier release kept + * for all the runs of one session ID: each counter up to the run's own. The + * success and correction flags describe the one session that release counted, + * so each run it covered reads them as they are. + */ +export function takeDailySession( + run: DailySessionSnapshot, + left: DailySessionSnapshot, +): { taken: DailySessionSnapshot; left: DailySessionSnapshot } { + const upTo = (own: number, rest: number) => Math.max(0, Math.min(own, rest)); + const leftDaily = reportedRequestDaily(left); + const leftCache = reportedSessionCache(left, leftDaily); + const takenDaily: Record = {}; + const restDaily: Record = { ...leftDaily }; + for (const [date, request] of Object.entries(run.requestDaily)) { + const rest = leftDaily[date]; + if (!rest) continue; + const taken: RequestCostMetrics = { + pricedRequests: upTo(request.pricedRequests, rest.pricedRequests), + costMicros: upTo(request.costMicros, rest.costMicros), + cacheReadTokens: upTo(request.cacheReadTokens, rest.cacheReadTokens), + cacheEligibleInputTokens: upTo(request.cacheEligibleInputTokens, rest.cacheEligibleInputTokens), + priceVersion: rest.priceVersion, + }; + takenDaily[date] = taken; + restDaily[date] = { + pricedRequests: rest.pricedRequests - taken.pricedRequests, + costMicros: rest.costMicros - taken.costMicros, + cacheReadTokens: rest.cacheReadTokens - taken.cacheReadTokens, + cacheEligibleInputTokens: rest.cacheEligibleInputTokens - taken.cacheEligibleInputTokens, + priceVersion: rest.priceVersion, + }; + } + const prompts = upTo(run.prompts, left.prompts); + const durationMs = upTo(run.durationMs, left.durationMs); + const cacheRead = upTo(run.sessionCacheReadTokens ?? 0, leftCache.read); + const cacheEligible = upTo(run.sessionCacheEligibleTokens ?? 0, leftCache.eligible); + return { + taken: { + date: run.date, prompts, durationMs, succeeded: left.succeeded, corrected: left.corrected, + requestDaily: takenDaily, sessionCacheReadTokens: cacheRead, sessionCacheEligibleTokens: cacheEligible, + }, + left: { + date: left.date, prompts: left.prompts - prompts, durationMs: left.durationMs - durationMs, + succeeded: left.succeeded, corrected: left.corrected, requestDaily: restDaily, + sessionCacheReadTokens: leftCache.read - cacheRead, sessionCacheEligibleTokens: leftCache.eligible - cacheEligible, + }, + }; +} + /** Compute idempotent daily deltas while keeping resumed work on the first Stop day. */ export function computeDailyStatsDelta( current: Map, @@ -121,28 +203,13 @@ export function computeDailyStatsDelta( bucket.sessionsCorrected += positiveDelta(snapshot.corrected, previous?.corrected); delta[date] = bucket; - const previousDaily = previous?.requestDaily ?? ( - previous?.pricedRequests || previous?.costMicros || previous?.cacheReadTokens || previous?.cacheEligibleInputTokens - ? { [previous.date]: { - pricedRequests: previous.pricedRequests ?? 0, - costMicros: previous.costMicros ?? 0, - cacheReadTokens: previous.cacheReadTokens ?? 0, - cacheEligibleInputTokens: previous.cacheEligibleInputTokens ?? 0, - priceVersion: previous.priceVersion ?? '', - } } - : {} - ); + const previousDaily = reportedRequestDaily(previous); // Cache-read share is pricing-independent: fold session-level cache tokens onto // the firstStop day (like prompts/duration), so it works even when the model - // can't be priced. Legacy fallback: a session reported under the old code has no - // sessionCache* fields but does have requestDaily cache — sum it as the baseline - // so this first post-upgrade report doesn't re-add already-counted tokens. - const prevCacheRead = previous?.sessionCacheReadTokens - ?? Object.values(previousDaily).reduce((sum, r) => sum + r.cacheReadTokens, 0); - const prevCacheEligible = previous?.sessionCacheEligibleTokens - ?? Object.values(previousDaily).reduce((sum, r) => sum + r.cacheEligibleInputTokens, 0); - bucket.cacheReadTokens += positiveDelta(snapshot.sessionCacheReadTokens ?? 0, prevCacheRead); - bucket.cacheEligibleInputTokens += positiveDelta(snapshot.sessionCacheEligibleTokens ?? 0, prevCacheEligible); + // can't be priced. + const prevCache = reportedSessionCache(previous, previousDaily); + bucket.cacheReadTokens += positiveDelta(snapshot.sessionCacheReadTokens ?? 0, prevCache.read); + bucket.cacheEligibleInputTokens += positiveDelta(snapshot.sessionCacheEligibleTokens ?? 0, prevCache.eligible); delta[date] = bucket; // Cost stays per-requestDate from pricing; cache no longer flows through here. for (const [requestDate, request] of Object.entries(snapshot.requestDaily)) { diff --git a/src/stats.ts b/src/stats.ts index e169a805f..b27e9c69f 100644 --- a/src/stats.ts +++ b/src/stats.ts @@ -158,16 +158,19 @@ async function unreportedDashboardStats( ): Promise { const { adoptBareKeys, computeInterventionDelta, computePromptTokenDelta, readReportedInterventions, readReportedPromptTokens, + takeInterventions, takePromptTokens, } = await import('./team-push.js'); // The scope's own snapshots, the ones its report compares against (#786). const currentInterventions = new Map( [...metrics].map(([sid, m]) => [sid, { interrupt: m.interrupt, toolReject: m.toolReject, correction: m.correction }]), ); + // Prompt tokens first: they decide which runs of a bare ID were reported. + const promptTokens = adoptBareKeys( + await readReportedPromptTokens(config), events, computePromptTokenDelta(metrics, {}).nextReported, takePromptTokens, + ); const interventions = adoptBareKeys( await readReportedInterventions(config), events, Object.fromEntries(currentInterventions), - ); - const promptTokens = adoptBareKeys( - await readReportedPromptTokens(config), events, computePromptTokenDelta(metrics, {}).nextReported, + takeInterventions((runId) => Object.hasOwn(promptTokens, runId)), ); const interventionDelta = computeInterventionDelta(currentInterventions, interventions); diff --git a/src/team-push.ts b/src/team-push.ts index e7104fd7a..49906ad3f 100644 --- a/src/team-push.ts +++ b/src/team-push.ts @@ -21,6 +21,8 @@ import { aggregateDailySessions, computeDailyStatsDelta, mergeDailyStats, + takeDailySession, + type DailySessionSnapshot, type ReportedDailySessions, } from './session-trends.js'; @@ -170,18 +172,20 @@ function scopeSnapshotPath(name: ReportedSnapshotName, config: LocalConfig | und /** * A scope's snapshot, seeded from the shared one when the scope has none yet. - * `current` gives each run's entry at its own totals, for {@link adoptBareKeys}. + * `split` gives each run's entry at its own totals and its share of a bare + * entry, for {@link adoptBareKeys}. */ async function readSnapshot( name: ReportedSnapshotName, config: LocalConfig | undefined, - current: (events: DashboardEvent[]) => Record, + split: (events: DashboardEvent[]) => Promise<{ current: Record; take: TakeReported }>, ): Promise | null> { const own = scopeSnapshotPath(name, config); if (!config || await pathExists(own)) return readJson>(own); const shared = await readJson>(sharedSnapshotPath(name)); const events = await filterEventsByScope(await readEvents(), config); - const adopted = adoptBareKeys(shared ?? {}, events, current(events), 'shared'); + const { current, take } = await split(events); + const adopted = adoptBareKeys(shared ?? {}, events, current, take, 'shared'); // Copy only resolved run IDs. An unmatched bare entry cannot be allowed to // attach to a future reuse after the source of the snapshot has been lost. const seed: Record = {}; @@ -198,8 +202,13 @@ async function readSnapshot( } export async function readReportedInterventions(config: LocalConfig | undefined): Promise { - const parsed = await readSnapshot('interventions', config, - (events) => Object.fromEntries(interventionCounts(aggregateSessionMetrics(events)))); + const parsed = await readSnapshot('interventions', config, async (events) => { + const promptTokens = await readReportedPromptTokens(config); + return { + current: Object.fromEntries(interventionCounts(aggregateSessionMetrics(events))), + take: takeInterventions((runId) => Object.hasOwn(promptTokens, runId)), + }; + }); return parsed && typeof parsed === 'object' ? parsed : {}; } @@ -216,6 +225,57 @@ function interventionCounts(metrics: Map): Map [sid, { interrupt: m.interrupt, toolReject: m.toolReject, correction: m.correction }])); } +/** A counter's share of what is left: up to the run's own. */ +function upTo(own: number, left: number): number { + return Math.max(0, Math.min(own, left)); +} + +/** + * A run's prompt and token share of a bare entry. This snapshot decides which + * runs of the ID that release reported: the ones its prompts and tokens reach. + */ +export const takePromptTokens: TakeReported = (run, left) => { + if (left.prompts <= 0 && !hasPromptTokenDelta({ prompts: 0, tokens: left.tokens })) return undefined; + const taken = { + prompts: upTo(run.prompts, left.prompts), + tokens: { + input: upTo(run.tokens.input, left.tokens.input), + output: upTo(run.tokens.output, left.tokens.output), + cacheRead: upTo(run.tokens.cacheRead, left.tokens.cacheRead), + cacheCreation: upTo(run.tokens.cacheCreation, left.tokens.cacheCreation), + }, + }; + return { taken, left: { prompts: left.prompts - taken.prompts, tokens: tokenDelta(left.tokens, taken.tokens) } }; +}; + +/** + * A run's intervention share, for the runs `covered`, the prompt-token + * snapshot's: these counts are mostly zero, so running out says nothing. + */ +export function takeInterventions(covered: (runId: string) => boolean): TakeReported { + return (run, left, runId) => { + if (!covered(runId)) return undefined; + const taken = { + interrupt: upTo(run.interrupt, left.interrupt), + toolReject: upTo(run.toolReject, left.toolReject), + correction: upTo(run.correction, left.correction), + }; + return { + taken, + left: { + interrupt: left.interrupt - taken.interrupt, + toolReject: left.toolReject - taken.toolReject, + correction: left.correction - taken.correction, + }, + }; + }; +} + +/** A run's daily share, for the runs `covered`, as for interventions. */ +function takeDaily(covered: (runId: string) => boolean): TakeReported { + return (run, left, runId) => (covered(runId) ? takeDailySession(run, left) : undefined); +} + /** * Compute the intervention delta to report: for each current session, the positive * change since it was last reported. A session not seen before contributes +1 to @@ -270,8 +330,10 @@ function hasInterventionDelta(d: UserInterventionStats): boolean { // export async function readReportedPromptTokens(config: LocalConfig | undefined): Promise { - const parsed = await readSnapshot('prompt-tokens', config, - (events) => computePromptTokenDelta(aggregateSessionMetrics(events), {}).nextReported); + const parsed = await readSnapshot('prompt-tokens', config, async (events) => ({ + current: computePromptTokenDelta(aggregateSessionMetrics(events), {}).nextReported, + take: takePromptTokens, + })); return parsed && typeof parsed === 'object' ? parsed : {}; } @@ -335,8 +397,13 @@ function hasPromptTokenDelta(d: PromptTokenDelta): boolean { } async function readReportedDailySessions(config: LocalConfig | undefined): Promise { - return (await readSnapshot('daily-sessions', config, - (events) => computeDailyStatsDelta(aggregateDailySessions(events), {}).nextReported)) ?? {}; + return (await readSnapshot('daily-sessions', config, async (events) => { + const promptTokens = await readReportedPromptTokens(config); + return { + current: computeDailyStatsDelta(aggregateDailySessions(events), {}).nextReported, + take: takeDaily((runId) => Object.hasOwn(promptTokens, runId)), + }; + })) ?? {}; } async function writeReportedDailySessions(data: ReportedDailySessions, config: LocalConfig | undefined): Promise { @@ -408,13 +475,14 @@ export async function filterEventsByScope( } return key; })); - // Each run is returned under its session ID plus its first event's - // timestamp, which stays the same whichever earlier runs compaction has - // dropped. A tool's own session ID is one run, whatever ends it records: - // `claude --resume` continues it, in a new process, and its Stop carries the - // whole transcript. A PID-fallback ID (`pid-…`) names one run until it ends, - // then comes back for a later one, maybe in another scope, so each run is - // decided on its own. A second end with nothing recorded since the first + // A tool's own session ID is one run, whatever ends it records: `claude + // --resume` continues it, in a new process, and its Stop carries the whole + // transcript. It is returned under the ID itself, so it stays the session + // already reported after compaction drops its events. A PID-fallback ID + // (`pid-…`) names one run until it ends, then comes back for a later one, + // maybe in another scope, so each run is decided on its own and returned + // under the ID plus its first event's timestamp, which stays the same + // whichever earlier runs compaction has dropped. A second end with nothing recorded since the first // (the dashboard monitor's process_exit after SessionEnd) belongs to the run // just closed. A start from another process than its open run's begins a new // run, though nothing ended that one (a crash with no dashboard running). @@ -443,7 +511,7 @@ export async function filterEventsByScope( } if (run === undefined) { run = deciding.push(undefined) - 1; - runIds.push(`${e.sessionId}@${e.timestamp}`); + runIds.push(fallback ? `${e.sessionId}@${e.timestamp}` : e.sessionId); } if (ends && fallback && (!observed || openRun.get(e.sessionId) === run)) { openRun.delete(e.sessionId); @@ -470,37 +538,45 @@ export async function filterEventsByScope( }); } +/** + * A run's share of what is left of a bare snapshot entry, or undefined when + * nothing is left: that run and every later one of its ID were not reported. + */ +export type TakeReported = (run: T, left: T, runId: string) => { taken: T; left: T } | undefined; + /** * A reported snapshot as the run IDs of {@link filterEventsByScope} read it. - * Snapshots written before runs had their own IDs are keyed by the bare - * session ID, summed over every run of that ID the release saw. Of those runs - * in `events` (the scope's, as that filter returns them), each but the last is - * taken as reported at its own totals, from `current`, and the last takes the - * bare entry, so none is sent again; a run's own entry is never replaced. The - * bare entry is retired, so no later run of that ID reads it. Only an earlier + * Snapshots written before fallback runs had their own IDs are keyed by the + * bare `pid-…` ID, holding the sum of the runs of that ID in the log at that + * release's last report; compaction keeps or drops the runs of an ID together. + * So the runs of that ID in `events` (the scope's, as that filter returns them) + * consume the entry in order, each taking its share, from its totals in + * `current`, of what is left; once nothing is left, the later runs take none + * and are reported as new. The first run always takes one: the entry means + * that release reported it. A run's own entry is never replaced, and the bare + * entry is retired, so no later run of that ID reads it. Only an earlier * release wrote bare entries, and only for runs it recorded, so a run whose * first event carries a `dataHomeKey`, and every later run of its ID, takes * none: the entry may be another scope's run under a reused PID-fallback ID. * Shared snapshots also exclude path-keyed runs: that release already had - * per-scope snapshots. - * Returns `reported` itself when there is nothing to retire; otherwise the - * caller persists the result. + * per-scope snapshots. A tool's own session ID keys its one run as it is, so + * its entry needs no adoption. Returns `reported` itself when there is nothing + * to retire; otherwise the caller persists the result. */ export function adoptBareKeys( reported: Record, events: Iterable, current: Record, + take: TakeReported, source: 'scope' | 'shared' = 'scope', ): Record { const legacyRuns = new Map(); const recorded = new Set(); const seen = new Set(); for (const { sessionId: runId, dataHomeKey, dataHome } of events) { - if (seen.has(runId)) continue; + if (seen.has(runId) || !runId.startsWith('pid-')) continue; seen.add(runId); - const at = runId.lastIndexOf('@'); - if (at < 0) continue; - const id = runId.slice(0, at); + const id = runId.slice(0, runId.lastIndexOf('@')); if (recorded.has(id) || !Object.hasOwn(reported, id)) continue; if (typeof dataHomeKey === 'string' || (source === 'shared' && typeof dataHome === 'string')) { recorded.add(id); @@ -511,11 +587,19 @@ export function adoptBareKeys( if (legacyRuns.size === 0) return reported; const adopted = { ...reported }; for (const [id, runs] of legacyRuns) { - runs.forEach((runId, i) => { - if (Object.hasOwn(adopted, runId)) return; - if (i === runs.length - 1) adopted[runId] = reported[id]; - else if (Object.hasOwn(current, runId)) adopted[runId] = current[runId]; - }); + let left = reported[id]; + let first = true; + for (const runId of runs) { + if (!Object.hasOwn(current, runId)) continue; + const share = take(current[runId], left, runId); + if (!share) { + if (first && !Object.hasOwn(adopted, runId)) adopted[runId] = left; + break; + } + if (!Object.hasOwn(adopted, runId)) adopted[runId] = share.taken; + left = share.left; + first = false; + } delete adopted[id]; } return adopted; @@ -568,29 +652,34 @@ export async function reportUsageToTeam( read: (config: LocalConfig | undefined) => Promise>, write: (data: Record, config: LocalConfig | undefined) => Promise, current: Record, + take: TakeReported, ): Promise> => { const stored = await read(reportsConfig); - const adopted = adoptBareKeys(stored, dashboardEvents, current); + const adopted = adoptBareKeys(stored, dashboardEvents, current, take); if (adopted !== stored) await write(adopted, reportsConfig); return adopted; }; - const reportedInterventions = await adopt( - readReportedInterventions, writeReportedInterventions, Object.fromEntries(currentInterventions), - ); - const { delta: interventionDelta, nextReported } = computeInterventionDelta( - currentInterventions, - reportedInterventions, - ); - + // Prompt tokens first: they decide which runs of a bare ID were reported. const reportedPromptTokens = await adopt( readReportedPromptTokens, writeReportedPromptTokens, computePromptTokenDelta(metrics, {}).nextReported, + takePromptTokens, ); + const covered = (runId: string) => Object.hasOwn(reportedPromptTokens, runId); const { delta: promptTokenDelta, nextReported: nextReportedPromptTokens } = computePromptTokenDelta( metrics, reportedPromptTokens, ); + const reportedInterventions = await adopt( + readReportedInterventions, writeReportedInterventions, Object.fromEntries(currentInterventions), + takeInterventions(covered), + ); + const { delta: interventionDelta, nextReported } = computeInterventionDelta( + currentInterventions, + reportedInterventions, + ); const reportedDailySessions = await adopt( readReportedDailySessions, writeReportedDailySessions, computeDailyStatsDelta(currentDaily, {}).nextReported, + takeDaily(covered), ); const { delta: dailyDelta, nextReported: nextReportedDailySessions } = computeDailyStatsDelta( currentDaily, From ed84421f315d31a59a2b12d1d3a43957118628c2 Mon Sep 17 00:00:00 2001 From: Saul Moro Date: Thu, 24 Sep 2026 18:54:50 +0200 Subject: [PATCH 17/41] fix(stats): address CI review (#785) Seeding a scope from the shared snapshot splits the whole log into runs, lets every scope's runs of a bare ID consume its entry in log order, and keeps the shares of the scope's own runs. The shared file summed every scope's runs, so one scope consuming it alone could spend another scope's baseline and suppress its own pending run. The scope that first reports a tool's own session ID records itself in ~/.teamai/dashboard/session-owners.jsonl (the ID and its data home key, no path), and a recorded session stays that scope's wherever it is resumed, after compaction dropped its events too. A dashboard started before processExitAfter existed reads the log and appends its exit in one pass, so an unannotated exit less than one PID check after the open fallback run began belongs to the run closed before it instead of closing the next invocation. --- docs/designs/data-directory-layout.md | 20 +- docs/usage-guide.md | 3 +- docs/usage-guide.zh-CN.md | 2 +- src/__tests__/dashboard-report-scope.test.ts | 57 +++++- src/__tests__/scope-filter.test.ts | 38 ++++ src/team-push.ts | 190 ++++++++++++++----- 6 files changed, 258 insertions(+), 52 deletions(-) diff --git a/docs/designs/data-directory-layout.md b/docs/designs/data-directory-layout.md index be57071c0..6a609ac7a 100644 --- a/docs/designs/data-directory-layout.md +++ b/docs/designs/data-directory-layout.md @@ -364,9 +364,19 @@ ID is not split this way, since Claude fires SessionStart again on resume, in a new process, and its Stop carries the whole transcript. The monitor's `process_exit` also records `processExitAfter`, the last event it observed, and closes only that run: an exit appended after the next run of the same ID began does not end it, -and one whose run compaction dropped is ignored. A tool's own session ID is +and one whose run compaction dropped is ignored. A dashboard started before +that field existed writes none, and it reads the log and appends its exit in +one pass, so an unannotated exit less than one PID check (15 s) after the open +fallback run began belongs to the run closed before it; a run that really +exits that soon is then left open, not counted twice. A tool's own session ID is reported and snapshotted under the ID itself, as before, so a session resumed -after compaction dropped its events still reads what its scope reported; a +after compaction dropped its events still reads what its scope reported. The +scope that first reports it also appends the ID and its own data home key to +`~/.teamai/dashboard/session-owners.jsonl` (never a path, #666), and a session +recorded there is that scope's wherever it is resumed later, whatever the log +still holds, so another scope never reports its transcript again; the first +line for an ID wins, and the file grows by one line per such session, like the +snapshots. A fallback run is reported and snapshotted as `@`, which does not change when compaction drops earlier runs. A snapshot entry keyed by a bare fallback ID (written before) is the sum of the runs of that ID in the @@ -389,8 +399,10 @@ ID is the parent PID): `/dashboard/reported-*.json`, and `~/.teamai/dashboard/user-reported-*.json` for the user scope. The first time a scope needs one it seeds it from the shared `~/.teamai/dashboard/reported-*.json`, so nothing reported before the upgrade is -sent again; after that it reads only its own. The seed takes only the entries of -the scope's runs in the log, under their run IDs, and none for a run recorded +sent again; after that it reads only its own. That file summed every scope's +runs of an ID, so the runs of the whole log consume it in log order, whichever +scope each belongs to, and the seed keeps the shares of the scope's own runs, +under their run IDs; none goes to a run recorded with a `dataHome` path: that release already kept per-scope snapshots, so a shared entry under its ID is another scope's. An unmatched entry is dropped, so a later reuse of the ID cannot inherit it. The shared file is no longer diff --git a/docs/usage-guide.md b/docs/usage-guide.md index 056a89f2a..2cc2912f8 100644 --- a/docs/usage-guide.md +++ b/docs/usage-guide.md @@ -1766,7 +1766,8 @@ no directory, or one removed since, are reported by no one. Each scope also keeps its own snapshot of what it already reported, and a session under a reused fallback ID (Copilot's PID-based ID when it sends none) counts as new, whichever scope reported the earlier one, while a resumed session -(`claude --resume`) keeps its ID and stays one session; the first +(`claude --resume`) keeps its ID and stays one session, reported by the scope +that first reported it wherever it is resumed; the first report after upgrading starts from the snapshot every scope used to share, so nothing is reported twice. A target removes its usage events only after it confirms success; failed pushes diff --git a/docs/usage-guide.zh-CN.md b/docs/usage-guide.zh-CN.md index 1ab490444..ef8721527 100644 --- a/docs/usage-guide.zh-CN.md +++ b/docs/usage-guide.zh-CN.md @@ -1696,7 +1696,7 @@ scope，即使之后切换到另一个项目：它的 Stop 带有整份 transcri 会重复计算。旧版本记录的事件没有该键：由其目录当前解析到的 scope 上报（项目下的嵌套 clone 解析到它自己的项目或 user scope，而不是外层项目）；没有目录或目录已删除的事件不由任何 scope 上报。 每个 scope 还各自保存已上报快照，且复用回退 ID 的新会话（Copilot 未提供会话 ID 时基于 PID 的 ID） -总算作新会话，无论先前那个由哪个 scope 上报；恢复的会话（`claude --resume`）保留原 ID，仍是同一个会话；升级后的首次上报从原先各 scope 共用的快照开始，不会重复上报。目标确认成功后才清理自己的使用事件， +总算作新会话，无论先前那个由哪个 scope 上报；恢复的会话（`claude --resume`）保留原 ID，仍是同一个会话，无论在哪里恢复，都由最先上报它的 scope 上报；升级后的首次上报从原先各 scope 共用的快照开始，不会重复上报。目标确认成功后才清理自己的使用事件， 推送失败会保留事件（最多保留最新 5,000 条，见下文）。上报完成前继续持有相关同步锁， 避免另一次 Pull 与尚未完成的上报竞争。 diff --git a/src/__tests__/dashboard-report-scope.test.ts b/src/__tests__/dashboard-report-scope.test.ts index 6aa9551c9..fff19f420 100644 --- a/src/__tests__/dashboard-report-scope.test.ts +++ b/src/__tests__/dashboard-report-scope.test.ts @@ -263,8 +263,8 @@ describe('each scope keeps its own reported snapshot (#786)', () => { expect(await reportedPrompts(user)).toBe(5); }); - it('a session that moves from one project into another is reported once, by the first', async () => { - const { root, project } = await setup(); + /** A second git project, Q, with its own team. */ + async function setupQ(): Promise<{ rootQ: string; projectQ: LocalConfig }> { const rootQ = path.join(tmp, 'project-q'); fs.mkdirSync(rootQ); execFileSync('git', ['init', '-q'], { cwd: rootQ }); @@ -276,6 +276,12 @@ describe('each scope keeps its own reported snapshot (#786)', () => { }); const projectQ = await resolveConfigForDir(rootQ); if (!projectQ) throw new Error('fixture config Q did not resolve'); + return { rootQ, projectQ }; + } + + it('a session that moves from one project into another is reported once, by the first', async () => { + const { root, project } = await setup(); + const { rootQ, projectQ } = await setupQ(); await hook('session-start', 'claude', { session_id: 'moved', cwd: root, hook_event_name: 'SessionStart' }); await prompts('moved', root, 3); await prompts('moved', rootQ, 2); @@ -463,6 +469,32 @@ describe('each scope keeps its own reported snapshot (#786)', () => { expect(await reportedPrompts(user)).toBe(2); }); + it('a session resumed in another project after compaction dropped its events stays its first project\'s', async () => { + const { root, project } = await setup(); + const { rootQ, projectQ } = await setupQ(); + await session('claude', { session_id: 'resumed', cwd: root }); + await hook('session-end', 'claude', { session_id: 'resumed', cwd: root, hook_event_name: 'SessionEnd' }); + expect(await reportedSessions(project)).toBe(1); + fs.writeFileSync(path.join(teamaiHome(), 'dashboard', 'events.jsonl'), ''); + // `claude --resume` in Q: its Stop carries P's transcript, which Q never reported. + await session('claude', { session_id: 'resumed', cwd: rootQ }); + + expect(await report(projectQ)).toBeNull(); + expect(await reportedSessions(project)).toBe(1); + expect(await reportedPrompts(project)).toBe(1); + }); + + it('the session owners a report keeps hold no path (#666)', async () => { + const { root, project } = await setup(); + await session('copilot', { session_id: 'copilot-p', cwd: root }); + await report(project); + + const owners = fs.readFileSync(path.join(teamaiHome(), 'dashboard', 'session-owners.jsonl'), 'utf-8'); + expect(owners).toContain('copilot-p'); + expect(owners).not.toContain(root); + expect(owners).not.toContain(getDataHome(project)); + }); + it('a session resumed after compaction dropped its events is still the one already reported', async () => { const { root, project } = await setup(); await session('claude', { session_id: 'resumed', cwd: root }); @@ -516,6 +548,27 @@ describe('each scope keeps its own reported snapshot (#786)', () => { expect(await reportedPrompts(user)).toBe(1); }); + it('a shared bare entry is consumed in the order of every scope\'s runs, not one scope\'s', async () => { + const { root, user, project } = await setup(); + const elsewhere = path.join(tmp, 'elsewhere'); + fs.mkdirSync(elsewhere); + // A release before #666 recorded Copilot's cwd. It reported the user-scope + // run of `pid-7` as 1 prompt, then recorded P's run of that ID unreported. + const timestamp = (s: number) => new Date(Date.now() - 600_000 + s * 1000).toISOString(); + const event = (type: string, s: number, cwd: string) => ({ type, timestamp: timestamp(s), sessionId: 'pid-7', tool: 'copilot', cwd }); + const log = [ + event('session_start', 0, elsewhere), event('prompt_submit', 1, elsewhere), event('stop', 2, elsewhere), event('session_end', 3, elsewhere), + event('session_start', 10, root), event('prompt_submit', 11, root), event('stop', 12, root), event('session_end', 13, root), + ]; + fs.mkdirSync(path.join(teamaiHome(), 'dashboard'), { recursive: true }); + fs.writeFileSync(path.join(teamaiHome(), 'dashboard', 'events.jsonl'), log.map((e) => JSON.stringify(e)).join('\n') + '\n'); + writeSharedSnapshots({ 'pid-7': 1 }, new Date().toISOString().slice(0, 10)); + + expect(await reportedSessions(project)).toBe(1); + expect(await reportedPrompts(project)).toBe(1); + expect(await report(user)).toBeNull(); + }); + it('the first report after the upgrade sends nothing a shared snapshot already reported', async () => { const { root, user, project } = await setup(); const elsewhere = path.join(tmp, 'elsewhere'); diff --git a/src/__tests__/scope-filter.test.ts b/src/__tests__/scope-filter.test.ts index 61f83e274..309be6ab7 100644 --- a/src/__tests__/scope-filter.test.ts +++ b/src/__tests__/scope-filter.test.ts @@ -118,6 +118,44 @@ describe('filterEventsByScope', () => { expect(before[2].sessionId).toBe(after[0].sessionId); }); + it('an exit a dashboard before processExitAfter wrote just after the next run began ends the earlier run', async () => { + // That monitor read the log before SessionEnd and the next SessionStart, + // then appended its exit, unannotated, after them. + const p = '/home/jeff/.teamai/projects/p'; + const at = async (type: DashboardEvent['type'], timestamp: string) => + ({ ...(await scopedEvent(undefined, 'pid-1', p)), type, timestamp }); + const log = [ + await at('session_start', '2026-01-01T00:00:00.000Z'), + await at('session_end', '2026-01-01T00:01:00.000Z'), + await at('session_start', '2026-01-01T00:01:01.000Z'), + await at('process_exit', '2026-01-01T00:01:02.000Z'), + await at('prompt_submit', '2026-01-01T00:01:03.000Z'), + ]; + const project = await filterEventsByScope(log, projectScope('/Users/jeff/project-a')); + expect(project.map((e) => e.sessionId)).toEqual([ + 'pid-1@2026-01-01T00:00:00.000Z', 'pid-1@2026-01-01T00:00:00.000Z', + 'pid-1@2026-01-01T00:01:01.000Z', 'pid-1@2026-01-01T00:00:00.000Z', 'pid-1@2026-01-01T00:01:01.000Z', + ]); + }); + + it('an unannotated exit well after the open run began still ends that run', async () => { + const p = '/home/jeff/.teamai/projects/p'; + const at = async (type: DashboardEvent['type'], timestamp: string) => + ({ ...(await scopedEvent(undefined, 'pid-1', p)), type, timestamp }); + const log = [ + await at('session_start', '2026-01-01T00:00:00.000Z'), + await at('session_end', '2026-01-01T00:01:00.000Z'), + await at('session_start', '2026-01-01T00:02:00.000Z'), + await at('process_exit', '2026-01-01T00:03:00.000Z'), + await at('prompt_submit', '2026-01-01T00:04:00.000Z'), + ]; + const project = await filterEventsByScope(log, projectScope('/Users/jeff/project-a')); + expect(project.map((e) => e.sessionId)).toEqual([ + 'pid-1@2026-01-01T00:00:00.000Z', 'pid-1@2026-01-01T00:00:00.000Z', + 'pid-1@2026-01-01T00:02:00.000Z', 'pid-1@2026-01-01T00:02:00.000Z', 'pid-1@2026-01-01T00:04:00.000Z', + ]); + }); + it('a delayed observed exit targets the earlier run across scopes and is ignored after that run is compacted', async () => { const old = { ...(await scopedEvent(undefined, 'pid-1', '/home/jeff/.teamai')), timestamp: '2026-01-01T00:00:00Z' }; const ended = { ...old, type: 'session_end' as const, timestamp: '2026-01-01T00:01:00Z' }; diff --git a/src/team-push.ts b/src/team-push.ts index 49906ad3f..d1d9aa45f 100644 --- a/src/team-push.ts +++ b/src/team-push.ts @@ -15,7 +15,10 @@ import { import { writeFile, readFileSafe, ensureDir, pathExists, readJson, writeJson } from './utils/fs.js'; import { log } from './utils/logger.js'; import type { UserStats, UserInterventionStats, SessionMetrics, TokenUsage, DashboardEvent, LocalConfig } from './types.js'; -import { getVotesDir, getDataHome, getTeamaiHomeDir, emptyTokenUsage, addTokenUsage, usesBranchWorktree } from './types.js'; +import { + getVotesDir, getDataHome, getTeamaiHomeDir, emptyTokenUsage, addTokenUsage, usesBranchWorktree, + DASHBOARD_PID_CHECK_INTERVAL_MS, +} from './types.js'; import { getUserHome } from './utils/home.js'; import { aggregateDailySessions, @@ -172,8 +175,10 @@ function scopeSnapshotPath(name: ReportedSnapshotName, config: LocalConfig | und /** * A scope's snapshot, seeded from the shared one when the scope has none yet. - * `split` gives each run's entry at its own totals and its share of a bare - * entry, for {@link adoptBareKeys}. + * The shared file summed every scope's runs of an ID, so they consume it in + * the order of the whole log, and the scope keeps its own runs' shares. + * `split` gives each run of the log its entry at its own totals and its share + * of a bare entry, for {@link adoptBareKeys}. */ async function readSnapshot( name: ReportedSnapshotName, @@ -183,9 +188,11 @@ async function readSnapshot( const own = scopeSnapshotPath(name, config); if (!config || await pathExists(own)) return readJson>(own); const shared = await readJson>(sharedSnapshotPath(name)); - const events = await filterEventsByScope(await readEvents(), config); - const { current, take } = await split(events); - const adopted = adoptBareKeys(shared ?? {}, events, current, take, 'shared'); + const logged = await readEvents(); + const all = await runsOfLog(logged); + const events = await filterEventsByScope(logged, config); + const { current, take } = await split(all); + const adopted = adoptBareKeys(shared ?? {}, all, current, take, 'shared'); // Copy only resolved run IDs. An unmatched bare entry cannot be allowed to // attach to a future reuse after the source of the snapshot has been lost. const seed: Record = {}; @@ -202,13 +209,10 @@ async function readSnapshot( } export async function readReportedInterventions(config: LocalConfig | undefined): Promise { - const parsed = await readSnapshot('interventions', config, async (events) => { - const promptTokens = await readReportedPromptTokens(config); - return { - current: Object.fromEntries(interventionCounts(aggregateSessionMetrics(events))), - take: takeInterventions((runId) => Object.hasOwn(promptTokens, runId)), - }; - }); + const parsed = await readSnapshot('interventions', config, async (events) => ({ + current: Object.fromEntries(interventionCounts(aggregateSessionMetrics(events))), + take: takeInterventions(await sharedCoverage(events)), + })); return parsed && typeof parsed === 'object' ? parsed : {}; } @@ -271,6 +275,14 @@ export function takeInterventions(covered: (runId: string) => boolean): TakeRepo }; } +/** The runs of the whole log (`all`) the shared prompt-token snapshot covers, for seeding. */ +async function sharedCoverage(all: DashboardEvent[]): Promise<(runId: string) => boolean> { + const shared = await readJson(sharedSnapshotPath('prompt-tokens')); + const current = computePromptTokenDelta(aggregateSessionMetrics(all), {}).nextReported; + const covered = adoptBareKeys(shared ?? {}, all, current, takePromptTokens, 'shared'); + return (runId) => Object.hasOwn(covered, runId); +} + /** A run's daily share, for the runs `covered`, as for interventions. */ function takeDaily(covered: (runId: string) => boolean): TakeReported { return (run, left, runId) => (covered(runId) ? takeDailySession(run, left) : undefined); @@ -397,13 +409,10 @@ function hasPromptTokenDelta(d: PromptTokenDelta): boolean { } async function readReportedDailySessions(config: LocalConfig | undefined): Promise { - return (await readSnapshot('daily-sessions', config, async (events) => { - const promptTokens = await readReportedPromptTokens(config); - return { - current: computeDailyStatsDelta(aggregateDailySessions(events), {}).nextReported, - take: takeDaily((runId) => Object.hasOwn(promptTokens, runId)), - }; - })) ?? {}; + return (await readSnapshot('daily-sessions', config, async (events) => ({ + current: computeDailyStatsDelta(aggregateDailySessions(events), {}).nextReported, + take: takeDaily(await sharedCoverage(events)), + }))) ?? {}; } async function writeReportedDailySessions(data: ReportedDailySessions, config: LocalConfig | undefined): Promise { @@ -435,7 +444,9 @@ function hasDailyDelta(delta: ReturnType['delta'] * written before events carried a key is decided by its first cwd, by the * dispatcher's rule: the scope that directory resolves to now, so a nested * clone under a project is not the project's; one with no cwd, or a cwd gone - * since, is no scope's. A caller without a scope config reads the whole log. + * since, is no scope's. A tool's own session ID a scope has recorded as its + * own (see `session-owners.jsonl`) is that scope's, whatever the log still + * holds. A caller without a scope config reads the whole log. */ export async function filterEventsByScope( events: DashboardEvent[], @@ -462,10 +473,33 @@ export async function filterEventsByScope( } return owns; }; - // The log is hand-editable: a key that is not a string counts as absent. - // An unreleased build of #795 recorded the data home path in place of its key. + const eventKeys = await keysOf(events); + const { runOf, runIds, deciding } = splitRuns(events, eventKeys); + const owners = await readSessionOwners(); + const owned = await Promise.all(deciding.map(async (i, run) => { + // Recorded by the scope that first reported it, so a session resumed + // elsewhere after compaction dropped its events stays that scope's. + const owner = owners.get(runIds[run]); + if (owner !== undefined) return keys.has(owner); + if (i === undefined) return false; + const key = eventKeys[i]; + const cwd = events[i].cwd; + return key !== undefined ? keys.has(key) : !!cwd && await ownsCwd(cwd); + })); + return events.flatMap((e, i) => { + const run = runOf[i]; + return run !== undefined && owned[run] ? [{ ...e, sessionId: runIds[run] }] : []; + }); +} + +/** + * Each event's data home key. The log is hand-editable: a key that is not a + * string counts as absent. An unreleased build of #795 recorded the data home + * path in place of its key. + */ +async function keysOf(events: DashboardEvent[]): Promise> { const hashes = new Map>(); - const eventKeys = await Promise.all(events.map((e) => { + return Promise.all(events.map((e) => { if (typeof e.dataHomeKey === 'string') return e.dataHomeKey; if (typeof e.dataHome !== 'string') return undefined; let key = hashes.get(e.dataHome); @@ -475,19 +509,36 @@ export async function filterEventsByScope( } return key; })); - // A tool's own session ID is one run, whatever ends it records: `claude - // --resume` continues it, in a new process, and its Stop carries the whole - // transcript. It is returned under the ID itself, so it stays the session - // already reported after compaction drops its events. A PID-fallback ID - // (`pid-…`) names one run until it ends, then comes back for a later one, - // maybe in another scope, so each run is decided on its own and returned - // under the ID plus its first event's timestamp, which stays the same - // whichever earlier runs compaction has dropped. A second end with nothing recorded since the first - // (the dashboard monitor's process_exit after SessionEnd) belongs to the run - // just closed. A start from another process than its open run's begins a new - // run, though nothing ended that one (a crash with no dashboard running). +} + +/** + * The runs of a log: each event's run (none for an exit whose run is gone), + * each run's ID, and the event that decides its scope. + * + * A tool's own session ID is one run, whatever ends it records: `claude + * --resume` continues it, in a new process, and its Stop carries the whole + * transcript. It is returned under the ID itself, so it stays the session + * already reported after compaction drops its events. A PID-fallback ID + * (`pid-…`) names one run until it ends, then comes back for a later one, + * maybe in another scope, so each run is decided on its own and returned under + * the ID plus its first event's timestamp, which stays the same whichever + * earlier runs compaction has dropped. A second end with nothing recorded since + * the first (the dashboard monitor's process_exit after SessionEnd) belongs to + * the run just closed. A start from another process than its open run's begins + * a new run, though nothing ended that one (a crash with no dashboard running). + * The monitor's exit names the last event it observed (`processExitAfter`); + * a dashboard started before that field existed wrote none, so its exit less + * than one PID check after the open run began may have been observed before + * that run, and belongs to the run closed before it. + */ +function splitRuns(events: DashboardEvent[], eventKeys: Array): { + runOf: Array; + runIds: string[]; + deciding: Array; +} { const runOf: Array = []; const runPid: Array = []; + const runStart: number[] = []; const observedRuns = new Map(); const openRun = new Map(); const closedRun = new Map(); @@ -501,8 +552,11 @@ export async function filterEventsByScope( const open = openRun.get(e.sessionId); if (starts && open !== undefined && fallback && typeof runPid[open] === 'number' && runPid[open] !== e.monitorPid) openRun.delete(e.sessionId); + const closed = closedRun.get(e.sessionId); + const early = e.type === 'process_exit' && !observed && fallback && open !== undefined && closed !== undefined + && Date.parse(e.timestamp) - runStart[open] < DASHBOARD_PID_CHECK_INTERVAL_MS; let run = observed ? observedRuns.get(`${e.sessionId}@${e.processExitAfter}`) - : openRun.get(e.sessionId) ?? (ends ? closedRun.get(e.sessionId) : undefined); + : early ? closed : openRun.get(e.sessionId) ?? (ends ? closed : undefined); // The observed run may have been compacted. Its delayed exit must not // manufacture a new session or close a later reuse of the same ID. if (observed && run === undefined) { @@ -512,8 +566,9 @@ export async function filterEventsByScope( if (run === undefined) { run = deciding.push(undefined) - 1; runIds.push(fallback ? `${e.sessionId}@${e.timestamp}` : e.sessionId); + runStart.push(Date.parse(e.timestamp)); } - if (ends && fallback && (!observed || openRun.get(e.sessionId) === run)) { + if (ends && fallback && (!(observed || early) || openRun.get(e.sessionId) === run)) { openRun.delete(e.sessionId); closedRun.set(e.sessionId, run); } else if (!ends || !fallback) { @@ -526,18 +581,64 @@ export async function filterEventsByScope( if (eventKeys[i] !== undefined ? current === undefined || eventKeys[current] === undefined : current === undefined && !!e.cwd) deciding[run] = i; }); - const owned = await Promise.all(deciding.map(async (i) => { - if (i === undefined) return false; - const key = eventKeys[i]; - const cwd = events[i].cwd; - return key !== undefined ? keys.has(key) : !!cwd && await ownsCwd(cwd); - })); + return { runOf, runIds, deciding }; +} + +/** Every event of the log under its run ID, whichever scope it belongs to. */ +async function runsOfLog(events: DashboardEvent[]): Promise { + const { runOf, runIds } = splitRuns(events, await keysOf(events)); return events.flatMap((e, i) => { const run = runOf[i]; - return run !== undefined && owned[run] ? [{ ...e, sessionId: runIds[run] }] : []; + return run !== undefined ? [{ ...e, sessionId: runIds[run] }] : []; }); } +// ─── Session owners ───────────────────────────────────── +// +// ~/.teamai/dashboard/session-owners.jsonl {"sessionId":"","dataHomeKey":""} per line +// +// A tool's own session ID can be resumed anywhere, long after compaction +// dropped its events, and its Stop carries the whole transcript. So the scope +// that first reports it records itself here, append-only, and the session +// stays that scope's. Only the ID and the key, never a path (#666). The first +// line for an ID wins. +// + +function sessionOwnersPath(): string { + return path.join(getTeamaiHomeDir(), 'dashboard', 'session-owners.jsonl'); +} + +async function readSessionOwners(): Promise> { + const owners = new Map(); + const content = await readFileSafe(sessionOwnersPath()); + for (const line of (content ?? '').split('\n')) { + if (!line.trim()) continue; + try { + const parsed: unknown = JSON.parse(line); + if (parsed && typeof parsed === 'object' && 'sessionId' in parsed && 'dataHomeKey' in parsed + && typeof parsed.sessionId === 'string' && typeof parsed.dataHomeKey === 'string' + && !owners.has(parsed.sessionId)) owners.set(parsed.sessionId, parsed.dataHomeKey); + } catch { + // A torn or hand-edited line records no owner. + } + } + return owners; +} + +/** Records `key` as the owner of the tool-own session IDs among `events` that have none yet. */ +async function recordSessionOwners(events: DashboardEvent[], key: string): Promise { + const owners = await readSessionOwners(); + const ids = new Set(events.map((e) => e.sessionId).filter((id) => !id.startsWith('pid-') && !owners.has(id))); + if (ids.size === 0) return; + try { + await ensureDir(path.dirname(sessionOwnersPath())); + await fs.promises.appendFile(sessionOwnersPath(), + [...ids].map((sessionId) => JSON.stringify({ sessionId, dataHomeKey: key })).join('\n') + '\n'); + } catch (e) { + log.debug(`Could not record session owners: ${(e as Error).message}`); + } +} + /** * A run's share of what is left of a bare snapshot entry, or undefined when * nothing is left: that run and every later one of its ID were not reported. @@ -642,6 +743,7 @@ export async function reportUsageToTeam( // both the intervention delta and the prompt-count/token delta from it. // Only the sessions recorded in this scope (#785). const dashboardEvents = await filterEventsByScope(await readEvents(), reportsConfig); + if (reportsConfig) await recordSessionOwners(dashboardEvents, await dataHomeKey(getDataHome(reportsConfig))); const metrics = aggregateSessionMetrics(dashboardEvents); const currentInterventions = interventionCounts(metrics); From 6be15ddea3106776195073c3ab992423ad8d91ca Mon Sep 17 00:00:00 2001 From: Saul Moro Date: Thu, 24 Sep 2026 19:12:26 +0200 Subject: [PATCH 18/41] fix(stats): address CI review (#785) A run taking its share of an earlier release's summed daily snapshot keeps its own success and correction flags: the sum's are no single run's (a successful run and an interrupted one sum to unsuccessful), so an adopted run changed sessionsSucceeded without sessionsEnded. An unannotated process_exit from a dashboard started before processExitAfter no longer ends the open fallback run when more events of that ID follow before the next start: a dead process records nothing more, so it was observed before that run and belongs to the run closed before it. This replaces the 15 s window, which a delayed callback or a skewed clock could miss. --- docs/designs/data-directory-layout.md | 12 ++++++---- src/__tests__/scope-filter.test.ts | 22 ++++++++++++++++- src/__tests__/session-trends.test.ts | 20 ++++++++++++++++ src/session-trends.ts | 7 +++--- src/team-push.ts | 34 ++++++++++++++++----------- 5 files changed, 72 insertions(+), 23 deletions(-) diff --git a/docs/designs/data-directory-layout.md b/docs/designs/data-directory-layout.md index 6a609ac7a..f4bac9a6a 100644 --- a/docs/designs/data-directory-layout.md +++ b/docs/designs/data-directory-layout.md @@ -365,10 +365,10 @@ new process, and its Stop carries the whole transcript. The monitor's `process_e records `processExitAfter`, the last event it observed, and closes only that run: an exit appended after the next run of the same ID began does not end it, and one whose run compaction dropped is ignored. A dashboard started before -that field existed writes none, and it reads the log and appends its exit in -one pass, so an unannotated exit less than one PID check (15 s) after the open -fallback run began belongs to the run closed before it; a run that really -exits that soon is then left open, not counted twice. A tool's own session ID is +that field existed writes none. A dead process records nothing more, so such +an exit followed by more events of its fallback ID before the next start did +not end the open run: it belongs to the run closed before it, however late it +was appended. A tool's own session ID is reported and snapshotted under the ID itself, as before, so a session resumed after compaction dropped its events still reads what its scope reported. The scope that first reports it also appends the ID and its own data home key to @@ -385,7 +385,9 @@ runs of an ID together. So the next time the scope reports, those runs consume the entry in log order, each taking up to its own totals of what is left; once the prompt-token entry is used up, the later runs were not reported and count as new sessions (the interventions and daily snapshots follow the prompt-token -one, since their counts say nothing when they run out). The first run always +one, since their counts say nothing when they run out). A run keeps its own +success and correction flags, since the sum's are no single run's, so an +adopted run changes no status total. The first run always takes a share, as the entry means that release reported it. The entry is then removed, so no later run of that ID reads it. Only an earlier release wrote bare entries, and a seeded one may be another scope's, so a run whose first event diff --git a/src/__tests__/scope-filter.test.ts b/src/__tests__/scope-filter.test.ts index 309be6ab7..f119f7c84 100644 --- a/src/__tests__/scope-filter.test.ts +++ b/src/__tests__/scope-filter.test.ts @@ -138,7 +138,9 @@ describe('filterEventsByScope', () => { ]); }); - it('an unannotated exit well after the open run began still ends that run', async () => { + it('an unannotated exit followed by more of the open run did not end it, however late it came', async () => { + // A delayed callback or skewed clock: a dead process records nothing more, + // so the prompt after the exit shows the open run went on. const p = '/home/jeff/.teamai/projects/p'; const at = async (type: DashboardEvent['type'], timestamp: string) => ({ ...(await scopedEvent(undefined, 'pid-1', p)), type, timestamp }); @@ -150,6 +152,24 @@ describe('filterEventsByScope', () => { await at('prompt_submit', '2026-01-01T00:04:00.000Z'), ]; const project = await filterEventsByScope(log, projectScope('/Users/jeff/project-a')); + expect(project.map((e) => e.sessionId)).toEqual([ + 'pid-1@2026-01-01T00:00:00.000Z', 'pid-1@2026-01-01T00:00:00.000Z', + 'pid-1@2026-01-01T00:02:00.000Z', 'pid-1@2026-01-01T00:00:00.000Z', 'pid-1@2026-01-01T00:02:00.000Z', + ]); + }); + + it('an unannotated exit with nothing more of the open run ends it', async () => { + const p = '/home/jeff/.teamai/projects/p'; + const at = async (type: DashboardEvent['type'], timestamp: string) => + ({ ...(await scopedEvent(undefined, 'pid-1', p)), type, timestamp }); + const log = [ + await at('session_start', '2026-01-01T00:00:00.000Z'), + await at('session_end', '2026-01-01T00:01:00.000Z'), + await at('session_start', '2026-01-01T00:02:00.000Z'), + await at('process_exit', '2026-01-01T00:03:00.000Z'), + await at('session_start', '2026-01-01T00:04:00.000Z'), + ]; + const project = await filterEventsByScope(log, projectScope('/Users/jeff/project-a')); expect(project.map((e) => e.sessionId)).toEqual([ 'pid-1@2026-01-01T00:00:00.000Z', 'pid-1@2026-01-01T00:00:00.000Z', 'pid-1@2026-01-01T00:02:00.000Z', 'pid-1@2026-01-01T00:02:00.000Z', 'pid-1@2026-01-01T00:04:00.000Z', diff --git a/src/__tests__/session-trends.test.ts b/src/__tests__/session-trends.test.ts index e760b378a..4277b8b46 100644 --- a/src/__tests__/session-trends.test.ts +++ b/src/__tests__/session-trends.test.ts @@ -5,6 +5,8 @@ import { computeDailyStatsDelta, mergeDailyStats, summarizeTrendWindow, + takeDailySession, + type DailySessionSnapshot, } from '../session-trends.js'; import type { DashboardEvent, DailyUserStats } from '../types.js'; @@ -125,6 +127,24 @@ describe('daily session trends', () => { expect(delta['2026-09-02']).toMatchObject({ cacheReadTokens: 0, cacheEligibleInputTokens: 0 }); }); + it('a run taking its share of an earlier release\'s sum reports no status change of its own', () => { + // Two reported runs of one ID, one successful and one interrupted: the sum is + // unsuccessful, but each run keeps its own status, so neither moves the totals. + const day = '2026-09-01'; + const snapshot = (prompts: number, succeeded: 0 | 1, corrected: 0 | 1): DailySessionSnapshot => + ({ date: day, prompts, durationMs: 1000, succeeded, corrected, requestDaily: {} }); + const runs = new Map([['pid-1@t1', snapshot(1, 1, 0)], ['pid-1@t2', snapshot(1, 0, 1)]]); + let left = snapshot(2, 0, 1); + const reported: Record = {}; + for (const [runId, run] of runs) { + const share = takeDailySession(run, left); + reported[runId] = share.taken; + left = share.left; + } + const { delta } = computeDailyStatsDelta(runs, reported); + expect(delta[day]).toMatchObject({ sessionsEnded: 0, sessionsSucceeded: 0, sessionsCorrected: 0, promptTurns: 0 }); + }); + it('compares the latest seven UTC days with the prior seven days', () => { const daily: Record = { '2026-08-27': { sessionsEnded: 10, sessionsSucceeded: 5, promptTurns: 80, durationMs: 600_000, sessionsCorrected: 4, pricedRequests: 10, costMicros: 1_000_000, cacheReadTokens: 20, cacheEligibleInputTokens: 100 }, diff --git a/src/session-trends.ts b/src/session-trends.ts index d3eb8b35d..c2d2f8d6f 100644 --- a/src/session-trends.ts +++ b/src/session-trends.ts @@ -131,8 +131,9 @@ function reportedSessionCache( /** * `run`'s share of `left`, what remains of the snapshot an earlier release kept * for all the runs of one session ID: each counter up to the run's own. The - * success and correction flags describe the one session that release counted, - * so each run it covered reads them as they are. + * sum's success and correction flags are not any one run's (a successful run + * and an interrupted one sum to unsuccessful), so each run takes its own and + * reports no status change: that release already counted the sum's. */ export function takeDailySession( run: DailySessionSnapshot, @@ -168,7 +169,7 @@ export function takeDailySession( const cacheEligible = upTo(run.sessionCacheEligibleTokens ?? 0, leftCache.eligible); return { taken: { - date: run.date, prompts, durationMs, succeeded: left.succeeded, corrected: left.corrected, + date: run.date, prompts, durationMs, succeeded: run.succeeded, corrected: run.corrected, requestDaily: takenDaily, sessionCacheReadTokens: cacheRead, sessionCacheEligibleTokens: cacheEligible, }, left: { diff --git a/src/team-push.ts b/src/team-push.ts index d1d9aa45f..cc7564f07 100644 --- a/src/team-push.ts +++ b/src/team-push.ts @@ -15,10 +15,7 @@ import { import { writeFile, readFileSafe, ensureDir, pathExists, readJson, writeJson } from './utils/fs.js'; import { log } from './utils/logger.js'; import type { UserStats, UserInterventionStats, SessionMetrics, TokenUsage, DashboardEvent, LocalConfig } from './types.js'; -import { - getVotesDir, getDataHome, getTeamaiHomeDir, emptyTokenUsage, addTokenUsage, usesBranchWorktree, - DASHBOARD_PID_CHECK_INTERVAL_MS, -} from './types.js'; +import { getVotesDir, getDataHome, getTeamaiHomeDir, emptyTokenUsage, addTokenUsage, usesBranchWorktree } from './types.js'; import { getUserHome } from './utils/home.js'; import { aggregateDailySessions, @@ -526,10 +523,11 @@ async function keysOf(events: DashboardEvent[]): Promise): { runOf: Array; @@ -538,12 +536,21 @@ function splitRuns(events: DashboardEvent[], eventKeys: Array = []; const runPid: Array = []; - const runStart: number[] = []; const observedRuns = new Map(); const openRun = new Map(); const closedRun = new Map(); const deciding: Array = []; const runIds: string[] = []; + // Whether each event is followed by activity of its ID (anything but a start + // or a monitor exit) before that ID's next start. + const continues: boolean[] = []; + const active = new Map(); + for (let i = events.length - 1; i >= 0; i--) { + const { sessionId, type } = events[i]; + continues[i] = active.get(sessionId) ?? false; + if (type === 'session_start') active.set(sessionId, false); + else if (type !== 'process_exit') active.set(sessionId, true); + } events.forEach((e, i) => { const fallback = e.sessionId.startsWith('pid-'); const ends = e.type === 'session_end' || e.type === 'process_exit'; @@ -553,10 +560,10 @@ function splitRuns(events: DashboardEvent[], eventKeys: Array Date: Thu, 24 Sep 2026 19:32:40 +0200 Subject: [PATCH 19/41] fix(stats): address CI review (#785) session-owners.jsonl is first written from the per-scope snapshots an earlier release left: a tool's own ID in the user scope's or a partition's prompt-token snapshot is that scope's, so a session main reported in P, compacted and resumed in Q, stays P's instead of being reported again to Q. An ID the shared snapshot also holds is left out: main copied the shared file into every scope, so it names no owner, and every scope already has its baseline. The file is created exclusively, so a concurrent report in another scope reads the one written first. --- docs/designs/data-directory-layout.md | 9 ++++- src/__tests__/dashboard-report-scope.test.ts | 12 ++++++ src/team-push.ts | 42 ++++++++++++++++++++ 3 files changed, 62 insertions(+), 1 deletion(-) diff --git a/docs/designs/data-directory-layout.md b/docs/designs/data-directory-layout.md index f4bac9a6a..ad1faa15c 100644 --- a/docs/designs/data-directory-layout.md +++ b/docs/designs/data-directory-layout.md @@ -376,7 +376,14 @@ scope that first reports it also appends the ID and its own data home key to recorded there is that scope's wherever it is resumed later, whatever the log still holds, so another scope never reports its transcript again; the first line for an ID wins, and the file grows by one line per such session, like the -snapshots. A +snapshots. An earlier release kept only per-scope snapshots, so the file is +first written from them: a tool's own ID in the user scope's or a partition's +prompt-token snapshot is that scope's, unless the shared snapshot holds it too +(that release copied the shared file into every scope, so such an ID names no +owner, and every scope already has its baseline). A project whose data home is +in its workspace cannot be found this way: a session of its that compaction +had already dropped gets no owner, and if resumed in another scope is decided +by its first retained event, as before. A fallback run is reported and snapshotted as `@`, which does not change when compaction drops earlier runs. A snapshot entry keyed by a bare fallback ID (written before) is the sum of the runs of that ID in the diff --git a/src/__tests__/dashboard-report-scope.test.ts b/src/__tests__/dashboard-report-scope.test.ts index fff19f420..938257f2c 100644 --- a/src/__tests__/dashboard-report-scope.test.ts +++ b/src/__tests__/dashboard-report-scope.test.ts @@ -484,6 +484,18 @@ describe('each scope keeps its own reported snapshot (#786)', () => { expect(await reportedPrompts(project)).toBe(1); }); + it('a session main reported in a project, resumed in another after compaction, stays the first project\'s', async () => { + const { project } = await setup(); + const { rootQ, projectQ } = await setupQ(); + // Main reported `resumed` in P: only P's own snapshot holds it, and main kept no owners. + writeSharedSnapshots({ resumed: 1 }, new Date().toISOString().slice(0, 10), project); + // Compaction dropped its events; `claude --resume` in Q. + await session('claude', { session_id: 'resumed', cwd: rootQ }); + + expect(await report(projectQ)).toBeNull(); + expect(await report(project)).toBeNull(); + }); + it('the session owners a report keeps hold no path (#666)', async () => { const { root, project } = await setup(); await session('copilot', { session_id: 'copilot-p', cwd: root }); diff --git a/src/team-push.ts b/src/team-push.ts index cc7564f07..d806bb6c3 100644 --- a/src/team-push.ts +++ b/src/team-push.ts @@ -17,6 +17,7 @@ import { log } from './utils/logger.js'; import type { UserStats, UserInterventionStats, SessionMetrics, TokenUsage, DashboardEvent, LocalConfig } from './types.js'; import { getVotesDir, getDataHome, getTeamaiHomeDir, emptyTokenUsage, addTokenUsage, usesBranchWorktree } from './types.js'; import { getUserHome } from './utils/home.js'; +import { projectsRootDir } from './utils/partition.js'; import { aggregateDailySessions, computeDailyStatsDelta, @@ -609,13 +610,54 @@ async function runsOfLog(events: DashboardEvent[]): Promise { // stays that scope's. Only the ID and the key, never a path (#666). The first // line for an ID wins. // +// A release before this file kept per-scope snapshots only, so the file is +// first written from them: each tool's own ID in the user scope's or a +// partition's prompt-token snapshot is that scope's. An ID the shared snapshot +// also holds is left out: that release copied the shared file into every +// scope, so it names no owner, and every scope already has its baseline. +// A project whose data home is in its workspace cannot be found this way. +// function sessionOwnersPath(): string { return path.join(getTeamaiHomeDir(), 'dashboard', 'session-owners.jsonl'); } +/** The owners the per-scope snapshots of an earlier release imply, as the file's first lines. */ +async function ownersFromSnapshots(): Promise { + const home = getTeamaiHomeDir(); + const shared = (await readJson>(sharedSnapshotPath('prompt-tokens'))) ?? {}; + const scopes = [{ dataHome: home, file: path.join(home, 'dashboard', 'user-reported-prompt-tokens.json') }]; + const slugs = await fs.promises.readdir(projectsRootDir()).catch(() => []); + for (const slug of [...slugs].sort()) { + const dataHome = path.join(projectsRootDir(), slug); + scopes.push({ dataHome, file: path.join(dataHome, 'dashboard', 'reported-prompt-tokens.json') }); + } + const lines: string[] = []; + const seen = new Set(); + for (const { dataHome, file } of scopes) { + const snapshot = await readJson>(file); + if (!snapshot || typeof snapshot !== 'object') continue; + const key = await dataHomeKey(dataHome); + for (const sessionId of Object.keys(snapshot)) { + if (sessionId.startsWith('pid-') || Object.hasOwn(shared, sessionId) || seen.has(sessionId)) continue; + seen.add(sessionId); + lines.push(JSON.stringify({ sessionId, dataHomeKey: key })); + } + } + return lines.map((line) => `${line}\n`).join(''); +} + async function readSessionOwners(): Promise> { const owners = new Map(); + if (!(await pathExists(sessionOwnersPath()))) { + try { + await ensureDir(path.dirname(sessionOwnersPath())); + // Exclusive: a report in another scope may be writing it too. + await fs.promises.writeFile(sessionOwnersPath(), await ownersFromSnapshots(), { flag: 'wx' }); + } catch (e) { + if ((e as NodeJS.ErrnoException).code !== 'EEXIST') log.debug(`Could not seed session owners: ${(e as Error).message}`); + } + } const content = await readFileSafe(sessionOwnersPath()); for (const line of (content ?? '').split('\n')) { if (!line.trim()) continue; From 9e8fef606b1d1e2e8510a6ab53a8c1051d89b1c3 Mon Sep 17 00:00:00 2001 From: Saul Moro Date: Thu, 24 Sep 2026 20:00:51 +0200 Subject: [PATCH 20/41] fix(stats): address CI review (#785) Owner migration reconciles every per-scope baseline of an ID: a tool's own ID in any of a scope's three snapshots is the scope's that holds its greatest total (prompts, then tokens). A session main split per event holds only part of it elsewhere, and a scope may have reported past the shared total it was seeded with, so neither the first holder nor leaving shared-held IDs out was right; a session reported with no prompts, only its intervention count, is found too. Besides the user scope and the partitions, it reads a project whose data home is in its workspace that a session still in the log leads to, and each report records the IDs of its own snapshots that no owner claims yet, for such a project the log no longer leads to. --- docs/designs/data-directory-layout.md | 14 +-- src/__tests__/dashboard-report-scope.test.ts | 72 +++++++++++++++ src/team-push.ts | 96 ++++++++++++++------ 3 files changed, 146 insertions(+), 36 deletions(-) diff --git a/docs/designs/data-directory-layout.md b/docs/designs/data-directory-layout.md index 98f95ee44..d2ad1e9bf 100644 --- a/docs/designs/data-directory-layout.md +++ b/docs/designs/data-directory-layout.md @@ -378,12 +378,14 @@ recorded there is that scope's wherever it is resumed later, whatever the log still holds, so another scope never reports its transcript again; the first line for an ID wins, and the file grows by one line per such session, like the snapshots. An earlier release kept only per-scope snapshots, so the file is -first written from them: a tool's own ID in the user scope's or a partition's -prompt-token snapshot is that scope's, unless the shared snapshot holds it too -(that release copied the shared file into every scope, so such an ID names no -owner, and every scope already has its baseline). A project whose data home is -in its workspace cannot be found this way: a session of its that compaction -had already dropped gets no owner, and if resumed in another scope is decided +first written from them: a tool's own ID in any of a scope's snapshots is the +scope's that holds its greatest total (prompts, then tokens), since a session +that release split per event holds only part of it elsewhere, and a scope may +have reported past the shared total it was seeded with. The scopes read are the +user scope, every partition, and a project whose data home is in its workspace +that a session still in the log leads to; each report also records the IDs of +its own snapshots that have no owner yet. A session of a workspace-data project +that neither reached gets no owner, and if resumed in another scope is decided by its first retained event, as before. A fallback run is reported and snapshotted as `@`, which does not change when compaction drops earlier runs. A snapshot entry keyed diff --git a/src/__tests__/dashboard-report-scope.test.ts b/src/__tests__/dashboard-report-scope.test.ts index 8a69c82d5..cee7bc595 100644 --- a/src/__tests__/dashboard-report-scope.test.ts +++ b/src/__tests__/dashboard-report-scope.test.ts @@ -511,6 +511,78 @@ describe('each scope keeps its own reported snapshot (#786)', () => { expect(await report(project)).toBeNull(); }); + /** `claude --resume` of `sessionId` in `cwd`, whose Stop carries the transcript's `prompts`. */ + async function resume(sessionId: string, cwd: string, prompts: number): Promise { + await session('claude', { session_id: sessionId, cwd }); + const log = path.join(teamaiHome(), 'dashboard', 'events.jsonl'); + fs.writeFileSync(log, fs.readFileSync(log, 'utf-8').split('\n').filter(Boolean).map((line) => { + const event = JSON.parse(line); + return JSON.stringify(event.sessionId === sessionId && event.type === 'stop' ? { ...event, prompts } : event); + }).join('\n') + '\n'); + } + + it('main split a session between two projects: the one holding the greater total owns it', async () => { + const { root, project } = await setup(); + const { projectQ } = await setupQ(); + const today = new Date().toISOString().slice(0, 10); + // Main reported 3 prompts to P before the session moved, then its cumulative 5 to Q. + writeSharedSnapshots({ moved: 3 }, today, project); + writeSharedSnapshots({ moved: 5 }, today, projectQ); + // Compaction dropped its events; resumed in P, its Stop carries 6. + await resume('moved', root, 6); + + expect(await report(project)).toBeNull(); + expect(await reportedPrompts(projectQ)).toBe(1); + }); + + it('a project that reported a session past the shared snapshot\'s total owns it', async () => { + const { project } = await setup(); + const { rootQ, projectQ } = await setupQ(); + const today = new Date().toISOString().slice(0, 10); + // The shared file held 1 prompt; P, seeded from it, then reported through prompt 2. + writeSharedSnapshots({ resumed: 1 }, today); + writeSharedSnapshots({ resumed: 2 }, today, project); + await resume('resumed', rootQ, 3); + + expect(await report(projectQ)).toBeNull(); + expect(await reportedPrompts(project)).toBe(1); + }); + + it('a session a project reported with no prompts, only its intervention count, stays that project\'s', async () => { + const { project } = await setup(); + const { rootQ, projectQ } = await setupQ(); + writeSharedSnapshots({ quiet: 0 }, new Date().toISOString().slice(0, 10), project); + for (const name of ['prompt-tokens', 'daily-sessions']) { + fs.rmSync(path.join(getDataHome(project), 'dashboard', `reported-${name}.json`)); + } + await resume('quiet', rootQ, 0); + + expect(await reportedInterventionSessions(projectQ)).toBe(0); + }); + + it('a session of a project whose data home is in its workspace stays that project\'s', async () => { + await setup(); + const { rootQ, projectQ } = await setupQ(); + // Not a git repo: its data home is `/.teamai`, under no partition. + const rootW = path.join(tmp, 'plain-dir'); + fs.mkdirSync(rootW); + const dataHomeW = await resolveProjectDataHome(rootW); + fs.mkdirSync(path.join(dataHomeW, 'team-repo'), { recursive: true }); + await saveLocalConfigForScope({ + repo: { localPath: path.join(dataHomeW, 'team-repo'), remote: 'https://example.test/acme/team-w.git', kind: 'git' }, + username: 'tester', scope: 'project', projectRoot: rootW, additionalRoles: [], dataHome: dataHomeW, + }); + const projectW = await resolveConfigForDir(rootW); + if (!projectW) throw new Error('fixture config W did not resolve'); + expect(getDataHome(projectW)).toBe(path.join(rootW, '.teamai')); + writeSharedSnapshots({ resumed: 1 }, new Date().toISOString().slice(0, 10), projectW); + // Another of W's sessions is still in the log. + await session('claude', { session_id: 'other-w', cwd: rootW }); + await resume('resumed', rootQ, 2); + + expect(await report(projectQ)).toBeNull(); + }); + it('the session owners a report keeps hold no path (#666)', async () => { const { root, project } = await setup(); await session('copilot', { session_id: 'copilot-p', cwd: root }); diff --git a/src/team-push.ts b/src/team-push.ts index d806bb6c3..eec696d82 100644 --- a/src/team-push.ts +++ b/src/team-push.ts @@ -155,7 +155,8 @@ export function mergeStats( // the whole log and reports into it). // -type ReportedSnapshotName = 'interventions' | 'prompt-tokens' | 'daily-sessions'; +const REPORTED_SNAPSHOTS = ['interventions', 'prompt-tokens', 'daily-sessions'] as const; +type ReportedSnapshotName = typeof REPORTED_SNAPSHOTS[number]; /** The machine-level snapshot every scope shared before #786 (evaluated at call time for tests). */ function sharedSnapshotPath(name: ReportedSnapshotName): string { @@ -164,8 +165,11 @@ function sharedSnapshotPath(name: ReportedSnapshotName): string { /** A scope's own snapshot. The user scope's data home holds the shared one, hence its prefix. */ function scopeSnapshotPath(name: ReportedSnapshotName, config: LocalConfig | undefined): string { - if (!config) return sharedSnapshotPath(name); - const dataHome = getDataHome(config); + return config ? snapshotPathIn(getDataHome(config), name) : sharedSnapshotPath(name); +} + +/** The snapshot of the scope whose data home is `dataHome`. */ +function snapshotPathIn(dataHome: string, name: ReportedSnapshotName): string { const file = `reported-${name}.json`; if (path.resolve(dataHome) !== path.resolve(getTeamaiHomeDir())) return path.join(dataHome, 'dashboard', file); return path.join(dataHome, 'dashboard', `user-${file}`); @@ -611,40 +615,65 @@ async function runsOfLog(events: DashboardEvent[]): Promise { // line for an ID wins. // // A release before this file kept per-scope snapshots only, so the file is -// first written from them: each tool's own ID in the user scope's or a -// partition's prompt-token snapshot is that scope's. An ID the shared snapshot -// also holds is left out: that release copied the shared file into every -// scope, so it names no owner, and every scope already has its baseline. -// A project whose data home is in its workspace cannot be found this way. +// first written from them: each tool's own ID in any snapshot of a scope is +// the scope's that holds its greatest total (prompts, then tokens), since a +// session that release split per event holds only part of it elsewhere. The +// scopes read are the user scope, every partition, and a project whose data +// home is in its workspace that a session still in the log leads to. Each +// report also records the IDs of its own snapshots that have no owner yet, +// for such a project the log no longer leads to. // function sessionOwnersPath(): string { return path.join(getTeamaiHomeDir(), 'dashboard', 'session-owners.jsonl'); } -/** The owners the per-scope snapshots of an earlier release imply, as the file's first lines. */ -async function ownersFromSnapshots(): Promise { - const home = getTeamaiHomeDir(); - const shared = (await readJson>(sharedSnapshotPath('prompt-tokens'))) ?? {}; - const scopes = [{ dataHome: home, file: path.join(home, 'dashboard', 'user-reported-prompt-tokens.json') }]; +/** The data homes whose snapshots an earlier release may have written. */ +async function knownDataHomes(): Promise { const slugs = await fs.promises.readdir(projectsRootDir()).catch(() => []); - for (const slug of [...slugs].sort()) { - const dataHome = path.join(projectsRootDir(), slug); - scopes.push({ dataHome, file: path.join(dataHome, 'dashboard', 'reported-prompt-tokens.json') }); + const homes = [getTeamaiHomeDir(), ...[...slugs].sort().map((slug) => path.join(projectsRootDir(), slug))]; + // A project whose data home is in its workspace is under no partition; a + // session of it still in the log leads to it. + const { resolveConfigForDir } = await import('./config.js'); + const cwds = new Set((await readEvents()).flatMap((e) => (typeof e.cwd === 'string' ? [e.cwd] : []))); + for (const cwd of cwds) { + const config = (await pathExists(cwd)) ? await resolveConfigForDir(cwd) : null; + if (config) homes.push(getDataHome(config)); } - const lines: string[] = []; - const seen = new Set(); - for (const { dataHome, file } of scopes) { - const snapshot = await readJson>(file); - if (!snapshot || typeof snapshot !== 'object') continue; + return [...new Set(homes.map((home) => path.resolve(home)))]; +} + +/** A snapshot entry's reported prompts and tokens, 0 for what it does not hold. */ +function reportedSize(entry: unknown): { prompts: number; tokens: number } { + if (!entry || typeof entry !== 'object') return { prompts: 0, tokens: 0 }; + const prompts = 'prompts' in entry && typeof entry.prompts === 'number' ? entry.prompts : 0; + const tokens = 'tokens' in entry && entry.tokens && typeof entry.tokens === 'object' + ? Object.values(entry.tokens).reduce((sum: number, n: unknown) => sum + (typeof n === 'number' ? n : 0), 0) + : 0; + return { prompts, tokens }; +} + +/** The owners the per-scope snapshots of an earlier release imply, as the file's first lines. */ +async function ownersFromSnapshots(): Promise { + const best = new Map(); + for (const dataHome of await knownDataHomes()) { + const [interventions, promptTokens, daily] = await Promise.all( + REPORTED_SNAPSHOTS.map((name) => readJson>(snapshotPathIn(dataHome, name))), + ); + const ids = new Set([interventions, promptTokens, daily].flatMap((snapshot) => + (snapshot && typeof snapshot === 'object' ? Object.keys(snapshot) : []))); const key = await dataHomeKey(dataHome); - for (const sessionId of Object.keys(snapshot)) { - if (sessionId.startsWith('pid-') || Object.hasOwn(shared, sessionId) || seen.has(sessionId)) continue; - seen.add(sessionId); - lines.push(JSON.stringify({ sessionId, dataHomeKey: key })); + for (const sessionId of ids) { + if (sessionId.startsWith('pid-')) continue; + const fromTokens = reportedSize(promptTokens?.[sessionId]); + const size = { prompts: Math.max(fromTokens.prompts, reportedSize(daily?.[sessionId]).prompts), tokens: fromTokens.tokens }; + const held = best.get(sessionId); + if (!held || size.prompts > held.prompts || (size.prompts === held.prompts && size.tokens > held.tokens)) { + best.set(sessionId, { key, ...size }); + } } } - return lines.map((line) => `${line}\n`).join(''); + return [...best].map(([sessionId, { key }]) => `${JSON.stringify({ sessionId, dataHomeKey: key })}\n`).join(''); } async function readSessionOwners(): Promise> { @@ -673,10 +702,10 @@ async function readSessionOwners(): Promise> { return owners; } -/** Records `key` as the owner of the tool-own session IDs among `events` that have none yet. */ -async function recordSessionOwners(events: DashboardEvent[], key: string): Promise { +/** Records `key` as the owner of the tool-own session IDs among `sessionIds` that have none yet. */ +async function recordSessionOwners(sessionIds: Iterable, key: string): Promise { const owners = await readSessionOwners(); - const ids = new Set(events.map((e) => e.sessionId).filter((id) => !id.startsWith('pid-') && !owners.has(id))); + const ids = new Set([...sessionIds].filter((id) => !id.startsWith('pid-') && !owners.has(id))); if (ids.size === 0) return; try { await ensureDir(path.dirname(sessionOwnersPath())); @@ -791,7 +820,6 @@ export async function reportUsageToTeam( // both the intervention delta and the prompt-count/token delta from it. // Only the sessions recorded in this scope (#785). const dashboardEvents = await filterEventsByScope(await readEvents(), reportsConfig); - if (reportsConfig) await recordSessionOwners(dashboardEvents, await dataHomeKey(getDataHome(reportsConfig))); const metrics = aggregateSessionMetrics(dashboardEvents); const currentInterventions = interventionCounts(metrics); @@ -835,6 +863,14 @@ export async function reportUsageToTeam( currentDaily, reportedDailySessions, ); + // This scope's sessions, and those of its snapshots no owner claims yet + // (a project whose data home is in its workspace, which the seed may miss). + if (reportsConfig) { + await recordSessionOwners([ + ...dashboardEvents.map((e) => e.sessionId), + ...Object.keys(reportedPromptTokens), ...Object.keys(reportedInterventions), ...Object.keys(reportedDailySessions), + ], await dataHomeKey(getDataHome(reportsConfig))); + } const hasUsage = events.length > 0; const hasInterventions = hasInterventionDelta(interventionDelta); From ae37f007b460c31b448c2fdfbc83eafd71d0d7fe Mon Sep 17 00:00:00 2001 From: Saul Moro Date: Thu, 24 Sep 2026 20:19:28 +0200 Subject: [PATCH 21/41] fix(stats): address CI review (#785) Owner migration assigns no owner when the greatest total ties across scopes: main copied the shared snapshot into every scope, so equal totals show only that copy, and each scope already holds the baseline. A report records an ID of its own snapshots only when they show it reported it (absent from the shared snapshot, or past its total there), so a copy no longer claims it either. A crashed fallback run a start from another process supersedes counts as the run closed before it, so a late unannotated exit of it no longer closes the new run. --- docs/designs/data-directory-layout.md | 10 ++-- src/__tests__/dashboard-report-scope.test.ts | 17 ++++++- src/__tests__/scope-filter.test.ts | 20 ++++++++ src/team-push.ts | 52 +++++++++++++++++--- 4 files changed, 88 insertions(+), 11 deletions(-) diff --git a/docs/designs/data-directory-layout.md b/docs/designs/data-directory-layout.md index d2ad1e9bf..2b9193590 100644 --- a/docs/designs/data-directory-layout.md +++ b/docs/designs/data-directory-layout.md @@ -360,7 +360,8 @@ Copilot's is the parent PID) names one run up to its `session_end` or recorded since the first (the dashboard monitor's `process_exit` after `SessionEnd`) belongs to the run it closed. A `session_start` on a fallback ID (`pid-…`) whose `monitorPid` differs from its open run's begins a new run even -though nothing ended that one (a crash with no dashboard running); a tool's own +though nothing ended that one (a crash with no dashboard running), and that one +counts as the run closed before it; a tool's own ID is not split this way, since Claude fires SessionStart again on resume, in a new process, and its Stop carries the whole transcript. The monitor's `process_exit` also records `processExitAfter`, the last event it observed, and closes only that @@ -381,10 +382,13 @@ snapshots. An earlier release kept only per-scope snapshots, so the file is first written from them: a tool's own ID in any of a scope's snapshots is the scope's that holds its greatest total (prompts, then tokens), since a session that release split per event holds only part of it elsewhere, and a scope may -have reported past the shared total it was seeded with. The scopes read are the +have reported past the shared total it was seeded with. A tie names no owner: +that release copied the shared file into every scope, so equal totals show only +the copy, and each scope already holds that baseline. The scopes read are the user scope, every partition, and a project whose data home is in its workspace that a session still in the log leads to; each report also records the IDs of -its own snapshots that have no owner yet. A session of a workspace-data project +its own snapshots that have no owner yet and show it reported them (absent from +the shared snapshot, or past its total there). A session of a workspace-data project that neither reached gets no owner, and if resumed in another scope is decided by its first retained event, as before. A fallback run is reported and snapshotted as `@`, diff --git a/src/__tests__/dashboard-report-scope.test.ts b/src/__tests__/dashboard-report-scope.test.ts index cee7bc595..959d51e47 100644 --- a/src/__tests__/dashboard-report-scope.test.ts +++ b/src/__tests__/dashboard-report-scope.test.ts @@ -237,8 +237,10 @@ describe('each scope keeps its own reported snapshot (#786)', () => { 'daily-sessions': entries((n) => ({ date, prompts: n, durationMs: 3_600_000, succeeded: 1, corrected: 0 })), }; const dir = path.join(config ? getDataHome(config) : teamaiHome(), 'dashboard'); + // The user scope's own snapshot sits beside the shared one, prefixed. + const prefix = config?.scope === 'user' ? 'user-' : ''; fs.mkdirSync(dir, { recursive: true }); - for (const name of SNAPSHOTS) fs.writeFileSync(path.join(dir, `reported-${name}.json`), JSON.stringify(values[name])); + for (const name of SNAPSHOTS) fs.writeFileSync(path.join(dir, `${prefix}reported-${name}.json`), JSON.stringify(values[name])); } async function prompts(sessionId: string, cwd: string, count: number): Promise { @@ -535,6 +537,19 @@ describe('each scope keeps its own reported snapshot (#786)', () => { expect(await reportedPrompts(projectQ)).toBe(1); }); + it('equal totals copied from the shared snapshot name no owner: the scope it resumes in reports it', async () => { + const { root, user, project } = await setup(); + const today = new Date().toISOString().slice(0, 10); + // Main copied the shared entry into both scopes' snapshots: they tie. + writeSharedSnapshots({ copied: 1 }, today); + writeSharedSnapshots({ copied: 1 }, today, user); + writeSharedSnapshots({ copied: 1 }, today, project); + await resume('copied', root, 2); + + expect(await report(user)).toBeNull(); + expect(await reportedPrompts(project)).toBe(1); + }); + it('a project that reported a session past the shared snapshot\'s total owns it', async () => { const { project } = await setup(); const { rootQ, projectQ } = await setupQ(); diff --git a/src/__tests__/scope-filter.test.ts b/src/__tests__/scope-filter.test.ts index f119f7c84..bf55b1e3d 100644 --- a/src/__tests__/scope-filter.test.ts +++ b/src/__tests__/scope-filter.test.ts @@ -158,6 +158,26 @@ describe('filterEventsByScope', () => { ]); }); + it('an unannotated exit of a run a new process superseded does not end the new run', async () => { + // The first run crashed with nothing ending it; a dashboard before + // processExitAfter appends its exit after the next invocation started. + const p = '/home/jeff/.teamai/projects/p'; + const at = async (type: DashboardEvent['type'], timestamp: string, fields: Partial = {}) => + ({ ...(await scopedEvent(undefined, 'pid-1', p)), type, timestamp, ...fields }); + const log = [ + await at('session_start', '2026-01-01T00:00:00.000Z', { monitorPid: 100 }), + await at('prompt_submit', '2026-01-01T00:00:30.000Z'), + await at('session_start', '2026-01-01T00:02:00.000Z', { monitorPid: 200 }), + await at('process_exit', '2026-01-01T00:02:05.000Z'), + await at('prompt_submit', '2026-01-01T00:03:00.000Z'), + ]; + const project = await filterEventsByScope(log, projectScope('/Users/jeff/project-a')); + expect(project.map((e) => e.sessionId)).toEqual([ + 'pid-1@2026-01-01T00:00:00.000Z', 'pid-1@2026-01-01T00:00:00.000Z', + 'pid-1@2026-01-01T00:02:00.000Z', 'pid-1@2026-01-01T00:00:00.000Z', 'pid-1@2026-01-01T00:02:00.000Z', + ]); + }); + it('an unannotated exit with nothing more of the open run ends it', async () => { const p = '/home/jeff/.teamai/projects/p'; const at = async (type: DashboardEvent['type'], timestamp: string) => diff --git a/src/team-push.ts b/src/team-push.ts index eec696d82..b0181dbd6 100644 --- a/src/team-push.ts +++ b/src/team-push.ts @@ -563,7 +563,11 @@ function splitRuns(events: DashboardEvent[], eventKeys: Array { // A release before this file kept per-scope snapshots only, so the file is // first written from them: each tool's own ID in any snapshot of a scope is // the scope's that holds its greatest total (prompts, then tokens), since a -// session that release split per event holds only part of it elsewhere. The +// session that release split per event holds only part of it elsewhere. A tie +// names no owner: that release copied the shared file into every scope. The // scopes read are the user scope, every partition, and a project whose data // home is in its workspace that a session still in the log leads to. Each // report also records the IDs of its own snapshots that have no owner yet, -// for such a project the log no longer leads to. +// for such a project the log no longer leads to, when they show it reported +// them: absent from the shared snapshot, or past its total there. // function sessionOwnersPath(): string { @@ -655,7 +661,7 @@ function reportedSize(entry: unknown): { prompts: number; tokens: number } { /** The owners the per-scope snapshots of an earlier release imply, as the file's first lines. */ async function ownersFromSnapshots(): Promise { - const best = new Map(); + const best = new Map(); for (const dataHome of await knownDataHomes()) { const [interventions, promptTokens, daily] = await Promise.all( REPORTED_SNAPSHOTS.map((name) => readJson>(snapshotPathIn(dataHome, name))), @@ -669,11 +675,16 @@ async function ownersFromSnapshots(): Promise { const size = { prompts: Math.max(fromTokens.prompts, reportedSize(daily?.[sessionId]).prompts), tokens: fromTokens.tokens }; const held = best.get(sessionId); if (!held || size.prompts > held.prompts || (size.prompts === held.prompts && size.tokens > held.tokens)) { - best.set(sessionId, { key, ...size }); + best.set(sessionId, { key, ...size, tied: false }); + } else if (size.prompts === held.prompts && size.tokens === held.tokens && key !== held.key) { + held.tied = true; } } } - return [...best].map(([sessionId, { key }]) => `${JSON.stringify({ sessionId, dataHomeKey: key })}\n`).join(''); + // A tie is a copy of one shared entry that release made in every scope: it + // names no owner, and each scope already holds that baseline. + return [...best].flatMap(([sessionId, { key, tied }]) => + (tied ? [] : [`${JSON.stringify({ sessionId, dataHomeKey: key })}\n`])).join(''); } async function readSessionOwners(): Promise> { @@ -702,6 +713,33 @@ async function readSessionOwners(): Promise> { return owners; } +/** + * The IDs of a scope's snapshots that show it reported them: absent from the + * shared snapshot, or past its total there. A copy of a shared entry, which + * an earlier release made in every scope, shows nothing. + */ +async function reportedBeyondShared( + promptTokens: Record, + interventions: Record, + daily: Record, +): Promise { + const [sharedTokens, sharedDaily] = await Promise.all([ + readJson>(sharedSnapshotPath('prompt-tokens')), + readJson>(sharedSnapshotPath('daily-sessions')), + ]); + const size = (tokens: Record | null, days: Record | null, id: string) => { + const fromTokens = reportedSize(tokens?.[id]); + return { prompts: Math.max(fromTokens.prompts, reportedSize(days?.[id]).prompts), tokens: fromTokens.tokens }; + }; + const ids = new Set([...Object.keys(promptTokens), ...Object.keys(interventions), ...Object.keys(daily)]); + return [...ids].filter((id) => { + if (!Object.hasOwn(sharedTokens ?? {}, id) && !Object.hasOwn(sharedDaily ?? {}, id)) return true; + const own = size(promptTokens, daily, id); + const shared = size(sharedTokens, sharedDaily, id); + return own.prompts > shared.prompts || (own.prompts === shared.prompts && own.tokens > shared.tokens); + }); +} + /** Records `key` as the owner of the tool-own session IDs among `sessionIds` that have none yet. */ async function recordSessionOwners(sessionIds: Iterable, key: string): Promise { const owners = await readSessionOwners(); @@ -868,7 +906,7 @@ export async function reportUsageToTeam( if (reportsConfig) { await recordSessionOwners([ ...dashboardEvents.map((e) => e.sessionId), - ...Object.keys(reportedPromptTokens), ...Object.keys(reportedInterventions), ...Object.keys(reportedDailySessions), + ...await reportedBeyondShared(reportedPromptTokens, reportedInterventions, reportedDailySessions), ], await dataHomeKey(getDataHome(reportsConfig))); } From fd1f163c3623caac59e29b7abe23eb2c51be3fcc Mon Sep 17 00:00:00 2001 From: Saul Moro Date: Thu, 24 Sep 2026 20:33:42 +0200 Subject: [PATCH 22/41] test(stats): pin a pre-upgrade exit reported before the next run's first prompt (#785) The run split is recomputed from the whole log on every report, so once the next run's first prompt follows the unannotated exit, the exit is the earlier run's and the next run keeps its ID: the second pull reports only its delta, not another session. --- src/__tests__/dashboard-report-scope.test.ts | 23 ++++++++++++++++++++ 1 file changed, 23 insertions(+) diff --git a/src/__tests__/dashboard-report-scope.test.ts b/src/__tests__/dashboard-report-scope.test.ts index 959d51e47..16458382c 100644 --- a/src/__tests__/dashboard-report-scope.test.ts +++ b/src/__tests__/dashboard-report-scope.test.ts @@ -33,6 +33,8 @@ const { hookDispatchCli } = await import('../hook-dispatch-cli.js'); const { resolveProjectDataHome, saveLocalConfigForScope, resolveConfigForDir, loadLocalConfig } = await import('../config.js'); const { reportUsageToTeam } = await import('../team-push.js'); const { getDataHome } = await import('../types.js'); +const { dataHomeKey } = await import('../dashboard-collector.js'); +const dataHomeKeyOf = (config: LocalConfig) => dataHomeKey(getDataHome(config)); let tmp: string; let originalHome: string | undefined; @@ -643,6 +645,27 @@ describe('each scope keeps its own reported snapshot (#786)', () => { expect(await reportedPrompts(project)).toBe(1); }); + it('a pre-upgrade exit reported before the next run\'s first prompt does not count that run twice', async () => { + const { project } = await setup(); + const p = await dataHomeKeyOf(project); + const log = path.join(teamaiHome(), 'dashboard', 'events.jsonl'); + const at = (type: string, s: number) => + JSON.stringify({ type, timestamp: new Date(Date.now() - 600_000 + s * 1000).toISOString(), sessionId: 'pid-9', tool: 'copilot', dataHomeKey: p }); + fs.mkdirSync(path.dirname(log), { recursive: true }); + // Run 1 ends; run 2 starts; a dashboard from before processExitAfter appends + // the exit it observed for run 1. A pull runs before run 2's first prompt. + fs.writeFileSync(log, [at('session_start', 0), at('prompt_submit', 1), at('stop', 2), at('session_end', 3), + at('session_start', 10), at('process_exit', 11)].join('\n') + '\n'); + expect(await reportedInterventionSessions(project)).toBe(2); + // Run 2 goes on: the exit is now followed by its activity, so it was run 1's, + // and run 2 keeps its ID, compared against what the first pull reported. + fs.appendFileSync(log, [at('prompt_submit', 20), at('stop', 21)].join('\n') + '\n'); + + expect(await reportedInterventionSessions(project)).toBe(2); + expect(await reportedSessions(project)).toBe(2); + expect(await reportedPrompts(project)).toBe(2); + }); + it('a fallback ID reused in a project after a user-scope run that never ended is the project\'s', async () => { const { root, user, project } = await setup(); const elsewhere = path.join(tmp, 'elsewhere'); From 10e9d92a291ebe996684f801e5cdeded95fae909 Mon Sep 17 00:00:00 2001 From: Saul Moro Date: Thu, 24 Sep 2026 20:51:27 +0200 Subject: [PATCH 23/41] fix(stats): give a compacted session resumed elsewhere to the project its transcript started in (#785) Once compaction dropped every event of a project whose data home is in its workspace, nothing outside it pointed to it, so a resume of its Claude session in another project reported the transcript there again. The transcript itself records where the session started: a Claude transcript keeps its first cwd when resumed from another project (the resume appends to the same file), as a Codex rollout keeps its session_meta. Hooks now record transcriptPath on UserPromptSubmit and SessionEnd as well as Stop (not SessionStart, whose path on such a resume names a file that never exists; never Copilot's). A tool's own session with no owner is the scope's that its origin resolves to, when that scope's snapshots already hold it; otherwise it is decided as before. --- docs/designs/data-directory-layout.md | 13 +++- src/__tests__/dashboard-report-scope.test.ts | 44 ++++++++++-- src/dashboard-collector.ts | 9 +++ src/team-push.ts | 72 +++++++++++++++++++- src/types.ts | 2 +- 5 files changed, 130 insertions(+), 10 deletions(-) diff --git a/docs/designs/data-directory-layout.md b/docs/designs/data-directory-layout.md index 2b9193590..5f2da8f05 100644 --- a/docs/designs/data-directory-layout.md +++ b/docs/designs/data-directory-layout.md @@ -388,9 +388,16 @@ the copy, and each scope already holds that baseline. The scopes read are the user scope, every partition, and a project whose data home is in its workspace that a session still in the log leads to; each report also records the IDs of its own snapshots that have no owner yet and show it reported them (absent from -the shared snapshot, or past its total there). A session of a workspace-data project -that neither reached gets no owner, and if resumed in another scope is decided -by its first retained event, as before. A +the shared snapshot, or past its total there). A session none of these reach +(a project whose data home is in its workspace, with no event left in the log) +is found by its transcript: hooks record `transcriptPath` on UserPromptSubmit, +Stop and SessionEnd (not SessionStart, whose path on a resume from another +project names a file that never exists; never Copilot's), and a Claude +transcript keeps its first `cwd` when resumed elsewhere, as a Codex rollout +keeps its `session_meta`. So a tool's own session with no owner is the scope's +that directory resolves to, when that scope's snapshots already hold it; else +it is decided as before (a fork under a new ID, a tool whose transcript records +no start, Copilot). A fallback run is reported and snapshotted as `@`, which does not change when compaction drops earlier runs. A snapshot entry keyed by a bare fallback ID (written before) is the sum of the runs of that ID in the diff --git a/src/__tests__/dashboard-report-scope.test.ts b/src/__tests__/dashboard-report-scope.test.ts index 16458382c..6b8dcbb2a 100644 --- a/src/__tests__/dashboard-report-scope.test.ts +++ b/src/__tests__/dashboard-report-scope.test.ts @@ -580,7 +580,18 @@ describe('each scope keeps its own reported snapshot (#786)', () => { it('a session of a project whose data home is in its workspace stays that project\'s', async () => { await setup(); const { rootQ, projectQ } = await setupQ(); - // Not a git repo: its data home is `/.teamai`, under no partition. + const { rootW, projectW } = await setupWorkspaceProject(); + expect(getDataHome(projectW)).toBe(path.join(rootW, '.teamai')); + writeSharedSnapshots({ resumed: 1 }, new Date().toISOString().slice(0, 10), projectW); + // Another of W's sessions is still in the log. + await session('claude', { session_id: 'other-w', cwd: rootW }); + await resume('resumed', rootQ, 2); + + expect(await report(projectQ)).toBeNull(); + }); + + /** A project with no git repo, so its data home is `/.teamai`, under no partition. */ + async function setupWorkspaceProject(): Promise<{ rootW: string; projectW: LocalConfig }> { const rootW = path.join(tmp, 'plain-dir'); fs.mkdirSync(rootW); const dataHomeW = await resolveProjectDataHome(rootW); @@ -591,15 +602,38 @@ describe('each scope keeps its own reported snapshot (#786)', () => { }); const projectW = await resolveConfigForDir(rootW); if (!projectW) throw new Error('fixture config W did not resolve'); - expect(getDataHome(projectW)).toBe(path.join(rootW, '.teamai')); + return { rootW, projectW }; + } + + it.each([ + ['claude', (cwd: string) => ({ type: 'user', sessionId: 'resumed', cwd, message: { role: 'user', content: 'start' } })], + ['codex', (cwd: string) => ({ type: 'session_meta', payload: { id: 'resumed', cwd } })], + ])('a %s session resumed elsewhere after compaction goes to the project its transcript started in', async (tool, first) => { + await setup(); + const { rootQ, projectQ } = await setupQ(); + const { rootW, projectW } = await setupWorkspaceProject(); + // An earlier release reported it in W; compaction dropped every W event. writeSharedSnapshots({ resumed: 1 }, new Date().toISOString().slice(0, 10), projectW); - // Another of W's sessions is still in the log. - await session('claude', { session_id: 'other-w', cwd: rootW }); - await resume('resumed', rootQ, 2); + // The resume appends to the transcript the session started, whose first cwd is W. + const transcript = path.join(tmp, `${tool}-resumed.jsonl`); + fs.writeFileSync(transcript, JSON.stringify(first(rootW)) + '\n'); + await session(tool, { session_id: 'resumed', cwd: rootQ, transcript_path: transcript }); expect(await report(projectQ)).toBeNull(); }); + it('a transcript that started in a project whose snapshot lacks the session does not hand it there', async () => { + await setup(); + const { rootQ, projectQ } = await setupQ(); + const { rootW } = await setupWorkspaceProject(); + const transcript = path.join(tmp, 'forked.jsonl'); + // A fork copies W's history under a new ID W never reported: Q's as usual. + fs.writeFileSync(transcript, JSON.stringify({ type: 'user', sessionId: 'forked', cwd: rootW }) + '\n'); + await session('claude', { session_id: 'forked', cwd: rootQ, transcript_path: transcript }); + + expect(await reportedSessions(projectQ)).toBe(1); + }); + it('the session owners a report keeps hold no path (#666)', async () => { const { root, project } = await setup(); await session('copilot', { session_id: 'copilot-p', cwd: root }); diff --git a/src/dashboard-collector.ts b/src/dashboard-collector.ts index c688c375b..b61fbafec 100644 --- a/src/dashboard-collector.ts +++ b/src/dashboard-collector.ts @@ -1225,6 +1225,15 @@ export async function parseHookEvent( if (!isCopilot) event.promptSummary = redactWithEnv(human).slice(0, 200); } + // The transcript a session runs in records where it started, which a report + // needs once compaction dropped the session's earlier events (#785). A + // SessionStart on a resume from another project names a file that never + // exists, so it is not kept there. + if ((eventType === 'prompt_submit' || eventType === 'session_end') && !isCopilot + && typeof hookData.transcript_path === 'string') { + event.transcriptPath = hookData.transcript_path; + } + // Extract transcript path, AI output and intervention counts from Stop event if (eventType === 'stop' && !isCopilot && typeof hookData.transcript_path === 'string') { event.transcriptPath = hookData.transcript_path; diff --git a/src/team-push.ts b/src/team-push.ts index b0181dbd6..e19a2a9a2 100644 --- a/src/team-push.ts +++ b/src/team-push.ts @@ -478,10 +478,16 @@ export async function filterEventsByScope( const eventKeys = await keysOf(events); const { runOf, runIds, deciding } = splitRuns(events, eventKeys); const owners = await readSessionOwners(); + const transcripts = new Map(); + events.forEach((e, i) => { + const run = runOf[i]; + if (run !== undefined && typeof e.transcriptPath === 'string') transcripts.set(run, [...(transcripts.get(run) ?? []), e.transcriptPath]); + }); const owned = await Promise.all(deciding.map(async (i, run) => { // Recorded by the scope that first reported it, so a session resumed // elsewhere after compaction dropped its events stays that scope's. - const owner = owners.get(runIds[run]); + const owner = owners.get(runIds[run]) ?? (runIds[run].startsWith('pid-') ? undefined + : await transcriptOwner(runIds[run], transcripts.get(run) ?? [], resolveConfigForDir)); if (owner !== undefined) return keys.has(owner); if (i === undefined) return false; const key = eventKeys[i]; @@ -494,6 +500,70 @@ export async function filterEventsByScope( }); } +/** + * The key of the scope a tool's own session started in, by its transcript, + * when that scope's snapshots already hold it: an earlier release reported it + * there, and compaction has since dropped the events that would say so. The + * latest transcript path first, since a tool may relocate the file. Undefined + * when there is no such transcript, origin, scope or snapshot entry (a fork + * under a new ID, a scope that never reported it). + */ +async function transcriptOwner( + sessionId: string, + paths: string[], + resolveConfigForDir: (dir: string) => Promise, +): Promise { + for (const transcript of [...paths].reverse()) { + const origin = await transcriptOrigin(transcript); + if (origin === undefined) continue; + const config = (await pathExists(origin)) ? await resolveConfigForDir(origin) : null; + if (!config) return undefined; + const dataHome = getDataHome(config); + const snapshots = await Promise.all(REPORTED_SNAPSHOTS.map((name) => + readJson>(snapshotPathIn(dataHome, name)))); + const held = snapshots.some((snapshot) => !!snapshot && typeof snapshot === 'object' && Object.hasOwn(snapshot, sessionId)); + return held ? dataHomeKey(dataHome) : undefined; + } + return undefined; +} + +/** + * The directory a transcript's session started in: the first `cwd` a Claude + * transcript records (a resume from another project appends to the same file), + * or a Codex rollout's `session_meta`. Reads a bounded head of the file; the + * format is the tool's own, so anything else is undefined. + */ +async function transcriptOrigin(transcript: string): Promise { + let head: string; + try { + const fh = await fs.promises.open(transcript, 'r'); + try { + const buffer = Buffer.alloc(64 * 1024); + const { bytesRead } = await fh.read(buffer, 0, buffer.length, 0); + head = buffer.subarray(0, bytesRead).toString('utf8'); + } finally { + await fh.close(); + } + } catch { + return undefined; + } + for (const line of head.split('\n')) { + let entry: unknown; + try { + entry = JSON.parse(line); + } catch { + continue; + } + if (!entry || typeof entry !== 'object') continue; + if ('cwd' in entry && typeof entry.cwd === 'string') return entry.cwd; + if ('type' in entry && entry.type === 'session_meta' && 'payload' in entry && entry.payload + && typeof entry.payload === 'object' && 'cwd' in entry.payload && typeof entry.payload.cwd === 'string') { + return entry.payload.cwd; + } + } + return undefined; +} + /** * Each event's data home key. The log is hand-editable: a key that is not a * string counts as absent. An unreleased build of #795 recorded the data home diff --git a/src/types.ts b/src/types.ts index 07d3aab22..d89e97677 100644 --- a/src/types.ts +++ b/src/types.ts @@ -1267,7 +1267,7 @@ export interface DashboardEvent { status?: DashboardSessionStatus; /** AI output captured from transcript at session stop (truncated to 500 chars) */ stoppedOutput?: string; - /** Path to Claude Code transcript file (from Stop hook STDIN) */ + /** The session's transcript (from the Stop, UserPromptSubmit and SessionEnd hook STDIN; never Copilot's) */ transcriptPath?: string; /** Resolved PID of the AI tool main process (for liveness monitoring) */ monitorPid?: number; From a56be8e48a01262da2a32533e0f4c50662e30508 Mon Sep 17 00:00:00 2001 From: Saul Moro Date: Thu, 24 Sep 2026 21:27:04 +0200 Subject: [PATCH 24/41] fix(stats): address CI review (#785) A session main split across scopes per event is credited once with every part it reported: for each scope its `dataHome` names, the shortest prefix of its events whose metrics reach its snapshot, and the owner takes the metrics of their union as reported when they exceed its own entry. Parts counted before any Stop carried the transcript's total are no longer sent again by the owner, and cumulative Stops are not credited twice. A Copilot session with an explicit ID is traced to where it started by Copilot's own session log, found by the session ID (TeamAI stores no path of it, #666): its session.start context names the directory. Compaction keeps a session whose tool process is still running, so a run that an exit from a dashboard before processExitAfter marked stopped keeps its start and its ID. --- docs/designs/data-directory-layout.md | 17 ++- src/__tests__/dashboard-collector.test.ts | 22 ++++ src/__tests__/dashboard-report-scope.test.ts | 63 +++++++++++ src/dashboard-collector.ts | 17 ++- src/stats.ts | 12 +- src/team-push.ts | 110 ++++++++++++++++--- 6 files changed, 216 insertions(+), 25 deletions(-) diff --git a/docs/designs/data-directory-layout.md b/docs/designs/data-directory-layout.md index 5f2da8f05..e2c3601f8 100644 --- a/docs/designs/data-directory-layout.md +++ b/docs/designs/data-directory-layout.md @@ -394,10 +394,19 @@ is found by its transcript: hooks record `transcriptPath` on UserPromptSubmit, Stop and SessionEnd (not SessionStart, whose path on a resume from another project names a file that never exists; never Copilot's), and a Claude transcript keeps its first `cwd` when resumed elsewhere, as a Codex rollout -keeps its `session_meta`. So a tool's own session with no owner is the scope's -that directory resolves to, when that scope's snapshots already hold it; else -it is decided as before (a fork under a new ID, a tool whose transcript records -no start, Copilot). A +keeps its `session_meta` and Copilot's own session log, found by the session ID +without storing its path, its `session.start` context. So a tool's own session +with no owner is the scope's that directory resolves to, when that scope's +snapshots already hold it; else it is decided as before (a fork under a new ID, +a tool whose transcript records no start). A session main split across scopes +per event, whose events are still in the log with each part's `dataHome`, is +credited once with every part reported: for each scope, the shortest prefix of +its events whose metrics reach its snapshot, and the owner takes the metrics of +their union as reported when they exceed its own entry, so parts counted before +any Stop carried the transcript's total are neither lost nor sent twice. +Compaction also keeps a session whose tool process is still running, so a run +an exit from a dashboard before `processExitAfter` marked stopped keeps its +start, and its ID. A fallback run is reported and snapshotted as `@`, which does not change when compaction drops earlier runs. A snapshot entry keyed by a bare fallback ID (written before) is the sum of the runs of that ID in the diff --git a/src/__tests__/dashboard-collector.test.ts b/src/__tests__/dashboard-collector.test.ts index bb6e09aef..09a1107ab 100644 --- a/src/__tests__/dashboard-collector.test.ts +++ b/src/__tests__/dashboard-collector.test.ts @@ -2122,6 +2122,28 @@ describe('compactEvents', () => { const content = fs.readFileSync(eventsPath, 'utf-8'); expect(content.trim().split('\n')).toHaveLength(1); }); + + it('keeps a session whose tool process is still running, though an exit marked it stopped', async () => { + // A dashboard from before processExitAfter may close a live run with an + // exit it observed for the run before; dropping that run's start would + // give its next activity a new run ID. + const eventsPath = path.join(tmpDir, '.teamai', 'dashboard', 'events.jsonl'); + fs.mkdirSync(path.dirname(eventsPath), { recursive: true }); + const old = new Date(Date.now() - 3_600_000).toISOString(); + const at = (sessionId: string, type: string, extra: Record = {}) => + JSON.stringify({ type, timestamp: old, sessionId, tool: 'copilot', ...extra }); + const lines = [ + at('pid-live', 'session_start', { monitorPid: process.pid }), at('pid-live', 'process_exit'), + at('pid-dead', 'session_start', { monitorPid: 2 ** 22 + 12345 }), at('pid-dead', 'process_exit'), + ...Array.from({ length: 5_000 }, (_, i) => at(`filler-${i}`, 'session_end')), + ]; + fs.writeFileSync(eventsPath, lines.join('\n') + '\n'); + + await compactEvents(eventsPath); + + const kept = fs.readFileSync(eventsPath, 'utf-8').split('\n').filter(Boolean).map((line) => JSON.parse(line).sessionId); + expect(kept).toEqual(['pid-live', 'pid-live']); + }); }); // ─── countInterventions (CodeBuddy index.json) ───────── diff --git a/src/__tests__/dashboard-report-scope.test.ts b/src/__tests__/dashboard-report-scope.test.ts index 6b8dcbb2a..462588768 100644 --- a/src/__tests__/dashboard-report-scope.test.ts +++ b/src/__tests__/dashboard-report-scope.test.ts @@ -622,6 +622,69 @@ describe('each scope keeps its own reported snapshot (#786)', () => { expect(await report(projectQ)).toBeNull(); }); + it('a Copilot session resumed elsewhere after compaction goes to the project its session log started in', async () => { + await setup(); + const { rootQ, projectQ } = await setupQ(); + const { rootW, projectW } = await setupWorkspaceProject(); + writeSharedSnapshots({ 'copilot-w': 1 }, new Date().toISOString().slice(0, 10), projectW); + // Copilot's own session log, found by the session ID: TeamAI stores no path of it (#666). + const log = path.join(tmp, 'home', '.copilot', 'session-state', 'copilot-w', 'events.jsonl'); + fs.mkdirSync(path.dirname(log), { recursive: true }); + fs.writeFileSync(log, JSON.stringify({ type: 'session.start', data: { context: { cwd: rootW } } }) + '\n'); + await session('copilot', { session_id: 'copilot-w', cwd: rootQ }); + + expect(await report(projectQ)).toBeNull(); + }); + + it('main split a session across projects before any Stop: the owner credits what each already reported', async () => { + const { root, project } = await setup(); + const { rootQ, projectQ } = await setupQ(); + // Main attributed each event to its scope by `dataHome` and reported each + // part as its own count: 3 prompts in P, then 2 in Q, no Stop yet. + const at = (s: number, type: string, cwd: string, dataHome: string) => JSON.stringify({ + type, timestamp: new Date(Date.now() - 600_000 + s * 1000).toISOString(), sessionId: 'split', tool: 'claude', cwd, dataHome, + }); + const p = getDataHome(project); + const q = getDataHome(projectQ); + const log = path.join(teamaiHome(), 'dashboard', 'events.jsonl'); + fs.mkdirSync(path.dirname(log), { recursive: true }); + fs.writeFileSync(log, [ + at(0, 'session_start', root, p), at(1, 'prompt_submit', root, p), at(2, 'prompt_submit', root, p), at(3, 'prompt_submit', root, p), + at(10, 'prompt_submit', rootQ, q), at(11, 'prompt_submit', rootQ, q), + ].join('\n') + '\n'); + const today = new Date().toISOString().slice(0, 10); + writeSharedSnapshots({ split: 3 }, today, project); + writeSharedSnapshots({ split: 2 }, today, projectQ); + + expect(await report(project)).toBeNull(); + expect(await report(projectQ)).toBeNull(); + }); + + it('main split a session whose parts each ended in a cumulative Stop: a new prompt is reported once', async () => { + const { root, project } = await setup(); + const { rootQ, projectQ } = await setupQ(); + const at = (s: number, type: string, cwd: string, dataHome: string, extra: Record = {}) => JSON.stringify({ + type, timestamp: new Date(Date.now() - 600_000 + s * 1000).toISOString(), sessionId: 'split', tool: 'claude', cwd, dataHome, ...extra, + }); + const p = getDataHome(project); + const q = getDataHome(projectQ); + const log = path.join(teamaiHome(), 'dashboard', 'events.jsonl'); + fs.mkdirSync(path.dirname(log), { recursive: true }); + fs.writeFileSync(log, [ + at(0, 'session_start', root, p), at(1, 'prompt_submit', root, p), at(2, 'stop', root, p, { prompts: 3 }), + at(10, 'prompt_submit', rootQ, q), at(11, 'stop', rootQ, q, { prompts: 5 }), + ].join('\n') + '\n'); + const today = new Date().toISOString().slice(0, 10); + // Each Stop carried the transcript's total, so Q's snapshot already spans P's part. + writeSharedSnapshots({ split: 3 }, today, project); + writeSharedSnapshots({ split: 5 }, today, projectQ); + expect(await report(project)).toBeNull(); + expect(await report(projectQ)).toBeNull(); + fs.appendFileSync(log, [at(20, 'prompt_submit', rootQ, q), at(21, 'stop', rootQ, q, { prompts: 6 })].join('\n') + '\n'); + + expect((await reportedPrompts(project)) + (await reportedPrompts(projectQ))).toBe(1); + }); + it('a transcript that started in a project whose snapshot lacks the session does not hand it there', async () => { await setup(); const { rootQ, projectQ } = await setupQ(); diff --git a/src/dashboard-collector.ts b/src/dashboard-collector.ts index b61fbafec..08131ffd5 100644 --- a/src/dashboard-collector.ts +++ b/src/dashboard-collector.ts @@ -7,7 +7,7 @@ import { deriveSessionId } from './utils/session-id.js'; import { resolveHookCwd } from './utils/hook-cwd.js'; import { ensureDir } from './utils/fs.js'; import { repoKeys, repoLabel } from './utils/repo-attribution.js'; -import { resolveMonitorPid } from './pid-monitor.js'; +import { isProcessAlive, resolveMonitorPid } from './pid-monitor.js'; import { normalizeToolName } from './utils/tool-names.js'; import { redactWithEnv } from './utils/redact.js'; import { @@ -784,7 +784,8 @@ async function waitForCopilotRunUsage( } /** Resolve Copilot's local event log without accepting path traversal via sessionId. */ -function resolveCopilotUsageTranscript( +/** Copilot's own session log for `sessionId`, derived from the ID alone (TeamAI stores no path of it, #666). */ +export function resolveCopilotUsageTranscript( sessionId: string, ): string | null { if (!COPILOT_SESSION_ID_RE.test(sessionId) || sessionId === '.' || sessionId === '..') return null; @@ -1926,7 +1927,10 @@ export function aggregateSessionInterventions( /** * Compact events.jsonl by keeping only events for active sessions. - * Active = not stopped and last activity within STALE_TIMEOUT. + * Active = not stopped and last activity within STALE_TIMEOUT, or its tool + * process still running: an exit a dashboard wrote before `processExitAfter` + * existed can mark a live run stopped, and dropping that run's start would give + * its next activity a new run ID (#785). * Called when file exceeds COMPACTION_THRESHOLD lines. */ export async function compactEvents(eventsPath?: string): Promise { @@ -1943,6 +1947,13 @@ export async function compactEvents(eventsPath?: string): Promise { const events = await readEventsRaw(filePath); const activeSessions = rebuildSessions(events); const activeIds = new Set(activeSessions.map(s => s.sessionId)); + const monitored = new Map(); + for (const e of events) { + if (e.type === 'session_start' && typeof e.monitorPid === 'number') monitored.set(e.sessionId, e.monitorPid); + } + for (const [sessionId, pid] of monitored) { + if (isProcessAlive(pid)) activeIds.add(sessionId); + } // Keep only events for active sessions const kept = events.filter(e => activeIds.has(e.sessionId)); diff --git a/src/stats.ts b/src/stats.ts index b27e9c69f..38025da5b 100644 --- a/src/stats.ts +++ b/src/stats.ts @@ -158,20 +158,24 @@ async function unreportedDashboardStats( ): Promise { const { adoptBareKeys, computeInterventionDelta, computePromptTokenDelta, readReportedInterventions, readReportedPromptTokens, - takeInterventions, takePromptTokens, + takeInterventions, takePromptTokens, creditSplitRuns, } = await import('./team-push.js'); // The scope's own snapshots, the ones its report compares against (#786). const currentInterventions = new Map( [...metrics].map(([sid, m]) => [sid, { interrupt: m.interrupt, toolReject: m.toolReject, correction: m.correction }]), ); // Prompt tokens first: they decide which runs of a bare ID were reported. - const promptTokens = adoptBareKeys( + const adoptedPromptTokens = adoptBareKeys( await readReportedPromptTokens(config), events, computePromptTokenDelta(metrics, {}).nextReported, takePromptTokens, ); - const interventions = adoptBareKeys( + const adoptedInterventions = adoptBareKeys( await readReportedInterventions(config), events, Object.fromEntries(currentInterventions), - takeInterventions((runId) => Object.hasOwn(promptTokens, runId)), + takeInterventions((runId) => Object.hasOwn(adoptedPromptTokens, runId)), ); + // As the report credits a session main split across scopes. + const { promptTokens, interventions } = await creditSplitRuns(events, { + interventions: adoptedInterventions, promptTokens: adoptedPromptTokens, daily: {}, + }); const interventionDelta = computeInterventionDelta(currentInterventions, interventions); const promptTokenDelta = computePromptTokenDelta(metrics, promptTokens); diff --git a/src/team-push.ts b/src/team-push.ts index e19a2a9a2..1de29ec45 100644 --- a/src/team-push.ts +++ b/src/team-push.ts @@ -3,7 +3,7 @@ import fs from 'node:fs'; import path from 'node:path'; import { readUsageEvents, truncateUsageAfterReport } from './usage-tracker.js'; import { aggregateUsage } from './stats.js'; -import { readEvents, aggregateSessionMetrics, dataHomeKey } from './dashboard-collector.js'; +import { readEvents, aggregateSessionMetrics, dataHomeKey, resolveCopilotUsageTranscript } from './dashboard-collector.js'; import { createGit, pushRepoDirectly, @@ -481,7 +481,12 @@ export async function filterEventsByScope( const transcripts = new Map(); events.forEach((e, i) => { const run = runOf[i]; - if (run !== undefined && typeof e.transcriptPath === 'string') transcripts.set(run, [...(transcripts.get(run) ?? []), e.transcriptPath]); + if (run === undefined) return; + // Copilot's session log is found by its ID; the event holds no path (#666). + const transcript = typeof e.transcriptPath === 'string' ? e.transcriptPath + : e.tool === 'copilot' ? resolveCopilotUsageTranscript(e.sessionId) : null; + const known = transcripts.get(run) ?? []; + if (transcript && !known.includes(transcript)) transcripts.set(run, [...known, transcript]); }); const owned = await Promise.all(deciding.map(async (i, run) => { // Recorded by the scope that first reported it, so a session resumed @@ -530,8 +535,9 @@ async function transcriptOwner( /** * The directory a transcript's session started in: the first `cwd` a Claude * transcript records (a resume from another project appends to the same file), - * or a Codex rollout's `session_meta`. Reads a bounded head of the file; the - * format is the tool's own, so anything else is undefined. + * a Codex rollout's `session_meta`, or a Copilot session log's `session.start`. + * Reads a bounded head of the file; the format is the tool's own, so anything + * else is undefined. */ async function transcriptOrigin(transcript: string): Promise { let head: string; @@ -560,6 +566,11 @@ async function transcriptOrigin(transcript: string): Promise && typeof entry.payload === 'object' && 'cwd' in entry.payload && typeof entry.payload.cwd === 'string') { return entry.payload.cwd; } + if ('type' in entry && entry.type === 'session.start' && 'data' in entry && entry.data && typeof entry.data === 'object' + && 'context' in entry.data && entry.data.context && typeof entry.data.context === 'object' + && 'cwd' in entry.data.context && typeof entry.data.context.cwd === 'string') { + return entry.data.context.cwd; + } } return undefined; } @@ -824,6 +835,64 @@ async function recordSessionOwners(sessionIds: Iterable, key: string): P } } +/** + * A session main split across scopes, crediting every part it reported. Main + * (#795) gave each event to the scope its `dataHome` names and reported each + * part against that scope's own snapshot, so a scope's entry may hold only its + * part (prompts counted before any Stop carried the transcript's total). The + * scope that owns the whole run now takes as reported the metrics of the union + * of those parts: for each scope, the shortest prefix of its events whose own + * metrics reach its snapshot. Only an entry that union exceeds is replaced, so a + * run already reported past it is left alone; runs whose events name at most + * one data home are untouched. + */ +export async function creditSplitRuns( + events: DashboardEvent[], + reported: { interventions: ReportedInterventions; promptTokens: ReportedPromptTokens; daily: ReportedDailySessions }, +): Promise<{ + interventions: ReportedInterventions; promptTokens: ReportedPromptTokens; daily: ReportedDailySessions; changed: boolean; +}> { + const byRun = new Map(); + for (const e of events) byRun.set(e.sessionId, [...(byRun.get(e.sessionId) ?? []), e]); + const result = { + interventions: { ...reported.interventions }, promptTokens: { ...reported.promptTokens }, daily: { ...reported.daily }, + changed: false, + }; + for (const [runId, runEvents] of byRun) { + const homes = new Set(runEvents.flatMap((e) => (typeof e.dataHome === 'string' ? [e.dataHome] : []))); + if (homes.size < 2) continue; + // Main keyed its snapshots by the session ID, before runs had their own. + const id = runId.startsWith('pid-') ? runId.slice(0, runId.lastIndexOf('@')) : runId; + const counted = new Set(); + for (const home of homes) { + const snapshot = await readJson>(snapshotPathIn(home, 'prompt-tokens')); + const target = reportedSize(snapshot?.[id]); + if (!snapshot || !Object.hasOwn(snapshot, id)) continue; + const part = runEvents.filter((e) => e.dataHome === home); + for (let k = 1; k <= part.length; k++) { + const size = aggregateSessionMetrics(part.slice(0, k)).get(runId); + const tokens = size ? size.tokens.input + size.tokens.output + size.tokens.cacheRead + size.tokens.cacheCreation : 0; + if (k === part.length || (size && size.prompts >= target.prompts && tokens >= target.tokens)) { + for (const e of part.slice(0, k)) counted.add(e); + break; + } + } + } + const union = runEvents.filter((e) => counted.has(e)); + if (union.length === 0) continue; + const metrics = aggregateSessionMetrics(union); + const promptTokens = computePromptTokenDelta(metrics, {}).nextReported[runId]; + if (!promptTokens || reportedSize(result.promptTokens[runId]).prompts >= promptTokens.prompts) continue; + result.promptTokens[runId] = promptTokens; + const interventions = interventionCounts(metrics).get(runId); + if (interventions) result.interventions[runId] = interventions; + const daily = computeDailyStatsDelta(aggregateDailySessions(union), {}).nextReported[runId]; + if (daily) result.daily[runId] = daily; + result.changed = true; + } + return result; +} + /** * A run's share of what is left of a bare snapshot entry, or undefined when * nothing is left: that run and every later one of its ID were not reported. @@ -946,27 +1015,40 @@ export async function reportUsageToTeam( return adopted; }; // Prompt tokens first: they decide which runs of a bare ID were reported. - const reportedPromptTokens = await adopt( + const adoptedPromptTokens = await adopt( readReportedPromptTokens, writeReportedPromptTokens, computePromptTokenDelta(metrics, {}).nextReported, takePromptTokens, ); - const covered = (runId: string) => Object.hasOwn(reportedPromptTokens, runId); + const covered = (runId: string) => Object.hasOwn(adoptedPromptTokens, runId); + const adoptedInterventions = await adopt( + readReportedInterventions, writeReportedInterventions, Object.fromEntries(currentInterventions), + takeInterventions(covered), + ); + const adoptedDailySessions = await adopt( + readReportedDailySessions, writeReportedDailySessions, computeDailyStatsDelta(currentDaily, {}).nextReported, + takeDaily(covered), + ); + // A session main split across scopes is credited with every part reported, + // once, before any delta: written out now, like a retired bare entry. + const credited = await creditSplitRuns(dashboardEvents, { + interventions: adoptedInterventions, promptTokens: adoptedPromptTokens, daily: adoptedDailySessions, + }); + if (credited.changed) { + await writeReportedInterventions(credited.interventions, reportsConfig); + await writeReportedPromptTokens(credited.promptTokens, reportsConfig); + await writeReportedDailySessions(credited.daily, reportsConfig); + } + const reportedPromptTokens = credited.promptTokens; + const reportedInterventions = credited.interventions; + const reportedDailySessions = credited.daily; const { delta: promptTokenDelta, nextReported: nextReportedPromptTokens } = computePromptTokenDelta( metrics, reportedPromptTokens, ); - const reportedInterventions = await adopt( - readReportedInterventions, writeReportedInterventions, Object.fromEntries(currentInterventions), - takeInterventions(covered), - ); const { delta: interventionDelta, nextReported } = computeInterventionDelta( currentInterventions, reportedInterventions, ); - const reportedDailySessions = await adopt( - readReportedDailySessions, writeReportedDailySessions, computeDailyStatsDelta(currentDaily, {}).nextReported, - takeDaily(covered), - ); const { delta: dailyDelta, nextReported: nextReportedDailySessions } = computeDailyStatsDelta( currentDaily, reportedDailySessions, From 442f4c289a2b402023dbf1da60fbdc89b5a6add3 Mon Sep 17 00:00:00 2001 From: Saul Moro Date: Thu, 24 Sep 2026 22:46:03 +0200 Subject: [PATCH 25/41] fix(stats): address CI review (#785) Owner migration takes a scope's entry as evidence only when its snapshots show it reported the ID: the shared snapshots (interventions included) hold none of it, or the scope is past their total. Main copied the shared file into every scope it ran in, so a copy, even the only one, names no owner, and the per-report recording follows the same rule. A session main split across scopes whose events are gone is credited from the parts' snapshots: a part whose daily entry shows a Stop holds the transcript's cumulative total, so the greatest counts once; a part with no Stop counted its own prompts, which add; intervention counts add, tokens take the greatest. The credit rides on the owner's line in session-owners.jsonl (numbers only) and is applied once as its baseline. --- docs/designs/data-directory-layout.md | 18 +- src/__tests__/dashboard-report-scope.test.ts | 48 +++++ src/session-trends.ts | 32 +++ src/team-push.ts | 212 +++++++++++++++---- 4 files changed, 266 insertions(+), 44 deletions(-) diff --git a/docs/designs/data-directory-layout.md b/docs/designs/data-directory-layout.md index e2c3601f8..6ee0bc0e0 100644 --- a/docs/designs/data-directory-layout.md +++ b/docs/designs/data-directory-layout.md @@ -379,12 +379,18 @@ recorded there is that scope's wherever it is resumed later, whatever the log still holds, so another scope never reports its transcript again; the first line for an ID wins, and the file grows by one line per such session, like the snapshots. An earlier release kept only per-scope snapshots, so the file is -first written from them: a tool's own ID in any of a scope's snapshots is the -scope's that holds its greatest total (prompts, then tokens), since a session -that release split per event holds only part of it elsewhere, and a scope may -have reported past the shared total it was seeded with. A tie names no owner: -that release copied the shared file into every scope, so equal totals show only -the copy, and each scope already holds that baseline. The scopes read are the +first written from them: a tool's own ID is the scope's whose snapshots show it +reported it with the greatest total (prompts, then tokens). They show it when the +shared snapshots (all three) hold none of it, or the scope is past their total: +that release copied the shared file into every scope it ran in, so a copy, even +the only one, shows nothing, and a tie names no owner. When several scopes +reported it (a session that release split per event), the owner's line also +carries the credit of their parts, applied once as its baseline: a part whose +daily entry shows it ended in a Stop holds the transcript's cumulative total, so +the greatest such part counts once, while a part with no Stop counted its own +prompts, which add; intervention counts add, tokens take the greatest. A part +with no Stop that came before another's Stop is then credited twice, which +undercounts once but never sends a prompt again. The scopes read are the user scope, every partition, and a project whose data home is in its workspace that a session still in the log leads to; each report also records the IDs of its own snapshots that have no owner yet and show it reported them (absent from diff --git a/src/__tests__/dashboard-report-scope.test.ts b/src/__tests__/dashboard-report-scope.test.ts index 462588768..aa7c2d54c 100644 --- a/src/__tests__/dashboard-report-scope.test.ts +++ b/src/__tests__/dashboard-report-scope.test.ts @@ -552,6 +552,54 @@ describe('each scope keeps its own reported snapshot (#786)', () => { expect(await reportedPrompts(project)).toBe(1); }); + it('a sole copy of a shared entry names no owner: a resume in the scope that reported it counts there', async () => { + const { user, project } = await setup(); + const elsewhere = path.join(tmp, 'elsewhere'); + fs.mkdirSync(elsewhere); + const today = new Date().toISOString().slice(0, 10); + // Only P pulled on main, so only P's snapshot holds a copy of the user-scope session. + writeSharedSnapshots({ 'user-session': 1 }, today); + writeSharedSnapshots({ 'user-session': 1 }, today, project); + await resume('user-session', elsewhere, 2); + + expect(await report(project)).toBeNull(); + expect(await reportedPrompts(user)).toBe(1); + }); + + it('a copy the shared intervention snapshot alone holds does not let a report claim it', async () => { + const { root, user, project } = await setup(); + const elsewhere = path.join(tmp, 'elsewhere'); + fs.mkdirSync(elsewhere); + const today = new Date().toISOString().slice(0, 10); + // A session reported with no prompts: only intervention entries, copied into P. + writeSharedSnapshots({ quiet: 0 }, today); + writeSharedSnapshots({ quiet: 0 }, today, project); + for (const dir of [path.join(teamaiHome(), 'dashboard'), path.join(getDataHome(project), 'dashboard')]) { + for (const name of ['prompt-tokens', 'daily-sessions']) fs.rmSync(path.join(dir, `reported-${name}.json`)); + } + // P reports another session first, and records owners from its snapshots. + await session('claude', { session_id: 'p-other', cwd: root }); + const reportedByP = await report(project); + await resume('quiet', elsewhere, 1); + + expect(await report(project)).toEqual(reportedByP); + expect(await reportedPrompts(user)).toBe(1); + }); + + it('main split a session before any Stop and compaction dropped it: the owner credits both parts', async () => { + const { root, project } = await setup(); + const { projectQ } = await setupQ(); + const today = new Date().toISOString().slice(0, 10); + // 3 prompts reported in P and 2 in Q, as submit counts: no Stop, so no daily entry. + writeSharedSnapshots({ split: 3 }, today, project); + writeSharedSnapshots({ split: 2 }, today, projectQ); + for (const config of [project, projectQ]) fs.rmSync(path.join(getDataHome(config), 'dashboard', 'reported-daily-sessions.json')); + // Resumed in P; its cumulative Stop carries 6. + await resume('split', root, 6); + + expect((await reportedPrompts(project)) + (await reportedPrompts(projectQ))).toBe(1); + }); + it('a project that reported a session past the shared snapshot\'s total owns it', async () => { const { project } = await setup(); const { rootQ, projectQ } = await setupQ(); diff --git a/src/session-trends.ts b/src/session-trends.ts index c2d2f8d6f..87da24fda 100644 --- a/src/session-trends.ts +++ b/src/session-trends.ts @@ -97,6 +97,38 @@ function positiveDelta(current: number, previous: number | undefined): number { return Math.max(0, current - (previous ?? 0)); } +/** A daily snapshot entry read from a file, or undefined when it is not one. */ +export function parseDailySnapshot(value: unknown): DailySessionSnapshot | undefined { + if (!value || typeof value !== 'object') return undefined; + if (!('date' in value) || typeof value.date !== 'string' || !('prompts' in value) || typeof value.prompts !== 'number' + || !('durationMs' in value) || typeof value.durationMs !== 'number') return undefined; + const requestDaily: Record = {}; + if ('requestDaily' in value && value.requestDaily && typeof value.requestDaily === 'object') { + for (const [date, request] of Object.entries(value.requestDaily)) { + if (!request || typeof request !== 'object') continue; + const num = (field: string) => { + const n: unknown = Object.entries(request).find(([k]) => k === field)?.[1]; + return typeof n === 'number' ? n : 0; + }; + const version: unknown = Object.entries(request).find(([k]) => k === 'priceVersion')?.[1]; + requestDaily[date] = { + pricedRequests: num('pricedRequests'), costMicros: num('costMicros'), cacheReadTokens: num('cacheReadTokens'), + cacheEligibleInputTokens: num('cacheEligibleInputTokens'), priceVersion: typeof version === 'string' ? version : '', + }; + } + } + return { + date: value.date, prompts: value.prompts, durationMs: value.durationMs, + succeeded: 'succeeded' in value && value.succeeded === 1 ? 1 : 0, + corrected: 'corrected' in value && value.corrected === 1 ? 1 : 0, + requestDaily, + ...('sessionCacheReadTokens' in value && typeof value.sessionCacheReadTokens === 'number' + ? { sessionCacheReadTokens: value.sessionCacheReadTokens } : {}), + ...('sessionCacheEligibleTokens' in value && typeof value.sessionCacheEligibleTokens === 'number' + ? { sessionCacheEligibleTokens: value.sessionCacheEligibleTokens } : {}), + }; +} + /** A reported snapshot's request costs per day, from its session-level fields if it predates them. */ function reportedRequestDaily(previous: DailySessionSnapshot | undefined): Record { return previous?.requestDaily ?? ( diff --git a/src/team-push.ts b/src/team-push.ts index 1de29ec45..b9de945c5 100644 --- a/src/team-push.ts +++ b/src/team-push.ts @@ -23,6 +23,7 @@ import { computeDailyStatsDelta, mergeDailyStats, takeDailySession, + parseDailySnapshot, type DailySessionSnapshot, type ReportedDailySessions, } from './session-trends.js'; @@ -740,32 +741,171 @@ function reportedSize(entry: unknown): { prompts: number; tokens: number } { return { prompts, tokens }; } -/** The owners the per-scope snapshots of an earlier release imply, as the file's first lines. */ +/** A scope's three snapshots (the shared ones for no data home), as a file holds them. */ +interface ScopeSnapshots { + interventions: Record | null; + promptTokens: Record | null; + daily: Record | null; +} + +async function readSnapshotsIn(dataHome: string | undefined): Promise { + const read = (name: ReportedSnapshotName) => + readJson>(dataHome === undefined ? sharedSnapshotPath(name) : snapshotPathIn(dataHome, name)); + const [interventions, promptTokens, daily] = await Promise.all(REPORTED_SNAPSHOTS.map(read)); + return { interventions, promptTokens, daily }; +} + +/** An intervention entry read from a file, zero for what it does not hold. */ +function interventionsEntry(value: unknown): ReportedInterventions[string] { + const count = (field: 'interrupt' | 'toolReject' | 'correction') => + (value && typeof value === 'object' && field in value && typeof Object.entries(value).find(([k]) => k === field)?.[1] === 'number' + ? Number(Object.entries(value).find(([k]) => k === field)?.[1]) : 0); + return { interrupt: count('interrupt'), toolReject: count('toolReject'), correction: count('correction') }; +} + +/** A prompt-token entry read from a file, zero for what it does not hold. */ +function promptTokensEntry(value: unknown): ReportedPromptTokens[string] { + const tokens = emptyTokenUsage(); + if (value && typeof value === 'object' && 'tokens' in value && value.tokens && typeof value.tokens === 'object') { + for (const [field, n] of Object.entries(value.tokens)) { + if (typeof n !== 'number') continue; + if (field === 'input' || field === 'output' || field === 'cacheRead' || field === 'cacheCreation') tokens[field] = n; + } + } + return { prompts: reportedSize(value).prompts, tokens }; +} + +/** What a scope's snapshots hold of `id`: prompts, tokens and intervention counts. */ +function heldIn(snapshots: ScopeSnapshots, id: string): { prompts: number; tokens: number; interventions: number } { + const fromTokens = reportedSize(snapshots.promptTokens?.[id]); + const iv = interventionsEntry(snapshots.interventions?.[id]); + return { + prompts: Math.max(fromTokens.prompts, reportedSize(snapshots.daily?.[id]).prompts), + tokens: fromTokens.tokens, + interventions: iv.interrupt + iv.toolReject + iv.correction, + }; +} + +/** + * Whether a scope's snapshots show it reported `id`: the shared snapshots hold + * none of it, or the scope is past their total. An earlier release copied the + * shared snapshots into every scope it ran in, so a copy shows nothing. + */ +function showsReported(own: ScopeSnapshots, shared: ScopeSnapshots, id: string): boolean { + const inShared = [shared.interventions, shared.promptTokens, shared.daily].some((snapshot) => + !!snapshot && typeof snapshot === 'object' && Object.hasOwn(snapshot, id)); + if (!inShared) return true; + const a = heldIn(own, id); + const b = heldIn(shared, id); + return a.prompts !== b.prompts ? a.prompts > b.prompts + : a.tokens !== b.tokens ? a.tokens > b.tokens : a.interventions > b.interventions; +} + +/** What several scopes had reported of one session, credited to its owner (numbers only). */ +interface OwnerCredit { + interventions: ReportedInterventions[string]; + promptTokens: ReportedPromptTokens[string]; + daily?: DailySessionSnapshot; +} + +/** + * The credit for a session an earlier release split across scopes per event, + * from their snapshots alone (its events are gone). A part whose daily entry + * shows it ended in a Stop carries the transcript's cumulative total, so the + * greatest such part counts once; a part with no Stop counted its own prompts, + * so those add. Intervention counts, per event, add; tokens, from Stops, take + * the greatest. A part without a Stop that came before another's Stop is + * credited twice: that undercounts, once, but never sends a prompt again. + */ +function creditOf(parts: ScopeSnapshots[], id: string): OwnerCredit { + const stops = parts.filter((part) => parseDailySnapshot(part.daily?.[id]) !== undefined); + const loose = parts.filter((part) => !stops.includes(part)); + const prompts = Math.max(0, ...stops.map((part) => heldIn(part, id).prompts)) + + loose.reduce((sum, part) => sum + heldIn(part, id).prompts, 0); + const tokens = emptyTokenUsage(); + const interventions = { interrupt: 0, toolReject: 0, correction: 0 }; + for (const part of parts) { + const entry = promptTokensEntry(part.promptTokens?.[id]); + for (const field of ['input', 'output', 'cacheRead', 'cacheCreation'] as const) { + tokens[field] = Math.max(tokens[field], entry.tokens[field]); + } + const iv = interventionsEntry(part.interventions?.[id]); + interventions.interrupt += iv.interrupt; + interventions.toolReject += iv.toolReject; + interventions.correction += iv.correction; + } + const days = stops.flatMap((part) => { + const day = parseDailySnapshot(part.daily?.[id]); + return day ? [day] : []; + }); + const latest = days.reduce((a, b) => (!a || b.prompts > a.prompts ? b : a), undefined); + return { + interventions, + promptTokens: { prompts, tokens }, + ...(latest ? { daily: { ...latest, prompts, durationMs: days.reduce((sum, day) => sum + day.durationMs, 0) } } : {}), + }; +} + +/** + * The owners the per-scope snapshots of an earlier release imply, as the file's + * first lines: for each tool's own ID, the scope whose snapshots show it reported + * it with the greatest total, and, when several did, the credit of their parts. + * A tie names no owner. + */ async function ownersFromSnapshots(): Promise { - const best = new Map(); + const shared = await readSnapshotsIn(undefined); + const parts = new Map>(); for (const dataHome of await knownDataHomes()) { - const [interventions, promptTokens, daily] = await Promise.all( - REPORTED_SNAPSHOTS.map((name) => readJson>(snapshotPathIn(dataHome, name))), - ); - const ids = new Set([interventions, promptTokens, daily].flatMap((snapshot) => + const snapshots = await readSnapshotsIn(dataHome); + const ids = new Set([snapshots.interventions, snapshots.promptTokens, snapshots.daily].flatMap((snapshot) => (snapshot && typeof snapshot === 'object' ? Object.keys(snapshot) : []))); const key = await dataHomeKey(dataHome); for (const sessionId of ids) { - if (sessionId.startsWith('pid-')) continue; - const fromTokens = reportedSize(promptTokens?.[sessionId]); - const size = { prompts: Math.max(fromTokens.prompts, reportedSize(daily?.[sessionId]).prompts), tokens: fromTokens.tokens }; - const held = best.get(sessionId); - if (!held || size.prompts > held.prompts || (size.prompts === held.prompts && size.tokens > held.tokens)) { - best.set(sessionId, { key, ...size, tied: false }); - } else if (size.prompts === held.prompts && size.tokens === held.tokens && key !== held.key) { - held.tied = true; - } + if (sessionId.startsWith('pid-') || !showsReported(snapshots, shared, sessionId)) continue; + parts.set(sessionId, [...(parts.get(sessionId) ?? []), { key, snapshots }]); } } - // A tie is a copy of one shared entry that release made in every scope: it - // names no owner, and each scope already holds that baseline. - return [...best].flatMap(([sessionId, { key, tied }]) => - (tied ? [] : [`${JSON.stringify({ sessionId, dataHomeKey: key })}\n`])).join(''); + const lines: string[] = []; + for (const [sessionId, held] of parts) { + const size = (part: { snapshots: ScopeSnapshots }) => heldIn(part.snapshots, sessionId); + const best = held.reduce((a, b) => { + const x = size(a); + const y = size(b); + return y.prompts > x.prompts || (y.prompts === x.prompts && y.tokens > x.tokens) ? b : a; + }); + const tied = held.some((part) => part.key !== best.key + && size(part).prompts === size(best).prompts && size(part).tokens === size(best).tokens); + if (tied) continue; + const credit = held.length > 1 ? creditOf(held.map((part) => part.snapshots), sessionId) : undefined; + lines.push(JSON.stringify({ sessionId, dataHomeKey: best.key, ...(credit ? { credit } : {}) })); + } + return lines.map((line) => `${line}\n`).join(''); +} + +/** The credits the file's first line for each ID carries (see {@link creditOf}). */ +async function readOwnerCredits(): Promise> { + const credits = new Map(); + const seen = new Set(); + for (const line of ((await readFileSafe(sessionOwnersPath())) ?? '').split('\n')) { + let parsed: unknown; + try { + parsed = JSON.parse(line); + } catch { + continue; + } + if (!parsed || typeof parsed !== 'object' || !('sessionId' in parsed) || typeof parsed.sessionId !== 'string') continue; + if (seen.has(parsed.sessionId)) continue; + seen.add(parsed.sessionId); + if (!('credit' in parsed) || !parsed.credit || typeof parsed.credit !== 'object') continue; + const credit = parsed.credit; + const daily = 'daily' in credit ? parseDailySnapshot(credit.daily) : undefined; + credits.set(parsed.sessionId, { + interventions: interventionsEntry('interventions' in credit ? credit.interventions : undefined), + promptTokens: promptTokensEntry('promptTokens' in credit ? credit.promptTokens : undefined), + ...(daily ? { daily } : {}), + }); + } + return credits; } async function readSessionOwners(): Promise> { @@ -794,31 +934,16 @@ async function readSessionOwners(): Promise> { return owners; } -/** - * The IDs of a scope's snapshots that show it reported them: absent from the - * shared snapshot, or past its total there. A copy of a shared entry, which - * an earlier release made in every scope, shows nothing. - */ +/** The IDs of a scope's snapshots that show it reported them (see {@link showsReported}). */ async function reportedBeyondShared( promptTokens: Record, interventions: Record, daily: Record, ): Promise { - const [sharedTokens, sharedDaily] = await Promise.all([ - readJson>(sharedSnapshotPath('prompt-tokens')), - readJson>(sharedSnapshotPath('daily-sessions')), - ]); - const size = (tokens: Record | null, days: Record | null, id: string) => { - const fromTokens = reportedSize(tokens?.[id]); - return { prompts: Math.max(fromTokens.prompts, reportedSize(days?.[id]).prompts), tokens: fromTokens.tokens }; - }; + const shared = await readSnapshotsIn(undefined); + const own: ScopeSnapshots = { interventions, promptTokens, daily }; const ids = new Set([...Object.keys(promptTokens), ...Object.keys(interventions), ...Object.keys(daily)]); - return [...ids].filter((id) => { - if (!Object.hasOwn(sharedTokens ?? {}, id) && !Object.hasOwn(sharedDaily ?? {}, id)) return true; - const own = size(promptTokens, daily, id); - const shared = size(sharedTokens, sharedDaily, id); - return own.prompts > shared.prompts || (own.prompts === shared.prompts && own.tokens > shared.tokens); - }); + return [...ids].filter((id) => showsReported(own, shared, id)); } /** Records `key` as the owner of the tool-own session IDs among `sessionIds` that have none yet. */ @@ -858,8 +983,19 @@ export async function creditSplitRuns( interventions: { ...reported.interventions }, promptTokens: { ...reported.promptTokens }, daily: { ...reported.daily }, changed: false, }; + const credits = await readOwnerCredits(); for (const [runId, runEvents] of byRun) { const homes = new Set(runEvents.flatMap((e) => (typeof e.dataHome === 'string' ? [e.dataHome] : []))); + // Its parts' events are gone: the credit the owners file seeded from their snapshots. + const credit = homes.size < 2 ? credits.get(runId) : undefined; + if (credit) { + if (reportedSize(result.promptTokens[runId]).prompts >= credit.promptTokens.prompts) continue; + result.promptTokens[runId] = credit.promptTokens; + result.interventions[runId] = credit.interventions; + if (credit.daily) result.daily[runId] = credit.daily; + result.changed = true; + continue; + } if (homes.size < 2) continue; // Main keyed its snapshots by the session ID, before runs had their own. const id = runId.startsWith('pid-') ? runId.slice(0, runId.lastIndexOf('@')) : runId; From 6930aba09ed076b1b2d9be470390fbca7e78df4e Mon Sep 17 00:00:00 2001 From: Saul Moro Date: Fri, 25 Sep 2026 00:03:28 +0200 Subject: [PATCH 26/41] fix(stats): keep a tool's own sessions in the first snapshot, parse legacy entries (#785) Seeding a scope's snapshot from the shared one kept only the runs still in the log, so a session reported before #795 and compacted before the scope's first pull was sent again in full when resumed. Only fallback entries need that filter, against a reused PID; a tool's own session ID is one session, so its entry is copied whole, as main did. Splitting a bare entry across runs read the snapshot entry as typed, and one without `tokens` (hand-edited or truncated) threw and skipped the whole report; the prompt-token and intervention shares now parse it as the owner-migration path already does. --- docs/designs/data-directory-layout.md | 6 +++-- src/__tests__/dashboard-report-scope.test.ts | 24 ++++++++++++++++++++ src/team-push.ts | 18 ++++++++++----- 3 files changed, 40 insertions(+), 8 deletions(-) diff --git a/docs/designs/data-directory-layout.md b/docs/designs/data-directory-layout.md index 6ee0bc0e0..4ba308be3 100644 --- a/docs/designs/data-directory-layout.md +++ b/docs/designs/data-directory-layout.md @@ -448,8 +448,10 @@ runs of an ID, so the runs of the whole log consume it in log order, whichever scope each belongs to, and the seed keeps the shares of the scope's own runs, under their run IDs; none goes to a run recorded with a `dataHome` path: that release already kept per-scope snapshots, so a -shared entry under its ID is another scope's. An unmatched entry is dropped, so a -later reuse of the ID cannot inherit it. The shared file is no longer +shared entry under its ID is another scope's. An unmatched fallback entry is +dropped, so a later reuse of the PID cannot inherit it; a tool's own session ID +is copied whole, as before, so a session resumed after compaction dropped its +events is not sent again. The shared file is no longer written, except by an earlier release after a rollback, so every scope seeds from what the machine had reported by then, never from another scope's later report. The seed holds a session's whole total, so a session still running at the diff --git a/src/__tests__/dashboard-report-scope.test.ts b/src/__tests__/dashboard-report-scope.test.ts index aa7c2d54c..ae1fa9354 100644 --- a/src/__tests__/dashboard-report-scope.test.ts +++ b/src/__tests__/dashboard-report-scope.test.ts @@ -914,6 +914,30 @@ describe('each scope keeps its own reported snapshot (#786)', () => { expect(await reportedSessions(project)).toBe(1); }); + it('a session the shared snapshot reported is not sent again after compaction dropped its events', async () => { + const { root, project } = await setup(); + // Reported before #795 into the shared snapshot and compacted; P's first + // pull seeds its own snapshot; then it is resumed in P, its Stop carrying 2. + writeSharedSnapshots({ resumed: 1 }, new Date().toISOString().slice(0, 10)); + expect(await report(project)).toBeNull(); + await resume('resumed', root, 2); + + expect(await reportedPrompts(project)).toBe(1); + }); + + it('a snapshot entry an earlier release wrote without tokens does not stop the report', async () => { + const { root, project } = await setup(); + const pid = `pid-${process.ppid}`; + await asEarlierRelease(() => session('copilot', { cwd: root }), getDataHome(project)); + // Hand-edited or truncated: no `tokens`, under the bare fallback ID. + const dir = path.join(getDataHome(project), 'dashboard'); + fs.mkdirSync(dir, { recursive: true }); + fs.writeFileSync(path.join(dir, 'reported-prompt-tokens.json'), JSON.stringify({ [pid]: { prompts: 1 } })); + + expect(await report(project)).not.toBeNull(); + expect(await reportedPrompts(project)).toBe(0); + }); + it('a scope first seeded after a rollback skips what the earlier release reported', async () => { const { root, project } = await setup(); await asEarlierRelease(() => session('claude', { session_id: 'rollback-p', cwd: root })); diff --git a/src/team-push.ts b/src/team-push.ts index b9de945c5..8f235ec04 100644 --- a/src/team-push.ts +++ b/src/team-push.ts @@ -196,11 +196,14 @@ async function readSnapshot( const events = await filterEventsByScope(logged, config); const { current, take } = await split(all); const adopted = adoptBareKeys(shared ?? {}, all, current, take, 'shared'); - // Copy only resolved run IDs. An unmatched bare entry cannot be allowed to - // attach to a future reuse after the source of the snapshot has been lost. + // A tool's own session ID is one session, so its entry is copied whole, as + // before: resumed after compaction dropped its events, it is not sent again. + // A fallback entry is copied only as a run of this scope: an unmatched bare + // one cannot be allowed to attach to a future reuse of its PID. + const ownRuns = new Set(events.map((e) => e.sessionId)); const seed: Record = {}; - for (const { sessionId } of events) { - if (Object.hasOwn(adopted, sessionId)) seed[sessionId] = adopted[sessionId]; + for (const [id, entry] of Object.entries(adopted)) { + if (!id.startsWith('pid-') || ownRuns.has(id)) seed[id] = entry; } try { await writeJson(own, seed); @@ -241,7 +244,9 @@ function upTo(own: number, left: number): number { * A run's prompt and token share of a bare entry. This snapshot decides which * runs of the ID that release reported: the ones its prompts and tokens reach. */ -export const takePromptTokens: TakeReported = (run, left) => { +export const takePromptTokens: TakeReported = (run, entry) => { + // Read from a snapshot file an earlier release wrote, so parsed, not trusted. + const left = promptTokensEntry(entry); if (left.prompts <= 0 && !hasPromptTokenDelta({ prompts: 0, tokens: left.tokens })) return undefined; const taken = { prompts: upTo(run.prompts, left.prompts), @@ -260,8 +265,9 @@ export const takePromptTokens: TakeReported = (run * snapshot's: these counts are mostly zero, so running out says nothing. */ export function takeInterventions(covered: (runId: string) => boolean): TakeReported { - return (run, left, runId) => { + return (run, entry, runId) => { if (!covered(runId)) return undefined; + const left = interventionsEntry(entry); const taken = { interrupt: upTo(run.interrupt, left.interrupt), toolReject: upTo(run.toolReject, left.toolReject), From 9d0b130f7024d55c0db60eb2cc44366d6cb58c76 Mon Sep 17 00:00:00 2001 From: Saul Moro Date: Fri, 25 Sep 2026 00:26:27 +0200 Subject: [PATCH 27/41] fix(stats): report a resumed Codex rollout after compaction dropped the earlier one (#785) A Codex build that writes a new rollout per resume restarts its transcript counters, and the session summed only the rollouts still in the log. Once compaction dropped rollout A, a resumed rollout B with smaller counters was compared against A's reported total and reported nothing until it passed it; routing the session back to the scope it started in made that loss reach the resume in another scope too. The prompt-token snapshot now keeps each rollout's reported prompts and tokens under a hash of its path (no path stored), and a rollout that is gone keeps its reported totals in the session's sum, so B is reported in full. A session's prompts also sum its rollouts' Stop counts, which restart per rollout like the tokens. An entry from before is compared as a whole once, then kept per rollout. --- docs/designs/data-directory-layout.md | 6 +++ .../conversation-token-metrics.test.ts | 29 +++++++++++ src/dashboard-collector.ts | 14 +++++- src/team-push.ts | 49 +++++++++++++++++-- src/types.ts | 5 ++ 5 files changed, 98 insertions(+), 5 deletions(-) diff --git a/docs/designs/data-directory-layout.md b/docs/designs/data-directory-layout.md index 4ba308be3..57be9b208 100644 --- a/docs/designs/data-directory-layout.md +++ b/docs/designs/data-directory-layout.md @@ -410,6 +410,12 @@ credited once with every part reported: for each scope, the shortest prefix of its events whose metrics reach its snapshot, and the owner takes the metrics of their union as reported when they exceed its own entry, so parts counted before any Stop carried the transcript's total are neither lost nor sent twice. +A transcript-scoped session (Codex) is kept per rollout: a rollout's counters +restart, so its prompt-token entry holds each rollout's reported prompts and +tokens under a hash of the rollout's path, and a rollout compaction has dropped +keeps its reported totals in the session's sum; a later rollout is then reported +in full. An entry written before rollouts were kept is compared as a whole once, +then kept per rollout. Compaction also keeps a session whose tool process is still running, so a run an exit from a dashboard before `processExitAfter` marked stopped keeps its start, and its ID. A diff --git a/src/__tests__/conversation-token-metrics.test.ts b/src/__tests__/conversation-token-metrics.test.ts index 0f34402c8..e59487e10 100644 --- a/src/__tests__/conversation-token-metrics.test.ts +++ b/src/__tests__/conversation-token-metrics.test.ts @@ -25,6 +25,8 @@ afterEach(() => { fs.rmSync(tmpDir, { recursive: true, force: true }); }); +const emptyTokens = (): TokenUsage => ({ input: 0, output: 0, cacheRead: 0, cacheCreation: 0 }); + // An assistant transcript line carrying token usage for one message id. function assistantLine(id: string, usage: Partial>, blockType = 'text'): string { return JSON.stringify({ @@ -368,6 +370,33 @@ describe('aggregateSessionMetrics', () => { .toEqual({ input: 0, output: 0, cacheRead: 0, cacheCreation: 0 }); }); + it('keeps a reported rollout\'s totals once compaction drops it, so a resumed rollout is reported', () => { + // A Codex build that writes a new rollout per resume restarts its counters. + const tokensA = { input: 500, output: 50, cacheRead: 1_000, cacheCreation: 0 }; + const tokensB = { input: 30, output: 3, cacheRead: 100, cacheCreation: 0 }; + const stop = (path: string, tokens: TokenUsage, prompts: number): DashboardEvent => ({ + type: 'stop', timestamp: new Date().toISOString(), sessionId: 's1', tool: 'codex', + transcriptPath: path, tokenScope: 'transcript', tokens, prompts, + }); + const first = computePromptTokenDelta(aggregateSessionMetrics([stop('/rollouts/a.jsonl', tokensA, 5)]), {}); + expect(first.delta).toEqual({ prompts: 5, tokens: tokensA }); + expect(JSON.stringify(first.nextReported)).not.toContain('/rollouts/'); + + // Compaction dropped rollout A; the resume wrote rollout B. + const resumed = computePromptTokenDelta(aggregateSessionMetrics([stop('/rollouts/b.jsonl', tokensB, 2)]), first.nextReported); + expect(resumed.delta).toEqual({ prompts: 2, tokens: tokensB }); + const again = computePromptTokenDelta(aggregateSessionMetrics([stop('/rollouts/b.jsonl', tokensB, 2)]), resumed.nextReported); + expect(again.delta).toEqual({ prompts: 0, tokens: emptyTokens() }); + }); + + it('counts a Codex session\'s prompts across its rollouts', () => { + const stop = (path: string, prompts: number): DashboardEvent => ({ + type: 'stop', timestamp: new Date().toISOString(), sessionId: 's1', tool: 'codex', + transcriptPath: path, tokenScope: 'transcript', tokens: emptyTokens(), prompts, + }); + expect(aggregateSessionMetrics([stop('/rollouts/a.jsonl', 5), stop('/rollouts/b.jsonl', 2)]).get('s1')?.prompts).toBe(7); + }); + it('does not add transcript-scoped Codex totals to a session-scoped snapshot', () => { const events: DashboardEvent[] = [ { diff --git a/src/dashboard-collector.ts b/src/dashboard-collector.ts index 08131ffd5..731a865aa 100644 --- a/src/dashboard-collector.ts +++ b/src/dashboard-collector.ts @@ -1835,6 +1835,7 @@ export function aggregateSessionMetrics( const unscopedTokens = new Map(); const sessionTokens = new Map(); const transcriptTokens = new Map>(); + const transcriptPrompts = new Map>(); for (const event of events) { let m = map.get(event.sessionId); @@ -1880,6 +1881,11 @@ export function aggregateSessionMetrics( // replace the same segment; a resumed rollout has a distinct path and adds // one new segment to the logical session total. setLatestTokenSnapshot(segments, event.transcriptPath, event); + if (event.type === 'stop' && typeof event.prompts === 'number') { + const prompts = transcriptPrompts.get(event.sessionId) ?? new Map(); + prompts.set(event.transcriptPath, event.prompts); + transcriptPrompts.set(event.sessionId, prompts); + } } else { // Claude, CodeBuddy, and pre-existing events retain latest-Stop semantics. setLatestTokenSnapshot(unscopedTokens, event.sessionId, event); @@ -1900,11 +1906,17 @@ export function aggregateSessionMetrics( let total = emptyTokenUsage(); for (const segment of segments.values()) total = addTokenUsage(total, segment.tokens); m.tokens = total; + const prompts = transcriptPrompts.get(sid); + m.segments = Object.fromEntries([...segments].map(([transcript, segment]) => + [transcript, { prompts: prompts?.get(transcript) ?? 0, tokens: { ...segment.tokens } }])); } else { const unscoped = unscopedTokens.get(sid); if (unscoped) m.tokens = { ...unscoped.tokens }; } - m.prompts = Math.max(submitCount.get(sid) ?? 0, stopPrompts.get(sid) ?? 0); + // A rollout's prompt count restarts too: its segments sum where they exist. + const rolloutPrompts = m.segments + ? Object.values(m.segments).reduce((sum, segment) => sum + segment.prompts, 0) : 0; + m.prompts = Math.max(submitCount.get(sid) ?? 0, stopPrompts.get(sid) ?? 0, rolloutPrompts); } return map; diff --git a/src/team-push.ts b/src/team-push.ts index 8f235ec04..7e53526f9 100644 --- a/src/team-push.ts +++ b/src/team-push.ts @@ -1,5 +1,6 @@ import YAML from 'yaml'; import fs from 'node:fs'; +import { createHash } from 'node:crypto'; import path from 'node:path'; import { readUsageEvents, truncateUsageAfterReport } from './usage-tracker.js'; import { aggregateUsage } from './stats.js'; @@ -31,8 +32,11 @@ import { /** Snapshot of already-reported per-session intervention counts (idempotency basis). */ type ReportedInterventions = Record; +/** A rollout's reported totals, keyed by a hash of its transcript path (no path is stored). */ +type ReportedSegments = Record; + /** Snapshot of already-reported per-session prompt counts + token usage (idempotency basis). */ -type ReportedPromptTokens = Record; +type ReportedPromptTokens = Record; /** Cumulative delta for conversation-turn count + token usage (Issue #75). */ interface PromptTokenDelta { @@ -376,11 +380,33 @@ function tokenDelta(cur: TokenUsage, prev: TokenUsage | undefined): TokenUsage { }; } +/** A rollout's key in a snapshot: its transcript path, hashed. */ +function segmentKey(transcript: string): string { + return createHash('sha256').update(transcript).digest('hex').slice(0, 16); +} + +/** The reported rollouts of a snapshot entry read from a file; empty when it holds none. */ +function reportedSegments(entry: unknown): ReportedSegments { + const segments: ReportedSegments = {}; + if (!entry || typeof entry !== 'object' || !('segments' in entry) || !entry.segments || typeof entry.segments !== 'object') { + return segments; + } + for (const [key, segment] of Object.entries(entry.segments)) segments[key] = promptTokensEntry(segment); + return segments; +} + /** * Compute the prompt-count + token delta to report: for each current session, the * positive change since it was last reported. Idempotent (a re-run reports nothing * new), and never negative if a snapshot shrinks. The next snapshot keeps only * sessions still present in events.jsonl (compacted sessions stay folded into totals). + * + * A transcript-scoped session (Codex) is reported per rollout: each rollout's + * counters restart, and the session's total sums those still in the log. A + * rollout already reported that compaction has since dropped keeps its reported + * totals in the sum, so a later rollout of the session is reported in full + * rather than against them. An entry written before rollouts were kept is + * compared as a whole, then kept per rollout. */ export function computePromptTokenDelta( current: Map, @@ -391,9 +417,24 @@ export function computePromptTokenDelta( for (const [sid, cur] of current) { const prev = reported[sid]; - delta.prompts += Math.max(0, cur.prompts - (prev?.prompts ?? 0)); - delta.tokens = addTokenUsage(delta.tokens, tokenDelta(cur.tokens, prev?.tokens)); - nextReported[sid] = { prompts: cur.prompts, tokens: cur.tokens }; + let prompts = cur.prompts; + let tokens = cur.tokens; + let segments: ReportedSegments | undefined; + if (cur.segments) { + segments = {}; + for (const [transcript, segment] of Object.entries(cur.segments)) { + segments[segmentKey(transcript)] = { prompts: segment.prompts, tokens: { ...segment.tokens } }; + } + for (const [key, gone] of Object.entries(reportedSegments(prev))) { + if (Object.hasOwn(segments, key)) continue; + segments[key] = gone; + prompts += gone.prompts; + tokens = addTokenUsage(tokens, gone.tokens); + } + } + delta.prompts += Math.max(0, prompts - (prev?.prompts ?? 0)); + delta.tokens = addTokenUsage(delta.tokens, tokenDelta(tokens, prev?.tokens)); + nextReported[sid] = { prompts, tokens, ...(segments ? { segments } : {}) }; } return { delta, nextReported }; diff --git a/src/types.ts b/src/types.ts index d89e97677..565bd9b7a 100644 --- a/src/types.ts +++ b/src/types.ts @@ -1214,6 +1214,11 @@ export interface SessionMetrics { prompts: number; /** Cumulative token usage across the logical session. */ tokens: TokenUsage; + /** + * Each rollout's own totals for a transcript-scoped session (Codex), keyed by + * its transcript path: a rollout's counters restart, so the session sums them. + */ + segments?: Record; } export type DashboardSessionStatus = 'running' | 'waiting_for_input' | 'error' | 'idle' | 'stopped'; From abe6f4012c0a9258efc3194de8d19e469b9ef586 Mon Sep 17 00:00:00 2001 From: Saul Moro Date: Fri, 25 Sep 2026 00:29:51 +0200 Subject: [PATCH 28/41] refactor(stats): move dashboard scope attribution and session owners out of team-push (#785) No behavior change. src/dashboard-scope.ts holds which scope reports a dashboard session (the log split into runs, each given whole to the scope it started in, and the transcript origin); src/session-owners.ts holds the machine-level owners index, its seeding from earlier snapshots, and the snapshot files it reads. team-push.ts keeps the snapshot adoption, deltas and push, and one reportedBaselines() now serves both the report and `teamai stats`, which repeated the adoption sequence. --- src/__tests__/scope-filter.test.ts | 3 +- src/dashboard-scope.ts | 270 ++++++++++++ src/session-owners.ts | 311 ++++++++++++++ src/stats.ts | 27 +- src/team-push.ts | 662 +++-------------------------- 5 files changed, 651 insertions(+), 622 deletions(-) create mode 100644 src/dashboard-scope.ts create mode 100644 src/session-owners.ts diff --git a/src/__tests__/scope-filter.test.ts b/src/__tests__/scope-filter.test.ts index bf55b1e3d..f632de5de 100644 --- a/src/__tests__/scope-filter.test.ts +++ b/src/__tests__/scope-filter.test.ts @@ -2,7 +2,8 @@ import { describe, it, expect } from 'vitest'; import fs from 'node:fs'; import os from 'node:os'; import path from 'node:path'; -import { adoptBareKeys, filterEventsByScope } from '../team-push.js'; +import { adoptBareKeys } from '../team-push.js'; +import { filterEventsByScope } from '../dashboard-scope.js'; import { dataHomeKey } from '../dashboard-collector.js'; import type { DashboardEvent, LocalConfig } from '../types.js'; diff --git a/src/dashboard-scope.ts b/src/dashboard-scope.ts new file mode 100644 index 000000000..e38dfdac3 --- /dev/null +++ b/src/dashboard-scope.ts @@ -0,0 +1,270 @@ +/** + * The dashboard events each scope reports (#785): the log split into runs, and + * each run given, whole, to the scope it started in. + */ +import fs from 'node:fs'; +import path from 'node:path'; +import { dataHomeKey, resolveCopilotUsageTranscript } from './dashboard-collector.js'; +import { pathExists, readJson } from './utils/fs.js'; +import type { DashboardEvent, LocalConfig } from './types.js'; +import { getDataHome } from './types.js'; +import { getUserHome } from './utils/home.js'; +import { REPORTED_SNAPSHOTS, readSessionOwners, snapshotPathIn } from './session-owners.js'; + +/** + * The dashboard events a scope reports (#785): every event of the sessions + * recorded in it. A session is the run of one ID up to its session_end or + * process_exit, since a PID-fallback ID comes back for a later run; each run + * is returned under its own ID (see {@link adoptBareKeys}). It is + * decided once, whole, by its first event keyed with a data home + * (`dataHomeKey`, or the path an unreleased build of #795 wrote), because a + * Stop carries the whole transcript's totals: split per event, a session that + * moved into another scope would count there again the part recorded before + * the move. A project also owns the key of its in-repo `.teamai`, where a + * hook recorded until migration moved the project to a partition. A session + * written before events carried a key is decided by its first cwd, by the + * dispatcher's rule: the scope that directory resolves to now, so a nested + * clone under a project is not the project's; one with no cwd, or a cwd gone + * since, is no scope's. A tool's own session ID a scope has recorded as its + * own (see `session-owners.jsonl`) is that scope's, whatever the log still + * holds. A caller without a scope config reads the whole log. + */ +export async function filterEventsByScope( + events: DashboardEvent[], + config?: LocalConfig, +): Promise { + if (!config) return events; + const ownKey = await dataHomeKey(getDataHome(config)); + const keys = new Set([ownKey]); + if (config.projectRoot) { + // Unless the project is rooted at HOME, where that is the user scope's. + const legacy = await dataHomeKey(path.join(config.projectRoot, '.teamai')); + if (legacy !== (await dataHomeKey(path.join(getUserHome(), '.teamai')))) keys.add(legacy); + } + const { resolveConfigForDir } = await import('./config.js'); + const resolvesHere = new Map>(); + const ownsCwd = (cwd: string): Promise => { + let owns = resolvesHere.get(cwd); + if (!owns) { + owns = pathExists(cwd).then(async (exists) => { + const resolved = exists ? await resolveConfigForDir(cwd) : null; + return !!resolved && keys.has(await dataHomeKey(getDataHome(resolved))); + }); + resolvesHere.set(cwd, owns); + } + return owns; + }; + const eventKeys = await keysOf(events); + const { runOf, runIds, deciding } = splitRuns(events, eventKeys); + const owners = await readSessionOwners(); + const transcripts = new Map(); + events.forEach((e, i) => { + const run = runOf[i]; + if (run === undefined) return; + // Copilot's session log is found by its ID; the event holds no path (#666). + const transcript = typeof e.transcriptPath === 'string' ? e.transcriptPath + : e.tool === 'copilot' ? resolveCopilotUsageTranscript(e.sessionId) : null; + const known = transcripts.get(run) ?? []; + if (transcript && !known.includes(transcript)) transcripts.set(run, [...known, transcript]); + }); + const owned = await Promise.all(deciding.map(async (i, run) => { + // Recorded by the scope that first reported it, so a session resumed + // elsewhere after compaction dropped its events stays that scope's. + const owner = owners.get(runIds[run]) ?? (runIds[run].startsWith('pid-') ? undefined + : await transcriptOwner(runIds[run], transcripts.get(run) ?? [], resolveConfigForDir)); + if (owner !== undefined) return keys.has(owner); + if (i === undefined) return false; + const key = eventKeys[i]; + const cwd = events[i].cwd; + return key !== undefined ? keys.has(key) : !!cwd && await ownsCwd(cwd); + })); + return events.flatMap((e, i) => { + const run = runOf[i]; + return run !== undefined && owned[run] ? [{ ...e, sessionId: runIds[run] }] : []; + }); +} + +/** + * The key of the scope a tool's own session started in, by its transcript, + * when that scope's snapshots already hold it: an earlier release reported it + * there, and compaction has since dropped the events that would say so. The + * latest transcript path first, since a tool may relocate the file. Undefined + * when there is no such transcript, origin, scope or snapshot entry (a fork + * under a new ID, a scope that never reported it). + */ +async function transcriptOwner( + sessionId: string, + paths: string[], + resolveConfigForDir: (dir: string) => Promise, +): Promise { + for (const transcript of [...paths].reverse()) { + const origin = await transcriptOrigin(transcript); + if (origin === undefined) continue; + const config = (await pathExists(origin)) ? await resolveConfigForDir(origin) : null; + if (!config) return undefined; + const dataHome = getDataHome(config); + const snapshots = await Promise.all(REPORTED_SNAPSHOTS.map((name) => + readJson>(snapshotPathIn(dataHome, name)))); + const held = snapshots.some((snapshot) => !!snapshot && typeof snapshot === 'object' && Object.hasOwn(snapshot, sessionId)); + return held ? dataHomeKey(dataHome) : undefined; + } + return undefined; +} + +/** + * The directory a transcript's session started in: the first `cwd` a Claude + * transcript records (a resume from another project appends to the same file), + * a Codex rollout's `session_meta`, or a Copilot session log's `session.start`. + * Reads a bounded head of the file; the format is the tool's own, so anything + * else is undefined. + */ +async function transcriptOrigin(transcript: string): Promise { + let head: string; + try { + const fh = await fs.promises.open(transcript, 'r'); + try { + const buffer = Buffer.alloc(64 * 1024); + const { bytesRead } = await fh.read(buffer, 0, buffer.length, 0); + head = buffer.subarray(0, bytesRead).toString('utf8'); + } finally { + await fh.close(); + } + } catch { + return undefined; + } + for (const line of head.split('\n')) { + let entry: unknown; + try { + entry = JSON.parse(line); + } catch { + continue; + } + if (!entry || typeof entry !== 'object') continue; + if ('cwd' in entry && typeof entry.cwd === 'string') return entry.cwd; + if ('type' in entry && entry.type === 'session_meta' && 'payload' in entry && entry.payload + && typeof entry.payload === 'object' && 'cwd' in entry.payload && typeof entry.payload.cwd === 'string') { + return entry.payload.cwd; + } + if ('type' in entry && entry.type === 'session.start' && 'data' in entry && entry.data && typeof entry.data === 'object' + && 'context' in entry.data && entry.data.context && typeof entry.data.context === 'object' + && 'cwd' in entry.data.context && typeof entry.data.context.cwd === 'string') { + return entry.data.context.cwd; + } + } + return undefined; +} + +/** + * Each event's data home key. The log is hand-editable: a key that is not a + * string counts as absent. An unreleased build of #795 recorded the data home + * path in place of its key. + */ +async function keysOf(events: DashboardEvent[]): Promise> { + const hashes = new Map>(); + return Promise.all(events.map((e) => { + if (typeof e.dataHomeKey === 'string') return e.dataHomeKey; + if (typeof e.dataHome !== 'string') return undefined; + let key = hashes.get(e.dataHome); + if (!key) { + key = dataHomeKey(e.dataHome); + hashes.set(e.dataHome, key); + } + return key; + })); +} + +/** + * The runs of a log: each event's run (none for an exit whose run is gone), + * each run's ID, and the event that decides its scope. + * + * A tool's own session ID is one run, whatever ends it records: `claude + * --resume` continues it, in a new process, and its Stop carries the whole + * transcript. It is returned under the ID itself, so it stays the session + * already reported after compaction drops its events. A PID-fallback ID + * (`pid-…`) names one run until it ends, then comes back for a later one, + * maybe in another scope, so each run is decided on its own and returned under + * the ID plus its first event's timestamp, which stays the same whichever + * earlier runs compaction has dropped. A second end with nothing recorded since + * the first (the dashboard monitor's process_exit after SessionEnd) belongs to + * the run just closed. A start from another process than its open run's begins + * a new run, though nothing ended that one (a crash with no dashboard running). + * The monitor's exit names the last event it observed (`processExitAfter`). + * A dashboard started before that field existed wrote none; since a dead + * process records nothing more, such an exit followed by more events of its + * fallback ID before the next start did not end the open run: it was observed + * before that run began, and belongs to the run closed before it. + */ +function splitRuns(events: DashboardEvent[], eventKeys: Array): { + runOf: Array; + runIds: string[]; + deciding: Array; +} { + const runOf: Array = []; + const runPid: Array = []; + const observedRuns = new Map(); + const openRun = new Map(); + const closedRun = new Map(); + const deciding: Array = []; + const runIds: string[] = []; + // Whether each event is followed by activity of its ID (anything but a start + // or a monitor exit) before that ID's next start. + const continues: boolean[] = []; + const active = new Map(); + for (let i = events.length - 1; i >= 0; i--) { + const { sessionId, type } = events[i]; + continues[i] = active.get(sessionId) ?? false; + if (type === 'session_start') active.set(sessionId, false); + else if (type !== 'process_exit') active.set(sessionId, true); + } + events.forEach((e, i) => { + const fallback = e.sessionId.startsWith('pid-'); + const ends = e.type === 'session_end' || e.type === 'process_exit'; + const observed = e.type === 'process_exit' && typeof e.processExitAfter === 'string'; + const starts = e.type === 'session_start' && typeof e.monitorPid === 'number'; + const open = openRun.get(e.sessionId); + if (starts && open !== undefined && fallback + && typeof runPid[open] === 'number' && runPid[open] !== e.monitorPid) { + // Nothing ended it (a crash), but a late exit of it must still find it. + openRun.delete(e.sessionId); + closedRun.set(e.sessionId, open); + } + const closed = closedRun.get(e.sessionId); + const stale = e.type === 'process_exit' && !observed && fallback && open !== undefined && closed !== undefined + && continues[i]; + let run = observed ? observedRuns.get(`${e.sessionId}@${e.processExitAfter}`) + : stale ? closed : openRun.get(e.sessionId) ?? (ends ? closed : undefined); + // The observed run may have been compacted. Its delayed exit must not + // manufacture a new session or close a later reuse of the same ID. + if (observed && run === undefined) { + runOf.push(undefined); + return; + } + if (run === undefined) { + run = deciding.push(undefined) - 1; + runIds.push(fallback ? `${e.sessionId}@${e.timestamp}` : e.sessionId); + } + if (ends && fallback && !stale && (!observed || openRun.get(e.sessionId) === run)) { + openRun.delete(e.sessionId); + closedRun.set(e.sessionId, run); + } else if (!ends || !fallback) { + openRun.set(e.sessionId, run); + } + if (starts) runPid[run] ??= e.monitorPid; + runOf.push(run); + observedRuns.set(`${e.sessionId}@${e.timestamp}`, run); + const current = deciding[run]; + if (eventKeys[i] !== undefined ? current === undefined || eventKeys[current] === undefined + : current === undefined && !!e.cwd) deciding[run] = i; + }); + return { runOf, runIds, deciding }; +} + +/** Every event of the log under its run ID, whichever scope it belongs to. */ +export async function runsOfLog(events: DashboardEvent[]): Promise { + const { runOf, runIds } = splitRuns(events, await keysOf(events)); + return events.flatMap((e, i) => { + const run = runOf[i]; + return run !== undefined ? [{ ...e, sessionId: runIds[run] }] : []; + }); +} + diff --git a/src/session-owners.ts b/src/session-owners.ts new file mode 100644 index 000000000..95817cd9b --- /dev/null +++ b/src/session-owners.ts @@ -0,0 +1,311 @@ +/** + * Which scope a tool's own session belongs to once compaction has dropped the + * events that would say so (#785): the machine-level session-owners index, its + * seeding from the snapshots an earlier release left, and the snapshot files it + * reads for that. + */ +import fs from 'node:fs'; +import path from 'node:path'; +import { readEvents, dataHomeKey } from './dashboard-collector.js'; +import { readFileSafe, ensureDir, pathExists, readJson } from './utils/fs.js'; +import { log } from './utils/logger.js'; +import type { TokenUsage } from './types.js'; +import { getDataHome, getTeamaiHomeDir, emptyTokenUsage } from './types.js'; +import { projectsRootDir } from './utils/partition.js'; +import { parseDailySnapshot, type DailySessionSnapshot } from './session-trends.js'; + +/** Snapshot of already-reported per-session intervention counts (idempotency basis). */ +export type ReportedInterventions = Record; + +/** A rollout's reported totals, keyed by a hash of its transcript path (no path is stored). */ +export type ReportedSegments = Record; + +/** Snapshot of already-reported per-session prompt counts + token usage (idempotency basis). */ +export type ReportedPromptTokens = Record; + + +export const REPORTED_SNAPSHOTS = ['interventions', 'prompt-tokens', 'daily-sessions'] as const; +export type ReportedSnapshotName = typeof REPORTED_SNAPSHOTS[number]; + +/** The machine-level snapshot every scope shared before #786 (evaluated at call time for tests). */ +export function sharedSnapshotPath(name: ReportedSnapshotName): string { + return path.join(getTeamaiHomeDir(), 'dashboard', `reported-${name}.json`); +} + +/** The snapshot of the scope whose data home is `dataHome`. */ +export function snapshotPathIn(dataHome: string, name: ReportedSnapshotName): string { + const file = `reported-${name}.json`; + if (path.resolve(dataHome) !== path.resolve(getTeamaiHomeDir())) return path.join(dataHome, 'dashboard', file); + return path.join(dataHome, 'dashboard', `user-${file}`); +} + +// ─── Session owners ───────────────────────────────────── +// +// ~/.teamai/dashboard/session-owners.jsonl {"sessionId":"","dataHomeKey":""} per line +// +// A tool's own session ID can be resumed anywhere, long after compaction +// dropped its events, and its Stop carries the whole transcript. So the scope +// that first reports it records itself here, append-only, and the session +// stays that scope's. Only the ID and the key, never a path (#666). The first +// line for an ID wins. +// +// A release before this file kept per-scope snapshots only, so the file is +// first written from them: each tool's own ID in any snapshot of a scope is +// the scope's that holds its greatest total (prompts, then tokens), since a +// session that release split per event holds only part of it elsewhere. A tie +// names no owner: that release copied the shared file into every scope. The +// scopes read are the user scope, every partition, and a project whose data +// home is in its workspace that a session still in the log leads to. Each +// report also records the IDs of its own snapshots that have no owner yet, +// for such a project the log no longer leads to, when they show it reported +// them: absent from the shared snapshot, or past its total there. +// + +function sessionOwnersPath(): string { + return path.join(getTeamaiHomeDir(), 'dashboard', 'session-owners.jsonl'); +} + +/** The data homes whose snapshots an earlier release may have written. */ +async function knownDataHomes(): Promise { + const slugs = await fs.promises.readdir(projectsRootDir()).catch(() => []); + const homes = [getTeamaiHomeDir(), ...[...slugs].sort().map((slug) => path.join(projectsRootDir(), slug))]; + // A project whose data home is in its workspace is under no partition; a + // session of it still in the log leads to it. + const { resolveConfigForDir } = await import('./config.js'); + const cwds = new Set((await readEvents()).flatMap((e) => (typeof e.cwd === 'string' ? [e.cwd] : []))); + for (const cwd of cwds) { + const config = (await pathExists(cwd)) ? await resolveConfigForDir(cwd) : null; + if (config) homes.push(getDataHome(config)); + } + return [...new Set(homes.map((home) => path.resolve(home)))]; +} + +/** A snapshot entry's reported prompts and tokens, 0 for what it does not hold. */ +export function reportedSize(entry: unknown): { prompts: number; tokens: number } { + if (!entry || typeof entry !== 'object') return { prompts: 0, tokens: 0 }; + const prompts = 'prompts' in entry && typeof entry.prompts === 'number' ? entry.prompts : 0; + const tokens = 'tokens' in entry && entry.tokens && typeof entry.tokens === 'object' + ? Object.values(entry.tokens).reduce((sum: number, n: unknown) => sum + (typeof n === 'number' ? n : 0), 0) + : 0; + return { prompts, tokens }; +} + +/** A scope's three snapshots (the shared ones for no data home), as a file holds them. */ +export interface ScopeSnapshots { + interventions: Record | null; + promptTokens: Record | null; + daily: Record | null; +} + +export async function readSnapshotsIn(dataHome: string | undefined): Promise { + const read = (name: ReportedSnapshotName) => + readJson>(dataHome === undefined ? sharedSnapshotPath(name) : snapshotPathIn(dataHome, name)); + const [interventions, promptTokens, daily] = await Promise.all(REPORTED_SNAPSHOTS.map(read)); + return { interventions, promptTokens, daily }; +} + +/** An intervention entry read from a file, zero for what it does not hold. */ +export function interventionsEntry(value: unknown): ReportedInterventions[string] { + const count = (field: 'interrupt' | 'toolReject' | 'correction') => + (value && typeof value === 'object' && field in value && typeof Object.entries(value).find(([k]) => k === field)?.[1] === 'number' + ? Number(Object.entries(value).find(([k]) => k === field)?.[1]) : 0); + return { interrupt: count('interrupt'), toolReject: count('toolReject'), correction: count('correction') }; +} + +/** A prompt-token entry read from a file, zero for what it does not hold. */ +export function promptTokensEntry(value: unknown): ReportedPromptTokens[string] { + const tokens = emptyTokenUsage(); + if (value && typeof value === 'object' && 'tokens' in value && value.tokens && typeof value.tokens === 'object') { + for (const [field, n] of Object.entries(value.tokens)) { + if (typeof n !== 'number') continue; + if (field === 'input' || field === 'output' || field === 'cacheRead' || field === 'cacheCreation') tokens[field] = n; + } + } + return { prompts: reportedSize(value).prompts, tokens }; +} + +/** What a scope's snapshots hold of `id`: prompts, tokens and intervention counts. */ +function heldIn(snapshots: ScopeSnapshots, id: string): { prompts: number; tokens: number; interventions: number } { + const fromTokens = reportedSize(snapshots.promptTokens?.[id]); + const iv = interventionsEntry(snapshots.interventions?.[id]); + return { + prompts: Math.max(fromTokens.prompts, reportedSize(snapshots.daily?.[id]).prompts), + tokens: fromTokens.tokens, + interventions: iv.interrupt + iv.toolReject + iv.correction, + }; +} + +/** + * Whether a scope's snapshots show it reported `id`: the shared snapshots hold + * none of it, or the scope is past their total. An earlier release copied the + * shared snapshots into every scope it ran in, so a copy shows nothing. + */ +export function showsReported(own: ScopeSnapshots, shared: ScopeSnapshots, id: string): boolean { + const inShared = [shared.interventions, shared.promptTokens, shared.daily].some((snapshot) => + !!snapshot && typeof snapshot === 'object' && Object.hasOwn(snapshot, id)); + if (!inShared) return true; + const a = heldIn(own, id); + const b = heldIn(shared, id); + return a.prompts !== b.prompts ? a.prompts > b.prompts + : a.tokens !== b.tokens ? a.tokens > b.tokens : a.interventions > b.interventions; +} + +/** What several scopes had reported of one session, credited to its owner (numbers only). */ +export interface OwnerCredit { + interventions: ReportedInterventions[string]; + promptTokens: ReportedPromptTokens[string]; + daily?: DailySessionSnapshot; +} + +/** + * The credit for a session an earlier release split across scopes per event, + * from their snapshots alone (its events are gone). A part whose daily entry + * shows it ended in a Stop carries the transcript's cumulative total, so the + * greatest such part counts once; a part with no Stop counted its own prompts, + * so those add. Intervention counts, per event, add; tokens, from Stops, take + * the greatest. A part without a Stop that came before another's Stop is + * credited twice: that undercounts, once, but never sends a prompt again. + */ +function creditOf(parts: ScopeSnapshots[], id: string): OwnerCredit { + const stops = parts.filter((part) => parseDailySnapshot(part.daily?.[id]) !== undefined); + const loose = parts.filter((part) => !stops.includes(part)); + const prompts = Math.max(0, ...stops.map((part) => heldIn(part, id).prompts)) + + loose.reduce((sum, part) => sum + heldIn(part, id).prompts, 0); + const tokens = emptyTokenUsage(); + const interventions = { interrupt: 0, toolReject: 0, correction: 0 }; + for (const part of parts) { + const entry = promptTokensEntry(part.promptTokens?.[id]); + for (const field of ['input', 'output', 'cacheRead', 'cacheCreation'] as const) { + tokens[field] = Math.max(tokens[field], entry.tokens[field]); + } + const iv = interventionsEntry(part.interventions?.[id]); + interventions.interrupt += iv.interrupt; + interventions.toolReject += iv.toolReject; + interventions.correction += iv.correction; + } + const days = stops.flatMap((part) => { + const day = parseDailySnapshot(part.daily?.[id]); + return day ? [day] : []; + }); + const latest = days.reduce((a, b) => (!a || b.prompts > a.prompts ? b : a), undefined); + return { + interventions, + promptTokens: { prompts, tokens }, + ...(latest ? { daily: { ...latest, prompts, durationMs: days.reduce((sum, day) => sum + day.durationMs, 0) } } : {}), + }; +} + +/** + * The owners the per-scope snapshots of an earlier release imply, as the file's + * first lines: for each tool's own ID, the scope whose snapshots show it reported + * it with the greatest total, and, when several did, the credit of their parts. + * A tie names no owner. + */ +async function ownersFromSnapshots(): Promise { + const shared = await readSnapshotsIn(undefined); + const parts = new Map>(); + for (const dataHome of await knownDataHomes()) { + const snapshots = await readSnapshotsIn(dataHome); + const ids = new Set([snapshots.interventions, snapshots.promptTokens, snapshots.daily].flatMap((snapshot) => + (snapshot && typeof snapshot === 'object' ? Object.keys(snapshot) : []))); + const key = await dataHomeKey(dataHome); + for (const sessionId of ids) { + if (sessionId.startsWith('pid-') || !showsReported(snapshots, shared, sessionId)) continue; + parts.set(sessionId, [...(parts.get(sessionId) ?? []), { key, snapshots }]); + } + } + const lines: string[] = []; + for (const [sessionId, held] of parts) { + const size = (part: { snapshots: ScopeSnapshots }) => heldIn(part.snapshots, sessionId); + const best = held.reduce((a, b) => { + const x = size(a); + const y = size(b); + return y.prompts > x.prompts || (y.prompts === x.prompts && y.tokens > x.tokens) ? b : a; + }); + const tied = held.some((part) => part.key !== best.key + && size(part).prompts === size(best).prompts && size(part).tokens === size(best).tokens); + if (tied) continue; + const credit = held.length > 1 ? creditOf(held.map((part) => part.snapshots), sessionId) : undefined; + lines.push(JSON.stringify({ sessionId, dataHomeKey: best.key, ...(credit ? { credit } : {}) })); + } + return lines.map((line) => `${line}\n`).join(''); +} + +/** The credits the file's first line for each ID carries (see {@link creditOf}). */ +export async function readOwnerCredits(): Promise> { + const credits = new Map(); + const seen = new Set(); + for (const line of ((await readFileSafe(sessionOwnersPath())) ?? '').split('\n')) { + let parsed: unknown; + try { + parsed = JSON.parse(line); + } catch { + continue; + } + if (!parsed || typeof parsed !== 'object' || !('sessionId' in parsed) || typeof parsed.sessionId !== 'string') continue; + if (seen.has(parsed.sessionId)) continue; + seen.add(parsed.sessionId); + if (!('credit' in parsed) || !parsed.credit || typeof parsed.credit !== 'object') continue; + const credit = parsed.credit; + const daily = 'daily' in credit ? parseDailySnapshot(credit.daily) : undefined; + credits.set(parsed.sessionId, { + interventions: interventionsEntry('interventions' in credit ? credit.interventions : undefined), + promptTokens: promptTokensEntry('promptTokens' in credit ? credit.promptTokens : undefined), + ...(daily ? { daily } : {}), + }); + } + return credits; +} + +export async function readSessionOwners(): Promise> { + const owners = new Map(); + if (!(await pathExists(sessionOwnersPath()))) { + try { + await ensureDir(path.dirname(sessionOwnersPath())); + // Exclusive: a report in another scope may be writing it too. + await fs.promises.writeFile(sessionOwnersPath(), await ownersFromSnapshots(), { flag: 'wx' }); + } catch (e) { + if ((e as NodeJS.ErrnoException).code !== 'EEXIST') log.debug(`Could not seed session owners: ${(e as Error).message}`); + } + } + const content = await readFileSafe(sessionOwnersPath()); + for (const line of (content ?? '').split('\n')) { + if (!line.trim()) continue; + try { + const parsed: unknown = JSON.parse(line); + if (parsed && typeof parsed === 'object' && 'sessionId' in parsed && 'dataHomeKey' in parsed + && typeof parsed.sessionId === 'string' && typeof parsed.dataHomeKey === 'string' + && !owners.has(parsed.sessionId)) owners.set(parsed.sessionId, parsed.dataHomeKey); + } catch { + // A torn or hand-edited line records no owner. + } + } + return owners; +} + +/** The IDs of a scope's snapshots that show it reported them (see {@link showsReported}). */ +export async function reportedBeyondShared( + promptTokens: Record, + interventions: Record, + daily: Record, +): Promise { + const shared = await readSnapshotsIn(undefined); + const own: ScopeSnapshots = { interventions, promptTokens, daily }; + const ids = new Set([...Object.keys(promptTokens), ...Object.keys(interventions), ...Object.keys(daily)]); + return [...ids].filter((id) => showsReported(own, shared, id)); +} + +/** Records `key` as the owner of the tool-own session IDs among `sessionIds` that have none yet. */ +export async function recordSessionOwners(sessionIds: Iterable, key: string): Promise { + const owners = await readSessionOwners(); + const ids = new Set([...sessionIds].filter((id) => !id.startsWith('pid-') && !owners.has(id))); + if (ids.size === 0) return; + try { + await ensureDir(path.dirname(sessionOwnersPath())); + await fs.promises.appendFile(sessionOwnersPath(), + [...ids].map((sessionId) => JSON.stringify({ sessionId, dataHomeKey: key })).join('\n') + '\n'); + } catch (e) { + log.debug(`Could not record session owners: ${(e as Error).message}`); + } +} + diff --git a/src/stats.ts b/src/stats.ts index 38025da5b..bdfea9f14 100644 --- a/src/stats.ts +++ b/src/stats.ts @@ -156,26 +156,11 @@ async function unreportedDashboardStats( metrics: Map, config: LocalConfig, ): Promise { - const { - adoptBareKeys, computeInterventionDelta, computePromptTokenDelta, readReportedInterventions, readReportedPromptTokens, - takeInterventions, takePromptTokens, creditSplitRuns, - } = await import('./team-push.js'); - // The scope's own snapshots, the ones its report compares against (#786). - const currentInterventions = new Map( - [...metrics].map(([sid, m]) => [sid, { interrupt: m.interrupt, toolReject: m.toolReject, correction: m.correction }]), - ); - // Prompt tokens first: they decide which runs of a bare ID were reported. - const adoptedPromptTokens = adoptBareKeys( - await readReportedPromptTokens(config), events, computePromptTokenDelta(metrics, {}).nextReported, takePromptTokens, - ); - const adoptedInterventions = adoptBareKeys( - await readReportedInterventions(config), events, Object.fromEntries(currentInterventions), - takeInterventions((runId) => Object.hasOwn(adoptedPromptTokens, runId)), - ); - // As the report credits a session main split across scopes. - const { promptTokens, interventions } = await creditSplitRuns(events, { - interventions: adoptedInterventions, promptTokens: adoptedPromptTokens, daily: {}, - }); + const { computeInterventionDelta, computePromptTokenDelta, interventionCounts, reportedBaselines } = await import('./team-push.js'); + const { aggregateDailySessions } = await import('./session-trends.js'); + // The scope's own snapshots, compared as its report compares them (#786). + const { promptTokens, interventions } = await reportedBaselines(events, metrics, aggregateDailySessions(events), config, false); + const currentInterventions = interventionCounts(metrics); const interventionDelta = computeInterventionDelta(currentInterventions, interventions); const promptTokenDelta = computePromptTokenDelta(metrics, promptTokens); @@ -241,7 +226,7 @@ export async function showStats(options: ShowStatsOptions = {}): Promise { // events.jsonl until compaction, so counting the full local aggregate would // count each one twice and pull in other projects' sessions). Same filter, // same scope config as the report path (#785). - const { filterEventsByScope } = await import('./team-push.js'); + const { filterEventsByScope } = await import('./dashboard-scope.js'); const scopedEvents = await filterEventsByScope(await readEvents(), config ?? undefined); const metricsMap = aggregateSessionMetrics(scopedEvents); // Only subtract what the team already holds. Two guards, because a scope's diff --git a/src/team-push.ts b/src/team-push.ts index 7e53526f9..b6e2c284e 100644 --- a/src/team-push.ts +++ b/src/team-push.ts @@ -1,10 +1,9 @@ import YAML from 'yaml'; -import fs from 'node:fs'; import { createHash } from 'node:crypto'; import path from 'node:path'; import { readUsageEvents, truncateUsageAfterReport } from './usage-tracker.js'; import { aggregateUsage } from './stats.js'; -import { readEvents, aggregateSessionMetrics, dataHomeKey, resolveCopilotUsageTranscript } from './dashboard-collector.js'; +import { readEvents, aggregateSessionMetrics, dataHomeKey } from './dashboard-collector.js'; import { createGit, pushRepoDirectly, @@ -16,28 +15,22 @@ import { import { writeFile, readFileSafe, ensureDir, pathExists, readJson, writeJson } from './utils/fs.js'; import { log } from './utils/logger.js'; import type { UserStats, UserInterventionStats, SessionMetrics, TokenUsage, DashboardEvent, LocalConfig } from './types.js'; -import { getVotesDir, getDataHome, getTeamaiHomeDir, emptyTokenUsage, addTokenUsage, usesBranchWorktree } from './types.js'; -import { getUserHome } from './utils/home.js'; -import { projectsRootDir } from './utils/partition.js'; +import { getVotesDir, getDataHome, emptyTokenUsage, addTokenUsage, usesBranchWorktree } from './types.js'; +import { filterEventsByScope, runsOfLog } from './dashboard-scope.js'; +import { + interventionsEntry, promptTokensEntry, readOwnerCredits, recordSessionOwners, reportedBeyondShared, reportedSize, + sharedSnapshotPath, snapshotPathIn, + type ReportedInterventions, type ReportedPromptTokens, type ReportedSegments, type ReportedSnapshotName, +} from './session-owners.js'; import { aggregateDailySessions, computeDailyStatsDelta, mergeDailyStats, takeDailySession, - parseDailySnapshot, type DailySessionSnapshot, type ReportedDailySessions, } from './session-trends.js'; -/** Snapshot of already-reported per-session intervention counts (idempotency basis). */ -type ReportedInterventions = Record; - -/** A rollout's reported totals, keyed by a hash of its transcript path (no path is stored). */ -type ReportedSegments = Record; - -/** Snapshot of already-reported per-session prompt counts + token usage (idempotency basis). */ -type ReportedPromptTokens = Record; - /** Cumulative delta for conversation-turn count + token usage (Issue #75). */ interface PromptTokenDelta { prompts: number; @@ -160,25 +153,11 @@ export function mergeStats( // the whole log and reports into it). // -const REPORTED_SNAPSHOTS = ['interventions', 'prompt-tokens', 'daily-sessions'] as const; -type ReportedSnapshotName = typeof REPORTED_SNAPSHOTS[number]; - -/** The machine-level snapshot every scope shared before #786 (evaluated at call time for tests). */ -function sharedSnapshotPath(name: ReportedSnapshotName): string { - return path.join(getTeamaiHomeDir(), 'dashboard', `reported-${name}.json`); -} - /** A scope's own snapshot. The user scope's data home holds the shared one, hence its prefix. */ function scopeSnapshotPath(name: ReportedSnapshotName, config: LocalConfig | undefined): string { return config ? snapshotPathIn(getDataHome(config), name) : sharedSnapshotPath(name); } -/** The snapshot of the scope whose data home is `dataHome`. */ -function snapshotPathIn(dataHome: string, name: ReportedSnapshotName): string { - const file = `reported-${name}.json`; - if (path.resolve(dataHome) !== path.resolve(getTeamaiHomeDir())) return path.join(dataHome, 'dashboard', file); - return path.join(dataHome, 'dashboard', `user-${file}`); -} /** * A scope's snapshot, seeded from the shared one when the scope has none yet. @@ -235,7 +214,7 @@ async function writeReportedInterventions(data: ReportedInterventions, config: L } /** Each session's intervention counts, the shape its snapshot entry holds. */ -function interventionCounts(metrics: Map): Map { +export function interventionCounts(metrics: Map): Map { return new Map([...metrics].map(([sid, m]) => [sid, { interrupt: m.interrupt, toolReject: m.toolReject, correction: m.correction }])); } @@ -248,7 +227,7 @@ function upTo(own: number, left: number): number { * A run's prompt and token share of a bare entry. This snapshot decides which * runs of the ID that release reported: the ones its prompts and tokens reach. */ -export const takePromptTokens: TakeReported = (run, entry) => { +const takePromptTokens: TakeReported = (run, entry) => { // Read from a snapshot file an earlier release wrote, so parsed, not trusted. const left = promptTokensEntry(entry); if (left.prompts <= 0 && !hasPromptTokenDelta({ prompts: 0, tokens: left.tokens })) return undefined; @@ -268,7 +247,7 @@ export const takePromptTokens: TakeReported = (run * A run's intervention share, for the runs `covered`, the prompt-token * snapshot's: these counts are mostly zero, so running out says nothing. */ -export function takeInterventions(covered: (runId: string) => boolean): TakeReported { +function takeInterventions(covered: (runId: string) => boolean): TakeReported { return (run, entry, runId) => { if (!covered(runId)) return undefined; const left = interventionsEntry(entry); @@ -480,533 +459,6 @@ function hasDailyDelta(delta: ReturnType['delta'] ); } -/** - * The dashboard events a scope reports (#785): every event of the sessions - * recorded in it. A session is the run of one ID up to its session_end or - * process_exit, since a PID-fallback ID comes back for a later run; each run - * is returned under its own ID (see {@link adoptBareKeys}). It is - * decided once, whole, by its first event keyed with a data home - * (`dataHomeKey`, or the path an unreleased build of #795 wrote), because a - * Stop carries the whole transcript's totals: split per event, a session that - * moved into another scope would count there again the part recorded before - * the move. A project also owns the key of its in-repo `.teamai`, where a - * hook recorded until migration moved the project to a partition. A session - * written before events carried a key is decided by its first cwd, by the - * dispatcher's rule: the scope that directory resolves to now, so a nested - * clone under a project is not the project's; one with no cwd, or a cwd gone - * since, is no scope's. A tool's own session ID a scope has recorded as its - * own (see `session-owners.jsonl`) is that scope's, whatever the log still - * holds. A caller without a scope config reads the whole log. - */ -export async function filterEventsByScope( - events: DashboardEvent[], - config?: LocalConfig, -): Promise { - if (!config) return events; - const ownKey = await dataHomeKey(getDataHome(config)); - const keys = new Set([ownKey]); - if (config.projectRoot) { - // Unless the project is rooted at HOME, where that is the user scope's. - const legacy = await dataHomeKey(path.join(config.projectRoot, '.teamai')); - if (legacy !== (await dataHomeKey(path.join(getUserHome(), '.teamai')))) keys.add(legacy); - } - const { resolveConfigForDir } = await import('./config.js'); - const resolvesHere = new Map>(); - const ownsCwd = (cwd: string): Promise => { - let owns = resolvesHere.get(cwd); - if (!owns) { - owns = pathExists(cwd).then(async (exists) => { - const resolved = exists ? await resolveConfigForDir(cwd) : null; - return !!resolved && keys.has(await dataHomeKey(getDataHome(resolved))); - }); - resolvesHere.set(cwd, owns); - } - return owns; - }; - const eventKeys = await keysOf(events); - const { runOf, runIds, deciding } = splitRuns(events, eventKeys); - const owners = await readSessionOwners(); - const transcripts = new Map(); - events.forEach((e, i) => { - const run = runOf[i]; - if (run === undefined) return; - // Copilot's session log is found by its ID; the event holds no path (#666). - const transcript = typeof e.transcriptPath === 'string' ? e.transcriptPath - : e.tool === 'copilot' ? resolveCopilotUsageTranscript(e.sessionId) : null; - const known = transcripts.get(run) ?? []; - if (transcript && !known.includes(transcript)) transcripts.set(run, [...known, transcript]); - }); - const owned = await Promise.all(deciding.map(async (i, run) => { - // Recorded by the scope that first reported it, so a session resumed - // elsewhere after compaction dropped its events stays that scope's. - const owner = owners.get(runIds[run]) ?? (runIds[run].startsWith('pid-') ? undefined - : await transcriptOwner(runIds[run], transcripts.get(run) ?? [], resolveConfigForDir)); - if (owner !== undefined) return keys.has(owner); - if (i === undefined) return false; - const key = eventKeys[i]; - const cwd = events[i].cwd; - return key !== undefined ? keys.has(key) : !!cwd && await ownsCwd(cwd); - })); - return events.flatMap((e, i) => { - const run = runOf[i]; - return run !== undefined && owned[run] ? [{ ...e, sessionId: runIds[run] }] : []; - }); -} - -/** - * The key of the scope a tool's own session started in, by its transcript, - * when that scope's snapshots already hold it: an earlier release reported it - * there, and compaction has since dropped the events that would say so. The - * latest transcript path first, since a tool may relocate the file. Undefined - * when there is no such transcript, origin, scope or snapshot entry (a fork - * under a new ID, a scope that never reported it). - */ -async function transcriptOwner( - sessionId: string, - paths: string[], - resolveConfigForDir: (dir: string) => Promise, -): Promise { - for (const transcript of [...paths].reverse()) { - const origin = await transcriptOrigin(transcript); - if (origin === undefined) continue; - const config = (await pathExists(origin)) ? await resolveConfigForDir(origin) : null; - if (!config) return undefined; - const dataHome = getDataHome(config); - const snapshots = await Promise.all(REPORTED_SNAPSHOTS.map((name) => - readJson>(snapshotPathIn(dataHome, name)))); - const held = snapshots.some((snapshot) => !!snapshot && typeof snapshot === 'object' && Object.hasOwn(snapshot, sessionId)); - return held ? dataHomeKey(dataHome) : undefined; - } - return undefined; -} - -/** - * The directory a transcript's session started in: the first `cwd` a Claude - * transcript records (a resume from another project appends to the same file), - * a Codex rollout's `session_meta`, or a Copilot session log's `session.start`. - * Reads a bounded head of the file; the format is the tool's own, so anything - * else is undefined. - */ -async function transcriptOrigin(transcript: string): Promise { - let head: string; - try { - const fh = await fs.promises.open(transcript, 'r'); - try { - const buffer = Buffer.alloc(64 * 1024); - const { bytesRead } = await fh.read(buffer, 0, buffer.length, 0); - head = buffer.subarray(0, bytesRead).toString('utf8'); - } finally { - await fh.close(); - } - } catch { - return undefined; - } - for (const line of head.split('\n')) { - let entry: unknown; - try { - entry = JSON.parse(line); - } catch { - continue; - } - if (!entry || typeof entry !== 'object') continue; - if ('cwd' in entry && typeof entry.cwd === 'string') return entry.cwd; - if ('type' in entry && entry.type === 'session_meta' && 'payload' in entry && entry.payload - && typeof entry.payload === 'object' && 'cwd' in entry.payload && typeof entry.payload.cwd === 'string') { - return entry.payload.cwd; - } - if ('type' in entry && entry.type === 'session.start' && 'data' in entry && entry.data && typeof entry.data === 'object' - && 'context' in entry.data && entry.data.context && typeof entry.data.context === 'object' - && 'cwd' in entry.data.context && typeof entry.data.context.cwd === 'string') { - return entry.data.context.cwd; - } - } - return undefined; -} - -/** - * Each event's data home key. The log is hand-editable: a key that is not a - * string counts as absent. An unreleased build of #795 recorded the data home - * path in place of its key. - */ -async function keysOf(events: DashboardEvent[]): Promise> { - const hashes = new Map>(); - return Promise.all(events.map((e) => { - if (typeof e.dataHomeKey === 'string') return e.dataHomeKey; - if (typeof e.dataHome !== 'string') return undefined; - let key = hashes.get(e.dataHome); - if (!key) { - key = dataHomeKey(e.dataHome); - hashes.set(e.dataHome, key); - } - return key; - })); -} - -/** - * The runs of a log: each event's run (none for an exit whose run is gone), - * each run's ID, and the event that decides its scope. - * - * A tool's own session ID is one run, whatever ends it records: `claude - * --resume` continues it, in a new process, and its Stop carries the whole - * transcript. It is returned under the ID itself, so it stays the session - * already reported after compaction drops its events. A PID-fallback ID - * (`pid-…`) names one run until it ends, then comes back for a later one, - * maybe in another scope, so each run is decided on its own and returned under - * the ID plus its first event's timestamp, which stays the same whichever - * earlier runs compaction has dropped. A second end with nothing recorded since - * the first (the dashboard monitor's process_exit after SessionEnd) belongs to - * the run just closed. A start from another process than its open run's begins - * a new run, though nothing ended that one (a crash with no dashboard running). - * The monitor's exit names the last event it observed (`processExitAfter`). - * A dashboard started before that field existed wrote none; since a dead - * process records nothing more, such an exit followed by more events of its - * fallback ID before the next start did not end the open run: it was observed - * before that run began, and belongs to the run closed before it. - */ -function splitRuns(events: DashboardEvent[], eventKeys: Array): { - runOf: Array; - runIds: string[]; - deciding: Array; -} { - const runOf: Array = []; - const runPid: Array = []; - const observedRuns = new Map(); - const openRun = new Map(); - const closedRun = new Map(); - const deciding: Array = []; - const runIds: string[] = []; - // Whether each event is followed by activity of its ID (anything but a start - // or a monitor exit) before that ID's next start. - const continues: boolean[] = []; - const active = new Map(); - for (let i = events.length - 1; i >= 0; i--) { - const { sessionId, type } = events[i]; - continues[i] = active.get(sessionId) ?? false; - if (type === 'session_start') active.set(sessionId, false); - else if (type !== 'process_exit') active.set(sessionId, true); - } - events.forEach((e, i) => { - const fallback = e.sessionId.startsWith('pid-'); - const ends = e.type === 'session_end' || e.type === 'process_exit'; - const observed = e.type === 'process_exit' && typeof e.processExitAfter === 'string'; - const starts = e.type === 'session_start' && typeof e.monitorPid === 'number'; - const open = openRun.get(e.sessionId); - if (starts && open !== undefined && fallback - && typeof runPid[open] === 'number' && runPid[open] !== e.monitorPid) { - // Nothing ended it (a crash), but a late exit of it must still find it. - openRun.delete(e.sessionId); - closedRun.set(e.sessionId, open); - } - const closed = closedRun.get(e.sessionId); - const stale = e.type === 'process_exit' && !observed && fallback && open !== undefined && closed !== undefined - && continues[i]; - let run = observed ? observedRuns.get(`${e.sessionId}@${e.processExitAfter}`) - : stale ? closed : openRun.get(e.sessionId) ?? (ends ? closed : undefined); - // The observed run may have been compacted. Its delayed exit must not - // manufacture a new session or close a later reuse of the same ID. - if (observed && run === undefined) { - runOf.push(undefined); - return; - } - if (run === undefined) { - run = deciding.push(undefined) - 1; - runIds.push(fallback ? `${e.sessionId}@${e.timestamp}` : e.sessionId); - } - if (ends && fallback && !stale && (!observed || openRun.get(e.sessionId) === run)) { - openRun.delete(e.sessionId); - closedRun.set(e.sessionId, run); - } else if (!ends || !fallback) { - openRun.set(e.sessionId, run); - } - if (starts) runPid[run] ??= e.monitorPid; - runOf.push(run); - observedRuns.set(`${e.sessionId}@${e.timestamp}`, run); - const current = deciding[run]; - if (eventKeys[i] !== undefined ? current === undefined || eventKeys[current] === undefined - : current === undefined && !!e.cwd) deciding[run] = i; - }); - return { runOf, runIds, deciding }; -} - -/** Every event of the log under its run ID, whichever scope it belongs to. */ -async function runsOfLog(events: DashboardEvent[]): Promise { - const { runOf, runIds } = splitRuns(events, await keysOf(events)); - return events.flatMap((e, i) => { - const run = runOf[i]; - return run !== undefined ? [{ ...e, sessionId: runIds[run] }] : []; - }); -} - -// ─── Session owners ───────────────────────────────────── -// -// ~/.teamai/dashboard/session-owners.jsonl {"sessionId":"","dataHomeKey":""} per line -// -// A tool's own session ID can be resumed anywhere, long after compaction -// dropped its events, and its Stop carries the whole transcript. So the scope -// that first reports it records itself here, append-only, and the session -// stays that scope's. Only the ID and the key, never a path (#666). The first -// line for an ID wins. -// -// A release before this file kept per-scope snapshots only, so the file is -// first written from them: each tool's own ID in any snapshot of a scope is -// the scope's that holds its greatest total (prompts, then tokens), since a -// session that release split per event holds only part of it elsewhere. A tie -// names no owner: that release copied the shared file into every scope. The -// scopes read are the user scope, every partition, and a project whose data -// home is in its workspace that a session still in the log leads to. Each -// report also records the IDs of its own snapshots that have no owner yet, -// for such a project the log no longer leads to, when they show it reported -// them: absent from the shared snapshot, or past its total there. -// - -function sessionOwnersPath(): string { - return path.join(getTeamaiHomeDir(), 'dashboard', 'session-owners.jsonl'); -} - -/** The data homes whose snapshots an earlier release may have written. */ -async function knownDataHomes(): Promise { - const slugs = await fs.promises.readdir(projectsRootDir()).catch(() => []); - const homes = [getTeamaiHomeDir(), ...[...slugs].sort().map((slug) => path.join(projectsRootDir(), slug))]; - // A project whose data home is in its workspace is under no partition; a - // session of it still in the log leads to it. - const { resolveConfigForDir } = await import('./config.js'); - const cwds = new Set((await readEvents()).flatMap((e) => (typeof e.cwd === 'string' ? [e.cwd] : []))); - for (const cwd of cwds) { - const config = (await pathExists(cwd)) ? await resolveConfigForDir(cwd) : null; - if (config) homes.push(getDataHome(config)); - } - return [...new Set(homes.map((home) => path.resolve(home)))]; -} - -/** A snapshot entry's reported prompts and tokens, 0 for what it does not hold. */ -function reportedSize(entry: unknown): { prompts: number; tokens: number } { - if (!entry || typeof entry !== 'object') return { prompts: 0, tokens: 0 }; - const prompts = 'prompts' in entry && typeof entry.prompts === 'number' ? entry.prompts : 0; - const tokens = 'tokens' in entry && entry.tokens && typeof entry.tokens === 'object' - ? Object.values(entry.tokens).reduce((sum: number, n: unknown) => sum + (typeof n === 'number' ? n : 0), 0) - : 0; - return { prompts, tokens }; -} - -/** A scope's three snapshots (the shared ones for no data home), as a file holds them. */ -interface ScopeSnapshots { - interventions: Record | null; - promptTokens: Record | null; - daily: Record | null; -} - -async function readSnapshotsIn(dataHome: string | undefined): Promise { - const read = (name: ReportedSnapshotName) => - readJson>(dataHome === undefined ? sharedSnapshotPath(name) : snapshotPathIn(dataHome, name)); - const [interventions, promptTokens, daily] = await Promise.all(REPORTED_SNAPSHOTS.map(read)); - return { interventions, promptTokens, daily }; -} - -/** An intervention entry read from a file, zero for what it does not hold. */ -function interventionsEntry(value: unknown): ReportedInterventions[string] { - const count = (field: 'interrupt' | 'toolReject' | 'correction') => - (value && typeof value === 'object' && field in value && typeof Object.entries(value).find(([k]) => k === field)?.[1] === 'number' - ? Number(Object.entries(value).find(([k]) => k === field)?.[1]) : 0); - return { interrupt: count('interrupt'), toolReject: count('toolReject'), correction: count('correction') }; -} - -/** A prompt-token entry read from a file, zero for what it does not hold. */ -function promptTokensEntry(value: unknown): ReportedPromptTokens[string] { - const tokens = emptyTokenUsage(); - if (value && typeof value === 'object' && 'tokens' in value && value.tokens && typeof value.tokens === 'object') { - for (const [field, n] of Object.entries(value.tokens)) { - if (typeof n !== 'number') continue; - if (field === 'input' || field === 'output' || field === 'cacheRead' || field === 'cacheCreation') tokens[field] = n; - } - } - return { prompts: reportedSize(value).prompts, tokens }; -} - -/** What a scope's snapshots hold of `id`: prompts, tokens and intervention counts. */ -function heldIn(snapshots: ScopeSnapshots, id: string): { prompts: number; tokens: number; interventions: number } { - const fromTokens = reportedSize(snapshots.promptTokens?.[id]); - const iv = interventionsEntry(snapshots.interventions?.[id]); - return { - prompts: Math.max(fromTokens.prompts, reportedSize(snapshots.daily?.[id]).prompts), - tokens: fromTokens.tokens, - interventions: iv.interrupt + iv.toolReject + iv.correction, - }; -} - -/** - * Whether a scope's snapshots show it reported `id`: the shared snapshots hold - * none of it, or the scope is past their total. An earlier release copied the - * shared snapshots into every scope it ran in, so a copy shows nothing. - */ -function showsReported(own: ScopeSnapshots, shared: ScopeSnapshots, id: string): boolean { - const inShared = [shared.interventions, shared.promptTokens, shared.daily].some((snapshot) => - !!snapshot && typeof snapshot === 'object' && Object.hasOwn(snapshot, id)); - if (!inShared) return true; - const a = heldIn(own, id); - const b = heldIn(shared, id); - return a.prompts !== b.prompts ? a.prompts > b.prompts - : a.tokens !== b.tokens ? a.tokens > b.tokens : a.interventions > b.interventions; -} - -/** What several scopes had reported of one session, credited to its owner (numbers only). */ -interface OwnerCredit { - interventions: ReportedInterventions[string]; - promptTokens: ReportedPromptTokens[string]; - daily?: DailySessionSnapshot; -} - -/** - * The credit for a session an earlier release split across scopes per event, - * from their snapshots alone (its events are gone). A part whose daily entry - * shows it ended in a Stop carries the transcript's cumulative total, so the - * greatest such part counts once; a part with no Stop counted its own prompts, - * so those add. Intervention counts, per event, add; tokens, from Stops, take - * the greatest. A part without a Stop that came before another's Stop is - * credited twice: that undercounts, once, but never sends a prompt again. - */ -function creditOf(parts: ScopeSnapshots[], id: string): OwnerCredit { - const stops = parts.filter((part) => parseDailySnapshot(part.daily?.[id]) !== undefined); - const loose = parts.filter((part) => !stops.includes(part)); - const prompts = Math.max(0, ...stops.map((part) => heldIn(part, id).prompts)) - + loose.reduce((sum, part) => sum + heldIn(part, id).prompts, 0); - const tokens = emptyTokenUsage(); - const interventions = { interrupt: 0, toolReject: 0, correction: 0 }; - for (const part of parts) { - const entry = promptTokensEntry(part.promptTokens?.[id]); - for (const field of ['input', 'output', 'cacheRead', 'cacheCreation'] as const) { - tokens[field] = Math.max(tokens[field], entry.tokens[field]); - } - const iv = interventionsEntry(part.interventions?.[id]); - interventions.interrupt += iv.interrupt; - interventions.toolReject += iv.toolReject; - interventions.correction += iv.correction; - } - const days = stops.flatMap((part) => { - const day = parseDailySnapshot(part.daily?.[id]); - return day ? [day] : []; - }); - const latest = days.reduce((a, b) => (!a || b.prompts > a.prompts ? b : a), undefined); - return { - interventions, - promptTokens: { prompts, tokens }, - ...(latest ? { daily: { ...latest, prompts, durationMs: days.reduce((sum, day) => sum + day.durationMs, 0) } } : {}), - }; -} - -/** - * The owners the per-scope snapshots of an earlier release imply, as the file's - * first lines: for each tool's own ID, the scope whose snapshots show it reported - * it with the greatest total, and, when several did, the credit of their parts. - * A tie names no owner. - */ -async function ownersFromSnapshots(): Promise { - const shared = await readSnapshotsIn(undefined); - const parts = new Map>(); - for (const dataHome of await knownDataHomes()) { - const snapshots = await readSnapshotsIn(dataHome); - const ids = new Set([snapshots.interventions, snapshots.promptTokens, snapshots.daily].flatMap((snapshot) => - (snapshot && typeof snapshot === 'object' ? Object.keys(snapshot) : []))); - const key = await dataHomeKey(dataHome); - for (const sessionId of ids) { - if (sessionId.startsWith('pid-') || !showsReported(snapshots, shared, sessionId)) continue; - parts.set(sessionId, [...(parts.get(sessionId) ?? []), { key, snapshots }]); - } - } - const lines: string[] = []; - for (const [sessionId, held] of parts) { - const size = (part: { snapshots: ScopeSnapshots }) => heldIn(part.snapshots, sessionId); - const best = held.reduce((a, b) => { - const x = size(a); - const y = size(b); - return y.prompts > x.prompts || (y.prompts === x.prompts && y.tokens > x.tokens) ? b : a; - }); - const tied = held.some((part) => part.key !== best.key - && size(part).prompts === size(best).prompts && size(part).tokens === size(best).tokens); - if (tied) continue; - const credit = held.length > 1 ? creditOf(held.map((part) => part.snapshots), sessionId) : undefined; - lines.push(JSON.stringify({ sessionId, dataHomeKey: best.key, ...(credit ? { credit } : {}) })); - } - return lines.map((line) => `${line}\n`).join(''); -} - -/** The credits the file's first line for each ID carries (see {@link creditOf}). */ -async function readOwnerCredits(): Promise> { - const credits = new Map(); - const seen = new Set(); - for (const line of ((await readFileSafe(sessionOwnersPath())) ?? '').split('\n')) { - let parsed: unknown; - try { - parsed = JSON.parse(line); - } catch { - continue; - } - if (!parsed || typeof parsed !== 'object' || !('sessionId' in parsed) || typeof parsed.sessionId !== 'string') continue; - if (seen.has(parsed.sessionId)) continue; - seen.add(parsed.sessionId); - if (!('credit' in parsed) || !parsed.credit || typeof parsed.credit !== 'object') continue; - const credit = parsed.credit; - const daily = 'daily' in credit ? parseDailySnapshot(credit.daily) : undefined; - credits.set(parsed.sessionId, { - interventions: interventionsEntry('interventions' in credit ? credit.interventions : undefined), - promptTokens: promptTokensEntry('promptTokens' in credit ? credit.promptTokens : undefined), - ...(daily ? { daily } : {}), - }); - } - return credits; -} - -async function readSessionOwners(): Promise> { - const owners = new Map(); - if (!(await pathExists(sessionOwnersPath()))) { - try { - await ensureDir(path.dirname(sessionOwnersPath())); - // Exclusive: a report in another scope may be writing it too. - await fs.promises.writeFile(sessionOwnersPath(), await ownersFromSnapshots(), { flag: 'wx' }); - } catch (e) { - if ((e as NodeJS.ErrnoException).code !== 'EEXIST') log.debug(`Could not seed session owners: ${(e as Error).message}`); - } - } - const content = await readFileSafe(sessionOwnersPath()); - for (const line of (content ?? '').split('\n')) { - if (!line.trim()) continue; - try { - const parsed: unknown = JSON.parse(line); - if (parsed && typeof parsed === 'object' && 'sessionId' in parsed && 'dataHomeKey' in parsed - && typeof parsed.sessionId === 'string' && typeof parsed.dataHomeKey === 'string' - && !owners.has(parsed.sessionId)) owners.set(parsed.sessionId, parsed.dataHomeKey); - } catch { - // A torn or hand-edited line records no owner. - } - } - return owners; -} - -/** The IDs of a scope's snapshots that show it reported them (see {@link showsReported}). */ -async function reportedBeyondShared( - promptTokens: Record, - interventions: Record, - daily: Record, -): Promise { - const shared = await readSnapshotsIn(undefined); - const own: ScopeSnapshots = { interventions, promptTokens, daily }; - const ids = new Set([...Object.keys(promptTokens), ...Object.keys(interventions), ...Object.keys(daily)]); - return [...ids].filter((id) => showsReported(own, shared, id)); -} - -/** Records `key` as the owner of the tool-own session IDs among `sessionIds` that have none yet. */ -async function recordSessionOwners(sessionIds: Iterable, key: string): Promise { - const owners = await readSessionOwners(); - const ids = new Set([...sessionIds].filter((id) => !id.startsWith('pid-') && !owners.has(id))); - if (ids.size === 0) return; - try { - await ensureDir(path.dirname(sessionOwnersPath())); - await fs.promises.appendFile(sessionOwnersPath(), - [...ids].map((sessionId) => JSON.stringify({ sessionId, dataHomeKey: key })).join('\n') + '\n'); - } catch (e) { - log.debug(`Could not record session owners: ${(e as Error).message}`); - } -} - /** * A session main split across scopes, crediting every part it reported. Main * (#795) gave each event to the scope its `dataHome` names and reported each @@ -1018,7 +470,7 @@ async function recordSessionOwners(sessionIds: Iterable, key: string): P * run already reported past it is left alone; runs whose events name at most * one data home are untouched. */ -export async function creditSplitRuns( +async function creditSplitRuns( events: DashboardEvent[], reported: { interventions: ReportedInterventions; promptTokens: ReportedPromptTokens; daily: ReportedDailySessions }, ): Promise<{ @@ -1076,6 +528,53 @@ export async function creditSplitRuns( return result; } +/** + * A scope's reported snapshots as its report and `teamai stats` compare them: + * bare entries adopted by the runs of `events` (prompt tokens first, since they + * decide which runs of a bare ID were reported), and a session an earlier + * release split across scopes credited with every part. With `persist`, what + * changed is written out at once, even with nothing to report, so the report's + * success writes, which merge into the files, cannot bring a retired entry back. + */ +export async function reportedBaselines( + events: DashboardEvent[], + metrics: Map, + currentDaily: Map, + config: LocalConfig | undefined, + persist: boolean, +): Promise<{ interventions: ReportedInterventions; promptTokens: ReportedPromptTokens; daily: ReportedDailySessions }> { + const adopt = async ( + read: (config: LocalConfig | undefined) => Promise>, + write: (data: Record, config: LocalConfig | undefined) => Promise, + current: Record, + take: TakeReported, + ): Promise> => { + const stored = await read(config); + const adopted = adoptBareKeys(stored, events, current, take); + if (persist && adopted !== stored) await write(adopted, config); + return adopted; + }; + const promptTokens = await adopt( + readReportedPromptTokens, writeReportedPromptTokens, computePromptTokenDelta(metrics, {}).nextReported, takePromptTokens, + ); + const covered = (runId: string) => Object.hasOwn(promptTokens, runId); + const interventions = await adopt( + readReportedInterventions, writeReportedInterventions, Object.fromEntries(interventionCounts(metrics)), + takeInterventions(covered), + ); + const daily = await adopt( + readReportedDailySessions, writeReportedDailySessions, computeDailyStatsDelta(currentDaily, {}).nextReported, + takeDaily(covered), + ); + const credited = await creditSplitRuns(events, { interventions, promptTokens, daily }); + if (persist && credited.changed) { + await writeReportedInterventions(credited.interventions, config); + await writeReportedPromptTokens(credited.promptTokens, config); + await writeReportedDailySessions(credited.daily, config); + } + return { interventions: credited.interventions, promptTokens: credited.promptTokens, daily: credited.daily }; +} + /** * A run's share of what is left of a bare snapshot entry, or undefined when * nothing is left: that run and every later one of its ID were not reported. @@ -1184,46 +683,9 @@ export async function reportUsageToTeam( const currentInterventions = interventionCounts(metrics); const currentDaily = aggregateDailySessions(dashboardEvents); - // A retired bare entry is written out now, even with nothing to report, so - // the success writes below, which merge into the file, cannot bring it back. - const adopt = async ( - read: (config: LocalConfig | undefined) => Promise>, - write: (data: Record, config: LocalConfig | undefined) => Promise, - current: Record, - take: TakeReported, - ): Promise> => { - const stored = await read(reportsConfig); - const adopted = adoptBareKeys(stored, dashboardEvents, current, take); - if (adopted !== stored) await write(adopted, reportsConfig); - return adopted; - }; - // Prompt tokens first: they decide which runs of a bare ID were reported. - const adoptedPromptTokens = await adopt( - readReportedPromptTokens, writeReportedPromptTokens, computePromptTokenDelta(metrics, {}).nextReported, - takePromptTokens, - ); - const covered = (runId: string) => Object.hasOwn(adoptedPromptTokens, runId); - const adoptedInterventions = await adopt( - readReportedInterventions, writeReportedInterventions, Object.fromEntries(currentInterventions), - takeInterventions(covered), - ); - const adoptedDailySessions = await adopt( - readReportedDailySessions, writeReportedDailySessions, computeDailyStatsDelta(currentDaily, {}).nextReported, - takeDaily(covered), - ); - // A session main split across scopes is credited with every part reported, - // once, before any delta: written out now, like a retired bare entry. - const credited = await creditSplitRuns(dashboardEvents, { - interventions: adoptedInterventions, promptTokens: adoptedPromptTokens, daily: adoptedDailySessions, - }); - if (credited.changed) { - await writeReportedInterventions(credited.interventions, reportsConfig); - await writeReportedPromptTokens(credited.promptTokens, reportsConfig); - await writeReportedDailySessions(credited.daily, reportsConfig); - } - const reportedPromptTokens = credited.promptTokens; - const reportedInterventions = credited.interventions; - const reportedDailySessions = credited.daily; + const { + interventions: reportedInterventions, promptTokens: reportedPromptTokens, daily: reportedDailySessions, + } = await reportedBaselines(dashboardEvents, metrics, currentDaily, reportsConfig, true); const { delta: promptTokenDelta, nextReported: nextReportedPromptTokens } = computePromptTokenDelta( metrics, reportedPromptTokens, From 37b531949b02f3542858a2c89fb0f2c1a6b2fea1 Mon Sep 17 00:00:00 2001 From: Saul Moro Date: Fri, 25 Sep 2026 00:33:10 +0200 Subject: [PATCH 29/41] test(stats): real CLI resume of a compacted session from a workspace-data project (#785) A non-git project W keeps its data home in its workspace. W reports a Claude session; with no owners index and every W event compacted, the session is resumed in git project Q through the real hook dispatcher, appending to W's transcript. Q reports only its own session and W the resumed turn; on a build without the transcript origin, Q reports both. The fixture gains a second project and hooks sent as the installed ones send them. --- src/__tests__/e2e/report-scope-reuse.test.ts | 98 ++++++++++++++++---- 1 file changed, 81 insertions(+), 17 deletions(-) diff --git a/src/__tests__/e2e/report-scope-reuse.test.ts b/src/__tests__/e2e/report-scope-reuse.test.ts index 252ade399..4fb9ce8be 100644 --- a/src/__tests__/e2e/report-scope-reuse.test.ts +++ b/src/__tests__/e2e/report-scope-reuse.test.ts @@ -32,10 +32,8 @@ function fixture() { const project = path.join(sandbox, 'project'); const dataHome = path.join(project, '.teamai'); const remote = path.join(sandbox, 'remote.git'); - const clone = path.join(dataHome, 'team-repo'); const logDir = path.join(home, '.teamai', 'dashboard'); const logPath = path.join(logDir, 'events.jsonl'); - fs.mkdirSync(path.join(project, '.claude'), { recursive: true }); fs.mkdirSync(logDir, { recursive: true }); const env = { ...process.env, HOME: home, USERPROFILE: home, XDG_CONFIG_HOME: path.join(home, '.config'), @@ -44,16 +42,40 @@ function fixture() { GIT_COMMITTER_NAME: 'Scope Test', GIT_COMMITTER_EMAIL: 'test@example.invalid', NO_COLOR: '1', }; const git = (...args: string[]) => execFileSync('git', args, { cwd: project, env, encoding: 'utf8', stdio: 'pipe' }); - git('init', '-q', '-b', 'main'); - fs.writeFileSync(path.join(project, 'teamai.yaml'), YAML.stringify({ team: 'scope-test', repo: remote, provider: 'git' })); - git('add', 'teamai.yaml'); - git('commit', '-qm', 'fixture'); - git('clone', '-q', '--bare', project, remote); - git('clone', '-q', remote, clone); - fs.writeFileSync(path.join(dataHome, 'config.yaml'), YAML.stringify({ - repo: { localPath: clone, remote, kind: 'git' }, username: 'tester', scope: 'project', projectRoot: project, - updatePolicy: 'skip', enabledAgents: ['claude'], additionalRoles: [], - })); + /** + * A project at `root` with its own team at `teamRemote`. A git project moves + * its data home to a partition on its first pull; without git (`git: false`) + * it stays in the workspace, `/.teamai`, under no partition. + */ + const makeProject = (root: string, teamRemote: string, { git: inGit = true } = {}) => { + const seed = inGit ? root : `${root}-team-seed`; + const at = (...args: string[]) => execFileSync('git', args, { cwd: seed, env, encoding: 'utf8', stdio: 'pipe' }); + fs.mkdirSync(path.join(root, '.claude'), { recursive: true }); + fs.mkdirSync(seed, { recursive: true }); + at('init', '-q', '-b', 'main'); + fs.writeFileSync(path.join(seed, 'teamai.yaml'), YAML.stringify({ team: 'scope-test', repo: teamRemote, provider: 'git' })); + at('add', 'teamai.yaml'); + at('commit', '-qm', 'fixture'); + at('clone', '-q', '--bare', seed, teamRemote); + const teamClone = path.join(root, '.teamai', 'team-repo'); + at('clone', '-q', teamRemote, teamClone); + fs.writeFileSync(path.join(root, '.teamai', 'config.yaml'), YAML.stringify({ + repo: { localPath: teamClone, remote: teamRemote, kind: 'git' }, username: 'tester', scope: 'project', projectRoot: root, + updatePolicy: 'skip', enabledAgents: ['claude'], additionalRoles: [], + })); + return { + pull: () => execFileSync(process.execPath, [cli, 'pull'], { cwd: root, env, encoding: 'utf8', stdio: 'pipe', timeout: 30_000 }), + stats: () => YAML.parse(at('--git-dir', teamRemote, 'show', 'teamai-reports:stats/tester.yaml')), + reportsHead: () => { + try { return at('--git-dir', teamRemote, 'rev-parse', '--verify', '-q', 'teamai-reports'); } catch { return ''; } + }, + /** A hook event the way the installed hooks send it. */ + hook: (hookName: string, data: Record) => execFileSync(process.execPath, + [cli, 'hook-dispatch', hookName, '--tool', 'claude', '--stdin'], + { cwd: root, env, input: JSON.stringify({ cwd: root, ...data }), encoding: 'utf8', stdio: 'pipe', timeout: 30_000 }), + }; + }; + const main = makeProject(project, remote); const writeLog = (events: DashboardEvent[]) => fs.writeFileSync(logPath, events.map(e => JSON.stringify(e)).join('\n') + '\n'); /** The shared snapshots an earlier release left, `prompts` reported per bare session ID. */ const writeSharedSnapshots = (prompts: Record, date: string) => { @@ -67,12 +89,13 @@ function fixture() { fs.writeFileSync(path.join(logDir, `reported-${name}.json`), JSON.stringify(value)); } }; - const pull = () => execFileSync(process.execPath, [cli, 'pull'], { cwd: project, env, encoding: 'utf8', stdio: 'pipe', timeout: 30_000 }); - const stats = () => YAML.parse(git('--git-dir', remote, 'show', 'teamai-reports:stats/tester.yaml')); - const reportsHead = () => { - try { return git('--git-dir', remote, 'rev-parse', '--verify', '-q', 'teamai-reports'); } catch { return ''; } + const { pull, stats, reportsHead, hook } = main; + /** Another project with its own team. */ + const addProject = (name: string, options: { git?: boolean } = {}) => { + const root = path.join(sandbox, name); + return { root, ...makeProject(root, path.join(sandbox, `${name}-remote.git`), options) }; }; - return { project, dataHome, remote, logPath, env, git, writeLog, writeSharedSnapshots, pull, stats, reportsHead }; + return { home, project, dataHome, remote, logPath, env, git, writeLog, writeSharedSnapshots, pull, stats, reportsHead, hook, addProject }; } it('real CLI preserves a path-keyed reuse and replays a delayed exit emitted by the real monitor', async () => { @@ -164,3 +187,44 @@ it('real CLI sends no run of an aggregated bare ID again and splits a reused ID pull(); expect(stats()).toMatchObject({ prompts: 2, interventions: { sessions: 2 } }); }, 60_000); + +it('real CLI gives a session resumed in another project after compaction to the project its transcript started in', () => { + const { home, logPath, addProject } = fixture(); + // W keeps its data home in the workspace; Q is a git project. + const w = addProject('project-w', { git: false }); + const q = addProject('project-q'); + const transcript = path.join(home, '.claude', 'projects', 'w', 'resumed-e2e.jsonl'); + fs.mkdirSync(path.dirname(transcript), { recursive: true }); + const turn = (sessionId: string, cwd: string, n: number) => JSON.stringify({ + type: 'user', sessionId, uuid: `${sessionId}-${n}`, cwd, timestamp: new Date().toISOString(), + message: { role: 'user', content: `turn ${n}` }, + }); + const run = (project: typeof w, sessionId: string, transcriptPath: string) => { + project.hook('prompt-submit', { session_id: sessionId, hook_event_name: 'UserPromptSubmit', prompt: 'turn', transcript_path: transcriptPath }); + project.hook('stop', { session_id: sessionId, hook_event_name: 'Stop', transcript_path: transcriptPath }); + }; + // W records and reports the session. + fs.writeFileSync(transcript, turn('resumed-e2e', w.root, 1) + '\n'); + run(w, 'resumed-e2e', transcript); + w.pull(); + expect(w.stats()).toMatchObject({ prompts: 1, interventions: { sessions: 1 } }); + expect(fs.existsSync(path.join(w.root, '.teamai', 'dashboard', 'reported-prompt-tokens.json'))).toBe(true); + // As if an earlier release had reported it: no owners index. Compaction then + // drops every W event, so nothing on disk outside W points to W. + fs.rmSync(path.join(home, '.teamai', 'dashboard', 'session-owners.jsonl'), { force: true }); + fs.writeFileSync(logPath, ''); + + // `claude --resume` in Q appends to the same transcript; Q also runs a session of its own. + fs.appendFileSync(transcript, turn('resumed-e2e', q.root, 2) + '\n'); + run(q, 'resumed-e2e', transcript); + const own = path.join(home, '.claude', 'projects', 'q', 'q-own.jsonl'); + fs.mkdirSync(path.dirname(own), { recursive: true }); + fs.writeFileSync(own, turn('q-own', q.root, 1) + '\n'); + run(q, 'q-own', own); + q.pull(); + w.pull(); + + // Q reports only its own session; W reports the resumed turn. + expect(q.stats()).toMatchObject({ prompts: 1, interventions: { sessions: 1 } }); + expect(w.stats()).toMatchObject({ prompts: 2, interventions: { sessions: 1 } }); +}, 90_000); From 79f2288edd3095e21620e65b68324933a7837c6a Mon Sep 17 00:00:00 2001 From: Saul Moro Date: Fri, 25 Sep 2026 00:35:52 +0200 Subject: [PATCH 30/41] fix(stats): credit a split session's Stop-derived interventions once (#785) Interruptions and tool rejections come from Stops, which carry the transcript's cumulative counts, so a compacted split session's credit takes the greatest part, as it does for tokens; summing them made the next cumulative Stop report nothing. Corrections are counted per prompt in each part's own events, so they still add. --- docs/designs/data-directory-layout.md | 3 ++- src/__tests__/dashboard-report-scope.test.ts | 28 ++++++++++++++++++++ src/session-owners.ts | 10 ++++--- 3 files changed, 36 insertions(+), 5 deletions(-) diff --git a/docs/designs/data-directory-layout.md b/docs/designs/data-directory-layout.md index 57be9b208..59eea0c11 100644 --- a/docs/designs/data-directory-layout.md +++ b/docs/designs/data-directory-layout.md @@ -388,7 +388,8 @@ reported it (a session that release split per event), the owner's line also carries the credit of their parts, applied once as its baseline: a part whose daily entry shows it ended in a Stop holds the transcript's cumulative total, so the greatest such part counts once, while a part with no Stop counted its own -prompts, which add; intervention counts add, tokens take the greatest. A part +prompts, which add; interruptions, rejections and tokens, from Stops, take the +greatest, and corrections, counted per prompt, add. A part with no Stop that came before another's Stop is then credited twice, which undercounts once but never sends a prompt again. The scopes read are the user scope, every partition, and a project whose data home is in its workspace diff --git a/src/__tests__/dashboard-report-scope.test.ts b/src/__tests__/dashboard-report-scope.test.ts index ae1fa9354..ba9785644 100644 --- a/src/__tests__/dashboard-report-scope.test.ts +++ b/src/__tests__/dashboard-report-scope.test.ts @@ -600,6 +600,34 @@ describe('each scope keeps its own reported snapshot (#786)', () => { expect((await reportedPrompts(project)) + (await reportedPrompts(projectQ))).toBe(1); }); + it('a compacted split session credits the cumulative Stop interventions once', async () => { + const { user, project } = await setup(); + const { rootQ, projectQ } = await setupQ(); + const today = new Date().toISOString().slice(0, 10); + // P's Stop reported 3 prompts and 1 interruption; Q's later, cumulative Stop + // 5 prompts and 2; the user scope 1 prompt before any Stop (no daily entry). + writeSharedSnapshots({ split: 3 }, today, project); + writeSharedSnapshots({ split: 5 }, today, projectQ); + writeSharedSnapshots({ split: 1 }, today, user); + fs.rmSync(path.join(teamaiHome(), 'dashboard', 'user-reported-daily-sessions.json')); + const interrupts = (config: LocalConfig, file: string, interrupt: number) => fs.writeFileSync( + path.join(getDataHome(config), 'dashboard', file), + JSON.stringify({ split: { interrupt, toolReject: 0, correction: 0 } })); + interrupts(project, 'reported-interventions.json', 1); + interrupts(projectQ, 'reported-interventions.json', 2); + // Resumed in Q: its next cumulative Stop carries 7 prompts and 3 interruptions. + await resume('split', rootQ, 7); + const log = path.join(teamaiHome(), 'dashboard', 'events.jsonl'); + fs.writeFileSync(log, fs.readFileSync(log, 'utf-8').split('\n').filter(Boolean).map((line) => { + const event = JSON.parse(line); + return JSON.stringify(event.type === 'stop' ? { ...event, interventions: { interrupt: 3, toolReject: 0 } } : event); + }).join('\n') + '\n'); + const stats = await report(projectQ); + const interventions = stats && typeof stats === 'object' && 'interventions' in stats ? stats.interventions : undefined; + + expect(interventions).toMatchObject({ interrupt: 1 }); + }); + it('a project that reported a session past the shared snapshot\'s total owns it', async () => { const { project } = await setup(); const { rootQ, projectQ } = await setupQ(); diff --git a/src/session-owners.ts b/src/session-owners.ts index 95817cd9b..d5e3639bf 100644 --- a/src/session-owners.ts +++ b/src/session-owners.ts @@ -162,8 +162,8 @@ export interface OwnerCredit { * from their snapshots alone (its events are gone). A part whose daily entry * shows it ended in a Stop carries the transcript's cumulative total, so the * greatest such part counts once; a part with no Stop counted its own prompts, - * so those add. Intervention counts, per event, add; tokens, from Stops, take - * the greatest. A part without a Stop that came before another's Stop is + * so those add. Interruptions, rejections and tokens, from Stops, take the + * greatest; corrections, counted per prompt, add. A part without a Stop that came before another's Stop is * credited twice: that undercounts, once, but never sends a prompt again. */ function creditOf(parts: ScopeSnapshots[], id: string): OwnerCredit { @@ -178,9 +178,11 @@ function creditOf(parts: ScopeSnapshots[], id: string): OwnerCredit { for (const field of ['input', 'output', 'cacheRead', 'cacheCreation'] as const) { tokens[field] = Math.max(tokens[field], entry.tokens[field]); } + // Interruptions and rejections come from Stops, cumulative; corrections + // are counted per prompt in each part's own events. const iv = interventionsEntry(part.interventions?.[id]); - interventions.interrupt += iv.interrupt; - interventions.toolReject += iv.toolReject; + interventions.interrupt = Math.max(interventions.interrupt, iv.interrupt); + interventions.toolReject = Math.max(interventions.toolReject, iv.toolReject); interventions.correction += iv.correction; } const days = stops.flatMap((part) => { From 09b872834631e7798b60467b04bad1e9104963fb Mon Sep 17 00:00:00 2001 From: Saul Moro Date: Fri, 25 Sep 2026 00:42:59 +0200 Subject: [PATCH 31/41] fix(stats): place a compacted split session's parts by its transcript (#785) A split session's credit added a part with no Stop to the greatest cumulative Stop, which already counts that part when it came before the Stop: after 3 prompts in P and a cumulative Stop of 5 in Q it credited 8, and the next Stop of 6 reported nothing. The credit now keeps each part (scope key, prompts, whether it ended in a Stop; numbers only), and the owner places them by the session's transcript, which keeps every prompt in order with the directory it was typed in: the Stop covers the first prompts, and only the part's prompts after those add. With no transcript to place them, they all add, as before. The Stop scan's human-turn test is now isHumanPromptEntry, shared by both, so the two count prompts alike. --- docs/designs/data-directory-layout.md | 9 ++- src/__tests__/dashboard-report-scope.test.ts | 25 +++++++ src/dashboard-collector.ts | 50 +++++++------ src/session-owners.ts | 79 +++++++++++++++++++- src/team-push.ts | 11 ++- 5 files changed, 139 insertions(+), 35 deletions(-) diff --git a/docs/designs/data-directory-layout.md b/docs/designs/data-directory-layout.md index 59eea0c11..f8a47d09f 100644 --- a/docs/designs/data-directory-layout.md +++ b/docs/designs/data-directory-layout.md @@ -389,9 +389,12 @@ carries the credit of their parts, applied once as its baseline: a part whose daily entry shows it ended in a Stop holds the transcript's cumulative total, so the greatest such part counts once, while a part with no Stop counted its own prompts, which add; interruptions, rejections and tokens, from Stops, take the -greatest, and corrections, counted per prompt, add. A part -with no Stop that came before another's Stop is then credited twice, which -undercounts once but never sends a prompt again. The scopes read are the +greatest, and corrections, counted per prompt, add. Whether a part with no +Stop came before another's cumulative Stop, which already counts it, is read +from the session's transcript when it has one (Claude): it keeps every prompt +in order with the directory it was typed in, so the Stop covers the first +prompts and only the part's prompts after those add. With no transcript to +place them they all add, which undercounts once but never sends a prompt again. The scopes read are the user scope, every partition, and a project whose data home is in its workspace that a session still in the log leads to; each report also records the IDs of its own snapshots that have no owner yet and show it reported them (absent from diff --git a/src/__tests__/dashboard-report-scope.test.ts b/src/__tests__/dashboard-report-scope.test.ts index ba9785644..03d051639 100644 --- a/src/__tests__/dashboard-report-scope.test.ts +++ b/src/__tests__/dashboard-report-scope.test.ts @@ -600,6 +600,31 @@ describe('each scope keeps its own reported snapshot (#786)', () => { expect((await reportedPrompts(project)) + (await reportedPrompts(projectQ))).toBe(1); }); + it.each([ + // P's 3 prompts came before Q's cumulative Stop of 5, which counts them already. + ['before', ['p', 'p', 'p', 'q', 'q'], 3, 5, 1], + // Q's Stop counted 2; P's 3 prompts came after it. + ['after', ['q', 'q', 'p', 'p', 'p'], 3, 2, 1], + ])('a compacted split session whose part with no Stop came %s the other part\'s Stop is credited by its transcript', + async (_, order, loose, stop, expected) => { + const { root, project } = await setup(); + const { rootQ, projectQ } = await setupQ(); + const today = new Date().toISOString().slice(0, 10); + writeSharedSnapshots({ split: loose }, today, project); + fs.rmSync(path.join(getDataHome(project), 'dashboard', 'reported-daily-sessions.json')); + writeSharedSnapshots({ split: stop }, today, projectQ); + // The transcript holds every prompt in order, each with the directory it was typed in. + const transcript = path.join(tmp, 'split.jsonl'); + const cwdOf = (at: string) => (at === 'p' ? root : rootQ); + fs.writeFileSync(transcript, [...order, 'q'].map((at, i) => JSON.stringify({ + type: 'user', sessionId: 'split', uuid: `u${i}`, cwd: cwdOf(at), message: { role: 'user', content: `turn ${i}` }, + })).join('\n') + '\n'); + // Resumed in Q, one new prompt: the Stop's transcript total is 6. + await session('claude', { session_id: 'split', cwd: rootQ, transcript_path: transcript }); + + expect((await reportedPrompts(project)) + (await reportedPrompts(projectQ))).toBe(expected); + }); + it('a compacted split session credits the cumulative Stop interventions once', async () => { const { user, project } = await setup(); const { rootQ, projectQ } = await setupQ(); diff --git a/src/dashboard-collector.ts b/src/dashboard-collector.ts index 731a865aa..391625657 100644 --- a/src/dashboard-collector.ts +++ b/src/dashboard-collector.ts @@ -224,6 +224,30 @@ interface JsonlTranscriptScan { } /** Scan the Claude/Codex JSONL transcript once. */ +/** + * Whether a Claude transcript entry is a human turn, what the Stop scan counts + * as a prompt: plain-string user content, or user content with text that is + * neither an interrupt marker nor a system injection; never meta or sidechain. + * One human turn per user entry (tool_result-only entries have no human text). + */ +export function isHumanPromptEntry(entry: unknown): boolean { + if (!entry || typeof entry !== 'object' || !('type' in entry) || entry.type !== 'user') return false; + if (('isMeta' in entry && entry.isMeta === true) || ('isSidechain' in entry && entry.isSidechain === true)) return false; + const message = 'message' in entry && entry.message && typeof entry.message === 'object' ? entry.message : undefined; + const content: unknown = message && 'content' in message ? message.content : undefined; + if (typeof content === 'string') { + const text = content.trim(); + return !!text && !text.startsWith(TRANSCRIPT_INTERRUPT_PREFIX) && !TRANSCRIPT_SYSTEM_PREFIXES.some((p) => text.startsWith(p)); + } + if (!Array.isArray(content)) return false; + return content.some((item: unknown) => { + if (!item || typeof item !== 'object' || !('type' in item) || item.type !== 'text' || !('text' in item)) return false; + if (typeof item.text !== 'string' || item.text.startsWith(TRANSCRIPT_INTERRUPT_PREFIX)) return false; + const text = item.text.trim(); + return !!text && !TRANSCRIPT_SYSTEM_PREFIXES.some((p) => text.startsWith(p)); + }); +} + async function scanJsonlTranscriptOnce(transcriptPath: string, modelAliases?: Record): Promise { let interrupt = 0; let toolReject = 0; @@ -352,33 +376,13 @@ async function scanJsonlTranscriptOnce(transcriptPath: string, modelAliases?: Re if (entry.type !== 'user') continue; - const isMeta = entry.isMeta === true || entry.isSidechain === true; + if (isHumanPromptEntry(entry)) prompts++; const content = entry.message?.content; - - // Plain-string user content = a genuine human prompt (older transcript shape). - if (typeof content === 'string') { - const trimContent = content.trim(); - if ( - !isMeta && - trimContent && - !trimContent.startsWith(TRANSCRIPT_INTERRUPT_PREFIX) && - !TRANSCRIPT_SYSTEM_PREFIXES.some((p) => trimContent.startsWith(p)) - ) { - prompts++; - } - continue; - } if (!Array.isArray(content)) continue; - let hasHumanText = false; for (const item of content as Array>) { if (item?.type === 'text' && typeof item.text === 'string') { - const txt = item.text.trim(); - if (item.text.startsWith(TRANSCRIPT_INTERRUPT_PREFIX)) { - interrupt++; - } else if (txt && !TRANSCRIPT_SYSTEM_PREFIXES.some((p) => txt.startsWith(p))) { - hasHumanText = true; - } + if (item.text.startsWith(TRANSCRIPT_INTERRUPT_PREFIX)) interrupt++; } else if (item?.type === 'tool_result' && item.is_error === true) { const text = typeof item.content === 'string' ? item.content @@ -395,8 +399,6 @@ async function scanJsonlTranscriptOnce(transcriptPath: string, modelAliases?: Re } } } - // One human turn per user entry (tool_result-only entries have no human text). - if (hasHumanText && !isMeta) prompts++; } } catch (e) { log.warn(`dashboard: failed to scan transcript: ${(e as Error).message}`); diff --git a/src/session-owners.ts b/src/session-owners.ts index d5e3639bf..92589df1d 100644 --- a/src/session-owners.ts +++ b/src/session-owners.ts @@ -6,7 +6,7 @@ */ import fs from 'node:fs'; import path from 'node:path'; -import { readEvents, dataHomeKey } from './dashboard-collector.js'; +import { readEvents, dataHomeKey, isHumanPromptEntry } from './dashboard-collector.js'; import { readFileSafe, ensureDir, pathExists, readJson } from './utils/fs.js'; import { log } from './utils/logger.js'; import type { TokenUsage } from './types.js'; @@ -155,8 +155,12 @@ export interface OwnerCredit { interventions: ReportedInterventions[string]; promptTokens: ReportedPromptTokens[string]; daily?: DailySessionSnapshot; + /** Each part's scope key, reported prompts, and whether it ended in a Stop, for {@link creditedPrompts}. */ + parts?: CreditPart[]; } +interface CreditPart { key: string; prompts: number; stop: boolean } + /** * The credit for a session an earlier release split across scopes per event, * from their snapshots alone (its events are gone). A part whose daily entry @@ -164,9 +168,11 @@ export interface OwnerCredit { * greatest such part counts once; a part with no Stop counted its own prompts, * so those add. Interruptions, rejections and tokens, from Stops, take the * greatest; corrections, counted per prompt, add. A part without a Stop that came before another's Stop is - * credited twice: that undercounts, once, but never sends a prompt again. + * credited twice: that undercounts, once, but never sends a prompt again, + * unless the session's transcript can place it (see {@link creditedPrompts}). */ -function creditOf(parts: ScopeSnapshots[], id: string): OwnerCredit { +function creditOf(held: Array<{ key: string; snapshots: ScopeSnapshots }>, id: string): OwnerCredit { + const parts = held.map((part) => part.snapshots); const stops = parts.filter((part) => parseDailySnapshot(part.daily?.[id]) !== undefined); const loose = parts.filter((part) => !stops.includes(part)); const prompts = Math.max(0, ...stops.map((part) => heldIn(part, id).prompts)) @@ -193,6 +199,9 @@ function creditOf(parts: ScopeSnapshots[], id: string): OwnerCredit { return { interventions, promptTokens: { prompts, tokens }, + parts: held.map((part) => ({ + key: part.key, prompts: heldIn(part.snapshots, id).prompts, stop: stops.includes(part.snapshots), + })), ...(latest ? { daily: { ...latest, prompts, durationMs: days.reduce((sum, day) => sum + day.durationMs, 0) } } : {}), }; } @@ -227,7 +236,7 @@ async function ownersFromSnapshots(): Promise { const tied = held.some((part) => part.key !== best.key && size(part).prompts === size(best).prompts && size(part).tokens === size(best).tokens); if (tied) continue; - const credit = held.length > 1 ? creditOf(held.map((part) => part.snapshots), sessionId) : undefined; + const credit = held.length > 1 ? creditOf(held, sessionId) : undefined; lines.push(JSON.stringify({ sessionId, dataHomeKey: best.key, ...(credit ? { credit } : {}) })); } return lines.map((line) => `${line}\n`).join(''); @@ -250,10 +259,18 @@ export async function readOwnerCredits(): Promise> { if (!('credit' in parsed) || !parsed.credit || typeof parsed.credit !== 'object') continue; const credit = parsed.credit; const daily = 'daily' in credit ? parseDailySnapshot(credit.daily) : undefined; + const parts: CreditPart[] = []; + if ('parts' in credit && Array.isArray(credit.parts)) { + for (const part of credit.parts) { + if (!part || typeof part !== 'object' || !('key' in part) || typeof part.key !== 'string') continue; + parts.push({ key: part.key, prompts: reportedSize(part).prompts, stop: 'stop' in part && part.stop === true }); + } + } credits.set(parsed.sessionId, { interventions: interventionsEntry('interventions' in credit ? credit.interventions : undefined), promptTokens: promptTokensEntry('promptTokens' in credit ? credit.promptTokens : undefined), ...(daily ? { daily } : {}), + ...(parts.length > 0 ? { parts } : {}), }); } return credits; @@ -311,3 +328,57 @@ export async function recordSessionOwners(sessionIds: Iterable, key: str } } + +/** + * The prompts a credit covers, placed by the session's transcript, which keeps + * every prompt in order with the directory it was typed in (Claude). A part + * that ended in a Stop counted the transcript's first prompts, cumulatively, + * so the greatest such part covers that many; a part with no Stop counted its + * own scope's first prompts, which add only where they come after those. + * Undefined when the credit mixes no such parts, or no transcript places them. + */ +export async function creditedPrompts(credit: OwnerCredit, transcripts: string[]): Promise { + const parts = credit.parts ?? []; + const stopped = parts.filter((part) => part.stop); + const loose = parts.filter((part) => !part.stop); + if (stopped.length === 0 || loose.length === 0) return undefined; + const covered = Math.max(...stopped.map((part) => part.prompts)); + const { resolveConfigForDir } = await import('./config.js'); + const keys = new Map>(); + const keyOf = (cwd: string) => { + let key = keys.get(cwd); + if (!key) { + key = pathExists(cwd).then(async (exists) => { + const config = exists ? await resolveConfigForDir(cwd) : null; + return config ? dataHomeKey(getDataHome(config)) : undefined; + }); + keys.set(cwd, key); + } + return key; + }; + for (const transcript of [...transcripts].reverse()) { + const content = await readFileSafe(transcript); + if (content === null) continue; + const prompts: string[] = []; + for (const line of content.split('\n')) { + let entry: unknown; + try { + entry = JSON.parse(line); + } catch { + continue; + } + if (!isHumanPromptEntry(entry) || !entry || typeof entry !== 'object' || !('cwd' in entry)) continue; + if (typeof entry.cwd === 'string') prompts.push(entry.cwd); + } + if (prompts.length < covered) continue; + const scopes = await Promise.all(prompts.map(keyOf)); + let total = covered; + for (const part of loose) { + const own = scopes.flatMap((key, i) => (key === part.key ? [i] : [])).slice(0, part.prompts); + if (own.length < part.prompts) return undefined; + total += own.filter((i) => i >= covered).length; + } + return total; + } + return undefined; +} diff --git a/src/team-push.ts b/src/team-push.ts index b6e2c284e..f7732edf9 100644 --- a/src/team-push.ts +++ b/src/team-push.ts @@ -18,7 +18,7 @@ import type { UserStats, UserInterventionStats, SessionMetrics, TokenUsage, Dash import { getVotesDir, getDataHome, emptyTokenUsage, addTokenUsage, usesBranchWorktree } from './types.js'; import { filterEventsByScope, runsOfLog } from './dashboard-scope.js'; import { - interventionsEntry, promptTokensEntry, readOwnerCredits, recordSessionOwners, reportedBeyondShared, reportedSize, + creditedPrompts, interventionsEntry, promptTokensEntry, readOwnerCredits, recordSessionOwners, reportedBeyondShared, reportedSize, sharedSnapshotPath, snapshotPathIn, type ReportedInterventions, type ReportedPromptTokens, type ReportedSegments, type ReportedSnapshotName, } from './session-owners.js'; @@ -488,10 +488,13 @@ async function creditSplitRuns( // Its parts' events are gone: the credit the owners file seeded from their snapshots. const credit = homes.size < 2 ? credits.get(runId) : undefined; if (credit) { - if (reportedSize(result.promptTokens[runId]).prompts >= credit.promptTokens.prompts) continue; - result.promptTokens[runId] = credit.promptTokens; + // Placed by the transcript where it can tell what the parts overlapped. + const transcripts = runEvents.flatMap((e) => (typeof e.transcriptPath === 'string' ? [e.transcriptPath] : [])); + const prompts = (await creditedPrompts(credit, transcripts)) ?? credit.promptTokens.prompts; + if (reportedSize(result.promptTokens[runId]).prompts >= prompts) continue; + result.promptTokens[runId] = { ...credit.promptTokens, prompts }; result.interventions[runId] = credit.interventions; - if (credit.daily) result.daily[runId] = credit.daily; + if (credit.daily) result.daily[runId] = { ...credit.daily, prompts }; result.changed = true; continue; } From de2552ac1d1c583218c67119c9dd8ea6e46df33f Mon Sep 17 00:00:00 2001 From: Saul Moro Date: Fri, 25 Sep 2026 01:04:44 +0200 Subject: [PATCH 32/41] fix(stats): keep a dropped Codex rollout's totals for daily and interventions, and migrate whole entries (#785) An entry from before rollouts were kept is one total. An earlier release rewrote every session in the log on each report, so it covers the rollouts begun by the time its file was last written, read before this report writes it: those still in the log consume it in order, what is left is the dropped rollouts', kept as one prior rollout, and a rollout begun later is new. Rollout B after a compacted A is no longer compared against A's total and lost. A rollout also keeps its Stop's interruptions and rejections, and its dropped totals now reach the intervention and daily sums too, not just prompts and tokens: daily prompt turns and intervention counts of a resumed rollout were compared against the dropped one's. --- docs/designs/data-directory-layout.md | 14 +- .../conversation-token-metrics.test.ts | 23 +++- src/__tests__/dashboard-report-scope.test.ts | 23 ++++ src/dashboard-collector.ts | 27 +++- src/session-owners.ts | 2 +- src/stats.ts | 14 +- src/team-push.ts | 130 ++++++++++++++++-- src/types.ts | 3 +- 8 files changed, 210 insertions(+), 26 deletions(-) diff --git a/docs/designs/data-directory-layout.md b/docs/designs/data-directory-layout.md index f8a47d09f..568b831f8 100644 --- a/docs/designs/data-directory-layout.md +++ b/docs/designs/data-directory-layout.md @@ -415,11 +415,15 @@ its events whose metrics reach its snapshot, and the owner takes the metrics of their union as reported when they exceed its own entry, so parts counted before any Stop carried the transcript's total are neither lost nor sent twice. A transcript-scoped session (Codex) is kept per rollout: a rollout's counters -restart, so its prompt-token entry holds each rollout's reported prompts and -tokens under a hash of the rollout's path, and a rollout compaction has dropped -keeps its reported totals in the session's sum; a later rollout is then reported -in full. An entry written before rollouts were kept is compared as a whole once, -then kept per rollout. +restart, so its prompt-token entry holds each rollout's reported prompts, tokens, +interruptions and rejections under a hash of the rollout's path, and a rollout +compaction has dropped keeps those totals in the session's prompt-token, +intervention and daily sums; a later rollout is then reported in full. An entry +written before rollouts were kept is one total: an earlier release rewrote every +session in the log on each report, so it covers the rollouts begun by the time +its file was last written (read before this report writes it). Those still in +the log consume it in order, what is left is the dropped rollouts', kept as one +prior rollout, and a rollout begun later is new. Compaction also keeps a session whose tool process is still running, so a run an exit from a dashboard before `processExitAfter` marked stopped keeps its start, and its ID. A diff --git a/src/__tests__/conversation-token-metrics.test.ts b/src/__tests__/conversation-token-metrics.test.ts index e59487e10..3fc42d215 100644 --- a/src/__tests__/conversation-token-metrics.test.ts +++ b/src/__tests__/conversation-token-metrics.test.ts @@ -7,7 +7,7 @@ import { aggregateSessionMetrics, rebuildSessions, } from '../dashboard-collector.js'; -import { computePromptTokenDelta, mergePromptTokenStats } from '../team-push.js'; +import { computePromptTokenDelta, droppedRollouts, mergePromptTokenStats } from '../team-push.js'; import { summarizeConversation, formatTokenCount } from '../digest.js'; import type { DashboardEvent, SessionMetrics, TokenUsage, UserStats } from '../types.js'; @@ -389,6 +389,27 @@ describe('aggregateSessionMetrics', () => { expect(again.delta).toEqual({ prompts: 0, tokens: emptyTokens() }); }); + it('an entry from before rollouts were kept covers only the rollouts that existed when it was written', () => { + const tokensA = { input: 500, output: 50, cacheRead: 1_000, cacheCreation: 0 }; + const tokensB = { input: 30, output: 3, cacheRead: 100, cacheCreation: 0 }; + const stop = (path: string, timestamp: string, tokens: TokenUsage, prompts: number): DashboardEvent => ({ + type: 'stop', timestamp, sessionId: 's1', tool: 'codex', transcriptPath: path, tokenScope: 'transcript', tokens, prompts, + }); + const writtenAt = Date.parse('2026-09-01T12:00:00Z'); + // An earlier release reported rollout A as a whole, with no rollouts. + const legacy = { s1: { prompts: 5, tokens: tokensA } }; + const a = stop('/rollouts/a.jsonl', '2026-09-01T11:00:00Z', tokensA, 5); + const b = stop('/rollouts/b.jsonl', '2026-09-01T13:00:00Z', tokensB, 2); + // A was compacted before this build's first report; B came after the entry was written. + const compacted = aggregateSessionMetrics([b]); + expect(computePromptTokenDelta(compacted, legacy, droppedRollouts(compacted, legacy, {}, writtenAt)).delta) + .toEqual({ prompts: 2, tokens: tokensB }); + // A still in the log takes the entry; B is new. + const retained = aggregateSessionMetrics([a, b]); + expect(computePromptTokenDelta(retained, legacy, droppedRollouts(retained, legacy, {}, writtenAt)).delta) + .toEqual({ prompts: 2, tokens: tokensB }); + }); + it('counts a Codex session\'s prompts across its rollouts', () => { const stop = (path: string, prompts: number): DashboardEvent => ({ type: 'stop', timestamp: new Date().toISOString(), sessionId: 's1', tool: 'codex', diff --git a/src/__tests__/dashboard-report-scope.test.ts b/src/__tests__/dashboard-report-scope.test.ts index 03d051639..73f47fb1f 100644 --- a/src/__tests__/dashboard-report-scope.test.ts +++ b/src/__tests__/dashboard-report-scope.test.ts @@ -625,6 +625,29 @@ describe('each scope keeps its own reported snapshot (#786)', () => { expect((await reportedPrompts(project)) + (await reportedPrompts(projectQ))).toBe(expected); }); + it('a Codex rollout resumed after compaction reports its daily prompts and interventions too', async () => { + const { project } = await setup(); + const key = await dataHomeKeyOf(project); + const log = path.join(teamaiHome(), 'dashboard', 'events.jsonl'); + fs.mkdirSync(path.dirname(log), { recursive: true }); + const stop = (rollout: string, prompts: number, toolReject: number) => JSON.stringify({ + type: 'stop', timestamp: new Date().toISOString(), sessionId: 'codex-s', tool: 'codex', dataHomeKey: key, + transcriptPath: path.join(tmp, rollout), tokenScope: 'transcript', prompts, + tokens: { input: 10, output: 1, cacheRead: 0, cacheCreation: 0 }, interventions: { interrupt: 0, toolReject }, + }); + fs.writeFileSync(log, stop('rollout-a.jsonl', 5, 1) + '\n'); + await report(project); + // Compaction dropped rollout A; the resumed rollout B restarts its counters. + fs.writeFileSync(log, stop('rollout-b.jsonl', 2, 1) + '\n'); + const stats = await report(project); + const day = stats && typeof stats === 'object' && 'daily' in stats && stats.daily && typeof stats.daily === 'object' + ? Object.values(stats.daily)[0] : undefined; + const interventions = stats && typeof stats === 'object' && 'interventions' in stats ? stats.interventions : undefined; + + expect(day).toMatchObject({ promptTurns: 7 }); + expect(interventions).toMatchObject({ toolReject: 2 }); + }); + it('a compacted split session credits the cumulative Stop interventions once', async () => { const { user, project } = await setup(); const { rootQ, projectQ } = await setupQ(); diff --git a/src/dashboard-collector.ts b/src/dashboard-collector.ts index 391625657..16d45ffcd 100644 --- a/src/dashboard-collector.ts +++ b/src/dashboard-collector.ts @@ -1838,6 +1838,8 @@ export function aggregateSessionMetrics( const sessionTokens = new Map(); const transcriptTokens = new Map>(); const transcriptPrompts = new Map>(); + const transcriptInterventions = new Map>(); + const transcriptSince = new Map>(); for (const event of events) { let m = map.get(event.sessionId); @@ -1845,6 +1847,12 @@ export function aggregateSessionMetrics( m = { interrupt: 0, toolReject: 0, correction: 0, prompts: 0, tokens: emptyTokenUsage() }; map.set(event.sessionId, m); } + if (typeof event.transcriptPath === 'string') { + const since = transcriptSince.get(event.sessionId) ?? new Map(); + const first = since.get(event.transcriptPath); + if (first === undefined || Date.parse(event.timestamp) < Date.parse(first)) since.set(event.transcriptPath, event.timestamp); + transcriptSince.set(event.sessionId, since); + } if (event.type === 'stop') { if (event.interventions) { @@ -1888,6 +1896,11 @@ export function aggregateSessionMetrics( prompts.set(event.transcriptPath, event.prompts); transcriptPrompts.set(event.sessionId, prompts); } + if (event.type === 'stop' && event.interventions) { + const counts = transcriptInterventions.get(event.sessionId) ?? new Map(); + counts.set(event.transcriptPath, { interrupt: event.interventions.interrupt, toolReject: event.interventions.toolReject }); + transcriptInterventions.set(event.sessionId, counts); + } } else { // Claude, CodeBuddy, and pre-existing events retain latest-Stop semantics. setLatestTokenSnapshot(unscopedTokens, event.sessionId, event); @@ -1909,8 +1922,18 @@ export function aggregateSessionMetrics( for (const segment of segments.values()) total = addTokenUsage(total, segment.tokens); m.tokens = total; const prompts = transcriptPrompts.get(sid); - m.segments = Object.fromEntries([...segments].map(([transcript, segment]) => - [transcript, { prompts: prompts?.get(transcript) ?? 0, tokens: { ...segment.tokens } }])); + const counts = transcriptInterventions.get(sid); + const since = transcriptSince.get(sid); + m.segments = Object.fromEntries([...segments].map(([transcript, segment]) => [transcript, { + prompts: prompts?.get(transcript) ?? 0, tokens: { ...segment.tokens }, + interrupt: counts?.get(transcript)?.interrupt ?? 0, toolReject: counts?.get(transcript)?.toolReject ?? 0, + since: since?.get(transcript) ?? segment.timestamp, + }])); + // A rollout's Stop counts restart too; the session sums them. + const sum = (field: 'interrupt' | 'toolReject') => + Object.values(m.segments ?? {}).reduce((total, segment) => total + segment[field], 0); + m.interrupt = Math.max(m.interrupt, sum('interrupt')); + m.toolReject = Math.max(m.toolReject, sum('toolReject')); } else { const unscoped = unscopedTokens.get(sid); if (unscoped) m.tokens = { ...unscoped.tokens }; diff --git a/src/session-owners.ts b/src/session-owners.ts index 92589df1d..44ba5d6e0 100644 --- a/src/session-owners.ts +++ b/src/session-owners.ts @@ -18,7 +18,7 @@ import { parseDailySnapshot, type DailySessionSnapshot } from './session-trends. export type ReportedInterventions = Record; /** A rollout's reported totals, keyed by a hash of its transcript path (no path is stored). */ -export type ReportedSegments = Record; +export type ReportedSegments = Record; /** Snapshot of already-reported per-session prompt counts + token usage (idempotency basis). */ export type ReportedPromptTokens = Record; diff --git a/src/stats.ts b/src/stats.ts index bdfea9f14..f69c20d8c 100644 --- a/src/stats.ts +++ b/src/stats.ts @@ -156,14 +156,20 @@ async function unreportedDashboardStats( metrics: Map, config: LocalConfig, ): Promise { - const { computeInterventionDelta, computePromptTokenDelta, interventionCounts, reportedBaselines } = await import('./team-push.js'); + const { + computeInterventionDelta, computePromptTokenDelta, droppedRollouts, interventionCounts, reportedBaselines, + snapshotWrittenAt, withDroppedRollouts, + } = await import('./team-push.js'); const { aggregateDailySessions } = await import('./session-trends.js'); // The scope's own snapshots, compared as its report compares them (#786). - const { promptTokens, interventions } = await reportedBaselines(events, metrics, aggregateDailySessions(events), config, false); - const currentInterventions = interventionCounts(metrics); + const writtenAt = await snapshotWrittenAt(config); + const currentDaily = aggregateDailySessions(events); + const { promptTokens, interventions } = await reportedBaselines(events, metrics, currentDaily, config, false); + const dropped = droppedRollouts(metrics, promptTokens, interventions, writtenAt); + const currentInterventions = withDroppedRollouts(interventionCounts(metrics), currentDaily, dropped).interventions; const interventionDelta = computeInterventionDelta(currentInterventions, interventions); - const promptTokenDelta = computePromptTokenDelta(metrics, promptTokens); + const promptTokenDelta = computePromptTokenDelta(metrics, promptTokens, dropped); return { sessions: interventionDelta.delta.sessions, diff --git a/src/team-push.ts b/src/team-push.ts index f7732edf9..1690be05e 100644 --- a/src/team-push.ts +++ b/src/team-push.ts @@ -1,4 +1,5 @@ import YAML from 'yaml'; +import fs from 'node:fs'; import { createHash } from 'node:crypto'; import path from 'node:path'; import { readUsageEvents, truncateUsageAfterReport } from './usage-tracker.js'; @@ -370,10 +371,73 @@ function reportedSegments(entry: unknown): ReportedSegments { if (!entry || typeof entry !== 'object' || !('segments' in entry) || !entry.segments || typeof entry.segments !== 'object') { return segments; } - for (const [key, segment] of Object.entries(entry.segments)) segments[key] = promptTokensEntry(segment); + for (const [key, segment] of Object.entries(entry.segments)) { + const iv = interventionsEntry(segment); + segments[key] = { ...promptTokensEntry(segment), interrupt: iv.interrupt, toolReject: iv.toolReject }; + } return segments; } +/** A reported rollout compaction has dropped: its key and the totals it was reported with. */ +export interface DroppedRollout { key: string; prompts: number; tokens: TokenUsage; interrupt: number; toolReject: number } + +/** + * The rollouts of each transcript-scoped session (Codex) that were reported and + * that compaction has since dropped. A rollout's counters restart, so the + * session's totals must keep what those were reported with, or a later rollout + * is compared against them and reported late or never. + * + * An entry written before rollouts were kept is one total. An earlier release + * rewrote every session in the log on each report, so that total covers the + * rollouts that had begun by `writtenAt`, when its file was last written: those + * still in the log consume it in order, and what is left is the dropped ones', + * kept as one prior rollout. A rollout begun after it is new. Without + * `writtenAt`, every rollout in the log is taken as covered. + */ +export function droppedRollouts( + current: Map, + promptTokens: ReportedPromptTokens, + interventions: ReportedInterventions, + writtenAt?: number, +): Map { + const dropped = new Map(); + for (const [sid, cur] of current) { + if (!cur.segments) continue; + const prev = promptTokens[sid]; + if (prev === undefined) continue; + const previous = reportedSegments(prev); + const present = new Set(Object.keys(cur.segments).map(segmentKey)); + const gone: DroppedRollout[] = []; + if (Object.keys(previous).length > 0) { + for (const [key, segment] of Object.entries(previous)) { + if (present.has(key)) continue; + gone.push({ key, prompts: segment.prompts, tokens: segment.tokens, interrupt: segment.interrupt ?? 0, toolReject: segment.toolReject ?? 0 }); + } + } else { + const total = promptTokensEntry(prev); + const iv = interventionsEntry(interventions[sid]); + const left = { prompts: total.prompts, tokens: { ...total.tokens }, interrupt: iv.interrupt, toolReject: iv.toolReject }; + const covered = Object.values(cur.segments) + .filter((segment) => writtenAt === undefined || Date.parse(segment.since) <= writtenAt) + .sort((a, b) => Date.parse(a.since) - Date.parse(b.since)); + for (const segment of covered) { + const take = (own: number, rest: number) => Math.max(0, Math.min(own, rest)); + left.prompts -= take(segment.prompts, left.prompts); + left.interrupt -= take(segment.interrupt, left.interrupt); + left.toolReject -= take(segment.toolReject, left.toolReject); + for (const field of ['input', 'output', 'cacheRead', 'cacheCreation'] as const) { + left.tokens[field] -= take(segment.tokens[field], left.tokens[field]); + } + } + if (left.prompts > 0 || left.interrupt > 0 || left.toolReject > 0 || hasPromptTokenDelta({ prompts: 0, tokens: left.tokens })) { + gone.push({ key: 'prior', ...left }); + } + } + if (gone.length > 0) dropped.set(sid, gone); + } + return dropped; +} + /** * Compute the prompt-count + token delta to report: for each current session, the * positive change since it was last reported. Idempotent (a re-run reports nothing @@ -381,15 +445,14 @@ function reportedSegments(entry: unknown): ReportedSegments { * sessions still present in events.jsonl (compacted sessions stay folded into totals). * * A transcript-scoped session (Codex) is reported per rollout: each rollout's - * counters restart, and the session's total sums those still in the log. A - * rollout already reported that compaction has since dropped keeps its reported - * totals in the sum, so a later rollout of the session is reported in full - * rather than against them. An entry written before rollouts were kept is - * compared as a whole, then kept per rollout. + * counters restart, and the session's total sums those still in the log plus + * the `dropped` ones (see {@link droppedRollouts}), which the next snapshot + * keeps, so a later rollout of the session is reported in full. */ export function computePromptTokenDelta( current: Map, reported: ReportedPromptTokens, + dropped: Map = droppedRollouts(current, reported, {}), ): { delta: PromptTokenDelta; nextReported: ReportedPromptTokens } { const delta: PromptTokenDelta = { prompts: 0, tokens: emptyTokenUsage() }; const nextReported: ReportedPromptTokens = {}; @@ -402,11 +465,12 @@ export function computePromptTokenDelta( if (cur.segments) { segments = {}; for (const [transcript, segment] of Object.entries(cur.segments)) { - segments[segmentKey(transcript)] = { prompts: segment.prompts, tokens: { ...segment.tokens } }; + segments[segmentKey(transcript)] = { + prompts: segment.prompts, tokens: { ...segment.tokens }, interrupt: segment.interrupt, toolReject: segment.toolReject, + }; } - for (const [key, gone] of Object.entries(reportedSegments(prev))) { - if (Object.hasOwn(segments, key)) continue; - segments[key] = gone; + for (const gone of dropped.get(sid) ?? []) { + segments[gone.key] = { prompts: gone.prompts, tokens: gone.tokens, interrupt: gone.interrupt, toolReject: gone.toolReject }; prompts += gone.prompts; tokens = addTokenUsage(tokens, gone.tokens); } @@ -419,6 +483,28 @@ export function computePromptTokenDelta( return { delta, nextReported }; } +/** + * The intervention counts and daily snapshots a report compares, with each + * session's dropped rollouts added (see {@link droppedRollouts}): Stop counts + * and prompts restart per rollout like the tokens do. + */ +export function withDroppedRollouts( + interventions: Map, + daily: Map, + dropped: Map, +): { interventions: Map; daily: Map } { + const nextInterventions = new Map(interventions); + const nextDaily = new Map(daily); + for (const [sid, gone] of dropped) { + const sum = (field: 'prompts' | 'interrupt' | 'toolReject') => gone.reduce((total, rollout) => total + rollout[field], 0); + const counts = nextInterventions.get(sid); + if (counts) nextInterventions.set(sid, { ...counts, interrupt: counts.interrupt + sum('interrupt'), toolReject: counts.toolReject + sum('toolReject') }); + const day = nextDaily.get(sid); + if (day) nextDaily.set(sid, { ...day, prompts: day.prompts + sum('prompts') }); + } + return { interventions: nextInterventions, daily: nextDaily }; +} + /** Accumulate a prompt/token delta onto the user's existing totals. */ export function mergePromptTokenStats( existingPrompts: number | undefined, @@ -531,6 +617,22 @@ async function creditSplitRuns( return result; } +/** + * When the scope's prompt-token snapshot was last written, before this report + * writes it: its own file's, else the shared one it would be seeded from. An + * entry an earlier release left covers what that release had seen by then. + */ +export async function snapshotWrittenAt(config: LocalConfig | undefined): Promise { + for (const file of [scopeSnapshotPath('prompt-tokens', config), sharedSnapshotPath('prompt-tokens')]) { + try { + return (await fs.promises.stat(file)).mtimeMs; + } catch { + // Not written yet: try the next. + } + } + return undefined; +} + /** * A scope's reported snapshots as its report and `teamai stats` compare them: * bare entries adopted by the runs of `events` (prompt tokens first, since they @@ -686,19 +788,23 @@ export async function reportUsageToTeam( const currentInterventions = interventionCounts(metrics); const currentDaily = aggregateDailySessions(dashboardEvents); + const writtenAt = await snapshotWrittenAt(reportsConfig); const { interventions: reportedInterventions, promptTokens: reportedPromptTokens, daily: reportedDailySessions, } = await reportedBaselines(dashboardEvents, metrics, currentDaily, reportsConfig, true); + const dropped = droppedRollouts(metrics, reportedPromptTokens, reportedInterventions, writtenAt); + const effective = withDroppedRollouts(currentInterventions, currentDaily, dropped); const { delta: promptTokenDelta, nextReported: nextReportedPromptTokens } = computePromptTokenDelta( metrics, reportedPromptTokens, + dropped, ); const { delta: interventionDelta, nextReported } = computeInterventionDelta( - currentInterventions, + effective.interventions, reportedInterventions, ); const { delta: dailyDelta, nextReported: nextReportedDailySessions } = computeDailyStatsDelta( - currentDaily, + effective.daily, reportedDailySessions, ); // This scope's sessions, and those of its snapshots no owner claims yet diff --git a/src/types.ts b/src/types.ts index 565bd9b7a..de4993cb1 100644 --- a/src/types.ts +++ b/src/types.ts @@ -1217,8 +1217,9 @@ export interface SessionMetrics { /** * Each rollout's own totals for a transcript-scoped session (Codex), keyed by * its transcript path: a rollout's counters restart, so the session sums them. + * `since` is the rollout's first event. */ - segments?: Record; + segments?: Record; } export type DashboardSessionStatus = 'running' | 'waiting_for_input' | 'error' | 'idle' | 'stopped'; From 0acbfe53d29f371e0378062159670e8083e13b07 Mon Sep 17 00:00:00 2001 From: Saul Moro Date: Fri, 25 Sep 2026 01:25:07 +0200 Subject: [PATCH 33/41] fix(stats): keep every metric of a dropped Codex rollout, with or without tokens (#785) A Codex session is now kept per rollout whenever its Stops name a rollout, not only once a Stop carries a token record, so a tokenless resumed rollout is not compared against the dropped one's totals. Each rollout also keeps its corrections (a correction goes to the rollout of its prompt), its active time (each gap to the rollout of the event it ends at) and its request costs, and a dropped rollout adds them to the intervention and daily sums, with its cache tokens from its tokens. The prompt-token snapshot, which holds the rollouts, is written with any delta, so a rollout whose rejections alone moved keeps its new totals. --- docs/designs/data-directory-layout.md | 12 ++- .../conversation-token-metrics.test.ts | 4 +- src/__tests__/dashboard-report-scope.test.ts | 76 ++++++++++++++++ src/dashboard-collector.ts | 78 ++++++++++++---- src/session-owners.ts | 7 +- src/session-trends.ts | 52 ++++++++--- src/stats.ts | 4 +- src/team-push.ts | 91 +++++++++++++++---- src/types.ts | 5 +- 9 files changed, 263 insertions(+), 66 deletions(-) diff --git a/docs/designs/data-directory-layout.md b/docs/designs/data-directory-layout.md index 568b831f8..1185df7b0 100644 --- a/docs/designs/data-directory-layout.md +++ b/docs/designs/data-directory-layout.md @@ -414,11 +414,13 @@ credited once with every part reported: for each scope, the shortest prefix of its events whose metrics reach its snapshot, and the owner takes the metrics of their union as reported when they exceed its own entry, so parts counted before any Stop carried the transcript's total are neither lost nor sent twice. -A transcript-scoped session (Codex) is kept per rollout: a rollout's counters -restart, so its prompt-token entry holds each rollout's reported prompts, tokens, -interruptions and rejections under a hash of the rollout's path, and a rollout -compaction has dropped keeps those totals in the session's prompt-token, -intervention and daily sums; a later rollout is then reported in full. An entry +A Codex session is kept per rollout, with or without a token record: a +rollout's counters restart, and compaction drops its events, so its prompt-token +entry holds each rollout's reported prompts, tokens, interruptions, rejections, +corrections, active time and request costs under a hash of the rollout's path, +written with any delta. A rollout compaction has dropped keeps those totals in +the session's prompt-token, intervention and daily sums (cache tokens from its +tokens), so a later rollout is reported in full. An entry written before rollouts were kept is one total: an earlier release rewrote every session in the log on each report, so it covers the rollouts begun by the time its file was last written (read before this report writes it). Those still in diff --git a/src/__tests__/conversation-token-metrics.test.ts b/src/__tests__/conversation-token-metrics.test.ts index 3fc42d215..456d339f3 100644 --- a/src/__tests__/conversation-token-metrics.test.ts +++ b/src/__tests__/conversation-token-metrics.test.ts @@ -402,11 +402,11 @@ describe('aggregateSessionMetrics', () => { const b = stop('/rollouts/b.jsonl', '2026-09-01T13:00:00Z', tokensB, 2); // A was compacted before this build's first report; B came after the entry was written. const compacted = aggregateSessionMetrics([b]); - expect(computePromptTokenDelta(compacted, legacy, droppedRollouts(compacted, legacy, {}, writtenAt)).delta) + expect(computePromptTokenDelta(compacted, legacy, droppedRollouts(compacted, legacy, {}, {}, writtenAt)).delta) .toEqual({ prompts: 2, tokens: tokensB }); // A still in the log takes the entry; B is new. const retained = aggregateSessionMetrics([a, b]); - expect(computePromptTokenDelta(retained, legacy, droppedRollouts(retained, legacy, {}, writtenAt)).delta) + expect(computePromptTokenDelta(retained, legacy, droppedRollouts(retained, legacy, {}, {}, writtenAt)).delta) .toEqual({ prompts: 2, tokens: tokensB }); }); diff --git a/src/__tests__/dashboard-report-scope.test.ts b/src/__tests__/dashboard-report-scope.test.ts index 73f47fb1f..1586b9d5d 100644 --- a/src/__tests__/dashboard-report-scope.test.ts +++ b/src/__tests__/dashboard-report-scope.test.ts @@ -625,6 +625,82 @@ describe('each scope keeps its own reported snapshot (#786)', () => { expect((await reportedPrompts(project)) + (await reportedPrompts(projectQ))).toBe(expected); }); + /** Codex events of one session, as the collector records them, keyed to `config`'s scope. */ + async function codexLog(config: LocalConfig) { + const key = await dataHomeKeyOf(config); + const log = path.join(teamaiHome(), 'dashboard', 'events.jsonl'); + fs.mkdirSync(path.dirname(log), { recursive: true }); + const base = Date.now() - 3_600_000; + const event = (rollout: string, type: string, minute: number, extra: Record = {}) => JSON.stringify({ + type, timestamp: new Date(base + minute * 60_000).toISOString(), sessionId: 'codex-s', tool: 'codex', dataHomeKey: key, + transcriptPath: path.join(tmp, rollout), ...extra, + }); + const write = (lines: string[]) => fs.writeFileSync(log, lines.join('\n') + '\n'); + const stats = async () => { + const reported = await report(config); + const day = reported && typeof reported === 'object' && 'daily' in reported && reported.daily && typeof reported.daily === 'object' + ? Object.values(reported.daily)[0] : undefined; + const interventions = reported && typeof reported === 'object' && 'interventions' in reported ? reported.interventions : undefined; + const prompts = reported && typeof reported === 'object' && 'prompts' in reported ? reported.prompts : undefined; + return { day, interventions, prompts }; + }; + return { event, write, stats }; + } + + it('a Codex rollout with no token record is kept per rollout too', async () => { + const { project } = await setup(); + const { event, write, stats } = await codexLog(project); + write([event('rollout-a.jsonl', 'stop', 0, { prompts: 5 })]); + await stats(); + write([event('rollout-b.jsonl', 'stop', 30, { prompts: 2 })]); + + expect((await stats()).prompts).toBe(7); + }); + + it('a dropped Codex rollout keeps its corrections', async () => { + const { project } = await setup(); + const { event, write, stats } = await codexLog(project); + const withCorrection = (rollout: string, minute: number) => [ + event(rollout, 'stop', minute, { prompts: 1 }), + event(rollout, 'prompt_submit', minute + 0.1, { correction: true, promptSummary: 'no, not that' }), + event(rollout, 'stop', minute + 0.2, { prompts: 2 }), + ]; + write(withCorrection('rollout-a.jsonl', 0)); + await stats(); + write(withCorrection('rollout-b.jsonl', 30)); + + expect((await stats()).interventions).toMatchObject({ correction: 2 }); + }); + + it('a dropped Codex rollout keeps its duration and cache tokens', async () => { + const { project } = await setup(); + const { event, write, stats } = await codexLog(project); + const tokens = (cacheRead: number) => ({ tokenScope: 'transcript', tokens: { input: 10, output: 1, cacheRead, cacheCreation: 0 } }); + // Rollout A: 8 active minutes and 100 cache-read tokens; B: 1 minute and 20. + write([event('rollout-a.jsonl', 'prompt_submit', 0), event('rollout-a.jsonl', 'prompt_submit', 4), + event('rollout-a.jsonl', 'stop', 8, { prompts: 2, ...tokens(100) })]); + await stats(); + write([event('rollout-b.jsonl', 'prompt_submit', 30), event('rollout-b.jsonl', 'stop', 31, { prompts: 1, ...tokens(20) })]); + const { day } = await stats(); + + expect(day).toMatchObject({ durationMs: 9 * 60_000, cacheReadTokens: 120 }); + }); + + it('a rollout\'s intervention change alone still updates its kept totals', async () => { + const { project } = await setup(); + const { event, write, stats } = await codexLog(project); + const stop = (rollout: string, minute: number, toolReject: number) => + event(rollout, 'stop', minute, { prompts: 1, interventions: { interrupt: 0, toolReject } }); + write([stop('rollout-a.jsonl', 0, 1)]); + await stats(); + // Only the rejection count moves; prompts and tokens stay the same. + write([stop('rollout-a.jsonl', 0, 1), stop('rollout-a.jsonl', 1, 2)]); + await stats(); + write([stop('rollout-b.jsonl', 30, 1)]); + + expect((await stats()).interventions).toMatchObject({ toolReject: 3 }); + }); + it('a Codex rollout resumed after compaction reports its daily prompts and interventions too', async () => { const { project } = await setup(); const key = await dataHomeKeyOf(project); diff --git a/src/dashboard-collector.ts b/src/dashboard-collector.ts index 16d45ffcd..6276790b6 100644 --- a/src/dashboard-collector.ts +++ b/src/dashboard-collector.ts @@ -1837,9 +1837,16 @@ export function aggregateSessionMetrics( const unscopedTokens = new Map(); const sessionTokens = new Map(); const transcriptTokens = new Map>(); + // Per rollout, for a transcript-scoped session (Codex), whose counters restart + // with each rollout: see SessionMetrics.segments. + const rolloutSessions = new Set(); const transcriptPrompts = new Map>(); const transcriptInterventions = new Map>(); + const transcriptRequests = new Map>>(); + const transcriptCorrections = new Map>(); const transcriptSince = new Map>(); + const lastTranscript = new Map(); + const timeline = new Map>(); for (const event of events) { let m = map.get(event.sessionId); @@ -1852,6 +1859,26 @@ export function aggregateSessionMetrics( const first = since.get(event.transcriptPath); if (first === undefined || Date.parse(event.timestamp) < Date.parse(first)) since.set(event.transcriptPath, event.timestamp); transcriptSince.set(event.sessionId, since); + lastTranscript.set(event.sessionId, event.transcriptPath); + } + if (event.tokenScope === 'transcript' || (event.tool === 'codex' && typeof event.transcriptPath === 'string')) { + rolloutSessions.add(event.sessionId); + } + const events = timeline.get(event.sessionId) ?? []; + events.push({ at: Date.parse(event.timestamp), transcript: event.transcriptPath ?? lastTranscript.get(event.sessionId) }); + timeline.set(event.sessionId, events); + if (event.type === 'stop' && typeof event.transcriptPath === 'string') { + const rollout = event.transcriptPath; + const record = (maps: Map>, value: T) => { + const perRollout = maps.get(event.sessionId) ?? new Map(); + perRollout.set(rollout, value); + maps.set(event.sessionId, perRollout); + }; + if (typeof event.prompts === 'number') record(transcriptPrompts, event.prompts); + if (event.interventions) { + record(transcriptInterventions, { interrupt: event.interventions.interrupt, toolReject: event.interventions.toolReject }); + } + if (event.requestDaily) record(transcriptRequests, event.requestDaily); } if (event.type === 'stop') { @@ -1872,6 +1899,12 @@ export function aggregateSessionMetrics( const isCorrection = event.correction ?? isCorrectionPrompt(event.promptSummary); if (gap >= 0 && gap <= CORRECTION_WINDOW_MS && isCorrection) { m.correction++; + const rollout = event.transcriptPath ?? lastTranscript.get(event.sessionId); + if (rollout !== undefined) { + const corrections = transcriptCorrections.get(event.sessionId) ?? new Map(); + corrections.set(rollout, (corrections.get(rollout) ?? 0) + 1); + transcriptCorrections.set(event.sessionId, corrections); + } } // Each stop is consumed once — a later prompt is a new task, not a correction. lastStopAt.delete(event.sessionId); @@ -1891,16 +1924,6 @@ export function aggregateSessionMetrics( // replace the same segment; a resumed rollout has a distinct path and adds // one new segment to the logical session total. setLatestTokenSnapshot(segments, event.transcriptPath, event); - if (event.type === 'stop' && typeof event.prompts === 'number') { - const prompts = transcriptPrompts.get(event.sessionId) ?? new Map(); - prompts.set(event.transcriptPath, event.prompts); - transcriptPrompts.set(event.sessionId, prompts); - } - if (event.type === 'stop' && event.interventions) { - const counts = transcriptInterventions.get(event.sessionId) ?? new Map(); - counts.set(event.transcriptPath, { interrupt: event.interventions.interrupt, toolReject: event.interventions.toolReject }); - transcriptInterventions.set(event.sessionId, counts); - } } else { // Claude, CodeBuddy, and pre-existing events retain latest-Stop semantics. setLatestTokenSnapshot(unscopedTokens, event.sessionId, event); @@ -1921,22 +1944,37 @@ export function aggregateSessionMetrics( let total = emptyTokenUsage(); for (const segment of segments.values()) total = addTokenUsage(total, segment.tokens); m.tokens = total; - const prompts = transcriptPrompts.get(sid); - const counts = transcriptInterventions.get(sid); - const since = transcriptSince.get(sid); - m.segments = Object.fromEntries([...segments].map(([transcript, segment]) => [transcript, { - prompts: prompts?.get(transcript) ?? 0, tokens: { ...segment.tokens }, - interrupt: counts?.get(transcript)?.interrupt ?? 0, toolReject: counts?.get(transcript)?.toolReject ?? 0, - since: since?.get(transcript) ?? segment.timestamp, + } else { + const unscoped = unscopedTokens.get(sid); + if (unscoped) m.tokens = { ...unscoped.tokens }; + } + if (rolloutSessions.has(sid) && !sessionSnapshot) { + // Active time per rollout: each gap goes to the rollout of the event it ends at. + const durations = new Map(); + const own = [...(timeline.get(sid) ?? [])].sort((a, b) => a.at - b.at); + for (let i = 1; i < own.length; i++) { + const gap = own[i].at - own[i - 1].at; + const rollout = own[i].transcript; + if (rollout !== undefined && Number.isFinite(gap) && gap >= 0 && gap <= DASHBOARD_IDLE_TIMEOUT_MS) { + durations.set(rollout, (durations.get(rollout) ?? 0) + gap); + } + } + const since = transcriptSince.get(sid) ?? new Map(); + m.segments = Object.fromEntries([...since].map(([transcript, first]) => [transcript, { + prompts: transcriptPrompts.get(sid)?.get(transcript) ?? 0, + tokens: { ...(segments?.get(transcript)?.tokens ?? emptyTokenUsage()) }, + interrupt: transcriptInterventions.get(sid)?.get(transcript)?.interrupt ?? 0, + toolReject: transcriptInterventions.get(sid)?.get(transcript)?.toolReject ?? 0, + correction: transcriptCorrections.get(sid)?.get(transcript) ?? 0, + durationMs: durations.get(transcript) ?? 0, + requestDaily: transcriptRequests.get(sid)?.get(transcript) ?? {}, + since: first, }])); // A rollout's Stop counts restart too; the session sums them. const sum = (field: 'interrupt' | 'toolReject') => Object.values(m.segments ?? {}).reduce((total, segment) => total + segment[field], 0); m.interrupt = Math.max(m.interrupt, sum('interrupt')); m.toolReject = Math.max(m.toolReject, sum('toolReject')); - } else { - const unscoped = unscopedTokens.get(sid); - if (unscoped) m.tokens = { ...unscoped.tokens }; } // A rollout's prompt count restarts too: its segments sum where they exist. const rolloutPrompts = m.segments diff --git a/src/session-owners.ts b/src/session-owners.ts index 44ba5d6e0..4ffb02c9a 100644 --- a/src/session-owners.ts +++ b/src/session-owners.ts @@ -9,7 +9,7 @@ import path from 'node:path'; import { readEvents, dataHomeKey, isHumanPromptEntry } from './dashboard-collector.js'; import { readFileSafe, ensureDir, pathExists, readJson } from './utils/fs.js'; import { log } from './utils/logger.js'; -import type { TokenUsage } from './types.js'; +import type { RequestCostMetrics, TokenUsage } from './types.js'; import { getDataHome, getTeamaiHomeDir, emptyTokenUsage } from './types.js'; import { projectsRootDir } from './utils/partition.js'; import { parseDailySnapshot, type DailySessionSnapshot } from './session-trends.js'; @@ -18,7 +18,10 @@ import { parseDailySnapshot, type DailySessionSnapshot } from './session-trends. export type ReportedInterventions = Record; /** A rollout's reported totals, keyed by a hash of its transcript path (no path is stored). */ -export type ReportedSegments = Record; +export type ReportedSegments = Record; +}>; /** Snapshot of already-reported per-session prompt counts + token usage (idempotency basis). */ export type ReportedPromptTokens = Record; diff --git a/src/session-trends.ts b/src/session-trends.ts index 87da24fda..4a71770bc 100644 --- a/src/session-trends.ts +++ b/src/session-trends.ts @@ -97,26 +97,48 @@ function positiveDelta(current: number, previous: number | undefined): number { return Math.max(0, current - (previous ?? 0)); } +/** Request costs per day read from a file; empty for what is not one. */ +export function parseRequestDaily(value: unknown): Record { + const requestDaily: Record = {}; + if (!value || typeof value !== 'object') return requestDaily; + for (const [date, request] of Object.entries(value)) { + if (!request || typeof request !== 'object') continue; + const num = (field: string) => { + const n: unknown = Object.entries(request).find(([k]) => k === field)?.[1]; + return typeof n === 'number' ? n : 0; + }; + const version: unknown = Object.entries(request).find(([k]) => k === 'priceVersion')?.[1]; + requestDaily[date] = { + pricedRequests: num('pricedRequests'), costMicros: num('costMicros'), cacheReadTokens: num('cacheReadTokens'), + cacheEligibleInputTokens: num('cacheEligibleInputTokens'), priceVersion: typeof version === 'string' ? version : '', + }; + } + return requestDaily; +} + +/** Request costs per day summed, `a`'s price version where both hold a day. */ +export function addRequestDaily( + a: Record, + b: Record, +): Record { + const sum = { ...a }; + for (const [date, request] of Object.entries(b)) { + const own = sum[date]; + sum[date] = own ? { + pricedRequests: own.pricedRequests + request.pricedRequests, costMicros: own.costMicros + request.costMicros, + cacheReadTokens: own.cacheReadTokens + request.cacheReadTokens, + cacheEligibleInputTokens: own.cacheEligibleInputTokens + request.cacheEligibleInputTokens, priceVersion: own.priceVersion, + } : request; + } + return sum; +} + /** A daily snapshot entry read from a file, or undefined when it is not one. */ export function parseDailySnapshot(value: unknown): DailySessionSnapshot | undefined { if (!value || typeof value !== 'object') return undefined; if (!('date' in value) || typeof value.date !== 'string' || !('prompts' in value) || typeof value.prompts !== 'number' || !('durationMs' in value) || typeof value.durationMs !== 'number') return undefined; - const requestDaily: Record = {}; - if ('requestDaily' in value && value.requestDaily && typeof value.requestDaily === 'object') { - for (const [date, request] of Object.entries(value.requestDaily)) { - if (!request || typeof request !== 'object') continue; - const num = (field: string) => { - const n: unknown = Object.entries(request).find(([k]) => k === field)?.[1]; - return typeof n === 'number' ? n : 0; - }; - const version: unknown = Object.entries(request).find(([k]) => k === 'priceVersion')?.[1]; - requestDaily[date] = { - pricedRequests: num('pricedRequests'), costMicros: num('costMicros'), cacheReadTokens: num('cacheReadTokens'), - cacheEligibleInputTokens: num('cacheEligibleInputTokens'), priceVersion: typeof version === 'string' ? version : '', - }; - } - } + const requestDaily = parseRequestDaily('requestDaily' in value ? value.requestDaily : undefined); return { date: value.date, prompts: value.prompts, durationMs: value.durationMs, succeeded: 'succeeded' in value && value.succeeded === 1 ? 1 : 0, diff --git a/src/stats.ts b/src/stats.ts index f69c20d8c..6c69d0224 100644 --- a/src/stats.ts +++ b/src/stats.ts @@ -164,8 +164,8 @@ async function unreportedDashboardStats( // The scope's own snapshots, compared as its report compares them (#786). const writtenAt = await snapshotWrittenAt(config); const currentDaily = aggregateDailySessions(events); - const { promptTokens, interventions } = await reportedBaselines(events, metrics, currentDaily, config, false); - const dropped = droppedRollouts(metrics, promptTokens, interventions, writtenAt); + const { promptTokens, interventions, daily } = await reportedBaselines(events, metrics, currentDaily, config, false); + const dropped = droppedRollouts(metrics, promptTokens, interventions, daily, writtenAt); const currentInterventions = withDroppedRollouts(interventionCounts(metrics), currentDaily, dropped).interventions; const interventionDelta = computeInterventionDelta(currentInterventions, interventions); diff --git a/src/team-push.ts b/src/team-push.ts index 1690be05e..f1ad72f18 100644 --- a/src/team-push.ts +++ b/src/team-push.ts @@ -15,7 +15,9 @@ import { } from './utils/git.js'; import { writeFile, readFileSafe, ensureDir, pathExists, readJson, writeJson } from './utils/fs.js'; import { log } from './utils/logger.js'; -import type { UserStats, UserInterventionStats, SessionMetrics, TokenUsage, DashboardEvent, LocalConfig } from './types.js'; +import type { + UserStats, UserInterventionStats, SessionMetrics, TokenUsage, DashboardEvent, LocalConfig, RequestCostMetrics, +} from './types.js'; import { getVotesDir, getDataHome, emptyTokenUsage, addTokenUsage, usesBranchWorktree } from './types.js'; import { filterEventsByScope, runsOfLog } from './dashboard-scope.js'; import { @@ -28,6 +30,9 @@ import { computeDailyStatsDelta, mergeDailyStats, takeDailySession, + parseDailySnapshot, + parseRequestDaily, + addRequestDaily, type DailySessionSnapshot, type ReportedDailySessions, } from './session-trends.js'; @@ -373,13 +378,21 @@ function reportedSegments(entry: unknown): ReportedSegments { } for (const [key, segment] of Object.entries(entry.segments)) { const iv = interventionsEntry(segment); - segments[key] = { ...promptTokensEntry(segment), interrupt: iv.interrupt, toolReject: iv.toolReject }; + const durationMs: unknown = segment && typeof segment === 'object' && 'durationMs' in segment ? segment.durationMs : 0; + segments[key] = { + ...promptTokensEntry(segment), interrupt: iv.interrupt, toolReject: iv.toolReject, correction: iv.correction, + durationMs: typeof durationMs === 'number' ? durationMs : 0, + requestDaily: parseRequestDaily(segment && typeof segment === 'object' && 'requestDaily' in segment ? segment.requestDaily : undefined), + }; } return segments; } /** A reported rollout compaction has dropped: its key and the totals it was reported with. */ -export interface DroppedRollout { key: string; prompts: number; tokens: TokenUsage; interrupt: number; toolReject: number } +export interface DroppedRollout { + key: string; prompts: number; tokens: TokenUsage; interrupt: number; toolReject: number; correction: number; + durationMs: number; requestDaily: Record; +} /** * The rollouts of each transcript-scoped session (Codex) that were reported and @@ -398,6 +411,7 @@ export function droppedRollouts( current: Map, promptTokens: ReportedPromptTokens, interventions: ReportedInterventions, + daily: ReportedDailySessions, writtenAt?: number, ): Map { const dropped = new Map(); @@ -411,27 +425,47 @@ export function droppedRollouts( if (Object.keys(previous).length > 0) { for (const [key, segment] of Object.entries(previous)) { if (present.has(key)) continue; - gone.push({ key, prompts: segment.prompts, tokens: segment.tokens, interrupt: segment.interrupt ?? 0, toolReject: segment.toolReject ?? 0 }); + gone.push({ + key, prompts: segment.prompts, tokens: segment.tokens, interrupt: segment.interrupt ?? 0, + toolReject: segment.toolReject ?? 0, correction: segment.correction ?? 0, durationMs: segment.durationMs ?? 0, + requestDaily: segment.requestDaily ?? {}, + }); } } else { const total = promptTokensEntry(prev); const iv = interventionsEntry(interventions[sid]); - const left = { prompts: total.prompts, tokens: { ...total.tokens }, interrupt: iv.interrupt, toolReject: iv.toolReject }; + const day = parseDailySnapshot(daily[sid]); + const left = { + prompts: total.prompts, tokens: { ...total.tokens }, interrupt: iv.interrupt, toolReject: iv.toolReject, + correction: iv.correction, durationMs: day?.durationMs ?? 0, requestDaily: { ...(day?.requestDaily ?? {}) }, + }; const covered = Object.values(cur.segments) .filter((segment) => writtenAt === undefined || Date.parse(segment.since) <= writtenAt) .sort((a, b) => Date.parse(a.since) - Date.parse(b.since)); + const take = (own: number, rest: number) => Math.max(0, Math.min(own, rest)); for (const segment of covered) { - const take = (own: number, rest: number) => Math.max(0, Math.min(own, rest)); - left.prompts -= take(segment.prompts, left.prompts); - left.interrupt -= take(segment.interrupt, left.interrupt); - left.toolReject -= take(segment.toolReject, left.toolReject); + for (const field of ['prompts', 'interrupt', 'toolReject', 'correction', 'durationMs'] as const) { + left[field] -= take(segment[field], left[field]); + } for (const field of ['input', 'output', 'cacheRead', 'cacheCreation'] as const) { left.tokens[field] -= take(segment.tokens[field], left.tokens[field]); } + for (const [date, request] of Object.entries(segment.requestDaily)) { + const rest = left.requestDaily[date]; + if (!rest) continue; + left.requestDaily[date] = { + ...rest, + pricedRequests: rest.pricedRequests - take(request.pricedRequests, rest.pricedRequests), + costMicros: rest.costMicros - take(request.costMicros, rest.costMicros), + cacheReadTokens: rest.cacheReadTokens - take(request.cacheReadTokens, rest.cacheReadTokens), + cacheEligibleInputTokens: rest.cacheEligibleInputTokens - take(request.cacheEligibleInputTokens, rest.cacheEligibleInputTokens), + }; + } } - if (left.prompts > 0 || left.interrupt > 0 || left.toolReject > 0 || hasPromptTokenDelta({ prompts: 0, tokens: left.tokens })) { - gone.push({ key: 'prior', ...left }); - } + const anyLeft = left.prompts > 0 || left.interrupt > 0 || left.toolReject > 0 || left.correction > 0 || left.durationMs > 0 + || hasPromptTokenDelta({ prompts: 0, tokens: left.tokens }) + || Object.values(left.requestDaily).some((r) => r.pricedRequests > 0 || r.costMicros > 0); + if (anyLeft) gone.push({ key: 'prior', ...left }); } if (gone.length > 0) dropped.set(sid, gone); } @@ -452,7 +486,7 @@ export function droppedRollouts( export function computePromptTokenDelta( current: Map, reported: ReportedPromptTokens, - dropped: Map = droppedRollouts(current, reported, {}), + dropped: Map = droppedRollouts(current, reported, {}, {}), ): { delta: PromptTokenDelta; nextReported: ReportedPromptTokens } { const delta: PromptTokenDelta = { prompts: 0, tokens: emptyTokenUsage() }; const nextReported: ReportedPromptTokens = {}; @@ -467,10 +501,12 @@ export function computePromptTokenDelta( for (const [transcript, segment] of Object.entries(cur.segments)) { segments[segmentKey(transcript)] = { prompts: segment.prompts, tokens: { ...segment.tokens }, interrupt: segment.interrupt, toolReject: segment.toolReject, + correction: segment.correction, durationMs: segment.durationMs, requestDaily: segment.requestDaily, }; } for (const gone of dropped.get(sid) ?? []) { - segments[gone.key] = { prompts: gone.prompts, tokens: gone.tokens, interrupt: gone.interrupt, toolReject: gone.toolReject }; + const { key, ...totals } = gone; + segments[key] = totals; prompts += gone.prompts; tokens = addTokenUsage(tokens, gone.tokens); } @@ -496,11 +532,27 @@ export function withDroppedRollouts( const nextInterventions = new Map(interventions); const nextDaily = new Map(daily); for (const [sid, gone] of dropped) { - const sum = (field: 'prompts' | 'interrupt' | 'toolReject') => gone.reduce((total, rollout) => total + rollout[field], 0); + const sum = (field: 'prompts' | 'interrupt' | 'toolReject' | 'correction' | 'durationMs') => + gone.reduce((total, rollout) => total + rollout[field], 0); + const tokens = gone.reduce((total, rollout) => addTokenUsage(total, rollout.tokens), emptyTokenUsage()); const counts = nextInterventions.get(sid); - if (counts) nextInterventions.set(sid, { ...counts, interrupt: counts.interrupt + sum('interrupt'), toolReject: counts.toolReject + sum('toolReject') }); + if (counts) { + nextInterventions.set(sid, { + interrupt: counts.interrupt + sum('interrupt'), toolReject: counts.toolReject + sum('toolReject'), + correction: counts.correction + sum('correction'), + }); + } const day = nextDaily.get(sid); - if (day) nextDaily.set(sid, { ...day, prompts: day.prompts + sum('prompts') }); + if (day) { + nextDaily.set(sid, { + ...day, + prompts: day.prompts + sum('prompts'), + durationMs: day.durationMs + sum('durationMs'), + sessionCacheReadTokens: (day.sessionCacheReadTokens ?? 0) + tokens.cacheRead, + sessionCacheEligibleTokens: (day.sessionCacheEligibleTokens ?? 0) + tokens.input + tokens.cacheRead + tokens.cacheCreation, + requestDaily: gone.reduce((total, rollout) => addRequestDaily(total, rollout.requestDaily), day.requestDaily), + }); + } } return { interventions: nextInterventions, daily: nextDaily }; } @@ -792,7 +844,7 @@ export async function reportUsageToTeam( const { interventions: reportedInterventions, promptTokens: reportedPromptTokens, daily: reportedDailySessions, } = await reportedBaselines(dashboardEvents, metrics, currentDaily, reportsConfig, true); - const dropped = droppedRollouts(metrics, reportedPromptTokens, reportedInterventions, writtenAt); + const dropped = droppedRollouts(metrics, reportedPromptTokens, reportedInterventions, reportedDailySessions, writtenAt); const effective = withDroppedRollouts(currentInterventions, currentDaily, dropped); const { delta: promptTokenDelta, nextReported: nextReportedPromptTokens } = computePromptTokenDelta( metrics, @@ -970,7 +1022,8 @@ export async function reportUsageToTeam( await writeReportedInterventions({ ...existingIv, ...nextReported }, reportsConfig); log.debug(`Reported intervention delta (${interventionDelta.sessions} new sessions) to team repo`); } - if (hasPromptTokens) { + // It also holds each rollout's totals, which any delta may have moved. + if (hasPromptTokens || hasInterventions || hasDaily) { const existingPt = await readReportedPromptTokens(reportsConfig); await writeReportedPromptTokens({ ...existingPt, ...nextReportedPromptTokens }, reportsConfig); log.debug(`Reported prompt/token delta (${promptTokenDelta.prompts} prompts) to team repo`); diff --git a/src/types.ts b/src/types.ts index de4993cb1..8edf4eb53 100644 --- a/src/types.ts +++ b/src/types.ts @@ -1219,7 +1219,10 @@ export interface SessionMetrics { * its transcript path: a rollout's counters restart, so the session sums them. * `since` is the rollout's first event. */ - segments?: Record; + segments?: Record; since: string; + }>; } export type DashboardSessionStatus = 'running' | 'waiting_for_input' | 'error' | 'idle' | 'stopped'; From 12bd7bcd45755d93e92f8800b8aa76b19ce8b980 Mon Sep 17 00:00:00 2001 From: Saul Moro Date: Fri, 25 Sep 2026 01:39:32 +0200 Subject: [PATCH 34/41] fix(stats): sum a Codex session's rollout costs and keep a dropped rollout's failure (#785) The daily snapshot took the request costs of the latest rollout only, so with rollout A still in the log a rollout B was compared against A's costs and clamped; a Codex session's daily costs now sum its rollouts. Each rollout also records whether it failed (an error, an interruption or a correction). A dropped rollout that failed keeps the session unsuccessful, and one with a correction keeps it corrected, so a clean later rollout does not turn it into a success. --- docs/designs/data-directory-layout.md | 9 ++++-- src/__tests__/dashboard-report-scope.test.ts | 30 ++++++++++++++++++++ src/dashboard-collector.ts | 8 +++++- src/session-owners.ts | 2 ++ src/session-trends.ts | 5 +++- src/team-push.ts | 12 ++++++-- src/types.ts | 2 ++ 7 files changed, 60 insertions(+), 8 deletions(-) diff --git a/docs/designs/data-directory-layout.md b/docs/designs/data-directory-layout.md index 1185df7b0..023d22946 100644 --- a/docs/designs/data-directory-layout.md +++ b/docs/designs/data-directory-layout.md @@ -418,9 +418,12 @@ A Codex session is kept per rollout, with or without a token record: a rollout's counters restart, and compaction drops its events, so its prompt-token entry holds each rollout's reported prompts, tokens, interruptions, rejections, corrections, active time and request costs under a hash of the rollout's path, -written with any delta. A rollout compaction has dropped keeps those totals in -the session's prompt-token, intervention and daily sums (cache tokens from its -tokens), so a later rollout is reported in full. An entry +written with any delta, and whether it failed (an error, an interruption or a +correction). The session's daily request costs sum its rollouts in the log. A +rollout compaction has dropped keeps those totals in the session's prompt-token, +intervention and daily sums (cache tokens from its tokens), and a failed one +keeps the session unsuccessful, so a later rollout is reported in full and does +not turn it into a success. An entry written before rollouts were kept is one total: an earlier release rewrote every session in the log on each report, so it covers the rollouts begun by the time its file was last written (read before this report writes it). Those still in diff --git a/src/__tests__/dashboard-report-scope.test.ts b/src/__tests__/dashboard-report-scope.test.ts index 1586b9d5d..25e470731 100644 --- a/src/__tests__/dashboard-report-scope.test.ts +++ b/src/__tests__/dashboard-report-scope.test.ts @@ -686,6 +686,36 @@ describe('each scope keeps its own reported snapshot (#786)', () => { expect(day).toMatchObject({ durationMs: 9 * 60_000, cacheReadTokens: 120 }); }); + it('a Codex session\'s request costs sum its rollouts still in the log', async () => { + const { project } = await setup(); + const { event, write } = await codexLog(project); + const day = new Date().toISOString().slice(0, 10); + const cost = (costMicros: number) => ({ + requestDaily: { [day]: { pricedRequests: 1, costMicros, cacheReadTokens: 0, cacheEligibleInputTokens: 0, priceVersion: 'v1' } }, + }); + write([event('rollout-a.jsonl', 'stop', 0, { prompts: 1, ...cost(100) })]); + await report(project); + // Rollout A is still in the log when rollout B adds its own cost. + write([event('rollout-a.jsonl', 'stop', 0, { prompts: 1, ...cost(100) }), event('rollout-b.jsonl', 'stop', 30, { prompts: 1, ...cost(20) })]); + const reported = await report(project); + const days = reported && typeof reported === 'object' && 'daily' in reported && reported.daily && typeof reported.daily === 'object' + ? Object.values(reported.daily) : []; + + expect(days.reduce((sum: number, d: unknown) => + sum + (d && typeof d === 'object' && 'costMicros' in d && typeof d.costMicros === 'number' ? d.costMicros : 0), 0)).toBe(120); + }); + + it('a dropped Codex rollout that failed keeps the session failed', async () => { + const { project } = await setup(); + const { event, write, stats } = await codexLog(project); + // Rollout A was interrupted; rollout B, after compaction, finishes cleanly. + write([event('rollout-a.jsonl', 'stop', 0, { prompts: 1, interventions: { interrupt: 1, toolReject: 0 } })]); + await stats(); + write([event('rollout-b.jsonl', 'stop', 30, { prompts: 1, interventions: { interrupt: 0, toolReject: 0 } })]); + + expect((await stats()).day).toMatchObject({ sessionsSucceeded: 0 }); + }); + it('a rollout\'s intervention change alone still updates its kept totals', async () => { const { project } = await setup(); const { event, write, stats } = await codexLog(project); diff --git a/src/dashboard-collector.ts b/src/dashboard-collector.ts index 6276790b6..a71f0b2b6 100644 --- a/src/dashboard-collector.ts +++ b/src/dashboard-collector.ts @@ -1844,6 +1844,7 @@ export function aggregateSessionMetrics( const transcriptInterventions = new Map>(); const transcriptRequests = new Map>>(); const transcriptCorrections = new Map>(); + const transcriptErrors = new Map>(); const transcriptSince = new Map>(); const lastTranscript = new Map(); const timeline = new Map>(); @@ -1865,8 +1866,12 @@ export function aggregateSessionMetrics( rolloutSessions.add(event.sessionId); } const events = timeline.get(event.sessionId) ?? []; - events.push({ at: Date.parse(event.timestamp), transcript: event.transcriptPath ?? lastTranscript.get(event.sessionId) }); + const rolloutOf = event.transcriptPath ?? lastTranscript.get(event.sessionId); + events.push({ at: Date.parse(event.timestamp), transcript: rolloutOf }); timeline.set(event.sessionId, events); + if (event.status === 'error' && rolloutOf !== undefined) { + transcriptErrors.set(event.sessionId, (transcriptErrors.get(event.sessionId) ?? new Set()).add(rolloutOf)); + } if (event.type === 'stop' && typeof event.transcriptPath === 'string') { const rollout = event.transcriptPath; const record = (maps: Map>, value: T) => { @@ -1969,6 +1974,7 @@ export function aggregateSessionMetrics( durationMs: durations.get(transcript) ?? 0, requestDaily: transcriptRequests.get(sid)?.get(transcript) ?? {}, since: first, + error: transcriptErrors.get(sid)?.has(transcript) ?? false, }])); // A rollout's Stop counts restart too; the session sums them. const sum = (field: 'interrupt' | 'toolReject') => diff --git a/src/session-owners.ts b/src/session-owners.ts index 4ffb02c9a..b762d9312 100644 --- a/src/session-owners.ts +++ b/src/session-owners.ts @@ -21,6 +21,8 @@ export type ReportedInterventions = Record; + /** The rollout errored, was interrupted or corrected: its session did not succeed. */ + failed?: boolean; }>; /** Snapshot of already-reported per-session prompt counts + token usage (idempotency basis). */ diff --git a/src/session-trends.ts b/src/session-trends.ts index 4a71770bc..37a089de8 100644 --- a/src/session-trends.ts +++ b/src/session-trends.ts @@ -77,7 +77,10 @@ export function aggregateDailySessions(events: DashboardEvent[]): Map event.status === 'error'); - const requestDaily = latestRequestDaily(own); + // A Codex rollout's costs restart with it: the session sums its rollouts. + const requestDaily = metric.segments + ? Object.values(metric.segments).reduce((sum, segment) => addRequestDaily(sum, segment.requestDaily), {}) + : latestRequestDaily(own); const corrected = metric.correction > 0 ? 1 : 0; result.set(sessionId, { date: firstStop.timestamp.slice(0, 10), diff --git a/src/team-push.ts b/src/team-push.ts index f1ad72f18..3ad54c721 100644 --- a/src/team-push.ts +++ b/src/team-push.ts @@ -383,6 +383,7 @@ function reportedSegments(entry: unknown): ReportedSegments { ...promptTokensEntry(segment), interrupt: iv.interrupt, toolReject: iv.toolReject, correction: iv.correction, durationMs: typeof durationMs === 'number' ? durationMs : 0, requestDaily: parseRequestDaily(segment && typeof segment === 'object' && 'requestDaily' in segment ? segment.requestDaily : undefined), + failed: !!segment && typeof segment === 'object' && 'failed' in segment && segment.failed === true, }; } return segments; @@ -391,7 +392,7 @@ function reportedSegments(entry: unknown): ReportedSegments { /** A reported rollout compaction has dropped: its key and the totals it was reported with. */ export interface DroppedRollout { key: string; prompts: number; tokens: TokenUsage; interrupt: number; toolReject: number; correction: number; - durationMs: number; requestDaily: Record; + durationMs: number; requestDaily: Record; failed: boolean; } /** @@ -428,7 +429,7 @@ export function droppedRollouts( gone.push({ key, prompts: segment.prompts, tokens: segment.tokens, interrupt: segment.interrupt ?? 0, toolReject: segment.toolReject ?? 0, correction: segment.correction ?? 0, durationMs: segment.durationMs ?? 0, - requestDaily: segment.requestDaily ?? {}, + requestDaily: segment.requestDaily ?? {}, failed: segment.failed ?? false, }); } } else { @@ -465,7 +466,8 @@ export function droppedRollouts( const anyLeft = left.prompts > 0 || left.interrupt > 0 || left.toolReject > 0 || left.correction > 0 || left.durationMs > 0 || hasPromptTokenDelta({ prompts: 0, tokens: left.tokens }) || Object.values(left.requestDaily).some((r) => r.pricedRequests > 0 || r.costMicros > 0); - if (anyLeft) gone.push({ key: 'prior', ...left }); + // What that release counted as the session's outcome stays with its part. + if (anyLeft) gone.push({ key: 'prior', ...left, failed: day?.succeeded === 0 }); } if (gone.length > 0) dropped.set(sid, gone); } @@ -502,6 +504,7 @@ export function computePromptTokenDelta( segments[segmentKey(transcript)] = { prompts: segment.prompts, tokens: { ...segment.tokens }, interrupt: segment.interrupt, toolReject: segment.toolReject, correction: segment.correction, durationMs: segment.durationMs, requestDaily: segment.requestDaily, + failed: segment.error || segment.interrupt > 0 || segment.correction > 0, }; } for (const gone of dropped.get(sid) ?? []) { @@ -551,6 +554,9 @@ export function withDroppedRollouts( sessionCacheReadTokens: (day.sessionCacheReadTokens ?? 0) + tokens.cacheRead, sessionCacheEligibleTokens: (day.sessionCacheEligibleTokens ?? 0) + tokens.input + tokens.cacheRead + tokens.cacheCreation, requestDaily: gone.reduce((total, rollout) => addRequestDaily(total, rollout.requestDaily), day.requestDaily), + // A session one of whose rollouts failed or was corrected did not succeed. + succeeded: gone.some((rollout) => rollout.failed) ? 0 : day.succeeded, + corrected: gone.some((rollout) => rollout.correction > 0) ? 1 : day.corrected, }); } } diff --git a/src/types.ts b/src/types.ts index 8edf4eb53..9b11890c7 100644 --- a/src/types.ts +++ b/src/types.ts @@ -1222,6 +1222,8 @@ export interface SessionMetrics { segments?: Record; since: string; + /** An event of the rollout ended in an error. */ + error: boolean; }>; } From 467be7e8d3e43132d2b35008b3c23816d6ee6fd0 Mon Sep 17 00:00:00 2001 From: Saul Moro Date: Fri, 25 Sep 2026 01:51:17 +0200 Subject: [PATCH 35/41] fix(stats): keep modern Codex rollouts, and their submit-counted prompts, per rollout (#785) A Codex session whose tokens come from the thread-level counter (tokenScope session) was not split into rollouts, so its prompts, interventions, active time, costs and failure were compared against a dropped rollout's. It is now kept per rollout like the others; the counter already spans the rollouts, so no rollout holds tokens of its own and the session total stays that counter's. A Codex Stop may count no prompts, so a rollout's prompts are its Stop's count or else its own submits: a dropped rollout's submit-counted prompts are no longer lost. --- docs/designs/data-directory-layout.md | 5 ++- src/__tests__/dashboard-report-scope.test.ts | 33 ++++++++++++++++++++ src/dashboard-collector.ts | 14 +++++++-- 3 files changed, 48 insertions(+), 4 deletions(-) diff --git a/docs/designs/data-directory-layout.md b/docs/designs/data-directory-layout.md index 023d22946..f864e0160 100644 --- a/docs/designs/data-directory-layout.md +++ b/docs/designs/data-directory-layout.md @@ -414,7 +414,10 @@ credited once with every part reported: for each scope, the shortest prefix of its events whose metrics reach its snapshot, and the owner takes the metrics of their union as reported when they exceed its own entry, so parts counted before any Stop carried the transcript's total are neither lost nor sent twice. -A Codex session is kept per rollout, with or without a token record: a +A Codex session is kept per rollout, with or without a token record, and when +its tokens come from a thread-level counter that already spans rollouts (then +no rollout holds tokens of its own); a rollout's prompts are its Stop's count or +else its submits. A rollout's counters restart, and compaction drops its events, so its prompt-token entry holds each rollout's reported prompts, tokens, interruptions, rejections, corrections, active time and request costs under a hash of the rollout's path, diff --git a/src/__tests__/dashboard-report-scope.test.ts b/src/__tests__/dashboard-report-scope.test.ts index 25e470731..8a3e23d18 100644 --- a/src/__tests__/dashboard-report-scope.test.ts +++ b/src/__tests__/dashboard-report-scope.test.ts @@ -716,6 +716,39 @@ describe('each scope keeps its own reported snapshot (#786)', () => { expect((await stats()).day).toMatchObject({ sessionsSucceeded: 0 }); }); + it('a Codex session with a session-scoped token counter still keeps its other metrics per rollout', async () => { + const { project } = await setup(); + const { event, write, stats } = await codexLog(project); + const day = new Date().toISOString().slice(0, 10); + // The thread-level token counter spans rollouts; prompts and costs restart with each. + const stop = (rollout: string, minute: number, prompts: number, costMicros: number, input: number) => event(rollout, 'stop', minute, { + prompts, tokenScope: 'session', tokens: { input, output: 0, cacheRead: 0, cacheCreation: 0 }, + requestDaily: { [day]: { pricedRequests: 1, costMicros, cacheReadTokens: 0, cacheEligibleInputTokens: 0, priceVersion: 'v1' } }, + }); + write([stop('rollout-a.jsonl', 0, 5, 100, 500)]); + await stats(); + write([stop('rollout-b.jsonl', 30, 2, 20, 530)]); + const after = await stats(); + const reported = await report(project); + const tokens = reported && typeof reported === 'object' && 'tokens' in reported ? reported.tokens : undefined; + + expect(after.prompts).toBe(7); + expect(after.day).toMatchObject({ costMicros: 120 }); + expect(tokens).toMatchObject({ input: 530 }); + }); + + it('a dropped Codex rollout keeps the prompts its submits counted', async () => { + const { project } = await setup(); + const { event, write, stats } = await codexLog(project); + // Codex Stops carry no prompt count: the submits of each rollout count its prompts. + write([event('rollout-a.jsonl', 'prompt_submit', 0), event('rollout-a.jsonl', 'prompt_submit', 1), + event('rollout-a.jsonl', 'prompt_submit', 2), event('rollout-a.jsonl', 'stop', 3)]); + await stats(); + write([event('rollout-b.jsonl', 'prompt_submit', 30), event('rollout-b.jsonl', 'stop', 31)]); + + expect((await stats()).prompts).toBe(4); + }); + it('a rollout\'s intervention change alone still updates its kept totals', async () => { const { project } = await setup(); const { event, write, stats } = await codexLog(project); diff --git a/src/dashboard-collector.ts b/src/dashboard-collector.ts index a71f0b2b6..9fd2768e2 100644 --- a/src/dashboard-collector.ts +++ b/src/dashboard-collector.ts @@ -1845,6 +1845,7 @@ export function aggregateSessionMetrics( const transcriptRequests = new Map>>(); const transcriptCorrections = new Map>(); const transcriptErrors = new Map>(); + const transcriptSubmits = new Map>(); const transcriptSince = new Map>(); const lastTranscript = new Map(); const timeline = new Map>(); @@ -1872,6 +1873,11 @@ export function aggregateSessionMetrics( if (event.status === 'error' && rolloutOf !== undefined) { transcriptErrors.set(event.sessionId, (transcriptErrors.get(event.sessionId) ?? new Set()).add(rolloutOf)); } + if (event.type === 'prompt_submit' && rolloutOf !== undefined) { + const submits = transcriptSubmits.get(event.sessionId) ?? new Map(); + submits.set(rolloutOf, (submits.get(rolloutOf) ?? 0) + 1); + transcriptSubmits.set(event.sessionId, submits); + } if (event.type === 'stop' && typeof event.transcriptPath === 'string') { const rollout = event.transcriptPath; const record = (maps: Map>, value: T) => { @@ -1953,7 +1959,7 @@ export function aggregateSessionMetrics( const unscoped = unscopedTokens.get(sid); if (unscoped) m.tokens = { ...unscoped.tokens }; } - if (rolloutSessions.has(sid) && !sessionSnapshot) { + if (rolloutSessions.has(sid)) { // Active time per rollout: each gap goes to the rollout of the event it ends at. const durations = new Map(); const own = [...(timeline.get(sid) ?? [])].sort((a, b) => a.at - b.at); @@ -1966,8 +1972,10 @@ export function aggregateSessionMetrics( } const since = transcriptSince.get(sid) ?? new Map(); m.segments = Object.fromEntries([...since].map(([transcript, first]) => [transcript, { - prompts: transcriptPrompts.get(sid)?.get(transcript) ?? 0, - tokens: { ...(segments?.get(transcript)?.tokens ?? emptyTokenUsage()) }, + // A Codex Stop may count no prompts: the rollout's submits then do. + prompts: Math.max(transcriptPrompts.get(sid)?.get(transcript) ?? 0, transcriptSubmits.get(sid)?.get(transcript) ?? 0), + // A session-scoped counter already spans the rollouts: none holds tokens of its own. + tokens: { ...(sessionSnapshot ? emptyTokenUsage() : segments?.get(transcript)?.tokens ?? emptyTokenUsage()) }, interrupt: transcriptInterventions.get(sid)?.get(transcript)?.interrupt ?? 0, toolReject: transcriptInterventions.get(sid)?.get(transcript)?.toolReject ?? 0, correction: transcriptCorrections.get(sid)?.get(transcript) ?? 0, From 53d1dab1ebe550d48ff41a4e632f46351af2c744 Mon Sep 17 00:00:00 2001 From: Saul Moro Date: Fri, 25 Sep 2026 02:06:31 +0200 Subject: [PATCH 36/41] fix(stats): no legacy tokens on a spanning Codex counter; teamai stats writes no seed (#785) A whole entry an earlier release left became a prior rollout carrying its tokens, which were then added to a thread-level counter that already holds them: rollout B's counter at 530 after A's 500 re-sent 500. A session whose counter spans its rollouts now takes no tokens from a dropped or prior rollout. `teamai stats` only reads, but seeding a scope's first snapshot wrote it with the current time, which a later report reads as the time an entry from before covers, taking a rollout begun earlier as reported. A read that does not persist now writes no seed, and a written seed keeps the shared file's time. --- docs/designs/data-directory-layout.md | 6 ++- src/__tests__/dashboard-report-scope.test.ts | 47 +++++++++++++++++++- src/dashboard-collector.ts | 1 + src/team-push.ts | 25 +++++++---- src/types.ts | 2 + 5 files changed, 70 insertions(+), 11 deletions(-) diff --git a/docs/designs/data-directory-layout.md b/docs/designs/data-directory-layout.md index f864e0160..2b5829d7a 100644 --- a/docs/designs/data-directory-layout.md +++ b/docs/designs/data-directory-layout.md @@ -416,7 +416,8 @@ their union as reported when they exceed its own entry, so parts counted before any Stop carried the transcript's total are neither lost nor sent twice. A Codex session is kept per rollout, with or without a token record, and when its tokens come from a thread-level counter that already spans rollouts (then -no rollout holds tokens of its own); a rollout's prompts are its Stop's count or +no rollout holds tokens of its own, nor does the prior rollout an entry from +before leaves); a rollout's prompts are its Stop's count or else its submits. A rollout's counters restart, and compaction drops its events, so its prompt-token entry holds each rollout's reported prompts, tokens, interruptions, rejections, @@ -429,7 +430,8 @@ keeps the session unsuccessful, so a later rollout is reported in full and does not turn it into a success. An entry written before rollouts were kept is one total: an earlier release rewrote every session in the log on each report, so it covers the rollouts begun by the time -its file was last written (read before this report writes it). Those still in +its file was last written (read before this report writes it; a seed keeps the +shared file's time, and `teamai stats`, which only reads, writes no seed). Those still in the log consume it in order, what is left is the dropped rollouts', kept as one prior rollout, and a rollout begun later is new. Compaction also keeps a session whose tool process is still running, so a run diff --git a/src/__tests__/dashboard-report-scope.test.ts b/src/__tests__/dashboard-report-scope.test.ts index 8a3e23d18..b2c1935ae 100644 --- a/src/__tests__/dashboard-report-scope.test.ts +++ b/src/__tests__/dashboard-report-scope.test.ts @@ -731,9 +731,14 @@ describe('each scope keeps its own reported snapshot (#786)', () => { const after = await stats(); const reported = await report(project); const tokens = reported && typeof reported === 'object' && 'tokens' in reported ? reported.tokens : undefined; + // Costs go to their request's day, which need not be the session's. + const days = reported && typeof reported === 'object' && 'daily' in reported && reported.daily && typeof reported.daily === 'object' + ? Object.values(reported.daily) : []; + const costMicros = days.reduce((sum: number, d: unknown) => + sum + (d && typeof d === 'object' && 'costMicros' in d && typeof d.costMicros === 'number' ? d.costMicros : 0), 0); expect(after.prompts).toBe(7); - expect(after.day).toMatchObject({ costMicros: 120 }); + expect(costMicros).toBe(120); expect(tokens).toMatchObject({ input: 530 }); }); @@ -749,6 +754,46 @@ describe('each scope keeps its own reported snapshot (#786)', () => { expect((await stats()).prompts).toBe(4); }); + it('a whole entry an earlier release left adds no tokens to a counter that spans rollouts', async () => { + const { project } = await setup(); + const { event, write, stats } = await codexLog(project); + // That release reported rollout A as one entry: 5 prompts, the counter at 500. + writeSharedSnapshots({ 'codex-s': 5 }, new Date().toISOString().slice(0, 10), project); + const file = path.join(getDataHome(project), 'dashboard', 'reported-prompt-tokens.json'); + fs.writeFileSync(file, JSON.stringify({ 'codex-s': { prompts: 5, tokens: { input: 500, output: 0, cacheRead: 0, cacheCreation: 0 } } })); + const past = new Date(Date.now() - 2 * 3_600_000); + for (const name of SNAPSHOTS) fs.utimesSync(path.join(getDataHome(project), 'dashboard', `reported-${name}.json`), past, past); + // A was compacted; rollout B, begun later, shows the thread-level counter at 530. + write([event('rollout-b.jsonl', 'stop', 30, { + prompts: 2, tokenScope: 'session', tokens: { input: 530, output: 0, cacheRead: 0, cacheCreation: 0 }, + })]); + await stats(); + const reported = await report(project); + const tokens = reported && typeof reported === 'object' && 'tokens' in reported ? reported.tokens : undefined; + + expect(tokens).toMatchObject({ input: 30 }); + }); + + it('reading the baselines without persisting them, as teamai stats does, does not move the time they cover', async () => { + const { project } = await setup(); + const { event, write, stats } = await codexLog(project); + // An earlier release reported rollout A into the shared snapshot, then compaction dropped it. + writeSharedSnapshots({ 'codex-s': 5 }, new Date().toISOString().slice(0, 10)); + const past = new Date(Date.now() - 2 * 3_600_000); + for (const name of SNAPSHOTS) fs.utimesSync(shared(name), past, past); + // Rollout B began after that, and the baselines are read before the next pull. + write([event('rollout-b.jsonl', 'stop', 30, { prompts: 2 })]); + const { reportedBaselines } = await import('../team-push.js'); + const { filterEventsByScope } = await import('../dashboard-scope.js'); + const { readEvents, aggregateSessionMetrics } = await import('../dashboard-collector.js'); + const { aggregateDailySessions } = await import('../session-trends.js'); + const events = await filterEventsByScope(await readEvents(), project); + await reportedBaselines(events, aggregateSessionMetrics(events), aggregateDailySessions(events), project, false); + + expect(fs.existsSync(path.join(getDataHome(project), 'dashboard', 'reported-prompt-tokens.json'))).toBe(false); + expect((await stats()).prompts).toBe(2); + }); + it('a rollout\'s intervention change alone still updates its kept totals', async () => { const { project } = await setup(); const { event, write, stats } = await codexLog(project); diff --git a/src/dashboard-collector.ts b/src/dashboard-collector.ts index 9fd2768e2..ec9977199 100644 --- a/src/dashboard-collector.ts +++ b/src/dashboard-collector.ts @@ -1951,6 +1951,7 @@ export function aggregateSessionMetrics( if (sessionSnapshot) { // The newer thread-level counter already spans rollout files. m.tokens = { ...sessionSnapshot.tokens }; + m.tokensSpanRollouts = true; } else if (segments && segments.size > 0) { let total = emptyTokenUsage(); for (const segment of segments.values()) total = addTokenUsage(total, segment.tokens); diff --git a/src/team-push.ts b/src/team-push.ts index 3ad54c721..c1cf33989 100644 --- a/src/team-push.ts +++ b/src/team-push.ts @@ -176,6 +176,7 @@ async function readSnapshot( name: ReportedSnapshotName, config: LocalConfig | undefined, split: (events: DashboardEvent[]) => Promise<{ current: Record; take: TakeReported }>, + persist = true, ): Promise | null> { const own = scopeSnapshotPath(name, config); if (!config || await pathExists(own)) return readJson>(own); @@ -194,8 +195,14 @@ async function readSnapshot( for (const [id, entry] of Object.entries(adopted)) { if (!id.startsWith('pid-') || ownRuns.has(id)) seed[id] = entry; } + // A read-only caller (`teamai stats`) leaves the seed to the report. + if (!persist) return seed; try { await writeJson(own, seed); + // The seed covers what the shared file had by its last write, and a later + // report reads that from this file's time (see droppedRollouts). + const sharedTime = await fs.promises.stat(sharedSnapshotPath(name)).then((stat) => stat.mtime, () => undefined); + if (sharedTime) await fs.promises.utimes(own, sharedTime, sharedTime); } catch (e) { // Seeded again next time: the shared file is not written any more. log.debug(`Could not seed ${own}: ${(e as Error).message}`); @@ -203,11 +210,11 @@ async function readSnapshot( return seed; } -export async function readReportedInterventions(config: LocalConfig | undefined): Promise { +export async function readReportedInterventions(config: LocalConfig | undefined, persist = true): Promise { const parsed = await readSnapshot('interventions', config, async (events) => ({ current: Object.fromEntries(interventionCounts(aggregateSessionMetrics(events))), take: takeInterventions(await sharedCoverage(events)), - })); + }), persist); return parsed && typeof parsed === 'object' ? parsed : {}; } @@ -339,11 +346,11 @@ function hasInterventionDelta(d: UserInterventionStats): boolean { // Separate snapshot from interventions so each metric stays independently idempotent. // -export async function readReportedPromptTokens(config: LocalConfig | undefined): Promise { +export async function readReportedPromptTokens(config: LocalConfig | undefined, persist = true): Promise { const parsed = await readSnapshot('prompt-tokens', config, async (events) => ({ current: computePromptTokenDelta(aggregateSessionMetrics(events), {}).nextReported, take: takePromptTokens, - })); + }), persist); return parsed && typeof parsed === 'object' ? parsed : {}; } @@ -469,6 +476,8 @@ export function droppedRollouts( // What that release counted as the session's outcome stays with its part. if (anyLeft) gone.push({ key: 'prior', ...left, failed: day?.succeeded === 0 }); } + // A thread-level counter already holds every rollout's tokens. + if (cur.tokensSpanRollouts) for (const rollout of gone) rollout.tokens = emptyTokenUsage(); if (gone.length > 0) dropped.set(sid, gone); } return dropped; @@ -581,11 +590,11 @@ function hasPromptTokenDelta(d: PromptTokenDelta): boolean { || d.tokens.cacheRead > 0 || d.tokens.cacheCreation > 0; } -async function readReportedDailySessions(config: LocalConfig | undefined): Promise { +async function readReportedDailySessions(config: LocalConfig | undefined, persist = true): Promise { return (await readSnapshot('daily-sessions', config, async (events) => ({ current: computeDailyStatsDelta(aggregateDailySessions(events), {}).nextReported, take: takeDaily(await sharedCoverage(events)), - }))) ?? {}; + }), persist)) ?? {}; } async function writeReportedDailySessions(data: ReportedDailySessions, config: LocalConfig | undefined): Promise { @@ -707,12 +716,12 @@ export async function reportedBaselines( persist: boolean, ): Promise<{ interventions: ReportedInterventions; promptTokens: ReportedPromptTokens; daily: ReportedDailySessions }> { const adopt = async ( - read: (config: LocalConfig | undefined) => Promise>, + read: (config: LocalConfig | undefined, persist: boolean) => Promise>, write: (data: Record, config: LocalConfig | undefined) => Promise, current: Record, take: TakeReported, ): Promise> => { - const stored = await read(config); + const stored = await read(config, persist); const adopted = adoptBareKeys(stored, events, current, take); if (persist && adopted !== stored) await write(adopted, config); return adopted; diff --git a/src/types.ts b/src/types.ts index 9b11890c7..c24d47bd3 100644 --- a/src/types.ts +++ b/src/types.ts @@ -1219,6 +1219,8 @@ export interface SessionMetrics { * its transcript path: a rollout's counters restart, so the session sums them. * `since` is the rollout's first event. */ + /** Its tokens come from one counter that spans its rollouts (Codex's thread-level counter). */ + tokensSpanRollouts?: true; segments?: Record; since: string; From f073b8f448db489890271a17886a458b9da4a4a1 Mon Sep 17 00:00:00 2001 From: Saul Moro Date: Fri, 25 Sep 2026 02:20:24 +0200 Subject: [PATCH 37/41] fix(stats): read a legacy daily entry's session cost fields as its day's costs (#785) parseDailySnapshot() dropped the top-level pricedRequests, costMicros, cache tokens and priceVersion a daily entry from before per-day costs held, so an entry from before rollouts were kept lost its cost in the prior rollout, and a later rollout's cost was compared against it and omitted. They are now read as the session day's request costs, as computeDailyStatsDelta already reads them. --- src/__tests__/dashboard-report-scope.test.ts | 25 ++++++++++++++++++++ src/session-trends.ts | 16 ++++++++++++- 2 files changed, 40 insertions(+), 1 deletion(-) diff --git a/src/__tests__/dashboard-report-scope.test.ts b/src/__tests__/dashboard-report-scope.test.ts index b2c1935ae..135f1206d 100644 --- a/src/__tests__/dashboard-report-scope.test.ts +++ b/src/__tests__/dashboard-report-scope.test.ts @@ -774,6 +774,31 @@ describe('each scope keeps its own reported snapshot (#786)', () => { expect(tokens).toMatchObject({ input: 30 }); }); + it('a whole daily entry from before request costs were kept per day keeps its cost for a later rollout', async () => { + const { project } = await setup(); + const { event, write } = await codexLog(project); + const today = new Date().toISOString().slice(0, 10); + // An earlier release reported rollout A: its daily entry holds the cost as session fields. + writeSharedSnapshots({ 'codex-s': 5 }, today, project); + const dir = path.join(getDataHome(project), 'dashboard'); + fs.writeFileSync(path.join(dir, 'reported-daily-sessions.json'), JSON.stringify({ 'codex-s': { + date: today, prompts: 5, durationMs: 0, succeeded: 1, corrected: 0, + pricedRequests: 1, costMicros: 100, cacheReadTokens: 0, cacheEligibleInputTokens: 0, priceVersion: 'v1', + } })); + const past = new Date(Date.now() - 2 * 3_600_000); + for (const name of SNAPSHOTS) fs.utimesSync(path.join(dir, `reported-${name}.json`), past, past); + // A was compacted; rollout B, begun later, costs $20 on the same day. + write([event('rollout-b.jsonl', 'stop', 30, { + prompts: 2, requestDaily: { [today]: { pricedRequests: 1, costMicros: 20, cacheReadTokens: 0, cacheEligibleInputTokens: 0, priceVersion: 'v1' } }, + })]); + const reported = await report(project); + const days = reported && typeof reported === 'object' && 'daily' in reported && reported.daily && typeof reported.daily === 'object' + ? Object.values(reported.daily) : []; + + expect(days.reduce((sum: number, d: unknown) => + sum + (d && typeof d === 'object' && 'costMicros' in d && typeof d.costMicros === 'number' ? d.costMicros : 0), 0)).toBe(20); + }); + it('reading the baselines without persisting them, as teamai stats does, does not move the time they cover', async () => { const { project } = await setup(); const { event, write, stats } = await codexLog(project); diff --git a/src/session-trends.ts b/src/session-trends.ts index 37a089de8..70849099a 100644 --- a/src/session-trends.ts +++ b/src/session-trends.ts @@ -141,7 +141,21 @@ export function parseDailySnapshot(value: unknown): DailySessionSnapshot | undef if (!value || typeof value !== 'object') return undefined; if (!('date' in value) || typeof value.date !== 'string' || !('prompts' in value) || typeof value.prompts !== 'number' || !('durationMs' in value) || typeof value.durationMs !== 'number') return undefined; - const requestDaily = parseRequestDaily('requestDaily' in value ? value.requestDaily : undefined); + // A snapshot from before costs were kept per day holds them as session fields, + // on the session's day: read them as that day's, as the delta does. + const legacy = (field: string): number => { + const n: unknown = Object.entries(value).find(([k]) => k === field)?.[1]; + return typeof n === 'number' ? n : 0; + }; + const version: unknown = Object.entries(value).find(([k]) => k === 'priceVersion')?.[1]; + const requestDaily = 'requestDaily' in value + ? parseRequestDaily(value.requestDaily) + : legacy('pricedRequests') || legacy('costMicros') || legacy('cacheReadTokens') || legacy('cacheEligibleInputTokens') + ? { [value.date]: { + pricedRequests: legacy('pricedRequests'), costMicros: legacy('costMicros'), cacheReadTokens: legacy('cacheReadTokens'), + cacheEligibleInputTokens: legacy('cacheEligibleInputTokens'), priceVersion: typeof version === 'string' ? version : '', + } } + : {}; return { date: value.date, prompts: value.prompts, durationMs: value.durationMs, succeeded: 'succeeded' in value && value.succeeded === 1 ? 1 : 0, From aade8eaac01031f737a9d25f0ce45466cb03b3f2 Mon Sep 17 00:00:00 2001 From: Saul Moro Date: Fri, 25 Sep 2026 02:38:04 +0200 Subject: [PATCH 38/41] fix(stats): keep every Codex variant per rollout, and an older Stop's request cost (#785) Rollout tracking recognized only `codex`, not `codex-internal` or `tcodex`, which write the same rollouts; it now uses isCodexTool(). A rollout's cost was read from requestDaily only, so an older Stop's requestMetrics left the rollout without cost, and the daily snapshot, which sums rollouts, omitted it; it is now that Stop's day's cost, as outside rollouts. --- docs/designs/data-directory-layout.md | 3 ++- src/__tests__/dashboard-report-scope.test.ts | 25 ++++++++++++++++++++ src/dashboard-collector.ts | 4 +++- 3 files changed, 30 insertions(+), 2 deletions(-) diff --git a/docs/designs/data-directory-layout.md b/docs/designs/data-directory-layout.md index 2b5829d7a..4f1a449d5 100644 --- a/docs/designs/data-directory-layout.md +++ b/docs/designs/data-directory-layout.md @@ -414,7 +414,8 @@ credited once with every part reported: for each scope, the shortest prefix of its events whose metrics reach its snapshot, and the owner takes the metrics of their union as reported when they exceed its own entry, so parts counted before any Stop carried the transcript's total are neither lost nor sent twice. -A Codex session is kept per rollout, with or without a token record, and when +A Codex session (any Codex variant: `codex`, `codex-internal`, `tcodex`) is kept +per rollout, with or without a token record, and when its tokens come from a thread-level counter that already spans rollouts (then no rollout holds tokens of its own, nor does the prior rollout an entry from before leaves); a rollout's prompts are its Stop's count or diff --git a/src/__tests__/dashboard-report-scope.test.ts b/src/__tests__/dashboard-report-scope.test.ts index 135f1206d..9c9354a8a 100644 --- a/src/__tests__/dashboard-report-scope.test.ts +++ b/src/__tests__/dashboard-report-scope.test.ts @@ -657,6 +657,31 @@ describe('each scope keeps its own reported snapshot (#786)', () => { expect((await stats()).prompts).toBe(7); }); + it('a tcodex rollout with no token record is kept per rollout too', async () => { + const { project } = await setup(); + const { event, write, stats } = await codexLog(project); + write([event('rollout-a.jsonl', 'stop', 0, { tool: 'tcodex', prompts: 5 })]); + await stats(); + write([event('rollout-b.jsonl', 'stop', 30, { tool: 'tcodex', prompts: 2 })]); + + expect((await stats()).prompts).toBe(7); + }); + + it('a Codex rollout whose Stop records its cost as requestMetrics keeps that cost', async () => { + const { project } = await setup(); + const { event, write } = await codexLog(project); + // An older Stop: its cost as one request record, on the Stop's day. + write([event('rollout-a.jsonl', 'stop', 0, { + prompts: 1, requestMetrics: { pricedRequests: 1, costMicros: 40, cacheReadTokens: 0, cacheEligibleInputTokens: 0, priceVersion: 'v1' }, + })]); + const reported = await report(project); + const days = reported && typeof reported === 'object' && 'daily' in reported && reported.daily && typeof reported.daily === 'object' + ? Object.values(reported.daily) : []; + + expect(days.reduce((sum: number, d: unknown) => + sum + (d && typeof d === 'object' && 'costMicros' in d && typeof d.costMicros === 'number' ? d.costMicros : 0), 0)).toBe(40); + }); + it('a dropped Codex rollout keeps its corrections', async () => { const { project } = await setup(); const { event, write, stats } = await codexLog(project); diff --git a/src/dashboard-collector.ts b/src/dashboard-collector.ts index ec9977199..4c5dc246e 100644 --- a/src/dashboard-collector.ts +++ b/src/dashboard-collector.ts @@ -1863,7 +1863,7 @@ export function aggregateSessionMetrics( transcriptSince.set(event.sessionId, since); lastTranscript.set(event.sessionId, event.transcriptPath); } - if (event.tokenScope === 'transcript' || (event.tool === 'codex' && typeof event.transcriptPath === 'string')) { + if (event.tokenScope === 'transcript' || (isCodexTool(event.tool) && typeof event.transcriptPath === 'string')) { rolloutSessions.add(event.sessionId); } const events = timeline.get(event.sessionId) ?? []; @@ -1889,7 +1889,9 @@ export function aggregateSessionMetrics( if (event.interventions) { record(transcriptInterventions, { interrupt: event.interventions.interrupt, toolReject: event.interventions.toolReject }); } + // An older Stop records its cost as one request, on its own day. if (event.requestDaily) record(transcriptRequests, event.requestDaily); + else if (event.requestMetrics) record(transcriptRequests, { [event.timestamp.slice(0, 10)]: event.requestMetrics }); } if (event.type === 'stop') { From c8b97ee23f1a0299f8e8f5ab859dc0ba1abd767c Mon Sep 17 00:00:00 2001 From: Saul Moro Date: Fri, 25 Sep 2026 02:49:48 +0200 Subject: [PATCH 39/41] fix(stats): keep a Codex rollout's latest Stop by timestamp (#785) A rollout's prompts, interventions and request costs took the last Stop appended, though background Stop handlers may append an older scan after a newer one, which then replaced the newer totals. They now keep the latest Stop by its timestamp, as the rollout's tokens already do. --- src/__tests__/conversation-token-metrics.test.ts | 14 ++++++++++++++ src/dashboard-collector.ts | 12 ++++++++++++ 2 files changed, 26 insertions(+) diff --git a/src/__tests__/conversation-token-metrics.test.ts b/src/__tests__/conversation-token-metrics.test.ts index 456d339f3..777858e60 100644 --- a/src/__tests__/conversation-token-metrics.test.ts +++ b/src/__tests__/conversation-token-metrics.test.ts @@ -410,6 +410,20 @@ describe('aggregateSessionMetrics', () => { .toEqual({ prompts: 2, tokens: tokensB }); }); + it('keeps a Codex rollout\'s latest Stop, whatever order the Stops come in', () => { + // A background handler appended the older scan after the newer one. + const cost = (costMicros: number) => ({ '2026-09-01': { + pricedRequests: 1, costMicros, cacheReadTokens: 0, cacheEligibleInputTokens: 0, priceVersion: 'v1', + } }); + const stop = (timestamp: string, prompts: number, costMicros: number, toolReject: number): DashboardEvent => ({ + type: 'stop', timestamp, sessionId: 's1', tool: 'codex', transcriptPath: '/rollouts/a.jsonl', + prompts, requestDaily: cost(costMicros), interventions: { interrupt: 0, toolReject }, + }); + const rollout = aggregateSessionMetrics([stop('2026-09-01T10:10:00Z', 3, 100, 2), stop('2026-09-01T10:05:00Z', 2, 40, 1)]) + .get('s1')?.segments?.['/rollouts/a.jsonl']; + expect(rollout).toMatchObject({ prompts: 3, toolReject: 2, requestDaily: cost(100) }); + }); + it('counts a Codex session\'s prompts across its rollouts', () => { const stop = (path: string, prompts: number): DashboardEvent => ({ type: 'stop', timestamp: new Date().toISOString(), sessionId: 's1', tool: 'codex', diff --git a/src/dashboard-collector.ts b/src/dashboard-collector.ts index 4c5dc246e..916777ef1 100644 --- a/src/dashboard-collector.ts +++ b/src/dashboard-collector.ts @@ -1846,6 +1846,7 @@ export function aggregateSessionMetrics( const transcriptCorrections = new Map>(); const transcriptErrors = new Map>(); const transcriptSubmits = new Map>(); + const stopAt = new Map>(); const transcriptSince = new Map>(); const lastTranscript = new Map(); const timeline = new Map>(); @@ -1880,7 +1881,18 @@ export function aggregateSessionMetrics( } if (event.type === 'stop' && typeof event.transcriptPath === 'string') { const rollout = event.transcriptPath; + // The latest Stop by its timestamp, as for tokens: background Stop + // handlers may append an older scan after a newer one. + const at = Date.parse(event.timestamp); + const latest = stopAt.get(event.sessionId) ?? new Map(); + const seen = latest.get(rollout); + const newest = seen === undefined || !Number.isFinite(at) || !Number.isFinite(seen) || at >= seen; + if (newest) { + latest.set(rollout, at); + stopAt.set(event.sessionId, latest); + } const record = (maps: Map>, value: T) => { + if (!newest) return; const perRollout = maps.get(event.sessionId) ?? new Map(); perRollout.set(rollout, value); maps.set(event.sessionId, perRollout); From 640c5471c3fdf8e82530c14653967f297f8318b8 Mon Sep 17 00:00:00 2001 From: Saul Moro Date: Fri, 25 Sep 2026 03:04:31 +0200 Subject: [PATCH 40/41] fix(stats): an entry from before covers a running Codex rollout only as far as it had got (#785) Migrating a whole entry from before rollouts were kept consumed it with each covered rollout's current totals, so a rollout begun before the entry was written but grown since had its later prompts taken as reported: an entry of 6 (A's 5, B's 1) with B now at 3 reported nothing. It now consumes it with each rollout's totals as of the entry's write, the metrics of the events up to then; what a rollout has done since is new. --- docs/designs/data-directory-layout.md | 2 +- src/__tests__/dashboard-report-scope.test.ts | 15 +++++++++++++ src/stats.ts | 4 ++-- src/team-push.ts | 22 ++++++++++++++------ 4 files changed, 34 insertions(+), 9 deletions(-) diff --git a/docs/designs/data-directory-layout.md b/docs/designs/data-directory-layout.md index 4f1a449d5..642ef3c30 100644 --- a/docs/designs/data-directory-layout.md +++ b/docs/designs/data-directory-layout.md @@ -433,7 +433,7 @@ written before rollouts were kept is one total: an earlier release rewrote every session in the log on each report, so it covers the rollouts begun by the time its file was last written (read before this report writes it; a seed keeps the shared file's time, and `teamai stats`, which only reads, writes no seed). Those still in -the log consume it in order, what is left is the dropped rollouts', kept as one +the log consume it in order, as far as each had got by that time, what is left is the dropped rollouts', kept as one prior rollout, and a rollout begun later is new. Compaction also keeps a session whose tool process is still running, so a run an exit from a dashboard before `processExitAfter` marked stopped keeps its diff --git a/src/__tests__/dashboard-report-scope.test.ts b/src/__tests__/dashboard-report-scope.test.ts index 9c9354a8a..82be32a31 100644 --- a/src/__tests__/dashboard-report-scope.test.ts +++ b/src/__tests__/dashboard-report-scope.test.ts @@ -824,6 +824,21 @@ describe('each scope keeps its own reported snapshot (#786)', () => { sum + (d && typeof d === 'object' && 'costMicros' in d && typeof d.costMicros === 'number' ? d.costMicros : 0), 0)).toBe(20); }); + it('a whole entry from before covers a rollout still running only as far as it had got', async () => { + const { project } = await setup(); + const { event, write, stats } = await codexLog(project); + const today = new Date().toISOString().slice(0, 10); + // That release reported 6 prompts: rollout A's 5 and rollout B's first. + writeSharedSnapshots({ 'codex-s': 6 }, today, project); + const dir = path.join(getDataHome(project), 'dashboard'); + const writtenAt = new Date(Date.now() - 20 * 60_000); + for (const name of SNAPSHOTS) fs.utimesSync(path.join(dir, `reported-${name}.json`), writtenAt, writtenAt); + // A was compacted; B had 1 prompt by then, and has 3 now. + write([event('rollout-b.jsonl', 'stop', 30, { prompts: 1 }), event('rollout-b.jsonl', 'stop', 50, { prompts: 3 })]); + + expect((await stats()).prompts).toBe(2); + }); + it('reading the baselines without persisting them, as teamai stats does, does not move the time they cover', async () => { const { project } = await setup(); const { event, write, stats } = await codexLog(project); diff --git a/src/stats.ts b/src/stats.ts index 6c69d0224..8e39969c6 100644 --- a/src/stats.ts +++ b/src/stats.ts @@ -157,7 +157,7 @@ async function unreportedDashboardStats( config: LocalConfig, ): Promise { const { - computeInterventionDelta, computePromptTokenDelta, droppedRollouts, interventionCounts, reportedBaselines, + computeInterventionDelta, computePromptTokenDelta, droppedRollouts, interventionCounts, metricsAsOf, reportedBaselines, snapshotWrittenAt, withDroppedRollouts, } = await import('./team-push.js'); const { aggregateDailySessions } = await import('./session-trends.js'); @@ -165,7 +165,7 @@ async function unreportedDashboardStats( const writtenAt = await snapshotWrittenAt(config); const currentDaily = aggregateDailySessions(events); const { promptTokens, interventions, daily } = await reportedBaselines(events, metrics, currentDaily, config, false); - const dropped = droppedRollouts(metrics, promptTokens, interventions, daily, writtenAt); + const dropped = droppedRollouts(metrics, promptTokens, interventions, daily, writtenAt, metricsAsOf(events, writtenAt)); const currentInterventions = withDroppedRollouts(interventionCounts(metrics), currentDaily, dropped).interventions; const interventionDelta = computeInterventionDelta(currentInterventions, interventions); diff --git a/src/team-push.ts b/src/team-push.ts index c1cf33989..97c17c3a0 100644 --- a/src/team-push.ts +++ b/src/team-push.ts @@ -410,10 +410,12 @@ export interface DroppedRollout { * * An entry written before rollouts were kept is one total. An earlier release * rewrote every session in the log on each report, so that total covers the - * rollouts that had begun by `writtenAt`, when its file was last written: those - * still in the log consume it in order, and what is left is the dropped ones', - * kept as one prior rollout. A rollout begun after it is new. Without - * `writtenAt`, every rollout in the log is taken as covered. + * rollouts that had begun by `writtenAt`, when its file was last written, as + * far as each had got by then (`before`: the metrics of the events up to + * `writtenAt`): those still in the log consume it in order, and what is left + * is the dropped ones', kept as one prior rollout. A rollout begun after it is + * new, as is what one begun before it has done since. Without `writtenAt`, + * every rollout in the log is taken as covered, as it is now. */ export function droppedRollouts( current: Map, @@ -421,6 +423,7 @@ export function droppedRollouts( interventions: ReportedInterventions, daily: ReportedDailySessions, writtenAt?: number, + before?: Map, ): Map { const dropped = new Map(); for (const [sid, cur] of current) { @@ -447,7 +450,7 @@ export function droppedRollouts( prompts: total.prompts, tokens: { ...total.tokens }, interrupt: iv.interrupt, toolReject: iv.toolReject, correction: iv.correction, durationMs: day?.durationMs ?? 0, requestDaily: { ...(day?.requestDaily ?? {}) }, }; - const covered = Object.values(cur.segments) + const covered = Object.values((writtenAt !== undefined && before?.get(sid)?.segments) || cur.segments) .filter((segment) => writtenAt === undefined || Date.parse(segment.since) <= writtenAt) .sort((a, b) => Date.parse(a.since) - Date.parse(b.since)); const take = (own: number, rest: number) => Math.max(0, Math.min(own, rest)); @@ -684,6 +687,11 @@ async function creditSplitRuns( return result; } +/** The metrics of `events` up to `at`, what an entry written then saw; undefined without `at`. */ +export function metricsAsOf(events: DashboardEvent[], at: number | undefined): Map | undefined { + return at === undefined ? undefined : aggregateSessionMetrics(events.filter((e) => Date.parse(e.timestamp) <= at)); +} + /** * When the scope's prompt-token snapshot was last written, before this report * writes it: its own file's, else the shared one it would be seeded from. An @@ -859,7 +867,9 @@ export async function reportUsageToTeam( const { interventions: reportedInterventions, promptTokens: reportedPromptTokens, daily: reportedDailySessions, } = await reportedBaselines(dashboardEvents, metrics, currentDaily, reportsConfig, true); - const dropped = droppedRollouts(metrics, reportedPromptTokens, reportedInterventions, reportedDailySessions, writtenAt); + const dropped = droppedRollouts( + metrics, reportedPromptTokens, reportedInterventions, reportedDailySessions, writtenAt, metricsAsOf(dashboardEvents, writtenAt), + ); const effective = withDroppedRollouts(currentInterventions, currentDaily, dropped); const { delta: promptTokenDelta, nextReported: nextReportedPromptTokens } = computePromptTokenDelta( metrics, From 2482931524d0e587a8fae79637c76a0166c93a71 Mon Sep 17 00:00:00 2001 From: Saul Moro Date: Fri, 25 Sep 2026 03:18:18 +0200 Subject: [PATCH 41/41] fix(stats): credit a split session counter by counter; read an old entry's cutoff before its push (#785) Both credit paths applied only when the parts' prompts exceeded the owner's, so a part that reported more active time, tokens or costs with no more prompts was sent again by the owner. The owner's entry is now raised counter by counter to at least the credit. An earlier release wrote its snapshot after the push, so events that arrived during the push predate the snapshot's time without being in it. The team stats file in the scope's reports checkout was written after that report read the log and before the push; the earlier of the two times is now the cutoff an entry from before covers. --- docs/designs/data-directory-layout.md | 10 +- src/__tests__/dashboard-report-scope.test.ts | 53 ++++++++++ src/team-push.ts | 103 +++++++++++++++---- 3 files changed, 142 insertions(+), 24 deletions(-) diff --git a/docs/designs/data-directory-layout.md b/docs/designs/data-directory-layout.md index 642ef3c30..57e489fc2 100644 --- a/docs/designs/data-directory-layout.md +++ b/docs/designs/data-directory-layout.md @@ -411,8 +411,9 @@ snapshots already hold it; else it is decided as before (a fork under a new ID, a tool whose transcript records no start). A session main split across scopes per event, whose events are still in the log with each part's `dataHome`, is credited once with every part reported: for each scope, the shortest prefix of -its events whose metrics reach its snapshot, and the owner takes the metrics of -their union as reported when they exceed its own entry, so parts counted before +its events whose metrics reach its snapshot, and the owner's entry is raised, +counter by counter, to at least the metrics of their union (a part may have +reported more time, tokens or costs with no more prompts), so parts counted before any Stop carried the transcript's total are neither lost nor sent twice. A Codex session (any Codex variant: `codex`, `codex-internal`, `tcodex`) is kept per rollout, with or without a token record, and when @@ -431,7 +432,10 @@ keeps the session unsuccessful, so a later rollout is reported in full and does not turn it into a success. An entry written before rollouts were kept is one total: an earlier release rewrote every session in the log on each report, so it covers the rollouts begun by the time -its file was last written (read before this report writes it; a seed keeps the +its file was last written, or, earlier, when that report wrote the team stats +file in this scope's reports checkout (after reading the log, before its push; +the snapshot came after the push). That is read before this report writes +anything; a seed keeps the shared file's time, and `teamai stats`, which only reads, writes no seed). Those still in the log consume it in order, as far as each had got by that time, what is left is the dropped rollouts', kept as one prior rollout, and a rollout begun later is new. diff --git a/src/__tests__/dashboard-report-scope.test.ts b/src/__tests__/dashboard-report-scope.test.ts index 82be32a31..b4ae939d6 100644 --- a/src/__tests__/dashboard-report-scope.test.ts +++ b/src/__tests__/dashboard-report-scope.test.ts @@ -839,6 +839,27 @@ describe('each scope keeps its own reported snapshot (#786)', () => { expect((await stats()).prompts).toBe(2); }); + it('a whole entry from before covers what its report read, not what came in during its push', async () => { + const { project } = await setup(); + const { event, write, stats } = await codexLog(project); + const today = new Date().toISOString().slice(0, 10); + // That release read the log (A 5, B 1), wrote the team stats file, pushed, then its snapshot. + writeSharedSnapshots({ 'codex-s': 6 }, today, project); + const dir = path.join(getDataHome(project), 'dashboard'); + const teamStats = path.join(path.dirname(project.repo.localPath), 'reports-wt', 'stats', 'tester.yaml'); + fs.mkdirSync(path.dirname(teamStats), { recursive: true }); + fs.writeFileSync(teamStats, YAML.stringify({ username: 'tester', skills: {} })); + const read = new Date(Date.now() - 25 * 60_000); + const pushed = new Date(Date.now() - 15 * 60_000); + fs.utimesSync(teamStats, read, read); + for (const name of SNAPSHOTS) fs.utimesSync(path.join(dir, `reported-${name}.json`), pushed, pushed); + // A was compacted; B reached 2 during that push and has 3 now. + write([event('rollout-b.jsonl', 'stop', 30, { prompts: 1 }), event('rollout-b.jsonl', 'stop', 40, { prompts: 2 }), + event('rollout-b.jsonl', 'stop', 55, { prompts: 3 })]); + + expect((await stats()).prompts).toBe(2); + }); + it('reading the baselines without persisting them, as teamai stats does, does not move the time they cover', async () => { const { project } = await setup(); const { event, write, stats } = await codexLog(project); @@ -1033,6 +1054,38 @@ describe('each scope keeps its own reported snapshot (#786)', () => { expect(await report(projectQ)).toBeNull(); }); + it('main split a session with the same prompts in each part: the owner still credits every part\'s active time', async () => { + const { root, project } = await setup(); + const { rootQ, projectQ } = await setupQ(); + const p = getDataHome(project); + const q = getDataHome(projectQ); + const tokens = (input: number) => ({ input, output: 0, cacheRead: 0, cacheCreation: 0 }); + const at = (minute: number, type: string, cwd: string, dataHome: string, extra: Record = {}) => JSON.stringify({ + type, timestamp: new Date(Date.now() - 3_600_000 + minute * 60_000).toISOString(), sessionId: 'split', tool: 'claude', cwd, dataHome, ...extra, + }); + const log = path.join(teamaiHome(), 'dashboard', 'events.jsonl'); + fs.mkdirSync(path.dirname(log), { recursive: true }); + // One prompt. P's part was active 4 minutes; Q's, after a pause, 3 more, with no new prompt. + fs.writeFileSync(log, [ + at(0, 'prompt_submit', root, p), at(4, 'stop', root, p, { prompts: 1, tokens: tokens(100) }), + at(10, 'tool_use', rootQ, q), at(12, 'tool_use', rootQ, q), at(13, 'stop', rootQ, q, { prompts: 1, tokens: tokens(150) }), + ].join('\n') + '\n'); + const date = new Date(Date.now() - 3_600_000).toISOString().slice(0, 10); + for (const [config, input, minutes] of [[project, 100, 4], [projectQ, 150, 3]] as const) { + const dir = path.join(getDataHome(config), 'dashboard'); + fs.mkdirSync(dir, { recursive: true }); + fs.writeFileSync(path.join(dir, 'reported-prompt-tokens.json'), JSON.stringify({ split: { prompts: 1, tokens: tokens(input) } })); + fs.writeFileSync(path.join(dir, 'reported-interventions.json'), JSON.stringify({ split: { interrupt: 0, toolReject: 0, correction: 0 } })); + fs.writeFileSync(path.join(dir, 'reported-daily-sessions.json'), JSON.stringify({ split: { + date, prompts: 1, durationMs: minutes * 60_000, succeeded: 1, corrected: 0, requestDaily: {}, + } })); + } + + // Both parts already reported their time: the owner sends none of it again. + expect(await report(projectQ)).toBeNull(); + expect(await report(project)).toBeNull(); + }); + it('main split a session whose parts each ended in a cumulative Stop: a new prompt is reported once', async () => { const { root, project } = await setup(); const { rootQ, projectQ } = await setupQ(); diff --git a/src/team-push.ts b/src/team-push.ts index 97c17c3a0..5f65065b2 100644 --- a/src/team-push.ts +++ b/src/team-push.ts @@ -18,7 +18,7 @@ import { log } from './utils/logger.js'; import type { UserStats, UserInterventionStats, SessionMetrics, TokenUsage, DashboardEvent, LocalConfig, RequestCostMetrics, } from './types.js'; -import { getVotesDir, getDataHome, emptyTokenUsage, addTokenUsage, usesBranchWorktree } from './types.js'; +import { getVotesDir, getDataHome, getReportsDir, emptyTokenUsage, addTokenUsage, usesBranchWorktree } from './types.js'; import { filterEventsByScope, runsOfLog } from './dashboard-scope.js'; import { creditedPrompts, interventionsEntry, promptTokensEntry, readOwnerCredits, recordSessionOwners, reportedBeyondShared, reportedSize, @@ -647,11 +647,11 @@ async function creditSplitRuns( // Placed by the transcript where it can tell what the parts overlapped. const transcripts = runEvents.flatMap((e) => (typeof e.transcriptPath === 'string' ? [e.transcriptPath] : [])); const prompts = (await creditedPrompts(credit, transcripts)) ?? credit.promptTokens.prompts; - if (reportedSize(result.promptTokens[runId]).prompts >= prompts) continue; - result.promptTokens[runId] = { ...credit.promptTokens, prompts }; - result.interventions[runId] = credit.interventions; - if (credit.daily) result.daily[runId] = { ...credit.daily, prompts }; - result.changed = true; + result.changed = raiseToCredit(result, runId, { + promptTokens: { ...credit.promptTokens, prompts }, + interventions: credit.interventions, + daily: credit.daily ? { ...credit.daily, prompts } : undefined, + }) || result.changed; continue; } if (homes.size < 2) continue; @@ -676,17 +676,78 @@ async function creditSplitRuns( if (union.length === 0) continue; const metrics = aggregateSessionMetrics(union); const promptTokens = computePromptTokenDelta(metrics, {}).nextReported[runId]; - if (!promptTokens || reportedSize(result.promptTokens[runId]).prompts >= promptTokens.prompts) continue; - result.promptTokens[runId] = promptTokens; - const interventions = interventionCounts(metrics).get(runId); - if (interventions) result.interventions[runId] = interventions; - const daily = computeDailyStatsDelta(aggregateDailySessions(union), {}).nextReported[runId]; - if (daily) result.daily[runId] = daily; - result.changed = true; + if (!promptTokens) continue; + result.changed = raiseToCredit(result, runId, { + promptTokens, + interventions: interventionCounts(metrics).get(runId), + daily: computeDailyStatsDelta(aggregateDailySessions(union), {}).nextReported[runId], + }) || result.changed; } return result; } +/** + * Raises a run's baselines to at least `credit`, counter by counter: a part + * may have reported more time, tokens or costs with no more prompts. Flags + * and dates stay the run's own where it has an entry. Whether anything moved. + */ +function raiseToCredit( + result: { interventions: ReportedInterventions; promptTokens: ReportedPromptTokens; daily: ReportedDailySessions }, + runId: string, + credit: { + promptTokens: ReportedPromptTokens[string]; + interventions: ReportedInterventions[string] | undefined; + daily: DailySessionSnapshot | undefined; + }, +): boolean { + const before = JSON.stringify([result.promptTokens[runId], result.interventions[runId], result.daily[runId]]); + const own = result.promptTokens[runId]; + const ownTokens = promptTokensEntry(own); + const greater = (a: TokenUsage, b: TokenUsage): TokenUsage => ({ + input: Math.max(a.input, b.input), output: Math.max(a.output, b.output), + cacheRead: Math.max(a.cacheRead, b.cacheRead), cacheCreation: Math.max(a.cacheCreation, b.cacheCreation), + }); + result.promptTokens[runId] = { + ...(own ?? {}), + prompts: Math.max(ownTokens.prompts, credit.promptTokens.prompts), + tokens: greater(ownTokens.tokens, credit.promptTokens.tokens), + }; + if (credit.interventions) { + const iv = interventionsEntry(result.interventions[runId]); + result.interventions[runId] = { + interrupt: Math.max(iv.interrupt, credit.interventions.interrupt), + toolReject: Math.max(iv.toolReject, credit.interventions.toolReject), + correction: Math.max(iv.correction, credit.interventions.correction), + }; + } + if (credit.daily) { + const day = parseDailySnapshot(result.daily[runId]); + if (!day) { + result.daily[runId] = credit.daily; + } else { + const requestDaily = { ...day.requestDaily }; + for (const [date, request] of Object.entries(credit.daily.requestDaily)) { + const mine = requestDaily[date]; + requestDaily[date] = mine ? { + ...mine, + pricedRequests: Math.max(mine.pricedRequests, request.pricedRequests), costMicros: Math.max(mine.costMicros, request.costMicros), + cacheReadTokens: Math.max(mine.cacheReadTokens, request.cacheReadTokens), + cacheEligibleInputTokens: Math.max(mine.cacheEligibleInputTokens, request.cacheEligibleInputTokens), + } : request; + } + result.daily[runId] = { + ...day, + prompts: Math.max(day.prompts, credit.daily.prompts), + durationMs: Math.max(day.durationMs, credit.daily.durationMs), + sessionCacheReadTokens: Math.max(day.sessionCacheReadTokens ?? 0, credit.daily.sessionCacheReadTokens ?? 0), + sessionCacheEligibleTokens: Math.max(day.sessionCacheEligibleTokens ?? 0, credit.daily.sessionCacheEligibleTokens ?? 0), + requestDaily, + }; + } + } + return JSON.stringify([result.promptTokens[runId], result.interventions[runId], result.daily[runId]]) !== before; +} + /** The metrics of `events` up to `at`, what an entry written then saw; undefined without `at`. */ export function metricsAsOf(events: DashboardEvent[], at: number | undefined): Map | undefined { return at === undefined ? undefined : aggregateSessionMetrics(events.filter((e) => Date.parse(e.timestamp) <= at)); @@ -698,14 +759,14 @@ export function metricsAsOf(events: DashboardEvent[], at: number | undefined): M * entry an earlier release left covers what that release had seen by then. */ export async function snapshotWrittenAt(config: LocalConfig | undefined): Promise { - for (const file of [scopeSnapshotPath('prompt-tokens', config), sharedSnapshotPath('prompt-tokens')]) { - try { - return (await fs.promises.stat(file)).mtimeMs; - } catch { - // Not written yet: try the next. - } - } - return undefined; + const mtime = (file: string) => fs.promises.stat(file).then((stat) => stat.mtimeMs, () => undefined); + const own = await mtime(scopeSnapshotPath('prompt-tokens', config)); + if (own === undefined) return mtime(sharedSnapshotPath('prompt-tokens')); + // An earlier release wrote the snapshot after its push, and the team stats + // file in this scope's reports checkout before it, after reading the log: the + // earlier of the two is nearer what that report had read. + const stats = config ? await mtime(path.join(getReportsDir(config), 'stats', `${config.username}.yaml`)) : undefined; + return stats === undefined ? own : Math.min(own, stats); } /**