fix(usage): cap usage.jsonl in scopes that never report (#788)

CHANGELOG.md

@@ -27,6 +27,7 @@ All notable changes to this project will be documented in this file. See [standa
 
 ### 🐛 Bug Fixes
 
+- A scope's usage file no longer grows without bound where it is never reported: an http source, a team with `usageReport: false`, or a remote that rejects every push. `teamai pull` keeps its newest 5,000 events, so `teamai stats` still shows recent usage there, the local file being its only source. In a reporting scope whose report does not complete while it holds more than 5,000 events, the dropped events were never reported. The cap runs after the report has removed the events it sent, so it cannot shift that removal onto events not yet reported, and a file at or below the cap is not rewritten. Hook appends, the report's truncate and the cap now share one lock beside the usage file, and a rewrite goes through a temp file that keeps the file's mode, so a rewrite no longer loses an event recorded while it runs, nor the file on a kill or a full disk. A hook that cannot take the lock within ~250 ms records its event in a `*.pending-<id>.jsonl` file beside it, which the next lock holder appends; a rewrite that cannot take it within ~5 s leaves the file as it is. A pending file gets no wider mode than the usage file (owner-only while there is none), and an in-workspace `.teamai/.gitignore` ignores the lock, a rewrite's temp copy and the pending files; `pull` and `push` add those entries to an existing single-repo one, and the first pending file or rewrite adds them to an existing project-scope one (for [#788](https://github.com/Tencent/teamai-cli/issues/788)).
 - `teamai stats` no longer counts a session twice, and no longer counts another project's sessions. Its dashboard section added the whole machine's local `events.jsonl` metrics to the scope's already-reported totals from the team repo, so every session that a `pull` had reported — and that stays in the event log until compaction — was counted once by the team total and once again locally, and sessions whose `cwd` belonged to a different project were added to this scope's as well. It now filters the event log the way `teamai pull` reports it (only the sessions recorded in the current scope, see #785) and adds only what that scope has not reported yet, derived from the same per-scope `reported-*` snapshots the report path advances — so the local figure agrees with the team's instead of exceeding it. When the reported totals could not be read at all (no stats file, an unreadable one, a reports worktree that is not there), or when they exist but hold nothing yet, nothing is subtracted — a snapshot can name a session the team file never received, so a non-empty team total is what licenses trusting them, and a session the member can see happening is never hidden. The per-repo and by-hour breakdowns read that same filtered log, so they stay inside the scope; they answer a different question from the headline — what this machine's retained event log holds, per repo — and both the headings and the `--by-repo` / `--by-time` flag descriptions now name that source instead of presenting them as a split of the headline (for [#768](https://github.com/Tencent/teamai-cli/issues/768)).
 - The data-partition migration no longer retires a legacy `<repo>/.teamai/` while the partition's `config.yaml` cannot be read. `teamai init`, `pull` and `push` took a partition config that merely existed as a finished migration and renamed the legacy directory to `.teamai.bak`, although it held the only config that still loaded. A partition `config.yaml` that is empty or cannot be opened, does not parse, does not validate, or is not `scope: project` now leaves the legacy directory in place and names the file to fix; once it is fixed, the next of those commands retires it as before. A partition directory whose `config.yaml` was moved aside is no longer replaced by a fresh copy of the legacy directory, which deleted what the partition held; the migration waits and says to restore the file or move the directory aside (for [#797](https://github.com/Tencent/teamai-cli/issues/797)).
 - `teamai recall <query>` searches nothing in a project whose config exists but cannot be read, and says why. Detection skipped the broken file and searched whatever loaded next: the user scope, or a legacy `.teamai/` behind a broken partition that may belong to another team, whose knowledge was returned and whose documents got recalled counts. It now prints ``Nothing was searched: <file>: <reason>. Fix the file, or move it aside and run `teamai init` to write a new one.`` and exits 1, and `recall --check` does the same instead of printing `NOT_RELEVANT`, which told the recall subagent the team had no knowledge. The `teamai-recall` subagent a pull from this release deploys relays that line to the member, telling the main conversation to move the file or re-run `teamai init` only with their consent. (for [#796](https://github.com/Tencent/teamai-cli/issues/796)).

docs/designs/team-intelligence-platform.md

@@ -22,7 +22,7 @@ Transform TeamAI from a simple skill-sharing CLI into a **Team Intelligence Plat
 | 7 | Session 内容来源 | AI 对话和工具使用（非 git diff） | 核心目的是记录 AI 用错工具的情况 |
 | 8 | Session 过滤 | 有价值才推送（包含工具错误/重试→有价值） | 避免团队仓库充斥无用信息 |
 | 9 | Hook 追踪范围 | 只追踪 Skill 工具调用 | 追踪所有工具会产生大量噪音数据，核心需求是 skill 使用统计 |
-| 10 | JSONL 生命周期 | 全部选中目标确认成功后截断已上报数据 | 失败保留事件；推送期间新增事件保留到下一批 |
+| 10 | JSONL 生命周期 | 全部选中目标确认成功后截断已上报数据 | 失败保留事件（最多最新 5,000 条）；推送期间新增事件保留到下一批 |
 | 11 | 错误处理策略 | 所有 I/O 操作 try-catch + graceful degrade | 零静默失败原则 |
 | 12 | Feature 3 推送方式 | 删除 `push --stats`，只用 auto-report | Decision 5 已决定自动上报，手动命令冗余 |
 | 13 | Hook 命令实现 | `teamai track` TypeScript CLI 命令 | 类型安全 + 输入验证，比 bash one-liner 稳健 |
@@ -108,6 +108,16 @@ reports worktree → git add → git commit → git push (teamai-reports)
 推送和本地确认，并在完成后释放相关同步锁。失败重试即使重建出与上次提交
 相同的文件，也会重新推送已有提交，无需创建空提交。
 
+所有目标处理完后，Pull 把每个活跃 scope 的 usage 文件截为最新 5,000 条（含
+HTTP 与 `usageReport: false` 这些从不上报的 scope，#788）。截断必须在清理之后：
+清理按上报读到的条数删除文件开头的行，中间插入截断会把删除挪到未上报的事件上。
+Hook 追加、清理与截断共用 usage 文件旁的一把锁（`<usage 文件>.lock`），改写经临时文件
+再 rename，保留原文件权限。Hook 约 250 ms 拿不到锁时写入 `*.pending-<id>.jsonl`（O_EXCL），
+由下一个持锁者并入；改写约 5 秒拿不到锁时不改文件。pending 文件权限不宽于 usage 文件
+（尚无 usage 文件时为 0600）。工作区内 `.teamai/.gitignore` 忽略 `usage.jsonl.*` 与
+`usage.pending-*.jsonl`，单仓库模式的已有文件由 `migrateSelfModeGitignoreContent` 补齐，
+项目级的已有文件在写 pending 文件或改写前由 `ignoreUsageSideFiles` 补齐。
+
 此修复不提供持久化批次或远端去重：进程在推送与确认之间终止、多仓库部分
 成功后的重试仍可能产生重复统计。等待超时不等于取消 Git 或强制退出进程。
 

docs/usage-guide.md

@@ -1778,7 +1778,7 @@ reported, so a session that moved into another project mid-session reaches both
 teams with the part recorded in each; the first report after upgrading starts
 from the snapshot every scope used to share, so nothing is reported twice. A
 target removes its usage events only after it confirms success; failed pushes
-preserve them. The affected sync locks remain
+preserve them, up to the newest 5,000 (see below). The affected sync locks remain
 held until reporting finishes, preventing another pull from racing the report.
 
 This is best-effort reporting, not crash-safe delivery: termination between a
@@ -1794,6 +1794,23 @@ can turn this off in `teamai.yaml`:
 usageReport: false
 ```
 
+Pull keeps each scope's usage file to its newest 5,000 events, dropping the
+oldest after the report step. For an http source or a `usageReport: false`
+team, that file is the only record `teamai stats` has, so it stays bounded
+without going empty; a reporting scope whose report does not complete while
+it holds more than 5,000 drops its oldest unreported events the same way. The
+cap runs only after a report has removed the events it sent. Hook appends, the
+report's truncate and the cap take one lock beside the usage file, so a rewrite
+does not lose an event recorded while it runs. A hook that cannot take the lock
+within ~250 ms records its event in a `*.pending-<id>.jsonl` file next to it,
+which the next lock holder appends to the usage file; a rewrite that cannot take
+it within ~5 s leaves the file as it is. A pending file gets no wider mode than
+the usage file (owner-only while there is none). An in-workspace
+`.teamai/.gitignore` ignores the lock, a rewrite's temp copy and the pending
+files; `pull` and `push` add those entries to an existing single-repo one, and
+the usage file's first pending file or rewrite adds them to an existing
+project-scope one.
+
 ### Git submodules
 
 If your team distributes skills as git submodules, opt in with `submodules: true`

docs/usage-guide.zh-CN.md

@@ -1709,7 +1709,7 @@ Copilot 会话以及从软链接路径启动的会话。旧版本记录的事件
 位于其根目录下的那些，user scope 一条也不上报。每个 scope 还各自保存已上报快照，
 因此会话中途切换到另一个项目时，两个团队各自收到在其中记录的那部分；升级后的首次上报
 从原先各 scope 共用的快照开始，不会重复上报。目标确认成功后才清理自己的使用事件，
-推送失败会保留事件。上报完成前继续持有相关同步锁，
+推送失败会保留事件（最多保留最新 5,000 条，见下文）。上报完成前继续持有相关同步锁，
 避免另一次 Pull 与尚未完成的上报竞争。
 
 这仍是尽力上报，不提供崩溃恢复保证：远端推送成功与本地确认之间如果进程
@@ -1723,6 +1723,16 @@ Copilot 会话以及从软链接路径启动的会话。旧版本记录的事件
 usageReport: false
 ```
 
+Pull 在上报步骤之后把每个 scope 的使用文件限制为最新 5,000 条事件，丢弃更早的
+事件。对 HTTP 源或 `usageReport: false` 的团队，该文件是 `teamai stats` 唯一的
+数据来源，因此文件保持有界而不会被清空；上报未完成且事件超过 5,000 条的上报
+scope 也以同样方式丢弃最早的未上报事件。该上限只在上报清理完已发送事件之后执行。Hook 追加、上报后的清理与该上限共用使用文件旁的一把锁，
+因此改写文件时不会丢失期间记录的事件。Hook 在约 250 ms 内拿不到锁时，把事件写入旁边的
+`*.pending-<id>.jsonl` 文件，由下一个持锁者追加进使用文件；改写在约 5 秒内拿不到锁时保持文件不变。
+pending 文件的权限不宽于使用文件（尚无使用文件时仅所有者可读写）。工作区内的 `.teamai/.gitignore`
+忽略该锁、改写的临时副本与 pending 文件；`pull` 与 `push` 会为已有的单仓库 `.gitignore` 补上这些条目，
+已有的项目级 `.gitignore` 则在使用文件第一次写 pending 文件或改写时补上。
+
 ### Git 子模块
 
 若团队以 git submodule 形式分发 skill，在 `teamai.yaml` 中开启 `submodules: true`：

src/__tests__/e2e/report-timeout.test.ts

@@ -15,19 +15,19 @@ function git(args: string[], cwd: string, env: NodeJS.ProcessEnv): string {
   return execFileSync('git', args, { cwd, env, encoding: 'utf8', windowsHide: true });
 }
 
-function fixture(agent: keyof typeof agents, provider: string) {
+function fixture(agent: keyof typeof agents, provider: string, team: Record<string, unknown> = {}) {
   sandbox = fs.mkdtempSync(path.join(os.tmpdir(), 'teamai-report-timeout-'));
   const home = path.join(sandbox, 'home');
   const seed = path.join(sandbox, 'seed');
   const remote = path.join(sandbox, 'remote.git');
   const clone = path.join(home, '.teamai/team-repo');
   fs.mkdirSync(seed, { recursive: true });
   fs.mkdirSync(path.join(home, agents[agent]), { recursive: true });
-  const env = { ...process.env, HOME: home, USERPROFILE: home, XDG_CONFIG_HOME: path.join(home, '.config'),
+  const env: NodeJS.ProcessEnv = { ...process.env, HOME: home, USERPROFILE: home, XDG_CONFIG_HOME: path.join(home, '.config'),
     GIT_CONFIG_GLOBAL: path.join(home, '.gitconfig'), GIT_CONFIG_NOSYSTEM: '1',
     GIT_AUTHOR_NAME: 'Report Test', GIT_AUTHOR_EMAIL: 'test@example.invalid',
     GIT_COMMITTER_NAME: 'Report Test', GIT_COMMITTER_EMAIL: 'test@example.invalid', FORCE_COLOR: '0' };
-  fs.writeFileSync(path.join(seed, 'teamai.yaml'), YAML.stringify({ team: 'report-test', repo: remote, provider }));
+  fs.writeFileSync(path.join(seed, 'teamai.yaml'), YAML.stringify({ team: 'report-test', repo: remote, provider, ...team }));
   git(['init', '-q', '-b', 'main'], seed, env);
   git(['add', '.'], seed, env);
   git(['commit', '-q', '-m', 'fixture'], seed, env);
@@ -91,7 +91,10 @@ function fixture(agent: keyof typeof agents, provider: string) {
     const p = path.join(dashboard, `user-reported-${name}.json`);
     return fs.existsSync(p) ? JSON.parse(fs.readFileSync(p, 'utf8')) : {};
   }
-  return { home, usage, usageLine, seedEvents, receiver, pull, stats, snapshot };
+  function run(args: string[]) {
+    return execFileSync(process.execPath, [cli, ...args], { cwd: sandbox, env, encoding: 'utf8', windowsHide: true });
+  }
+  return { home, env, clone, usage, usageLine, dashboard, seedEvents, receiver, pull, stats, snapshot, run };
 }
 
 afterEach(() => {
@@ -153,3 +156,73 @@ describe('real CLI report completion', () => {
     expect(fs.readFileSync(f.usage, 'utf8')).toBe('');
   }, 60_000);
 });
+
+describe('real CLI usage cap (#788)', () => {
+  const cap = 5_000;
+  const line = (skill: string) => JSON.stringify({ skill, tool: 'claude', timestamp: new Date().toISOString() });
+  /** `legacy` oldest, `review` newest: only the newest survive a cap. */
+  function seedUsage(file: string, legacy: number, review: number) {
+    fs.writeFileSync(file, [...Array(legacy).fill(line('legacy')), ...Array(review).fill(line('review'))].join('\n') + '\n');
+  }
+  const lines = (file: string) => fs.readFileSync(file, 'utf8').split('\n').filter(Boolean);
+
+  it('caps a usageReport: false scope and stats still shows its newest usage', async () => {
+    const f = fixture('claude', 'git', { usageReport: false });
+    seedUsage(f.usage, 100, cap);
+    await f.pull();
+    expect(lines(f.usage)).toEqual(Array(cap).fill(expect.stringContaining('"review"')));
+    const stats = f.run(['stats']);
+    expect(stats).toMatch(/review\s+5000 uses/);
+    expect(stats).not.toContain('legacy');
+  }, 60_000);
+
+  it('caps an http scope and stats still shows its newest usage', async () => {
+    const f = fixture('claude', 'git');
+    fs.writeFileSync(path.join(f.home, '.teamai/config.yaml'), YAML.stringify({
+      repo: { localPath: f.clone, remote: 'http://127.0.0.1:9/team', kind: 'http', url: 'http://127.0.0.1:9' },
+      username: 'alice', scope: 'user', updatePolicy: 'skip', enabledAgents: ['claude'], additionalRoles: [],
+    }));
+    f.env.TEAMAI_API_TOKEN = 'unused';
+    seedUsage(f.usage, 100, cap);
+    await f.pull();
+    expect(lines(f.usage)).toEqual(Array(cap).fill(expect.stringContaining('"review"')));
+    const stats = f.run(['stats']);
+    expect(stats).toMatch(/review\s+5000 uses/);
+    expect(stats).not.toContain('legacy');
+  }, 60_000);
+
+  it('does not rewrite a usage file below the cap', async () => {
+    const f = fixture('claude', 'git', { usageReport: false });
+    seedUsage(f.usage, 0, 10);
+    const past = new Date('2026-01-01T00:00:00Z');
+    fs.utimesSync(f.usage, past, past);
+    await f.pull();
+    expect(fs.statSync(f.usage).mtimeMs).toBe(past.getTime());
+    expect(lines(f.usage)).toHaveLength(10);
+  }, 60_000);
+
+  it('caps a reporting scope whose push is rejected and reports its newest events on retry', async () => {
+    const f = fixture('claude', 'git');
+    await f.pull(); // Warm reports worktree without any session data.
+    seedUsage(f.usage, 100, cap);
+    f.receiver('reject');
+    await f.pull();
+    expect(lines(f.usage)).toEqual(Array(cap).fill(expect.stringContaining('"review"')));
+    f.receiver('normal');
+    await f.pull();
+    expect(f.stats().skills.review.count).toBe(cap);
+    expect(f.stats().skills.legacy).toBeUndefined();
+    expect(lines(f.usage)).toEqual([]);
+  }, 60_000);
+
+  it('a reporting scope over the cap keeps every event written after the report read it', async () => {
+    const f = fixture('claude', 'git');
+    await f.pull(); // Warm reports worktree without any session data.
+    seedUsage(f.usage, 100, cap);
+    f.receiver('slow');
+    await f.pull(() => fs.appendFileSync(f.usage, line('late') + '\n' + line('late') + '\n'));
+    expect(lines(f.usage)).toEqual([expect.stringContaining('"late"'), expect.stringContaining('"late"')]);
+    expect(f.stats().skills.legacy.count).toBe(100);
+    expect(f.stats().skills.review.count).toBe(cap);
+  }, 60_000);
+});

src/__tests__/pull-learnings-deletion.test.ts

@@ -48,6 +48,7 @@ vi.mock('../team-push.js', () => ({ reportUsageToTeam: vi.fn().mockResolvedValue
 vi.mock('../usage-tracker.js', () => ({
   readUsageEvents: vi.fn().mockResolvedValue([]),
   truncateUsageAfterReport: vi.fn().mockResolvedValue(undefined),
+  capUsageEvents: vi.fn().mockResolvedValue(undefined),
 }));
 vi.mock('../roles.js', () => ({
   loadRolesManifest: vi.fn().mockResolvedValue({

src/__tests__/pull-placement-reconcile.test.ts

@@ -56,6 +56,7 @@ vi.mock('../team-push.js', () => ({ reportUsageToTeam: vi.fn().mockResolvedValue
 vi.mock('../usage-tracker.js', () => ({
   readUsageEvents: vi.fn().mockResolvedValue([]),
   truncateUsageAfterReport: vi.fn().mockResolvedValue(undefined),
+  capUsageEvents: vi.fn().mockResolvedValue(undefined),
 }));
 vi.mock('../roles.js', () => ({
   loadRolesManifest: vi.fn().mockResolvedValue({ version: 1, roles: [], defaults: { shareTarget: 'primary-role' } }),

src/__tests__/pull-publishes-queue.test.ts

@@ -53,6 +53,7 @@ vi.mock('../team-push.js', () => ({ reportUsageToTeam: vi.fn().mockResolvedValue
 vi.mock('../usage-tracker.js', () => ({
   readUsageEvents: vi.fn().mockResolvedValue([]),
   truncateUsageAfterReport: vi.fn().mockResolvedValue(undefined),
+  capUsageEvents: vi.fn().mockResolvedValue(undefined),
 }));
 vi.mock('../roles.js', () => ({
   loadRolesManifest: vi.fn().mockResolvedValue({