Repository navigation
Git hooks fire for every worktree with no concurrency cap — 36 detached indexers, ~66GB RAM, host unusable #159
Description
Summary
install_hooks writes the auto-reindex hook into the main repository's .git/hooks/. Git worktrees share that directory through the common dir, so every git worktree add, commit and merge inside any worktree fires the hook and detaches another unbounded cce index.
For anyone running AI agents in git worktrees this scales with agent parallelism. On my machine it reached 36 concurrent indexers at ~1.85 GB RSS each (~66 GB on a 96 GB box), load average 426, and took down an unrelated production service on the same host — its origin stopped answering and Cloudflare served a connection error to users. The web app itself was healthy; the machine was simply out of memory.
This is distinct from #139 (68 cce serve processes). Same class of failure, different process and different trigger — and, as noted below, the 0.4.26 resource governor does not cover this path.
Root cause
src/context_engine/indexer/git_hooks.py:
HOOK_NAMES = ["post-commit", "post-checkout", "post-merge"]
def _hook_script() -> str:
bin_path = shlex.quote(_resolve_cce_binary())
return f"""{HOOK_MARKER}
{bin_path} index >/dev/null 2>&1 &
"""
def install_hooks(project_dir: str) -> list[str]:
hooks_dir = Path(project_dir) / ".git" / "hooks"Two properties combine badly:
.git/hooksis shared with every worktree. In a worktree.gitis a file pointing at<main>/.git/worktrees/<name>/, and hook lookup resolves to the common dir — so a hook installed once in the main checkout runs for all N worktrees. Improve git worktree support: hook installer + storage path collision #48 observed the inverse (hooks don't install from inside a worktree); the consequence that main-repo hooks fire for every worktree seems not to have been considered.- The spawn is unbounded.
&detaches it, so the process is orphaned to PID 1, git returns instantly, and nothing serialises, nices, or caps the result. There is no lock and no awareness of sibling indexers.
Creating a worktree fires post-checkout; the agent's first commit fires post-commit; a merge fires post-merge. I observed three separate indexers against a single worktree from one agent's activity.
Observed
- 35–36 concurrent
cce index, all PPID 1, oldest ~3h49m - ~1.85 GB RSS each, ≈66 GB of 96 GB
- Load average 426 (15-min average sustained >200 for hours)
- 8 concurrent agents, each cutting its own worktree under
/private/tmp/w-* - Host: macOS (Apple silicon), CCE 0.4.25, embedding backend ollama
- Recovery required killing all of them by hand; memory returned to 83% free immediately
Why the 0.4.26 resource governor (#142) does not cover this
I upgraded to 0.4.26 expecting it to help, and on reading it, it does not address this path:
| Mechanism | Why it misses |
|---|---|
Per-project index lock (fcntl.flock) |
Keyed per project, and every worktree resolves to a distinct project slug — so the lock never engages between worktrees. It dedupes same-project duplicates only. |
| Linux PSI memory-pressure deferral | Reads /proc/pressure/memory; explicitly a no-op off Linux. Nothing on macOS, which is where the agent-worktree workflow is common. |
| Idle auto-shutdown, ORT thread caps | Both apply to cce serve. This failure is standalone cce index invoked from a git hook. |
Suggested fixes
Any one of these would have prevented it:
- A global (not per-project) concurrency cap for hook-triggered indexing — the lock in fix(serve): resource governor for multi-instance system freezes #142 is the right mechanism at the wrong scope. Hook-spawned indexers should contend machine-wide.
- Detect worktrees in
install_hooks/ the hook itself and skip, or coalesce to the main checkout.git rev-parse --git-common-dirdistinguishes a worktree from a main checkout in one call. nicethe hook-spawned process. It's background maintenance; it should never compete with the user's foreground work.- Skip ephemeral worktrees. Agent harnesses create throwaway trees (
/private/tmp/w-*,<repo>/.claude/worktrees/*) that are deleted minutes later — indexing them is pure cost, since the index dies with the tree.
Related: #48 (hooks and worktrees), #52 (slug collision), #139/#142 (multi-instance resource exhaustion).
Workaround for others hitting this
Repoint the hooks at a wrapper that holds an fcntl semaphore (macOS has no flock(1)), caps concurrency machine-wide, dedupes per directory, runs at nice 10, and skips ephemeral worktree paths. After that change the same workload produces at most 2 concurrent indexers and the machine stays responsive.
Worth noting for anyone applying a local fix: cce init rewrites the hook files, so re-running it silently discards the workaround.
Activity
itsandyking commentedon Aug 13, 2026
Following up with a source-level check, since the obvious first response to this is "isn't that fixed by the 0.4.26 resource governor (#142)?"
It isn't, and the code is explicit about why.
resource_governoris imported exactly once in the whole package, insidecce serve:The
indexcommand contains no references tocap_ort_threads,ProjectIndexLock, oris_memory_pressured. So the hook-spawned indexer this issue is about gets no thread cap, no lock, no memory-pressure check — and, by that first comment, deliberately the multiprocess embedding path, i.e. the heavier one.Two different failure modes, easy to conflate because both end in an unusable machine:
cce servecce index, from the git hookscce initinstallsworktree add, commit, mergeThe comment "bulk
cce indexrun from a separate shell" is the crux: it assumes a human typing in a terminal. The hooks make it fire automatically and unattended, and because worktrees share.git/hooksthrough the common dir, one hook install fans out across every worktree.ProjectIndexLockalso wouldn't help if it were simply extended tocce index, because it keys onstorage_baseand every worktree resolves to its own slug — N worktrees means N locks that never contend. That's the part that makes this orthogonal to #142 rather than a subset of it.For anyone hitting this before an upstream fix, a wrapper holding an
fcntlsemaphore across all projects (not per-project), plusnice, plus skipping ephemeral worktree paths, took the same workload from 36 concurrent indexers to a cap of 2. macOS has noflock(1), so the lock has to come fromfcntlin Python or similar.rajkumarsakthivel commentedon Aug 14, 2026
Good analysis throughout, especially the distinction from #139/#142 and explaining exactly why the per-project lock doesn't help here.
You're right that the resource governor doesn't cover this path. The
ProjectIndexLockis keyed per storage slug, and every worktree resolves to its own slug, so N worktrees means N locks that never contend. The hook script itself had no concurrency awareness at all.Fix is up in #160. Three changes:
Global machine-wide lock via
mkdir(POSIX atomic, noflockdependency, works on macOS). One hook-triggered indexer at a time across all projects and worktrees. Stale locks are reclaimed when the owner PID is dead.Skip ephemeral worktree paths (
/tmp/*,.claude/worktrees/*). Indexing throwaway trees is pure cost.nice -n 10on the spawned process.Also fixed the install side:
install_hooks()now resolves throughgit rev-parse --git-common-dirso it works from inside a worktree, andcce initreplaces old hook blocks instead of silently skipping when the marker is already present (so existing installs pick up the cap on upgrade).Re: your point about
cce initoverwriting custom wrappers. The new_install_single_hookonly replaces content between the CCE markers. User lines outside the block are preserved.fix: global concurrency cap for hook-triggered indexers (#160)