tier2: dead_code is 83% of background scan time, and its cost distribution is bimodal (p50 326ms, p95 19s)

[iceteaSA]

Census from a live daemon's module log — 110 perf ticks, ~2h uptime, several project roots bound by concurrent sessions:

category	runs	total	share	p50	p95	max
dead_code	34	52,586 ms	83%	326 ms	19,087 ms	19,655 ms
duplicates	30	4,863 ms	7%	42 ms	788 ms	2,046 ms
unused_exports	30	2,902 ms	4%	38 ms	239 ms	889 ms
cycles	30	2,656 ms	4%	23 ms	485 ms	768 ms

63 s of background scan time in the window, and dead_code is five sixths of it.

The interesting part isn't the total, it's the shape. p50 is 326 ms and p95 is 19 s — a 58× spread. Sorted run times:

19655  19087  2741  1167  771  637  597  466  ...

Two runs account for 74% of all dead_code time. This isn't a uniformly slow scanner; it's a fast one with a rare catastrophic mode, and that shape usually means a specific input class rather than general cost.

What I could rule out from the log: it isn't watcher volume, and it isn't callgraph churn.

13:11:11    2741ms  watcher_paths= 25  callgraph_invalidations=3
14:18:21   19655ms  watcher_paths= 52  callgraph_invalidations=0
14:21:21   19087ms  watcher_paths=218  callgraph_invalidations=0

The 218-path tick ran faster than the 52-path one, and both 19 s runs had zero callgraph invalidations. Whatever drives the tail, it isn't how much changed.

Where I'd look, and why I'm asking rather than patching:

project_dead_code_snapshot opens the callgraph SQLite read-only and does what looks like a full projection — SELECT path FROM files ORDER BY path, then all nodes, then all refs — per scan. If that's per-root-per-scan with no incremental path, a large root would give exactly this bimodal profile: cheap on small roots, cliff on the big one. I haven't confirmed that's the code path taken in the slow runs, and the log doesn't carry the root, so I can't attribute it from here.

Two things that would make this diagnosable from a log without guessing:

1. Tag tier2 timings with the root (or a stable root id). Right now a 19 s dead_code run is anonymous — you can't tell whether it's one pathological repo or a general regression, which is the first question anyone asks.
2. A per-category soft deadline for tier2. InspectManager::soft_deadline bounds a caller waiting on an outcome, but a background scan appears to run to completion regardless. A 19 s scan on a shared daemon competes with interactive work for the same process; a bounded scan that reports complete: false fits the honest-reporting convention better than an unbounded one that finishes eventually.

Happy to dig further on either if useful, but you'll know immediately whether the projection is the right suspect and whether the tail is expected on a large root.

Method note so the numbers can be discounted appropriately: single host, single daemon instance, ~2 h, mixed real workload from several concurrent sessions. Not a controlled benchmark — a census of what one production instance actually spent.

[ualtinok]

Verified at source, and your instinct about where to look was exact: project_dead_code_snapshot (crates/aft/src/callgraph_store/dead_code_projection.rs) opens the store read-only and projects the full graph — SELECT path FROM files ORDER BY path, all nodes, all refs — on every scan, per root, with no caching between scans. Your own log data is the conviction: both 19 s runs had callgraph_invalidations=0, so they re-projected byte-identical data the previous scan had already projected. The bimodal shape is exactly what you called it — cheap on small stores, cliff on large ones, and the tail is pure duplicate work.

Fix going out now: cache the projected snapshot keyed on the store's generation pointer (the .current file that already serializes atomic swaps — when the generation hasn't moved, the store bytes cannot have changed by construction, so the cached projection is exact, not heuristic). Invalidation is generation movement; no TTL, no staleness window. On your census that converts ~74% of dead_code's total time into a cache hit, and the p95 should collapse toward the p50.

The ORDER BY path and full-refs shape stay as-is — with the cache in front, the projection runs once per graph change instead of once per scan, which is the right cost point. Will note here when it ships.