Skip to content

--quality-delta grows to 13–22 GB RSS on a large repository and freezes hosts without swap #387

Open
Open
--quality-delta grows to 13–22 GB RSS on a large repository and freezes hosts without swap#387

Description

@tonibergholm-codento
tonibergholm-codento
opened on Oct 12, 2026

Summary

ripwire . --quality-delta=<base>..<head> on a large private repository grows to 12.5–22 GB RSS and keeps growing until something outside ripwire stops it. ripwire --mcp on the same repository reached 11.4 GB. On hosts without swap, this left the machine unresponsive for 5–30 minutes before the process ended.

Environment

  • ripwire 0.6.2 (linux/amd64, in a Debian 12 container)
  • Linux 6.1, no swap; hosts with 16 GB and 32 GB RAM
  • The repository is a private enterprise codebase, so I can't share it. I can run diagnostic builds or commands against it and report the results.
  • Invoked by a coding agent following a review prompt: ripwire . --quality-delta=<base>..<head>, run from the repository root of a git worktree.

Observed

Host RAM Command Peak RSS Outcome
16 GB --quality-delta (one range, run three times) 12.5–13.2 GB Each run thrashed the page cache for 5–10 min, then was killed externally
16 GB --quality-delta, four runs on another checkout 12.8–13.7 GB Same
16 GB --mcp 11.4 GB Still growing when sampled
32 GB --quality-delta, three concurrent runs about 22 GB each Host froze until reset

RSS grows steadily over several minutes; it does not spike at startup. The kernel never OOM-killed ripwire. With no swap, it evicted file-backed pages instead, and the host stopped responding.

Expected

Memory bounded by the size of the diff, or a documented cap (a flag or env var) that makes ripwire fail fast with a clear message instead of growing without limit.

Notes

  • We now run ripwire under a 2 GiB cgroup limit, so on this repository the quality delta fails rather than finishing. We'd rather it finish.
  • I have not yet retested on 0.6.7. Its release notes mention a peak-memory fix for --whereis but not for --quality-delta. I'll retest and add the result here.
  • --mcp and --help-task use unbounded memory on a large non-git root (e.g. the home directory) #350 looks related (unbounded --mcp growth), but this root is a git repository, so .gitignore applies.

Activity

  1. joyful-ii-V-I commented on Oct 12, 2026

    @joyful-ii-V-I
    joyful-ii-V-I
    on Oct 12, 2026
    Collaborator

    Thank you for such a precise report. The RSS table, the no-swap detail and the cgroup workaround tell us a lot.

    0.6.2 is a few releases behind; 0.6.7 is current. One change there is relevant: 0.6.7 is the first release with a memory guard (#350). Every ingest is now watched, with a limit of 65% of physical RAM or of the cgroup v2 memory.max on ripwire's own cgroup, whichever is lower. Past that limit, ripwire stops with exit 5 and one stderr line instead of growing. You can set the limit with --max-memory=N[K|M|G] or RIPWIRE_MAX_MEMORY (the flag wins). Two caveats: a limit set on a parent slice, or cgroup v1, is not read, so set --max-memory explicitly there; and in 0.6.7 the guard only covers the parse phases, not the analysis that follows, so treat it as a brake, not a hard ceiling.

    What we found in the code: --quality-delta=A..B extracts and fully indexes both commits, and it also indexes the working tree, which the range form never uses. All three stay in memory together, so memory follows the size of the repository rather than the size of the diff. We plan to look at dropping the unused working-tree index and releasing the base tree before the head tree is loaded.

    If you can retest on 0.6.7, these would help most (none of them reveal code):

    1. ripwire --version, and the peak RSS and exit code of the same --quality-delta=<base>..<head> run, with and without --max-memory=6G.
    2. The repository's size as ripwire sees it: ripwire . --legend=compact | head -c 2000 (the header has files= and symbols=), plus git diff --shortstat <base>..<head> to show how wide the range is.
    3. The opening <quality-delta …> tag from a small range (e.g. HEAD~1..HEAD) and the per-kind counts: ripwire . --quality-delta=HEAD~1..HEAD | grep -o 'kind="[^"]*"' | sort | uniq -c. That tells us whether a large repo or a large number of findings drives the growth.
    4. Whether TMPDIR (or XDG_CACHE_HOME) is on tmpfs. Both commits are extracted there, so on tmpfs those files also take RAM.

    There is no flag to run the kinds one at a time.

    If it still grows on 0.6.7, we can send a diagnostic build that logs peak memory per phase (each ingest, the graphs, each analysis pass). You offered to run diagnostics, and that would pinpoint the cause without the code leaving your machines.

    What we don't know yet: how 0.6.7 behaves on Linux with no swap on a repository of your size. Our largest local test has about 10k files, and we have not tested inside a cgroup. We don't know whether the indexes or the analysis passes dominate on your repo either. We'll post what we learn here.

  2. tonibergholm-codento commented on Oct 12, 2026

    @tonibergholm-codento
    tonibergholm-codento
    on Oct 12, 2026
    ContributorAuthor

    Thanks, that's very helpful. The three-index explanation matches what we see: memory grows with repository size, not with the range.

    Answers we can give now:

    • 4. TMPDIR / XDG_CACHE_HOME: neither is on tmpfs. ripwire runs in a container where TMPDIR is unset, so /tmp is on the container's overlay filesystem on the VM boot disk. XDG_CACHE_HOME is on a separate persistent disk. The extracted commits don't take RAM, but they do add write IO to the boot disk.
    • cgroup layout: ripwire runs directly in its own cgroup v2 leaf with memory.max = 2 GiB, memory.oom.group=1 and memory.swap.max=0. The limit is on ripwire's own cgroup, not a parent slice.

    We're moving from 0.6.2 to 0.6.7 and will treat exit 5 as "memory limit". We also set RIPWIRE_MAX_MEMORY explicitly to 90% of the leaf's memory.max, so parsing doesn't stop at half of the default 65%. After that we'll rerun the same range on one of the affected hosts and post items 1–3:

    • version, peak RSS and exit code, with and without --max-memory=6G;
    • the files=/symbols= header;
    • --shortstat of the range;
    • the per-kind counts on HEAD~1..HEAD.

    If 0.6.7 still grows, we'd gladly run the per-phase diagnostic build. A linux-x64 binary works for us.

  3. tonibergholm-codento commented on Oct 12, 2026

    @tonibergholm-codento
    tonibergholm-codento
    on Oct 12, 2026
    ContributorAuthor

    Retest results on 0.6.7: the official ripwire-0.6.7-linux-x64.tar.gz, sha256 verified, ripwire 0.6.7 (Release, GNU 14.2.1, emit=std::print, built_from=unknown).

    Setup. An idle 32 GB VM: Debian 12, kernel 6.1, cgroup v2, no swap. ripwire ran in a transient systemd scope with MemoryMax=8G and MemorySwapMax=0, which is ripwire's own cgroup, as an unprivileged user. HOME, XDG_CACHE_HOME and TMPDIR pointed at fresh empty directories on disk, so every run started from a cold cache. The working tree was a git clone --shared checked out at the head commit.

    1. Peak memory and exit code

    Run Range Result
    default the original range (10 files changed, 1196 insertions, 4 deletions) killed by the kernel OOM killer at the 8 GiB scope limit after 235 s, exit 137, no stdout
    --max-memory=6G same same: OOM at 8 GiB after 216 s, exit 137
    default HEAD~1..HEAD same: OOM at 8 GiB after 198 s, exit 137
    • The memory guard never fired in any run. There was no exit 5 and no guard line on stderr, either at the default limit (65% of 8 GiB) or at --max-memory=6G.
    • The width of the range makes no difference. A one-commit range dies the same way.
    • During the --max-memory=6G run, one sample of the scope's memory.stat showed about 4.2 GB anon and 4.2 GB file. The extracted commits under TMPDIR are charged to the cgroup as page cache. I don't know whether the guard counts only RSS.

    2. Repository size (--legend=compact header): files=20356 symbols=595752 edges=2089193. git tracks about 33.5k files. A plain map (--legend=compact) finishes under the same 8 GiB limit.

    3. Small range: the run was killed before printing anything, so I have no opening tag or per-kind counts.

    So on this repository, any --quality-delta needs more than 8 GiB however small the range, and the 0.6.7 guard does not stop it. We'd be glad to run the per-phase diagnostic build (linux-x64) under the same setup.

  4. tonibergholm-codento commented on Oct 13, 2026

    @tonibergholm-codento
    tonibergholm-codento
    on Oct 13, 2026
    ContributorAuthor

    We read the v0.6.7 source to see why the guard never fired and where the range form spends memory. Here is what we found, plus an offer to send a PR.

    Why the guard did not stop it

    • On Linux, os::mem_footprint reads resident pages from /proc/self/statm (src/infra/os.h:583-592). The cgroup also charges the extracted trees as page cache, which the guard cannot see. That matches the ~4.2 GB file we saw.
    • The hard line runs at the three points in main.cpp before the verb dispatches (:5253, :5279, :5354). Inside --quality-delta=A..B, loadRefTree ingests and builds a graph twice (src/quality.h:4335-4366) with only the time-gated soft watch, and does not act on memoryStop.

    Where the memory goes in the range form

    • The working tree is ingested and graphed before dispatch. Range mode reads it only for diagnoseRegisterMacroConfig (src/verbs_quality.h:1516).
    • RefPairDelta keeps both RefTrees (rich ingest plus graph) until the run ends (src/verbs_quality.h:72-87). The base is used only to compute its snapshot (:220).
    • The ingest cache is used only when the sha is HEAD (src/quality.h:4347), so the base always parses cold.

    What we would propose, as one PR or three:

    1. Range mode dispatches before the working-tree ingest and passes the root to the config diagnostic.
    2. Compute the base snapshot inside loadRefPairDelta, release the base tree and its temp dir, then load the target. sameRef keeps its current path.
    3. When the cgroup's memory.current is readable, the guard reads it too, and the hard line is checked after each loadRefTree and after the snapshot, so a range run exits 5 instead of being OOM-killed.

    Have you already started on 1 or 2? If not, we can open a PR against main with tests in the existing gates (qdrefpaircheck.sh, memguardcheck.sh), in whatever shape you prefer. We are still glad to run the diagnostic build.

  5. tonibergholm-codento commented on Oct 13, 2026

    @tonibergholm-codento
    tonibergholm-codento
    on Oct 13, 2026
    ContributorAuthor

    Measured peaks on the same repository, ripwire 0.6.7 (official linux-x64), 2026-10-13. Each run in its own cgroup v2 scope with MemoryMax=20G, no swap, cold cache, RIPWIRE_MAX_MEMORY=18G, fresh shared clone, 8 vCPU. Header: files=20349 symbols=595607 edges=2088997.

    Run Exit Wall cgroup memory.peak max anon max RSS
    ripwire . --legend=compact 0 41–46 s 2.8 GiB 2.5 GiB 2.6 GiB
    ripwire --mcp, one analyze {"legend":"compact"} call 0 133 s 14.2 GiB 12.5 GiB 12.5 GiB
    ripwire . --quality-delta=HEAD~1..HEAD (4 files, +83 −61) 5 380 s 20.0 GiB (at the limit) 17.3 GiB 17.5 GiB
    • With RIPWIRE_MAX_MEMORY set below the scope, the guard fired on the delta (exit 5, "the memory guard stopped an ingest this answer depends on"). The cgroup sat at its limit with heavy reclaim (memory.events max=67264) and 7.5 GiB of page cache before that.
    • The MCP server's first index build takes about 5× the plain map. We had expected it to be close to the map. Is the difference the rich ingest (value uses), the HEAD snapshot prefetch thread running at the same time, or both? If it is the prefetch, a way to turn it off would let us run --mcp in a much smaller limit.
    • tools/list alone stays at 7 MB, so the index is built on the first tool call.

  6. tonibergholm-codento commented on Oct 13, 2026

    @tonibergholm-codento
    tonibergholm-codento
    on Oct 13, 2026
    ContributorAuthor

    Opened #391 (range form holds one commit index at a time; hard line inside it) and #392 (guard counts the cgroup leaf charge the kernel cannot reclaim). They apply in either order. On golang/go, HEAD~1..HEAD peak RSS goes from 7.08 GB to 4.67 GB, and findings are byte-identical to 0.6.7. --mcp is not addressed: its first index build peaked at 14.2 GiB here, against 2.8 GiB for the plain map, so our question above about the prefetch thread still stands.

  7. joyful-ii-V-I commented on Oct 13, 2026

    @joyful-ii-V-I
    joyful-ii-V-I
    on Oct 13, 2026
    Collaborator

    @tonibergholm-codento thank you, both for the measurements on 0.6.7 and for following up with fixes. #391 and #392 are exactly the two problems we found when reading the code: the range form holding several whole-tree indexes, and a guard that couldn't see the cgroup's charge. We're reviewing both today, and they're in CI now.

    On your --mcp question (the first index build at about 5× the plain map): we'll look at whether the HEAD snapshot prefetch thread accounts for it, and answer here.

  8. tonibergholm-codento
    added a commit that references this issue on Oct 13, 2026

    refactor(quality-delta): score the base and load the target in their …

    d2ae363
  9. joyful-ii-V-I
    added 4 commits that reference this issue on Oct 14, 2026

    fix(quality-delta): the range form indexed the working tree it never …

    6e40c1f

    fix(quality-delta): the base commit's index outlived the only read of…

    c1e453e

    fix(memguard): the hard line never ran inside --quality-delta=A..B (#387

    7b221ee

    Merge pull request #391 from tonibergholm-codento/qd-range-memory

    36753d6

  10. joyful-ii-V-I commented on Oct 14, 2026

    @joyful-ii-V-I
    joyful-ii-V-I
    on Oct 14, 2026
    Collaborator

    @tonibergholm-codento thank you for an exact question with numbers behind it. Here is what we measured.

    Short answer: it isn't the prefetch thread. On the first tool call the MCP server builds its index with a "rich" ingest: it records value-use sites (reads and writes) plus extra subtoken data for the uses/for/exemplar/quality tools, and writes a larger cache file. The plain map uses the lean ingest. The HEAD snapshot prefetch never runs on the first call. The first time it sees HEAD it only records it. It starts only when HEAD moves while the server is running (a commit, checkout or reset).

    Measured on macOS (16 GB host, origin/main, cold cache, 2 runs each), with peak memory footprint and one analyze call:

    repo map --mcp first call --mcp, prefetch off --mcp with the index built lean (local test patch)
    textual (Python, 1.3k files, 16k symbols) 0.10 GB 0.28 GB 0.28 GB 0.10 GB
    envoy (C++, 10k files, 167k symbols) 1.72 GB 2.77 GB 2.83 GB 1.54 GB

    Building the MCP index lean brings it back to the map's peak, and forcing the map to rich brings the map up to MCP's peak. Turning the prefetch off changes nothing on the first call. The peak is a transient inside the ingest (parsing plus the cache write), not the graph or the ranking.

    Where the prefetch does cost memory: after a commit during a session, it ingests the HEAD tree a second time while the index is still resident. On envoy that took the peak from about 2.5 GB to 4.1–4.9 GB. Today you can turn it off with RIPWIRE_QSNAP_PREFETCH_MIN_FILES=1000000000. It's a test knob, but it's safe: quality_delta then computes the HEAD snapshot when you call it.

    What we don't know yet: we see 1.6× (C++) to 2.8× (Python), not 5×. The mechanism is the same, but the size of the rich overhead depends on the language and the repo, and we haven't measured Linux/glibc at your size (595k symbols). If you can share the language mix, or run a diagnostic build that prints memory per phase, that would close the gap.

    On --max-memory for this path: the limit applies to the first build's parse, but it's checked only every few seconds. A 1G cap gave a partial answer (flagged _memory_stop) but still peaked at about 2 GB. The per-call hard check runs before the build, not inside it.

    What we plan to change:

    1. Make the prefetch aware of memory headroom (skip it when a second tree would not fit), and give it a documented off switch.
    2. Build the resident MCP index lean and add the value-use data only when a tool that needs it is called.
    3. Put the hard memory check inside the first index build (after the ingest and after the graph), so a cap is respected rather than overshot.
    4. Look at writing the rich cache file without the extra transient memory.

    Thanks again. The cgroup numbers and the tools/list detail made this quick to narrow down.

  11. joyful-ii-V-I
    added 4 commits that reference this issue on Oct 14, 2026

    fix(memguard): on Linux the guard read only the resident set the cgro…

    8925096

    regen: COMMANDS.md and the help_all golden for --max-memory's cgroup …

    3aa3df8

    Merge pull request #392 from tonibergholm-codento/memguard-cgroup-charge

    b274e0c

    feat(quality): --quality-delta=A..B answers the kinds it can under me…

    a466e66
  12. mpapis
    added a commit that references this issue on Oct 17, 2026

    test: qddegradecheck — --quality-delta=A..B answers under memory pres…

    3413622
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions