OOM killer / memory pressure issue on Steam Deck after long uptime and 7z archive compression

[allians00758]

OOM killer / memory pressure issue on Steam Deck after long uptime and 7z archive compression

Environment

Device: Steam Deck OLED
BIOS: F7G0114
Kernel: 6.16.12-valve21-1-charcoal-616-g6e6a05c0d53f-dirty
SteamOS: 3.8.x

Problem description

After long uptime (several hours or more), the system starts experiencing severe memory pressure during creation/compression of large archives using 7z/PeaZip.

The issue does not happen immediately after reboot, but appears only after the system has been running for a long time (games, suspend/resume, Steam UI usage, browser, etc.).

When compressing large solid 7z archives:

• the system becomes extremely slow,
• swap thrashing begins,
• OOM killer activates,
• unrelated processes are killed (often steamwebhelper first),
• sometimes Steam UI becomes unstable.

After reboot the issue temporarily disappears.

Current memory configuration

ZRAM:

• 32 GB zstd
• swap priority 100

Memory state during normal operation:

• plenty of available RAM remains,
• swap usage is low,
• no permanent memory exhaustion.

/tmp is mounted as tmpfs:

tmpfs on /tmp type tmpfs (rw,nosuid,nodev,size=7579844k,...)

Relevant kernel logs

OOM triggered by 7z:

7z invoked oom-killer: gfp_mask=0x140cca(...)
Out of memory: Killed process 3314 (steamwebhelper)

Call trace includes:

• do_swap_page
• handle_mm_fault
• __alloc_pages_slowpath

Large inactive file cache observed:

inactive_file:2579918

Additional observation

The issue does not reproduce on the stock SteamOS kernel.

The problem appears only on:

• 6.16.12-valve21-1-charcoal-616-g6e6a05c0d53f-dirty

Under the same workload on the stock SteamOS kernel:

• OOM killer does not trigger,
• swap thrashing is significantly reduced,
• the system remains stable even after long uptime and large archive compression.

Suspected cause

Possible interaction between:

• tmpfs-backed /tmp,
• aggressive page cache growth,
• shared RAM/VRAM architecture on Steam Deck,
• long uptime memory fragmentation,
• heavy multithreaded 7z compression,
• high memory usage of LZMA2 solid compression,
• changes in memory reclaim / VM subsystem behavior in the current kernel.

The issue appears to be related more to memory reclaim / fragmentation / cache pressure than to actual lack of RAM or swap space.

Expected behavior

System should reclaim cache and memory more gracefully under pressure instead of entering swap storm and triggering OOM killer for unrelated processes.

Additional notes

Using /var/tmp instead of /tmp significantly reduces the issue frequency.

The problem is reproducible mostly after long uptime and not immediately after fresh boot.

[zarpon]

It's probably related to kcompressd patch, under several hours of memory stress the system might stops doing the async swapping. Try setting vm.kcompressd=256, and vm.vfs_cache_pressure=125. Have you noticed any issue on a gaming cenário? Also you might try to use zswap instead of zram, under heavy memory load the system will behave better.

[allians00758]

It's probably related to kcompressd patch, under several hours of memory stress the system might stops doing the async swapping. Try setting vm.kcompressd=256, and vm.vfs_cache_pressure=125. Have you noticed any issue on a gaming cenário? Also you might try to use zswap instead of zram, under heavy memory load the system will behave better.

Thanks, this actually helped a lot!

After applying these settings the system became much more stable under heavy memory pressure during 7z compression, and the OOM issues completely stopped occurring.

I wrote a small script for convenience, maybe it will help someone else in the future:

#!/bin/bash
set -e

CONF="/etc/sysctl.d/99-charcoal-memory.conf"

sudo tee "$CONF" >/dev/null <<'EOC'
vm.kcompressd=256
vm.vfs_cache_pressure=125
vm.page-cluster=0
EOC

sudo sysctl --system >/dev/null

[zarpon]

It's probably related to kcompressd patch, under several hours of memory stress the system might stops doing the async swapping. Try setting vm.kcompressd=256, and vm.vfs_cache_pressure=125. Have you noticed any issue on a gaming cenário? Also you might try to use zswap instead of zram, under heavy memory load the system will behave better.

Thanks, this actually helped a lot!

After applying these settings the system became much more stable under heavy memory pressure during 7z compression, and the OOM issues completely stopped occurring.

I wrote a small script for convenience, maybe it will help someone else in the future:

#!/bin/bash
set -e

CONF="/etc/sysctl.d/99-charcoal-memory.conf"

sudo tee "$CONF" >/dev/null <<'EOC'
vm.kcompressd=256
vm.vfs_cache_pressure=125
vm.page-cluster=0
EOC

sudo sysctl --system >/dev/null

I'm glad it helped