tts validation pack: peak_memory_bound fails with no memory growth when the RSS sample after generate is low

[leehack]

peak_memory_bound, added in #594 for #325, divides the largest resident set sampled after generate by the single sample taken right after generate, against a 1.10 budget. On a loaded macOS host the tts pack on CPU failed it 3 times in 68 runs with no sign of memory growth. The failing runs had baselines of 3.71–5.03 GB, where the normal baseline is 5.69–5.74 GB. Each of their peaks was lower than the peak of every run with a normal baseline.

Measured

Apple M4 Max, 64 GB, macOS arm64, Flutter 3.47.1, --backend cpu. The locked Qwen3-TTS-12Hz-1.7B-Base Q4_K_M and mmproj Q8_0 (revision ca27d74b) were checksum-verified by the pack. The sample is all 68 such runs in this machine's result files from 2026-09-22 and 2026-09-23, on main, PR branches and local native builds. Mutation, revert and instrumented runs are excluded:

TTS / CPU runs	n	baseline (after generate)	peak	growth	peak_memory_bound FAIL
normal baseline	62	5.686–5.743 GB	5.690–5.748 GB	1.0000–1.0057	0
low baseline	6	3.713–5.170 GB	5.412–5.595 GB	1.0494–1.4874	3

The failures, all on 2026-09-23:

UTC	tree	1-min loadavg at start	after load	baseline	peak	growth
15:44	local commit on 0f88c09f6, validation-speech-tts scenario	17.46	3.272	5.034	5.595	1.1115
15:46	main 0f88c09f6, validation-speech-tts scenario	33.58	3.048	5.002	5.595	1.1186
19:24	#630 head e2e9fe957	9.32	3.263	3.713	5.523	1.4874

The first two also failed cancel_latency_bound. The third failed only peak_memory_bound. Its samples in GB were load 3.26, generate 3.71, cancel_immediate 3.71, cancel 5.09, after_cancel through after_invalid 5.15, reload 5.50, and 5.51–5.52 for the cleanup cycles and latency bounds. Its generate WAV is byte-identical (sha256 ac7ff037…) to the generate WAV of the 7 other runs checked, 2 of which had a normal baseline. So generate presumably touched the same memory, and about 2 GB of it would then not have been resident when the baseline was sampled. Swap, first recorded 1.6 minutes after that run ended, was 11.1 of 12 GB used. The two runs after it passed, with baselines of 5.05 and 5.17 GB. packages/llamadart_validation is identical at all three trees and at main 8f6651f35.

Pressure can also hide growth. In one passing e2e9fe957 run (swap 10.9 of 12 GB used), the samples fell from 5.41 GB after the second cleanup cycle to 4.93 GB after the third.

Likely cause (hypothesis)

• ProcessInfo.currentRss on macOS returns mach_task_basic_info.resident_size (Dart process_macos.cc#L1005-L1015), which counts only the pages resident at the moment of the call.
• The pack loads the model with load_mode = mmap. ModelParams.useMmap defaults to true (model_params.dart#L328, load_param_helpers.dart#L84-L90), and the pack does not override it (speech_runner.dart#L139-L146). The run log reports a CPU_Mapped buffer of 970.55 MiB.
• A probe (below) mapped 64 MiB of the model file and read every page. resident_size rose by 64 MiB, while phys_footprint stayed at 1.0 MiB. msync(MS_INVALIDATE) dropped those clean pages, and resident_size fell by 64 MiB with nothing freed. Reading the pages again restored them.
• The 3.71 GB baseline is about 2.0 GB below the normal 5.69–5.74 GB, which is more than the whole 1.04 GB model file. Evicting the model's clean file pages alone cannot explain that gap, so anonymous pages (KV, compute or repack buffers) were presumably compressed or swapped out as well.

Scope

• stt: no failures in 108 macOS runs (CPU and Metal) or in 4 Linux x64 CPU runs. Growth was at most 1.0309.
• tts / Metal: 2 of 63 runs had a low baseline (3.696–3.753 GB against 4.011–4.136 GB), with growth up to 1.0408. None failed.
• LiteRT ASR shows a different pattern. Its 5 failures (1.2009–1.2654x, reported in #594) have a steady 0.47–0.48 GB baseline, and RSS never falls during a run, for example 537, 552 and 566 MiB after the three cleanup cycles.

Code at 8f6651f35

• speech_runner.dart#L512-L517: one residentBytes() sample after each check.
• #L654-L681: the baseline is the single generate sample, and the peak is the maximum of the samples after it.
• #L442-L449: speechPeakRssGrowthBudget = 1.10. #594 measured at most 1.0266x on the GGUF packs.
• process_memory_io.dart#L11-L18: ProcessInfo.currentRss.

Fix direction

• Sample a metric that clean-page eviction does not shrink. On macOS, TASK_VM_INFO phys_footprint stayed at 1.0 MiB in the probe output below while clean file pages entered and left residency. Whether it holds steady when anonymous pages are compressed or swapped is untested. On Linux the candidates are RssAnon and VmSwap from /proc/self/status. dart:io exposes only resident-set values (currentRss, maxRss), so this needs FFI.
• Or record evidence that the process lost residency (for example TASK_VM_INFO compressed) with each sample, and report SKIP with a reason instead of FAIL when that evidence covers the baseline.
• Raising the budget is not a fix: it would have to exceed 1.49x to pass these runs.

There is no on-demand repro, and no memory pressure was induced to make one. The pack command:

dart run tool/testing/validation.dart speech --pack tts --backend cpu \
  --out <new dir> --model <Q4_K_M.gguf> --projector <mmproj-Q8_0.gguf>

Probe (cc -O1 probe.c && ./a.out <Qwen3-TTS Q4_K_M.gguf>)

#include <fcntl.h>
#include <mach/mach.h>
#include <stdio.h>
#include <sys/mman.h>
#include <unistd.h>
static void show(const char *tag) {
  task_vm_info_data_t v; mach_msg_type_number_t c = TASK_VM_INFO_COUNT;
  task_info(mach_task_self(), TASK_VM_INFO, (task_info_t)&v, &c);
  mach_task_basic_info_data_t b; mach_msg_type_number_t bc = MACH_TASK_BASIC_INFO_COUNT;
  task_info(mach_task_self(), MACH_TASK_BASIC_INFO, (task_info_t)&b, &bc);
  printf("%-26s resident_size=%6.1f MiB external=%5.1f internal=%4.1f compressed=%4.1f phys_footprint=%4.1f\n", tag,
         b.resident_size / 1048576.0, v.external / 1048576.0, v.internal / 1048576.0, v.compressed / 1048576.0, v.phys_footprint / 1048576.0);
}
int main(int argc, char **argv) {
  size_t len = (size_t)64 << 20;
  int fd = open(argv[1], O_RDONLY);
  show("start");
  volatile unsigned char *p = mmap(0, len, PROT_READ, MAP_SHARED, fd, 0);
  show("mapped 64 MiB, untouched");
  unsigned long s = 0;
  for (size_t i = 0; i < len; i += 4096) s += p[i];
  show("read every page");
  msync((void *)p, len, MS_INVALIDATE);
  show("after MS_INVALIDATE");
  for (size_t i = 0; i < len; i += 4096) s += p[i];
  show("read every page again");
  return (int)(s & 1);
}

start                      resident_size=   1.4 MiB external=  0.6 internal= 0.8 compressed= 0.0 phys_footprint= 1.0
mapped 64 MiB, untouched   resident_size=   1.4 MiB external=  0.6 internal= 0.8 compressed= 0.0 phys_footprint= 1.0
read every page            resident_size=  65.4 MiB external= 64.6 internal= 0.8 compressed= 0.0 phys_footprint= 1.0
after MS_INVALIDATE        resident_size=   1.4 MiB external=  0.6 internal= 0.8 compressed= 0.0 phys_footprint= 1.0
read every page again      resident_size=  65.4 MiB external= 64.6 internal= 0.8 compressed= 0.0 phys_footprint= 1.0