You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
{{ message }}
Repository navigation
Health reports critical from heapUsed/heapTotal, so /agentmemory/health returns 503 on a healthy process #1223
evaluateHealth measures memory severity against heapTotal, which is the heap V8 has already committed, not the ceiling it is allowed to grow to. V8 sizes heapTotal to demand, so a busy healthy process sits near 100% of it more or less permanently.
Since heapTotal <= heap_size_limit always holds, this can only ever over-report severity, never under-report it.
What I saw
On a live deployment:
bytes
heapUsed
390,771,624
heapTotal
404,930,560
rss
584,327,168
v8.getHeapStatistics().heap_size_limit
6,492,782,592 (6192 MB)
That is 6.0% of the heap the process may actually use. The alert read memory_critical_97%_rss557mb and status was critical.
GET /agentmemory/health then returned HTTP 503, because src/triggers/api.ts:274 maps critical to 503:
conststatusCode=status==="critical" ? 503 : 200;
/agentmemory/livez returned 200 at the same moment. So an uptime monitor pointed at /health reads the service as down while it is serving normally. At that point the container was using 3.0 GB of an 8 GB limit, the circuit breaker was closed, CPU was 3.4%, and event loop lag was 0.1 ms.
Worth noting for anyone deploying from this repo: deploy/railway/railway.json sets healthcheckPath to /agentmemory/livez, so Railway's own check keeps passing and the service never restart-loops. The shipped template dodges this by accident. It is the richer /agentmemory/health endpoint, the one you would point an external monitor at, that reports the false 503.
What I expected
healthy, and HTTP 200. Six percent of the heap limit is not memory pressure.
#158 reported the same formula misfiring at heapUsed ~45 MB, heapTotal ~46 MB, and rss ~120 MB. The fix that landed added memoryRssFloorBytes (512 MB, src/health/thresholds.ts:20) so the alert stays quiet on small processes. That covers the case as reported, but the ratio itself was never corrected, so the bug comes back for any process whose RSS clears the floor.
There is also no way to tune around it from the outside. src/health/monitor.ts:87 calls evaluateHealth(snapshot) with no config argument, so DEFAULTS always wins and neither the percentages nor the RSS floor are reachable from the environment.
Repro
Run agentmemory under a workload that keeps RSS above 512 MB.
curl -si "$AGENTMEMORY_URL/agentmemory/health" | head -1
You get 503, with memory_critical_NN% in health.alerts, while the process is using a small fraction of v8.getHeapStatistics().heap_size_limit.
The viewer shows the same thing. src/viewer/index.html:1582 recomputes the identical heapUsed / heapTotal ratio client-side, so the HEAP gauge renders red on a healthy process.
Environment
agentmemory 0.9.28 (latest published: 0.9.29)
Node v22.23.2
Platform: Railway, self-hosted from this repo's own deploy/railway template
Linux container, 8 GB memory limit, single replica
Reproduced against the deployed service, not a local dev server
Three things in this comment, from a systemd host on 0.9.29:
Confirmation. The health endpoint reported this process as critically short of memory and answered 503 while it was fine, and reported the previous, smaller process the same way. Fifteen minutes of samples below show the ratio moving with garbage collection while real memory use was flat.
A gap none of the PRs touch. Status is asserted from a single 30-second sample, so it flips between 200 and 503 with nothing changing. Any monitor or readiness probe that trusts the status code pages or restarts the service on a healthy process. There is also no configuration path on 0.9.29, so an operator cannot raise the threshold to work around it.
The rest of this comment is the evidence: the samples beside the cgroup's own counters, why the ratio is wrong in both directions, and the four asks in order of preference.
Still present on 0.9.29, and one 15-minute window on a systemd host caught both halves of it: the ratio moving between 87% and 96% with RSS flat, and the flip from 200 to 503 on the last sample.
The host is a systemd user unit under cgroup v2 running @agentmemory/agentmemory 0.9.29 on Node v22.23.2, Linux x64, as node /usr/bin/agentmemory with no NODE_OPTIONS, so v8.getHeapStatistics().heap_size_limit is 2,197,815,296 (2,096 MB). Process up 6 h at the first sample, RSS 1.06 GB, a 22,000-entry index loaded. All line numbers are dist/index.mjs of the published 0.9.29 tarball.
Thirty samples 30 s apart of the GET /agentmemory/health body, the HTTP code from a separate curl -o /dev/null -w '%{http_code}', and the unit's cgroup read beside each sample (memory.current, cumulative memory.eventshigh, memory.pressuresome total, memory.swap.current):
Every other field in those bodies was green throughout: kvConnectivity ok at 1.8 to 2.0 ms, event loop lag 0.07 to 0.17 ms, circuit breaker closed. The window shows:
The ratio tracks heapTotal. RSS moved from 1,062 MB to 1,084 MB in 15 minutes. heapTotal moved 321, 311, 379, 348, 316, 309, 277 MB as V8 grew and released committed space, and the ratio moved with it, 93, 87, 92, 87, 94, 89, 96.5. Between the last two samples heapUsed fell by 10 MB and heapTotal by 32 MB. The process used less memory, and the status went from degraded to critical. Against the limit it may actually grow into, that last sample is 268 MB of 2,096 MB, 12.2%.
One sample decides: collectHealth samples every 30 s (15719-15721), evaluateHealth(snapshot) (15711) sees one snapshot, and api::health (16226) maps whatever record is latest to status === "critical" ? 503 : 200 (16232). Sample 29 was 200 at 90%, sample 30 was 503 at 96.5%, and the next collection will put it back. There is no hold in either direction. None of fix: correct memory health and supervise engine failures #1224, fix(health): measure heap against heap_size_limit, not heapTotal #1285 or fix(health): measure heap saturation against the heap ceiling, not heapTotal #1317 add one. With heap_size_limit as the denominator the ratio moves slowly enough that this matters less, but a status external systems act on should still have to hold across more than one sample before it becomes critical, and hold below the threshold for the same window before it clears.
The previous instance sat on the other side of the threshold. The process this one replaced, measured 2026-09-13 on the same host, eight samples over 40 s: heapUsed 196 MB, heapTotal 206 MB, 95.5% flat, status: critical, HTTP 503 on every sample, with kvConnectivity ok at 0.53 ms, lag 0.047 ms, breaker closed, and 24 observations captured in the preceding two minutes. The smaller heap was critical and the larger one is degraded. The collector inverts the two.
The field is wrong in both directions. The service's cgroup during the same window: memory.current 4.04 to 4.06 GB against memory.high 4 GiB and memory.max 5 GiB, memory.eventshigh 63,893 cumulative, memory.swap.current 994 MB. Two processes live in it, the node worker at 1.06 GB RSS and the iii engine at 2.7 GB RSS with 902 MB swapped out. The high counter and PSI did not advance during the window, so the cgroup was not being actively throttled, but it has spent its life within 6% of the limit that throttles it, with 63,893 throttling events behind it and a gigabyte in swap, none of which appears in the health body. The memory field reports a process at 12% of its heap limit as critical, and the limit this host enforces is invisible to it. #1285 names cgroup awareness as out of scope for the denominator fix.
#1019 shows the same alert on a process that was in trouble: its health body shows memory_critical_98%_rss744mb next to 6-million-millisecond function latencies and 669 active invocations. The alert here has the same shape on a process with 0.07 ms lag.
There is no configuration path on 0.9.29. memoryWarnPercent, memoryCriticalPercent and memoryRssFloorBytes appear in DEFAULTS (15585-15593) and in evaluateHealth and nowhere else in the bundle. Nothing reads them from the environment, and the only caller passes no config (15711, inside registerHealthMonitor(sdk, kv) at 15643, invoked at 22845). So "raise the threshold" is not available to an operator until #1237 or #226 lands.
This host's own poller survives the 503 by accident: it reads the body with curl -s and never inspects the status code, so it alerts only when the body fails to parse. Written the obvious way, checking %{http_code}, it would have paged on every run since the process first crossed the 512 MB memoryRssFloorBytes. Any check that trusts the code, a readiness probe for one, does that, and under a restart loop the process comes back, loads its index, crosses the floor, and reports the same ratio again.
The ask, in order:
Measure against a limit the process will hit. cgroup v2 memory.current against memory.high or memory.max when /sys/fs/cgroup is present, because that is the limit under systemd and under every container runtime, and it is the only signal that sees sibling processes in the same cgroup. fix(health): measure heap against heap_size_limit, not heapTotal #1285's own caveat, that V8 sizes heap_size_limit from host memory, ignoring the cgroup, unless --max-old-space-size is set, is another reason to read the cgroup first.
Failing that, RSS against an explicit budget.
heapUsed / heap_size_limit only if the intent is specifically to catch V8 heap exhaustion, which the three open PRs do. heapUsed / heapTotal should not be under any threshold.
Whatever the signal, require it to hold across more than one 30 s sample before critical is asserted, and to hold clear for the same window before it drops.
#1372 (the MAX_STRING_LENGTH ceiling in VectorIndex.serialize()) and #1373 (resolveDimensions' 1536 fallback) each ask collectHealth (15650) to report vector size, BM25 size and last successful save per stream. Neither mentions that the memory field the function already reports is wrong. Whoever adds the index fields should correct this one in the same pass.
Patch against v0.9.29 for the points above, with tests:
Activity
fix(viewer): compute the heap percentage from raw bytes
test(viewer): cover the heap gauge ceiling and precision
test(viewer): note why the gauge test asserts on source
fix(viewer): pick the gauge colour on the unrounded percentage
refactor(types): drop the redundant heapSizeLimit comment
possiblyneal commentedon Sep 14, 2026
Three things in this comment, from a systemd host on 0.9.29:
heap_size_limit. That removes the garbage-collection noise, and it is the weakest of the options listed at the end: it still ignores the cgroup limit, which is the one the process will actually hit and the only one that sees the sibling engine process in the same unit.The rest of this comment is the evidence: the samples beside the cgroup's own counters, why the ratio is wrong in both directions, and the four asks in order of preference.
Still present on 0.9.29, and one 15-minute window on a systemd host caught both halves of it: the ratio moving between 87% and 96% with RSS flat, and the flip from 200 to 503 on the last sample.
The host is a systemd user unit under cgroup v2 running
@agentmemory/agentmemory0.9.29 on Node v22.23.2, Linux x64, asnode /usr/bin/agentmemorywith noNODE_OPTIONS, sov8.getHeapStatistics().heap_size_limitis 2,197,815,296 (2,096 MB). Process up 6 h at the first sample, RSS 1.06 GB, a 22,000-entry index loaded. All line numbers aredist/index.mjsof the published 0.9.29 tarball.Thirty samples 30 s apart of the
GET /agentmemory/healthbody, the HTTP code from a separatecurl -o /dev/null -w '%{http_code}', and the unit's cgroup read beside each sample (memory.current, cumulativememory.eventshigh,memory.pressuresome total,memory.swap.current):Every other field in those bodies was green throughout:
kvConnectivityok at 1.8 to 2.0 ms, event loop lag 0.07 to 0.17 ms, circuit breaker closed. The window shows:heapTotal. RSS moved from 1,062 MB to 1,084 MB in 15 minutes.heapTotalmoved 321, 311, 379, 348, 316, 309, 277 MB as V8 grew and released committed space, and the ratio moved with it, 93, 87, 92, 87, 94, 89, 96.5. Between the last two samplesheapUsedfell by 10 MB andheapTotalby 32 MB. The process used less memory, and the status went fromdegradedtocritical. Against the limit it may actually grow into, that last sample is 268 MB of 2,096 MB, 12.2%.collectHealthsamples every 30 s (15719-15721),evaluateHealth(snapshot)(15711) sees one snapshot, andapi::health(16226) maps whatever record is latest tostatus === "critical" ? 503 : 200(16232). Sample 29 was 200 at 90%, sample 30 was 503 at 96.5%, and the next collection will put it back. There is no hold in either direction. None of fix: correct memory health and supervise engine failures #1224, fix(health): measure heap against heap_size_limit, not heapTotal #1285 or fix(health): measure heap saturation against the heap ceiling, not heapTotal #1317 add one. Withheap_size_limitas the denominator the ratio moves slowly enough that this matters less, but a status external systems act on should still have to hold across more than one sample before it becomescritical, and hold below the threshold for the same window before it clears.heapUsed196 MB,heapTotal206 MB, 95.5% flat,status: critical, HTTP 503 on every sample, withkvConnectivityok at 0.53 ms, lag 0.047 ms, breaker closed, and 24 observations captured in the preceding two minutes. The smaller heap wascriticaland the larger one isdegraded. The collector inverts the two.The field is wrong in both directions. The service's cgroup during the same window:
memory.current4.04 to 4.06 GB againstmemory.high4 GiB andmemory.max5 GiB,memory.eventshigh63,893 cumulative,memory.swap.current994 MB. Two processes live in it, the node worker at 1.06 GB RSS and theiiiengine at 2.7 GB RSS with 902 MB swapped out. Thehighcounter and PSI did not advance during the window, so the cgroup was not being actively throttled, but it has spent its life within 6% of the limit that throttles it, with 63,893 throttling events behind it and a gigabyte in swap, none of which appears in the health body. The memory field reports a process at 12% of its heap limit ascritical, and the limit this host enforces is invisible to it. #1285 names cgroup awareness as out of scope for the denominator fix.#1019 shows the same alert on a process that was in trouble: its health body shows
memory_critical_98%_rss744mbnext to 6-million-millisecond function latencies and 669 active invocations. The alert here has the same shape on a process with 0.07 ms lag.There is no configuration path on 0.9.29.
memoryWarnPercent,memoryCriticalPercentandmemoryRssFloorBytesappear inDEFAULTS(15585-15593) and inevaluateHealthand nowhere else in the bundle. Nothing reads them from the environment, and the only caller passes no config (15711, insideregisterHealthMonitor(sdk, kv)at 15643, invoked at 22845). So "raise the threshold" is not available to an operator until #1237 or #226 lands.This host's own poller survives the 503 by accident: it reads the body with
curl -sand never inspects the status code, so it alerts only when the body fails to parse. Written the obvious way, checking%{http_code}, it would have paged on every run since the process first crossed the 512 MBmemoryRssFloorBytes. Any check that trusts the code, a readiness probe for one, does that, and under a restart loop the process comes back, loads its index, crosses the floor, and reports the same ratio again.The ask, in order:
memory.currentagainstmemory.highormemory.maxwhen/sys/fs/cgroupis present, because that is the limit under systemd and under every container runtime, and it is the only signal that sees sibling processes in the same cgroup. fix(health): measure heap against heap_size_limit, not heapTotal #1285's own caveat, that V8 sizesheap_size_limitfrom host memory, ignoring the cgroup, unless--max-old-space-sizeis set, is another reason to read the cgroup first.heapUsed / heap_size_limitonly if the intent is specifically to catch V8 heap exhaustion, which the three open PRs do.heapUsed / heapTotalshould not be under any threshold.criticalis asserted, and to hold clear for the same window before it drops.#1372 (the
MAX_STRING_LENGTHceiling inVectorIndex.serialize()) and #1373 (resolveDimensions' 1536 fallback) each askcollectHealth(15650) to report vector size, BM25 size and last successful save per stream. Neither mentions that the memory field the function already reports is wrong. Whoever adds the index fields should correct this one in the same pass.Patch against v0.9.29 for the points above, with tests:
0001-health-measure-memory-pressure-against-cgroup-rss-budget-or-heap-limit-hold-before-critical.patch
inix-x commentedon Sep 15, 2026
Thank you @possiblyneal for this detailed review, Ill take a look into it
fix(health): measure heap against heap_size_limit, not heapTotal (roh…
test(health): cover heapLimit producer wiring (rohitg00#1223)
rohitg00 commentedon Oct 8, 2026
@inix-x fixed in v0.9.30: health measures heap saturation against
heap_size_limit, so the Memory elevated alert no longer fires on a healthy process. https://github.com/rohitg00/agentmemory/releases/tag/v0.9.30