Skip to content

Stress dictionary (555k words) is loaded eagerly at boot though production never uses it: about 140 MB resident, 310 MB peak, on a 1 GiB swapping host #963

Closed

Description

@alex-mextner

What

src/bootstrap/speech.ts loads data/dictionaries/stress-dict.json (555,461 entries, 27 MB) on every start through StressDictionary.loadFromFile: JSON parse, a zod z.record copy, Object.entries, a Map and a sorted key array. Measured 2026-11-04 with the production file (bun 1.4.4): RSS 51 MB before, 362 MB peak, 190 MB after GC, 79 MB of heap retained. Its only consumers are Silero TTS (SILERO_PYTHON_PATH is unset in production), the voice-call TTS path (the MTProto service tier is off in production) and the lookup_stress AI tool. The container (1 GiB limit, 384 MiB memory.low) shares an overcommitted 3.7 GB odroid with 2.4 GB in swap, so this is memory taken from Chromium, redis and the neighbours for nothing. Load the dictionary on first use instead of at boot: the object exists from startup, its contents load on the first lookup, and a start without a lookup never reads the file.

Why (motivation)

On 2026-11-04 the fresh production container showed 410-460 MiB against 138-163 MiB seen on the old 28 h container; the investigation found no leak and no change between the images (identical module-load RSS, identical dependencies), but did find this eager load as the largest single item in the process, on a host whose memory pressure is the open problem of #800.

User impact

People who ask the bot for a calendar picture or a reply wait because the host swaps: this takes 80-140 MB of the bot's memory for a dictionary no one asks for, and freeing it means fewer slow pictures and fewer stalled answers for them.

Cost of inaction

The bot keeps holding memory it never uses, stays near its 384 MiB protection line, and users keep meeting slow pictures and replies in the next swap storm; the 410 MiB figure stays unexplained to the next reader.

Acceptance criteria

  • A bot start that does no lookup does not read or parse the stress dictionary file (regression test with a synthetic dictionary and a read counter) — proof: proof
  • The first lookup, lookupMany or findSimilar loads it once and answers exactly as before (existing stress-dictionary tests unchanged and passing) — proof: proof
  • A missing file leaves the dictionary absent at startup with the existing warning; a file that exists but cannot be parsed is logged once at the first load and every later lookup finds nothing, and the lookup_stress tool answers 'Stress dictionary not loaded' instead of reporting every word as missing — proof: proof
  • Measured on the production container after deploy: bot RSS right after start is lower by roughly the retained heap (about 80-140 MB) and /ready is ok — proof: proof

Screenshots

(none)

Deploy

Deployed: no

Links

Skipped gates

Activity

  1. alex-mextner commented on Nov 4, 2026

    @alex-mextner
    OwnerAuthor

    Memory investigation 2026-11-04 (read-only first), result for the 410 MiB question; ticket 800 holds the host-pressure side.

    Measured: a fresh e4185dd container sat at 413 MiB one minute after start and 449-461 MiB over the next 16 minutes (16 samples a minute apart): flat, no growth; cgroup anon 405 MB falling to 363 MB while the kernel swapped cold pages out (swap.current 108 MB at minute 20, memory.low 384 MiB). The bun process held 328 MB anon plus 93 MB swapped; the render pool's Chromium processes (started at boot by playwrightPool.initialize(), unchanged code) are mostly shared pages.
    Not the cause: dependencies, Dockerfile, compose limits and bunfig are identical between ff93220 and e4185dd, and importing every startup module costs the same in both trees (130.8-133.1 MB RSS old, 133.0-133.4 MB new, 29 MB heap, two runs each). The 138-163 MiB seen on the 28 h old container is the same working set after the host swapped its idle part out (the 2026-10-10 figure of 71 MB resident is the same effect), so compare containers of the same age.
    Largest single item found: the stress dictionary, parsed at boot though production never uses it (SILERO_PYTHON_PATH and the MTProto service tier are unset): in a bare bun process with the production file RSS goes 51 MB -> 362 MB peak -> 190 MB after GC, 79 MB heap retained, 1.9 s on a loaded PC.
    Fix in review: it loads on first use (or at startup when SILERO_PYTHON_PATH is set). Two single-seat Opus rounds so far; round 1 findings fixed (Silero startup read, unreadable file keeps Silero off, README, a Silero-path test). Not shipped yet: branch lazy-stress-dict-20261104, version 0.3.5, round 2 review job 20261104T115927-a1c373d4. Post-deploy acceptance (criterion 4: bot RSS right after start lower by roughly the retained heap, /ready ok) is NOT measured yet.

  2. alex-mextner commented on Nov 4, 2026

    @alex-mextner
    OwnerAuthor

    PR for this ticket: #965 (version 0.3.4, Refs #800 and #963). Two single-seat Opus rounds, round 2 clean. Hosted CI and Stage QA were pending when this run ended; not shipped, not deployed. Acceptance criterion 4 (bot RSS right after start on the deployed container) is still unmeasured.

  3. alex-mextner commented on Nov 4, 2026

    @alex-mextner
    OwnerAuthor

    screenshot: p963-tests.png

  4. alex-mextner commented on Nov 4, 2026

    @alex-mextner
    OwnerAuthor

    screenshot: p963-rss.png

  5. alex-mextner commented on Nov 4, 2026

    @alex-mextner
    OwnerAuthor

    Shipped and verified on production (2026-11-04): PR #965 merged as main bbedb86; the release receipt on the odroid names that revision (image id sha256:c81a46f8..., verified_at 2026-11-04T17:00:36Z), the container restarted 0 times, /health is ok and /ready is "ok (unverified)", and the container logs of the first 15 minutes hold 40 records, all level 30.

    Acceptance criterion 4, measured on the production container (the bun process of the container, /proc//status):

    • 3 min 24 s after start: VmRSS 290 MB, RssAnon 247 MB, VmSwap 1.4 MB; docker stats 435 MiB.
    • 15 min after start: VmRSS 285 MB, RssAnon 252 MB, VmSwap 1.4 MB; docker stats 345 MiB.
    • "Stress dictionary loaded" appears 0 times in the logs: the file was not read.
    • Before (main e4185dd, same host, runbook measurement): 328 MB anon plus 93 MB swapped in the bun process, 413 MiB in docker stats one minute after start, 449-461 MiB afterwards.
      So the bot's anonymous memory is about 80 MB lower and flat over 15 minutes. Not measured: a first lookup_stress call on production (nobody asked for a stress mark while I watched), so the stall and the later memory of a loaded dictionary there are unmeasured; follow-up ticket: Lazy stress dictionary: first lookup_stress stalls the event loop, a transient read error disables it until restart, and Silero paths ignore a failed load #966.

    All four criteria are checked with proofs (tests of the lazy load; the RSS table).

  6. alex-mextner commented on Nov 4, 2026

    @alex-mextner
    OwnerAuthor

    Shipped: merged via #965.
    Merge commit: bbedb86
    This does NOT close the ticket — acceptance still needs proof. Run task read #963 for the outstanding criteria, then task check/task done.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions