Conversation
…OTICE The LICENSE file still carried the Apache 2.0 template placeholders (Copyright [yyyy] [name of copyright owner]), so the notice was never actually applied to this work. Fill it in, add the NOTICE file Apache 2.0 expects for the copyright line, and point the four READMEs at both.
license: name the copyright holder and add NOTICE
Eviction-safe port of the qwen36 CUDA expert tier for glm53: - qwen36_tier.c/h: stream-int4 ownership path (qt_note deep-copies all six pieces before returning; uploader never observes a recycled RAM slot) - backend_cuda.cu/h + backend_loader.c: stream promote/issue support - glm53.c: cuda_tier_owner flag + per-token resident-route issue in ffn_layer - Makefile: glm53 tier wiring + stream-int4 test target - tests: new test_qwen36_tier_stream_int4 (ok); shutdown updated (ok); fp8/int8/invariants contract updates — invariants still core-dumps (open) Status at this commit: stream_int4 ok, shutdown ok, invariants core dump OPEN. Homeserver acceptance (A1-A6) not yet run. WIP — do not merge. Co-Authored-By: <none>
…r turn Counts CUDA-resident routed experts via qt_is_resident() as VRAM, LRU cache slots as RAM, remainder as disk — same cadence as deepseek_v4. Includes the newline-escape fix for the TIERS serve_line so the frame no longer glues into the next protocol line (float() ValueError on the python side made every chat turn fail).
The fake-backend tests prove tier bookkeeping, not numbers. This runs one expert's six pieces through quant.h's CPU reference and the tier's full note->upload->issue->take round trip on a real device at glm53-flash geometry (D=4096, Ih=2048, gs=64, swiglu clamp 10), covering single expert, weighted K-expert group, and the clamped epilogue. Builds everywhere; skips at runtime without CUDA or a device.
glm53 packs its expert nibbles offset-binary (matmul_i4_grouped decodes nibble-8), which is exactly the upload format backend_cuda converts with offset_to_signed_s4 before the kernels sign-decode. The stream tier's staging XORed 0x88 first -- the convention qwen36's two's-complement containers need -- so the upload conversion double-flipped every sign bit and every tier-resident expert computed garbage. Verified at expert level on an RTX 4000 Ada: XOR-staged uploads mismatched the CPU reference by ~1e1 rel, raw uploads matched to 3e-6. Key the staging on G_int4_stream (set only by qt_init_stream_int4, the glm53 path): stream tiers copy verbatim, qwen36's warmstart tier keeps the XOR. The fake-backend stream test now asserts the verbatim staging.
Same defect class as glm53 (#A3): cap came from argv alone, tier_warmstart loaded ALL experts into RAM with no ceiling. RAM_GB is now a whole-process ceiling: measured RSS + activations/KV reserve subtracted, argv cap clamped with named terms, refuse-to-start (exit 2) when even cap=1 overflows MemAvailable unless COLI_RAM_OVERCOMMIT=1. Warmstart loads the planned VRAM set first, then only what the cap allows; the rest stays in the container until LRU demand. Telemetry at init and every 64 loads.
Xore
changed the title
…tier path glm53_emit_tiers double-counted promoted experts: an LRU slot the qt tier had already promoted appeared in BOTH the RAM sum and the VRAM registry count, driving disk negative and silently clamped -> vram+ram+disk drifted from the routed total and the dashboard lied. Counts are now exclusive: VRAM = qt residency registry, RAM = live LRU slots not resident in qt, disk = remainder; reconciliation holds by construction. Non-streaming branch now counts only the routed (sparse) layer range instead of every layer. HITS audited end to end: no tier-path gap exists (routes marked at issue time, emitted every completed turn, parser format matches); no web change needed.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
qt expert tier + expert-cache RAM ceilings (glm53 & qwen36), A3 soak-verified
What this PR is
Two related things, honestly labelled:
The qt expert tier feature branch (7 commits, unchanged scope):
eviction-safe expert tier stream path with glm53 integration
(ef416fa), TIERS tier telemetry at boot and per turn (885b31a),
real-GPU parity harness + Makefile/test wiring (6d8be48, b26bb9f,
dd541f5).
Two expert-cache RAM-ceiling fixes landed in this branch while the
tier was under A3 acceptance soak. They fix the same defect class and
were verified together on the soak box, which is why they ride here
instead of standalone:
9a2df8bfix(qtier): honor RAM_GB in glm53 cache00c58f8fix(qwen36): honor RAM_GB in expert cache + warmstart(touches only
c/qwen36.c; the tier path does not depend on it andvice versa — both compile/verify independently, see evidence below)
The A3 RAM bug (why the two fixes exist)
An explicit
--ram Nnever reached the expert-cache sizing code in thetier engines: glm53's
expert_cache_initsized the LRU without readingthe resolved budget, and qwen36's
tier_warmstartdeliberately loadedALL experts into RAM with no ceiling at all. Under sustained load the
engine's anon RSS grew unbounded. Observed twice on the 92 GB soak box:
--ram 45soak: 81.5 GB anon RSS, kernel OOM-killer.--ram 20rerun: 82 GB RSS, resident experts 0/12960, zero uploads(tier never engaged), box starved to 596 MB available.
The fixes
Both give the engine the contract its siblings (kimi_k3 #855, olmoe,
colibri.c
cap_for_ram) already have:c/glm53.conly):RAM_GBis a whole-process ceiling —cache budget =
RAM_GB − measured RSS − 3 GBreserve, computed atcache-init. Explicit cache requests (
GLM53_EXPERT_GB/ cap override)are clamped under it with a named-terms notice. Hard
exit(1)at bootif even one slot/layer exceeds the ceiling.
[glm53][warm-start]telemetry (cap/RSS/anon-RSS/loads) every 64 loads.
c/qwen36.conly): same contract — measured resident +activations/KV reserve subtracted, argv cap clamped with named terms,
refuse-to-start (
exit(2)) when even cap=1 overflows MemAvailableunless
COLI_RAM_OVERCOMMIT=1. Warmstart loads the planned VRAM setfirst, then only what the cap allows; the rest stays in the container
until LRU demand. Per-64-load telemetry.
Sibling audit (read-only): kimi_k3, olmoe and colibri.c
cap_for_ramalready enforce the ceiling — unchanged. deepseek_v4 enforces
memory_limit_bytesincoli_v4_resource_plan_compute— unchanged.Evidence
make -C c glm53andmake -C c qwen36build clean.make -C c tests/test_qwen36_tier_stream_int4 && ./c/tests/…→test_qwen36_tier_stream_int4: ok(real-GPU parity harness greenafter both fixes).
(no dependency on the tier commits); the tier commits do not touch
c/qwen36.c.--ram 20:RSS pinned flat at 17.4 GB across 150,000+ expert loads (bug was
82 GB and climbing); TIERS emitted at boot in ~60 s (previously never
appeared); box stayed at 58 GB available throughout the 50-request
loop.
Notes
unmodified (no re-quant), so no container regeneration is needed for
existing gs64 containers.