π gave it a star if you find something useful......
FluidLayer is a memory layer you can attach on top of any HuggingFace
causal model (5 Colab cells β see the
Colab guide). It remembers what it has seen
(12-gram replay recall 0.9945, measured), keeps a bounded KV
cache that never grows past kv_max_size (default 256), and pushes
every evicted token into long-term memory with its REAL hidden
state β no token is silently dropped.
Every number in this README was measured on this repo's CI, not claimed. Full logs: the verification-results branch and docs/VERIFICATION.md.
| Claim | Measured |
|---|---|
| 12-gram replay recall | 0.9945 base-only, 0.9945 base+KV (exact parity) β e19 |
| Fresh (never-seen) recall | 0.0 by construction (the exact tier only recalls contexts it has seen) |
| Bounded KV cache | max 256 seen while 2000 tokens streamed; 1744 evictions, all stored to long-term memory (e19) |
| Attach Ξ loss (tiny- | |
| gpt2) | 0.0000 nats with the main layer (e19); β +0.0005 untrained with the base store |
| Worst-case memory | 21.4 MB measured (ceiling: 100 MB) β e19 |
| Thread safety | 8 threads, 0 errors (e19) |
| Optional graft security | Session masking is available in optional graft APIs; MILITARY exposes Fernet helpers (AES-128-CBC + HMAC-SHA256). Neither is enabled by the recommended BaseKVLayer (SecurityLevel.NONE), which provides no tenant isolation or encryption. |
| GGUF export | valid GGUF container (magic b'GGUF'); llama.cpp execution needs a backend patch |
| Quantization (e21) | int8: 4.0x smaller, round-trip error 0.3936% β e21 |
| Multi-GPU (e21) | sharded bounded KV: 4 balanced shards [64, 64, 64, 64], order preserved β e21 |
| Hardware acceleration (e21) | vectorized fast path 40.34x faster than the loop path on CI CPU β e21 |
| Overhead | full forward ~5-9 ms / 256 tokens; 12-gram keygen ~3-6 Β΅s; store lookup ~3-5 Β΅s (CI CPU) |
The verification pipeline itself found and fixed seven real bugs along the way β the story is in docs/VERIFICATION.md.
pip install fluidlayer # from PyPI (numpy-only core)
pip install "fluidlayer[gpu]" # + torch, transformers, accelerate, safetensors
# from source
git clone https://github.com/kapildhinwa2010-dev/fluidtech.git
cd fluidtech && pip install .
pip install cryptography # optional: MILITARY-level Fernet encryption
pip install pytest # optional: to run the test suitesimport torch
from fluidlayer import BaseKVConfig, BaseKVLayer # the main layer (lazy import, needs torch)
layer = BaseKVLayer(BaseKVConfig(d=64, kv_max_size=256))
h = torch.randn(8, 64) # hidden states
tok = torch.randint(0, 32000, (8,)) # token ids
out = layer.forward(h, tok) # attach inside any forward hook
print(layer.stats()) # base recall, KV size, evic
tions, long-termNumPy-only (no torch):
import numpy as np
from fluidlayer.graft_bulletproof import SAABranchBulletproof, BulletproofConfig, HashMode, SecurityLevel
from fluidlayer import quantize_int8, dequantize_int8 # e21: int8 quantization (numpy-only)
layer = SAABranchBulletproof(M=16, r=4, d=64, config=BulletproofConfig(
ctx_len=12, hash_mode=HashMode.DOUBLE_64, security_level=SecurityLevel.NONE))
h_out, info = layer.forward(np.random.randn(1, 64), np.random.randn(1, 64), np.array([7]))
q, scale = quantize_int8(h_out) # float32 -> int8 (4x smaller)
back = dequantize_int8(q, scale) # round-trip error < 0.4% (measured, e21)More: docs/USAGE.md (configuration, performance, security, GGUF, examples, troubleshooting) and docs/COLAB_GUIDE.md (attach to a real model in 5 Colab cells).
| Layer | What it is | Status | Where |
|---|---|---|---|
| base + KV cache | exact-tier store + bounded KV + long-term memory | THE main layer β RECOMMENDED | fluidlayer/base_kv_layer.py |
| base only | exact-tier store, no KV window, no long-term | optional (Colab guide 4.1) | fluidlayer/graft_bulletproof.py |
| mamba ssm | SuperfluidMambaFluidLayer | optional (Colab guide 4.2) | experiments/superfluid_mamba_layer.py |
| KV cache only | the wrapper standalone | optional (Colab guide 4.3) | kv_cache/constant_kv_superfluid.py |
| ultra | UltraFluidStack (stacked + cross-attention) | optional (Colab guide 4.4) | experiments/ultra_fluid_layer.py |
| combined | all branches fused into one layer | optional (Colab guide 4.5) | combined/combined_layer.py |
The optional layers are in the repo and importable, but nothing depends on them β use them only if you want them.
base + KV cache β CI run [e19](https://github.com/kapildhinwa2010-dev/fluidtech/blob/verification-results/results/base-kv/run.jso n):
| Metric | base only | base + KV cache |
|---|---|---|
| 12-gram replay recall | 0.9945 | 0.9945 (exact parity) |
| fresh recall | 0.0 (by construction) | 0.0 (by construction) |
| speed (ms / 256 tokens, d=16) | 3.9 | 6.7 (1.72Γ) |
| worst-case memory | 10.7 MB | 21.4 MB (ceiling: 100 MB) |
| thread errors (8Γ20) | 0 | 0 |
| bounded KV | β | max 256 seen; 1744 evictions β all stored to long-term |
| attach Ξ loss (tiny-gpt2) | 0.0000 | 0.0000 nats |
All five layers in one identical harness β CI run e18:
| Layer | replay recall | ms/256tok | worst-case MB | threads | bounded KV | GPT-2 Ξ |
|---|---|---|---|---|---|---|
| base (main) | 0.9945 | 3.9 | 10.7 | 0 | β | 0.0000 |
| mamba (experiments) | 0.9940 | 8.8 | 10.7 | 0 | β | +0.0017 |
| ultra (experiments) | 0.9940 | 24.7 | 21.4 | 0 | β | +0.0001 |
| kv (kv_cache) | n/a (wrapper) | 5.0 | 10.7 | 0 | β | 0.0000 |
| combined (all 4) | 0.9945 | 36.4 | 42.9 | 0 | β | +0.0001 |
Honest reading:
- fresh recall is 0 BY CONSTRUCTION β the exact tier only recalls contexts it has seen before;
- the attach deltas are untrained layers: proof that the layer does no harm, NOT a quality claim;
- speeds were measured at d=16 on CI CPU runners and scale with d.
- base tier β every 12-gram of token ids becomes a digest key in an exact-tier store; when the same 12-gram context is seen again, the stored token is retrieved and written into the output (verbatim replay recall β 0.9945);
- bounded KV cache β the fused hidden states enter a fixed-size KV window (
kv_max_size, default 256). The window never grows: when it is full, the oldest entries are evicted; - long-term memory β every evicted token is stored in the long-term memory together with its REAL hidden state (a sliding stream keeps the last
kv_max_sizestates, so an ev iction always has its true state available). Nothing is dropped.
- KV compression (architectural): the KV window is bounded at
kv_max_size, so KV memory is O(1) with respect to context length β stream 100 tokens or 1,000,000, the window stays 256. Evicted tokens are stored once (token id + one hidden-state row) instead of per-head K and V tensors, which is where the memory saving comes from. - Exact-tier dedup: repeated 12-gram contexts collapse to a single store entry (digest key). The store evicts by LRU + frequency once
max_capacityis reached. - Worst-case accounting (measured in e19): each store entry β 112 B β 2 stores Γ 100,000 entries β 21.4 MiB, plus the KV window (256 Γ 2 Γ d Γ 4 B) β well under the 100 MiB ceiling.
- What this is NOT: the main layer has no sparsity and no low-rank tensor compression β its compression is architectural (bounded window + dedup + eviction), and the numbers above are measured, not projected. (An OPTIONAL int8 quantization module now exists β see the e21 section below β measured in CI, but NOT applied to the main layer by default.)
| Doc | What it covers |
|---|---|
| docs/GETTING_STARTED.md | first steps |
| docs/USAGE.md | quick start, configuration, performance, security, GGUF, examples, troubleshooting |
| docs/COLAB_GUIDE.md | attach the layer to a real model on a Colab T4 β 5 steps |
| docs/API.md | the API surface |
| docs/VERIFICATION.md | what is measured and how |
| docs/CHANGELOG.md | release history |
| docs/REPRODUCING.md Β· docs/COMPARISON.md Β· docs/LIMITATION.md Β· docs/NEGATIVE-RESULTS.md | honest deep-dives |
| [verification-results](https://github.com/kapildhinwa2010-dev/flui | |
| dtech/tree/verification-results/results) | the raw measured results (run.json + logs, refreshed on every push) |
fluidtech/
βββ fluidlayer/ # the core package (numpy-only at import time)
β βββ base_kv_layer.py # β THE main layer: base + bounded KV + long-term memory
β βββ graft_bulletproof.py # the base layer (exact-tier store, security, GGUF)
β βββ quantization.py # e21: int8 quantization (4.0x smaller)
β βββ multi_gpu.py # e21: device detection + sharded bounded KV cache
β βββ acceleration.py # e21: framework detection + vectorized fast path
β βββ torch_graft_bulletproof.py # the PyTorch twin
β βββ sheaf.py, diffusion.py, topology.py, model.py, autodiff.py, ... # the fluid core
βββ kv_cache/ # bounded KV cache + long-term memory companions
βββ combined/ # the optional combined layer (all four branches)
βββ experiments/ # optional layers + the measurement scripts e1βe21
βββ deployment/ # session-registry companion
βββ tests/ # pytest suites (torch-dependent tests skip without torch)
βββ docs/ # USAGE.md, COLAB_GUIDE.md, VERIFICATION.md, CHANGELOG.md, ...
βββ examples/ # quickstart + demos
βββ tools/ # bundle + release helpers
βββ .github/workflows/ # CI, verify-claims, features-test, e20-finetune, release
βββ corpus.txt, selftest.py, run_without_pytest.py, test_bulletproof.py
python -m pytest -q tests # the suite (torch-dependent tests skip without torch)
python tests/test_base_kv.py # the main layer's own self-test (CPU, ~1 min)
ruff check fluidlayer experiments tools tests examples selftest.py run_without_pytest.pyEvery push to main runs: CI (Python 3.9-3.12 verify matrix, package build + clean-venv install, lint, optional PyTorch parity), verify-claims (re-measures e19/e18 and publishes to
verification-results), features-test (the 11 e21 unit tests + the e21 measurement) and e20-finetune (on e20 changes).
- The layers are untrained: attaching them does not improve a model's loss (measured: Ξ β 0 on tiny-gpt2). The value is the memory behaviour, not perplexity.
- The exact tier recalls repeated contexts; genuinely fresh text recalls 0 by construction.
- The layer runs on CPU (NumPy) inside the hook: keep demo prompts short; the KV window is O(1) but the per-token store work is not free.
- The optional mamba/ultra/combined layers are experimental (see each module's honest docstring).
| Run | What it measured | Where |
|---|---|---|
| e17 | combined layer, 9/9 claims | measured on the combined-layer branch before the merge; the layer is now the optional combined module |
| e18 | all 5 layers, one harness | results/comparison/run.json |
| e19 | base + kv cache (this layer) | results/base-kv/run.json |
| e21 | quantization + multi-GPU + acceleration (the 3 coming-soon features) | results/e21-features/run.json |
| e20 | short-budget fine-tune + KV wrapper (tiny-gpt2, 5-min LoRA) | results/e20-finetune/run.json |
Reproduce any of them: the scripts live on main β experiments/e18_compare.py, experiments/e19_base_kv.py β each is a standalone script that measures, prints, and publishes its own results (e18/e19 re-run on every push to main).
Idea: attach the base+KV wrapper layer onto a model, run a 5-10 min LoRA fine-tune, and MEASURE what changes. Script: e20_finetune_kv.py; workflow e20-finetune.yml runs on CI on every e20 change. The KV wrapper is OFF during training (evictions would corrupt the training context) β it is attached only after the fine-tune.
CI measured (GitHub Actions CPU, commit 4912034e): 840 steps / 300.3 s (5 min budget, LoRA: 128 of 102842 params trainable), train loss flat (10.8249 -> 10.825), eval 10.826 -> 10.8254 (delta 0.0006 nats = noise-level), attach delta 0.0, KV after fine-tune: max 256 + 1280 evictions β bounded.
Honest reading: tiny-gpt2 (2-dim hidden, ~100k params) + 128 LoRA params shows no measurable effect in 5 min β try it on a bigger model: python experiments/e20_finetune_kv.py --model Qwen/Qwen2.5-1.5B --minutes 10 (Colab T4). Full results: run.json
e21: the 3 "coming soon" features β quantization, multi-GPU, hardware acceleration (MEASURED, merged to main)
These 3 features are implemented + CI-tested β tests 11/11 PASS, all results came back positive, so they were merged to main. Modules: quantization.py, multi_gpu.py, acceleration.py; workflow features-test.yml re-runs and re-publishes them on every push.
CI measured (GitHub Actions CPU, commit 4912034e):
| Feature | Measured |
|---|---|
| quantization (int8) | float32 -> int8 = 4.0x smaller; round-trip error 0.3936% (theoretical bound < 0.4%) |
| multi-GPU | device detection (cuda -> mps -> cpu fallback); 1000 inserts into 4 shards -> size 256, 744 evictions, shard sizes [64, 64, 64, 64] balanced, insertion order preserved |
| hardware acceleration | torch 2.15 (CPU) detected, cuda/mps/numba/cupy flags honest; vectorized fast path 40.34x faster than the loop path (0.000674 s -> 0.000017 s), outputs bit-consistent |
Honest reading: CI runners have no GPU β the multi-GPU sharding + eviction logic was verified on CPU virtual shards (placement switches automatically on real GPU machines), and the measured acceleration speedup comes from CPU vectorization (GPU backend detection is ready; run-to-run speedup varies a bit). Full results: run.json, features.md aur tests.log (11/11 PASSED) on the verification-results branch.
MIT β see LICENSE. Created by Kapil Dhinwa. Questions, issues, contributions: GitHub Β· kapildhinwa2010@gmail.com Β· CONTRIBUTING.md Β· SECURITY.md
β Star this repository if you find it useful!
π Every claim in this README is measured β see docs/VERIFICATION.md and the verification-results branch.
Note: this experiment shows that fine-tuning has no measurable effect on very small models.
You can try training or fine-tuning on larger models if you are interested.
π STAR IT IF YOU FIND THIS USEFUL