Skip to content
kapildhinwa2010-devPublic

About

A bounded memory layer for language models, with exact recall for retained token IDs. NumPy core; optional PyTorch adapter.

Topics

Resources

Code of conduct

Contributing

Security policy

Stars

5 stars

Watchers

1 watching

Forks

Latest commit

Β 

History

178 Commits

Folders and files

NameName
Last commit message
Last commit date
Β 
Β 
Β 
Β 
Β 
Β 
Β 
Β 
Β 
Β 
Β 
Β 
Β 
Β 
Β 
Β 
Β 
Β 
Β 
Β 
Β 
Β 
Β 
Β 
Β 
Β 
Β 
Β 
Β 
Β 
Β 
Β 
Β 
Β 
Β 
Β 
Β 
Β 
Β 
Β 
Β 
Β 
Β 
Β 
Β 
Β 
Β 
Β 
Β 
Β 
Β 
Β 
Β 
Β 
Β 
Β 
Β 
Β 
Β 
Β 
Β 
Β 
Β 
Β 
Β 
Β 
Β 
Β 
Β 
Β 
Β 
Β 
Β 
Β 
Β 
Β 

Repository files navigation

FluidTech wave

FluidTech

the measured memory layer for language models

CI PyPI License: MIT Python 3.9+ NumPy 1.23+ 🌟 gave it a star if you find something useful...... FluidLayer is a memory layer you can attach on top of any HuggingFace causal model (5 Colab cells β€” see the Colab guide). It remembers what it has seen (12-gram replay recall 0.9945, measured), keeps a bounded KV cache that never grows past kv_max_size (default 256), and pushes every evicted token into long-term memory with its REAL hidden state β€” no token is silently dropped.

Every number in this README was measured on this repo's CI, not claimed. Full logs: the verification-results branch and docs/VERIFICATION.md.


βœ… What is measured (re-measured on CI on every push to main)

Claim Measured
12-gram replay recall 0.9945 base-only, 0.9945 base+KV (exact parity) β€” e19
Fresh (never-seen) recall 0.0 by construction (the exact tier only recalls contexts it has seen)
Bounded KV cache max 256 seen while 2000 tokens streamed; 1744 evictions, all stored to long-term memory (e19)
Attach Ξ” loss (tiny-
gpt2) 0.0000 nats with the main layer (e19); β‰ˆ +0.0005 untrained with the base store
Worst-case memory 21.4 MB measured (ceiling: 100 MB) β€” e19
Thread safety 8 threads, 0 errors (e19)
Optional graft security Session masking is available in optional graft APIs; MILITARY exposes Fernet helpers (AES-128-CBC + HMAC-SHA256). Neither is enabled by the recommended BaseKVLayer (SecurityLevel.NONE), which provides no tenant isolation or encryption.
GGUF export valid GGUF container (magic b'GGUF'); llama.cpp execution needs a backend patch
Quantization (e21) int8: 4.0x smaller, round-trip error 0.3936% β€” e21
Multi-GPU (e21) sharded bounded KV: 4 balanced shards [64, 64, 64, 64], order preserved β€” e21
Hardware acceleration (e21) vectorized fast path 40.34x faster than the loop path on CI CPU β€” e21
Overhead full forward ~5-9 ms / 256 tokens; 12-gram keygen ~3-6 Β΅s; store lookup ~3-5 Β΅s (CI CPU)

The verification pipeline itself found and fixed seven real bugs along the way β€” the story is in docs/VERIFICATION.md.


πŸ“¦ Installation

pip install fluidlayer            # from PyPI (numpy-only core)
pip install "fluidlayer[gpu]"     # + torch, transformers, accelerate, safetensors

# from source
git clone https://github.com/kapildhinwa2010-dev/fluidtech.git
cd fluidtech && pip install .

pip install cryptography          # optional: MILITARY-level Fernet encryption
pip install pytest                # optional: to run the test suites

πŸš€ Quick start

import torch
from fluidlayer import BaseKVConfig, BaseKVLayer   # the main layer (lazy import, needs torch)

layer = BaseKVLayer(BaseKVConfig(d=64, kv_max_size=256))
h   = torch.randn(8, 64)                       # hidden states
tok = torch.randint(0, 32000, (8,))            # token ids
out = layer.forward(h, tok)                    # attach inside any forward hook
print(layer.stats())                           # base recall, KV size, evic
tions, long-term

NumPy-only (no torch):

import numpy as np
from fluidlayer.graft_bulletproof import SAABranchBulletproof, BulletproofConfig, HashMode, SecurityLevel
from fluidlayer import quantize_int8, dequantize_int8   # e21: int8 quantization (numpy-only)

layer = SAABranchBulletproof(M=16, r=4, d=64, config=BulletproofConfig(
    ctx_len=12, hash_mode=HashMode.DOUBLE_64, security_level=SecurityLevel.NONE))
h_out, info = layer.forward(np.random.randn(1, 64), np.random.randn(1, 64), np.array([7]))

q, scale = quantize_int8(h_out)               # float32 -> int8 (4x smaller)
back = dequantize_int8(q, scale)              # round-trip error < 0.4% (measured, e21)

More: docs/USAGE.md (configuration, performance, security, GGUF, examples, troubleshooting) and docs/COLAB_GUIDE.md (attach to a real model in 5 Colab cells).


🧠 The layers (and the optional ones)

Layer What it is Status Where
base + KV cache exact-tier store + bounded KV + long-term memory THE main layer β€” RECOMMENDED fluidlayer/base_kv_layer.py
base only exact-tier store, no KV window, no long-term optional (Colab guide 4.1) fluidlayer/graft_bulletproof.py
mamba ssm SuperfluidMambaFluidLayer optional (Colab guide 4.2) experiments/superfluid_mamba_layer.py
KV cache only the wrapper standalone optional (Colab guide 4.3) kv_cache/constant_kv_superfluid.py
ultra UltraFluidStack (stacked + cross-attention) optional (Colab guide 4.4) experiments/ultra_fluid_layer.py
combined all branches fused into one layer optional (Colab guide 4.5) combined/combined_layer.py

The optional layers are in the repo and importable, but nothing depends on them β€” use them only if you want them.

Measured results (honest)

base + KV cache β€” CI run [e19](https://github.com/kapildhinwa2010-dev/fluidtech/blob/verification-results/results/base-kv/run.jso n):

Metric base only base + KV cache
12-gram replay recall 0.9945 0.9945 (exact parity)
fresh recall 0.0 (by construction) 0.0 (by construction)
speed (ms / 256 tokens, d=16) 3.9 6.7 (1.72Γ—)
worst-case memory 10.7 MB 21.4 MB (ceiling: 100 MB)
thread errors (8Γ—20) 0 0
bounded KV β€” max 256 seen; 1744 evictions β†’ all stored to long-term
attach Ξ” loss (tiny-gpt2) 0.0000 0.0000 nats

All five layers in one identical harness β€” CI run e18:

Layer replay recall ms/256tok worst-case MB threads bounded KV GPT-2 Ξ”
base (main) 0.9945 3.9 10.7 0 β€” 0.0000
mamba (experiments) 0.9940 8.8 10.7 0 β€” +0.0017
ultra (experiments) 0.9940 24.7 21.4 0 β€” +0.0001
kv (kv_cache) n/a (wrapper) 5.0 10.7 0 βœ… 0.0000
combined (all 4) 0.9945 36.4 42.9 0 βœ… +0.0001

Honest reading:

  • fresh recall is 0 BY CONSTRUCTION β€” the exact tier only recalls contexts it has seen before;
  • the attach deltas are untrained layers: proof that the layer does no harm, NOT a quality claim;
  • speeds were measured at d=16 on CI CPU runners and scale with d.

How the main layer works

  1. base tier β€” every 12-gram of token ids becomes a digest key in an exact-tier store; when the same 12-gram context is seen again, the stored token is retrieved and written into the output (verbatim replay recall β‰ˆ 0.9945);
  2. bounded KV cache β€” the fused hidden states enter a fixed-size KV window (kv_max_size, default 256). The window never grows: when it is full, the oldest entries are evicted;
  3. long-term memory β€” every evicted token is stored in the long-term memory together with its REAL hidden state (a sliding stream keeps the last kv_max_size states, so an ev iction always has its true state available). Nothing is dropped.

Compression (what is compressed, and how β€” honestly)

  • KV compression (architectural): the KV window is bounded at kv_max_size, so KV memory is O(1) with respect to context length β€” stream 100 tokens or 1,000,000, the window stays 256. Evicted tokens are stored once (token id + one hidden-state row) instead of per-head K and V tensors, which is where the memory saving comes from.
  • Exact-tier dedup: repeated 12-gram contexts collapse to a single store entry (digest key). The store evicts by LRU + frequency once max_capacity is reached.
  • Worst-case accounting (measured in e19): each store entry β‰ˆ 112 B β†’ 2 stores Γ— 100,000 entries β‰ˆ 21.4 MiB, plus the KV window (256 Γ— 2 Γ— d Γ— 4 B) β€” well under the 100 MiB ceiling.
  • What this is NOT: the main layer has no sparsity and no low-rank tensor compression β€” its compression is architectural (bounded window + dedup + eviction), and the numbers above are measured, not projected. (An OPTIONAL int8 quantization module now exists β€” see the e21 section below β€” measured in CI, but NOT applied to the main layer by default.)

πŸ“š Documentation

Doc What it covers
docs/GETTING_STARTED.md first steps
docs/USAGE.md quick start, configuration, performance, security, GGUF, examples, troubleshooting
docs/COLAB_GUIDE.md attach the layer to a real model on a Colab T4 β€” 5 steps
docs/API.md the API surface
docs/VERIFICATION.md what is measured and how
docs/CHANGELOG.md release history
docs/REPRODUCING.md Β· docs/COMPARISON.md Β· docs/LIMITATION.md Β· docs/NEGATIVE-RESULTS.md honest deep-dives
[verification-results](https://github.com/kapildhinwa2010-dev/flui
dtech/tree/verification-results/results) the raw measured results (run.json + logs, refreshed on every push)

πŸ“ Repository layout

fluidtech/
β”œβ”€β”€ fluidlayer/            # the core package (numpy-only at import time)
β”‚   β”œβ”€β”€ base_kv_layer.py         # ⭐ THE main layer: base + bounded KV + long-term memory
β”‚   β”œβ”€β”€ graft_bulletproof.py     # the base layer (exact-tier store, security, GGUF)
β”‚   β”œβ”€β”€ quantization.py          # e21: int8 quantization (4.0x smaller)
β”‚   β”œβ”€β”€ multi_gpu.py             # e21: device detection + sharded bounded KV cache
β”‚   β”œβ”€β”€ acceleration.py          # e21: framework detection + vectorized fast path
β”‚   β”œβ”€β”€ torch_graft_bulletproof.py  # the PyTorch twin
β”‚   └── sheaf.py, diffusion.py, topology.py, model.py, autodiff.py, ...   # the fluid core
β”œβ”€β”€ kv_cache/              # bounded KV cache + long-term memory companions
β”œβ”€β”€ combined/              # the optional combined layer (all four branches)
β”œβ”€β”€ experiments/           # optional layers + the measurement scripts e1–e21
β”œβ”€β”€ deployment/            # session-registry companion
β”œβ”€β”€ tests/                 # pytest suites (torch-dependent tests skip without torch)
β”œβ”€β”€ docs/                  # USAGE.md, COLAB_GUIDE.md, VERIFICATION.md, CHANGELOG.md, ...
β”œβ”€β”€ examples/              # quickstart + demos
β”œβ”€β”€ tools/                 # bundle + release helpers
β”œβ”€β”€ .github/workflows/     # CI, verify-claims, features-test, e20-finetune, release
└── corpus.txt, selftest.py, run_without_pytest.py, test_bulletproof.py

πŸ§ͺ Tests & CI

python -m pytest -q tests      # the suite (torch-dependent tests skip without torch)
python tests/test_base_kv.py   # the main layer's own self-test (CPU, ~1 min)
ruff check fluidlayer experiments tools tests examples selftest.py run_without_pytest.py

Every push to main runs: CI (Python 3.9-3.12 verify matrix, package build + clean-venv install, lint, optional PyTorch parity), verify-claims (re-measures e19/e18 and publishes to

verification-results), features-test (the 11 e21 unit tests + the e21 measurement) and e20-finetune (on e20 changes).


🧯 Limitations (honest)

  • The layers are untrained: attaching them does not improve a model's loss (measured: Ξ” β‰ˆ 0 on tiny-gpt2). The value is the memory behaviour, not perplexity.
  • The exact tier recalls repeated contexts; genuinely fresh text recalls 0 by construction.
  • The layer runs on CPU (NumPy) inside the hook: keep demo prompts short; the KV window is O(1) but the per-token store work is not free.
  • The optional mamba/ultra/combined layers are experimental (see each module's honest docstring).

Verification runs (for the skeptical)

Run What it measured Where
e17 combined layer, 9/9 claims measured on the combined-layer branch before the merge; the layer is now the optional combined module
e18 all 5 layers, one harness results/comparison/run.json
e19 base + kv cache (this layer) results/base-kv/run.json
e21 quantization + multi-GPU + acceleration (the 3 coming-soon features) results/e21-features/run.json
e20 short-budget fine-tune + KV wrapper (tiny-gpt2, 5-min LoRA) results/e20-finetune/run.json

Reproduce any of them: the scripts live on main β€” experiments/e18_compare.py, experiments/e19_base_kv.py β€” each is a standalone script that measures, prints, and publishes its own results (e18/e19 re-run on every push to main).


e20: short-budget fine-tune + KV wrapper (measured on main)

Idea: attach the base+KV wrapper layer onto a model, run a 5-10 min LoRA fine-tune, and MEASURE what changes. Script: e20_finetune_kv.py; workflow e20-finetune.yml runs on CI on every e20 change. The KV wrapper is OFF during training (evictions would corrupt the training context) β€” it is attached only after the fine-tune.

CI measured (GitHub Actions CPU, commit 4912034e): 840 steps / 300.3 s (5 min budget, LoRA: 128 of 102842 params trainable), train loss flat (10.8249 -> 10.825), eval 10.826 -> 10.8254 (delta 0.0006 nats = noise-level), attach delta 0.0, KV after fine-tune: max 256 + 1280 evictions β€” bounded.

Honest reading: tiny-gpt2 (2-dim hidden, ~100k params) + 128 LoRA params shows no measurable effect in 5 min β€” try it on a bigger model: python experiments/e20_finetune_kv.py --model Qwen/Qwen2.5-1.5B --minutes 10 (Colab T4). Full results: run.json


e21: the 3 "coming soon" features β€” quantization, multi-GPU, hardware acceleration (MEASURED, merged to main)

These 3 features are implemented + CI-tested β€” tests 11/11 PASS, all results came back positive, so they were merged to main. Modules: quantization.py, multi_gpu.py, acceleration.py; workflow features-test.yml re-runs and re-publishes them on every push.

CI measured (GitHub Actions CPU, commit 4912034e):

Feature Measured
quantization (int8) float32 -> int8 = 4.0x smaller; round-trip error 0.3936% (theoretical bound < 0.4%)
multi-GPU device detection (cuda -> mps -> cpu fallback); 1000 inserts into 4 shards -> size 256, 744 evictions, shard sizes [64, 64, 64, 64] balanced, insertion order preserved
hardware acceleration torch 2.15 (CPU) detected, cuda/mps/numba/cupy flags honest; vectorized fast path 40.34x faster than the loop path (0.000674 s -> 0.000017 s), outputs bit-consistent

Honest reading: CI runners have no GPU β€” the multi-GPU sharding + eviction logic was verified on CPU virtual shards (placement switches automatically on real GPU machines), and the measured acceleration speedup comes from CPU vectorization (GPU backend detection is ready; run-to-run speedup varies a bit). Full results: run.json, features.md aur tests.log (11/11 PASSED) on the verification-results branch.


πŸ“œ License & contact

MIT β€” see LICENSE. Created by Kapil Dhinwa. Questions, issues, contributions: GitHub Β· kapildhinwa2010@gmail.com Β· CONTRIBUTING.md Β· SECURITY.md


⭐ Star this repository if you find it useful!

πŸš€ Every claim in this README is measured β€” see docs/VERIFICATION.md and the verification-results branch.

Note: this experiment shows that fine-tuning has no measurable effect on very small models.

You can try training or fine-tuning on larger models if you are interested.

🌟 STAR IT IF YOU FIND THIS USEFUL

About

A bounded memory layer for language models, with exact recall for retained token IDs. NumPy core; optional PyTorch adapter.

Topics

Resources

Code of conduct

Contributing

Security policy

Stars

5 stars

Watchers

1 watching

Forks

Releases

Packages

Used by

Contributors

Languages