A tour of how dfkv is put together: the layers a request flows through, the pluggable storage engines, the optional RAM hot tier, and the wire protocol. For deployment see DEPLOY.md; for metrics see METRICS.md; for engine connectors and the client-side env/config reference see CONNECTORS.md.
Client vs server config: the storage engine (§5) and RAM tier (§6) are server-side only. Clients must supply an explicit namespace and object key; the supported connectors derive both from runtime model/layout metadata. See CONNECTORS.md §1.
LLM inference engine (SGLang HiCache / LMCache / vLLM)
│ thin adapter (ctypes / KVConnector)
▼
┌──────────────────────────────────────────────────────────────┐
│ libdfkv client — namespace + object key→128-bit id, Ketama route │
│ opaque raw Put/Get/Exist/Remove + batch/SG APIs │
└───────────────┬──────────────────────────────┬───────────────┘
MDS discovery (etcd epoch) transport: TCP or RDMA (RC)
│ │ versioned wire frame
▼ ▼
┌──────────────────────┐ ┌──────────────────────────────────────┐
│ dfkv_mds (+ etcd) │ │ dfkv_server (cache node) │
│ membership directory │ │ ┌────────────────────────────────┐ │
└──────────────────────┘ │ │ RamTier (opt) — registered RAM │ │
│ │ promotion + zero-copy RDMA GET │ │
│ └───────────────┬────────────────┘ │
│ miss ▼ │ flush (write-back) │
│ ┌──────────────────────────────────┐│
│ │ DiskCacheGroup — Ketama over N ││
│ │ disks, each a StoreEngine: ││
│ │ file (KVStore) | slab ││
│ └──────────────────────────────────┘│
└──────────────────────────────────────┘
Distribution is client-side consistent hashing; there is no replication (regenerable KV → a node loss is a miss → recompute). Membership is dynamic via the MDS tier — see the README for the two-layer offline detection.
PUT client.Put(key, value)
- The connector supplies the handle's namespace and a canonical object key. The client derives the native block identity, routes it by Ketama, and sends the caller's bytes unchanged.
- Server
ProcessRequest(kCache)/CacheDirectstores the exact raw bytes and their length. If the RAM tier is on, admission makes the value visible in RAM, but the request does not returnkOkuntil its disk flush commits. An overlapping same-key PUT joins that commit result rather than reporting uncommitted success. Genuine RAM capacity backpressure uses the synchronous disk path.
GET client.Get(key, dst)
- The same namespace and object key select the same block. The server checks RAM first, then disk; RDMA writes the raw payload directly into the caller's registered buffer.
- The stored length is returned separately.
GetAutoaccepts any destination capacity large enough for that length; dfkv does not prepend or validate a value envelope. - Every backend has one range contract:
length=0means the complete remainder;offset == stored_lengthsucceeds with zero bytes;offset > stored_lengthiskInvalid; an oversized length is clamped.value_lenalways reports the exact full stored payload on success.
The native C ABI has one versioned constructor:
dfkv_open_v2(const dfkv_client_options_v2*). The size-delimited descriptor
owns static members or MDS discovery/registration settings, immutable binary
namespace bytes, and batch concurrency. There are no post-open membership
mutators and no model-hash or geometry arguments.
Every scalar and batch/SG operation passes object keys as (const void*, uint64_t)
binary spans (batch APIs add a parallel key_lens[] array); embedded NUL bytes
are identity bytes, not terminators. Null, zero-length, or unrepresentable key
spans fail closed. PUT values must also be non-empty: scalar, batch, SG, TCP,
RDMA, RAM-tier, and persistent-store admission all reject zero-byte objects.
Every exported extern C function is a no-throw boundary: output slots are
initialized to failure and C++ exceptions translate to the documented result.
MDS group and registration client IDs are validated synchronously by
dfkv_open_v2 before either background worker starts.
Connectors construct one automatic cross-language NamespaceDescriptor
encoding beginning with b"DFKVNS\x00\x02". Length-framed fields bind tenant,
exact model ID and revision, connector raw-layout ID, dtype, layout fingerprint,
block-token and layer geometry, group, and replicated topology sizes.
Connector-specific shape/stride fields feed the deterministic 64-bit layout
fingerprint.
The source-controlled layout IDs are sglang-hicache/raw-v1, vllm-multiwr-v4,
and lmcache/raw-v1. Model/revision strings are preserved verbatim. Operators
cannot alias namespaces through configuration: an identity-bearing schema
change requires code review and a layout-ID bump, which deliberately starts a
cold cache. Python and C++ serializers share a checked golden vector.
Every connector renders an object key as:
dfkv/pool/v2|pool=...|hash=...|dp=S:R|tp=S:R|pcp=S:R|dcp=S:R|pp=S:R|group=N|component=...
Scatter-grouped objects append |sg=W:G. Both size and rank are identity;
R=-1 means replicated. Object identity is SHA-256 over the length-framed
namespace and object-key bytes, truncated to 128 bits. BlockKey also carries a
stable 64-bit tenant hash. For canonical DFKVNS\0\2, tenant identity is the
first length-framed field; malformed or other namespace forms use the complete
namespace bytes. The hash is the first eight SHA-256 bytes (big-endian) of
"DFKVTENANT1" || u64le(identity_length) || identity. C++ and Python share
golden vectors.
Identity ownership is strict:
- namespace: exact model/deployment identity plus a deployment-wide raw layout/schema version; connector defaults include their raw-layout ID
- object key: pool, full content hash, parallel coordinates, cache group, component, and optional scatter width/group
- control metadata: members, MDS endpoints/group, transport, library path, telemetry, and capacity knobs; none changes cache identity
- raw payload: exactly the caller-provided bytes; dfkv stores no envelope, geometry, dtype, checksum, or model tag in the value
Different namespace or key bytes produce a cold miss. Reusing the same namespace and key for a different byte layout is a type-safety violation, not a guarded miss: encode every identity-bearing geometry/layout change in the namespace or object key and bump the connector's source-controlled layout ID. This is a clean cutover: native clients do not read old keys or dual-write the retired identity/value format.
src/transport/wire.h defines two deliberately distinct transports:
| path | request/control frame | response/control frame | payload |
|---|---|---|---|
| native TCP | 50-byte fixed prefix + inline payload | 18-byte stored-length prefix + inline payload | versioned stream |
| native RDMA v2 | 50-byte prefix; GET adds 4 bytes + 16 bytes per destination | 18-byte stored-length prefix; Members allows at most 32 KiB data |
PUT WRITE_WITH_IMM into a leased server slot; GET server RDMA_WRITE into client MRs |
Both prefixes start with an explicit 1-byte protocol epoch: TCP accepts epoch 6
and native RDMA v2 accepts epoch 7. An unknown or unexpected epoch fails fast
instead of being mis-parsed. RDMA v2 is negotiated during the TCP bootstrap
before QPs exchange frames. Replies remain strict FIFO on each RC QP. RDMA
control buffers have an explicit 18-byte prefix + 32-KiB response capacity;
kMembers uses the control lane and an oversized declaration is rejected before
response copy/post rather than truncated or allowed to resize a receive buffer.
Production discovery uses MDS.
RdmaServer::Startcommits one aligned receive chunk. Additional chunks are committed on allocation misses up toDFKV_RDMA_RECV_SEGMENT_SIZE. Resident receive slots pin their chunk through endpoint lifetime; operation leases use exact MRs rather than retaining growth-chunk MRs in endpoint caches.DFKV_RDMA_MAX_BLOCK_BYTESremains the logical safety ceiling for every API. Inline operations select a power-of-two resident class. Leased PUT excludes oversized objects from that resident geometry; dynamic direct GET uses the minimum class independently of its operation's logical capacity.- The client selects a second power-of-two class from the operation's requested window: scalar QPs open at depth 1, while batches reuse/open the smallest sufficient depth up to the client/server ceiling.
- Optional PUT and dynamic-pull capabilities are negotiated before geometry selection and cached by peer identity/publication. Legacy peers retain receive plus pull slots and their 73-byte readiness frame. Dynamic peers omit the resident pull arena and use the 33-byte retirement readiness frame.
- A resident slot is
align4K(4096 + connection_class). Both resident slots and transient PUT/GET leases consume the same process hard budget. Empty non-initial chunks return afterDFKV_RDMA_RECV_CHUNK_IDLE_MS. - Successful PUT deregisters its exact remote-WRITE MR before recycling the range. Dynamic GET publishes an exact READ MR and waits for an explicit generation-checked RELEASE; the client sends RELEASE only after local READ completions and waits for ACK before idling. Failed connections destroy the QP and revoke MRs before RAII returns backing ranges and decrements gauges.
- Numeric rkeys can be recycled by providers after fresh registration; they are not permanent nonces. Correctness relies on completion/release ordering, generation checks for control messages, and endpoint fencing on failures.
- Client host/device pools are registered once per rail at declaration time.
Re-declaring the same base with a larger size registers the larger extent; the
registration call returns false/nonzero unless the full range is ready. Buffers
outside those pools use operation-scoped MRs and are deregistered only after
the corresponding completion; no cached MR may outlive caller-owned
std::string,CacheSrc, destination, or SG buffers. - Every multi-completion wait uses one absolute window deadline. A partial CQ
drain reduces the remaining budget and cannot restart it.
DFKV_RDMA_BATCH_OP_TIMEOUT_MSapplies to every multi-item method, including Cache/Range/Exist batches, zero-copy variants, and both SG variants; unset followsDFKV_RDMA_OP_TIMEOUT_MS. - With an empty device selector each side chooses its own first ACTIVE local
HCA, so host-local names may differ. An explicit comma list is a cross-host
whitelist and therefore requires the same names/fabric on both peers. A
device name is announced inside the fixed 32-byte v2 bootstrap frame
alongside the NUL terminator and the
"DCP2"/max_block_bytes/protocol capability trailer, so names longer than 18 bytes (32 − 1 − 4 − 8 − 1, seesrc/transport/dev_frame.h) are rejected fail-fast at client whitelist and server anchor validation time rather than silently losing the trailer; the fix is a shorter udev/bond alias, not a longer name. - Rail health is host-local evidence only. Local device open, verbs/QP
transition, post, or CQ failures increment the selected rail's failure streak
and may quarantine that rail. TCP bootstrap, unreachable/incompatible peer,
and malformed peer frames return the admission credit and feed client
PeerHealth; they do not penalize an HCA shared with unrelated nodes. DFKV_RDMA_NUMA=1derives the calling thread's NUMA node for every admission. If any enabled configured rail has matching discovery metadata, only that stable-index subset competes for credits. Unknown caller topology or no local rail falls back to all enabled rails; an explicit device whitelist remains the authoritative candidate set. When every NUMA-local rail is inadmissible (quarantined or credit-bound), admission retries once across every enabled rail — locality is a preference, never an availability gate.
dfkv_rdma_v2_ready, receive-segment total/free bytes, registered-rail count,
opened connections, v2 PUT/GET WRITE counters, client rail-vs-endpoint failure
and quarantine counters, and bounded NUMA-fallback counters make these
invariants observable. See CONNECTORS.md §1.2.1 for capacity
arithmetic and DEPLOY.md §3 for rollout settings.
DiskCacheGroup routes each block to one NVMe disk (Ketama over the full
BlockKey.Filename()) and owns one StoreEngine per disk. With no option and
no DFKV_STORE_ENGINE, every DiskCacheGroup user (including dfkv_server)
selects slab. --store-engine=file / DFKV_STORE_ENGINE=file selects the
explicit diagnostic fallback; invalid slab capacity or persistent geometry
fails startup and never retries with file:
StoreEngine: Cache / CacheDirect[Batch] / Range[*] / Lookup / Remove + stats
├── slab → DiskSlabStore (production: extent pool + sparse slots.tbl)
└── file → KVStore (diagnostic fallback: one file per block)
Each configured disk is a fixed pool of pre-allocated extent files. A
media-agnostic SlabAllocator binds extents to a deterministic startup-built
size-class lattice and carves them into uniform slots:
- Dense resident metadata: one reusable
SlotMetaper live value; class and extent lists store integer indices, and the only owned key string lives in the key→slot map. There is no per-class key mirror or stale slot-state scan. - Variable-size workloads: aligned geometric classes keep bounded internal waste across partial chunks and mixed model layouts. Placement is independent of request/restore order. Rebalancing uses useful bytes and decayed read heat, not object count alone.
- Concurrent I/O: payload reads/writes run outside metadata locks while slot pins and deferred removal prevent reuse. Prepared disk reads return one move-only RAII lease; async completion, fallback, error, and teardown release descriptor + pin exactly once. Batched flushes group by disk and run participating NVMe devices concurrently.
- Commit and removal ordering: a reserved same-key PUT has one leader; scalar and batch followers wait for its payload+record result. REMOVE hides the key immediately and fences in-flight I/O, so a deferred write cannot republish it after REMOVE returns.
- Restart: each published slot has a CRC-protected 64-byte tenant-scoped
fixed-offset record in sparse
slots.tbl. Rebuild visits allocated sparse ranges withSEEK_DATA/SEEK_HOLEwhen available (bounded chunked fallback otherwise), parses chunks, validates all records, restores allocator state in one bulk transaction, and rebuilds per-tenant committed-byte usage—no per-object open and no per-record syscall. - Crash safety:
slab_statemarks the running epoch dirty before traffic and clean only after payload/table sync on graceful shutdown. An unclean epoch is cold-reset rather than risking stale resurrection. Torn/invalid records are rejected and cleared durably; metadata I/O failure fails the store closed. Format/geometry mismatches are refused without rewriting existing data.
Payload I/O uses resident extent descriptors and O_DIRECT by default; a filesystem that cannot support it reports and uses the bounded buffered fallback. The v3 tenant-scoped on-disk format is intentionally incompatible with earlier slab metadata, so upgrading a slab node requires a clean cache directory.
KVStore keeps one file per block under blocks/<bucket>/…, with O_DIRECT
read/write and a sharded CLOCK index. It remains useful for diagnosis or
rollback, but is not the daemon's production default: inode growth,
open-per-GET, tmp cleanup, and unlink/ENOSPC behavior make it unsuitable as the
main large-capacity path.
Its v3 path is strict lower-case hex:
blocks/<tenant-prefix>/<tenant-prefix>/<16-hex-tenant><32-hex-object>;
startup rejects old or malformed cache filenames rather than decoding them.
Rename-to-trash is part of the file engine's logical commit: rename failure
returns kIOError and leaves the index, global bytes, and tenant bytes intact.
Both Cache and CacheDirect make exactly one bounded force-evict-and-retry
attempt after ENOSPC.
DiskCacheGroup optionally loads immutable startup quotas from
DFKV_TENANT_QUOTAS_FILE, with DFKV_TENANT_DEFAULT_QUOTA_BYTES as the
unlisted-tenant limit (0/unset means unlimited). Quotas are deliberately
per cache node, not a synchronous cluster-global reservation protocol.
Admission uses 256 fixed tenant stripes. PUT/REMOVE for the same stripe
serialize; GET remains lock-free relative to quota. A batch acquires unique
stripes in ascending order, charges only new keys, and may reject one tenant
with kQuotaExceeded while admitting unrelated tenants. Locks remain held
through storage writes, so concurrent single/batch requests cannot oversubscribe.
Both engines account committed payload bytes through publish, failure rollback,
eviction, removal, and restart rebuild. A limited tenant bypasses the
asynchronous RAM write-back admission path so the PUT result reflects durable
quota admission immediately.
A COLD dfkv load is disk-bound (~480 MB/s O_DIRECT) and dominates PD-decode TTFT.
The RAM tier fronts the disk with a pre-registered RAM arena so a PD-warm GET is
served straight from RAM over RDMA — no open, no pread, no disk. Enabled with
DFKV_RAM_TIER=1 (DFKV_RAM_TIER_BYTES sizes the arena; default off).
PUT policy and acknowledgement: writeback (default) admits into a visible,
flush-pinned RAM slot and normally returns kOk before asynchronous disk flush.
At a positive dirty-byte watermark (80% by default), new keys are rejected
before admission with kCacheFull; that rejection never becomes a direct-disk
fallback. Zero watermark explicitly selects disk-ACK. Set
DFKV_PUT_ACK_MODE=disk for durable acknowledgement of every admitted write.
writearound writes straight to disk and populates RAM on a later whole-value
GET. Existing RAM owners resolve first, including cancellation and duplicate
completion. For a key without a RAM owner, the committed backing index is
checked before a new RAM generation is admitted: a retained disk value wins,
so an idempotent disk kOk cannot falsely certify different RAM bytes.
Foreground write-back PUTs also hold a key-striped mutex through admission and
any synchronous capacity-bypass write. This covers a disk owner that is not yet
visible in the committed index. The fixed 1,024 stripes require no per-request
allocation; unrelated keys can collide on a stripe. Sampled PUT latency includes
lock wait. Flush workers and REMOVE do not acquire these foreground locks, so
existing flush completion and cancellation can still unblock a waiting PUT.
State machine = allocator pin refcount. The slot lifecycle maps directly onto the allocator's pin count — this is why the allocator was built media-agnostic:
| state | pin holders | evictable? |
|---|---|---|
| RESERVED (cold-read I/O) | reservation pin | no |
| RAM_ONLY (flush pending) | flush-pin | no |
| in-flight (RDMA transfer reading arena) | transfer-pin | no |
| DURABLE & idle | none (refcount 0) | yes |
- reservation pin — hides a cold-read slot until disk completion commits it as durable; on success the same pin becomes the first transfer-pin.
- flush-pin — taken on write-back admission, released when the flush reaches disk or is canceled.
- transfer-pin — taken on
GetPrep; responder-WRITE replies release it at their signaled SEND fence. Dynamic pull holds it through the peer READ and revokes the exact READ MR before releasing the pin on PullRelease. REMOVE hides an entry immediately but cannot reuse its pinned allocation.
RDMA zero-copy serve: the arena is registered on each selected device's shared PD. Dynamic pull lends an exact, revocable READ capability for the requested resident bytes, avoiding a staging copy. Successful read accounting is committed at PullRelease; connection teardown first fences the endpoint, then aborts the pin without counting a successful read. The responder-WRITE path still sends arena bytes to advertised client targets and holds its source until the final status SEND completes. Reply buffers also remain owned until their SEND fence, but a consumed control/PUT receive is posted before its reply can reach the peer. If the next request arrives before a reply buffer is free, a FIFO retains its CQE and receive-buffer ownership (at most the negotiated depth, with no per-request allocation). Dispatch preserves receive order. Receive slots still used as outbound data sources retain their SEND fence. This avoids receiver-not-ready stalls without reusing an in-flight reply. With RAM tier off none of the arena path is wired.
Direct cold-read promotion: on a whole-value io_uring miss, the server
reserves the final arena slot before submitting disk I/O. O_DIRECT reads into
that registered slot, completion publishes it as born-durable, and the same pin
holds it through the first RDMA SEND. The request therefore performs no
staging-to-arena memcpy and no promotion flush. An identical in-flight
follower waits only for publication, takes its own transfer-pin, and sends from
the same slot instead of copying the leader payload into connection staging.
Partial reads, values larger than an arena extent, allocation pressure,
timeouts, failures, and same-thread pipelined duplicates retain the bounded
independent/staging fallback. Identity-checked slot handles and serialized key
lifecycles prevent a stale reservation or SEND from pinning a replacement slot.
Values no larger than one RAM extent live inside that registered arena and keep
the zero-copy path. Larger accepted values use a dedicated aligned allocation
charged to the same hard byte budget; they retain the same flush/durable/send-pin
lifecycle but copy through the connection's bounded registered buffer because
their address is not part of the arena MR. A large pointer is never paired with
the arena's memory registration.
The fixed arena and dedicated allocations partition DFKV_RAM_TIER_BYTES;
their resident sum cannot exceed that limit. The automatic dedicated reserve is
two extents. DFKV_RAM_TIER_LARGE_RESERVE_BYTES=0 gives all bytes to the arena;
set an explicit reserve when values larger than an extent are expected.
Backpressure, durability, and removal: if the arena fills with non-evictable slots, PUT admission declines and the caller takes one synchronous disk path; a cold GET declines its direct reservation and retains the bounded staging path. A queued REMOVE cancels the queue item; an active flush or hidden cold-read reservation is fenced before slot reuse, so no later completion can resurrect the key. Exhausting flush retries is terminal RAM/store health and removes readiness. Only a DURABLE idle slot may be capacity-evicted. Requested arena allocation or an unsupported NUMA mode rejects startup instead of silently running disk-only.
Observability: dfkv_ram_hit/miss/put/put_bypass/flushed/flush_dropped/ evictions_total, dfkv_ram_healthy, actual dfkv_ram_flush_threads,
object/backlog gauges, and explicit total/arena/dedicated budget/usage gauges
(emitted only when enabled) — see METRICS.md.
The production disk path is slab by default; RAM and RDMA remain explicit.
RDMA requires DFKV_RDMA=1; selected peers must negotiate the v2 one-sided
data plane or the connection fails.
| Feature | Enable with | Default | Notes |
|---|---|---|---|
| disk storage engine | `--store-engine=slab | file/DFKV_STORE_ENGINE` |
slab (all server/store construction paths) |
| RAM hot tier | DFKV_RAM_TIER=1 (+ DFKV_RAM_TIER_BYTES) |
off | write-back/write-around + direct cold-read promotion + RDMA zero-copy GET; requested allocation failure rejects startup |
| RAM PUT policy | `--ram-write-mode=writeback | writearound/DFKV_RAM_WRITE_MODE` |
writeback |
| RAM PUT acknowledgement | `DFKV_PUT_ACK_MODE=ram | disk` | ram for write-back; disk otherwise |
| RAM NUMA policy | `--ram-tier-numa=interleave | off/DFKV_RAM_TIER_NUMA` |
interleave |
| RAM flush workers | --ram-flush-threads / DFKV_RAM_FLUSH_THREADS |
4× disk count, cap 16 | actual count is at least shard count and is reported after adjustment |
| RAM dedicated-value reserve | DFKV_RAM_TIER_LARGE_RESERVE_BYTES |
two extents | partitions the hard total budget; 0 disables oversized residency |
| RAM tier lock shards | --ram-tier-shards / DFKV_RAM_TIER_SHARDS |
8 | 1-64; auto-halved while a shard would hold <32 extents |
| read convoy/direct promotion | DFKV_READ_COALESCE=1 |
off | collapses identical reads; with RAM + io_uring, whole-value disk misses read directly into a hidden registered arena reservation |
| RDMA transport | build -DDFKV_WITH_RDMA=ON, DFKV_RDMA=1 |
TCP | active-HCA discovery; DFKV_RDMA_DEV is an optional whitelist |
| RDMA v2 | DFKV_RDMA=1 |
TCP when RDMA was not requested | bounded control buffers + adaptive data-QP classes + lazily committed receive chunks |
| io_uring async GET | build -DDFKV_WITH_URING; DFKV_SERVER_URING=0 disables |
on when built, unavailable otherwise | RDMA v2 disk-read path |
| first-request absolute deadline | DFKV_TCP_FIRST_REQ_MS / DFKV_MDS_FIRST_REQ_MS / DFKV_METRICS_FIRST_REQ_MS |
30000 ms; 0 = off |
anchored at accept: the first complete frame/request line is due within this window, so a drip feeder cannot pin a handler thread; dfkv_server also publishes its resolved value as the dfkv_tcp_first_req_ms gauge |
| MDS legacy control-plane shim | DFKV_MDS_ACCEPT_LEGACY=1 |
off (strict epoch gate) | widens the MDS control-plane version gate by exactly one epoch so v1.x peers are served with legacy 42/10-byte framing during a mixed-generation migration; data-plane listeners stay strictly epoch 6/7; drop the env once dfkv_mds_legacy_frames_total drains to zero |
| MDS /readyz etcd probe debounce | DFKV_MDS_PROBE_CACHE_MS |
2500 ms (clamp 600000; 0 = per-request probe) |
TTL-debounced etcd reachability probe so kubelet scrape cadence cannot amplify into etcd read load during an incident |
src/common/ BlockKey, namespace codec, Status — portable shared types
src/transport/ wire.h (current TCP/RDMA v2 frames) · rdma_protocol / rdma_recv_segment ·
rdma_topology · tcp_transport · rdma_transport / rdma_verbs
src/cache/ store_engine.h (interface) · kv_store (file) ·
slab_allocator + disk_slab_store (slab) · ram_tier (RAM hot tier) ·
disk_cache_group (per-disk routing) · kv_node_server · rdma_server ·
dfkv_server_main
src/client/ kv_client + C ABI · key_map (128-bit native identity)
src/mds/ membership service + dfkv_mds
src/tools/ dfkvctl / dfkv_smoke / dfkv_bench