A replicated, chunk-distributed file system written from scratch in C++17. The server and client stand on POSIX sockets, threads, and the standard library alone: no storage engine, no RPC framework, no third-party libraries. The wire protocol is written by hand.
Clients connect over TCP and upload, download, list, and delete files. The coordinator splits each file into chunks, places every chunk independently on the least-loaded live storage nodes, replicates it, and transparently fails over to another replica when a node dies mid-transfer. Cluster state is reconstructed from the nodes themselves at startup, so there is no metadata journal to corrupt or replay.
About this fork. This started as a four-person honors project for CS 341 (Systems Programming) at UIUC. This fork is where I continue it independently. I wrote the distributed layer (where chunks get placed, how they are replicated, and how the system detects node failure and keeps serving through it), along with the storage-node service and the networking layer beneath it. My teammates' contributions are credited in Attribution.
┌─────────────────┐
│ client │ TUI or CLI
└────────┬────────┘
│ TCP
▼
╔═══════════════════════════════════════════════════════════╗
║ COORDINATOR ║
║ ║
║ file table placement heap ║
║ name → chunk layout nodes ordered by bytes stored ║
║ ║
║ thread per client · replication · failover · recovery ║
╚═══════════════════════════════════════════════════════════╝
│ │ │
│ persistent TCP, one connection per live node
▼ ▼ ▼
┌───────────┐ ┌───────────┐ ┌───────────┐
│ node 0 │ │ node 1 │ ··· │ node N-1 │
│ │ │ │ │ │
│ storage │ │ storage │ │ storage │
└───────────┘ └───────────┘ └───────────┘
Each node runs its own accept loop on its own port and owns its own storage directory, so a failed node genuinely stops answering rather than setting a flag. The node count is a build-time parameter, and the coordinator reaches nodes over ordinary TCP, so moving them onto separate machines is a change of address, not of design.
A file is never pinned to a node. It is split into chunks, and each chunk is placed independently on whichever nodes are least loaded at that moment, then replicated:
file.bin
│
└── split ──▶ c0 c1 c2 c3
│ │ │ │
every chunk placed independently, then replicated,
so one file's replicas spread across different pairs
node 0 node 1 node 2
┌─────────┐ ┌─────────┐ ┌─────────┐
│ c0 c2 │ │ c0 c1 │ │ c1 c2 │
│ c3 │ │ c3 │ │ │
└─────────┘ └─────────┘ └─────────┘
Placement is driven by a min-heap keyed on bytes stored, a std::set<pair<off_t,int>>, so the least-loaded node is begin() in O(1) and repositioning a node after a write is O(log N).
The important detail is when the accounting happens. Bytes are charged to a node at selection time, before the write completes, so concurrent uploads immediately see the updated load and route around each other instead of all stampeding the same node. A write that fails has its reservation rolled back.
Because placement happens per chunk rather than per file, a large file spreads across the whole cluster and its replica set differs from chunk to chunk.
Every chunk is written to several nodes, controlled by a replication factor that caps automatically at the number of live nodes. At the start of an upload the coordinator opens one connection to each live node and holds them all open for the duration, so no chunk pays a TCP handshake.
Replicas for a chunk are written in parallel by worker threads, so one slow node cannot hold up the others. A chunk enters the file table only after its replicas acknowledge; any replica that fails is rolled back out of the load accounting, its node is marked dead, and its connection is dropped for the remainder of the upload. When a chunk lands on fewer nodes than requested, the transfer continues at reduced redundancy and reports it, rather than discarding an otherwise good upload.
Failure handling operates at four levels:
During an operation. A node that fails to respond is added to a dead-node set and excluded from routing immediately. Send and receive timeouts are set explicitly on every node socket, because a node that stops accepting while its listen socket stays open would otherwise let connect() succeed and then stall the coordinator indefinitely.
In the background. A health-check thread re-probes dead nodes on an interval and returns them to the placement heap once they answer again, so a recovered node rejoins without restarting the cluster.
On download. Each chunk carries its replica list. The coordinator walks it, and when a node is unreachable it fails over to the next replica and keeps streaming. The client sees one uninterrupted byte stream and never learns a node died.
At startup. There is no metadata journal. The coordinator asks every node for its chunk inventory and rebuilds the file table (chunk indices, sizes, and replica sets) from what the nodes actually hold, which means metadata cannot drift out of sync with storage. Each node independently detects whether it died mid-write and discards the incomplete file before accepting connections, so a partially written chunk is never advertised as real.
A KILL_NODE command takes a node down on demand, which makes failover demonstrable live rather than only under test.
Each client is handled on its own thread and each storage node runs its own accept loop, so slow clients and slow nodes stay isolated from one another. Shared state (the file table, per-node byte counts, the placement heap, and the dead-node set) is guarded by a single mutex taken narrowly: the coordinator selects replicas and reserves bytes under the lock, then releases it before doing any network I/O. Console output has its own mutex so concurrent threads never interleave mid-line.
Details that took real debugging to get right:
- Concurrent uploads of the same filename are serialized through a pending-upload set, so two clients racing on one name cannot leave orphaned chunks behind.
- Aborted uploads drain the client's unsent payload, so the next command on that connection starts at a clean protocol boundary instead of parsing file bytes as a command.
- Filenames are validated against directory traversal before they ever reach a node.
SIGPIPEis ignored, so a write to a disconnected client returns an error instead of killing the server.- Nagle is disabled on node connections; otherwise small per-chunk acknowledgements get held by the receiver's delayed-ACK timer and uploads stall once kernel buffers fill.
Cluster shape is set in include/constants.hpp at compile time. Everything downstream derives from these values: array sizing, the node spawn loop, placement, replication, and startup recovery alike.
| Constant | Controls |
|---|---|
NUM_NODES |
How many storage nodes the coordinator spawns |
NUM_REDUNDANCIES |
Replicas per chunk. 1 stores a single copy; N stores every chunk on every node. Capped at the live node count |
CHUNK_SIZE |
Chunk granularity |
NODE_BASE_PORT |
Node n listens on NODE_BASE_PORT + n |
Chunk size is the one parameter with a genuine tradeoff behind it: smaller chunks distribute a file more finely and spread load more evenly, but multiply per-chunk round trips, and below a certain size the acknowledgement traffic begins interacting with receiver-side delayed ACK in ways that dominate transfer time.
Requirements: Linux (developed against a Raspberry Pi), a C++17 toolchain, and Python 3 with pip for the TUI.
make # builds bin/server and bin/client
make run-server # or ./bin/serverThe coordinator spawns the storage nodes itself; there is nothing else to start.
Client, from another machine or another shell:
python -m venv venv && source venv/bin/activate
make tui # install TUI dependencies
make run-tui # or: python3 src/dfs_tui.py [host] [port]With no arguments the TUI connects to localhost; pass a hostname or IP to reach a Pi on the network. A C++ CLI client is also available at ./bin/client <host> <port> with an interactive dfs> prompt.
Text commands, newline-terminated, with raw bytes following the header where applicable.
Client → Coordinator
UPLOAD <filename> <filesize>\n<filesize bytes>
DOWNLOAD <filename>\n
LIST\n
DELETE <filename>\n
STATUS\n
STATUS <filename>\n
KILL_NODE <node_id>\n
QUIT\n
Coordinator → Node
NODE_STORE <chunkname> <size>\n<raw bytes>
NODE_RETRIEVE <chunkname>\n
NODE_DELETE <chunkname>\n
NODE_LIST\n
NODE_STATUS\n
STATUS reports per-node chunk counts and byte totals. STATUS <filename> returns that file's chunk layout (index, size, and which nodes hold each replica), which is what drives the distribution view in the TUI.
tcpServer/server.cpp coordinator: routing, placement, replication, recovery
src/node_server.cpp storage node: TCP layer over local file storage
src/tcp_helpers.cpp reliable send/recv helpers
src/client.cpp interactive C++ CLI client
src/dfs_tui.py Textual TUI client
node/nodeinternal.cpp local filesystem layer used by each node
include/constants.hpp cluster configuration
testing/ Python integration tests
Who built what on the original CS 341 Honors project, Spring 2026:
Jacob Lopez. The distributed layer: chunk placement and load balancing, parallel replication over pooled connections, node-failure detection and download failover, plus the storage-node service, the networking layer, and the C++ client.
Joey Li. The original multi-client TCP file server, whose accept loop, command dispatch, and socket primitives still sit underneath; the Textual TUI; most of the integration tests.
Hayden Fountain. The local filesystem layer each node stands on, plus the error-detection metadata that makes mid-write crash cleanup possible and a fault-injection mode for testing.
Kyle Li. The chunk model: indexed splitting, the chunk naming convention, per-chunk replica tracking, the startup routine that rebuilds a file's layout from node inventories, and the chunk-distribution command.