feat: FileRegistry and Streaming File Upload Architecture

[AndriiS-DevBrother]

Streaming File Upload & Distributed Data Availability

This PR implements the complete Streaming File Upload feature for F1R3FLY — enabling native support for large binary file handling (10GB+) directly on the blockchain, without inflating the LMDB state database.

Dependency

• Python SDK: F1R3FLY-io/pyf1r3fly#4 — client-side uploadFile and downloadFile gRPC streaming implementation

Integration Tests

• System Integration: F1R3FLY-io/system-integration#36 — end-to-end integration tests for the streaming file upload and download flow

What Was Added

1. gRPC Streaming Upload (uploadFile)

Client-streaming gRPC endpoint that accepts multi-gigabyte files in 1–4 MB chunks. Files are written directly to disk via FileChannel with direct ByteBuffer allocation (O(1) memory, zero JVM heap pressure). A Blake2b-256 hash is computed incrementally during ingestion and verified at EOF with an atomic .tmp → <hash> rename. Interrupted uploads leave no trace — the .tmp file is immediately cleaned up.

2. Automated Synthetic Deploy

After file ingestion, the node automatically constructs and submits a blockchain transaction (file!("register", ...) on the rho:io:file system channel) linking the file to the chain. The client makes ONE call — if uploadFile succeeds, the file is on disk AND the deploy is in the mempool. The response includes a deployId for lifecycle tracking via existing findDeploy/isFinalized RPCs.

3. FileRegistry.rho — On-Chain Ownership & Deduplication

A Rholang smart contract (modeled after SystemVault.rho) that maintains a per-file registry map (fileHash → { deployers: {pk1: true, ...}, name: "file.ext" }). Provides:

• Zero-cost deduplication — duplicate uploads skip physical storage but add the new deployer to the reference-counted ownership map
• AuthKey-gated deletion — only file owners can delete; physical file removal happens only when the last reference is removed
• Self-initializing genesis — no genesis ceremony dependency

4. P2P Validator File Sync (Layer 2)

Pull-based file replication over the existing TransportLayer. When a validator receives a block referencing files it does not have, it fetches them from the block proposer in 16 MB streaming chunks with Blake2b integrity verification. Hash mismatch → block rejected.

5. gRPC Streaming Download (downloadFile)

Server-streaming gRPC endpoint (observer-only in production, validator-allowed in devMode). Downloads are gated by finalization — the handler queries the FileRegistry at the Last Finalized Block post-state via exploratoryDeploy before serving bytes. Includes resume support via offset field.

6. DA-Optimistic Consensus (Layer 4)

Casper block validation is gated by data availability — validators do not build on blocks until all referenced files are downloaded and verified. Finalization naturally implies DA across participating validators. Includes configurable backpressure: max-file-data-size-per-block (50 GB) and max-file-deploys-per-block (10).

7. Storage-Proportional Phlo Pricing (Layer 5)

File uploads are charged BASE_REGISTER_PHLO + fileSize × phloPerStorageByte with enforcement at both upload-time (early rejection) and block-execution time (via CostAccounting.charge()). Downloads are free.

8. Orphan File Cleanup

When a file-registration deploy expires or is evicted from the mempool, the associated physical file is automatically deleted. The deployLifespan was increased from 50 to 250 blocks to prevent premature cleanup during large P2P replication.

How It Works (High Level)

Client                          Node                         Validators
  │                               │                               │
  ├── uploadFile(stream) ───────► │ Write to disk (O(1) mem)      │
  │   metadata + chunks           │ Blake2b hash verification     │
  │                               │ Atomic .tmp → <hash> rename   │
  │                               │ Auto-create synthetic deploy  │
  │  ◄── FileUploadResponse ────--┤ Push deploy to mempool        │
  │   (fileHash, deployId)        │                               │
  │                               │── Propose block ─────────────►│
  │                               │                               │ Fetch missing files (P2P)
  │                               │                               │ Verify Blake2b hash
  │                               │                               │ Execute & validate block
  │                               │                               │ Build on block (DA-gated)
  │── findDeploy(deployId) ─────► │                               │
  │── isFinalized(blockHash) ───► │                               │
  │                               │                               │
  │── downloadFile(hash) ────────►│ (observer-only)               │
  │  ◄── stream file chunks ─────-┤ Finalization-gated            │

Architecture Documentation

For the full five-layer architecture specification, see streaming_file_upload.md

[spreston8]

Initial Code Review — PR #407

This is a first-pass review covering the Scala infrastructure, P2P protocol, gRPC layer, and integration points. A deeper review of the Rholang contracts (FileRegistry.rho, system process interactions, and consensus replay correctness) will follow separately.

Overall the architecture is well-designed — the five-layer separation, content-addressed storage, and the SystemVault.rho-style AuthKey pattern are solid foundations. The issues below are things that should be addressed before merge.

---

1. Path traversal in FileRequester — remote peer can read/write arbitrary files

FileRequester.scala:321 — dataDir.resolve(msg.fileHash) uses the peer-supplied fileHash with no validation. A malicious peer can send fileHash = "../../.rnode/node.key.pem" to exfiltrate the node's private key via handleFileRequest, or write to arbitrary paths via handleFilePacket (lines 43, 56, 63, 191).

FileDownloadAPI already has both regex validation (^[a-f0-9]{64}$, line 29) and normalize().startsWith() defense-in-depth (line 97). FileRequester needs the same treatment on all entry points.

Similarly, SystemProcesses.scala:870 (fileDelete) does dir.resolve(fileHash) with no validation on the Rholang-supplied hash.

2. _deleteTemplate sends to wrong channel — physical file deletion is broken

FileRegistry.rho:252 — fileProc(`rho:io:file`) resolves to FILE_IO (arity 5, wired to fileRegister). The fileDelete handler lives on rho:io:file:delete / FILE_IO_DELETE (arity 4), as wired in RhoRuntime.scala:544-551. The delete message will never match the register handler's pattern and will block forever.

Tests pass because: (a) the Rholang delete test deliberately uses two deployers so remainingDeployers > 0 and never enters the physical-delete branch, and (b) FileSystemProcessSpec sends directly to FILE_IO_DELETE, bypassing FileRegistry.rho entirely.

Fix: change line 252 to fileProc(`rho:io:file:delete`) and add a test that exercises last-deployer deletion through the contract.

3. FILE_REGISTER missing from nonDeterministicCalls — replay divergence

SystemProcesses.scala:210-218 — FILE_DELETE (31L) is correctly listed as non-deterministic with a replay branch (line 847-853). FILE_REGISTER (30L) is not listed, and the fileRegister handler has no replay branch. However, it produces side-effect data to FILE_REGISTRY_NOTIFY (line 812). During block replay on other validators, the handler re-executes and double-produces to that channel, breaking consensus.

Fix: add FILE_REGISTER to nonDeterministicCalls and add a case isContractCall(produce, true, previousOutput, ...) replay branch.

4. Signature verification bypassed — any deploy can register phantom files

SystemProcesses.scala:781 — val verified = true is hardcoded. The rho:io:file channel is exposed as a write-only bundle via the URN map, so any user deploy can call file!("register", "nonexistent_hash", 1, "fake.txt", *ack). The handler constructs a real GSysAuthToken (line 800-802) and produces to FILE_REGISTRY_NOTIFY, which FileRegistry.rho consumes and creates a real on-chain entry. The attacker pays phlo but pollutes the registry with hashes that don't exist on any node's disk, triggering phantom P2P fetch attempts across the network.

5. MissingFileData blocks are permanently discarded instead of held for retry

MultiParentCasperImpl.scala:665-670 — MissingFileData falls through to the generic InvalidBlock handler which calls CasperBufferStorage.remove(). The comment at BlockStatus.scala:94-97 correctly notes this shouldn't be slashable because files may still be in transit, but the block is still deleted rather than held. If P2P transfer completes later (which can take hours per the design doc), the block is already gone.

Fix: MissingFileData should return the block to a pending state for retry, not discard it.

6. No retry on P2P download hash mismatch

FileRequester.scala:309-316 — when finalizeDownload detects a hash mismatch, it deletes the temp file and removes download state but does not re-request from another peer. Combined with #5, a single corrupted transfer can permanently prevent a node from validating a block.

Fix: on mismatch, reset download state and re-request from a different peer. Consider negative peer scoring (the TODO at line 310-311).

7. Peer-controlled OOM in handleFileRequest

FileRequester.scala:333 — ByteBuffer.allocate(msg.chunkSize) uses the peer-supplied chunkSize directly. A malicious peer can send chunkSize = Int.MaxValue to trigger a 2GB allocation.

Fix: clamp to local configured max, e.g. math.min(msg.chunkSize, this.chunkSize).

8. FileChannel resource leak in handleFileRequest

FileRequester.scala:331-337 — FileChannel.open() and channel.close() are not in try-finally. An IOException during channel.read(buf) leaks a file descriptor.

9. TOCTOU race in handleHasFile

FileRequester.scala:168-184 — three separate Ref.get calls plus a filesystem check, then conditionally calls startDownload. startDownload (line 193) does downloads.update(_ + ...) which unconditionally overwrites, potentially resetting an in-progress download. requestFiles (line 60-68) correctly uses atomic downloads.modify — handleHasFile should use the same pattern.

10. DA-gate broadcasts FileRequest to ALL connected peers

FileReplicationSetup.scala:57 — peers.toList.traverse_ sends to every peer. All peers with the file respond with full chunk streams, but only the fastest peer's chunks are accepted (offset matching). For a 6GB file with 10 peers, ~90% of P2P bandwidth is wasted.

Consider sending only to the block proposer (who is guaranteed to have the file) plus 1-2 fallback peers.

11. _sysAuthTokenCh unbounded tuplespace accumulation

FileRegistry.rho:150 — every registerNotify call unconditionally produces to _sysAuthTokenCh. Rholang channels are multiset queues, not registers. After N file registrations, N copies of the token accumulate. _deleteTemplate reads with <<- (peek) so correctness is preserved, but tuplespace storage grows linearly with file count.

Fix: produce only once using a flag channel, or use consume-then-reproduce pattern.

12. registerNotify silently discards errors

FileRegistry.rho:148-152 — regAckCh is created but never consumed. If FileRegistry!("register", ...) fails, the system process has already returned (true, fileHash) to the deploy caller, creating a disconnect between system process result and on-chain state.

13. System.out.println debug statements bypass logging framework

CasperLaunch.scala:111,185,224 — "sendBufferPendantsToCasper", "connectAsGenesisValidator", "initBootstrap" print directly to stdout, bypassing SLF4J. These should use Log[F].info(...) or be removed.

14. extractFileSize returns 0 on parse failure, bypassing block backpressure

OrphanFileCleanup.scala:57-58 — if the size field in a file-registration deploy is malformed, extractFileSize returns 0. This causes the deploy to bypass the per-block maxFileDataSizePerBlock limit in BlockCreator.

Fix: return Option[Long] and exclude unrecognizable deploys from file-deploy selection.

15. completed set in FileRequester never pruned

FileRequester.scala:36 — Set[String] grows monotonically with every downloaded file hash. Should be pruned after confirmation on disk or use a bounded structure.

16. FileAvailability.findMissingFiles does blocking I/O outside F

FileAvailability.scala:69-71 — pure function calls Files.exists() synchronously, returning List[String] not F[List[String]]. Blocks the calling fiber's thread.

17. Metadata sidecar race on concurrent dedup uploads

FileUploadAPI.scala:327-328 — two concurrent uploads of the same content both write .meta.json via Files.write(). Last writer wins with potentially incorrect fileName.

18. deployLifespan = 250 hardcoded

MultiParentCasperImpl.scala:684 — compile-time constant coupled to network bandwidth assumptions. Should be configurable via CasperShardConf.

19. Minor cleanup

• SystemProcesses.scala:768 — blockInfo <- blockData.get result never used. Dead code from removed sig verification.
• FileSystemProcessSpec:44-46 — empty test body for "register with malformed hex signature". Remove or mark pending.

Co-Authored-By: Claude noreply@anthropic.com

[metaweta]

This is great, thanks! How many of these attacks apply to the code this PR was based on?

[NazarY-DevBrother]

Additional Review Findings (not covered by @spreston8's review)

Security / Bugs

1. DeployGrpcServiceV1.scala:uploadFile — On signature validation failure, cleanup deletes uploadDir.resolve(hash). If this was a deduplicated upload (file already existed before this request), the pre-existing file is deleted, breaking other clients who successfully uploaded the same content earlier.

2. FileDownloadAPI.scala — _maxIpEntries is a mutable var on a singleton object, overwritten on every streamFile call without synchronization. Two concurrent requests with different maxCacheEntries race on this write. Should be set once at initialization.

3. OrphanFileCleanup.scala:isFileRegistrationDeploy — Since this string-matching detection also drives DA-gating in MultiParentCasperImpl, a crafted regular deploy containing "rho:io:file", "register", and a 64-char hex string in a comment could force validators to stall for fileSyncTimeout (2h) waiting for a non-existent file.

4. FileRequester.scala:awaitFiles — Timeout cleanup races with finalizeDownload. Between deciding a hash is "still missing" and running downloads.modify, a concurrent fiber could complete the download. This can leave inconsistent state where the file exists on disk but is in neither downloads nor completed.

5. FileRequester.scala — Mutable Blake2bDigest object is shared across atomic downloads.modify boundary via shallow .copy(). If a retry or concurrent operation touches the old DownloadState reference after modify, the digest corrupts silently.

Edge Cases

6. FileUploadAPI.scala:validateMetadata — maxFileSize is checked only against the client-declared metadata.fileSize, not actual bytes received. A client can declare a valid size but stream more data.

7. FileUploadAPI.scala:finalizeUpload — Atomic rename (.tmp → final) and .meta.json write are two separate operations. JVM crash between them leaves a file without metadata sidecar, with no recovery path.

8. Costs.scala:fileStorageCost — Math.multiplyExact throws ArithmeticException on overflow but callers don't catch it. At high phloPerStorageByte or large files, this propagates as an unhandled exception through the evaluator.

9. FileReplicationSetup.scala:create — Falls back to relative path Paths.get("file-replication") when fileReplicationDir is None. Location depends on JVM working directory — unpredictable in production.

10. DeployGrpcServiceV1.scala:checkFileFinalized — FileRegistry contract address hardcoded as a string literal in exploratoryDeploy. If the address changes (upgrade, different shard), all downloads silently fail.

11. FileDownloadAPI.scala:doStream — Files.size(filePath) called twice (once for logging, once for metadata chunk). File could be deleted between calls by orphan cleanup. Should capture size once.

12. FileDownloadAPI.scala — No validation that offset < fileSize. An offset beyond EOF silently returns an empty data stream with a metadata chunk claiming the full file size.

13. DeployServiceV1.proto:FileUploadMetadata.term — Client provides raw Rholang term. If the signed term isn't the canonical rho:io:file registration, deploy enters mempool but never registers the file, creating an orphan with no cleanup trigger.

DRY

14. Hex regex "^[a-f0-9]{64}$" hardcoded in 3 places. toHex conversion duplicated in 3 places. Both should be shared.

15. File cleanup logic (deleteIfExists for data + .meta.json) copy-pasted in two error branches within DeployGrpcServiceV1.uploadFile.

16. FileReplicationSetup.create + destructure pattern repeated identically in 3 call sites (CasperLaunch, GenesisCeremonyMaster, Initializing).

Tests

17. FileDeploySelectionSpec:applyBackpressure — Uses takeWhile (stops at first over-budget), but production BlockCreator uses foldLeft (skips and continues). Divergent behavior not covered by tests.

Co-Authored-By: Claude noreply@anthropic.com