Mmap allows to map whole files into addressable memory regions and even share that memory accross processes.
A solid mmap implementation would hence make any manual read/write buffer logic unneccessary and even allow to use the same mmap for any amount of reader processes running on the same machine.
Reads and writes can happen directly to the memory region (in form of a nodejs Buffer) without copying.
All paging of the memory is then fully offloaded to the OS kernel and large files (partitions) would be handled exactly like smaller files and be fully transparent to the library.
There are various mmap implementations for nodejs, many forked under the mmap-io name, with the most up to date one being https://www.npmjs.com/package/@riaskov/mmap-io
Unfortunately, there is no build for nodejs 24+
Also, writing to mmaped files has the issue that the file needs to be resized first and remapped. Hence the remapping needs to happen in chunks to be viably performant. This then leads to the file size no longer being a correct measure for the actual data inside and truncation to actual data size or some other system needs to be implemented to keep track.
Mmap allows to map whole files into addressable memory regions and even share that memory accross processes.
A solid mmap implementation would hence make any manual read/write buffer logic unneccessary and even allow to use the same mmap for any amount of reader processes running on the same machine.
Reads and writes can happen directly to the memory region (in form of a nodejs Buffer) without copying.
All paging of the memory is then fully offloaded to the OS kernel and large files (partitions) would be handled exactly like smaller files and be fully transparent to the library.
There are various mmap implementations for nodejs, many forked under the mmap-io name, with the most up to date one being https://www.npmjs.com/package/@riaskov/mmap-io
Unfortunately, there is no build for nodejs 24+
Also, writing to mmaped files has the issue that the file needs to be resized first and remapped. Hence the remapping needs to happen in chunks to be viably performant. This then leads to the file size no longer being a correct measure for the actual data inside and truncation to actual data size or some other system needs to be implemented to keep track.