feat(store): Add tiered cache (disk + mem) by Anton-Kalpakchiev · Pull Request #650 · uber/kraken

Anton-Kalpakchiev · 2026-07-29T19:31:52Z

Check #633 for more details.

Implement the handle returned by the tiered store's Create and Open methods. When the handle is initially opened, it may operate on the blob's corresponding `memory.File` in the mem cache. However, at some point after the blob is marked complete, the blob will get evicted to disk. `tiered.File` seemlessly handles this transition by opening the blob in the disk store and continuing to operate on it in there, ensuring store clients don't even notice the transition. Alternatively, if the memory store had no capacity and the blob was instead directly initialized on disk, `tiered.File` handles that case too.

Next commit will implement the store itself. The flusher's job is to asynchronously flush data from the memory store to disk. I tried implementing it in a way where the muteces contend as little as possible and we don't block store APIs on disk IO, but it became a little complex so I'm open on any feedback on how to simplify the flusher.

Now that the flusher is done, implementing the store APIs was much easier. I also changed the signature of the disk store's DeleteMetadata API to match that of memory store's. Also fixed a bug in the tiered.File where if we fallback to disk when creating a file (since mem cache is full of inevictable blobs), we'd get a nil pointer dereference and thus a panic. TODO - tests

The API interface is the same as disk.Store and memory.Store's scoping APIs. However, considering that tiered.Store maintains state in 2 different stores that is not fully synced, deciding if a blob is in or out of scope is more complex, thus a general function like `isOutOfScope` was not the cleanest way to implement scoping. - To make scoping the tiered store easier, I added the `Scoped(scope store.BlobScope)` API to the disk.Store and memory.Store. - Documented tiered.Store - Added an error check in tiered store's constructor to protect against RebootIncompleteBlobs being turned on for disk store (doing so would leak blobs on crash) - Added a comment about a theoretically-possible bug that I expect won't happen in prod, just in case. We fail-open from that bug if it does happen, so it shouldn't be dangeroud to roll out to prod. - Will add tests in next commit. - Will create a proper tiered.Config in the next commits that allows disabling the memory cache

thijmv · 2026-08-05T15:29:18Z

+
+// File represents an open handle to a blob in [Store], similar to how an [os.File] is an open file descriptor to a file on disk.
+// File, just like a Linux file with its page cache, continues to work seemlessly if the blob is initially in memory, but later flushed to disk and evicted from memory.
+type File struct {


Can we abstract the backing file to an interface, such that this struct holds only a single reference to the backing file? That would also greatly simplify the implementation of its functions, since nil checks between the mem and disk file are no longer needed.

If not, the construction currently assumes a mem xor a disk file is provided, but this is not enforced. Do you think we should do so?

thijmv · 2026-08-05T15:29:41Z

+	if diskF != nil {
+		f.once.Do(func() { f.diskF = diskF })
+	}


What is the reason for not just passing diskF when initialising the struct? Which concurrency are you accounting for?

Flushing is usually asynchronous. The flow is:

User creates a file in tiered store.

Tiered store thus creates it in memory and returns a tiered.File to the user, where the tiered.File has memF set and diskF as nil.

Once the file moves from mem to disk, tiered.File needs to do diskStore.Open() to get the diskF so the user that holds tiered.File can continue operating on the disk.File

thijmv · 2026-08-05T15:30:26Z

+// A blob might be evicted from the mem store (after being flushed to disk), while a tiered store user is holding a [File].
+// To allow the user to continue operating on the [File] seemlessly, we open the blob in the disk store and
+// operate on it instead. This is done once, on the first call to [File] after eviction from memory.
+func (f *File) openDiskFileIfNeeed() error {


Suggested change

func (f *File) openDiskFileIfNeeed() error {

func (f *File) openDiskFileIfNeeded() error {

thijmv · 2026-08-05T15:30:42Z

+func (f *File) openDiskFileIfNeeed() error {
+	f.once.Do(func() {
+		diskF, err := f.diskStore.Open(f.key)
+		if errors.Is(err, os.ErrNotExist) {


You stated that ErrEvicted may also be used for deleted blobs.

However, that means this branch is reachable when trying to operate on a File for a deleted blob, causing an incorrect invariant violation.

thijmv · 2026-08-05T15:31:44Z

+}
+
+func errBadSwitch(subErr error) error {
+	return fmt.Errorf("tiered.File could not switch over from pointing to memory.File to pointing to disk.File after blob was evicted from memory: %w", subErr)


nit: Consider shortening this error message and the three Error(...)s above.

Suggested change

return fmt.Errorf("tiered.File could not switch over from pointing to memory.File to pointing to disk.File after blob was evicted from memory: %w", subErr)

return fmt.Errorf("mem to disk switch failed after eviction: %w", subErr)

thijmv · 2026-08-05T15:33:21Z

+// Store is a tiered (disk + memory), thread-safe, LRU cache for blobs and their [metadata.Metadata].
+//
+//   - New blobs are initially created in the memory cache to speed up writes/reads and asynchronously flushed to disk
+//     once MarkComplete is called on them. If the memory cache is full with inevctable blobs, the new blob is created on disk.


Suggested change

// once MarkComplete is called on them. If the memory cache is full with inevctable blobs, the new blob is created on disk.

// once MarkComplete is called on them. If the memory cache is full with inevictable blobs, the new blob is created on disk.

thijmv · 2026-08-05T15:33:34Z

+//   - New blobs are initially created in the memory cache to speed up writes/reads and asynchronously flushed to disk
+//     once MarkComplete is called on them. If the memory cache is full with inevctable blobs, the new blob is created on disk.
+//
+//   - Partially crash-resistant - all complete blobs that were fully flushed to disk are persisted. Use [disk.Store] if you need full crash resistence.


Suggested change

// - Partially crash-resistant - all complete blobs that were fully flushed to disk are persisted. Use [disk.Store] if you need full crash resistence.

// - Partially crash-resistant - all complete blobs that were fully flushed to disk are persisted. Use [disk.Store] if you need full crash resistance.

thijmv · 2026-08-05T15:33:51Z

+// Has reports if the blob is 1) in the store and 2) in scope.
+// If you don't care about the blob's scope and just want to check membership, do:
+//
+//	_, ok := store.Has(key)


Suggested change

// _, ok := store.Has(key)

// inStore, _ := store.Has(key)

thijmv · 2026-08-05T15:36:17Z

+		return newFile(key, memF, nil, s.disk, s.log), nil
+	}
+
+	diskF, err := s.disk.Create(key, sizeBytes)


Can we add observability for falling back to disk because of memory.ErrNoSpace?

thijmv · 2026-08-05T15:49:06Z

+		return false, storelib.ErrOutOfScope
+	}
+	if err != nil && !errors.Is(err, os.ErrNotExist) {
+		return false, fmt.Errorf("mem store get metadarta: %w", err)


Suggested change

return false, fmt.Errorf("mem store get metadarta: %w", err)

return false, fmt.Errorf("mem store get metadata: %w", err)

github-actions Bot added the size/xl label Jul 29, 2026

Anton-Kalpakchiev force-pushed the lru-cache-tiered branch from 84eace5 to 6994d38 Compare July 29, 2026 19:34

Anton-Kalpakchiev added 3 commits July 31, 2026 20:00

thijmv reviewed Aug 5, 2026

View reviewed changes

test(tiered.Store): Add tests, add 2 sanity checks, and fix a typo

cb4e93b

Navigation Menu

Uh oh!

feat(store): Add tiered cache (disk + mem) - #650

feat(store): Add tiered cache (disk + mem)#650
Anton-Kalpakchiev wants to merge 5 commits into
lru-cache-memfrom
lru-cache-tiered

Anton-Kalpakchiev commented Jul 29, 2026

Uh oh!

thijmv Aug 5, 2026

Uh oh!

thijmv Aug 5, 2026

Uh oh!

Anton-Kalpakchiev Aug 7, 2026

Uh oh!

thijmv Aug 5, 2026

Uh oh!

thijmv Aug 5, 2026

Uh oh!

thijmv Aug 5, 2026

Uh oh!

thijmv Aug 5, 2026

Uh oh!

thijmv Aug 5, 2026

Uh oh!

thijmv Aug 5, 2026

Uh oh!

thijmv Aug 5, 2026

Uh oh!

thijmv Aug 5, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

	func (f *File) openDiskFileIfNeeed() error {
	func (f *File) openDiskFileIfNeeded() error {

	return fmt.Errorf("tiered.File could not switch over from pointing to memory.File to pointing to disk.File after blob was evicted from memory: %w", subErr)
	return fmt.Errorf("mem to disk switch failed after eviction: %w", subErr)

	// once MarkComplete is called on them. If the memory cache is full with inevctable blobs, the new blob is created on disk.
	// once MarkComplete is called on them. If the memory cache is full with inevictable blobs, the new blob is created on disk.

	// - Partially crash-resistant - all complete blobs that were fully flushed to disk are persisted. Use [disk.Store] if you need full crash resistence.
	// - Partially crash-resistant - all complete blobs that were fully flushed to disk are persisted. Use [disk.Store] if you need full crash resistance.

	return false, fmt.Errorf("mem store get metadarta: %w", err)
	return false, fmt.Errorf("mem store get metadata: %w", err)

Uh oh!

Conversation

Anton-Kalpakchiev commented Jul 29, 2026

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants