Add integration test for invoker yield under memory pressure by tillrohrmann · Pull Request #405 · restatedev/e2e

tillrohrmann · 2026-03-25T09:43:09Z

Test fires 50 concurrent invocations that each generate 10x64KB of side-effect output against a tight invoker memory budget (1MiB pool, 256KiB per-invocation limit). This forces invocations to yield when they exceed their memory budget and resume when memory frees up. Verifies all invocations complete with correct results.

This is an integration test case for restatedev/restate#4426.

slinkydeveloper

looks good, two minor nits

src/main/kotlin/dev/restate/sdktesting/tests/InvokerYieldTest.kt

tillrohrmann · 2026-03-25T12:01:33Z

Thanks a lot for your review @slinkydeveloper. I've addressed your comments. Asking for another round of reviews because I added some more tests.

src/main/kotlin/dev/restate/sdktesting/tests/InvokerMemoryTest.kt

slinkydeveloper · 2026-03-25T12:35:19Z

src/main/kotlin/dev/restate/sdktesting/tests/InvokerMemoryTest.kt

+    private fun randomString(length: Int): String {
+      val chars = "ABCDEFGHIJKLMNOPQRSTUVWXYZabcdefghijklmnopqrstuvwxyz0123456789"
+      return String(CharArray(length) { chars[Random.nextInt(chars.length)] })
+    }


perhaps place this in the utils, will be handy to other tests as well

Ok, will do.

src/main/kotlin/dev/restate/sdktesting/tests/InvokerMemoryTest.kt

slinkydeveloper · 2026-03-25T12:53:12Z

src/main/kotlin/dev/restate/sdktesting/tests/InvokerMemoryTest.kt

+      }
+
+  @Test
+  @DisplayName("Virtual object invocations yield when state loading exceeds memory budget")


the yield terminology is not very clear to me,what is that? suspends? is this some new concept?

It's an internal concept of handing invocations back to the scheduler (basically freeing the current invoker slot and but be eligible for scheduling again.

gotcha, was trying to understand if that's something we need to describe users/expose

slinkydeveloper · 2026-03-25T12:54:34Z

src/main/kotlin/dev/restate/sdktesting/tests/InvokerMemoryTest.kt

+  }
+
+  @Test
+  @DisplayName("Invocation is paused when virtual object state exceeds per-invocation memory limit")


interesting behavior here... is this some new behavior we need to document, for which we need new troubleshootings/runbooks?

Also what will it be the cascading effect of this, when you have many virtual object invocations enqueued?

also, reading the test, my understanding is that state will be lazily loaded and then the invocation gets paused, right?

That you could even assert in theory (if it makes sense for the test), getting the journal and checking the entry types.

A paused vo invocation will block all subsequent invocation to this vo instance. So it should effectively stop the processing of the vo instance. It behaves similar to having an error that depletes the retry attempts.

The invocation won't be executed on the deployment since we are failing at reading the initial key value pairs to construct the start message.

the initial key value pairs to construct the start message.

Can this be solved by simple loading up to some point, and then not sending the other entries?

This can certainly be made more clever in a follow-up. But in the general case you might have the situation where you have single key-value pair that is larger than your configured per invocation memory budget (by default it's the max message size which should also be the max size of a key-value pair). At the latest point when you try to access it no matter whether its lazy or eager, the system won't be able to do it.

tillrohrmann

Thanks for the second round of reviews. I've left some answers to your questions and will address the other comments swfitly.

src/main/kotlin/dev/restate/sdktesting/tests/InvokerMemoryTest.kt

tillrohrmann · 2026-03-25T14:20:24Z

src/main/kotlin/dev/restate/sdktesting/tests/InvokerMemoryTest.kt

+    private fun randomString(length: Int): String {
+      val chars = "ABCDEFGHIJKLMNOPQRSTUVWXYZabcdefghijklmnopqrstuvwxyz0123456789"
+      return String(CharArray(length) { chars[Random.nextInt(chars.length)] })
+    }


Ok, will do.

tillrohrmann · 2026-03-25T14:21:04Z

src/main/kotlin/dev/restate/sdktesting/tests/InvokerMemoryTest.kt

+      }
+
+  @Test
+  @DisplayName("Virtual object invocations yield when state loading exceeds memory budget")


It's an internal concept of handing invocations back to the scheduler (basically freeing the current invoker slot and but be eligible for scheduling again.

src/main/kotlin/dev/restate/sdktesting/tests/InvokerMemoryTest.kt

tillrohrmann · 2026-03-25T14:24:12Z

src/main/kotlin/dev/restate/sdktesting/tests/InvokerMemoryTest.kt

+  }
+
+  @Test
+  @DisplayName("Invocation is paused when virtual object state exceeds per-invocation memory limit")


A paused vo invocation will block all subsequent invocation to this vo instance. So it should effectively stop the processing of the vo instance. It behaves similar to having an error that depletes the retry attempts.

The invocation won't be executed on the deployment since we are failing at reading the initial key value pairs to construct the start message.

Add InvokerMemoryTest with four test scenarios: 1. Side-effect pressure: 50 concurrent invocations each generating 10x64KiB of side-effect output against a 1MiB invoker memory pool with 256KiB per-invocation limit. Forces yield/resume cycles. 2. State loading pressure: 50 virtual objects each with 64KiB of state (2x32KiB entries) invoked concurrently. 50x64KiB = 3.2MiB exceeds the 1MiB pool, forcing yields during state loading. 3. Oversized run payload: Single invocation producing 512KiB run output exceeding the 256KiB per-invocation limit. Verifies the server pauses the invocation rather than looping forever. 4. Oversized state: Virtual object with 512KiB state injected via admin API, exceeding the 256KiB per-invocation limit. Verifies the server pauses when it cannot load the state. Configures explicit retry policy (pause on max attempts) so pause tests don't break if the default changes.

tillrohrmann requested a review from slinkydeveloper March 25, 2026 09:43

slinkydeveloper reviewed Mar 25, 2026

View reviewed changes

src/main/kotlin/dev/restate/sdktesting/tests/InvokerYieldTest.kt Outdated Show resolved Hide resolved

src/main/kotlin/dev/restate/sdktesting/tests/InvokerYieldTest.kt Outdated Show resolved Hide resolved

tillrohrmann force-pushed the invoker-memory-limit-test branch from fa7c797 to ad20e7e Compare March 25, 2026 12:00

tillrohrmann requested a review from slinkydeveloper March 25, 2026 12:01

tillrohrmann mentioned this pull request Mar 25, 2026

Invoker memory budget: bounded memory management for invoker data paths restatedev/restate#4426

Open

slinkydeveloper reviewed Mar 25, 2026

View reviewed changes

tillrohrmann commented Mar 25, 2026

View reviewed changes

tillrohrmann force-pushed the invoker-memory-limit-test branch from ad20e7e to 6c30ef6 Compare March 25, 2026 22:35

Conversation

tillrohrmann commented Mar 25, 2026

Uh oh!

slinkydeveloper left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

tillrohrmann commented Mar 25, 2026

Uh oh!

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

tillrohrmann left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants