Support Worker-variant callbacks - #11589
Open
chrsmith wants to merge 3 commits into
Open
Conversation
Contributor
|
Claude finished @chrsmith's task in 8m 48s —— View job Reviewed 20 files, 3 findings. |
![github-actions[bot]](https://avatars.githubusercontent.com/in/15368?s=60&v=4){kind=small}
chrsmith
force-pushed
the
chrsmith/wc-support-worker-variant-callbacks
branch
from
August 14, 2026 23:13
9e161f5 to
5719f95
Compare
chrsmith
commented
Aug 17, 2026
chrsmith
added a commit
to temporalio/api
that referenced
this pull request
Aug 19, 2026
⚠️ This is to be merged into the `feature/worker-callbacks` branch, and not `main`. Only after the feature is complete will that branch be rebased and merged into `main`. --- This PR makes three changes, all so that resources spanwed from the invocation of a worker callback can be linked correctly. (1) Remove the `Link_NexusOperationCallback` variant with a more general `Link_Callback` proto Previously we were scoping the feature to only be applicable for SANO callbacks. But if we are going to support worker callbacks for any async operation, having a general link type (that uses the existing [Execution proto](https://github.com/temporalio/api/blob/0066de621239ca9ddc6c976e091e27a6bc474752/temporal/api/common/v1/message.proto#L73-L77)) will avoid needing to create additional link variants in the future. (2) Add a `callbackpb.CallbackInfo::request_id` field This type is used in the `Describe-` operations for standalone Activities and standalone Nexus operations. Without it, there would be no way to determine _which_ completion callback is being referred to. (Instead, we couldn't be any more accurate than to have the link point to "one of these N" callbacks.) (3) Add `workflowpb.CallbackInfo::{request_id, result}` The `workflowpb` namespace forked rather than embedded the `callbackpb.CallbackInfo` message. The changes here add the missing fields, so that `DescribeWorkflowExecution` can disambiguate callbacks as well. (In addition to carrying the result of those callbacks.) **Why?** With these changes, the server will be able to properly cross-link resources spawned from completion callbacks. On the Caller-side, any resources spawned from the completion callbacks would be available on the `commonpb.Callback::links` field. (*) ```graphql query GetSpawnedResourceLinks(workflowID: string { DescribeWorkflowExecution(workflowID) { completion_callbacks { callback { links } } } } ``` > (*) Only the resources _initially_ created from the worker callback invocation will be present. e.g. the Workflow that backs an asynchronous Nexus handler. It would not contain links for any subsequent resources created. On the Handler-side, a single `Link_Callback` would be supplied to the Nexus handler receiving the worker callback. (This would be in the form of a `nexuspb.Link`.) **Breaking changes** Yes, this PR contains breaking proto changes. However, in the context of a PR into a long-lived feature branch for an unshipped feature this is safe. (The protos haven't ever been persisted by a production service.) **Server PR** It isn't out yet, but will be stacked on top of this: temporalio/temporal#11589
chrsmith
added a commit
to temporalio/api
that referenced
this pull request
Aug 19, 2026
⚠️ This is to be merged into the `feature/worker-callbacks` branch, and not `main`. Only after the feature is complete will that branch be rebased and merged into `main`. --- This PR makes three changes, all so that resources spanwed from the invocation of a worker callback can be linked correctly. (1) Remove the `Link_NexusOperationCallback` variant with a more general `Link_Callback` proto Previously we were scoping the feature to only be applicable for SANO callbacks. But if we are going to support worker callbacks for any async operation, having a general link type (that uses the existing [Execution proto](https://github.com/temporalio/api/blob/0066de621239ca9ddc6c976e091e27a6bc474752/temporal/api/common/v1/message.proto#L73-L77)) will avoid needing to create additional link variants in the future. (2) Add a `callbackpb.CallbackInfo::request_id` field This type is used in the `Describe-` operations for standalone Activities and standalone Nexus operations. Without it, there would be no way to determine _which_ completion callback is being referred to. (Instead, we couldn't be any more accurate than to have the link point to "one of these N" callbacks.) (3) Add `workflowpb.CallbackInfo::{request_id, result}` The `workflowpb` namespace forked rather than embedded the `callbackpb.CallbackInfo` message. The changes here add the missing fields, so that `DescribeWorkflowExecution` can disambiguate callbacks as well. (In addition to carrying the result of those callbacks.) **Why?** With these changes, the server will be able to properly cross-link resources spawned from completion callbacks. On the Caller-side, any resources spawned from the completion callbacks would be available on the `commonpb.Callback::links` field. (*) ```graphql query GetSpawnedResourceLinks(workflowID: string { DescribeWorkflowExecution(workflowID) { completion_callbacks { callback { links } } } } ``` > (*) Only the resources _initially_ created from the worker callback invocation will be present. e.g. the Workflow that backs an asynchronous Nexus handler. It would not contain links for any subsequent resources created. On the Handler-side, a single `Link_Callback` would be supplied to the Nexus handler receiving the worker callback. (This would be in the form of a `nexuspb.Link`.) **Breaking changes** Yes, this PR contains breaking proto changes. However, in the context of a PR into a long-lived feature branch for an unshipped feature this is safe. (The protos haven't ever been persisted by a production service.) **Server PR** It isn't out yet, but will be stacked on top of this: temporalio/temporal#11589
bergundy
reviewed
Aug 20, 2026
chrsmith
force-pushed
the
chrsmith/wc-support-worker-variant-callbacks
branch
2 times, most recently
from
August 23, 2026 02:33
843d371 to
ed07d2d
Compare
chrsmith
force-pushed
the
chrsmith/wc-support-worker-variant-callbacks
branch
3 times, most recently
from
August 23, 2026 21:04
820cd76 to
a0a9d57
Compare
This was referenced Aug 23, 2026
chrsmith
force-pushed
the
chrsmith/wc-support-worker-variant-callbacks
branch
from
August 24, 2026 18:47
a0a9d57 to
6166f74
Compare
Contributor
Author
|
@bergundy PTAL. I took care of your suggestions, as well as addressed that TODO comment about enforcing an aggregate max size for all worker callback source contexts attached to an execution. I'll add support for enabling Worker-callbacks for other execution types (#11566 (comment)) and rebase this on top. So there will be a few more functional tests to sanity check other execution types work end-to-end with worker callbacks. |
chrsmith
force-pushed
the
chrsmith/wc-support-worker-variant-callbacks
branch
from
August 24, 2026 22:39
6b6d677 to
6acf89f
Compare
This was referenced Aug 24, 2026
bergundy
reviewed
Aug 26, 2026
bergundy
left a comment
bergundy
left a comment
Member
There was a problem hiding this comment.
I did not review the tests FTR.
chrsmith
force-pushed
the
chrsmith/wc-support-worker-variant-callbacks
branch
from
August 26, 2026 21:59
6acf89f to
18b6e0d
Compare
chrsmith
added a commit
to temporalio/api
that referenced
this pull request
Aug 29, 2026
⚠️ This is to be merged into the `feature/worker-callbacks` branch, and not `main`. Only after the feature is complete will that branch be rebased and merged into `main`. --- This PR makes three changes, all so that resources spanwed from the invocation of a worker callback can be linked correctly. (1) Remove the `Link_NexusOperationCallback` variant with a more general `Link_Callback` proto Previously we were scoping the feature to only be applicable for SANO callbacks. But if we are going to support worker callbacks for any async operation, having a general link type (that uses the existing [Execution proto](https://github.com/temporalio/api/blob/0066de621239ca9ddc6c976e091e27a6bc474752/temporal/api/common/v1/message.proto#L73-L77)) will avoid needing to create additional link variants in the future. (2) Add a `callbackpb.CallbackInfo::request_id` field This type is used in the `Describe-` operations for standalone Activities and standalone Nexus operations. Without it, there would be no way to determine _which_ completion callback is being referred to. (Instead, we couldn't be any more accurate than to have the link point to "one of these N" callbacks.) (3) Add `workflowpb.CallbackInfo::{request_id, result}` The `workflowpb` namespace forked rather than embedded the `callbackpb.CallbackInfo` message. The changes here add the missing fields, so that `DescribeWorkflowExecution` can disambiguate callbacks as well. (In addition to carrying the result of those callbacks.) **Why?** With these changes, the server will be able to properly cross-link resources spawned from completion callbacks. On the Caller-side, any resources spawned from the completion callbacks would be available on the `commonpb.Callback::links` field. (*) ```graphql query GetSpawnedResourceLinks(workflowID: string { DescribeWorkflowExecution(workflowID) { completion_callbacks { callback { links } } } } ``` > (*) Only the resources _initially_ created from the worker callback invocation will be present. e.g. the Workflow that backs an asynchronous Nexus handler. It would not contain links for any subsequent resources created. On the Handler-side, a single `Link_Callback` would be supplied to the Nexus handler receiving the worker callback. (This would be in the form of a `nexuspb.Link`.) **Breaking changes** Yes, this PR contains breaking proto changes. However, in the context of a PR into a long-lived feature branch for an unshipped feature this is safe. (The protos haven't ever been persisted by a production service.) **Server PR** It isn't out yet, but will be stacked on top of this: temporalio/temporal#11589
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
feature/worker-callbacks. This will not go directly intomain, until the overall feature is code complete.What changed?
THIS IS IT! The actual PR that implements Worker callbacks!
This PR provides the implementation of the Worker-variant callback in the CHASM Callback component. A Worker-variant completion callback attached to a Workflow, Workflow Update, standalone Activity, or standalone Nexus Operation will result in a Nexus operation being invoked within the same namespace.
Additional changes/refactorings
The
CallbackInfo.BlockedReasonis now properly set. Previously it was left unimplemented in CHASM, and only callbacks attached to Workflows would report their status. We now wire through adestinationBlocked DestinationBlockedFnwith the CHASM context for Callbacks, and inject it viaCallbackDestinationBlockedProviderinservice/history/fx.go.Quirks/issues
We deliver all completion callbacks using the same request ID. This means however, that if multiple completion callbacks to the same (task queue, service, operation) were added in the same request (e.g. to
StartNexusOperationExecution.) then all of the worker callback invocations would carry the same request ID. The Nexus handler could interpret subsequent calls as duplicates and ignore them.This is something we can fix, by persisting a little more information in
CallbackStatebut it didn't seem like a realistic scenario; attaching multiple worker callbacks, in the same request, to the same taskqueue/operation.Why?
This is the new worker callbacks feature that will make developing "Nexus Connectors" (AKA fronting a Nexus service by a different API protocol) easier.
How did you test it?
Potential risks
While behind configuration flags, it's possible to attach a Worker-variant callback to Workflows, Workflow Updates, standalone Activities, and standalone Nexus Operations. In all cases they should behave the exact same. But there is opportunity for some subtle difference to cause problems.
NOTE: Worker-variant callbacks are NOT supported for HSM-backed workflows. Attaching a Worker callback to an HSM workflow will fail in glorious ways.