Skip to content

Always run chained callbacks on deferred's executor - #258

Open
DerGuteMoritz wants to merge 8 commits into
masterfrom
fix-151-deferred-executors
Open

Always run chained callbacks on deferred's executor#258
DerGuteMoritz wants to merge 8 commits into
masterfrom
fix-151-deferred-executors

Conversation

@DerGuteMoritz

@DerGuteMoritz DerGuteMoritz commented Jan 8, 2026

Copy link
Copy Markdown
Collaborator

Addresses #151 by implementing @ztellman's idea from #151 (comment). I decided to split this up into three main commits:

  1. Extract a helper function (execute-callback) for executing chained callbacks and use it in all places where we have to decide whether to invoke the callback directly or whether to hand it off to the executor.
  2. Modify chaining functions to always execute callbacks on a deferred's executor even when already realized
  3. Keep track of the current executor and make execute-callback execute callbacks directly when the given executor is the same as the current one.

I also tried to add tests for all of this. See individual commits for more details. The other commits are small collateral changes.

The only major missing piece is this:

add a mechanism in manifold.executor for wrapping a preexisting executor such that submitted Runnables are wrapped with thread-local handlers (ideally this will be idempotent and not double-wrap executors like instrumented-executor which already has the necessary stuff)

Not quite sure how to do this, yet. Maybe via proxy? Suggestions welcome.

UPDATE: Just pushed two more commits to provide an API for wrapping existing executors.

Copilot AI left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

This PR implements a significant improvement to Manifold's executor handling by ensuring that chained deferred callbacks are always executed on the deferred's executor, even when the deferred is already realized. This addresses issue #151 by preventing callbacks from being invoked directly on the calling thread when chaining operations on realized deferreds.

Changes:

  • Introduced execute-callback helper function to centralize callback execution logic with executor affinity checking
  • Added current-executor-thread-local tracking to detect when already running on the target executor and avoid unnecessary re-scheduling
  • Implemented wrap-executor API to wrap existing Java executors with Manifold's executor tracking capabilities

Reviewed changes

Copilot reviewed 6 out of 6 changed files in this pull request and generated 3 comments.

Show a summary per file
File Description
src/manifold/executor.clj Added current-executor tracking, wrap-executor function, and IExecutorTracking interface; refactored thread-factory and instrumented-executor to use new executor promise structure with :executor and :onto? keys
src/manifold/deferred.clj Added execute-callback function, executor inline accessor, and different-executor? helper; refactored catch' and finally' to always use executor for realized deferreds; updated unwrap' and unwrap to stop unwrapping at executor boundaries
src/manifold/stream.clj Changed dropping-stream and sliding-stream implementations from d/let-flow to d/chain' for proper executor handling
test/manifold/executor_test.clj Added comprehensive tests for wrap-executor functionality
test/manifold/deferred_test.clj Added extensive tests for executor behavior with all chaining functions (chain, chain', catch, catch', finally, finally') and executor affinity tracking
test/manifold/stream_test.clj Added await-drained helper and tests to verify stream behavior with the new executor changes

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

Comment thread src/manifold/executor.clj Outdated
Comment thread test/manifold/executor_test.clj Outdated
Comment thread test/manifold/executor_test.clj Outdated
@DerGuteMoritz
DerGuteMoritz force-pushed the fix-151-deferred-executors branch from fd65c00 to 7cd82a5 Compare January 11, 2026 16:08
Comment thread src/manifold/executor.clj
Comment thread src/manifold/executor.clj
Comment thread src/manifold/executor.clj
@DerGuteMoritz
DerGuteMoritz force-pushed the fix-151-deferred-executors branch from 787771e to 4049ad2 Compare January 20, 2026 09:47
@DerGuteMoritz
DerGuteMoritz requested a review from Copilot January 20, 2026 09:50
Comment thread src/manifold/deferred.clj Outdated
To that end, extract `execute-callback` helper which is used in all places where callbacks are
invoked. As a side-effect, all callback invocation sites now also catch and log errors in executor
callbacks (before, only invocations via `set-deferred` did so).
* For `chain[']`, change `unwrap[']` so that it also stops when reaching a
  realized deferred with an explicit executor.
* Adjust `catch[']` and `finally[']` accordingly.
* Fill in `executor` method for all `IDeferred` implementations
* Add tests
To that end, keep track of the current executor in a new thread local variable.
`test-window-streams` relied on the source being drained before invoking `stream->seq`. Due to
`dropping-stream` using `let-flow`, the timing has changed now that executors are used in all cases
so that this isn't always the case anymore. To fix this, explicitly await the channel to be drained
first.
Specifically, don't use `let-flow` to avoid switching between different executors. Also, use
`d/chain'` instead of `d/chain` since `try-put!` is guaranteed to return a deferred.
@DerGuteMoritz
DerGuteMoritz force-pushed the fix-151-deferred-executors branch from 4049ad2 to 35eb48a Compare January 20, 2026 09:52

Copilot AI left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

Copilot reviewed 6 out of 6 changed files in this pull request and generated 8 comments.


💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

Comment thread src/manifold/executor.clj
Comment thread src/manifold/executor.clj
Comment thread src/manifold/executor.clj
Comment on lines +75 to +77
| `executor` | a `java.util.concurrent.Executor` or a function which accepts a `java.util.concurrent.ThreadFactory` and returns an executor for it. |
| `thread-factory` | an optional `java.util.concurrent.ThreadFactory` that creates the executor's threads. When given, `executor` must be a function. |
| `onto?` | if true, all streams and deferred generated in the scope of this executor will also be 'on' this executor. |"

Copilot AI Jan 20, 2026

Copy link

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

When wrap-executor is called with a function as the first argument and a thread-factory option, the function is expected to accept a ThreadFactory and return an Executor. However, the docstring doesn't clearly explain this dual-mode behavior where the first parameter can be either an Executor instance or a function that creates one. This could be confusing to API consumers. Consider clarifying the documentation to explicitly state that when thread-factory is provided, the executor parameter must be a function of type (ThreadFactory) -> Executor.

Suggested change
| `executor` | a `java.util.concurrent.Executor` or a function which accepts a `java.util.concurrent.ThreadFactory` and returns an executor for it. |
| `thread-factory` | an optional `java.util.concurrent.ThreadFactory` that creates the executor's threads. When given, `executor` must be a function. |
| `onto?` | if true, all streams and deferred generated in the scope of this executor will also be 'on' this executor. |"
| `executor` | Either (1) a `java.util.concurrent.Executor` instance, or (2) a function of type `(java.util.concurrent.ThreadFactory -> java.util.concurrent.Executor)` that, given a `ThreadFactory`, constructs and returns the underlying executor. |
| `thread-factory` | An optional `java.util.concurrent.ThreadFactory` that creates the executor's threads. When this option is provided, `executor` **must** be a function as described above and will be invoked with a wrapped `ThreadFactory`. |
| `onto?` | If true, all streams and deferreds generated in the scope of this executor will also be 'on' this executor. |"

Copilot uses AI. Check for mistakes.
Comment thread src/manifold/deferred.clj
Comment thread src/manifold/executor.clj
Comment thread src/manifold/executor.clj
Comment thread src/manifold/deferred.clj
Comment thread test/manifold/stream_test.clj
Comment thread src/manifold/deferred.clj
"Recursively unwraps a deferred or deferrable until either 1) a non-deferred
value is reached, or 2) an unrealized deferrable is reached."
"Recursively unwraps a deferred or deferrable until either 1) a non-deferred value is reached, or 2)
an unrealized deferrable is reached, or 3) a realized deferrable with a different executor than

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggested change
an unrealized deferrable is reached, or 3) a realized deferrable with a different executor than
an unrealized deferrable is reached, or 3) a deferrable with a different executor than

Maybe I am missing something, but it seems to me this would also stop when reaching an unrealized deferrable with a different executor than the current one. Same for the corresponding commit message – which additionally has 'explicit' executor rather than 'different' executor.

Side question: is it manifold parlance that a deferred in error state is considered 'unrealized' here?

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants