Task dispatcher speaks the rhapsody execution dialect (pools phase 2) - #117
Merged
Conversation
New bulk submit route (`submit_rh/{sid}`): rhapsody-style task dicts in
rhapsody's own wire format -- cloudpickled fields ride as base64 strings
the dispatcher never decodes -- with a `pool` key per task. Mixed-pool
batches group server-side; the dicts forward verbatim to pilot rhapsody
sessions, uid-namespaced by the owning session because a pilot's session
is shared while client-side uid counters are not unique across clients.
`TaskDispatcherClient.submit_tasks` mirrors `RhapsodyClient.submit_tasks`
(same serialization, same frame-bounded batching), which is the verb
rhapsody's OrbitExecutionBackend calls -- pointing it at the dispatcher
now needs nothing but `plugin_name`.
The per-task fsync goes: one ledger persist per touched pool per submit
batch and per drain, and completion-side persists coalesce on the same
NOTIFY_WINDOW plugin_rhapsody uses. Dialect completions emit rhapsody-
shaped `task_status` / `task_status_batch` frames (uid, state, encoded
return_value forwarded from the pilot), so the backend's existing
notification consumer works unchanged; exec-style tasks keep their
classic immediate frames for the Explorer page.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_016Npyz3Hbnwos12ESsdJ2YU
`register_session` forwards `sid` / `lifetime` / `ttl` -- pools are keyed per session, so a second client (the workflow engine's execution backend) that should see a session's pools joins it by sid instead of minting its own. `pool_detail` and `endpoint_name` in pool summaries give such a client the executing endpoint, which is what the python-version handshake for cloudpickled tasks has to be resolved against. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_016Npyz3Hbnwos12ESsdJ2YU
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
The orbit half of digital.twins#7 phase 2 — pools behind the workflow engine's existing execution backend.
Bulk submit in rhapsody's wire format. New
submit_rh/{sid}route: rhapsody-style task dicts (cloudpickled fields ride as base64 strings the dispatcher never decodes) with apoolkey per task. Mixed-pool batches group server-side; dicts forward verbatim to pilot rhapsody sessions, uid-namespaced by the owning session (a pilot's rhapsody session is shared, client-side uid counters are not unique across clients).TaskDispatcherClient.submit_tasksmirrorsRhapsodyClient.submit_tasks— the verb rhapsody'sOrbitExecutionBackendcalls, so pointing that backend at the dispatcher needs nothing butplugin_name.The per-task fsync goes. One ledger persist per touched pool per submit batch and per drain; completion-side persists coalesce on the same NOTIFY_WINDOW plugin_rhapsody uses. Per the earlier latency analysis, this was the dominant cost of pool mode.
Batched notifications. Dialect completions emit rhapsody-shaped
task_status/task_status_batchframes (uid, state, encoded return_value forwarded from the pilot) under the dispatcher's namespace — the backend's existing notification consumer works unchanged. Exec-style tasks keep their classic immediate frames, so the Explorer page and existing consumers see no difference.Client plumbing for a second session member.
register_sessionforwardssid/lifetime/ttl(pools are per-session; an execution backend joins the owning session by sid),pool_detailandendpoint_namein pool summaries expose the executing endpoint for the cloudpickle python-version handshake.Unit tests: 7 new (grouped bulk submit + one-persist-per-pool, atomic validation, uid/pool requirement, cached-DONE resubmit, bulk namespaced pilot forward, batched terminal notifications with result forwarding, client pool-key guard). Full suite: 1048 passed.
Companion PRs to follow: rhapsody
OrbitExecutionBackendpool targeting, digital.twins pool wiring.🤖 Generated with Claude Code
https://claude.ai/code/session_016Npyz3Hbnwos12ESsdJ2YU