Skip to content

Fix concurrent StreamablePipe writes - #889

Open
nalves599 wants to merge 1 commit into
Bogdanp:masterfrom
nalves599:fix/streamable-pipe-writes
Open

Fix concurrent StreamablePipe writes#889
nalves599 wants to merge 1 commit into
Bogdanp:masterfrom
nalves599:fix/streamable-pipe-writes

Conversation

@nalves599

Copy link
Copy Markdown

Summary

  • serialize StreamablePipe.write() calls with a per-pipe lock
  • recreate the lock after unpickling to support spawned worker processes
  • add regression coverage for concurrent writes and pickling

Connection.send_bytes() splits large frames into separate header and payload writes. Without serialization, concurrent writers sharing one StreamablePipe can interleave those writes and corrupt the parent log watcher stream.

Fixes #888.

Validation

  • focused StreamablePipe regression tests
  • flake8 dramatiq/compat.py tests/test_compat.py
  • five-writer, 128 KiB frame reproduction: all 500 frames are received with this patch

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

StreamablePipe corrupts concurrent large writes and can stall the log watcher

1 participant