fix(monitor): bound pending tasks, not just concurrent scans - #59
Merged
Merged
Conversation
build_snapshot admitted one asyncio Task per skill to gather() and bounded
only how many ran, via a semaphore inside the worker. Live Tasks therefore
scaled with the registry rather than with the concurrency limit.
Reproduced independently at the reporter's parameters and at the sweep's real
ceiling of --max-scans 12000:
n=512 gather-all tasks_alive=513 peak_active=8 707 KiB
n=512 worker-pool tasks_alive=9 peak_active=8 26 KiB
n=12000 gather-all tasks_alive=12001 peak_active=8 16,579 KiB
n=12000 worker-pool tasks_alive=9 peak_active=8 477 KiB
So ~16 MiB of Tasks to keep 8 of them busy. Not a correctness or security
problem -- the concurrency limit worked as documented, and no extra load ever
reached the registry -- but it is waste that grows with the registry, and the
registry is the thing that grows.
run_bounded inverts the structure: a fixed pool pulls from a queue, so live
Tasks equal the limit. Applied to all three sites (scan, escalate, enrich),
not just the one reported; the escalation and enrichment phases had the same
shape.
On failure the pool is cancelled rather than left running, so a sweep that is
already failing stops hitting the registry.
The tests were checked against the old implementation and fail there. That
matters here: peak concurrency was already correct before this change, so a
test asserting only the limit would have passed against the bug.
Reported by @Nievesjyl.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01DNoTXU8k3pfSBzR7aJubqL
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Fixes #57.
Confirmed
The report is accurate.
build_snapshotadmitted one Task per skill toasyncio.gatherand bounded only how many ran, via a semaphore inside the worker. Live Tasks therefore scaled with the registry rather than with the concurrency limit.Reproduced independently — without running the attached PoC — at the reporter's parameters and at the sweep's real ceiling of
--max-scans 12000:~16 MiB of Tasks to keep 8 of them busy. The reporter's counts match ours exactly.
Severity
Low. The concurrency limit worked as documented and no extra load ever reached the registry — the report doesn't claim otherwise. But it is waste that grows with the registry, and the registry is the thing that grows: 74k skills today, up from 66k when that code was written.
Fix
malwar.core.concurrency.run_boundedinverts the structure — a fixed pool pulls from a queue, so live Tasks equal the limit instead of the input size.Applied to all three sites, not just the one reported. The escalation and enrichment phases had the identical shape:
gather(*(_worker(m) for m in to_scan))gather(*(_escalate(s) for s in candidates))gather(*(_enrich(s) for s in flagged))On failure the pool is cancelled rather than left running, so a sweep that is already failing stops hitting the registry.
On the tests
They were run against the old implementation and fail there:
That check matters here. Peak concurrency was already correct before this change, so a test asserting only the limit would have passed against the bug — which is the failure mode that has bitten this repo twice recently, where a test encoded the same assumption as the code it guarded.
Also covered: every item runs exactly once, empty input, fewer items than workers,
concurrency=0making progress rather than deadlocking, and no workers surviving a failure.Thanks @Nievesjyl — clear report with reproducible numbers.
🤖 Generated with Claude Code
https://claude.ai/code/session_01DNoTXU8k3pfSBzR7aJubqL
Generated by Claude Code