Skip to content

fix(monitor): bound pending tasks, not just concurrent scans - #59

Merged
Ap6pack merged 2 commits into
mainfrom
claude/fix-57-bounded-tasks
Sep 11, 2026
Merged

Ap6pack merged 2 commits into
mainfrom
claude/fix-57-bounded-tasks

Conversation

@Ap6pack

@Ap6pack Ap6pack commented Sep 11, 2026

Copy link
Copy Markdown
Owner

Fixes #57.

Confirmed

The report is accurate. build_snapshot admitted one Task per skill to asyncio.gather and bounded only how many ran, via a semaphore inside the worker. Live Tasks therefore scaled with the registry rather than with the concurrency limit.

Reproduced independently — without running the attached PoC — at the reporter's parameters and at the sweep's real ceiling of --max-scans 12000:

n=512    gather-all   tasks_alive=513    peak_active=8      707 KiB
n=512    worker-pool  tasks_alive=9      peak_active=8       26 KiB
n=12000  gather-all   tasks_alive=12001  peak_active=8   16,579 KiB
n=12000  worker-pool  tasks_alive=9      peak_active=8      477 KiB

~16 MiB of Tasks to keep 8 of them busy. The reporter's counts match ours exactly.

Severity

Low. The concurrency limit worked as documented and no extra load ever reached the registry — the report doesn't claim otherwise. But it is waste that grows with the registry, and the registry is the thing that grows: 74k skills today, up from 66k when that code was written.

Fix

malwar.core.concurrency.run_bounded inverts the structure — a fixed pool pulls from a queue, so live Tasks equal the limit instead of the input size.

Applied to all three sites, not just the one reported. The escalation and enrichment phases had the identical shape:

site was
Phase 1 scan gather(*(_worker(m) for m in to_scan))
Phase 2 escalate gather(*(_escalate(s) for s in candidates))
Phase 3 enrich gather(*(_enrich(s) for s in flagged))

On failure the pool is cancelled rather than left running, so a sweep that is already failing stops hitting the registry.

On the tests

They were run against the old implementation and fail there:

OLD code  n=12000  peak=8 (ok)  live=12001  -> FAILS
NEW code  n=12000  peak=8 (ok)  live=9      -> PASSES

That check matters here. Peak concurrency was already correct before this change, so a test asserting only the limit would have passed against the bug — which is the failure mode that has bitten this repo twice recently, where a test encoded the same assumption as the code it guarded.

Also covered: every item runs exactly once, empty input, fewer items than workers, concurrency=0 making progress rather than deadlocking, and no workers surviving a failure.

1727 passed, 16 skipped   (9 new)

Thanks @Nievesjyl — clear report with reproducible numbers.

🤖 Generated with Claude Code

https://claude.ai/code/session_01DNoTXU8k3pfSBzR7aJubqL


Generated by Claude Code

build_snapshot admitted one asyncio Task per skill to gather() and bounded
only how many ran, via a semaphore inside the worker. Live Tasks therefore
scaled with the registry rather than with the concurrency limit.

Reproduced independently at the reporter's parameters and at the sweep's real
ceiling of --max-scans 12000:

    n=512    gather-all   tasks_alive=513    peak_active=8      707 KiB
    n=512    worker-pool  tasks_alive=9      peak_active=8       26 KiB
    n=12000  gather-all   tasks_alive=12001  peak_active=8   16,579 KiB
    n=12000  worker-pool  tasks_alive=9      peak_active=8      477 KiB

So ~16 MiB of Tasks to keep 8 of them busy. Not a correctness or security
problem -- the concurrency limit worked as documented, and no extra load ever
reached the registry -- but it is waste that grows with the registry, and the
registry is the thing that grows.

run_bounded inverts the structure: a fixed pool pulls from a queue, so live
Tasks equal the limit. Applied to all three sites (scan, escalate, enrich),
not just the one reported; the escalation and enrichment phases had the same
shape.

On failure the pool is cancelled rather than left running, so a sweep that is
already failing stops hitting the registry.

The tests were checked against the old implementation and fail there. That
matters here: peak concurrency was already correct before this change, so a
test asserting only the limit would have passed against the bug.

Reported by @Nievesjyl.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01DNoTXU8k3pfSBzR7aJubqL
@Ap6pack
Ap6pack merged commit f9bf35f into main Sep 11, 2026
5 checks passed
@Ap6pack
Ap6pack deleted the claude/fix-57-bounded-tasks branch September 11, 2026 22:24
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

build_snapshot creates input-proportional pending tasks despite bounded scan concurrency

2 participants