Repository navigation
fix(collections): refresh smart collection counts after a scan - #3707
Conversation
A server smart collection's stored count, which unrestricted viewers see on its card, only refreshed at startup, hourly and when an admin saved the collection. After a scan added or removed titles it stayed stale for up to an hour while the collection itself listed the new members. The node that runs a scan now observes its own scan_complete publish and, about 10 seconds later, recounts the smart collections whose rule draws from that library (or from every library). Observing the publish rather than subscribing counts each scan once on a multi-node deployment and still works without Redis. ObservedEventBus now keeps a list of observers so this does not replace the plugin event dispatcher's. Closes Silo-Server#3697
This comment has been minimized.
This comment has been minimized.
|
Warning Review limit reachedThis organization requires a seat for paid usage-based reviews, including automatic and on-demand reviews. Ask an admin to assign a seat or enable paid usage-based reviews for users without a seat in Billing. Next included review available in 25 minutes. View limit details
No actionable comments were generated in the recent review. Walkthrough
Merge Risk:
|
Quick104
left a comment
There was a problem hiding this comment.
🤖 Claude Code (claude-opus-5-5)
Review of 850c8d2. The approach holds: observing the publish counts each scan once on the node that ran it, with or without Redis, and the scheduler's coalescing and re-arm logic is sound. There are two findings on the lines below. I'm fixing both in a commit on this branch, so there's no need to change anything yourself.
Maintainer decision: #3547, which is also open, removes cache.ObservedEventBus, and this PR's scan hook is built on it. Whichever of the two lands second needs rework. The maintainers will settle the order; nothing here needs to change for it.
A write to a library_collections row advances the collection's revision, which ends open catalog cursors and fails editor saves. Recounting after every scan made that happen every few seconds during an import, even when the count was unchanged. Write the count only when it differs. The DB test no longer fails on another test's uncountable smart collection: CI runs the DB contract packages in parallel on one database, and the refresh recounts every collection that names no library. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Silo Kody — review completeReview finished. Check the inline comments for findings and verify each suggestion against the code and tests. Reviewing changes in Silo
Review settingsReview OptionsThe following review options are enabled or disabled:
|
|
🤖 Claude Code (claude-opus-5-5) Review summaryReviewed head: db4ec3a (850c8d2 plus a review fix commit and a doc note). Addressed
Left for the maintainers
Checks
Fit: 9/10 · Readiness: 8/10 — conditional (assuming CI is green) Fit: it fixes a wrong number on a core 1.0 surface at the point where scans already announce changes, and it adds no setting or API surface. Readiness: the fix is sound and its tests hold. The open item is the merge order with #3547, plus the absence of a test that fails on main for the stale count itself. |
| scheduler.LibraryChanged(libraryID) | ||
| } | ||
| }) | ||
| go scheduler.Run(appCtx) |
There was a problem hiding this comment.
Concurrent refreshes can overwrite a post-scan count with an older result
The new scheduler runs concurrently with the hourly and save-triggered refreshes, so an older preview can finish its update after a post-scan preview and restore the stale count for another hour. Serialize refreshes across nodes or make the update conditional on the collection definition and a monotonic refresh generation.
Prompt for LLM
File internal/api/router.go, line 2309:
Concurrent refreshes can overwrite a post-scan count with an older result
The new scheduler runs concurrently with the hourly and save-triggered refreshes, so an older preview can finish its update after a post-scan preview and restore the stale count for another hour. Serialize refreshes across nodes or make the update conditional on the collection definition and a monotonic refresh generation.
Talk to Kody by mentioning @kody
Was this suggestion helpful? React with 👍 or 👎 to help Kody learn from this interaction.
There was a problem hiding this comment.
🤖 Claude Code (claude-opus-5-5)
The ordering is possible, but the window is narrow and it doesn't regress anything. The scheduler recounts at least 10 seconds after scan_complete. For an older result to land after it, a startup or hourly count of that same collection has to take its snapshot before the scan's writes commit and still be running 10 seconds after the publish. Each count runs sequentially with a 30-second timeout, so that takes an unusually slow count query. When it does happen, the collection shows the count it would have shown on main, and the next hourly pass corrects it. The same ordering already exists between the hourly pass and the recount on save, with no scan involved, so a fix belongs in the refresher for every writer, not in this PR. Leaving this open for the maintainers.
There was a problem hiding this comment.
The race is real, but I agree that fixing every count writer can reasonably be a follow-up, if maintainers accept the residual risk.
The 10-second delay narrows the window for a single scan, but it is not a general guarantee that an older recount must run for 10 seconds after a scan finishes. The scheduler batches from the first notification: a second scan can finish just before the batch runs, leaving little delay between that scan and its recount. Recounts from schedulers on different nodes can also overlap.
The current update in internal/catalog/smart_count_refresher.go checks whether the count differs, not whether the result is newer or was computed from the current definition. So an older result can overwrite a newer one. This does not make the count staler than it would have been on main in that scenario, but it does qualify the PR’s claim that the count follows a scan after roughly 10 seconds.
I’d leave the suggestion open for the maintainers’ decision. If deferred, the PR should document the concurrency exception and track a fix covering all writers; a deterministic test that forces the older result to write last would help validate that fix.
There was a problem hiding this comment.
Agreed: a scan that finishes just before a batch fires, or schedulers on two nodes, can narrow the gap. db4ec3a adds the exception to docs/architecture/collection-posters.md: recounts are unordered, an older one can finish last, and the next recount corrects it. Whether to make every count writer safe, and whether to track that separately, is for the maintainers to decide, so I'm leaving this thread open.
There was a problem hiding this comment.
That sounds reasonable. The documented exception makes the remaining risk clear, and leaving the thread open gives maintainers a chance to decide whether to address or track it separately.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Silo Kody — review completeReview finished. Check the inline comments for findings and verify each suggestion against the code and tests. Reviewing changes in Silo
Review settingsReview OptionsThe following review options are enabled or disabled:
|
Problem
Closes #3697
Related issue: #192
Validation tasks: unblocks #193 S4; unblocks the #3697 part of #194 A2 (still blocked by #3694, #3695, #3696); unblocks #195 W4; unblocks Silo-Server/silo-apple#20 C4, Silo-Server/silo-apple#21 C4, Silo-Server/silo-android#5 C4, Silo-Server/silo-android#6 C4
After a library scan adds or removes titles, a server smart collection's card keeps its old title count for up to an hour for admins and other accounts with no library or rating limits. The collection itself already lists the new titles. The stale number shows on the Collections page, the library Collections tab and the admin list, and the apps and Jellyfin-compatible clients read the same value.
Unrestricted viewers get the stored
library_collections.item_count_cached.SmartCountRefresheronly recounted it shortly after startup, hourly, and when an admin saved the collection. Nothing recounted it when a scan changed a library. With this change the stored count follows the scan about 10 seconds after it finishes.Approach
Scans already publish
scan_completewith the library ID on the catalog channel (the ingest executor, and the item and library refresh jobs). The node that publishes it now also hands it to a newcatalog.SmartCountScheduler. After a 10 second delay, the scheduler recounts the smart collections whose rule draws from a changed library, plus those whose rule draws from every library. Libraries that change during the delay are recounted in the same pass, and a library that changes during a recount is recounted again afterwards.I used the publish-side observer on
cache.ObservedEventBusrather than a subscription for two reasons:ObservedEventBusheld a single observer that a second registration replaced, and the plugin event dispatcher already uses it. It now keeps a list, so the dispatcher and the count refresh both see every publish.The recount itself is the existing per-collection query. Its cost per scan is one count per affected smart collection, not one per file.
Jellyfin-compatible clients get the fix with no extra change, because a BoxSet's
ChildCountreads the same stored count. The API contract is unchanged, sosilo-appleandsilo-androidneed no follow-up. The user manual doesn't describe when the count refreshes, so it needs no change. I updated the two architecture docs that described the hourly-only refresh and the single observer.Validation
internal/catalogandinternal/cachepass, including the new scheduler and observer cases.scripts/ci/db-contracts.txt) andTestServerCollectionCountsFollowTheViewerDBpass.go veton the touched packages,gofmtandmake verify-local-pathsare clean.make lint-changedand the full gate were left to CI. The full local lint is too heavy for my machine.Evidence
Evidence: https://evidence.siloserver.org/r/silo-server/pr-3707/
Surface and build: admin collections list (
GET /api/v1/admin/collections) next to a live preview of the same rule. Before is upstream main4018816da, after is this branch, on the same local database and collection. On main, after two titles were added and the scan finished, the stored count stayed at 1 while the rule matched 3, and nothing else recounts it before the hourly refresh. On this branch the stored count matched the rule about 10 seconds after each scan, both when a title was added (3 to 4) and when one was removed (4 to 3).Risks
internal/catalog/smart_count_refresher.go. My edit there is small and additive (aRefreshLibrariesentry point that filters the existing loop), so it should rebase cleanly either way.scan_complete;library_idsonly, while the collection's member list also intersects them with the collection's own libraries, so the two can differ for a collection whose rule names other libraries.Checklist
AI Disclosure
claude-opus-5-5-race. It found no code defects. It flagged two architecture docs that still described the hourly-only count and a single observer, and both are fixed here.Note
Refresh smart collection counts after library scans
SmartCountScheduler, which batches changed library IDs and triggersSmartCountRefresher.RefreshLibrariesafter a 10-second delay, so scans no longer wait up to an hour for counts to update. See smart_count_scheduler.goscan_completeevents when the event bus supports observation. See router.gocache.ObservedEventBusnow supports multiple observers invoked in registration order, instead of only the latest one. See observed_bus.goMacroscope summarized db4ec3a.