Fix handling of storage that has multiple shared mounts - #271
Fix handling of storage that has multiple shared mounts#271christoph-bessei wants to merge 2 commits into
Conversation
bd9f527 to
546a21e
Compare
546a21e to
0c91783
Compare
|
Confirming this on a large instance, in case another data point helps it along. We hit exactly the symptom from #254 on Nextcloud 34.0.3 with context_chat 5.4.0 and context_chat_backend 5.4.1: roughly 1.5 M eligible files, heavy use of shares and group folders, and a steady stream of on What made it costly for us is what happens next to those entries. if ($queueMapper->lock($document->getId())) {
try {
$files[$document->getId()] = $this->getFileSource(...);
} catch (\Exception $e) {
$this->logger->warning($e->getMessage(), ['exception' => $e]);
$queueMapper->delete($document);
}
}so the queue drained quickly while almost nothing reached the vector database — from the outside it looked like indexing had stopped, when in fact files were being consumed and discarded. Because the warning goes to the Nextcloud log rather than the backend container, it is easy to spend a long time looking in the wrong place. With this patch applied, indexing ran to 32,879 documents and 490,387 chunks before we paused it for unrelated reasons, and the remaining per-file failures are the ordinary ones — One note on Thanks for writing it — happy to test further revisions on an instance of this size if that is useful. |
|
Hello there, We hope that the review process is going smooth and is helpful for you. We want to ensure your pull request is reviewed to your satisfaction. If you have a moment, our community management team would very much appreciate your feedback on your experience with this PR review process. Your feedback is valuable to us as we continuously strive to improve our community developer experience. Please take a moment to complete our short survey by clicking on the following link: https://cloud.nextcloud.com/apps/forms/s/i9Ago4EQRZ7TWxjfmeEpPkf6 Thank you for contributing to Nextcloud and we hope to hear from you soon! (If you believe you should not receive this message, you can add yourself to the blocklist.) |
…le shared mounts Signed-off-by: Christoph Bessei <28066477+christoph-bessei@users.noreply.github.com>
…le shared mounts Signed-off-by: Christoph Bessei <28066477+christoph-bessei@users.noreply.github.com>
0c91783 to
391b4f1
Compare
Fixes #254
Description
As described in #254 there are some edge cases where
$mounts[0]->getUser()->getUID()returns a user ID that has no access to a given file. This PR fixes this by iterating through the mounts until it finds a user with access to the given file.There might be cleaner solutions, but I wanted to reduce the changes as much as possible for an easier review. Assuming
$mounts[0]is working in most cases, the performance impact should be minimal.Notes
🤖 AI (if applicable)