Proposal to leverage FTS into Nextcloud

[madbob]

Recently, I had the chance to tinker with Nextcloud's FTS system, and the whole thing felt a bit overly convoluted. The impression I got is that it was designed almost exclusively with Files in mind, though that doesn't quite justify the architectural split into Providers and Platforms (which hints at the original intention of extending this mechanism to other types of content as well).

I'd like to propose a few changes - some fairly substantial - to the API, aimed at making it simpler and more expressive at the same time. Why does this matter? Because search capabilities are notoriously a weak spot in Nextcloud, and while introducing a third-party indexing service makes deployment more complex, it would deliver much better results in enterprise environments, where running an indexer shouldn't be much of an issue.

The notes below stem from analyzing the files_fulltextsearch and fulltextsearch_elasticsearch packages, which are the only two "official" Nextcloud packages hooked into the FTS subsystem.

• The payload for each indexed document is accompanied by several attributes, most of which are obscure. Nowhere is the (theoretical) difference between tags, subtags, metatags, parts, and more documented. You can only guess that info is used to exchange volatile data (between Provider and Platform, and/or between Provider and client). The role of these data points during search also seems undefined: fulltextsearch_elasticsearch handles some of them with AND logic and others with OR, for no apparent reason
• The *WildcardField() and *RegexFilter() functions in ISearchRequest seem redundant given the existence of *SimpleQuery(), which lets you specify search criteria in a fairly granular way (even though, again in fulltextsearch_elasticsearch, those search conditions are treated strictly with OR logic)
• There is no way for a Provider to explicitly request specific data attached to a document from the Platform. fulltextsearch_elasticsearch returns some fields but omits others. While it makes sense not to return everything every time, it would be equally reasonable to leverage data already stored in the indexer to speed up building the final result set. PR feat: specify portions of document to be passed from platform to provider #960 already targets this issue
• Overall, the FTS system is completely isolated from applications. Many apps implement their own internal search mechanisms, exposing them via the OCP\Search\IFilteringProvider interface to feed into Unified Search - effectively competing with FTS itself

In short, here is what I would propose:

• In the IIndexDocument model (and consequently elsewhere), drop at least subtags, parts, and more to simplify the interface
• Use ISearchRequestSimpleQuery as the sole mechanism for building queries. By adding functions like addAnd(ISearchRequestSimpleQuery $and) and addOr(ISearchRequestSimpleQuery $or), multiple ISearchRequestSimpleQuery instances could be nested to define complex queries that any given Platform can execute unambiguously
• In IFullTextSearchProvider, introduce a function allowing the Provider to declare what type of content it handles. The same type identifier could be used by individual apps to trigger a matching process when registering different applications. These identifiers could be arbitrary strings: for example, the Mail app declares it handles "mail" content, and the relevant FTS Providers match it by declaring their type as "mail"
• Align the OCP\Search\IFilter interfaces with OCP\FullTextSearch\Model\ISearchTemplate, and more broadly transparently route an app-generated OCP\Search\ISearchQuery either to the internal search engine (serving as a basic, essential fallback) or to a compatible IFullTextSearchProvider. The goal here is to consolidate Unified Search, Full Text Search, and app-internal search. This is likely the most complex part, as it involves breaking changes across several existing apps

I've almost certainly missed a few factors in my analysis, but I'd love to discuss this with the community and developers, as I believe making better use of FTS within the Nextcloud ecosystem is crucial.