Skip to content

Commit 03bfc83

Browse files
melvincarvalhomelvincarvalho
committed
search plugin: full-text search over pod resources
TF-IDF ranking (smoothed idf, sublinear length-normalised tf), AND semantics, phrase queries, snippet windows. Text from JSON-LD text props (by local name), HTML (tag-strip), plain. Bounded recursive loopback crawl forwarding the caller's auth -> WAC by construction. CORS. Finding: THIRD and sharpest api.events consumer — after notifications (late) and sparql (wrong results), search is where staleness is most user-visible (freshness is the property users most expect of search). Read-time (default) is correct+fresh+O(N); cached-index mode (config.index) is documented unsafe without a write hook (stale/leak), patches output-WAC via per-hit HEAD recheck but can't fix content staleness. 11/11.
1 parent 1bc41ed commit 03bfc83

3 files changed

Lines changed: 1026 additions & 0 deletions

File tree

‎search/README.md‎

Lines changed: 182 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,182 @@
1+
# search/ — full-text search over a pod
2+
3+
A new feature built straight onto the #206 loader plugin api: full-text
4+
search over the resources under a pod container.
5+
6+
```
7+
GET /search?q=<terms>&container=/path/
8+
→ { "query": "...", "hits": [ { "url", "title", "snippet", "score } ] }
9+
```
10+
11+
The container subtree is walked over **loopback HTTP** carrying the caller's
12+
own `Authorization` header (the pattern notifications/, sparql/, rss/ and
13+
webdav/ established), each member's text is extracted, tokenised, and ranked
14+
by TF-IDF over the query terms. No npm dependencies: the tokeniser, the
15+
text extractor, and the ranker are hand-rolled on node builtins.
16+
17+
## Usage
18+
19+
```js
20+
plugins: [{
21+
module: 'search/plugin.js',
22+
prefix: '/search',
23+
config: {
24+
baseUrl: 'http://localhost:3000', // required: the server's public origin
25+
// loopbackUrl: 'http://127.0.0.1:3000', // where the plugin reaches its host
26+
// defaultContainer: '/alice/', // scope when the request names none
27+
// maxHits: 20, // result cap
28+
// maxResources: 200, // loopback fetches per crawl
29+
// maxDepth: 3, // container recursion below the scope
30+
// snippetRadius: 80, // chars either side of the first match
31+
// index: false, // cached-index mode (see Findings)
32+
// podsRoot: './data', // OPTIONAL fs.watch refresh (index mode)
33+
// reindexMs: 60000, // cached-index TTL
34+
},
35+
}]
36+
```
37+
38+
```bash
39+
curl "http://localhost:3000/search?q=fishing%20boat&container=/alice/notes/" \
40+
-H "Authorization: Bearer $TOKEN"
41+
```
42+
43+
```json
44+
{
45+
"query": "fishing boat",
46+
"hits": [
47+
{ "url": "http://localhost:3000/alice/notes/boats.jsonld",
48+
"title": "The old fishing boat",
49+
"snippet": "An old fishing boat rests in the harbor. The fishing crew …",
50+
"score": 0.7421 }
51+
]
52+
}
53+
```
54+
55+
**Scope** — the caller names which container subtree to search:
56+
`?container=/path/` (this server's origin) or `config.defaultContainer`. The
57+
plugin walks it recursively over loopback: containers via their `ldp:contains`
58+
listing, leaf members fetched and scanned (`.acl`/`.meta` sidecars and
59+
non-text bodies skipped), bounded by `maxDepth`/`maxResources`. When the walk
60+
is cut short the response carries `X-Search-Scope-Truncated: true`;
61+
`X-Search-Scope-Resources` always counts the resources visited, and
62+
`X-Search-Mode` reports `read-time` or `cached-index`.
63+
64+
**Authorization** — every loopback fetch carries the caller's own
65+
`Authorization`, so the searchable set is exactly what the caller could `GET`
66+
themselves. Anonymous callers search only public data; WAC can never disagree
67+
with the server, because the server itself answers every fetch.
68+
69+
**CORS** — the endpoint sets `Access-Control-Allow-Origin: *` and answers the
70+
`OPTIONS` preflight (allowing the `Authorization` header), so a browser search
71+
UI on another origin can call it. A bearer token is an explicit request
72+
header, not a cookie, so the wildcard origin is safe here.
73+
74+
## Ranking
75+
76+
Deliberately small and exactly this:
77+
78+
- **Tokenising** — Unicode-aware, lowercased, split on non-alphanumerics.
79+
`?q=` is parsed into terms: a `"quoted run"` is a **phrase** term (matched
80+
as a normalised-whitespace substring), bare words are single terms.
81+
- **AND semantics** — a document is a hit only when it contains **every**
82+
query term; a term absent from a doc drops it.
83+
- **TF-IDF** — per term, `idf = ln((N+1)/(df+1)) + 1` (smoothed, always
84+
positive) over the `N` crawled docs; per doc, a sublinear, length-normalised
85+
term frequency `(1 + ln(tf)) / sqrt(docLength)`. Score is the sum across
86+
terms. Hits sort by score descending (ties broken by URL for stability).
87+
- **Snippet** — a window of `snippetRadius` chars either side of the first
88+
matching term in the original text, whitespace-collapsed, elided with `…`.
89+
90+
**Text extraction** — from JSON/JSON-LD, the text-ish props matched by *local
91+
name* so any vocabulary works (`schema:`, `dcterms:`, a bare term): `title`,
92+
`name`, `headline`, `label`, `content`, `articleBody`, `text`, `description`,
93+
`body`, `summary`, `abstract` (arrays, `@value`, `@graph` and one level of
94+
nested objects unwrapped); title from the first title-ish of those. From
95+
`text/html`, tags are crudely stripped (`<script>`/`<style>`/comments dropped
96+
first, a few entities decoded) and the title taken from `<title>`. From
97+
`text/plain` (and markdown/xml) the body *is* the text; the title falls back
98+
to the filename.
99+
100+
## Read-time vs cached-index
101+
102+
| | **read-time** (default) | **cached-index** (`config.index: true`) |
103+
|---|---|---|
104+
| how | walk + scan the container per query | in-memory inverted index (persisted under `pluginDir`), rebuilt on TTL / `fs.watch` |
105+
| correctness | always fresh, WAC-correct by construction | **can be stale or wrong** — see Findings |
106+
| cost | O(N) loopback fetches per query | O(1) after the first crawl, + a per-hit HEAD recheck |
107+
| default | ✅ (deterministic; what the tests exercise) | off |
108+
109+
Cached mode restores WAC on its *output* with a per-hit loopback `HEAD`
110+
against the current caller (the index is a cross-agent store, so it can't be
111+
WAC-correct by itself). Staleness of the *content* it cannot fix — that is
112+
the finding below.
113+
114+
## Test
115+
116+
```bash
117+
node --test --test-concurrency=1 search/test.js
118+
```
119+
120+
Boots a real JSS from npm with `idp: true`, registers two pods, seeds
121+
`/alice/notes/` with JSON-LD articles, a plaintext file and an HTML page over
122+
authenticated `PUT`, then exercises: relevance ranking (the densest doc ranks
123+
first) and snippets, single-term recall, a non-matching term (empty), phrase
124+
queries (adjacent-only), multi-term AND, HTML title + tag stripping (script
125+
text not indexed), the WAC property (owner sees her notes; anonymous and a
126+
second registered agent see nothing), the `400`s and CORS preflight, and
127+
**cached-index mode on the same server** (a second plugin entry at `/isearch`
128+
with `index: true`) proving cached ranking matches read-time and does not leak
129+
private docs across agents. **11 tests.** Both index modes run on ONE server
130+
(a second entry, not a second boot) to sidestep the process-global
131+
`DATA_ROOT` footgun; the misconfiguration test is ordered first for the same
132+
reason.
133+
134+
## Findings
135+
136+
1. **Search is the THIRD — and sharpest — consumer of the missing
137+
`api.events.onResourceChange` seam.** After notifications/ (late
138+
notifications) and sparql/ (wrong query results), search makes the seam
139+
*most user-visible*: freshness is the one property users expect of search
140+
above all else, and it is exactly what a plugin cannot guarantee. The
141+
read-time path is correct because it re-crawls every query; the cached
142+
index — the mode users would actually turn on for a large pod — goes
143+
**stale** (a write the OS coalesces past `fs.watch`, or a non-filesystem
144+
storage backend the watcher can't see at all) or **wrong** (a shared
145+
in-memory index holds one agent's text and would serve it to another). See
146+
[sparql/'s finding 1](../sparql/README.md#findings) — this is its
147+
write-index sibling; the two ports together make the seam's rank in
148+
[NOTES §2](../NOTES.md) empirical, not speculative: two independent
149+
read-over-pod features both hit the same wall, and search is the one whose
150+
failure a user notices first.
151+
2. **`fs.watch` on a config-supplied `podsRoot` is a workaround, not a
152+
fix** — the same one notifications/ uses. It can drop or coalesce events,
153+
fires only for the filesystem backend, and gives no per-resource ACL
154+
context, so cache invalidation is guesswork. This plugin therefore keeps
155+
read-time as the default and treats the watcher as *optional* best-effort
156+
refresh on top of a TTL; the honest answer to "is my search fresh?" is
157+
"only in read-time mode."
158+
3. **A cross-agent cache cannot be WAC-correct without help.** The index is
159+
built under whichever caller first triggered it, so its contents span
160+
whatever *that* agent could read. Serving it to a second agent would leak
161+
private text. The plugin patches the *output* with a per-hit loopback
162+
`HEAD` recheck against the current caller — correct, but it re-introduces
163+
per-request HTTP (partly defeating the cache) and still leaves the *stored*
164+
text as a cross-tenant blob. A faithful cached search wants either
165+
per-agent indexes keyed by identity, or `api.events` carrying the ACL
166+
context to invalidate precisely — neither is expressible today. Read-time's
167+
forward-the-Authorization crawl gets this right for free (same property
168+
sparql/ and rss/ rely on).
169+
4. **`config.baseUrl` again** — same `api.serverInfo` finding as
170+
notifications/sparql/rss: the plugin needs its host's public origin (the
171+
`?container=` origin check, the loopback target, absolute hit URLs) and
172+
must be told it in config; the test does the probe-port-then-boot dance.
173+
`config.serverInfo` repetition now spans a dozen ports (see
174+
[NOTES §3](../NOTES.md)).
175+
5. **Process-global `DATA_ROOT` bites multi-boot tests** (host-side quirk,
176+
not a plugin-api gap): the misconfiguration test boots a second, failing
177+
`createServer`, which repoints the env var every `createServer` shares, so
178+
it must be ordered *before* the long-lived boot or the main server's token
179+
verification silently reads the wrong keys directory (this port learned it
180+
the hard way — a `before()` hook runs first and poisoned every authed
181+
query until the boot was moved into an ordered `it`). Both index modes are
182+
proven on one server precisely to avoid a second long-lived boot.

0 commit comments

Comments
 (0)